R12.2, Service Manager is not start
Recently, I was working with a client when the managers in their 12.2 environment quit working. We had shutdown the applications tier to reboot the server, and when it came back up, only the Internal Manager (ICM) would run. After working through all the usual steps, relinking, autoconfig, cmclean, answering the same questions from Oracle repeatedly, etc., we still could not get the managers to start. The Administer Concurrent Managers form showed the Internal Manager running with a matching target (1), and the Service Manager running with a target of 0. This was the state every time we started the managers even after rebooting both the database and applications servers. To make things even stranger, if you viewed the processes under the Service Manager from the Administer Concurrent Managers form, all the processes were Terminated or Deactivated and no new entries were being made in spite of the parent form showing that one was running (If the managers were shutdown, the running count went to 0). Even starting the concurrent managers with DIAG=Y did not add any useful information to the logs for this issue.
I reached out to one of my contacts through OAUG, to see if he could think of something I had not tried. One of his suggestions was to run FND_CONC_CLONE.SETUP_CLEAN and then rerun AutoConfig. This did not solve the problem, but it did cause a new message in the ICM log, we now had a message that the apps tier could not ping. We tried ping from the command line and got the error, ping: imp open socket: operation not permitted. As a result of this, we discovered that /bin/ping had permissions and ownership:
-rwxrwxr-x root root
instead of the expected:
-rwsrwxr-x root root
As soon as we issued chmod u+s /bin/ping, the service manager (and therefore all the other managers) started. This occurred on a Redhat 6.4 server, and we do not know why the permissions changed on ping between the previous time we started the managers and the reboot. I also am not clear why the ping error did not show up every time we attempted to start the managers (we did find it in the first ICM log after the reboot, but it was not in any subsequent logs until after running doing the setup clean and autoconfig).
If you have an issue with the concurrent managers not starting, check the permissions, verify that you can run ping as the applications owner.