OIDMOS status stop and OID not starting up after crash
Our system has a system shutdown in which OID 11.1.1.9 was running and not starting up using opmnctl after server is brought up. A few things I went through to fix it up is posted below.
Check the status from the Oracle Instance home as below.
[apploid@oam01 bin]$ ./opmnctl status
Processes in Instance: asinst_1
———————————+——————–+———+———
ias-component | process-type | pid | status
———————————+——————–+———+———
ohs1 | OHS | 19283 | Alive
ovd1 | OVD | 19282 | Alive
oid1 | oidldapd | N/A | Down
oid1 | oidldapd | N/A | Down
oid1 | oidmon | 5562 | Stop
EMAGENT | EMAGENT | 19281 | Alive
You can see oidmon is stopped and oidldap is down. Normally opmnctl startall can bring up in almost all the cases, but I was getting time outs and stuck at below in the oidmon****.log file.
[2015-09-04T13:23:59.800201+10:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19273] [tid: 0] OIDMON_STOP: Thread started
[2015-09-04T13:23:59.807343+10:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19273] [tid: 0] OIDMON_STOP: Connecting to database, connect string is oiddb
I did check the database connectivity from the OID application server, checked the ODS schema status and almost everything including the status in ODS_PROCESS_STATUS (I will post about this table soon).
Since we had a abrupt shutdown, I guessed we have some uncleaned files in the directory below. Removed the files after taking the backup after shutting down everything, then try starting up.
$ORACLE_INSTANCE/bin/opmnctl stopall
cd $ORACLE_INSTANCE/config/OPMN/opmn/states
[apploid@oam01 states]$ ls
p1515733570
[apploid@oam01 states]$ cd ..
[apploid@oam01 opmn]$ mv states states_bak
[apploid@oam01 opmn]$ mkdir states
$ORACLE_INSTANCE/bin/opmnctl startall
Check the log file
[apploid@oam01 opmn]$ tail -f /u01/oid/Oracle/Middleware/asinst_1/diagnostics/logs/OID/oid1/oidmon-0000.log
[2015-09-04T13:33:43.438248+10:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19771] [tid: 0] Guardian: Starting OIDLDAPD cmdLine args = “instance=1 inst=1 “
[2015-09-04T13:33:43.446457+10:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19771] [tid: 0] Guardian: Starting OIDLDAP Server,PID=19818
[2015-09-04T13:33:43.449019+10:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19771] [tid: 0] Guardian: Updating Process Table…
[2015-09-04T13:33:43.452984+10:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19771] [tid: 0] Guardian: gsldda_DeleteEntry:gsldfgGetEntryID Error..1010
[2015-09-04T13:33:43.707887+10:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19771] [tid: 0] Guardian: CreateInstance: Successfuly created entry cn=asinst_1_oid1_1_oam01.saroj.com.au,cn=osdldapd,cn=subregistrysubentry
[2015-09-04T13:33:53.403166+10:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19771] [tid: 0] Guardian: gslmmonOpmnProcStatus: Sending OIDLDAPD PID=,19956 status to OPMN
[2015-09-04T13:33:55.405243+10:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19771] [tid: 0] Guardian: gslmmonOpmnProcStatus:Sending OID Host and ports info to OPMN
HOST=oam01.saroj.com.au LDAP NONSSL PORT = 3060 LDAP SSL PORT = 3131
[apploid@oam01 bin]$ ./opmnctl status
Processes in Instance: asinst_1
———————————+——————–+———+———
ias-component | process-type | pid | status
———————————+——————–+———+———
ohs1 | OHS | 19773 | Alive
ovd1 | OVD | 19772 | Alive
oid1 | oidldapd | 19956 | Alive
oid1 | oidldapd | 19818 | Alive
oid1 | oidmon | 19771 | Alive
EMAGENT | EMAGENT | 19770 | Alive