In 3 node rac setup. In the process of root.sh script execution in first 2 nodes were completed successfully. But when we run the script in the 3'rd we faced the issue "Failed to start Nodeapps" .
#1 Error Detail:
Using configuration parameter file: /CRS/app/11.2.0/grid_new/crs/install/crsconfig_params
User ignored Prerequisites during installation
PRKO-2190 : VIP exists for node testdb6, VIP name testdb6-vip
/CRS/app/11.2.0/grid_new/bin/srvctl start nodeapps -n testdb6 ... failed
Failed to start Nodeapps at /CRS/app/11.2.0/grid_new/crs/install/crsconfig_lib.pm line 9217.
[NEW root@testdb6 ~]#
Cause analysis:
#1 Check the Logfile:
Logfile : /CRS/app/11.2.0/grid_new/cfgtoollogs/crsconfig/rootcrs_testdb6.log
2017-05-31 00:12:51: starting nodeapps...
2017-05-31 00:12:51: DHCP_flag=0
2017-05-31 00:12:51: nodes_to_start=testdb6
2017-05-31 00:15:05: exit value of start nodeapps/vip is 1
2017-05-31 00:15:05: output for start nodeapps is PRKO-2421 : Network resource is already started on node(s): testdb6 PRKO-2420 : VIP is already started on node(s): testdb6 PRCR-1013 : Failed to start resource ora.ons PRCR-1064 : Failed to start resource ora.ons on node testdb6 CRS-5014: Agent "/CRS/app/11.2.0/grid_new/bin/oraagent.bin" timed out
#2 Check Cluster Resource Status:
[NEW oracle@testdb6 crsconfig]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ons
ONLINE ONLINE testdb4
ONLINE ONLINE testdb5
OFFLINE OFFLINE testdb6 CHECK TIMED OUT
#3 Check the nodeapps status:
srvctl config nodeapps
Network exists: 1/10.80.1.0/255.255.255.0/eth0, type static
VIP exists: /testdb4-vip/10.80.1.72/10.80.1.0/255.255.255.0/eth0, hosting node testdb4
VIP exists: /testdb5-vip/10.80.1.65/10.80.1.0/255.255.255.0/eth0, hosting node testdb5
VIP exists: /testdb6-vip/10.80.1.165/10.80.1.0/255.255.255.0/eth0, hosting node testdb6
GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016
#4 Check the connection status:
[NEW oracle@testdb6 ~]$ /CRS/app/11.2.0/grid/bin/onsctli ping
Communication error with the OPMN server local port.
Check the OPMN log files
^Cons is not running ...
Resolve this issue:
#1 Check the ons process:
[NEW oracle@testdb6 ~]$ ps -ef | grep ons
root 2770 1 0 May22 ? 00:00:27 sendmail: accepting connections
root 2909 1 0 May22 hvc0 00:00:00 /sbin/agetty serial_console 9600 vt100-nav
oracle 18960 1 0 May26 ? 00:00:00 /CRS/app/11.2.0/grid/opmn/bin/ons -d
oracle 18961 18960 0 May26 ? 00:05:55 /CRS/app/11.2.0/grid/opmn/bin/ons -d
oracle 20786 20701 0 01:23 pts/0 00:00:00 grep ons
#2 Kill the process manually:
[NEW oracle@testdb6 bin]$ kill -9 18960
[NEW oracle@testdb6 bin]$ kill -9 18961
#3 Check the ons process:
[NEW oracle@testdb6 bin]$ ps -ef | grep ons
root 2770 1 0 May22 ? 00:00:27 sendmail: accepting connections
root 2909 1 0 May22 hvc0 00:00:00 /sbin/agetty serial_console 9600 vt100-nav
oracle 20823 20701 0 01:26 pts/0 00:00:00 grep ons
Note : no ons process is running.
#4 Start the Ons Process:
[NEW oracle@testdb6 bin]$ ./onsctl start
CRS-2672: Attempting to start 'ora.net1.network' on 'testdb6'
CRS-2676: Start of 'ora.net1.network' on 'testdb6' succeeded
CRS-2679: Attempting to clean 'ora.ons' on 'testdb6'
CRS-2681: Clean of 'ora.ons' on 'testdb6' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'testdb6'
CRS-2676: Start of 'ora.ons' on 'testdb6' succeeded
#5 Check the ons status:
[NEW oracle@testdb6 bin]$ ./onsctl ping
ons is running ...
[NEW oracle@testdb6 bin]$ exit
#6 Re-run the root script:
[NEW root@testdb6 crsconfig]# /CRS/app/11.2.0/grid_new/perl/bin/perl -I/CRS/app/11.2.0/grid_new/perl/lib -I/CRS/app/11.2.0/grid_new/crs/install /CRS/app/11.2.0/grid_new/crs/install/rootcrs.pl
Using configuration parameter file: /CRS/app/11.2.0/grid_new/crs/install/crsconfig_params
User ignored Prerequisites during installation
PRKO-2190 : VIP exists for node testdb6, VIP name testdb6-vip
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[NEW root@testdb6 crsconfig]#
#7 Check the resource status:
[NEW oracle@testdb6 crsconfig]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ons
ONLINE ONLINE testdb4
ONLINE ONLINE testdb5
ONLINE ONLINE testdb6
Reference:
In 11.2 and 12c, stopping ons fails. The rootupgrade.sh, "crsctl stop crs -f", and "srvctl stop nodeapps" fail because ons fails to stop. (Doc ID 2038830.1)