root.sh fails Reporting 'srvctl start nodeapps -n <nodename> ... failed' as ons won't start
APPLIES TO:
Oracle Database - Enterprise Edition - Version 11.2.0.1 and later
SYMPTOMS:
New installation of 11.2 Grid Infrastructure, root.sh is fails, reporting to screen :
/u01/app/11.2.0.4/grid/bin/srvctl start nodeapps -n Node1 ... failed FirstNode configuration failed at /u01/app/11.2.0.4/grid/crs/install/crsconfig_lib.pm line 9379. /u01/app/11.2.0.4/grid/perl/bin/perl -I/u01/app/11.2.0.4/grid/perl/lib –I /u01/app/11.2.0.4/grid/crs/install /u01/app/11.2.0.4/grid/crs/install/rootcrs.pl execution failed
Rootcrs log file shows :
2015-01-26 15:26:11: nodes_to_start=hqi-ebsdb-1
2015-01-26 15:26:32: exit value of start nodeapps/vip is 1
2015-01-26 15:26:32: output for start nodeapps is PRCR-1013 : Failed to start resource ora.ons PRCR-1064 : Failed to start resource ora.ons on node Node1 CRS-5016: Process "/u01/app/11.2.0.4/grid/opmn/bin/onsctli" spawned by agent "/u01/app/11.2.0.4/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0.4/grid/log/Node1/agent/crsd/oraagent_oracle/oraagent_oracle.log" CRS-2674: Start of 'ora.ons' on 'Node1' failed
2015-01-26 15:26:32: output of startnodeapp after removing already started mesgs is PRCR-1013 : Failed to start resource ora.ons PRCR-1064 : Failed to start resource ora.ons on node Node1 CRS-5016: Process "/u01/app/11.2.0.4/grid/opmn/bin/onsctli" spawned by agent "/u01/app/11.2.0.4/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0.4/grid/log/Node1/agent/crsd/oraagent_oracle/oraagent_oracle.log" CRS-2674: Start of 'ora.ons' on 'Node1' failed
2015-01-26 15:26:32: /u01/app/11.2.0.4/grid/bin/srvctl start nodeapps -n Node1 ... failed
2015-01-26 15:26:32: Running as user oracle: /u01/app/11.2.0.4/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_NODECONFIG -state FAIL
2015-01-26 15:26:32: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/11.2.0.4/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_NODECONFIG -state FAIL '
2015-01-26 15:26:33: Removing file /tmp/fileoy7dDY
2015-01-26 15:26:33: Successfully removed file: /tmp/fileoy7dDY
2015-01-26 15:26:33: /bin/su successfully executed
ONS log file shows:
[2015-01-26T15:26:15+04:00] [ons] [NOTIFICATION:1] [104] [ons-internal] ONS server initiated
[2015-01-26T15:26:15+04:00] [ons] [ERROR:1] [17] [ons-listener] 172.0.0.1,6100: BIND (Cannot assign requested address)
CAUSE:
Incorrect value used for localhost in /etc/hosts
IP address 127.0.0.1 should only map to localhost and/or localhost.localdomain, not anything else
SOLUTION:
Edit localhost and ensure that entry for localhost is
127.0.0.1 localhost.localdomain localhost
How to Validate Network and Name Resolution Setup for the Clusterware and RAC (Doc ID 1054902.1)