WLS Domain Validation failed during EBS 12.2 upgrade
In this particular case, one of the post installation check “WLS Domain Validation” failed as shown below .
WLS_Check_Failed
Click on Red mark button ( ! ) to check why WLS Validation failed. In our case, it was showing:
EBS Domain validation failed for RUN filesystem.
Check Domain deployment logs and re-deploy the RUN domain.
After checking the log file, we came to know that there is some issue with OHS instance as shown in below error messages.
Successfully updated context variable s_ohs_installed with the value generateEBSOHSConfigFiles
opmnctl startall: starting opmn and all managed processes…
EBS_web_PRD84 is failed to start !!
.end std out.
===================
opmn id=<hostname>:6285
Response: 0 of 1 processes started.
ias-instance id=EBS_web_PRD84_OHS1
+++++++++++++++++++++++
———————————
ias-component/process-type/process-set:
EBS_web_PRD84/OHS/OHS/
Error
–> Process (index=1,uid=1443244297,pid=10563)
failed to start a managed process after the maximum retry limit
Log:
/u01/oracle/ PRD84/ fs2/FMW_Home/ webtier/instances/ EBS_web_PRD84_OHS1/ diagnostics/ logs/OHS/EBS_web_PRD84/ console~OHS~1.log
.end err out.
ERROR: Unable to create OHS instance.
Failed to create the OHS instance
So we checked in the given OHS log file (/<base_directory>/fs2/FMW_Home/webtier/instances/<instance> /diagnostics /logs/OHS/<instance>/console~OHS~1.log) and got the exact error behind the issue that was “Address already in use“.
[2015-08-20T14:28:40.4043+01:00] [OHS] [INCIDENT_ERROR:32] [OHS-9999] [worker.c] [host_id: <Hostname>] [host_addr: <IP of Host>] [pid: 10527] [tid: 47435944011936] [user: <application_user>] [VirtualHost: main] (98)Address already in use: make_sock: could not bind to address [::]:8084
Root Cause:
As shown in the above log, 8084 port (OHS Port) is already being used by other application that’s why unable to bind to that address.Solution:
To check which process is using port 8084 (PID 2823 in our case)[root@ebstemp ~]# lsof | grep 8084 | grep LISTEN
httpd 2823 oracle84 18u IPv4 6925093 TCP *:8084 (LISTEN)
[root@ebstemp ~]# ps -ef | grep 2823
oracle84 2823 2745 0 Aug17 ? 00:00:00 /u01/applebs/EBSTEMP/apps/tech_st/10.1.3/Apache/Apache/bin/httpd -d /u01/applebs/EBSTEMP/apps/tech_st/10.1.3/Apache/Apache -DSSL -f /u01/applebs/EBSTEMP/inst/apps/EBSTEMP_ebstemp/ora/10.1.3/Apache/Apache/conf/httpd.conf
To kill the process, run command:
[root@ebstemp ~]# kill -9 2823 <PID>
The solution provided above resolved the issue (after this start services manually and click on Retry)