Alert log and Trace Location centralized for Oracle 12.1.0.2 cluster environment :-
Troubleshooting a RAC issue is now made easier as we can easily locate all the required log/trace files under a centralized location. We need not jump around different directories to find the Clusterware related log files
From 12c,the Clusterware log/trace files are stored in the Automatic Diagnostic Repository (ADR) under the ADR_HOME location $ADR_BASE/diag/crs/`hostname`/crs.
#---// Finding ADR_HOME for Clusterware logs //---
myrac1 {/home/oracle}: adrci
ADRCI: Release 12.1.0.2.0 - Production on Wed Jan 13 04:01:00 2018
Copyright (c) 1982, 2014, Oracle and/or its affiliates. All rights reserved.
ADR base = "/app/oracle"
adrci> show homes
ADR Homes:
diag/tnslsnr/rac01/mgmtlsnr
diag/rdbms/_mgmtdb/-MGMTDB
diag/crs/rac1/crs
New centralized Clusterware log location to identify the related Clusterware log/trace files.
Oracle stores all the Clusterware diagnostic log/trace files under trace directory of Clusterware ADR_HOME.
#---// Locating Clusterware centralized log repository //----
rac1 {/home/oracle}: ls -ld $ADR_BASE/diag/crs/`hostname`/crs/trace
drwxrwxr-x 2 oracle oinstall 876544 Jan 13 04:02 /app/oracle/diag/crs/rac1/crs/trace
#---// moving to Clusterware centralized log repository //----
myrac01 {/home/oracle}: cd $ADR_BASE/diag/crs/`hostname`/crs/trace
Clusterware alert log file
#---// Clusterware alert log file //----
myrac01 {/app/oracle/diag/crs/rac1/crs/trace}: ls -lrt alert.log
-rw-rw---- 1 oracle oinstall 1024107 Jan 13 04:04 alert.log
Cluster Ready Services (CRS) daemon/agent trace files
#---// crs daemon and agent trace files //----
rac1 {/app/oracle/diag/crs/rac1/crs/trace}: ls -lrt crs* | tail -6
-rw-rw---- 1 root oinstall 1340371 Jan 13 04:03 crsd.trm
-rw-rw---- 1 root oinstall 7238701 Jan 13 04:03 crsd.trc
-rw-rw---- 1 oracle oinstall 739563 Jan 13 04:04 crsd_oraagent_oracle.trm
-rw-rw---- 1 oracle oinstall 4371314 Jan 13 04:04 crsd_oraagent_oracle.trc
-rw-rw---- 1 root oinstall 650733 Jan 13 04:04 crsd_orarootagent_root.trm
-rw-rw---- 1 root oinstall 3565651 Jan 13 04:04 crsd_orarootagent_root.trc
Cluster Synchronization Services (CSS) daemon trace files
#---// css daemon trace files //----
rac1 {/app/oracle/diag/crs/rac1/crs/trace}: ls -lrt ocssd* | tail -2
-rw-rw---- 1 oracle oinstall 443750 Jan 13 04:04 ocssd.trm
-rw-rw---- 1 oracle oinstall 2868877 Jan 13 04:04 ocssd.trc
High Availability Service (HAS) daemon/agent trace files
#---// has daemon and agent trace files //----
rac1 {/app/oracle/diag/crs/rac1/crs/trace}: ls -lrt ohasd* | tail -6
-rw-rw---- 1 root oinstall 783540 Jan 13 04:04 ohasd_orarootagent_root.trm
-rw-rw---- 1 root oinstall 4805435 Jan 13 04:04 ohasd_orarootagent_root.trc
-rw-rw---- 1 root oinstall 211308 Jan 13 04:04 ohasd.trm
-rw-rw---- 1 root oinstall 1225033 Jan 13 04:04 ohasd.trc
-rw-rw---- 1 oracle oinstall 1382492 Jan 13 04:04 ohasd_oraagent_oracle.trm
-rw-rw---- 1 oracle oinstall 6877368 Jan 13 04:04 ohasd_oraagent_oracle.trc
Event Management (EVM) daemon trace files.
#---// evm daemon trace files //----
rac1 {/app/oracle/diag/crs/rac1/crs/trace}: ls -lrt evm* | tail -2
-rw-rw---- 1 oracle oinstall 1562665 Jan 13 04:04 evmd.trm
-rw-rw---- 1 oracle oinstall 10052660 Jan 13 04:04 evmd.trc
Grid Interprocess Communication (GIPC) daemon trace files
#---//
#---// gipc daemon trace files //----
#---//
rac1 {/app/oracle/diag/crs/rac1/crs/trace}: ls -lrt gip* | tail -2
-rw-rw---- 1 oracle oinstall 763055 Jan 13 04:04 gipcd.trm
-rw-rw---- 1 oracle oinstall 5507597 Jan 13 04:04 gipcd.trc
Cluster System Monitor daemon trace files
#---//
#---// sysmon daemon trace files //----
#---//
rac1 {/app/oracle/diag/crs/rac1/crs/trace}: ls -lrt osysmon* | tail -2
-rw-rw---- 1 root oinstall 557926 Jan 13 03:46 osysmond.trm
-rw-rw---- 1 root oinstall 2867760 Jan 13 03:46 osysmond.trc
Cluster Logger Service daemon trace files
#---//
#---// logger daemon trace files //----
#---//
rac1 {/app/oracle/diag/crs/rac1/crs/trace}: ls -lrt ologgerd* | tail -2
-rw-rw---- 1 root oinstall 410752 Jan 13 03:48 ologgerd.trm
-rw-rw---- 1 root oinstall 2742627 Jan 13 03:48 ologgerd.trc
Apart from the Clusterware service (daemon) trace files, the diagnostic location also contains the client utility trace files as shown below
OCRCHECK utility trace files
#---//
#---// ocrcheck client trace files //----
#---//
rac1 {/app/oracle/diag/crs/rac1/crs/trace}: ls -lrt ocrcheck* | tail -2
-rw-rw---- 1 oracle oinstall 137 Jan 13 01:30 ocrcheck_31338.trm
-rw-rw---- 1 oracle oinstall 310 Jan 13 01:30 ocrcheck_31338.trc
OCRCONFIG utility trace files.
#---//
#---// ocrconfig client trace files //----
#---//
rac1 {/app/oracle/diag/crs/rac1/crs/trace}: ls -lrt ocrconfig* | tail -2
-rw-rw---- 1 oracle oinstall 137 Jan 13 01:30 ocrconfig_30866.trm
-rw-rw---- 1 oracle oinstall 312 Jan 13 01:30 ocrconfig_30866.trc
CRSCTL utility trace files.
#---//
#---// crsctl client trace files //----
#---//
rac1 {/app/oracle/diag/crs/rac1/crs/trace}: ls -lrt crsctl* | tail -2
-rw-rw---- 1 oracle oinstall 175 Jan 13 01:31 crsctl_1103.trm
-rw-rw---- 1 oracle oinstall 729 Jan 13 01:31 crsctl_1103.trc
Cluster Monitor utility (oclumon) trace files
#---//
#---// oclumon client trace files //----
#---//
rac1 {/app/oracle/diag/crs/rac1/crs/trace}: ls -lrt oclumon* | tail -2
-rw-rw---- 1 oracle oinstall 166 Dec 17 03:54 oclumon_2175.trm
-rw-rw---- 1 oracle oinstall 387 Dec 17 03:54 oclumon_2175.trc
Grid Plug and Play utility (GPNP) trace files
#---//
#---// gpnp client trace files //----
#---//
rac1 {/app/oracle/diag/crs/rac1/crs/trace}: ls -lrt gpn* | tail -2
-rw-rw---- 1 oracle oinstall 205 Jan 13 01:31 gpnptool_664.trm
-rw-rw---- 1 oracle oinstall 608 Jan 13 01:31 gpnptool_664.trc
Therefore,Clusterware trace/log files are placed under a single centralized log repository. From 12c Release 1,we can troubleshooting Clusterware related issues easily under one location.