EM 13c: Host Target Remains in Down Status in Enterprise Manager 13c Cloud Control and/or Host Metrics do not Collect (No Data Available)
A host target (typically with an EM 13c Agent upgraded from 12c) either remains in DOWN status, or is UP but does not collect any metrics.
All the charts on the Host target's homepage show "No Data Available".
The EM Agent which is monitoring the host shows a target status of Up.
Database targets on this host show a target status of Up.
Stopping and starting the upgraded EM 13c Agent will return the Host target status to Up, but once the Host target is involved in a blackout, it is shown as Down when the blackout is over.
From the Host target's homepage, Host > Monitoring > Metric and Collection Settings
The Response metric has comparison operator > (greater than) instead of = (equals), putting the target in CRITICAL state despite being up.
Normally, a target CRITICAL status is determined by: if Response = 0 mark target as CRITICAL
where 0=DOWN and 1=UP
By having the wrong comparison operator, this becomes: if Response > 0 mark target as CRITICAL
effectively reversing the logic and putting the target in CRITICAL state when it's up.
Once the host target is in CRITICAL state, its metrics are no longer collected
To obtain the state of the target:
Symptoms
Enterprise Manager (EM) 13c Cloud Control (13.1 or 13.2)A host target (typically with an EM 13c Agent upgraded from 12c) either remains in DOWN status, or is UP but does not collect any metrics.
All the charts on the Host target's homepage show "No Data Available".
The EM Agent which is monitoring the host shows a target status of Up.
Database targets on this host show a target status of Up.
Stopping and starting the upgraded EM 13c Agent will return the Host target status to Up, but once the Host target is involved in a blackout, it is shown as Down when the blackout is over.
Changes
This issue is seen where an agent was upgraded from version 12c to 13c.Cause
Bug 23046988 - Host Target status showing downFrom the Host target's homepage, Host > Monitoring > Metric and Collection Settings
The Response metric has comparison operator > (greater than) instead of = (equals), putting the target in CRITICAL state despite being up.
Normally, a target CRITICAL status is determined by: if Response = 0 mark target as CRITICAL
where 0=DOWN and 1=UP
By having the wrong comparison operator, this becomes: if Response > 0 mark target as CRITICAL
effectively reversing the logic and putting the target in CRITICAL state when it's up.
Once the host target is in CRITICAL state, its metrics are no longer collected
To obtain the state of the target:
<agent_inst>/bin/emctl getmetric agent <hostname>,host,Response
[oracle@example ~]$ /u01/app/agent/agent_inst/bin/emctl getmetric agent example.domain.com,host,Response
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
Status
1
Status 1 means the target is UP. If this shows 0, something else is causing the host to show as Down, and this document is not applicable.Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
Status
1
<agent_inst>/bin/emctl status agent target <hostname>,host
[oracle@example ~]$ /u01/app/agent/agent_inst/bin/emctl status agent target example.domain.com,host
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Target Name : example.domain.com
Target Type : host
Current severity state
----------------------
Metric Column name Key State Timestamp
--------------------------------------------------------------------------------
DiskActivity DiskActivitybusy dm-0 CLEAR Fri Mar 03 18:54:42 MST 2017
...
Response Status n/a CRITICAL Fri Mar 03 19:07:08 MST 2017
It is this CRITICAL status that is causing the issue. If this says CLEAR, something else is causing the issue, and this document is not applicable.Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Target Name : example.domain.com
Target Type : host
Current severity state
----------------------
Metric Column name Key State Timestamp
--------------------------------------------------------------------------------
DiskActivity DiskActivitybusy dm-0 CLEAR Fri Mar 03 18:54:42 MST 2017
...
Response Status n/a CRITICAL Fri Mar 03 19:07:08 MST 2017
Solution
- Download the workaround monitoring template responsefix-host.zip from (https://support.oracle.com/epmos/main/downloadattachmentprocessor?parent=DOCUMENT&sourceId=2236697.1&attachid=2236697.1:RESPONSEFIXTEMPALTE&clickstream=yes)
- Import the template into EM Cloud Control as SYSMAN:
Enterprise > Monitoring > Monitoring Templates > Actions > Import - Apply the template to the problematic host target(s):
- Select the responsefix template > Apply > (x) Template will only override metrics that are common to both template and target
- Destination Targets > Add > the host(s)
- Finish - If the target does not come up, restart the agent.