Since about an hour before the transition from daylight savings time back to standard time, I’ve noticed some events showing up in the incident manager that make it appear as though my agents are time traveling.
The events are showing up as metric evaluation errors on database instance and agent targets stating, for example: “Row(8): Supplied date is in the future : now = Mon Nov 05 08:42:41 EST 2012 supplied value = Mon Nov 05 09:38:42 EST 2012”. Sometimes the times reported are exactly one hour apart: “Row(1): Supplied date is in the future : now = Sun Nov 04 01:59:52 EDT 2012 supplied value = Sun Nov 04 01:59:52 EST 2012”. I see these from the repository database instance, several monitored database instances, and four different agents.
I’m seeing cases where a monitored database instance has this event and the monitoring agent does NOT, and I’m also seeing cases where both a monitored database and the agent performing the monitoring both show the event.
It seems that stopping the agent, running an emctl clearstate agent, then starting up the agent on each affected server will, after a few minutes, cause these events to clear. I only noticed this right when it started since I was logged in to take down SAP instances that can’t handle the time change, I wasn’t expecting to hit a related problem with EM12c.
emctl stop agent ; emctl clearstate agent ; emctl start agent
If you’re impatient, you can force collection of the EMDStatus metric collection to speed things up in clearing the events:
emctl control agent runCollection agenthost.domain.com:3872:oracle_emd EMDStatusCollection