OTN Discussion Forums : How can we help to make EM12c better?

A recent post in the OTN forum for Enterprise Manager deserves some more attention: How can we help to make EM12c better?. Thanks to blzysh for having the chutzpah to toss this post out there. I’m personally rather pleased with EM12c, but I only make use of the database monitoring tools which have to be the most mature part of EM12c. Seems like the people monitoring middleware, provisioning, working with engineered systems and so on have a few more issues. The OTN thread looks like a great place to just air it all out.

I’m really impressed with all the Oracle employees monitoring the OTN forums. Very rapid responses to questions that don’t really require all the hassle of an SR and quick suggestions to go ahead and file an SR when something looks unusual. Thanks, everyone!

Advertisement

How To Setup Out Of Bound Email Notification In 12c [ID 1472854.1]

Huge thanks to Laurence on the MOS Community forum for posting a link to this document. On more than one occasion I’ve spent a half hour trolling through the EM12c documentation trying to find out how to set up OOB (shouldn’t it be out of BAND, rather than out of BOUND?) email notification for the EM12c repository. I was concerned the feature had been left out after making significant use of it in EM11g. It almost looks like Oracle is keeping this hidden so that only those of us with active MOS accounts can find it — can’t find it in the docs, and if you remember modifying emd.properties from your 11g days, you’ll notice that the parameters to set to enable this aren’t listed in the file. I’ve confirmed that this works as advertised in EM12c.

Followup: EM12cR2 reports Job Purge repository job as Down

This is a follow up to EM12cR2 reports Job Purge repository job as Down. Erik at the OTN Forum has provided a fix:

EM12cR2 – “Job Purge” repository scheduler job falsely(?) reported down .

Simply run the following update connected as user SYSMAN on your repository database, and the repository jobs will all show as clear and the open incident will be closed:

UPDATE MGMT_PERFORMANCE_NAMES SET display_name = NULL,
dbms_jobname = NULL, is_dbmsjob = 'N', is_deleted = 'Y'
WHERE UPPER(DBMS_JOBNAME) LIKE 'MGMT_JOB_ENGINE.APPLY_PURGE_POLICIES%';
COMMIT;

Patch 6895422 confirmed to fix bug 13729794 (TOO_MANY_OPEN_FILES) in EM12cR2

Update 20130619: While patch 6895422 is still available, Oracle has released patch 16087066, which contains the fix from 6895422 along with a fix for bug 13583799.  The general recommendation from Oracle now is to use patch 16087066.  Also, based on a comment in this MOS post, the 12.1.0.3 EM agent will include this patch and we will no longer need to apply it manually once the next version of the agent is released.

I’ve confirmed that patch 6895422 fixes the TOO_MANY_OPEN_FILES bug encountered with 12.1.0.2.0 agents in an Oracle Enterprise Manager Cloud Control 12c Release 2 environment.

Witness the following metric value graphs of unpatched and patched agents for the “number files open” metric over seven days:

BUG 13729794 still impacts Enterprise Manager 12cR2

Those of us with busy servers may have run into bug 13729794, causing OEM 12c agents to eventually stop monitoring after raising a TOO_MANY_OPEN_FILES error. On 12cR1, patch 6895422 took care of this problem. Oracle’s bug database reports this bug as “fixed in 12.1.0.2.0” but in status 75 “Closed, Code fix resolution not verified.”

I had two 12cR2 agents bomb out over the weekend (nothing like logging into MOS on a day off) with TOO_MANY_OPEN_FILES. The OMS wouldn’t recognize either of them after bouncing the agent. I had to run an emctl clearstate on each of them with the agent down, then start it back up, before the 12cR2 OMS would report the agents as reachable again.

So it seems 12cR2 does not fix this bug, at least not on Linux x86-64. I should know in about two days whether or not this patch still fixes the problem.

EM12cR2 reports Job Purge repository job as Down

Crossposted from the OTN Forum post. According to Oracle this is a bug and a script to resolve the reported problematic job will be forthcoming.

On the Repository page (Setup -> Manage Cloud Control -> Repository), in the Repository Scheduler Jobs Status section, the “Job Purge” job is shown as down with a red arrow and blanks for both Next Scheduled Run and Last Scheduled Run. All the other jobs look good and are running successfully.

This also appears as an incident stating “DBMS Job UpDown for Job Purge exceeded the critical threshold (DOWN). Current value: DOWN”.

The framework metric reference manual for this metric (link here) indicates that this metric equates to the DBMS_JOB ‘broken’ status and suggests retrieving the job_id and re-submitting it, but the provided instructions don’t find anything useful.

SQL> sho user
USER is "SYSMAN"
SQL> select dbms_jobname from mgmt_performance_names where display_name = 'Job Purge';

DBMS_JOBNAME
--------------------------------------------------------------------------------
MGMT_JOB_ENGINE.apply_purge_policies()

SQL> select job from all_jobs where what='MGMT_JOB_ENGINE.apply_purge_policies()';

no rows selected

SQL> select count(*) from all_jobs;

COUNT(*)
----------
0

No ORA-12012 messages found in the alert/trace logs.

Repvfy also reports this as a missing scheduler job:

2. Missing DBMS_SCHEDULER jobs
------------------------------

DISPLAY_NAME DBMS_JOBNAME
---------------------------------------- -----------------------------------------------------
Job Purge MGMT_JOB_ENGINE.apply_purge_policies()

The MGMT_JOB_ENGINE package no longer contains apply_purge_policies. It seems to have moved to EM_JOB_PURGE.

According to dba_scheduler_jobs, EM_JOB_PURGE.APPLY_PURGE_POLICIES is running just fine:

SQL> select job_action, enabled, state, failure_count, last_start_date, last_run_duration
2 from dba_scheduler_jobs where job_name = 'EM_JOB_PURGE_POLICIES';

JOB_ACTION ENABL STATE FAILURE_COUNT
---------------------------------------- ----- ---------- -------------
LAST_START_DATE
----------------------------------------
LAST_RUN_DURATION
----------------------------------------
EM_JOB_PURGE.APPLY_PURGE_POLICIES(); TRUE SCHEDULED 0
14-SEP-12 05.00.00.010318 AM US/EASTERN
+000000000 00:00:09.464552

It seems to me this is some kind of metadata inconsistency between mgmt_performance_names and the DBMS scheduler, and probably isn’t something to be concerned with. Any thoughts?

EM12cR2 increased redo logging on repository database

I’ve posted this on the OTN forum here, but perhaps it’s worth a blog post.

Has anyone else noticed increased redo logging in their repository database after an upgrade to EM12cR2?

I’m running on Linux x86-64 with an 11.2.0.3 repository database. Prior to the upgrade I was running 12.1.0.1.0 with BP1, and my repository database was producing, on average, about 80-120MB of archived redo logs per hour. Since the upgrade to 12.1.0.2.0, my repository database is producing, on average, 450-500MB of archived redo logs per hour.

Upgrading the deployed agents from 12.1.0.1.0 to 12.1.0.2.0 hasn’t seemed to help; I’ve upgraded about 10 of 15 without any apparent decrease in redo logging.

Just wondering if this is comparable to what others are seeing or if I have a local issue I should investigate.