Tag Archives: upgrade

Fix: Plugin error when upgrading EM13cR1 agent to EM13cR2 (13.2) on Windows 2008 R2 x64

I ran into the following issue while attempting to upgrade to the Oracle management agent on my one Windows (2008 R2, x64) server to the 13.2.0.0.0 version distributed with Oracle Enterprise Manager 13.2. The agent upgrade repeatedly failed in the “Upgrading Management Agent” step with an error message:


Exit Code :0
The version is 13.1.0.0.0
Checking for the version 13.2.0.0.0
The agent is not upgraded successfully
[...]
Plugins upgrade failed.
Plugin upgrade failed.
Check the file E:/agent13c/agent_inst/install/plugins_upgrade.txt.status on agent for plugin upgrade status.
Check latest E:/agent13c/agent_inst/install/logs/agentplugindeploy_.log
Plugins upgrade failed.0

The log file referenced in the error message contains some, but not much, additional information:


The command executed for discovery at _2016_10_12_11_48_28 is : E:\agent13c\agent_inst\bin\emctl_upgrade.bat set_discovery_root plugin oracle.sysman.si E:\agent13c\agent_13.2.0.0.0\plugins\oracle.sysman.si.discovery.plugin_13.2.1.0.0 13.2.1.0.0
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
return value is : 65280
Install case : set_discovery_root failure...existing on error value 65280

I fixed this issue by installing the Microsoft Visual C++ 2010 Service Pack 1 Redistributable Package MFC Security Update package on the server. Finally, create a new directory called “prerelogs” inside of the …\agent_13.2.0.0.0\cfgtoollogs\ directory, or else my deployment hung forever during the “Performing prerequisite checks” step. Note that “prerelogs” does not contain the letter q, do not use ‘prereqlogs’. The agent deployment uses this directory to extract software for running the prerequisite checks; possibly setting SCRATCHPATH or another extra parameter would resolve this issue without manually creating the directory, but this worked for me.

After installation of the VC++ 2010 package, which did not require a reboot, followed by creating of the prerelogs directory, I repeated the agent upgrade and it completed successfully.

This issue may or may not occur on other Windows versions, I have not heard any other reports of problems. The first release of EM13c did not seem to require this package.

My Production EM12c Upgrade From R2 (12.1.0.2) to R3 (12.1.0.3)

This post covers my production upgrade from EM12c R2 to EM12c R3 on Linux x86-64 (SLES11 SP2).  To stress test the upgrade and keep it interesting, this system also has BI Publisher integrated into OEM, and also has the plugin from NetApp (version 12.1.0.1.0), the VMware plugin from BlueMedora (version 12.1.0.5.0), and the MySQL plugin from Pythian (version 12.1.0.1.1).  The repository database is running on version 11.2.0.3.

I’m feeling lucky today.  So this is just going straight into production.  Famous last words…

Preparation

  1. Go to edelivery and search for the Oracle Enterprise Manager product pack and platform Linux x86-64
  2. Follow the link titled “Oracle Enterprise Manager Cloud Control 12c Release 3 (12.1.0.3) Media Pack for Linux x86-64
  3. Download the three files needed for EM12c R3: V38641-01.zip, V38642-01.zip, and V38643-01.zip
  4. Download patch 13349651 for WebLogic, you will need it during the post-upgrade steps
  5. View the digest and run md5sum against each downloaded file to confirm that the files downloaded correctly
  6. Transfer the EM12c R3 files to a staging area on your Oracle Enterprise Manager server and unzip all three of them
  7. Delete the three downloaded .zip files if you are short on space (but don’t just “rm *.zip” or you’ll remove the necessary WT.zip file at the top level of your staging directory)
  8. Review the upgrade guide
  9. Create backups of your current OMS home, agent home, software library and Oracle inventory
  10. Create a backup of your current repository database that you can restore from if necessary
  11. If you use a dedicated filesystem for your Oracle Management Agents, make sure that dedicated filesystem has enough free space.  I used a small, 2GB filesystem, and this was barely large enough, except for the one sandbox server where it was too small and I was not able to complete the agent upgrade without adding space
  12. Stop the BI Publisher WebLogic Server (if you have it installed) via the WebLogic Admin Console
  13. Make sure your repository database does not have snapshots created on any tables by running “select master, log_table from all_mview_logs where log_owner = ‘SYSMAN'” — if any snapshots are found, follow the instructions in the upgrade guide to drop them
  14. If your repository database is version 11.1.0.7 or 11.2.0.1, follow the steps in the upgrade guide to apply the prerequisite patches needed to proceed
  15. Copy the emkey from the existing OMS to the existing management repository by running “$OMS_HOME/bin/emctl config emkey -copy_to_repos” and enter the SYSMAN password when prompted
  16. Confirm the emkey was copied by running “$OMS_HOME/bin/emctl status emkey” and enter the SYSMAN password when prompted
  17. Stop every OMS in your environment by running “$OMS_HOME/bin/emctl stop oms -all”
  18. Stop the management agent monitoring the management services and repository target by running “$AGENT_HOME/bin/emctl stop agent”

Running The Upgrade

Due to some issues I had with the upgrade from EM12c R1 to R2, I highly recommend that you do NOT use Cygwin to ssh to your OMS host and display the installer over ssh using Cygwin’s X server.  Use VNC instead.  I’m not going to try Cygwin this time through.

First, I start vncserver on the OMS host.  Then I connect to it to using TightVNC from my desktop machine.

  1. Navigate to the staging directory where you unzipped the EM12c R3 distribution files and run “./runInstaller” as your oracle software owner
  2. As an SAP customer, we are not allowed to use OCM, so I skipped the steps involving entering my My Oracle Support credentialsStep 1
  3. I also skipped the search for updates as this is a new enough release at the moment there should not be any necessaryStep 2
  4. The prerequisite checks run.  I received a warning about libstdc++43-4.3 not being found, but libstdc++43-devel-4.3.3-x86_64 fulfills this need, so I click ignore and then Next to continueStep 3
  5. Only the one system upgrade is supported when upgrading from EM12cR2, so I selected “Upgrade an existing system”, “One system upgrade”, and my existing middleware home.  Click NextStep 4
  6. This out of place upgrade will go into a new middleware home.  I am using /oracle/oem/Middleware12cR3.  Click Next and the installer will confirm that you have enough free space availableStep 5
  7. Here I had to pause and request more space from my storage admin, as the installer wants at least 14.0GB of free space.  Once that was done, I proceeded
  8. Enter the SYS and SYSMAN passwords and check the box to confirm that you have backed up the repository (you should have your OMS, agent, etc all backed up as well), then click NextStep 6
  9. The installer will check various parameters on your repository database and offer the chance to fix them if any need to be changed.  I accept the fixes and click YesStep 6(b)
  10. The installer checks some additional settings and notes that they should be reviewed after the installation or fixed now.  I explicitly granted execute on DBMS_RANDOM to DBSNMP and then clicked OKStep 6(c)
  11. The installer lists the plugin versions that will change and the plugins that will migrate. Confirm this all looks right and then click NextStep 7
  12. The installer lists additional plugins you can choose to deploy at install time.  I do not use any of these so I left them all unchecked and clicked NextStep 8
  13. The installer requests the password for your WebLogic adminserver and confirmation of the hostname, port and username.  Provide the password and click Next.  You may be able to change the OMS instance directory here but I do not suggest doing soStep 9
  14. You now have a chance to review your settings, then click Install to proceed with the upgradeStep 10
  15. Installation proceedsStep 11
  16. You are then prompted to run the allroot.sh file.  Login to the server and execute it as root or via sudoallroot.sh
  17. Once the install/upgrade is complete, the installer will display an installation summary.  Review it, save the URLs it gives you for the OMS and adminserver, then click CloseUpgrade Summary Report

Overall, the upgrade installation steps took 1 hour and 15 minutes in my environment.  This was on a physical server with 126GB RAM, 16 dual core processors and 200 managed targets.  This does not include the post-upgrade steps shown below.

Post Upgrade Steps

Now that the upgrade is complete, return to the upgrade guide to complete post upgrade steps.  Your environment may differ from mine, but these are the steps I had to follow.

  1. Start your central agent by running “$AGENT_HOME/bin/emctl start agent”.  At this point the load on my system went up very high and began responding very slowly.  I walked away for 10 minutes to let things settle down
  2. Open your web browser and go to the URL provided at the end of the installation and login as SYSMAN.  When I first tried to do so using Firefox I received an error indicating an invalid certificate.  I had to delete the old certificates and authorities from my previous installation and restart Firefox before it would allow me in.  MSIE worked fine though
  3. Update the central agent (the management agent installed on the OMS host) by clicking on the Setup menu, then Manage Cloud Control, then Upgrade Agents.  Click the Add button and select your central agent.  I choose “Override preferred credentials” since I have not configured sudo.  Click Submit to continue, then OK when warned that you may have to run root.sh manuallyUpgrade Central Agent
  4. My first upgrade attempt on the central agent failed due to a prerequisite check for package libstdc++-43.  The easy thing to do here is expand the Additional Inputs region and provide “-ignorePrereqs” as an additional parameter, but I chose to complete this agent upgrade using emcli and describe that process in the next two steps
  5. First run “$OMS_HOME/bin/emcli login -username=sysman” and enter the SYSMAN password when prompted.  Then run “$OMS_HOME/bin/emcli sync”
  6. Upgrade the agent by running “$OMS_HOME/bin/emcli upgrade_agents -additional_parameters=”-ignorePrereqs” -agents=”hostname.domain.com:3872″
  7. Wait while the upgrade proceeds.  You can view upgrade progress by running “$OMS_HOME/bin/emcli get_agent_upgrade_status”, or in the GUI by clicking on “Agent Upgrade Results” in the “Upgrade Agents” pageAgent Upgrade Progress
  8. Click Done once the central agent upgrade completes.  Go to the new agent home and run root.sh as root or via sudo
  9. Then repeat this process for the rest of your agents.  Try to install them first WITHOUT using the “-ignorePrereqs” flag, because if there are missing prerequisites you need to identify the issue and find out if it is something that it is appropriate to ignore, as the libstdc++ version was in my case.  Execute root.sh for each agent afterwards, unless you have sudo configured in which case it will happen automatically
  10. Two of my agents that run on different platforms could not be upgraded right away.  The new versions of the agent software needed to be downloaded from Self Update.  I am skipping them for now
  11. Next, apply patch 13349651 to WebLogic, following the instructions in the README file.  I attempted to do so, but the patch was already installed so I skipped this step
  12. There are a few other optional, post-upgrade steps like deleting obsolete targets.  These are documented in the upgrade guide and I will not note them here
  13. As a final step, make sure you update your Oracle user’s environment variables to reflect the new middleware home, OMS home, agent home, and so on

Conclusion

At this point my EM12cR3 production upgrade is complete!  Everything I have checked so far appears fully functional.  The only problem I had was a small filesystem for the management agent on my sandbox server causing the agent upgrade to run out of space, forcing manual intervention to resolve.  Don’t be stingy with space like I am and you should be fine.

I haven’t taken any time to investigate the new features yet, but I will be now.

(EDITED TO ADD: I forgot to mention the steps to get BI Publisher working again.  Please refer to the EM12cR3 Advanced Installation and Configuration Guide, chapter 15.  Essentially you will need to perform a software-only installation into the new middleware home, then execute the configureBIP script with an -upgrade flag to complete the BIP setup.)

Stale EM12c patch recommendations? Get patch 14822626

I don’t use the automated patching functionality provided by EM12c.  I do, however, get value out of the patch recommendations since they serve as a good reminder when I’ve missed a patch that should be applied to one of my targets.  For this reason I was disappointed when, after upgrading from EM12cR1 to EM12cR2, the patch recommendations it gave me became stale and stopped getting updated when I loaded the em_catalog.zip file.

If you DO use the automated patching functionality, you have probably already followed all of the advice and installed the required patches documented in MOS note 427577.1, “Enterprise Manager patches required for setting up Provisioning, Patching and Cloning (Deployment Procedures)”.  In that case you already have this patch installed and don’t need to read any further, but if not, read on.

After upgrading to EM12cR2, I also upgraded several databases from 10gR2 to 11gR2.  Months passed, and yet the patch recommendations EM12c gave me continued to refer to 10gR2 patches which I knew weren’t applicable as I was running 11gR2.  I tried several things, like setting EM12c to offline mode, to online mode, loading em_catalog.zip, re-running the various “Refresh From My Oracle Support” jobs, all without ever receiving fresh patch recommendations.

So to sort this out, I did what I usually do, and asked about it on Twitter.  Big thanks to Sudip Datta, Vice President of Product Management at Oracle, who pointed me to bug 14822626 and its associated patch.  The bug does not appear to be public, but MOS note 1522918.1, “12C – Patch Recommendations Not Updating After Upgrade To 12.1.0.2 Cloud Control – ‘…Patch Recommendations Computation is disabled … skipping …'” documents the problem as a known issue after upgrading from 12.1.0.1 to 12.1.0.2 and clearly matches the behavior I saw.

As soon as I applied patch 14822626, the old stale patch recommendations were cleared out, and once I loaded the current em_catalog.zip file, I had accurate patch recommendations for my environment that I can now use to make decisions going forward.

Thank you, Sudip!

Final (!) update/fix for EM12c increased redo from EM_JOB_METRICS after upgrade to R2 (bug/patch 14726136)

Update 13 Nov 2012: Patch 14726136 has been obsoleted. The note on MOS indicated that the patch caused some metrics to be calculated incorrectly. The patch has been replaced by patch 14833587. I have applied the new patch and all appears well — jobs are running and redo generation on the repository database remains where it was before the upgrade to EM12c R2.

Following up again on the EM12cR2 upgrade issue from BP1 that causes significantly increased redo logging on the repository database due to heavy insert activity in EM_JOB_METRICS. I first covered this here, with a followup here, a partial workaround here, and then a warning here.

Patch 14726136 has been re-released on MOS. The initial release caused problems for me as it prevented all OEM scheduled jobs from running, eventually causing me to rollback the patch. I am very pleased to report that the new update of the patch (from Oct 30th) applies cleanly and all of my jobs are now running on-schedule and completing successfully. EM_JOB_METRICS is showing no more than 11 inserts per second, and more than a minute is passing between the batches of inserts. My redo volume is already down significantly.

Big thanks to Oracle support and development and the EM12c team!

Patch 14726136

You may have seen the patch in the title drop regarding increased redo logging after upgrading to EM12cR2.

I tried the patch Friday morning, and while it did decrease my redo logging, it also prevented any OEM scheduled jobs from running. I eventually rolled back the patch and jobs ran again.

I would be very interested to know if the patch works for you AND your jobs still run.

Fix for ASH Analytics deployment error after upgrade to R2

When upgrading from Oracle Enterprise Manager 12c R1 to R2, the one-system upgrade requires that you place the OMS into a new middleware home. After you’ve validated your R2 installation, and followed the instructions for removing the R1 install, the OMS host’s agent is still running out of your original middleware home so you can’t just remove the entire old directory.

You may have decided to clean up the leftover files from your old middleware home, taking care not to remove the agent. That’s all well and good. But after doing so, you may run into an error the next time you try to deploy the ASH Analytics PL/SQL packages to a database instance.

As a reminder, deploying these packages is done by going to the database target home page, and selecting “ASH Analytics” from the Performance menu.

If you’ve done a cleanup of your old middleware home, you are very likely to receive the following error text in the deployment job:
Package deployment driver file is at /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/ashviewer_pkgdefs.sql
Package deployment driver file is at /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/ashviewer_pkgbodys.sql
Package deployment driver file is at /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/prvs_awr_data.sql
Package deployment driver file is at /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/prvt_awr_data.sql
Package deployment driver file is at /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/prvs_awr_data_cp.sql
Package deployment driver file is at /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/prvt_awr_data_cp.sql
Package deployment driver file is at /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/dbms_compare_period.sql
Package deployment driver file is at /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/prvt_compare_period.sql
Package deployment driver file is at /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/eaddm_pkgdef.sql
Package deployment driver file is at /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/eaddm_pkgbody.sql
Instantiated JDBC Engine with connect string host.domain.com:1521:SID


You can view the output of this execution at the following location /tmp/DB_Deploy_201210171243.log
Altering session to set schema to DBSNMP

Currently executing null :: ALTER SESSION SET CURRENT_SCHEMA = DBSNMP
Altered session to set schema to DBSNMP

Executing /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/ashviewer_pkgdefs.sql

Error while executing the script : oracle.sysman.assistants.common.dbutil.SQLFatalErrorException: java.io.FileNotFoundException: /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/ashviewer_pkgdefs.sql (No such file or directory) at line number - -1
Driver SQL Script encountered errors.
Fail

At least the error text makes the problem clear. The SQL files containing the needed packages can’t be found, because EM12cR2 is still looking for them in the old middleware home.

The fix is easy. Just add a symlink in your old middleware home for the ‘plugins’ directory that points to the new middleware home’s plugins directory:

oracle@omshost$ cd /oracle/oem/Middleware
oracle@omshost$ ln -s ../Middleware12cR2/plugins/ ./plugins

Once you’ve finished that, return to the ASH Analytics deployment page and re-run the deployment. It should succeed and you can now take advantage of ASH Analytics!

If you need a primer on ASH Analytics, I highly recommend viewing the set of webinars presented by Doug Burns through AllThingsOracle.

How and why you should set target lifecycle status properties in EM12c

If, like me, you’re using EM12c after you were already plenty familiar with EM11g, you may have missed an important detail in the EM12c new features guide.

From New Features in Oracle Enterprise Manager Cloud Control 12c

Each target now has a lifecycle status target property which can be set to one of the following values: mission critical, production, staging, test, or development. This target property is used to specify the priority by which data from the target should be handled. When Enterprise Manager is under a heavy load, targets where the value of the lifecycle property is mission critical or production are treated with higher priority than targets where the value of the lifecycle property is staging, test, or development. Setting the priorities of your most important targets ensures that even as your data center grows and the number of managed targets grows, your most important targets continue to be treated at high priority.

You may not use some of the other new features like administration groups or lifecycle management, but it’s still very much worth your while to set the lifecycle status target property. After all, you’re more concerned about alerts and monitoring on your mission critical and other production systems than you are on the staging and test systems, so why not tell EM12c about that and gain the benefits of target prioritization?

If you have quite a few targets it can be quite tedious to step through them all in the GUI interface to set this property. It works, but it’ll take a while. Enter emcli. Rob Zoeteweij has covered the setup of EMCLI in his blog post Installing EMCLI on EM12c so I won’t repeat that here other than to add that with the release of EM12cR2 there is no longer a JDK in the $OMS_HOME so if you’re running 12.1.0.2 you should amend his instructions as follows:


oracle@omshost$ export JAVA_HOME=$OMS_HOME/../jdk16/jdk
oracle@omshost$ export PATH=$JAVA_HOME/bin:$PATH
oracle@omshost$ export ORACLE_HOME=$OMS_HOME
oracle@omshost$ cd $ORACLE_HOME
oracle@omshost$ mkdir emcli
oracle@omshost$ java -jar $ORACLE_HOME/sysman/jlib/emclikit.jar client -install_dir=$ORACLE_HOME/emcli
Oracle Enterprise Manager 12c Release 2.
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.

EM CLI client-side install completed successfully.
oracle@omshost$ $ORACLE_HOME/emcli/emcli setup -url=https://omshost.domain.com:7803/em -username=sysman
Oracle Enterprise Manager Cloud Control 12c Release 2.
Copyright (c) 1996, 2012 Oracle Corporation and/or its affiliates. All rights reserved.


Enter password

Warning: This certificate has not been identified as trusted in the local trust store
--------------------------------------
[certificate details snipped]
--------------------------------------
Do you trust the certificate chain? [yes/no] yes
Emcli setup successful

Once you have emcli running you can easily create files containing the target properties you would like to set, including the target lifecycle status, and apply them in bulk to your EM12c installation.

For the example I will demonstrate setting the target lifecycle status property for host targets. First you need to produce a list of your host targets, formatted for eventual input to the set_target_property_value verb:

oracle@omshost$ ./emcli get_targets -noheader -script -targets=host | awk '{print $4":"$3":LifeCycle Status:"}' > /tmp/targets

As another example, from commenter Bill Korb, to produce a list of database targets, including any currently under blackout, run:

oracle@omshost$ ./emcli get_targets -noheader -format='name:script;column_separator:|;' -targets='%database%' | awk -F\| '{print $4":"$3":LifeCycle Status:"}' > /tmp/targets

Edit the resulting file, appending the host or database’s lifecycle stage to each line. Be aware of the predefined lifecycle stages provided by Oracle, which are listed below in order of precedence:

  1. MissionCritical
  2. Production
  3. Stage
  4. Test
  5. Development

You can modify the names of these lifecycle stages with the modify_lifecycle_stage_name verb if you wish. Your file should now look something like:

dev1.domain.com:host:LifeCycle Status:Development
omshost.domain.com:host:LifeCycle Status:MissionCritical
prod1.domain.com:host:LifeCycle Status:Production
prod2.domain.com:host:LifeCycle Status:Production
prod3.domain.com:host:LifeCycle Status:Production
prod4.domain.com:host:LifeCycle Status:Production
stage1.domain.com:host:LifeCycle Status:Stage
test1.domain.com:host:LifeCycle Status:Test
test2.domain.com:host:LifeCycle Status:Test

Now make the call to emcli to load your target property definitions into the OMS:

oracle@omshost$ ./emcli set_target_property_value -property_records="REC_FILE" -input_file="REC_FILE:/tmp/targets" -separator=property_records="\n"

There you go. Your hosts are now updated with appropriate target lifecycle stages and the OMS will prioritize them based on these settings whenever the OMS is under high load. Repeat this for your listeners (-targets=oracle_listener), database instances (-targets=oracle_database) and so on until all of your targets have a lifecycle stage assigned. I’ve broken these out by target type for simplicity of documentation, but you can also just produce a single large file containing the lifecycle status for all of your targets and load the whole thing at once. This same technique works to assign a contact, comment, location, or any other target property you find useful.

Partial workaround for increased redo generation on EM12cR2 repository database after upgrade from R1

Another followup to my previous posts on increased redo logging on the repository database after upgrading to EM12cR2.

On the OTN Forum thread, slohit posted a suggestion with a way to reduce the frequency of job dispatcher runs:

To reduce the frequency of inserts, it would be better to slow down the frequency of the dispatcher. This is OK only if you are fine with a lower response time between steps for your jobs. If it is only a few jobs/job steps scheduled to run together, this is all right

emctl set property -name oracle.sysman.core.jobs.minDispatcherSleepInterval -value 0.5
emctl set property -name oracle.sysman.core.jobs.linearBackoff -value 0.5

By making this change, followed by a bounce of the OMS, the redo generation on my repository database decreased by about 50%. It’s still 200-300% more than before the upgrade, but it’s quite an improvement. The EM_JOB_METRICS table now only receives about 20 inserts per second compared to 60. My 13GB repository database should now only produce about 5GB of redo per day instead of 10GB.

Prior to the change, emctl get property reported these property values as null so presumably the OMS was working with whatever the built-in default values were.

In my case, a delay between job steps is a tradeoff that I can tolerate. This OMS runs simple database backups, SQL and RMAN scripts, so I do not have any time-critical multi-step jobs that could be negatively impacted by delayed response to completion of individual job steps. Your mileage may vary.

Oracle Support is continuing to investigate this bug. At least one other poster on the forum has reported that they too are experiencing this issue so I’m looking forward to finding out if this change reduces their redo generation to an acceptable rate. That same poster also reports that they do not see the issue on a new install of EM12cR2, only on a system upgraded from R1. It’s not clear to me why an upgraded system would experience this when processing the raw metric data while a fresh install would not, nor is it clear how many upgraded systems are experiencing the issue. But at least we now have a partial workaround for bug 14726136.

BUG 14726136 – INCREASED REDO LOGGING ON REPOSITORY DB AFTER UPGRADING TO OEM 12.1.0.2

Following up on my previous post about increased redo logging on the repository DB after upgrading to OEM 12cR2, we now have bug 14726136 filed.

The heavy redo activity is coming from SYSMAN inserts into the EM_JOB_METRICS table. It’s doing 60 inserts a second, in batches of 10 per LOG_TIMESTAMP, where each insert looks like:

insert into "SYSMAN"."EM_JOB_METRICS"("SOURCE","LOG_TIMESTAMP","METRIC_NAME","METRIC_QUALIFIER","METRIC_VALUE") values
('omshost.domain.com:4890_Management_Service',TO_TIMESTAMP('26-SEP-12 03.09.25.215415 PM'),'DISP_REQ_STEPS','0','25');

The 10 inserts differ only in the values of the METRIC_QUALIFIER and METRIC_VALUE columns:

'0', '25'
'0', '0'
'1', '12'
'1', '0'
'2', '25'
'2', '0'
'3', '10'
'3', '0'
'4', '10'
'4', '0'

LogMiner shows this as a significant proportion of the total database activity, with the following extracts based on about 20 minutes (or about 200MB) of repository DB operation:

SQL> select TABLE_NAME, count(*) from v$logmnr_contents group by table_name order by 2 desc;

                                                       TABLE_NAME           COUNT(*)
                                                     EM_JOB_METRICS           201740
                                                                              163843
                                                    EM_METRIC_ITEMS            13374
                                                    EM_METRIC_VALUES           12524
                                                MGMT_AVAILABILITY_MARKER        3845
                                                       COL_USAGE$               2403
                                                          ROUT                  1453
                                                     SCHEDULER$_JOB             1067


SQL> select seg_owner, operation, count(*) from v$logmnr_contents group by seg_owner, operation order by 1;

                                              SEG_OWNER     OPERATION     COUNT(*)  
                                               DBSNMP        DELETE             13  
                                               DBSNMP        INSERT             13  
                                               DBSNMP        UPDATE              4  
                                                RCAT         DELETE            269  
                                                RCAT         INSERT           1372  
                                                RCAT    SELECT_FOR_UPDATE       50  
                                                RCAT         UPDATE            884  
                                                 SYS         DELETE             88  
                                                 SYS         INSERT           1041
                                                 SYS    SELECT_FOR_UPDATE      265
                                                 SYS         UPDATE           4802  
                                               SYSMAN        DELETE          97325  
                                               SYSMAN        INSERT         106610  
                                               SYSMAN       INTERNAL           196  
                                               SYSMAN       LOB_TRIM             5  
                                               SYSMAN       LOB_WRITE          326  
                                               SYSMAN   SELECT_FOR_UPDATE     1203  
                                               SYSMAN    SEL_LOB_LOCATOR       103  
                                               SYSMAN      UNSUPPORTED       13626  
                                               SYSMAN        UPDATE          21808
                                               UNKNOWN       DELETE             82  
                                               UNKNOWN       INSERT             80  
                                               UNKNOWN  SELECT_FOR_UPDATE       65
                                               UNKNOWN       UPDATE             65
                                                             COMMIT          19792
                                                          DPI SAVEPOINT         15
                                                            INTERNAL        124178
                                                            ROLLBACK            33
                                                              START          19825

This is not a particularly busy OMS, it’s monitoring fewer than 200 targets and only runs a single job every couple minutes.

Followup: EM12cR2 reports Job Purge repository job as Down

This is a follow up to EM12cR2 reports Job Purge repository job as Down. Erik at the OTN Forum has provided a fix:

EM12cR2 – “Job Purge” repository scheduler job falsely(?) reported down .

Simply run the following update connected as user SYSMAN on your repository database, and the repository jobs will all show as clear and the open incident will be closed:

UPDATE MGMT_PERFORMANCE_NAMES SET display_name = NULL,
dbms_jobname = NULL, is_dbmsjob = 'N', is_deleted = 'Y'
WHERE UPPER(DBMS_JOBNAME) LIKE 'MGMT_JOB_ENGINE.APPLY_PURGE_POLICIES%';
COMMIT;

EM12cR2 reports Job Purge repository job as Down

Crossposted from the OTN Forum post. According to Oracle this is a bug and a script to resolve the reported problematic job will be forthcoming.

On the Repository page (Setup -> Manage Cloud Control -> Repository), in the Repository Scheduler Jobs Status section, the “Job Purge” job is shown as down with a red arrow and blanks for both Next Scheduled Run and Last Scheduled Run. All the other jobs look good and are running successfully.

This also appears as an incident stating “DBMS Job UpDown for Job Purge exceeded the critical threshold (DOWN). Current value: DOWN”.

The framework metric reference manual for this metric (link here) indicates that this metric equates to the DBMS_JOB ‘broken’ status and suggests retrieving the job_id and re-submitting it, but the provided instructions don’t find anything useful.

SQL> sho user
USER is "SYSMAN"
SQL> select dbms_jobname from mgmt_performance_names where display_name = 'Job Purge';

DBMS_JOBNAME
--------------------------------------------------------------------------------
MGMT_JOB_ENGINE.apply_purge_policies()

SQL> select job from all_jobs where what='MGMT_JOB_ENGINE.apply_purge_policies()';

no rows selected

SQL> select count(*) from all_jobs;

COUNT(*)
----------
0

No ORA-12012 messages found in the alert/trace logs.

Repvfy also reports this as a missing scheduler job:

2. Missing DBMS_SCHEDULER jobs
------------------------------

DISPLAY_NAME DBMS_JOBNAME
---------------------------------------- -----------------------------------------------------
Job Purge MGMT_JOB_ENGINE.apply_purge_policies()

The MGMT_JOB_ENGINE package no longer contains apply_purge_policies. It seems to have moved to EM_JOB_PURGE.

According to dba_scheduler_jobs, EM_JOB_PURGE.APPLY_PURGE_POLICIES is running just fine:

SQL> select job_action, enabled, state, failure_count, last_start_date, last_run_duration
2 from dba_scheduler_jobs where job_name = 'EM_JOB_PURGE_POLICIES';

JOB_ACTION ENABL STATE FAILURE_COUNT
---------------------------------------- ----- ---------- -------------
LAST_START_DATE
----------------------------------------
LAST_RUN_DURATION
----------------------------------------
EM_JOB_PURGE.APPLY_PURGE_POLICIES(); TRUE SCHEDULED 0
14-SEP-12 05.00.00.010318 AM US/EASTERN
+000000000 00:00:09.464552

It seems to me this is some kind of metadata inconsistency between mgmt_performance_names and the DBMS scheduler, and probably isn’t something to be concerned with. Any thoughts?

EM12cR2 increased redo logging on repository database

I’ve posted this on the OTN forum here, but perhaps it’s worth a blog post.

Has anyone else noticed increased redo logging in their repository database after an upgrade to EM12cR2?

I’m running on Linux x86-64 with an 11.2.0.3 repository database. Prior to the upgrade I was running 12.1.0.1.0 with BP1, and my repository database was producing, on average, about 80-120MB of archived redo logs per hour. Since the upgrade to 12.1.0.2.0, my repository database is producing, on average, 450-500MB of archived redo logs per hour.

Upgrading the deployed agents from 12.1.0.1.0 to 12.1.0.2.0 hasn’t seemed to help; I’ve upgraded about 10 of 15 without any apparent decrease in redo logging.

Just wondering if this is comparable to what others are seeing or if I have a local issue I should investigate.