Tag Archives: upgrade

Fix: Plugin error when upgrading EM13cR1 agent to EM13cR2 (13.2) on Windows 2008 R2 x64

I ran into the following issue while attempting to upgrade to the Oracle management agent on my one Windows (2008 R2, x64) server to the 13.2.0.0.0 version distributed with Oracle Enterprise Manager 13.2. The agent upgrade repeatedly failed in the “Upgrading Management Agent” step with an error message:


Exit Code :0
The version is 13.1.0.0.0
Checking for the version 13.2.0.0.0
The agent is not upgraded successfully
[...]
Plugins upgrade failed.
Plugin upgrade failed.
Check the file E:/agent13c/agent_inst/install/plugins_upgrade.txt.status on agent for plugin upgrade status.
Check latest E:/agent13c/agent_inst/install/logs/agentplugindeploy_.log
Plugins upgrade failed.0

The log file referenced in the error message contains some, but not much, additional information:


The command executed for discovery at _2016_10_12_11_48_28 is : E:\agent13c\agent_inst\bin\emctl_upgrade.bat set_discovery_root plugin oracle.sysman.si E:\agent13c\agent_13.2.0.0.0\plugins\oracle.sysman.si.discovery.plugin_13.2.1.0.0 13.2.1.0.0
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
return value is : 65280
Install case : set_discovery_root failure...existing on error value 65280

I fixed this issue by installing the Microsoft Visual C++ 2010 Service Pack 1 Redistributable Package MFC Security Update package on the server. Finally, create a new directory called “prerelogs” inside of the …\agent_13.2.0.0.0\cfgtoollogs\ directory, or else my deployment hung forever during the “Performing prerequisite checks” step. Note that “prerelogs” does not contain the letter q, do not use ‘prereqlogs’. The agent deployment uses this directory to extract software for running the prerequisite checks; possibly setting SCRATCHPATH or another extra parameter would resolve this issue without manually creating the directory, but this worked for me.

After installation of the VC++ 2010 package, which did not require a reboot, followed by creating of the prerelogs directory, I repeated the agent upgrade and it completed successfully.

This issue may or may not occur on other Windows versions, I have not heard any other reports of problems. The first release of EM13c did not seem to require this package.

Advertisements

My Production EM12c Upgrade From R2 (12.1.0.2) to R3 (12.1.0.3)

This post covers my production upgrade from EM12c R2 to EM12c R3 on Linux x86-64 (SLES11 SP2).  To stress test the upgrade and keep it interesting, this system also has BI Publisher integrated into OEM, and also has the plugin from NetApp (version 12.1.0.1.0), the VMware plugin from BlueMedora (version 12.1.0.5.0), and the MySQL plugin from Pythian (version 12.1.0.1.1).  The repository database is running on version 11.2.0.3.

I’m feeling lucky today.  So this is just going straight into production.  Famous last words…

Preparation

  1. Go to edelivery and search for the Oracle Enterprise Manager product pack and platform Linux x86-64
  2. Follow the link titled “Oracle Enterprise Manager Cloud Control 12c Release 3 (12.1.0.3) Media Pack for Linux x86-64
  3. Download the three files needed for EM12c R3: V38641-01.zip, V38642-01.zip, and V38643-01.zip
  4. Download patch 13349651 for WebLogic, you will need it during the post-upgrade steps
  5. View the digest and run md5sum against each downloaded file to confirm that the files downloaded correctly
  6. Transfer the EM12c R3 files to a staging area on your Oracle Enterprise Manager server and unzip all three of them
  7. Delete the three downloaded .zip files if you are short on space (but don’t just “rm *.zip” or you’ll remove the necessary WT.zip file at the top level of your staging directory)
  8. Review the upgrade guide
  9. Create backups of your current OMS home, agent home, software library and Oracle inventory
  10. Create a backup of your current repository database that you can restore from if necessary
  11. If you use a dedicated filesystem for your Oracle Management Agents, make sure that dedicated filesystem has enough free space.  I used a small, 2GB filesystem, and this was barely large enough, except for the one sandbox server where it was too small and I was not able to complete the agent upgrade without adding space
  12. Stop the BI Publisher WebLogic Server (if you have it installed) via the WebLogic Admin Console
  13. Make sure your repository database does not have snapshots created on any tables by running “select master, log_table from all_mview_logs where log_owner = ‘SYSMAN'” — if any snapshots are found, follow the instructions in the upgrade guide to drop them
  14. If your repository database is version 11.1.0.7 or 11.2.0.1, follow the steps in the upgrade guide to apply the prerequisite patches needed to proceed
  15. Copy the emkey from the existing OMS to the existing management repository by running “$OMS_HOME/bin/emctl config emkey -copy_to_repos” and enter the SYSMAN password when prompted
  16. Confirm the emkey was copied by running “$OMS_HOME/bin/emctl status emkey” and enter the SYSMAN password when prompted
  17. Stop every OMS in your environment by running “$OMS_HOME/bin/emctl stop oms -all”
  18. Stop the management agent monitoring the management services and repository target by running “$AGENT_HOME/bin/emctl stop agent”

Running The Upgrade

Due to some issues I had with the upgrade from EM12c R1 to R2, I highly recommend that you do NOT use Cygwin to ssh to your OMS host and display the installer over ssh using Cygwin’s X server.  Use VNC instead.  I’m not going to try Cygwin this time through.

First, I start vncserver on the OMS host.  Then I connect to it to using TightVNC from my desktop machine.

  1. Navigate to the staging directory where you unzipped the EM12c R3 distribution files and run “./runInstaller” as your oracle software owner
  2. As an SAP customer, we are not allowed to use OCM, so I skipped the steps involving entering my My Oracle Support credentialsStep 1
  3. I also skipped the search for updates as this is a new enough release at the moment there should not be any necessaryStep 2
  4. The prerequisite checks run.  I received a warning about libstdc++43-4.3 not being found, but libstdc++43-devel-4.3.3-x86_64 fulfills this need, so I click ignore and then Next to continueStep 3
  5. Only the one system upgrade is supported when upgrading from EM12cR2, so I selected “Upgrade an existing system”, “One system upgrade”, and my existing middleware home.  Click NextStep 4
  6. This out of place upgrade will go into a new middleware home.  I am using /oracle/oem/Middleware12cR3.  Click Next and the installer will confirm that you have enough free space availableStep 5
  7. Here I had to pause and request more space from my storage admin, as the installer wants at least 14.0GB of free space.  Once that was done, I proceeded
  8. Enter the SYS and SYSMAN passwords and check the box to confirm that you have backed up the repository (you should have your OMS, agent, etc all backed up as well), then click NextStep 6
  9. The installer will check various parameters on your repository database and offer the chance to fix them if any need to be changed.  I accept the fixes and click YesStep 6(b)
  10. The installer checks some additional settings and notes that they should be reviewed after the installation or fixed now.  I explicitly granted execute on DBMS_RANDOM to DBSNMP and then clicked OKStep 6(c)
  11. The installer lists the plugin versions that will change and the plugins that will migrate. Confirm this all looks right and then click NextStep 7
  12. The installer lists additional plugins you can choose to deploy at install time.  I do not use any of these so I left them all unchecked and clicked NextStep 8
  13. The installer requests the password for your WebLogic adminserver and confirmation of the hostname, port and username.  Provide the password and click Next.  You may be able to change the OMS instance directory here but I do not suggest doing soStep 9
  14. You now have a chance to review your settings, then click Install to proceed with the upgradeStep 10
  15. Installation proceedsStep 11
  16. You are then prompted to run the allroot.sh file.  Login to the server and execute it as root or via sudoallroot.sh
  17. Once the install/upgrade is complete, the installer will display an installation summary.  Review it, save the URLs it gives you for the OMS and adminserver, then click CloseUpgrade Summary Report

Overall, the upgrade installation steps took 1 hour and 15 minutes in my environment.  This was on a physical server with 126GB RAM, 16 dual core processors and 200 managed targets.  This does not include the post-upgrade steps shown below.

Post Upgrade Steps

Now that the upgrade is complete, return to the upgrade guide to complete post upgrade steps.  Your environment may differ from mine, but these are the steps I had to follow.

  1. Start your central agent by running “$AGENT_HOME/bin/emctl start agent”.  At this point the load on my system went up very high and began responding very slowly.  I walked away for 10 minutes to let things settle down
  2. Open your web browser and go to the URL provided at the end of the installation and login as SYSMAN.  When I first tried to do so using Firefox I received an error indicating an invalid certificate.  I had to delete the old certificates and authorities from my previous installation and restart Firefox before it would allow me in.  MSIE worked fine though
  3. Update the central agent (the management agent installed on the OMS host) by clicking on the Setup menu, then Manage Cloud Control, then Upgrade Agents.  Click the Add button and select your central agent.  I choose “Override preferred credentials” since I have not configured sudo.  Click Submit to continue, then OK when warned that you may have to run root.sh manuallyUpgrade Central Agent
  4. My first upgrade attempt on the central agent failed due to a prerequisite check for package libstdc++-43.  The easy thing to do here is expand the Additional Inputs region and provide “-ignorePrereqs” as an additional parameter, but I chose to complete this agent upgrade using emcli and describe that process in the next two steps
  5. First run “$OMS_HOME/bin/emcli login -username=sysman” and enter the SYSMAN password when prompted.  Then run “$OMS_HOME/bin/emcli sync”
  6. Upgrade the agent by running “$OMS_HOME/bin/emcli upgrade_agents -additional_parameters=”-ignorePrereqs” -agents=”hostname.domain.com:3872″
  7. Wait while the upgrade proceeds.  You can view upgrade progress by running “$OMS_HOME/bin/emcli get_agent_upgrade_status”, or in the GUI by clicking on “Agent Upgrade Results” in the “Upgrade Agents” pageAgent Upgrade Progress
  8. Click Done once the central agent upgrade completes.  Go to the new agent home and run root.sh as root or via sudo
  9. Then repeat this process for the rest of your agents.  Try to install them first WITHOUT using the “-ignorePrereqs” flag, because if there are missing prerequisites you need to identify the issue and find out if it is something that it is appropriate to ignore, as the libstdc++ version was in my case.  Execute root.sh for each agent afterwards, unless you have sudo configured in which case it will happen automatically
  10. Two of my agents that run on different platforms could not be upgraded right away.  The new versions of the agent software needed to be downloaded from Self Update.  I am skipping them for now
  11. Next, apply patch 13349651 to WebLogic, following the instructions in the README file.  I attempted to do so, but the patch was already installed so I skipped this step
  12. There are a few other optional, post-upgrade steps like deleting obsolete targets.  These are documented in the upgrade guide and I will not note them here
  13. As a final step, make sure you update your Oracle user’s environment variables to reflect the new middleware home, OMS home, agent home, and so on

Conclusion

At this point my EM12cR3 production upgrade is complete!  Everything I have checked so far appears fully functional.  The only problem I had was a small filesystem for the management agent on my sandbox server causing the agent upgrade to run out of space, forcing manual intervention to resolve.  Don’t be stingy with space like I am and you should be fine.

I haven’t taken any time to investigate the new features yet, but I will be now.

(EDITED TO ADD: I forgot to mention the steps to get BI Publisher working again.  Please refer to the EM12cR3 Advanced Installation and Configuration Guide, chapter 15.  Essentially you will need to perform a software-only installation into the new middleware home, then execute the configureBIP script with an -upgrade flag to complete the BIP setup.)

Stale EM12c patch recommendations? Get patch 14822626

I don’t use the automated patching functionality provided by EM12c.  I do, however, get value out of the patch recommendations since they serve as a good reminder when I’ve missed a patch that should be applied to one of my targets.  For this reason I was disappointed when, after upgrading from EM12cR1 to EM12cR2, the patch recommendations it gave me became stale and stopped getting updated when I loaded the em_catalog.zip file.

If you DO use the automated patching functionality, you have probably already followed all of the advice and installed the required patches documented in MOS note 427577.1, “Enterprise Manager patches required for setting up Provisioning, Patching and Cloning (Deployment Procedures)”.  In that case you already have this patch installed and don’t need to read any further, but if not, read on.

After upgrading to EM12cR2, I also upgraded several databases from 10gR2 to 11gR2.  Months passed, and yet the patch recommendations EM12c gave me continued to refer to 10gR2 patches which I knew weren’t applicable as I was running 11gR2.  I tried several things, like setting EM12c to offline mode, to online mode, loading em_catalog.zip, re-running the various “Refresh From My Oracle Support” jobs, all without ever receiving fresh patch recommendations.

So to sort this out, I did what I usually do, and asked about it on Twitter.  Big thanks to Sudip Datta, Vice President of Product Management at Oracle, who pointed me to bug 14822626 and its associated patch.  The bug does not appear to be public, but MOS note 1522918.1, “12C – Patch Recommendations Not Updating After Upgrade To 12.1.0.2 Cloud Control – ‘…Patch Recommendations Computation is disabled … skipping …'” documents the problem as a known issue after upgrading from 12.1.0.1 to 12.1.0.2 and clearly matches the behavior I saw.

As soon as I applied patch 14822626, the old stale patch recommendations were cleared out, and once I loaded the current em_catalog.zip file, I had accurate patch recommendations for my environment that I can now use to make decisions going forward.

Thank you, Sudip!

Final (!) update/fix for EM12c increased redo from EM_JOB_METRICS after upgrade to R2 (bug/patch 14726136)

Update 13 Nov 2012: Patch 14726136 has been obsoleted. The note on MOS indicated that the patch caused some metrics to be calculated incorrectly. The patch has been replaced by patch 14833587. I have applied the new patch and all appears well — jobs are running and redo generation on the repository database remains where it was before the upgrade to EM12c R2.

Following up again on the EM12cR2 upgrade issue from BP1 that causes significantly increased redo logging on the repository database due to heavy insert activity in EM_JOB_METRICS. I first covered this here, with a followup here, a partial workaround here, and then a warning here.

Patch 14726136 has been re-released on MOS. The initial release caused problems for me as it prevented all OEM scheduled jobs from running, eventually causing me to rollback the patch. I am very pleased to report that the new update of the patch (from Oct 30th) applies cleanly and all of my jobs are now running on-schedule and completing successfully. EM_JOB_METRICS is showing no more than 11 inserts per second, and more than a minute is passing between the batches of inserts. My redo volume is already down significantly.

Big thanks to Oracle support and development and the EM12c team!

Patch 14726136

You may have seen the patch in the title drop regarding increased redo logging after upgrading to EM12cR2.

I tried the patch Friday morning, and while it did decrease my redo logging, it also prevented any OEM scheduled jobs from running. I eventually rolled back the patch and jobs ran again.

I would be very interested to know if the patch works for you AND your jobs still run.

Fix for ASH Analytics deployment error after upgrade to R2

When upgrading from Oracle Enterprise Manager 12c R1 to R2, the one-system upgrade requires that you place the OMS into a new middleware home. After you’ve validated your R2 installation, and followed the instructions for removing the R1 install, the OMS host’s agent is still running out of your original middleware home so you can’t just remove the entire old directory.

You may have decided to clean up the leftover files from your old middleware home, taking care not to remove the agent. That’s all well and good. But after doing so, you may run into an error the next time you try to deploy the ASH Analytics PL/SQL packages to a database instance.

As a reminder, deploying these packages is done by going to the database target home page, and selecting “ASH Analytics” from the Performance menu.

If you’ve done a cleanup of your old middleware home, you are very likely to receive the following error text in the deployment job:
Package deployment driver file is at /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/ashviewer_pkgdefs.sql
Package deployment driver file is at /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/ashviewer_pkgbodys.sql
Package deployment driver file is at /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/prvs_awr_data.sql
Package deployment driver file is at /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/prvt_awr_data.sql
Package deployment driver file is at /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/prvs_awr_data_cp.sql
Package deployment driver file is at /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/prvt_awr_data_cp.sql
Package deployment driver file is at /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/dbms_compare_period.sql
Package deployment driver file is at /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/prvt_compare_period.sql
Package deployment driver file is at /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/eaddm_pkgdef.sql
Package deployment driver file is at /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/eaddm_pkgbody.sql
Instantiated JDBC Engine with connect string host.domain.com:1521:SID


You can view the output of this execution at the following location /tmp/DB_Deploy_201210171243.log
Altering session to set schema to DBSNMP

Currently executing null :: ALTER SESSION SET CURRENT_SCHEMA = DBSNMP
Altered session to set schema to DBSNMP

Executing /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/ashviewer_pkgdefs.sql

Error while executing the script : oracle.sysman.assistants.common.dbutil.SQLFatalErrorException: java.io.FileNotFoundException: /oracle/oem/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.2.0/sql/db/latest/instance/ashviewer_pkgdefs.sql (No such file or directory) at line number - -1
Driver SQL Script encountered errors.
Fail

At least the error text makes the problem clear. The SQL files containing the needed packages can’t be found, because EM12cR2 is still looking for them in the old middleware home.

The fix is easy. Just add a symlink in your old middleware home for the ‘plugins’ directory that points to the new middleware home’s plugins directory:

oracle@omshost$ cd /oracle/oem/Middleware
oracle@omshost$ ln -s ../Middleware12cR2/plugins/ ./plugins

Once you’ve finished that, return to the ASH Analytics deployment page and re-run the deployment. It should succeed and you can now take advantage of ASH Analytics!

If you need a primer on ASH Analytics, I highly recommend viewing the set of webinars presented by Doug Burns through AllThingsOracle.

How and why you should set target lifecycle status properties in EM12c

If, like me, you’re using EM12c after you were already plenty familiar with EM11g, you may have missed an important detail in the EM12c new features guide.

From New Features in Oracle Enterprise Manager Cloud Control 12c

Each target now has a lifecycle status target property which can be set to one of the following values: mission critical, production, staging, test, or development. This target property is used to specify the priority by which data from the target should be handled. When Enterprise Manager is under a heavy load, targets where the value of the lifecycle property is mission critical or production are treated with higher priority than targets where the value of the lifecycle property is staging, test, or development. Setting the priorities of your most important targets ensures that even as your data center grows and the number of managed targets grows, your most important targets continue to be treated at high priority.

You may not use some of the other new features like administration groups or lifecycle management, but it’s still very much worth your while to set the lifecycle status target property. After all, you’re more concerned about alerts and monitoring on your mission critical and other production systems than you are on the staging and test systems, so why not tell EM12c about that and gain the benefits of target prioritization?

If you have quite a few targets it can be quite tedious to step through them all in the GUI interface to set this property. It works, but it’ll take a while. Enter emcli. Rob Zoeteweij has covered the setup of EMCLI in his blog post Installing EMCLI on EM12c so I won’t repeat that here other than to add that with the release of EM12cR2 there is no longer a JDK in the $OMS_HOME so if you’re running 12.1.0.2 you should amend his instructions as follows:


oracle@omshost$ export JAVA_HOME=$OMS_HOME/../jdk16/jdk
oracle@omshost$ export PATH=$JAVA_HOME/bin:$PATH
oracle@omshost$ export ORACLE_HOME=$OMS_HOME
oracle@omshost$ cd $ORACLE_HOME
oracle@omshost$ mkdir emcli
oracle@omshost$ java -jar $ORACLE_HOME/sysman/jlib/emclikit.jar client -install_dir=$ORACLE_HOME/emcli
Oracle Enterprise Manager 12c Release 2.
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.

EM CLI client-side install completed successfully.
oracle@omshost$ $ORACLE_HOME/emcli/emcli setup -url=https://omshost.domain.com:7803/em -username=sysman
Oracle Enterprise Manager Cloud Control 12c Release 2.
Copyright (c) 1996, 2012 Oracle Corporation and/or its affiliates. All rights reserved.


Enter password

Warning: This certificate has not been identified as trusted in the local trust store
--------------------------------------
[certificate details snipped]
--------------------------------------
Do you trust the certificate chain? [yes/no] yes
Emcli setup successful

Once you have emcli running you can easily create files containing the target properties you would like to set, including the target lifecycle status, and apply them in bulk to your EM12c installation.

For the example I will demonstrate setting the target lifecycle status property for host targets. First you need to produce a list of your host targets, formatted for eventual input to the set_target_property_value verb:

oracle@omshost$ ./emcli get_targets -noheader -script -targets=host | awk '{print $4":"$3":LifeCycle Status:"}' > /tmp/targets

As another example, from commenter Bill Korb, to produce a list of database targets, including any currently under blackout, run:

oracle@omshost$ ./emcli get_targets -noheader -format='name:script;column_separator:|;' -targets='%database%' | awk -F\| '{print $4":"$3":LifeCycle Status:"}' > /tmp/targets

Edit the resulting file, appending the host or database’s lifecycle stage to each line. Be aware of the predefined lifecycle stages provided by Oracle, which are listed below in order of precedence:

  1. MissionCritical
  2. Production
  3. Stage
  4. Test
  5. Development

You can modify the names of these lifecycle stages with the modify_lifecycle_stage_name verb if you wish. Your file should now look something like:

dev1.domain.com:host:LifeCycle Status:Development
omshost.domain.com:host:LifeCycle Status:MissionCritical
prod1.domain.com:host:LifeCycle Status:Production
prod2.domain.com:host:LifeCycle Status:Production
prod3.domain.com:host:LifeCycle Status:Production
prod4.domain.com:host:LifeCycle Status:Production
stage1.domain.com:host:LifeCycle Status:Stage
test1.domain.com:host:LifeCycle Status:Test
test2.domain.com:host:LifeCycle Status:Test

Now make the call to emcli to load your target property definitions into the OMS:

oracle@omshost$ ./emcli set_target_property_value -property_records="REC_FILE" -input_file="REC_FILE:/tmp/targets" -separator=property_records="\n"

There you go. Your hosts are now updated with appropriate target lifecycle stages and the OMS will prioritize them based on these settings whenever the OMS is under high load. Repeat this for your listeners (-targets=oracle_listener), database instances (-targets=oracle_database) and so on until all of your targets have a lifecycle stage assigned. I’ve broken these out by target type for simplicity of documentation, but you can also just produce a single large file containing the lifecycle status for all of your targets and load the whole thing at once. This same technique works to assign a contact, comment, location, or any other target property you find useful.