Tag Archives: repository

Securing Oracle Enterprise Manager 13cR2

Oracle released Oracle Enterprise Manager 13cR2 at the beginning of October 2016. I have upgraded my production system to this new version, and here I provide a 13cR2-compatible version of my EM13c security checkup script. In addition to updating the script for EM13cR2, I have also updated it to take account of Oracle’s recommendation that single-instance non-RAC databases such as OEM repositories should now apply the DBBP Bundle Patch (previously known as the engineered systems bundle patch).

Latest Updates

Latest release: April 21, 2017, version 2.6. This release covers the 20170418 patch release and latest batch of recommended EM13.2 patches. See the changelog contained in the script for more details.

Download the latest release from https://raw.githubusercontent.com/brianpardy/em13c/master/checksec13R2.sh

EMCLI

If you have used this script for a while, you can download the latest release and just run it. It will continue to work the way it always has. If you would like to enable additional, optional functionality, enable the checksec13R2.sh EMCLI integration by logging in to EMCLI with an OEM administrator account before running checksec13R2.sh. The script will use EMCLI and attempt to check for plugin bundle patches on ALL of your OEM agents, not only the chained agent as it used to. It will also use EMCLI to attempt to validate the Java versions on all of your agents. This functionality requires that the EMCLI user account has access to run the execute_sql and execute_hostcmd, and also requires that the EMCLI user account has preferred credentials set for the repository database (normal and sysdba), repository database host, and for every host with a management agent.

To simplify the process, I have created a script to create a CHECKSEC user account in your OEM environment. The script will prompt you for the named credentials that the new account should use your repository database and each host. If you run this script after logging in to EMCLI as SYSMAN, it will create the new OEM user, grant acccess to all specified credentials, and grant EM_ALL_OPERATOR and VIEW_ANY_TARGET privileges so that the new account will have all the access needed to run all the optional checksec13R2.sh checks. I have included sample output from the user creation script at the end of this post. You can download the user creation script at create_user_for_checksec13R2.sh.

Download

You can access my EM13c script repository at https://github.com/brianpardy/em13c. To directly access the EM13cR2 security checkup script, use https://raw.githubusercontent.com/brianpardy/em13c/master/checksec13R2.sh.

Example Output – checksec13R2.sh


Performing EM13c R2 security checkup version 2.6 on omshost.domain.com at Fri Apr 21 09:32:30 EDT 2017.

Gathering info...
EM13c config... OK
Repos DB... 12.1.0.2.0 OK
OPatch-OMS... OK
OPatch-Agent... OK
OPatch-Repos DB... OK
OMSPatcher-OMS... OK
EMCLI login... OK
EMCLI-Agent list... OK
EMCLI-Agent patches... OK
EMCLI-Agent homes... OK

Using port definitions from configuration files
/etc/oragchomelist
/oracle/oem/gc_inst1/em/EMGC_OMS1/emgc.properties
/oracle/oem/gc_inst1/em/EMGC_OMS1/embip.properties
/oracle/oem/agent13cR1/agent_13.2.0.0.0/../agent_inst/sysman/emd/targets.xml

Agent port found at omshost.domain.com:3872
BIPublisher port found at omshost.domain.com:9803
BIPublisherOHS port found at omshost.domain.com:9852
NodeManager port found at omshost.domain.com:7403
OMSconsole port found at omshost.domain.com:7802
OMSproxy port found at omshost.domain.com:7301
OMSupload port found at omshost.domain.com:4903
WLSadmin found at omshost.domain.com:7102

Repository DB version=12.1.0.2.0 SID=oemdb host=omshost.domain.com
Repository DB target name=oemdb.domain.com

Using OPENSSL=/usr/bin/openssl1 (has TLS1_2=2)
Repository DB on OMS server, will check patches/parameters in /oracle/oem/product/12.1.0/db

(1) Checking SSL/TLS configuration (see notes 2138391.1, 2212006.1)

(1a) Forbid SSLv2 connections
Confirming ssl2 disabled for Agent at omshost.domain.com:3872... OK
Confirming ssl2 disabled for BIPublisher at omshost.domain.com:9803... OK
Confirming ssl2 disabled for NodeManager at omshost.domain.com:7403... OK
Confirming ssl2 disabled for BIPublisherOHS at omshost.domain.com:9852... OK
Confirming ssl2 disabled for OMSconsole at omshost.domain.com:7802... OK
Confirming ssl2 disabled for OMSproxy at omshost.domain.com:7301... OK
Confirming ssl2 disabled for OMSupload at omshost.domain.com:4903... OK
Confirming ssl2 disabled for WLSadmin at omshost.domain.com:7102... OK

Checking SSLv2 on all agents

Confirming ssl2 disabled for Agent at host01.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host02.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host04.usa.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host03.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host05.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host06.domain.com:1830... OK
Confirming ssl2 disabled for Agent at host07.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host08.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host09.domain.com:1830... OK
Confirming ssl2 disabled for Agent at host10.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host11.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host12.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host13.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host14.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host15.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host16.domain.com:3872... OK
Confirming ssl2 disabled for Agent at omshost.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host17.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host18.domain.com:3872... OK

(1b) Forbid SSLv3 connections
Confirming ssl3 disabled for Agent at omshost.domain.com:3872... OK
Confirming ssl3 disabled for BIPublisher at omshost.domain.com:9803... OK
Confirming ssl3 disabled for NodeManager at omshost.domain.com:7403... OK
Confirming ssl3 disabled for BIPublisherOHS at omshost.domain.com:9852... OK
Confirming ssl3 disabled for OMSconsole at omshost.domain.com:7802... OK
Confirming ssl3 disabled for OMSproxy at omshost.domain.com:7301... OK
Confirming ssl3 disabled for OMSupload at omshost.domain.com:4903... OK
Confirming ssl3 disabled for WLSadmin at omshost.domain.com:7102... OK

Checking SSLv3 on all agents

Confirming ssl3 disabled for Agent at host01.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host02.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host04.usa.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host03.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host05.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host06.domain.com:1830... OK
Confirming ssl3 disabled for Agent at host07.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host08.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host09.domain.com:1830... OK
Confirming ssl3 disabled for Agent at host10.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host11.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host12.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host13.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host14.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host15.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host16.domain.com:3872... OK
Confirming ssl3 disabled for Agent at omshost.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host17.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host18.domain.com:3872... OK

(1c) Forbid TLSv1 connections
Confirming tls1 disabled for Agent at omshost.domain.com:3872... OK
Confirming tls1 disabled for BIPublisher at omshost.domain.com:9803... OK
Confirming tls1 disabled for NodeManager at omshost.domain.com:7403... OK
Confirming tls1 disabled for BIPublisherOHS at omshost.domain.com:9852... OK
Confirming tls1 disabled for OMSconsole at omshost.domain.com:7802... OK
Confirming tls1 disabled for OMSproxy at omshost.domain.com:7301... OK
Confirming tls1 disabled for OMSupload at omshost.domain.com:4903... OK
Confirming tls1 disabled for WLSadmin at omshost.domain.com:7102... OK

Checking TLSv1 on all agents

Confirming tls1 disabled for Agent at host01.domain.com:3872... OK
Confirming tls1 disabled for Agent at host02.domain.com:3872... OK
Confirming tls1 disabled for Agent at host04.usa.domain.com:3872... OK
Confirming tls1 disabled for Agent at host03.domain.com:3872... OK
Confirming tls1 disabled for Agent at host05.domain.com:3872... OK
Confirming tls1 disabled for Agent at host06.domain.com:1830... OK
Confirming tls1 disabled for Agent at host07.domain.com:3872... OK
Confirming tls1 disabled for Agent at host08.domain.com:3872... OK
Confirming tls1 disabled for Agent at host09.domain.com:1830... OK
Confirming tls1 disabled for Agent at host10.domain.com:3872... OK
Confirming tls1 disabled for Agent at host11.domain.com:3872... OK
Confirming tls1 disabled for Agent at host12.domain.com:3872... OK
Confirming tls1 disabled for Agent at host13.domain.com:3872... OK
Confirming tls1 disabled for Agent at host14.domain.com:3872... OK
Confirming tls1 disabled for Agent at host15.domain.com:3872... OK
Confirming tls1 disabled for Agent at host16.domain.com:3872... OK
Confirming tls1 disabled for Agent at omshost.domain.com:3872... OK
Confirming tls1 disabled for Agent at host17.domain.com:3872... OK
Confirming tls1 disabled for Agent at host18.domain.com:3872... OK

(1d) Forbid TLSv1.1 connections
Confirming tls1_1 disabled for Agent at omshost.domain.com:3872... OK
Confirming tls1_1 disabled for BIPublisher at omshost.domain.com:9803... OK
Confirming tls1_1 disabled for NodeManager at omshost.domain.com:7403... OK
Confirming tls1_1 disabled for BIPublisherOHS at omshost.domain.com:9852... OK
Confirming tls1_1 disabled for OMSconsole at omshost.domain.com:7802... OK
Confirming tls1_1 disabled for OMSproxy at omshost.domain.com:7301... OK
Confirming tls1_1 disabled for OMSupload at omshost.domain.com:4903... OK
Confirming tls1_1 disabled for WLSadmin at omshost.domain.com:7102... OK

Checking TLSv1.1 on all agents

Confirming tls1_1 disabled for Agent at host01.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host02.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host04.usa.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host03.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host05.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host06.domain.com:1830... OK
Confirming tls1_1 disabled for Agent at host07.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host08.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host09.domain.com:1830... OK
Confirming tls1_1 disabled for Agent at host10.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host11.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host12.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host13.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host14.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host15.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host16.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at omshost.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host17.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host18.domain.com:3872... OK

(1e) Permit TLSv1.2 connections
Confirming tls1_2 available for Agent at omshost.domain.com:3872... OK
Confirming tls1_2 available for BIPublisher at omshost.domain.com:9803... OK
Confirming tls1_2 available for NodeManager at omshost.domain.com:7403... OK
Confirming tls1_2 available for BIPublisherOHS at omshost.domain.com:9852... OK
Confirming tls1_2 available for OMSconsole at omshost.domain.com:7802... OK
Confirming tls1_2 available for OMSproxy at omshost.domain.com:7301... OK
Confirming tls1_2 available for OMSupload at omshost.domain.com:4903... OK
Confirming tls1_2 available for WLSadmin at omshost.domain.com:7102... OK

Checking TLSv1.2 on all agents

Confirming tls1_2 available for Agent at host01.domain.com:3872... OK
Confirming tls1_2 available for Agent at host02.domain.com:3872... OK
Confirming tls1_2 available for Agent at host04.usa.domain.com:3872... OK
Confirming tls1_2 available for Agent at host03.domain.com:3872... OK
Confirming tls1_2 available for Agent at host05.domain.com:3872... OK
Confirming tls1_2 available for Agent at host06.domain.com:1830... OK
Confirming tls1_2 available for Agent at host07.domain.com:3872... OK
Confirming tls1_2 available for Agent at host08.domain.com:3872... OK
Confirming tls1_2 available for Agent at host09.domain.com:1830... OK
Confirming tls1_2 available for Agent at host10.domain.com:3872... OK
Confirming tls1_2 available for Agent at host11.domain.com:3872... OK
Confirming tls1_2 available for Agent at host12.domain.com:3872... OK
Confirming tls1_2 available for Agent at host13.domain.com:3872... OK
Confirming tls1_2 available for Agent at host14.domain.com:3872... OK
Confirming tls1_2 available for Agent at host15.domain.com:3872... OK
Confirming tls1_2 available for Agent at host16.domain.com:3872... OK
Confirming tls1_2 available for Agent at omshost.domain.com:3872... OK
Confirming tls1_2 available for Agent at host17.domain.com:3872... OK
Confirming tls1_2 available for Agent at host18.domain.com:3872... OK

(2) Checking supported ciphers at SSL/TLS endpoints (see notes 2138391.1, 1067411.1)
(2a) Checking LOW strength ciphers on Agent (omshost.domain.com:3872, protocol tls1_2)... OK
(2a) Checking MEDIUM strength ciphers on Agent (omshost.domain.com:3872)... OK
(2a) Checking HIGH strength ciphers on Agent (omshost.domain.com:3872)... OK

(2b) Checking LOW strength ciphers on BIPublisher (omshost.domain.com:9803, protocol tls1_2)... OK
(2b) Checking MEDIUM strength ciphers on BIPublisher (omshost.domain.com:9803)... OK
(2b) Checking HIGH strength ciphers on BIPublisher (omshost.domain.com:9803)... OK

(2c) Checking LOW strength ciphers on NodeManager (omshost.domain.com:7403, protocol tls1_2)... OK
(2c) Checking MEDIUM strength ciphers on NodeManager (omshost.domain.com:7403)... OK
(2c) Checking HIGH strength ciphers on NodeManager (omshost.domain.com:7403)... OK

(2d) Checking LOW strength ciphers on BIPublisherOHS (omshost.domain.com:9852, protocol tls1_2)... OK
(2d) Checking MEDIUM strength ciphers on BIPublisherOHS (omshost.domain.com:9852)... OK
(2d) Checking HIGH strength ciphers on BIPublisherOHS (omshost.domain.com:9852)... OK

(2e) Checking LOW strength ciphers on OMSconsole (omshost.domain.com:7802, protocol tls1_2)... OK
(2e) Checking MEDIUM strength ciphers on OMSconsole (omshost.domain.com:7802)... OK
(2e) Checking HIGH strength ciphers on OMSconsole (omshost.domain.com:7802)... OK

(2f) Checking LOW strength ciphers on OMSproxy (omshost.domain.com:7301, protocol tls1_2)... OK
(2f) Checking MEDIUM strength ciphers on OMSproxy (omshost.domain.com:7301)... OK
(2f) Checking HIGH strength ciphers on OMSproxy (omshost.domain.com:7301)... OK

(2g) Checking LOW strength ciphers on OMSupload (omshost.domain.com:4903, protocol tls1_2)... OK
(2g) Checking MEDIUM strength ciphers on OMSupload (omshost.domain.com:4903)... OK
(2g) Checking HIGH strength ciphers on OMSupload (omshost.domain.com:4903)... OK

(2h) Checking LOW strength ciphers on WLSadmin (omshost.domain.com:7102, protocol tls1_2)... OK
(2h) Checking MEDIUM strength ciphers on WLSadmin (omshost.domain.com:7102)... OK
(2h) Checking HIGH strength ciphers on WLSadmin (omshost.domain.com:7102)... OK

Checking supported ciphers on all agents

(2i) Checking LOW strength ciphers on Agent (host01.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host01.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host01.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host02.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host02.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host02.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host04.usa.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host04.usa.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host04.usa.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host03.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host03.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host03.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host05.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host05.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host05.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host06.domain.com:1830, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host06.domain.com:1830)... OK
(2i) Checking HIGH strength ciphers on Agent (host06.domain.com:1830)... OK

(2i) Checking LOW strength ciphers on Agent (host07.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host07.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host07.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host08.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host08.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host08.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host09.domain.com:1830, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host09.domain.com:1830)... OK
(2i) Checking HIGH strength ciphers on Agent (host09.domain.com:1830)... OK

(2i) Checking LOW strength ciphers on Agent (host10.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host10.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host10.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host11.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host11.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host11.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host12.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host12.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host12.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host13.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host13.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host13.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host14.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host14.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host14.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host15.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host15.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host15.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host16.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host16.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host16.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (omshost.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (omshost.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (omshost.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host17.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host17.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host17.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host18.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host18.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host18.domain.com:3872)... OK

(3) Checking self-signed and demonstration certificates at SSL/TLS endpoints (see notes 2202569.1, 1367988.1, 1914184.1, 2213661.1, 2220788.1, 123033.1, 1937457.1)

(3a) Checking for self-signed certificates on OMS components
Checking certificate at Agent (omshost.domain.com:3872, protocol tls1_2)... OK
Checking certificate at BIPublisherOHS (omshost.domain.com:9852, protocol tls1_2)... OK
Checking certificate at BIPublisher (omshost.domain.com:9803, protocol tls1_2)... OK
Checking certificate at NodeManager (omshost.domain.com:7403, protocol tls1_2)... OK
Checking certificate at OMSconsole (omshost.domain.com:7802, protocol tls1_2)... OK
Checking certificate at OMSproxy (omshost.domain.com:7301, protocol tls1_2)... OK
Checking certificate at OMSupload (omshost.domain.com:4903, protocol tls1_2)... OK
Checking certificate at WLSadmin (omshost.domain.com:7102, protocol tls1_2)... OK

(3b) Checking for demonstration certificates on OMS components
Checking demo certificate at Agent (omshost.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at BIPublisherOHS (omshost.domain.com:9852, protocol tls1_2)... OK
Checking demo certificate at BIPublisher (omshost.domain.com:9803, protocol tls1_2)... OK
Checking demo certificate at NodeManager (omshost.domain.com:7403, protocol tls1_2)... OK
Checking demo certificate at OMSconsole (omshost.domain.com:7802, protocol tls1_2)... OK
Checking demo certificate at OMSproxy (omshost.domain.com:7301, protocol tls1_2)... OK
Checking demo certificate at OMSupload (omshost.domain.com:4903, protocol tls1_2)... OK
Checking demo certificate at WLSadmin (omshost.domain.com:7102, protocol tls1_2)... OK

(3c) Checking for self-signed certificates on all agents

Checking certificate at Agent (host01.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host02.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host04.usa.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host03.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host05.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host06.domain.com:1830, protocol tls1_2)... OK
Checking certificate at Agent (host07.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host08.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host09.domain.com:1830, protocol tls1_2)... OK
Checking certificate at Agent (host10.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host11.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host12.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host13.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host14.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host15.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host16.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (omshost.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host17.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host18.domain.com:3872, protocol tls1_2)... OK

(3d) Checking for demonstration certificates on all agents

Checking demo certificate at Agent (host01.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host02.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host04.usa.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host03.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host05.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host06.domain.com:1830, protocol tls1_2)... OK
Checking demo certificate at Agent (host07.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host08.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host09.domain.com:1830, protocol tls1_2)... OK
Checking demo certificate at Agent (host10.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host11.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host12.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host13.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host14.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host15.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host16.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (omshost.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host17.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host18.domain.com:3872, protocol tls1_2)... OK

(4) Checking EM13c Oracle home patch levels against 18 Apr 2017 baseline (see notes 1664074.1, 2219797.1, 822485.1, 1470197.1)

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) DATABASE BUNDLE PATCH: 12.1.0.2.170418 (APR2017) (25397136)... OK

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) Database PSU 12.1.0.2.170418, Oracle JavaVM Component (APR2017) (25437695)... OK

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) OCW Interim patch for 25481150 (25481150)... OK

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) EM QUERY WITH SQL_ID 4RQ83FNXTF39U PERFORMS POORLY ON ORACLE 12C RELATIVE TO 11G (20243268)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_TYPES_SERVER parameter (76629.1, 2167682.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_SERVER parameter (76629.1, 2167682.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_TYPES_CLIENT parameter (76629.1, 2167682.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_CLIENT parameter (76629.1, 2167682.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_TYPES_SERVER parameter (76629.1, 2167682.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_SERVER parameter (76629.1, 2167682.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_TYPES_CLIENT parameter (76629.1, 2167682.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_CLIENT parameter (76629.1, 2167682.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SSL_VERSION parameter (1545816.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SSL_CIPHER_SUITES parameter (1545816.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) listener.ora SSL_VERSION parameter (1545816.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) listener.ora SSL_CIPHER_SUITES parameter (1545816.1)... OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) ENTERPRISE MANAGER BASE PLATFORM - OMS 13.2.0.0.170418 PSU (25387277)... OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) TRACKING BUG TO REGISTER META VERSION FROM PS4 AND 13.1 BUNDLE PATCHES IN 13.2 (SYSTEM PATCH) (23603592)... OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) MERGE REQUEST ON TOP OF 12.1.3.0.0 FOR BUGS 24571979 24335626 (25322055)... OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) MERGE REQUEST ON TOP OF 12.1.3.0.0 FOR BUGS 22557350 19901079 20222451 (24329181)... OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) MERGE REQUEST ON TOP OF 12.1.3.0.0 FOR BUGS 19485414 20022048 (21849941)... OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) OPSS BUNDLE PATCH 12.1.3.0.170418 (22748215)... OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) ENTERPRISE MANAGER FOR OMS PLUGINS 13.2.0.0.170331 (Not used for 13.2.2 plugins) (25672422)... OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) WLS PATCH SET UPDATE 12.1.3.0.170418 (25388793)... OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) TOPLINK SECURITY PATCH UPDATE CPUJUL2016 (24327938)... OK

Using EMCLI to check for agent bundle patch on all agents

(4d) Agent host01.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host02.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host04.usa.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host03.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host05.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host06.domain.com:1830 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host07.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host08.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host09.domain.com:1830 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host10.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host11.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host12.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host13.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host14.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host15.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host16.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent omshost.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host17.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host18.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(5) Checking EM13cR2 Java patch levels against 18 Apr 2017 baseline (see notes 1506916.1, 2241373.1, 2241358.1)

(5a) Common Java (/oracle/oem/Middleware13cR2/oracle_common/jdk) JAVA SE JDK VERSION 1.7.0_141 (13079846)... OK

Using EMCLI to check Java patch levels on all agents

(5b) Agent host01.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host02.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host04.usa.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host03.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host05.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host06.domain.com:1830 Java VERSION 1.7.0_141... OK

(5b) Agent host07.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host08.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host09.domain.com:1830 Java VERSION 1.7.0_141... OK

(5b) Agent host10.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host11.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host12.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host13.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host14.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host15.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host16.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent omshost.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host17.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host18.domain.com:3872 Java VERSION 1.7.0_141... OK

(6) Checking EM13cR2 OPatch/OMSPatcher patch levels against 18 Apr 2017 requirements (see patch 25197714 README, patches 6880880 and 19999993)

(6a) OMS OPatch (/oracle/oem/Middleware13cR2/OPatch) VERSION 13.9.1.3.0 or newer... OK

(6b) OMSPatcher (/oracle/oem/Middleware13cR2/OPatch) VERSION 13.8.0.0.2 or newer... OK

Checking OPatch patch levels on all agents

(6c) Agent host01.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host02.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host04.usa.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host03.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host05.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host06.domain.com:1830 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host07.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host08.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host09.domain.com:1830 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host10.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host11.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host12.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host13.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host14.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host15.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host16.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent omshost.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host17.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host18.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(7) Agent plugin bundle patch checks on all agents...
(7a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host01.domain.com:3872 (25501452)... OK - plugin not installed

(7b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host01.domain.com:3872 (25197692)... OK - plugin not installed

(7c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host01.domain.com:3872 (25501427)... OK - plugin not installed

(7d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host01.domain.com:3872 (25501430)... OK

(7e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host01.domain.com:3872 (25501408)... OK - plugin not installed

(7f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host01.domain.com:3872 (25162444)... OK - plugin not installed

(7g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host01.domain.com:3872 (25501436)... OK

(7h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host01.domain.com:3872 (25362875)... OK - plugin not installed

(7i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host01.domain.com:3872 (25522944)... OK - plugin not installed

(7j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host01.domain.com:3872 (25501416)... OK - plugin not installed

(7k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host01.domain.com:3872 (25362898)... OK - plugin not installed

(7l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host01.domain.com:3872 (25362890)... OK - plugin not installed

(7m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host01.domain.com:3872 (25197712)... OK - plugin not installed

(8a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host02.domain.com:3872 (25501452)... OK - plugin not installed

(8b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host02.domain.com:3872 (25197692)... OK - plugin not installed

(8c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host02.domain.com:3872 (25501427)... OK - plugin not installed

(8d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host02.domain.com:3872 (25501430)... OK

(8e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host02.domain.com:3872 (25501408)... OK - plugin not installed

(8f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host02.domain.com:3872 (25162444)... OK - plugin not installed

(8g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host02.domain.com:3872 (25501436)... OK

(8h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host02.domain.com:3872 (25362875)... OK - plugin not installed

(8i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host02.domain.com:3872 (25522944)... OK - plugin not installed

(8j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host02.domain.com:3872 (25501416)... OK - plugin not installed

(8k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host02.domain.com:3872 (25362898)... OK - plugin not installed

(8l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host02.domain.com:3872 (25362890)... OK - plugin not installed

(8m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host02.domain.com:3872 (25197712)... OK - plugin not installed

(9a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host04.usa.domain.com:3872 (25501452)... OK - plugin not installed

(9b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host04.usa.domain.com:3872 (25197692)... OK - plugin not installed

(9c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host04.usa.domain.com:3872 (25501427)... OK - plugin not installed

(9d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host04.usa.domain.com:3872 (25501430)... OK - plugin not installed

(9e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host04.usa.domain.com:3872 (25501408)... OK - plugin not installed

(9f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host04.usa.domain.com:3872 (25162444)... OK - plugin not installed

(9g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host04.usa.domain.com:3872 (25501436)... OK - plugin not installed

(9h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host04.usa.domain.com:3872 (25362875)... OK - plugin not installed

(9i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host04.usa.domain.com:3872 (25522944)... OK - plugin not installed

(9j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host04.usa.domain.com:3872 (25501416)... OK - plugin not installed

(9k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host04.usa.domain.com:3872 (25362898)... OK - plugin not installed

(9l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host04.usa.domain.com:3872 (25362890)... OK - plugin not installed

(9m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host04.usa.domain.com:3872 (25197712)... OK - plugin not installed

(10a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host03.domain.com:3872 (25501452)... OK - plugin not installed

(10b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host03.domain.com:3872 (25197692)... OK - plugin not installed

(10c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host03.domain.com:3872 (25501427)... OK - plugin not installed

(10d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host03.domain.com:3872 (25501430)... OK

(10e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host03.domain.com:3872 (25501408)... OK - plugin not installed

(10f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host03.domain.com:3872 (25162444)... OK - plugin not installed

(10g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host03.domain.com:3872 (25501436)... OK

(10h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host03.domain.com:3872 (25362875)... OK - plugin not installed

(10i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host03.domain.com:3872 (25522944)... OK - plugin not installed

(10j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host03.domain.com:3872 (25501416)... OK - plugin not installed

(10k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host03.domain.com:3872 (25362898)... OK - plugin not installed

(10l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host03.domain.com:3872 (25362890)... OK - plugin not installed

(10m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host03.domain.com:3872 (25197712)... OK - plugin not installed

(11a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host05.domain.com:3872 (25501452)... OK - plugin not installed

(11b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host05.domain.com:3872 (25197692)... OK - plugin not installed

(11c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host05.domain.com:3872 (25501427)... OK - plugin not installed

(11d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host05.domain.com:3872 (25501430)... OK

(11e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host05.domain.com:3872 (25501408)... OK - plugin not installed

(11f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host05.domain.com:3872 (25162444)... OK - plugin not installed

(11g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host05.domain.com:3872 (25501436)... OK

(11h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host05.domain.com:3872 (25362875)... OK - plugin not installed

(11i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host05.domain.com:3872 (25522944)... OK - plugin not installed

(11j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host05.domain.com:3872 (25501416)... OK - plugin not installed

(11k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host05.domain.com:3872 (25362898)... OK - plugin not installed

(11l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host05.domain.com:3872 (25362890)... OK - plugin not installed

(11m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host05.domain.com:3872 (25197712)... OK - plugin not installed

(12a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host06.domain.com:1830 (25501452)... OK - plugin not installed

(12b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host06.domain.com:1830 (25197692)... OK - plugin not installed

(12c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host06.domain.com:1830 (25501427)... OK - plugin not installed

(12d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host06.domain.com:1830 (25501430)... OK

(12e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host06.domain.com:1830 (25501408)... OK - plugin not installed

(12f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host06.domain.com:1830 (25162444)... OK - plugin not installed

(12g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host06.domain.com:1830 (25501436)... OK - plugin not installed

(12h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host06.domain.com:1830 (25362875)... OK - plugin not installed

(12i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host06.domain.com:1830 (25522944)... OK - plugin not installed

(12j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host06.domain.com:1830 (25501416)... OK - plugin not installed

(12k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host06.domain.com:1830 (25362898)... OK - plugin not installed

(12l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host06.domain.com:1830 (25362890)... OK - plugin not installed

(12m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host06.domain.com:1830 (25197712)... OK - plugin not installed

(13a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host07.domain.com:3872 (25501452)... OK - plugin not installed

(13b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host07.domain.com:3872 (25197692)... OK - plugin not installed

(13c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host07.domain.com:3872 (25501427)... OK - plugin not installed

(13d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host07.domain.com:3872 (25501430)... OK

(13e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host07.domain.com:3872 (25501408)... OK - plugin not installed

(13f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host07.domain.com:3872 (25162444)... OK - plugin not installed

(13g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host07.domain.com:3872 (25501436)... OK

(13h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host07.domain.com:3872 (25362875)... OK - plugin not installed

(13i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host07.domain.com:3872 (25522944)... OK - plugin not installed

(13j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host07.domain.com:3872 (25501416)... OK - plugin not installed

(13k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host07.domain.com:3872 (25362898)... OK - plugin not installed

(13l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host07.domain.com:3872 (25362890)... OK - plugin not installed

(13m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host07.domain.com:3872 (25197712)... OK - plugin not installed

(14a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host08.domain.com:3872 (25501452)... OK - plugin not installed

(14b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host08.domain.com:3872 (25197692)... OK - plugin not installed

(14c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host08.domain.com:3872 (25501427)... OK - plugin not installed

(14d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host08.domain.com:3872 (25501430)... OK

(14e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host08.domain.com:3872 (25501408)... OK - plugin not installed

(14f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host08.domain.com:3872 (25162444)... OK - plugin not installed

(14g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host08.domain.com:3872 (25501436)... OK

(14h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host08.domain.com:3872 (25362875)... OK - plugin not installed

(14i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host08.domain.com:3872 (25522944)... OK - plugin not installed

(14j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host08.domain.com:3872 (25501416)... OK - plugin not installed

(14k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host08.domain.com:3872 (25362898)... OK - plugin not installed

(14l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host08.domain.com:3872 (25362890)... OK - plugin not installed

(14m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host08.domain.com:3872 (25197712)... OK - plugin not installed

(15a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host09.domain.com:1830 (25501452)... OK - plugin not installed

(15b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host09.domain.com:1830 (25197692)... OK - plugin not installed

(15c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host09.domain.com:1830 (25501427)... OK - plugin not installed

(15d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host09.domain.com:1830 (25501430)... OK

(15e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host09.domain.com:1830 (25501408)... OK - plugin not installed

(15f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host09.domain.com:1830 (25162444)... OK - plugin not installed

(15g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host09.domain.com:1830 (25501436)... OK

(15h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host09.domain.com:1830 (25362875)... OK - plugin not installed

(15i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host09.domain.com:1830 (25522944)... OK - plugin not installed

(15j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host09.domain.com:1830 (25501416)... OK - plugin not installed

(15k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host09.domain.com:1830 (25362898)... OK - plugin not installed

(15l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host09.domain.com:1830 (25362890)... OK - plugin not installed

(15m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host09.domain.com:1830 (25197712)... OK - plugin not installed

(16a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host10.domain.com:3872 (25501452)... OK - plugin not installed

(16b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host10.domain.com:3872 (25197692)... OK - plugin not installed

(16c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host10.domain.com:3872 (25501427)... OK - plugin not installed

(16d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host10.domain.com:3872 (25501430)... OK

(16e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host10.domain.com:3872 (25501408)... OK - plugin not installed

(16f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host10.domain.com:3872 (25162444)... OK - plugin not installed

(16g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host10.domain.com:3872 (25501436)... OK

(16h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host10.domain.com:3872 (25362875)... OK - plugin not installed

(16i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host10.domain.com:3872 (25522944)... OK - plugin not installed

(16j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host10.domain.com:3872 (25501416)... OK - plugin not installed

(16k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host10.domain.com:3872 (25362898)... OK - plugin not installed

(16l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host10.domain.com:3872 (25362890)... OK - plugin not installed

(16m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host10.domain.com:3872 (25197712)... OK - plugin not installed

(17a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host11.domain.com:3872 (25501452)... OK - plugin not installed

(17b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host11.domain.com:3872 (25197692)... OK - plugin not installed

(17c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host11.domain.com:3872 (25501427)... OK - plugin not installed

(17d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host11.domain.com:3872 (25501430)... OK

(17e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host11.domain.com:3872 (25501408)... OK - plugin not installed

(17f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host11.domain.com:3872 (25162444)... OK - plugin not installed

(17g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host11.domain.com:3872 (25501436)... OK

(17h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host11.domain.com:3872 (25362875)... OK - plugin not installed

(17i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host11.domain.com:3872 (25522944)... OK - plugin not installed

(17j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host11.domain.com:3872 (25501416)... OK - plugin not installed

(17k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host11.domain.com:3872 (25362898)... OK - plugin not installed

(17l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host11.domain.com:3872 (25362890)... OK - plugin not installed

(17m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host11.domain.com:3872 (25197712)... OK - plugin not installed

(18a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host12.domain.com:3872 (25501452)... OK - plugin not installed

(18b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host12.domain.com:3872 (25197692)... OK - plugin not installed

(18c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host12.domain.com:3872 (25501427)... OK - plugin not installed

(18d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host12.domain.com:3872 (25501430)... OK

(18e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host12.domain.com:3872 (25501408)... OK - plugin not installed

(18f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host12.domain.com:3872 (25162444)... OK - plugin not installed

(18g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host12.domain.com:3872 (25501436)... OK

(18h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host12.domain.com:3872 (25362875)... OK - plugin not installed

(18i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host12.domain.com:3872 (25522944)... OK - plugin not installed

(18j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host12.domain.com:3872 (25501416)... OK - plugin not installed

(18k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host12.domain.com:3872 (25362898)... OK - plugin not installed

(18l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host12.domain.com:3872 (25362890)... OK - plugin not installed

(18m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host12.domain.com:3872 (25197712)... OK - plugin not installed

(19a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host13.domain.com:3872 (25501452)... OK - plugin not installed

(19b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host13.domain.com:3872 (25197692)... OK - plugin not installed

(19c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host13.domain.com:3872 (25501427)... OK - plugin not installed

(19d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host13.domain.com:3872 (25501430)... OK

(19e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host13.domain.com:3872 (25501408)... OK - plugin not installed

(19f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host13.domain.com:3872 (25162444)... OK - plugin not installed

(19g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host13.domain.com:3872 (25501436)... OK

(19h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host13.domain.com:3872 (25362875)... OK - plugin not installed

(19i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host13.domain.com:3872 (25522944)... OK - plugin not installed

(19j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host13.domain.com:3872 (25501416)... OK - plugin not installed

(19k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host13.domain.com:3872 (25362898)... OK - plugin not installed

(19l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host13.domain.com:3872 (25362890)... OK - plugin not installed

(19m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host13.domain.com:3872 (25197712)... OK - plugin not installed

(20a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host14.domain.com:3872 (25501452)... OK - plugin not installed

(20b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host14.domain.com:3872 (25197692)... OK - plugin not installed

(20c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host14.domain.com:3872 (25501427)... OK - plugin not installed

(20d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host14.domain.com:3872 (25501430)... OK

(20e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host14.domain.com:3872 (25501408)... OK - plugin not installed

(20f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host14.domain.com:3872 (25162444)... OK - plugin not installed

(20g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host14.domain.com:3872 (25501436)... OK

(20h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host14.domain.com:3872 (25362875)... OK - plugin not installed

(20i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host14.domain.com:3872 (25522944)... OK - plugin not installed

(20j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host14.domain.com:3872 (25501416)... OK - plugin not installed

(20k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host14.domain.com:3872 (25362898)... OK - plugin not installed

(20l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host14.domain.com:3872 (25362890)... OK - plugin not installed

(20m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host14.domain.com:3872 (25197712)... OK - plugin not installed

(21a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host15.domain.com:3872 (25501452)... OK - plugin not installed

(21b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host15.domain.com:3872 (25197692)... OK - plugin not installed

(21c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host15.domain.com:3872 (25501427)... OK - plugin not installed

(21d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host15.domain.com:3872 (25501430)... OK

(21e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host15.domain.com:3872 (25501408)... OK - plugin not installed

(21f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host15.domain.com:3872 (25162444)... OK - plugin not installed

(21g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host15.domain.com:3872 (25501436)... OK

(21h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host15.domain.com:3872 (25362875)... OK - plugin not installed

(21i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host15.domain.com:3872 (25522944)... OK - plugin not installed

(21j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host15.domain.com:3872 (25501416)... OK - plugin not installed

(21k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host15.domain.com:3872 (25362898)... OK - plugin not installed

(21l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host15.domain.com:3872 (25362890)... OK - plugin not installed

(21m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host15.domain.com:3872 (25197712)... OK - plugin not installed

(22a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host16.domain.com:3872 (25501452)... OK - plugin not installed

(22b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host16.domain.com:3872 (25197692)... OK - plugin not installed

(22c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host16.domain.com:3872 (25501427)... OK - plugin not installed

(22d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host16.domain.com:3872 (25501430)... OK

(22e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host16.domain.com:3872 (25501408)... OK - plugin not installed

(22f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host16.domain.com:3872 (25162444)... OK - plugin not installed

(22g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host16.domain.com:3872 (25501436)... OK

(22h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host16.domain.com:3872 (25362875)... OK - plugin not installed

(22i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host16.domain.com:3872 (25522944)... OK - plugin not installed

(22j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host16.domain.com:3872 (25501416)... OK - plugin not installed

(22k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host16.domain.com:3872 (25362898)... OK - plugin not installed

(22l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host16.domain.com:3872 (25362890)... OK - plugin not installed

(22m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host16.domain.com:3872 (25197712)... OK - plugin not installed

(23a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ omshost.domain.com:3872 (25501452)... OK - plugin not installed

(23b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ omshost.domain.com:3872 (25197692)... OK - plugin not installed

(23c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ omshost.domain.com:3872 (25501427)... OK - plugin not installed

(23d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ omshost.domain.com:3872 (25501430)... OK - plugin not installed

(23e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ omshost.domain.com:3872 (25501408)... OK - plugin not installed

(23f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ omshost.domain.com:3872 (25162444)... OK

(23g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ omshost.domain.com:3872 (25501436)... OK

(23h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ omshost.domain.com:3872 (25362875)... OK - plugin not installed

(23i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ omshost.domain.com:3872 (25522944)... OK - plugin not installed

(23j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ omshost.domain.com:3872 (25501416)... OK - plugin not installed

(23k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ omshost.domain.com:3872 (25362898)... OK - plugin not installed

(23l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ omshost.domain.com:3872 (25362890)... OK - plugin not installed

(23m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ omshost.domain.com:3872 (25197712)... OK - plugin not installed

(24a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host17.domain.com:3872 (25501452)... OK - plugin not installed

(24b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host17.domain.com:3872 (25197692)... OK - plugin not installed

(24c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host17.domain.com:3872 (25501427)... OK - plugin not installed

(24d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host17.domain.com:3872 (25501430)... OK

(24e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host17.domain.com:3872 (25501408)... OK - plugin not installed

(24f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host17.domain.com:3872 (25162444)... OK - plugin not installed

(24g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host17.domain.com:3872 (25501436)... OK

(24h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host17.domain.com:3872 (25362875)... OK - plugin not installed

(24i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host17.domain.com:3872 (25522944)... OK - plugin not installed

(24j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host17.domain.com:3872 (25501416)... OK - plugin not installed

(24k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host17.domain.com:3872 (25362898)... OK - plugin not installed

(24l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host17.domain.com:3872 (25362890)... OK - plugin not installed

(24m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host17.domain.com:3872 (25197712)... OK - plugin not installed

(25a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host18.domain.com:3872 (25501452)... OK - plugin not installed

(25b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host18.domain.com:3872 (25197692)... OK - plugin not installed

(25c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host18.domain.com:3872 (25501427)... OK - plugin not installed

(25d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host18.domain.com:3872 (25501430)... OK

(25e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host18.domain.com:3872 (25501408)... OK - plugin not installed

(25f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host18.domain.com:3872 (25162444)... OK - plugin not installed

(25g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host18.domain.com:3872 (25501436)... OK

(25h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host18.domain.com:3872 (25362875)... OK - plugin not installed

(25i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host18.domain.com:3872 (25522944)... OK - plugin not installed

(25j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host18.domain.com:3872 (25501416)... OK - plugin not installed

(25k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host18.domain.com:3872 (25362898)... OK - plugin not installed

(25l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host18.domain.com:3872 (25362890)... OK - plugin not installed

(25m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host18.domain.com:3872 (25197712)... OK - plugin not installed

Cleaning up temporary files... done
All tests succeeded.

Visit https://pardydba.wordpress.com/2016/10/28/securing-oracle-enterprise-manager-13cr2/ for more information.
Download the latest release from https://raw.githubusercontent.com/brianpardy/em13c/master/checksec13R2.sh
Download the latest beta release from https://raw.githubusercontent.com/brianpardy/em13c/beta/checksec13R2.sh

Example Output – create_user_for_checksec13R2.sh


Welcome to ./create_user_for_checksec13R2.sh, version 1.0, released 20170314.

Download the latest release of this script at any time from:
https://raw.githubusercontent.com/brianpardy/em13c/master/create_user_for_checksec13R2.sh

This script exists to supplement checksec13R2.sh and enable additional checks. When run, this
script will create a user named CHECKSEC in your EM13cR2 environment and give that user a
random password, which gets printed to the screen at the end of the script. The script then
grants CHECKSEC VIEW_ANY_TARGET and EM_ALL_OPERATOR privilege, and then prompts you to supply
the names of credentials existing in your EM13cR2 environment. The script validates the names of
credentials supplied, grants VIEW access to CHECKSEC for each credential, and assigns those
credentials as preferred credentials for CHECKSEC for each relevant target.

Providing credentials for the repository database enables the following additional checks in
checksec13R2.sh:
* Check for presence/absence of plugin bundle patches on all agents

Providing host credentials for every monitored host running an agent enables the following
additional checks in checksec13R2.sh:
* Check for presence/absence of the latest Java version on all agents

Login to EMCLI as SYSMAN before running this script. If you already have an CHECKSEC account,
running this script will delete and recreate it with a new password.

Continue? [y/n]
Continuing...

Synchronized successfully
User "CHECKSEC" deleted successfully

User "CHECKSEC" created successfully

Created user CHECKSEC with password: [redacted]

Now CHECKSEC needs preferred credentials for the repository DB and repository DB host.
Your repository DB target name is oemdb.domain.com
Enter the credential name for the repository DB Normal Database Credentials: DB-OEMDB-SYSTEM
Enter the credential name for the repository DB SYSDBA Database Credentials: DB-OEMDB-SYS
Enter the credential name for the repository DB Database Host Credentials: HOST-OMSHOST-ORACLE

Validating that supplied credentials exist.

Credentials "DB-OEMDB-SYSTEM:SYSMAN" tested successfully
Credentials "DB-OEMDB-SYS:SYSMAN" tested successfully
Credentials "HOST-OMSHOST-ORACLE:SYSMAN" tested successfully

Granting CHECKSEC GET_CREDENTIAL access to supplied credentials.
Privileges granted to user/role "CHECKSEC" successfully

Confirmed supplied credentials exist and granted to CHECKSEC.

The next section asks you to supply credentials for the account used to run the Oracle Management Agent.

You will receive a separate prompt for each agent. Enter 'done' (without quotes) to skip this step.

If you provide these credentials, checksec13R2.sh can report on the Java version used by your agents.

Generating a list of all agent targets.
Now loop through all agent targets and provide named credentials for the agent user account on each host.

Enter the credential name to login as the agent user for host1.domain.com:3872: HOST-HOST1-ORAAGENT
Credentials "HOST-HOST1-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host2.domain.com:3872: HOST-HOST2-ORAAGENT
Credentials "HOST-HOST2-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host3.domain.com:3872: HOST-HOST3-ORAAGENT
Credentials "HOST-HOST3-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host4.domain.com:1830: HOST-HOST4-ORAAGENT
Credentials "HOST-HOST4-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host5.domain.com:3872: HOST-HOST5-ORAAGENT
Credentials "HOST-HOST5-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host6.domain.com:1830: HOST-HOST6-ORAAGENT
Credentials "HOST-HOST6-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host7.domain.com:3872: HOST-HOST7-ORAAGENT
Credentials "HOST-HOST7-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host8.domain.com:3872: HOST-HOST8-ORAAGENT
Credentials "HOST-HOST8-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host9.domain.com:1830: HOST-HOST9-ORAAGENT
Credentials "HOST-HOST9-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host10.domain.com:3872: HOST-HOST10-ORAAGENT
Credentials "HOST-HOST10-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host11.domain.com:3872: HOST-HOST11-ORAAGENT
Credentials "HOST-HOST11-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host12.domain.com:3872: HOST-HOST12-ORAAGENT
Credentials "HOST-HOST12-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host13.domain.com:3872: HOST-HOST13-ORAAGENT
Credentials "HOST-HOST13-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host14.domain.com:3872: HOST-HOST14-ORAAGENT
Credentials "HOST-HOST14-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host15.domain.com:3872: HOST-HOST15-ORAAGENT
Credentials "HOST-HOST15-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host16.domain.com:3872: HOST-HOST16-ORAAGENT
Credentials "HOST-HOST16-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for omshost.domain.com:3872: HOST-OMSHOST-ORACLE
Credentials "HOST-OMSHOST-ORACLE:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host17.domain.com:3872: HOST-HOST17-ORAAGENT
Credentials "HOST-HOST17-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host18.domain.com:3872: HOST-HOST18-ORAAGENT
Credentials "HOST-HOST18-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Logging out of EMCLI
Logout successful

Logging in to EMCLI as CHECKSEC
Login successful

Setting preferred credentials DB-OEMDB-SYSTEM for CHECKSEC on oemdb.domain.com
Successfully set preferred credentials for target oemdb.domain.com:oracle_database.

Setting preferred credentials DB-OEMDB-SYS for CHECKSEC on oemdb.domain.com
Successfully set preferred credentials for target oemdb.domain.com:oracle_database.

Setting preferred credentials HOST-OMSHOST-ORACLE for CHECKSEC on oemdb.domain.com
Successfully set preferred credentials for target oemdb.domain.com:oracle_database.

Now assigning preferred credentials for agent targets.

Setting preferred credentials for CHECKSEC on host1.domain.com:3872
Successfully set preferred credentials for target host1.domain.com:host.

Setting preferred credentials for CHECKSEC on host2.domain.com:3872
Successfully set preferred credentials for target host2.domain.com:host.

Setting preferred credentials for CHECKSEC on host3.domain.com:3872
Successfully set preferred credentials for target host3.domain.com:host.

Setting preferred credentials for CHECKSEC on host4.domain.com:1830
Successfully set preferred credentials for target host4.domain.com:host.

Setting preferred credentials for CHECKSEC on host5.domain.com:3872
Successfully set preferred credentials for target host5.domain.com:host.

Setting preferred credentials for CHECKSEC on host6.domain.com:1830
Successfully set preferred credentials for target host6.domain.com:host.

Setting preferred credentials for CHECKSEC on host7.domain.com:3872
Successfully set preferred credentials for target host7.domain.com:host.

Setting preferred credentials for CHECKSEC on host8.domain.com:3872
Successfully set preferred credentials for target host8.domain.com:host.

Setting preferred credentials for CHECKSEC on host9.domain.com:1830
Successfully set preferred credentials for target host9.domain.com:host.

Setting preferred credentials for CHECKSEC on host10.domain.com:3872
Successfully set preferred credentials for target host10.domain.com:host.

Setting preferred credentials for CHECKSEC on host11.domain.com:3872
Successfully set preferred credentials for target host11.domain.com:host.

Setting preferred credentials for CHECKSEC on host12.domain.com:3872
Successfully set preferred credentials for target host12.domain.com:host.

Setting preferred credentials for CHECKSEC on host13.domain.com:3872
Successfully set preferred credentials for target host13.domain.com:host.

Setting preferred credentials for CHECKSEC on host14.domain.com:3872
Successfully set preferred credentials for target host14.domain.com:host.

Setting preferred credentials for CHECKSEC on host15.domain.com:3872
Successfully set preferred credentials for target host15.domain.com:host.

Setting preferred credentials for CHECKSEC on host16.domain.com:3872
Successfully set preferred credentials for target host16.domain.com:host.

Setting preferred credentials for CHECKSEC on omshost.domain.com:3872
Successfully set preferred credentials for target omshost.domain.com:host.

Setting preferred credentials for CHECKSEC on host17.domain.com:3872
Successfully set preferred credentials for target host17.domain.com:host.

Setting preferred credentials for CHECKSEC on host18.domain.com:3872
Successfully set preferred credentials for target host18.domain.com:host.

All finished. User CHECKSEC now logged in to EMCLI.

Now go run the checksec13R2.sh script.

As a reminder, user CHECKSEC has password [redacted].

Previous Versions

Finding the AWR Warehouse link

Configuring AWR Warehouse (AWRW) in EM12c

Oracle Enterprise Manager 12cR4 introduces the new “AWR Warehouse (AWRW)” feature, permitting administrators to consolidate AWR statistics from many individual databases managed by OEM into a single AWRW repository database.

As with all Oracle features, you must pay attention to licensing here.  I will not discuss licensing other than to point you to the relevant documents which you must read and understand yourself: Oracle Database Licensing Information 12c Release 1.

Documentation

At the moment official documentation appears limited to MOS note 1907335.1 and one section of the Oracle Database 2-Day Database + Performance Tuning Guide.  Get familiar with it.

Prerequisites

Repository Database

You must use Enterprise Edition for the AWRW repository database.  You must use version 12.1.0.2 or higher, or version 11.2.0.4 with patch 18547891 applied. Oracle recommends you use a database not used for any other purpose. I strongly agree with that recommendation.  Do not use your OEM repository. Note that I had to enable the diagnostic and tuning packs on the AWRW repository database by setting the control_management_pack_access initialization parameter to “DIAG+TUNING” before EM12c would allow me to select it for the repository.  I cannot reiterate enough how much I wish Oracle would explicitly state that users may enable management packs on their limited-use repository databases that support EM12c, RMAN catalogs and AWRW, but only a sucker expects license clarity out of Oracle.

I have selected 11.2.0.4 with patch 18547891 for my AWRW repository.

Oracle Enterprise Manager

You must use Oracle Enterprise Manager 12cR4 (12.1.0.4), and your OMS must have at least the August 31, 2014 bundle patch (19391521, or a later bundle patch) applied.  Your agents must run version 12.1.0.4.2 or later (requiring patch 19051570).

Licensing

Double check your licensing one more time.  Do not use features you have not licensed or you will pay a lot of money once you get audited, and you will get audited.

Configuration

For the purposes of this post I will skip the database installation and configuration steps.  If you have not yet gained proficiency with base installation and configuration tasks, you should probably gain some experience there before diving in to the AWR Warehouse.  Install a database of the appropriate version and register it with EM12c.

Planning

Think about your architecture.  With the recent release of AWRW functionality, some rough edges still exist.  These will probably get cleaned up over the next few releases but they took me by surprise and I have not seen them documented anywhere.

Oracle Enterprise Manager Agent Considerations

Do you use a separate dedicated user account on your servers to run the OEM Agent?  I do. Your AGENT_INSTANCE_DIR will get used by AWRW as a place to hold Data Pump output containing each source database’s AWR data.  I had to make this directory group writable by the dba group.  You also need to make sure the volume where this directory resides has enough free space to hold AWR extracts, which end up quite large on a busy system.  You may need to add more space if you keep your agent on a dedicated filesystem, as I do.

Do you run multiple instances under isolated accounts that don’t share a group membership?  You will probably need to create a group they all share that can write to the AGENT_INSTANCE_DIR.

Preferred Credential Considerations

AWRW strongly depends on the preferred credentials set for a database instance by the user that adds the database to AWRW.  If you already heavily use preferred credentials and want to use a different preferred database login for AWRW extraction compared to your usual DBA activities, you may elect to create a dedicated EM12c administrator to maintain AWRW to avoid conflicts.

The AWRW extraction user in the target database must have the DBA role and must also have an explicit execute grant on package SYS.DBMS_SWRF_INTERNAL.  I have chosen to use the SYSTEM account, to match my other preferred credential usage, but a more secure setup would use an account dedicated to this task.

Space Considerations

Take a look at how much space AWR consumes in your SYSAUX tablespaces already.  Your AWRW repository will need at least this much space, multiplied by however long you plan to keep these AWR snapshots around.  This will get very large, very quickly.

Added 20140912: I highly recommend that you disable data file autogrowth on your AWRW repository database.  I experienced repeated hangs until I determined that my jobs continually got stuck when SYSTEM or SYSAUX nearly filled and they sat there waiting on data file operations I/O as the system failed to resize the data files or identify a deadlock.  Do not rely on data file autogrowth, at least when using an 11.2.0.4 AWRW repository.

Initialize The AWR Warehouse

To begin configuring the AWR Warehouse, you must login using an EM12c super administrator account, like SYSMAN.  Once logged in, go to the Databases target list screen.  Unfortunately for this purpose you must use the “Database Load Map” form of the screen and not the infinitely more useful “Deprecated Search List (with Metrics)” that I have up on screen 99.9% of the time. Click the Targets menu, select Databases from the submenu that appears, and then if you do not see a “Performance” menu, enable the “Database Load Map” radio button.

Click the Performance menu and select the “AWR Warehouse” item.

Finding the AWR Warehouse link

This button makes things happen

At this point, if you used a super administrator account, you should see a summary screen that briefly describes the feature with a link to click to begin configuration.  If you don’t, log out and come back with the SYSMAN account.

Begin AWRW Configuration

Click Configure to continue

The next screen offers a search box to select the database to use as your AWRW repository and the usual credential selector.  Select the correct database, choose a database credential (I first selected SYSTEM, which failed, so use SYS as SYSDBA) and provide host credentials.

Database Selection

Rough edge: no warning that you must use SYSDBA

Once you click Next, the tool will pop up a dialog box warning you to make sure that your repository database has the necessary patch installed, and then asks you to select how long the system should keep AWRW data.  You can also select a staging location for AWR data extract storage prior to data loading.

Repository Configuration (Continued)

Diamonds and AWR Warehouses are forever

Click Submit on this screen and OEM will submit a job to initialize the AWRW repository.  To find this job later, if needed, go to the advanced job activity page and search for jobs of type “dbSetupCAW”.  The job should complete successfully if you have done everything correct so far.  On my system it only took six seconds, so just wait a moment and then reload the page, which should now look like this.

Repository Ready

That was easy

Click on the database icon at the upper left to switch away from the repository configuration tab to the database selection tab.

Database Selection

No data yet

As of this point you no longer need to use the SYSMAN account.  I switched back to my regular account, then returned to this screen.

Click the Add button to begin adding your first database(s). OEM will prompt you with the usual target selection screen.  Choose one or more databases and then click the Select button.  AWRW will NOT prompt you for credentials at this time.  Instead it will silently use the database host and normal database user preferred credentials you have established for the database target.  Another rough edge I expect to work better in future versions.  AWRW will perform some initial validations of those credentials to make sure that the database user has the DBA role and the previously mentioned execute grant on SYS.DBMS_SWRF_INTERNAL.  If you have missed any of these steps OEM will tell you about it and prevent you from adding the database.  Again, later I expect this to include an automated setup that will fix those issues.

First Target DB

I can’t show you the name

At this point you can just walk away and within about 24 hours you should have AWR data loaded into the warehouse.  If you feel impatient, click on one of the lines for a database to select it, then choose “Upload Snapshots Now” from the Actions menu.  This will submit a job to extract and load the AWR data, which you can find later under the job type “Run AWR Extract, Transfer and Load”.  In the background, this job extracts AWR data to the AGENT_INSTANCE_DIR on the target database’s server, compresses the data, transfers it to the staging area defined during AWRW repository setup, then loads the transferred data into the consolidated AWR Warehouse repository.

Loaded

One database in there. So many to go.

Summary

The size of and load on your selected database, along with the amount of AWR history you keep online, will influence how long each load takes.  My first load appeared to hang, with the AWRW repository database full of sessions waiting on enq: HW contention and buffer busy waits.  I ended up doing a shutdown abort and following the workaround instructions in MOS note 1912230.1.  I do not know if I truly needed to do this or not, but the symptoms sounded similar.  I’ve also noticed that some limits appear to exist.  I keep 42 days worth of hourly snapshots in each AWR, and my initial load only picked up 20 days / 500 snapshots.  This may represent rate-limiting as a way to trickle-load the AWRW, or it may mean AWRW does not yet play nicely with large AWR data.  Time will tell, and I fully expect future versions to shake out any bugs and to hold the DBA’s hand a bit more through the process.

I hope to cover using AWRW for performance tuning in a later post and I look forward to comments.

More Information

See these other fine posts for more information.

How to migrate EM12c R3 OMS and repository to a new host

(EDIT 20130917: If you simply need to change the IP address of your OEM server, please review MOS note 1562029.1.  The procedure in that note may allow you to change your OEM server’s IP address without following the lengthy process I describe below.)

In order to save power in our data center, I need to migrate my EM12c R3 environment from the host where it currently runs to a new host.  I have a simple configuration, with a single OMS, no load balancer, and the repository database runs on the same host as EM12c R3 itself.  I also have BI Publisher installed and integrated with EM12c, and a few third party plugins as I’ve detailed elsewhere on this blog.  If you use an OS other than Linux x86-64 I suggest you research thoroughly as this procedure may or may not apply to your environment.  Further, if you have a multi-OMS setup or use a load balancer, you must read the documentation and adapt the process accordingly to match your system’s needs.  Note that I wrote this as I did the migration, live, on my production system, so I have text in a few places showing where I would have done things differently if I knew what to expect in the first place.  It all ended up working, but it could have been simpler.

Oracle documents the procedure for this migration in the EM12c Administrator’s Guide, Part VII, section 29, “Backing Up and Recovering Enterprise Manager“.  As a first step, my system administrator installed SLES 11 SP3 on the new server and created an account for me along with the ‘oracle’ account for EM12c. I have a 70GB volume to use for the database and OEM binaries, a 1GB volume for the DB control files and a 2GB volume for redo logs supplemented with a 15GB FRA volume to support flashback.  Due to our tape backup strategy I use the FRA only for flashback, which we don’t wish to backup, and use a separate volume for RMAN backupsets.  To avoid a backup/restore cycle, the volumes holding the database datafiles will just be moved over to the new host on the storage side.

First I will relocate the management repository database to the new host, then complete the process by relocating the OMS.

Relocating the Management Repository Database

I run Oracle Database 11.2.0.3, Enterprise Edition, plus PSU Jul 2013.  Rather than installing the database software from scratch and patching it, I will clone the existing Oracle home to the new server.  Unfortunately I cannot use EM12c to do the cloning, as cloning via EM12c requires a management agent on the new host.  The software-only install of EM12c that I will run later installs a management agent as part of the process and I do not wish these two to conflict, so I do not want to install an agent on the new host at this time.

I will clone the database home according to the procedure in Appendix B of the 11gR2 database documentation.  You should review the documentation for full details.

Cloning the Database Home

Stop the OMS, database and management agent before cloning the existing Oracle home.

oracle$ $OMS_HOME/bin/emctl stop oms -all ; $AGENT_HOME/bin/emctl stop agent ; $ORACLE_HOME/bin/dbshut $ORACLE_HOME

Create a zip file of the existing database home.  Run this step as root (or using sudo) to make sure that you get all the files.

oracle$ sudo zip -r dbhome_1.zip /oracle/oem/product/11.2.0/dbhome_1

Now I will start the original database back up so that OEM continues running while I prepare the cloned Oracle home.  I will perform this migration over a few days, as I have time, so I need to keep OEM up and running as much as possible to support and manage my other databases.

oracle$ $ORACLE_HOME/bin/dbstart $ORACLE_HOME ; sleep 10 ; $OMS_HOME/bin/emctl start oms ; sleep 10 ; $AGENT_HOME/bin/emctl start agent

Copy this zip file to the new host.

oracle$  scp dbhome_1.zip oracle@newhost:/oracle/oem

On the new host, extract this zip file to the target directory.

oracle@newhost$ unzip -d / dbhome_1.zip

Remove all “*.ora” files from the extracted $ORACLE_HOME/network/admin directory.

oracle@newhost$  rm /oracle/oem/product/11.2.0/dbhome_1/network/admin/*.ora

Execute clone.pl from $ORACLE_HOME/clone/bin.

oracle@newhost$ export ORACLE_HOME=/oracle/oem/product/11.2.0/dbhome_1
oracle@newhost$ $ORACLE_HOME/perl/bin/perl clone.pl ORACLE_BASE="/oracle/oem" ORACLE_HOME="/oracle/oem/product/11.2.0/dbhome_1" OSDBA_GROUP=dba OSOPER_GROUP=oper -defaultHomeName

Unfortunately this creates an oraInventory directory in the oracle user’s home directory.  I prefer to keep oraInventory under ORACLE_BASE, so I moved it and edited the generated files to change the path from /home/oracle/oraInventory to /oracle/oem/oraInventory.  Most likely some environment variable, or a previously existing /etc/oraInst.loc would have prevented this optional step.

oracle@newhost$ cp -a ~/oraInventory /oracle/oem
oracle@newhost$ cd /oracle/oem/oraInventory
oracle@newhost$ perl -pi.bak -e 's#/home/oracle#/oracle/oem#' oraInst.loc orainstRoot.sh

Complete the cloning steps by running the orainstRoot.sh and root.sh scripts.

oracle@newhost$ sudo /oracle/oem/oraInventory/orainstRoot.sh
Changing permissions of /oracle/oem/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /oracle/oem/oraInventory to dba.
The execution of the script is complete.
oracle@newhost$ sudo /oracle/oem/product/11.2.0/dbhome_1/root.sh
Check /oracle/oem/product/11.2.0/dbhome_1/install/root_newhost_2013-08-27_13-04-51.log for the output of root script

I do not want to use netca to configure the listener, so I will just copy the $ORACLE_HOME/network/admin/*.ora files back over from the original server to the new server, and edit them accordingly.

oracle$ scp *.ora oracle@newhost:/oracle/oem/product/11.2.0/dbhome_1/network/admin/ 

oracle@newhost$ cd $ORACLE_HOME/network/admin
oracle@newhost$ perl -pi.bak -e 's#oldhost#newhost#' *.ora

This completes the database cloning.

Start Management Repository Database On New Host

At this point you will probably use RMAN to create a backup of your original repository database, then restore that backup onto the new host.  Instead, I will cheat a bit, shut down OEM and the database, and ask my sysadmin to move the repository database’s datafile LUN over to the new host and mount it at the same location.

Before moving the LUN, create directories that the database needs for a successful startup.  These include the admin/SID/adump directory, and in my case, the /oracle/mirror/SID/cntrl and /oracle/mirror/SID/log directories where I keep the multiplexed copies of my redo logs and controlfiles.

oracle@newhost$ mkdir -p /oracle/oem/admin/emrep/adump
oracle@newhost$ mkdir -p /oracle/mirror/emrep/cntrl ; mkdir -p /oracle/mirror/emrep/log

As a sanity check, you should try starting up the listener on the new server and starting the database in NOMOUNT mode before proceeding.  This will help catch any issues that may exist in your environment before you start the outage on your original server.  Investigate and resolve any issues found before proceeding.

Shutdown the OMS, agent and database on the original server.

oracle$ $OMS_HOME/bin/emctl stop oms -all ; $AGENT_HOME/bin/emctl stop agent ; $ORACLE_HOME/bin/dbshut $ORACLE_HOME

Copy the controlfiles and redo logs from the original server to the new server.

oracle$ scp /oracle/oem/cntrl/control01.ctl oracle@newhost:/oracle/oem/cntrl/control01.ctl
oracle$ scp /oracle/mirror/emrep/cntrl/control02.ctl oracle@newhost:/oracle/mirror/emrep/cntrl/control02.ctl
oracle$ scp /oracle/oem/log/redo* oracle@newhost:/oracle/oem/log
oracle$ scp /oracle/mirror/emrep/log/redo* oracle@newhost:/oracle/mirror/emrep/log

Back on the new server, start up the listener, then the database.  I probably should have disabled flashback first.

oracle@newhost$ lsnrctl start LISTENER
oracle@newhost$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Wed Aug 28 10:09:01 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup;
ORACLE instance started.

Total System Global Area 9620525056 bytes
Fixed Size                  2236488 bytes
Variable Size            6241128376 bytes
Database Buffers         3355443200 bytes
Redo Buffers               21716992 bytes

Database mounted.
ORA-38760: This database instance failed to turn on flashback database
SQL> select open_mode from v$database;

OPEN_MODE
--------------------
MOUNTED

SQL> alter database flashback off;

Database altered.

SQL> alter database open;

Database altered.

Reconfigure Existing OMS For New Repository Database

Start the OMS and agent on the original server.  OMS startup will fail, as you have not yet reconfigured the repository.

oracle$ $OMS_HOME/bin/emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server is not functioning because of the following reason:
Failed to connect to repository database. OMS will be automatically restarted once it identifies that database and listener are up.
Check EM Server log file for details: /oracle/oem/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
oracle$ $AGENT_HOME/bin/emctl start agent

Reconfigure the OMS repository database connection.  Provide SYSMAN’s password when prompted.

oracle$ $OMS_HOME/bin/emctl config oms -store_repos_details -repos_conndesc "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=newhost)(PORT=1521)))(CONNECT_DATA=(SID=emrep)))" -repos_user sysman
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Enter Repository User's Password : 
Successfully updated datasources and stored repository details in Credential Store.
If there are multiple OMSs in this environment, run this store_repos_details command on all of them.
And finally, restart all the OMSs using 'emctl stop oms -all' and 'emctl start oms'.
It is also necessary to restart the BI Publisher Managed Server.

Stop, then restart the OMS.

oracle$ $OMS_HOME/bin/emctl stop oms -all ; sleep 5 ; $OMS_HOME/bin/emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
AdminServer Successfully Stopped
Oracle Management Server is Down
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up

Login to OEM and confirm proper operation of the system.  I had a lot of alerts for failed backup jobs since my repository database hosts my RMAN catalog.  These can wait for now.  Also expect your repository target to show as down, since you have not yet updated the monitoring configuration.  Reconfigure it now, providing the SYSMAN password when prompted.

oracle$ $OMS_HOME/bin/emctl config emrep -conn_desc "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=newhost)(PORT=1521)))(CONNECT_DATA=(SID=emrep)))"
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Please enter repository password:                                    Enter password :                                               Login successful
Target "Management Services and Repository:oracle_emrep" modified successfully
Command completed successfully!

At this point you have successfully moved your repository database.  Don’t worry about any errors for now, though if you rely on an RMAN catalog and stored scripts for your backups, and these all live in your OEM repository database, you should go through now and update the monitoring configuration for the repository database and listener so that backups of your other databases do not fail.  I had to edit the recovery catalog and specify the host, port, and SID manually, since for some reason when I told it to use the repository database it kept trying to use the old hostname.  I will fix this after I complete the rest of the migration.

IMPORTANT NOTE: Since you have not yet migrated the repository database target to an agent local to that machine, backups of your repository database may not run.  Monitor your archived log directory on this system until you complete the rest of the migration, and manually run backups when necessary.

Installing OMS On A New Host

To install the OMS on a new host, perform a software-only installation from the same EM12c R3 installer that was used to install on the original host.  You will need to identify and retrieve all of the plugins that you have installed on the current OMS, as well as any patches that are currently installed on the OMS.  You must also make sure to use the same directory layout as on the original OMS.

Identifying Installed Patches

oracle$ $OMS_HOME/OPatch/opatch lsinv -oh $OMS_HOME
[...]
Interim patches (1) :

Patch  13983293     : applied on Thu Jul 11 09:56:16 EDT 2013
Unique Patch ID:  14779750
   Created on 25 Apr 2012, 02:18:06 hrs PST8PDT
   Bugs fixed:
     13587457, 13425845, 11822929

This patch gets installed by the EM12c R3 installer, so no need to bother with it any further.  If you have other patches installed, go fetch them, and install them after you have completed the plugin installation (see below).

Identifying Installed Plugins

Identify all plugins installed on your system using the query provided in the documentation, run as SYSMAN against your repository database.

SELECT epv.display_name, epv.plugin_id, epv.version, epv.rev_version,decode(su.aru_file, null, 'Media/External', 'https://updates.oracle.com/Orion/Services/download/'||aru_file||'?aru='||aru_id||chr(38)||'patch_file='||aru_file) URL
FROM em_plugin_version epv, em_current_deployed_plugin ecp, em_su_entities su
WHERE epv.plugin_type NOT IN ('BUILT_IN_TARGET_TYPE', 'INSTALL_HOME')
AND ecp.dest_type='2'
AND epv.plugin_version_id = ecp.plugin_version_id
AND su.entity_id = epv.su_entity_id;

Oracle-provided plugins will show a URL from which you must download the plugin.  Third-party plugins will not; you will need to make sure you have the appropriate downloaded plugin install .opar file from when you initially installed it.  Gather up all of these plugin files into a single directory on your NEW OMS host, changing the “.zip” filename extension to “.opar” for the Oracle-provided plugins.  You need EVERY plugin returned by this query or else your installation will NOT work.  I placed mine in /oracle/oem/migration/plugins.

You also need to copy over the three .zip files containing the OEM 12cR3 distribution: V38641-01.zip, V38642-01.zip and V38643-01.zip.  Save them into a convenient staging area on the new server (I use /oracle/oem/stage).

Perform Software-Only Installation Of EM12c R3

Go to the staging area on the new server and extract the three .zip files containing the EM12c R3 distribution, then start the installer.

oracle@newhost$ unzip V38641-01.zip ; unzip V38642-01.zip ; unzip V38643-01.zip 
[...]
oracle@newhost$ ./runInstaller

You can follow my previous post about upgrading EM12c R2 to R3 for more information about the installation process, just make sure you run it as a software only install and use the exact same path names as configured on the original OMS.  In my case this means a middleware home of /oracle/oem/Middleware12cR3 and an agent base directory of /oracle/oem/agent12c.

While the software installation proceeds, you should run an exportconfig on your current OMS to produce the configuration backup file you will need to use to reconfigure the new one.  Enter the SYSMAN password when prompted.

oracle$ $OMS_HOME/bin/emctl exportconfig oms
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password : 
ExportConfig started...
Machine is Admin Server host. Performing Admin Server backup...
Exporting emoms properties...
Exporting secure properties...

Export has determined that the OMS is not fronted 
by an SLB. The local hostname was NOT exported. 
The exported data can be imported on any host but 
resecure of all agents will be required. Please 
see the EM Advanced Configuration Guide for more 
details.

Exporting configuration for pluggable modules...
Preparing archive file...
Backup has been written to file: /oracle/oem/gc_inst/em/EMGC_OMS1/sysman/backup/opf_ADMIN_20130828_120424.bka

The export file contains sensitive data. 
 You must keep it secure.

ExportConfig completed successfully!

Copy that backup file to the new server.

oracle$  scp /oracle/oem/gc_inst/em/EMGC_OMS1/sysman/backup/opf_ADMIN_20130828_120424.bka oracle@newhost:/oracle/oem

Once the software-only install finishes, it will prompt you to run allroot.sh.  Do so.

oracle@newhost$ sudo /oracle/oem/Middleware12cR3/oms/allroot.sh 

Starting to execute allroot.sh ......... 

Starting to execute /oracle/oem/Middleware12cR3/oms/root.sh ......
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /oracle/oem/Middleware12cR3/oms

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: 
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: 
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: 

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
/etc exist

Creating /etc/oragchomelist file...
/oracle/oem/Middleware12cR3/oms
Finished execution of  /oracle/oem/Middleware12cR3/oms/root.sh ......

Starting to execute /oracle/oem/agent12c/core/12.1.0.3.0/root.sh ......
Finished product-specific root actions.
/etc exist
/oracle/oem/agent12c/core/12.1.0.3.0
Finished execution of  /oracle/oem/agent12c/core/12.1.0.3.0/root.sh ......

After running allroot.sh, you need to run the PluginInstall.sh script with the path where you saved the .opar files.  Make sure you select every plugin listed when you ran the query to retrieve the plugin list earlier, then hit install.

oracle@newhost$ /oracle/oem/Middleware12cR3/oms/sysman/install/PluginInstall.sh -pluginLocation /oracle/oem/migration/plugins
This must match the list you generated previously

This must match the list you generated previously

Prepare the Software Library

Go to the original server, and copy the contents of the software library to the new server.

oracle$ scp -r /oracle/oem/software_library/ oracle@newhost:/oracle/oem

Recreate the OMS with OMSCA

Shut everything down on your old server.

oracle$ $OMS_HOME/bin/emctl stop oms -all ; sleep 5 ; $AGENT_HOME/bin/emctl stop agent

Run OMSCA using the exportconfig backup file you generated earlier.  Enter the administration server, node manager, repository database user and agent registration passwords when prompted.

oracle@newhost$ $OMS_HOME/bin/omsca recover -as -ms -nostart -backup_file /oracle/oem/opf_ADMIN_20130828_120424.bka
Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.3.0
Copyright (c) 1996, 2013, Oracle. All rights reserved.

OS check passed.
OMS version check passed.
Performing Admin Server Recovery...
Retrieved Admin Server template.
Source Instance Host name where configuration is exported : [deleted]
Populated install params from backup...
Enter Administration Server user password:
Confirm Password:
Enter Node Manager Password:
Confirm Password:
Enter Repository database user password:
Enter Agent Registration password:
Confirm Password:
Doing pre requisite checks ......
Pre requisite checks completed successfully

Checking Plugin software bits
Proceed to recovery
Setting up domain from template...
Setup EM infrastructure succeeded!
Admin Server recovered from backup.
Now performing cleanup of OMS EMGC_OMS1...
Now launching DeleteOMS...
OMS Deleted successfully

Delete finished successfully
Now launching AddOMS...
Infrastructure setup of EM completed successfully.

Doing pre deployment operations ......
Pre deployment of EM completed successfully.

Deploying EM ......
Deployment of EM completed successfully.

Configuring webtier ......
Configuring webTier completed successfully.

Importing OMS configuration from recovery file...

If you have software library configured 
please make sure it is functional and accessible 
from this OMS by visiting:
 Setup->Provisioning and Patching->Software Library

Securing OMS ......
Adapter already exists: emgc_USER
Adapter already exists: emgc_GROUP
Post "Deploy and Repos Setup" operations completed successfully.

Performing Post deploy operations ....
Total 0 errors, 78 warnings. 0 entities imported.
pluginID:oracle.sysman.core
Done with csg import
pluginID:oracle.sysman.core
Done with csg import
No logging has been configured and default agent logging support is unavailable.
Post deploy operations completed successfully.

EM configuration completed successfully.
EM URL is:https://newhost:7803/em

Add OMS finished successfully
Recovery of server EMGC_OMS1 completed successfully
OMSCA Recover completed successfully

Start the OMS on the new server.

oracle@newhost$ $OMS_HOME/bin/emctl start oms

Configure the central agent on the new server, then run the root.sh script.

oracle@newhost$ /oracle/oem/agent12c/core/12.1.0.3.0/sysman/install/agentDeploy.sh AGENT_BASE_DIR=/oracle/oem/agent12c AGENT_INSTANCE_HOME=/oracle/oem/agent12c/agent_inst AGENT_PORT=3872 -configOnly OMS_HOST=newhost EM_UPLOAD_PORT=4902 AGENT_REGISTRATION_PASSWORD=password
[...]
oracle@newhost$ sudo /oracle/oem/agent12c/core/12.1.0.3.0/root.sh

Relocate the oracle_emrep target to the new OMS host.

oracle@newhost$ $OMS_HOME/bin/emcli login -username=sysman
Enter password : 

Login successful
oracle@newhost$ $OMS_HOME/bin/emcli sync
Synchronized successfully
oracle@newhost$ $OMS_HOME/bin/emctl config emrep -agent newhost:3872
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Please enter repository password: 
Enter password :                                                               
Login successful
Moved all targets from oldhost:3872 to newhost:3872
Command completed successfully!
Enter password :                                                               
Login successful
Moved all targets from oldhost:3872 to newhost:3872
Command completed successfully!

Step through each of your existing agents to re-secure them against the new OMS.  Provide the OMS HTTP port (not HTTPS) in this command, and enter the agent registration password when prompted.

$ $AGENT_INSTANCE_DIR/bin/emctl secure agent -emdWalletSrcUrl "http://newhost:4890/em"
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Agent successfully stopped...   Done.
Securing agent...   Started.
Enter Agent Registration Password : 
Agent successfully restarted...   Done.
EMD gensudoprops completed successfully
Securing agent...   Successful.

Start the agent on the old OMS server.  You should not need to do this, but I could not update the WebLogic Domain monitoring configuration without doing so first.  Also re-secure this agent to point to the new OMS.

oracle$ $AGENT_HOME/bin/emctl start agent
oracle$ $AGENT_INSTANCE_DIR/bin/emctl secure agent -emdWalletSrcUrl "http://newhost:4890/em"

Login to the OEM GUI running on the new server and navigate to the WebLogic Domain target for the Cloud Control domain.  In the Target Setup -> Monitoring Credentials section, update the Administration server host value to the new server name, then hit OK.  Then execute a Refresh WebLogic Domain, selecting Add/Update Targets, to move all WebLogic targets to the new central agent.

I use third-party plugins to monitor VMWare targets, NetApp storage and MySQL servers.  I had many of them set up to run from the OMS agent (except for the VMWare ones, since Blue Medora helpfully advised not to use the OMS agent for this — great advice).  I now need to relocate each of these targets to the new central agent using emcli.  You won’t need to do this step unless you also have things set up this way.  If I had to do this again, I would not use the OMS agent for these targets, since I would not need to change anything if I just had these on some other agent.

oracle@newhost$ ./emcli relocate_targets -src_agent=oldhost:3872 -dest_agent=newhost:3872 -copy_from_src -target_name=nameoftarget -target_type=typeoftarget

Final Cleanup Steps

By now you have completed the bulk of the work necessary to migrate your EM12c stack to a new server.  Only a few steps remain.  If you use any utility scripts on the old server, go ahead and copy those over now.  I have scripts to automate starting/stopping the OMS and agent, so I’ve copied those over.  Also make sure the oracle user on the new server has all the environment variables set up in their shell initialization files.

oracle$ scp ~/bin/CCstart ~/bin/CCstop oracle@newhost:bin/

The GCDomain Oracle WebLogic Domain target did not get moved to my new agent.  If this happened to you, go to the target home page and select the Modify Agents menu item.  Click Continue, then find GCDomain in the list, scroll to the right, and assign the new OMS server’s agent as the monitoring agent for this target, then click the Modify Agents button.

Reinstall BI Publisher

Since I had BI Publisher installed on the old server, I need to install it again on the new one.  Retrieve the 11.1.1.6.0 BI Publisher installation files used previously, and copy them to your staging area.  Run the “runInstaller” program from bishiphome/Disk1, and perform a software-only installation with the middleware home set to your EM12c installation middleware home, and leave the Oracle home as Oracle_BI1.

Instead of running the configureBIP script as you normally would to integrate BI Publisher with EM12c, just go to the WebLogic administration console after the software-only install completes, and navigate to the BIP server configuration page.  Lock the configuration for editing, and edit the configuration to change the listen address to reference the new server’s hostname and change the machine to the machine name where the admin server runs (in my case it showed up as EMGC_MACHINE2).  Save and activate the changes, then start the BIP server.

After the server has started, return to the WebLogic Domain page and re-run the Refresh WebLogic Domain step, again with Add/Update targets, to move BIP to your new OMS agent.

I actually had to do the Refresh WebLogic Domain step here twice.  I may have simply not waited long enough after starting BIP before I ran it, but I do not know for sure.

Update EM Console Service

I have only one target showing down at this point, the EM Console Service.  Go to the target, and click on the Monitoring Configuration tab.  Click on Service Tests and Beacons.  Select the EM Console Service Test, and click the Edit button.  Make sure you have the “Access Login page” step selected, and click Edit.  Change the URL to reflect your new OEM server, and save the changes.

Remove Previous OMS Server From OEM

Stop the agent on your original OMS server.

oracle$ $AGENT_HOME/bin/emctl stop agent

Remove the host target where your original OMS ran.  Then remove the agent target.

One Last Bounce

Finally, bounce the whole thing one last time, then start it back up.  All green.

Conclusion

I would prefer a simpler process to migrate the EM12c stack to a new server, but this works.  If you find yourself in a similar position to mine, I hope this helps you.  I’ve spent a lot of time working in EM12c so I feel capable to diagnose and resolve issues encountered during the process, but if you run into problems do not hesitate to contact Oracle Support and file a service requests.  If you want your system to stay supportable, stick with the experts and just use blogs as a guide to get started.  Good luck.

My Production EM12c Upgrade From R2 (12.1.0.2) to R3 (12.1.0.3)

This post covers my production upgrade from EM12c R2 to EM12c R3 on Linux x86-64 (SLES11 SP2).  To stress test the upgrade and keep it interesting, this system also has BI Publisher integrated into OEM, and also has the plugin from NetApp (version 12.1.0.1.0), the VMware plugin from BlueMedora (version 12.1.0.5.0), and the MySQL plugin from Pythian (version 12.1.0.1.1).  The repository database is running on version 11.2.0.3.

I’m feeling lucky today.  So this is just going straight into production.  Famous last words…

Preparation

  1. Go to edelivery and search for the Oracle Enterprise Manager product pack and platform Linux x86-64
  2. Follow the link titled “Oracle Enterprise Manager Cloud Control 12c Release 3 (12.1.0.3) Media Pack for Linux x86-64
  3. Download the three files needed for EM12c R3: V38641-01.zip, V38642-01.zip, and V38643-01.zip
  4. Download patch 13349651 for WebLogic, you will need it during the post-upgrade steps
  5. View the digest and run md5sum against each downloaded file to confirm that the files downloaded correctly
  6. Transfer the EM12c R3 files to a staging area on your Oracle Enterprise Manager server and unzip all three of them
  7. Delete the three downloaded .zip files if you are short on space (but don’t just “rm *.zip” or you’ll remove the necessary WT.zip file at the top level of your staging directory)
  8. Review the upgrade guide
  9. Create backups of your current OMS home, agent home, software library and Oracle inventory
  10. Create a backup of your current repository database that you can restore from if necessary
  11. If you use a dedicated filesystem for your Oracle Management Agents, make sure that dedicated filesystem has enough free space.  I used a small, 2GB filesystem, and this was barely large enough, except for the one sandbox server where it was too small and I was not able to complete the agent upgrade without adding space
  12. Stop the BI Publisher WebLogic Server (if you have it installed) via the WebLogic Admin Console
  13. Make sure your repository database does not have snapshots created on any tables by running “select master, log_table from all_mview_logs where log_owner = ‘SYSMAN'” — if any snapshots are found, follow the instructions in the upgrade guide to drop them
  14. If your repository database is version 11.1.0.7 or 11.2.0.1, follow the steps in the upgrade guide to apply the prerequisite patches needed to proceed
  15. Copy the emkey from the existing OMS to the existing management repository by running “$OMS_HOME/bin/emctl config emkey -copy_to_repos” and enter the SYSMAN password when prompted
  16. Confirm the emkey was copied by running “$OMS_HOME/bin/emctl status emkey” and enter the SYSMAN password when prompted
  17. Stop every OMS in your environment by running “$OMS_HOME/bin/emctl stop oms -all”
  18. Stop the management agent monitoring the management services and repository target by running “$AGENT_HOME/bin/emctl stop agent”

Running The Upgrade

Due to some issues I had with the upgrade from EM12c R1 to R2, I highly recommend that you do NOT use Cygwin to ssh to your OMS host and display the installer over ssh using Cygwin’s X server.  Use VNC instead.  I’m not going to try Cygwin this time through.

First, I start vncserver on the OMS host.  Then I connect to it to using TightVNC from my desktop machine.

  1. Navigate to the staging directory where you unzipped the EM12c R3 distribution files and run “./runInstaller” as your oracle software owner
  2. As an SAP customer, we are not allowed to use OCM, so I skipped the steps involving entering my My Oracle Support credentialsStep 1
  3. I also skipped the search for updates as this is a new enough release at the moment there should not be any necessaryStep 2
  4. The prerequisite checks run.  I received a warning about libstdc++43-4.3 not being found, but libstdc++43-devel-4.3.3-x86_64 fulfills this need, so I click ignore and then Next to continueStep 3
  5. Only the one system upgrade is supported when upgrading from EM12cR2, so I selected “Upgrade an existing system”, “One system upgrade”, and my existing middleware home.  Click NextStep 4
  6. This out of place upgrade will go into a new middleware home.  I am using /oracle/oem/Middleware12cR3.  Click Next and the installer will confirm that you have enough free space availableStep 5
  7. Here I had to pause and request more space from my storage admin, as the installer wants at least 14.0GB of free space.  Once that was done, I proceeded
  8. Enter the SYS and SYSMAN passwords and check the box to confirm that you have backed up the repository (you should have your OMS, agent, etc all backed up as well), then click NextStep 6
  9. The installer will check various parameters on your repository database and offer the chance to fix them if any need to be changed.  I accept the fixes and click YesStep 6(b)
  10. The installer checks some additional settings and notes that they should be reviewed after the installation or fixed now.  I explicitly granted execute on DBMS_RANDOM to DBSNMP and then clicked OKStep 6(c)
  11. The installer lists the plugin versions that will change and the plugins that will migrate. Confirm this all looks right and then click NextStep 7
  12. The installer lists additional plugins you can choose to deploy at install time.  I do not use any of these so I left them all unchecked and clicked NextStep 8
  13. The installer requests the password for your WebLogic adminserver and confirmation of the hostname, port and username.  Provide the password and click Next.  You may be able to change the OMS instance directory here but I do not suggest doing soStep 9
  14. You now have a chance to review your settings, then click Install to proceed with the upgradeStep 10
  15. Installation proceedsStep 11
  16. You are then prompted to run the allroot.sh file.  Login to the server and execute it as root or via sudoallroot.sh
  17. Once the install/upgrade is complete, the installer will display an installation summary.  Review it, save the URLs it gives you for the OMS and adminserver, then click CloseUpgrade Summary Report

Overall, the upgrade installation steps took 1 hour and 15 minutes in my environment.  This was on a physical server with 126GB RAM, 16 dual core processors and 200 managed targets.  This does not include the post-upgrade steps shown below.

Post Upgrade Steps

Now that the upgrade is complete, return to the upgrade guide to complete post upgrade steps.  Your environment may differ from mine, but these are the steps I had to follow.

  1. Start your central agent by running “$AGENT_HOME/bin/emctl start agent”.  At this point the load on my system went up very high and began responding very slowly.  I walked away for 10 minutes to let things settle down
  2. Open your web browser and go to the URL provided at the end of the installation and login as SYSMAN.  When I first tried to do so using Firefox I received an error indicating an invalid certificate.  I had to delete the old certificates and authorities from my previous installation and restart Firefox before it would allow me in.  MSIE worked fine though
  3. Update the central agent (the management agent installed on the OMS host) by clicking on the Setup menu, then Manage Cloud Control, then Upgrade Agents.  Click the Add button and select your central agent.  I choose “Override preferred credentials” since I have not configured sudo.  Click Submit to continue, then OK when warned that you may have to run root.sh manuallyUpgrade Central Agent
  4. My first upgrade attempt on the central agent failed due to a prerequisite check for package libstdc++-43.  The easy thing to do here is expand the Additional Inputs region and provide “-ignorePrereqs” as an additional parameter, but I chose to complete this agent upgrade using emcli and describe that process in the next two steps
  5. First run “$OMS_HOME/bin/emcli login -username=sysman” and enter the SYSMAN password when prompted.  Then run “$OMS_HOME/bin/emcli sync”
  6. Upgrade the agent by running “$OMS_HOME/bin/emcli upgrade_agents -additional_parameters=”-ignorePrereqs” -agents=”hostname.domain.com:3872″
  7. Wait while the upgrade proceeds.  You can view upgrade progress by running “$OMS_HOME/bin/emcli get_agent_upgrade_status”, or in the GUI by clicking on “Agent Upgrade Results” in the “Upgrade Agents” pageAgent Upgrade Progress
  8. Click Done once the central agent upgrade completes.  Go to the new agent home and run root.sh as root or via sudo
  9. Then repeat this process for the rest of your agents.  Try to install them first WITHOUT using the “-ignorePrereqs” flag, because if there are missing prerequisites you need to identify the issue and find out if it is something that it is appropriate to ignore, as the libstdc++ version was in my case.  Execute root.sh for each agent afterwards, unless you have sudo configured in which case it will happen automatically
  10. Two of my agents that run on different platforms could not be upgraded right away.  The new versions of the agent software needed to be downloaded from Self Update.  I am skipping them for now
  11. Next, apply patch 13349651 to WebLogic, following the instructions in the README file.  I attempted to do so, but the patch was already installed so I skipped this step
  12. There are a few other optional, post-upgrade steps like deleting obsolete targets.  These are documented in the upgrade guide and I will not note them here
  13. As a final step, make sure you update your Oracle user’s environment variables to reflect the new middleware home, OMS home, agent home, and so on

Conclusion

At this point my EM12cR3 production upgrade is complete!  Everything I have checked so far appears fully functional.  The only problem I had was a small filesystem for the management agent on my sandbox server causing the agent upgrade to run out of space, forcing manual intervention to resolve.  Don’t be stingy with space like I am and you should be fine.

I haven’t taken any time to investigate the new features yet, but I will be now.

(EDITED TO ADD: I forgot to mention the steps to get BI Publisher working again.  Please refer to the EM12cR3 Advanced Installation and Configuration Guide, chapter 15.  Essentially you will need to perform a software-only installation into the new middleware home, then execute the configureBIP script with an -upgrade flag to complete the BIP setup.)

Final (!) update/fix for EM12c increased redo from EM_JOB_METRICS after upgrade to R2 (bug/patch 14726136)

Update 13 Nov 2012: Patch 14726136 has been obsoleted. The note on MOS indicated that the patch caused some metrics to be calculated incorrectly. The patch has been replaced by patch 14833587. I have applied the new patch and all appears well — jobs are running and redo generation on the repository database remains where it was before the upgrade to EM12c R2.

Following up again on the EM12cR2 upgrade issue from BP1 that causes significantly increased redo logging on the repository database due to heavy insert activity in EM_JOB_METRICS. I first covered this here, with a followup here, a partial workaround here, and then a warning here.

Patch 14726136 has been re-released on MOS. The initial release caused problems for me as it prevented all OEM scheduled jobs from running, eventually causing me to rollback the patch. I am very pleased to report that the new update of the patch (from Oct 30th) applies cleanly and all of my jobs are now running on-schedule and completing successfully. EM_JOB_METRICS is showing no more than 11 inserts per second, and more than a minute is passing between the batches of inserts. My redo volume is already down significantly.

Big thanks to Oracle support and development and the EM12c team!

Patch 14726136

You may have seen the patch in the title drop regarding increased redo logging after upgrading to EM12cR2.

I tried the patch Friday morning, and while it did decrease my redo logging, it also prevented any OEM scheduled jobs from running. I eventually rolled back the patch and jobs ran again.

I would be very interested to know if the patch works for you AND your jobs still run.

Partial workaround for increased redo generation on EM12cR2 repository database after upgrade from R1

Another followup to my previous posts on increased redo logging on the repository database after upgrading to EM12cR2.

On the OTN Forum thread, slohit posted a suggestion with a way to reduce the frequency of job dispatcher runs:

To reduce the frequency of inserts, it would be better to slow down the frequency of the dispatcher. This is OK only if you are fine with a lower response time between steps for your jobs. If it is only a few jobs/job steps scheduled to run together, this is all right

emctl set property -name oracle.sysman.core.jobs.minDispatcherSleepInterval -value 0.5
emctl set property -name oracle.sysman.core.jobs.linearBackoff -value 0.5

By making this change, followed by a bounce of the OMS, the redo generation on my repository database decreased by about 50%. It’s still 200-300% more than before the upgrade, but it’s quite an improvement. The EM_JOB_METRICS table now only receives about 20 inserts per second compared to 60. My 13GB repository database should now only produce about 5GB of redo per day instead of 10GB.

Prior to the change, emctl get property reported these property values as null so presumably the OMS was working with whatever the built-in default values were.

In my case, a delay between job steps is a tradeoff that I can tolerate. This OMS runs simple database backups, SQL and RMAN scripts, so I do not have any time-critical multi-step jobs that could be negatively impacted by delayed response to completion of individual job steps. Your mileage may vary.

Oracle Support is continuing to investigate this bug. At least one other poster on the forum has reported that they too are experiencing this issue so I’m looking forward to finding out if this change reduces their redo generation to an acceptable rate. That same poster also reports that they do not see the issue on a new install of EM12cR2, only on a system upgraded from R1. It’s not clear to me why an upgraded system would experience this when processing the raw metric data while a fresh install would not, nor is it clear how many upgraded systems are experiencing the issue. But at least we now have a partial workaround for bug 14726136.