Category Archives: Cloud Control

EM13cR2 AWR Warehouse “Error communicating with agent” during transfer step with custom certificates

I have just noticed and resolved an issue in my EM13c R2 AWR Warehouse environment that I brought upon myself, hence a blog post for any others who might run into this, which also seems like a good time to release the scripts I use to generate and populate Oracle wallets for my EM13c agents.

After moving an AWRW source database from one EM13c managed server to a different EM13c managed server (same OS, same DB release), AWRW loads from that server began to fail. While debugging the issue, I first had to resolve an already-documented issue (see MOS note 2075341.1) where the source database had a NULL definition for the CAW_EXTR directory object, then fix up the data in the DBSNMP.CAW_EXTRACT_PROPERTIES table to reflect the CAW_EXTR directory. After resolving that, AWRW extracts ran successfully from the source database, but began to hang indefinitely during the CAW_RUN_ETL_NOW job in the transferAWR/transferFile job step, displaying only a cryptic error message:

An unhelpful error message

A helpful error message

I ran through many debugging steps: changing preferred credentials, bouncing the agents, checking for firewalls blocking connectivity, none seemed to help. Eventually I realized the step I had missed in setting up the new managed server where the source database now runs: I had not generated an Oracle wallet for the agent on the new server, while I did have an Oracle wallet for the agent on the previous, now-retired server. This created an issue because I have secured the agent on my OMS host (where my AWRW repository database runs) with a custom third party certificate, and the new agent, lacking a wallet containing a trusted root certificate to which it could trace the repository agent’s certificate, could not initiate a connection from the AWRW source DB host agent to the AWRW repository DB host agent.

I generated a wallet for the new agent, added the trusted root certificate and a certificate for the host to the wallet, stopped the agent, deployed the wallet, and started the agent. After those steps, running the AWRW load from this source database completed successfully. I believe that the missing trusted root certificate prevented the creation of a secure channel between the two agents. I probably did not need to add the host certificate to resolve this problem, but consider it a good practice anyway.

If you read this far, you may find my create_agent_wallets.sh script useful to generate wallets and certificate signing requests for every agent in your environment. If you find the wallet creation script useful, you may also find my import_agent_wallets.sh script useful to populate those wallets with signed certificates received from your CA.

Securing Oracle Enterprise Manager 13cR2

Oracle released Oracle Enterprise Manager 13cR2 at the beginning of October 2016. I have upgraded my production system to this new version, and here I provide a 13cR2-compatible version of my EM13c security checkup script. In addition to updating the script for EM13cR2, I have also updated it to take account of Oracle’s recommendation that single-instance non-RAC databases such as OEM repositories should now apply the DBBP Bundle Patch (previously known as the engineered systems bundle patch).

Latest Updates

Latest release: April 21, 2017, version 2.6. This release covers the 20170418 patch release and latest batch of recommended EM13.2 patches. See the changelog contained in the script for more details.

Download the latest release from https://raw.githubusercontent.com/brianpardy/em13c/master/checksec13R2.sh

EMCLI

If you have used this script for a while, you can download the latest release and just run it. It will continue to work the way it always has. If you would like to enable additional, optional functionality, enable the checksec13R2.sh EMCLI integration by logging in to EMCLI with an OEM administrator account before running checksec13R2.sh. The script will use EMCLI and attempt to check for plugin bundle patches on ALL of your OEM agents, not only the chained agent as it used to. It will also use EMCLI to attempt to validate the Java versions on all of your agents. This functionality requires that the EMCLI user account has access to run the execute_sql and execute_hostcmd, and also requires that the EMCLI user account has preferred credentials set for the repository database (normal and sysdba), repository database host, and for every host with a management agent.

To simplify the process, I have created a script to create a CHECKSEC user account in your OEM environment. The script will prompt you for the named credentials that the new account should use your repository database and each host. If you run this script after logging in to EMCLI as SYSMAN, it will create the new OEM user, grant acccess to all specified credentials, and grant EM_ALL_OPERATOR and VIEW_ANY_TARGET privileges so that the new account will have all the access needed to run all the optional checksec13R2.sh checks. I have included sample output from the user creation script at the end of this post. You can download the user creation script at create_user_for_checksec13R2.sh.

Download

You can access my EM13c script repository at https://github.com/brianpardy/em13c. To directly access the EM13cR2 security checkup script, use https://raw.githubusercontent.com/brianpardy/em13c/master/checksec13R2.sh.

Example Output – checksec13R2.sh


Performing EM13c R2 security checkup version 2.6 on omshost.domain.com at Fri Apr 21 09:32:30 EDT 2017.

Gathering info...
EM13c config... OK
Repos DB... 12.1.0.2.0 OK
OPatch-OMS... OK
OPatch-Agent... OK
OPatch-Repos DB... OK
OMSPatcher-OMS... OK
EMCLI login... OK
EMCLI-Agent list... OK
EMCLI-Agent patches... OK
EMCLI-Agent homes... OK

Using port definitions from configuration files
/etc/oragchomelist
/oracle/oem/gc_inst1/em/EMGC_OMS1/emgc.properties
/oracle/oem/gc_inst1/em/EMGC_OMS1/embip.properties
/oracle/oem/agent13cR1/agent_13.2.0.0.0/../agent_inst/sysman/emd/targets.xml

Agent port found at omshost.domain.com:3872
BIPublisher port found at omshost.domain.com:9803
BIPublisherOHS port found at omshost.domain.com:9852
NodeManager port found at omshost.domain.com:7403
OMSconsole port found at omshost.domain.com:7802
OMSproxy port found at omshost.domain.com:7301
OMSupload port found at omshost.domain.com:4903
WLSadmin found at omshost.domain.com:7102

Repository DB version=12.1.0.2.0 SID=oemdb host=omshost.domain.com
Repository DB target name=oemdb.domain.com

Using OPENSSL=/usr/bin/openssl1 (has TLS1_2=2)
Repository DB on OMS server, will check patches/parameters in /oracle/oem/product/12.1.0/db

(1) Checking SSL/TLS configuration (see notes 2138391.1, 2212006.1)

(1a) Forbid SSLv2 connections
Confirming ssl2 disabled for Agent at omshost.domain.com:3872... OK
Confirming ssl2 disabled for BIPublisher at omshost.domain.com:9803... OK
Confirming ssl2 disabled for NodeManager at omshost.domain.com:7403... OK
Confirming ssl2 disabled for BIPublisherOHS at omshost.domain.com:9852... OK
Confirming ssl2 disabled for OMSconsole at omshost.domain.com:7802... OK
Confirming ssl2 disabled for OMSproxy at omshost.domain.com:7301... OK
Confirming ssl2 disabled for OMSupload at omshost.domain.com:4903... OK
Confirming ssl2 disabled for WLSadmin at omshost.domain.com:7102... OK

Checking SSLv2 on all agents

Confirming ssl2 disabled for Agent at host01.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host02.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host04.usa.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host03.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host05.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host06.domain.com:1830... OK
Confirming ssl2 disabled for Agent at host07.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host08.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host09.domain.com:1830... OK
Confirming ssl2 disabled for Agent at host10.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host11.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host12.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host13.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host14.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host15.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host16.domain.com:3872... OK
Confirming ssl2 disabled for Agent at omshost.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host17.domain.com:3872... OK
Confirming ssl2 disabled for Agent at host18.domain.com:3872... OK

(1b) Forbid SSLv3 connections
Confirming ssl3 disabled for Agent at omshost.domain.com:3872... OK
Confirming ssl3 disabled for BIPublisher at omshost.domain.com:9803... OK
Confirming ssl3 disabled for NodeManager at omshost.domain.com:7403... OK
Confirming ssl3 disabled for BIPublisherOHS at omshost.domain.com:9852... OK
Confirming ssl3 disabled for OMSconsole at omshost.domain.com:7802... OK
Confirming ssl3 disabled for OMSproxy at omshost.domain.com:7301... OK
Confirming ssl3 disabled for OMSupload at omshost.domain.com:4903... OK
Confirming ssl3 disabled for WLSadmin at omshost.domain.com:7102... OK

Checking SSLv3 on all agents

Confirming ssl3 disabled for Agent at host01.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host02.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host04.usa.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host03.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host05.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host06.domain.com:1830... OK
Confirming ssl3 disabled for Agent at host07.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host08.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host09.domain.com:1830... OK
Confirming ssl3 disabled for Agent at host10.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host11.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host12.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host13.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host14.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host15.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host16.domain.com:3872... OK
Confirming ssl3 disabled for Agent at omshost.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host17.domain.com:3872... OK
Confirming ssl3 disabled for Agent at host18.domain.com:3872... OK

(1c) Forbid TLSv1 connections
Confirming tls1 disabled for Agent at omshost.domain.com:3872... OK
Confirming tls1 disabled for BIPublisher at omshost.domain.com:9803... OK
Confirming tls1 disabled for NodeManager at omshost.domain.com:7403... OK
Confirming tls1 disabled for BIPublisherOHS at omshost.domain.com:9852... OK
Confirming tls1 disabled for OMSconsole at omshost.domain.com:7802... OK
Confirming tls1 disabled for OMSproxy at omshost.domain.com:7301... OK
Confirming tls1 disabled for OMSupload at omshost.domain.com:4903... OK
Confirming tls1 disabled for WLSadmin at omshost.domain.com:7102... OK

Checking TLSv1 on all agents

Confirming tls1 disabled for Agent at host01.domain.com:3872... OK
Confirming tls1 disabled for Agent at host02.domain.com:3872... OK
Confirming tls1 disabled for Agent at host04.usa.domain.com:3872... OK
Confirming tls1 disabled for Agent at host03.domain.com:3872... OK
Confirming tls1 disabled for Agent at host05.domain.com:3872... OK
Confirming tls1 disabled for Agent at host06.domain.com:1830... OK
Confirming tls1 disabled for Agent at host07.domain.com:3872... OK
Confirming tls1 disabled for Agent at host08.domain.com:3872... OK
Confirming tls1 disabled for Agent at host09.domain.com:1830... OK
Confirming tls1 disabled for Agent at host10.domain.com:3872... OK
Confirming tls1 disabled for Agent at host11.domain.com:3872... OK
Confirming tls1 disabled for Agent at host12.domain.com:3872... OK
Confirming tls1 disabled for Agent at host13.domain.com:3872... OK
Confirming tls1 disabled for Agent at host14.domain.com:3872... OK
Confirming tls1 disabled for Agent at host15.domain.com:3872... OK
Confirming tls1 disabled for Agent at host16.domain.com:3872... OK
Confirming tls1 disabled for Agent at omshost.domain.com:3872... OK
Confirming tls1 disabled for Agent at host17.domain.com:3872... OK
Confirming tls1 disabled for Agent at host18.domain.com:3872... OK

(1d) Forbid TLSv1.1 connections
Confirming tls1_1 disabled for Agent at omshost.domain.com:3872... OK
Confirming tls1_1 disabled for BIPublisher at omshost.domain.com:9803... OK
Confirming tls1_1 disabled for NodeManager at omshost.domain.com:7403... OK
Confirming tls1_1 disabled for BIPublisherOHS at omshost.domain.com:9852... OK
Confirming tls1_1 disabled for OMSconsole at omshost.domain.com:7802... OK
Confirming tls1_1 disabled for OMSproxy at omshost.domain.com:7301... OK
Confirming tls1_1 disabled for OMSupload at omshost.domain.com:4903... OK
Confirming tls1_1 disabled for WLSadmin at omshost.domain.com:7102... OK

Checking TLSv1.1 on all agents

Confirming tls1_1 disabled for Agent at host01.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host02.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host04.usa.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host03.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host05.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host06.domain.com:1830... OK
Confirming tls1_1 disabled for Agent at host07.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host08.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host09.domain.com:1830... OK
Confirming tls1_1 disabled for Agent at host10.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host11.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host12.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host13.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host14.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host15.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host16.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at omshost.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host17.domain.com:3872... OK
Confirming tls1_1 disabled for Agent at host18.domain.com:3872... OK

(1e) Permit TLSv1.2 connections
Confirming tls1_2 available for Agent at omshost.domain.com:3872... OK
Confirming tls1_2 available for BIPublisher at omshost.domain.com:9803... OK
Confirming tls1_2 available for NodeManager at omshost.domain.com:7403... OK
Confirming tls1_2 available for BIPublisherOHS at omshost.domain.com:9852... OK
Confirming tls1_2 available for OMSconsole at omshost.domain.com:7802... OK
Confirming tls1_2 available for OMSproxy at omshost.domain.com:7301... OK
Confirming tls1_2 available for OMSupload at omshost.domain.com:4903... OK
Confirming tls1_2 available for WLSadmin at omshost.domain.com:7102... OK

Checking TLSv1.2 on all agents

Confirming tls1_2 available for Agent at host01.domain.com:3872... OK
Confirming tls1_2 available for Agent at host02.domain.com:3872... OK
Confirming tls1_2 available for Agent at host04.usa.domain.com:3872... OK
Confirming tls1_2 available for Agent at host03.domain.com:3872... OK
Confirming tls1_2 available for Agent at host05.domain.com:3872... OK
Confirming tls1_2 available for Agent at host06.domain.com:1830... OK
Confirming tls1_2 available for Agent at host07.domain.com:3872... OK
Confirming tls1_2 available for Agent at host08.domain.com:3872... OK
Confirming tls1_2 available for Agent at host09.domain.com:1830... OK
Confirming tls1_2 available for Agent at host10.domain.com:3872... OK
Confirming tls1_2 available for Agent at host11.domain.com:3872... OK
Confirming tls1_2 available for Agent at host12.domain.com:3872... OK
Confirming tls1_2 available for Agent at host13.domain.com:3872... OK
Confirming tls1_2 available for Agent at host14.domain.com:3872... OK
Confirming tls1_2 available for Agent at host15.domain.com:3872... OK
Confirming tls1_2 available for Agent at host16.domain.com:3872... OK
Confirming tls1_2 available for Agent at omshost.domain.com:3872... OK
Confirming tls1_2 available for Agent at host17.domain.com:3872... OK
Confirming tls1_2 available for Agent at host18.domain.com:3872... OK

(2) Checking supported ciphers at SSL/TLS endpoints (see notes 2138391.1, 1067411.1)
(2a) Checking LOW strength ciphers on Agent (omshost.domain.com:3872, protocol tls1_2)... OK
(2a) Checking MEDIUM strength ciphers on Agent (omshost.domain.com:3872)... OK
(2a) Checking HIGH strength ciphers on Agent (omshost.domain.com:3872)... OK

(2b) Checking LOW strength ciphers on BIPublisher (omshost.domain.com:9803, protocol tls1_2)... OK
(2b) Checking MEDIUM strength ciphers on BIPublisher (omshost.domain.com:9803)... OK
(2b) Checking HIGH strength ciphers on BIPublisher (omshost.domain.com:9803)... OK

(2c) Checking LOW strength ciphers on NodeManager (omshost.domain.com:7403, protocol tls1_2)... OK
(2c) Checking MEDIUM strength ciphers on NodeManager (omshost.domain.com:7403)... OK
(2c) Checking HIGH strength ciphers on NodeManager (omshost.domain.com:7403)... OK

(2d) Checking LOW strength ciphers on BIPublisherOHS (omshost.domain.com:9852, protocol tls1_2)... OK
(2d) Checking MEDIUM strength ciphers on BIPublisherOHS (omshost.domain.com:9852)... OK
(2d) Checking HIGH strength ciphers on BIPublisherOHS (omshost.domain.com:9852)... OK

(2e) Checking LOW strength ciphers on OMSconsole (omshost.domain.com:7802, protocol tls1_2)... OK
(2e) Checking MEDIUM strength ciphers on OMSconsole (omshost.domain.com:7802)... OK
(2e) Checking HIGH strength ciphers on OMSconsole (omshost.domain.com:7802)... OK

(2f) Checking LOW strength ciphers on OMSproxy (omshost.domain.com:7301, protocol tls1_2)... OK
(2f) Checking MEDIUM strength ciphers on OMSproxy (omshost.domain.com:7301)... OK
(2f) Checking HIGH strength ciphers on OMSproxy (omshost.domain.com:7301)... OK

(2g) Checking LOW strength ciphers on OMSupload (omshost.domain.com:4903, protocol tls1_2)... OK
(2g) Checking MEDIUM strength ciphers on OMSupload (omshost.domain.com:4903)... OK
(2g) Checking HIGH strength ciphers on OMSupload (omshost.domain.com:4903)... OK

(2h) Checking LOW strength ciphers on WLSadmin (omshost.domain.com:7102, protocol tls1_2)... OK
(2h) Checking MEDIUM strength ciphers on WLSadmin (omshost.domain.com:7102)... OK
(2h) Checking HIGH strength ciphers on WLSadmin (omshost.domain.com:7102)... OK

Checking supported ciphers on all agents

(2i) Checking LOW strength ciphers on Agent (host01.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host01.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host01.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host02.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host02.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host02.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host04.usa.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host04.usa.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host04.usa.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host03.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host03.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host03.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host05.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host05.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host05.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host06.domain.com:1830, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host06.domain.com:1830)... OK
(2i) Checking HIGH strength ciphers on Agent (host06.domain.com:1830)... OK

(2i) Checking LOW strength ciphers on Agent (host07.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host07.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host07.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host08.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host08.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host08.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host09.domain.com:1830, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host09.domain.com:1830)... OK
(2i) Checking HIGH strength ciphers on Agent (host09.domain.com:1830)... OK

(2i) Checking LOW strength ciphers on Agent (host10.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host10.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host10.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host11.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host11.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host11.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host12.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host12.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host12.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host13.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host13.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host13.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host14.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host14.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host14.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host15.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host15.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host15.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host16.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host16.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host16.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (omshost.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (omshost.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (omshost.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host17.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host17.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host17.domain.com:3872)... OK

(2i) Checking LOW strength ciphers on Agent (host18.domain.com:3872, protocol tls1_2)... OK
(2i) Checking MEDIUM strength ciphers on Agent (host18.domain.com:3872)... OK
(2i) Checking HIGH strength ciphers on Agent (host18.domain.com:3872)... OK

(3) Checking self-signed and demonstration certificates at SSL/TLS endpoints (see notes 2202569.1, 1367988.1, 1914184.1, 2213661.1, 2220788.1, 123033.1, 1937457.1)

(3a) Checking for self-signed certificates on OMS components
Checking certificate at Agent (omshost.domain.com:3872, protocol tls1_2)... OK
Checking certificate at BIPublisherOHS (omshost.domain.com:9852, protocol tls1_2)... OK
Checking certificate at BIPublisher (omshost.domain.com:9803, protocol tls1_2)... OK
Checking certificate at NodeManager (omshost.domain.com:7403, protocol tls1_2)... OK
Checking certificate at OMSconsole (omshost.domain.com:7802, protocol tls1_2)... OK
Checking certificate at OMSproxy (omshost.domain.com:7301, protocol tls1_2)... OK
Checking certificate at OMSupload (omshost.domain.com:4903, protocol tls1_2)... OK
Checking certificate at WLSadmin (omshost.domain.com:7102, protocol tls1_2)... OK

(3b) Checking for demonstration certificates on OMS components
Checking demo certificate at Agent (omshost.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at BIPublisherOHS (omshost.domain.com:9852, protocol tls1_2)... OK
Checking demo certificate at BIPublisher (omshost.domain.com:9803, protocol tls1_2)... OK
Checking demo certificate at NodeManager (omshost.domain.com:7403, protocol tls1_2)... OK
Checking demo certificate at OMSconsole (omshost.domain.com:7802, protocol tls1_2)... OK
Checking demo certificate at OMSproxy (omshost.domain.com:7301, protocol tls1_2)... OK
Checking demo certificate at OMSupload (omshost.domain.com:4903, protocol tls1_2)... OK
Checking demo certificate at WLSadmin (omshost.domain.com:7102, protocol tls1_2)... OK

(3c) Checking for self-signed certificates on all agents

Checking certificate at Agent (host01.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host02.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host04.usa.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host03.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host05.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host06.domain.com:1830, protocol tls1_2)... OK
Checking certificate at Agent (host07.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host08.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host09.domain.com:1830, protocol tls1_2)... OK
Checking certificate at Agent (host10.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host11.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host12.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host13.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host14.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host15.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host16.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (omshost.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host17.domain.com:3872, protocol tls1_2)... OK
Checking certificate at Agent (host18.domain.com:3872, protocol tls1_2)... OK

(3d) Checking for demonstration certificates on all agents

Checking demo certificate at Agent (host01.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host02.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host04.usa.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host03.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host05.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host06.domain.com:1830, protocol tls1_2)... OK
Checking demo certificate at Agent (host07.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host08.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host09.domain.com:1830, protocol tls1_2)... OK
Checking demo certificate at Agent (host10.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host11.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host12.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host13.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host14.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host15.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host16.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (omshost.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host17.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (host18.domain.com:3872, protocol tls1_2)... OK

(4) Checking EM13c Oracle home patch levels against 18 Apr 2017 baseline (see notes 1664074.1, 2219797.1, 822485.1, 1470197.1)

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) DATABASE BUNDLE PATCH: 12.1.0.2.170418 (APR2017) (25397136)... OK

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) Database PSU 12.1.0.2.170418, Oracle JavaVM Component (APR2017) (25437695)... OK

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) OCW Interim patch for 25481150 (25481150)... OK

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) EM QUERY WITH SQL_ID 4RQ83FNXTF39U PERFORMS POORLY ON ORACLE 12C RELATIVE TO 11G (20243268)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_TYPES_SERVER parameter (76629.1, 2167682.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_SERVER parameter (76629.1, 2167682.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_TYPES_CLIENT parameter (76629.1, 2167682.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_CLIENT parameter (76629.1, 2167682.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_TYPES_SERVER parameter (76629.1, 2167682.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_SERVER parameter (76629.1, 2167682.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_TYPES_CLIENT parameter (76629.1, 2167682.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_CLIENT parameter (76629.1, 2167682.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SSL_VERSION parameter (1545816.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SSL_CIPHER_SUITES parameter (1545816.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) listener.ora SSL_VERSION parameter (1545816.1)... OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) listener.ora SSL_CIPHER_SUITES parameter (1545816.1)... OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) ENTERPRISE MANAGER BASE PLATFORM - OMS 13.2.0.0.170418 PSU (25387277)... OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) TRACKING BUG TO REGISTER META VERSION FROM PS4 AND 13.1 BUNDLE PATCHES IN 13.2 (SYSTEM PATCH) (23603592)... OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) MERGE REQUEST ON TOP OF 12.1.3.0.0 FOR BUGS 24571979 24335626 (25322055)... OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) MERGE REQUEST ON TOP OF 12.1.3.0.0 FOR BUGS 22557350 19901079 20222451 (24329181)... OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) MERGE REQUEST ON TOP OF 12.1.3.0.0 FOR BUGS 19485414 20022048 (21849941)... OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) OPSS BUNDLE PATCH 12.1.3.0.170418 (22748215)... OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) ENTERPRISE MANAGER FOR OMS PLUGINS 13.2.0.0.170331 (Not used for 13.2.2 plugins) (25672422)... OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) WLS PATCH SET UPDATE 12.1.3.0.170418 (25388793)... OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) TOPLINK SECURITY PATCH UPDATE CPUJUL2016 (24327938)... OK

Using EMCLI to check for agent bundle patch on all agents

(4d) Agent host01.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host02.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host04.usa.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host03.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host05.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host06.domain.com:1830 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host07.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host08.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host09.domain.com:1830 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host10.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host11.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host12.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host13.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host14.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host15.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host16.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent omshost.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host17.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(4d) Agent host18.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170331 (25580746)... OK

(5) Checking EM13cR2 Java patch levels against 18 Apr 2017 baseline (see notes 1506916.1, 2241373.1, 2241358.1)

(5a) Common Java (/oracle/oem/Middleware13cR2/oracle_common/jdk) JAVA SE JDK VERSION 1.7.0_141 (13079846)... OK

Using EMCLI to check Java patch levels on all agents

(5b) Agent host01.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host02.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host04.usa.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host03.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host05.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host06.domain.com:1830 Java VERSION 1.7.0_141... OK

(5b) Agent host07.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host08.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host09.domain.com:1830 Java VERSION 1.7.0_141... OK

(5b) Agent host10.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host11.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host12.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host13.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host14.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host15.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host16.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent omshost.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host17.domain.com:3872 Java VERSION 1.7.0_141... OK

(5b) Agent host18.domain.com:3872 Java VERSION 1.7.0_141... OK

(6) Checking EM13cR2 OPatch/OMSPatcher patch levels against 18 Apr 2017 requirements (see patch 25197714 README, patches 6880880 and 19999993)

(6a) OMS OPatch (/oracle/oem/Middleware13cR2/OPatch) VERSION 13.9.1.3.0 or newer... OK

(6b) OMSPatcher (/oracle/oem/Middleware13cR2/OPatch) VERSION 13.8.0.0.2 or newer... OK

Checking OPatch patch levels on all agents

(6c) Agent host01.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host02.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host04.usa.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host03.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host05.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host06.domain.com:1830 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host07.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host08.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host09.domain.com:1830 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host10.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host11.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host12.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host13.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host14.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host15.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host16.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent omshost.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host17.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(6c) Agent host18.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0... OK

(7) Agent plugin bundle patch checks on all agents...
(7a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host01.domain.com:3872 (25501452)... OK - plugin not installed

(7b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host01.domain.com:3872 (25197692)... OK - plugin not installed

(7c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host01.domain.com:3872 (25501427)... OK - plugin not installed

(7d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host01.domain.com:3872 (25501430)... OK

(7e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host01.domain.com:3872 (25501408)... OK - plugin not installed

(7f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host01.domain.com:3872 (25162444)... OK - plugin not installed

(7g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host01.domain.com:3872 (25501436)... OK

(7h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host01.domain.com:3872 (25362875)... OK - plugin not installed

(7i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host01.domain.com:3872 (25522944)... OK - plugin not installed

(7j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host01.domain.com:3872 (25501416)... OK - plugin not installed

(7k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host01.domain.com:3872 (25362898)... OK - plugin not installed

(7l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host01.domain.com:3872 (25362890)... OK - plugin not installed

(7m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host01.domain.com:3872 (25197712)... OK - plugin not installed

(8a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host02.domain.com:3872 (25501452)... OK - plugin not installed

(8b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host02.domain.com:3872 (25197692)... OK - plugin not installed

(8c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host02.domain.com:3872 (25501427)... OK - plugin not installed

(8d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host02.domain.com:3872 (25501430)... OK

(8e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host02.domain.com:3872 (25501408)... OK - plugin not installed

(8f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host02.domain.com:3872 (25162444)... OK - plugin not installed

(8g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host02.domain.com:3872 (25501436)... OK

(8h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host02.domain.com:3872 (25362875)... OK - plugin not installed

(8i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host02.domain.com:3872 (25522944)... OK - plugin not installed

(8j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host02.domain.com:3872 (25501416)... OK - plugin not installed

(8k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host02.domain.com:3872 (25362898)... OK - plugin not installed

(8l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host02.domain.com:3872 (25362890)... OK - plugin not installed

(8m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host02.domain.com:3872 (25197712)... OK - plugin not installed

(9a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host04.usa.domain.com:3872 (25501452)... OK - plugin not installed

(9b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host04.usa.domain.com:3872 (25197692)... OK - plugin not installed

(9c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host04.usa.domain.com:3872 (25501427)... OK - plugin not installed

(9d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host04.usa.domain.com:3872 (25501430)... OK - plugin not installed

(9e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host04.usa.domain.com:3872 (25501408)... OK - plugin not installed

(9f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host04.usa.domain.com:3872 (25162444)... OK - plugin not installed

(9g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host04.usa.domain.com:3872 (25501436)... OK - plugin not installed

(9h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host04.usa.domain.com:3872 (25362875)... OK - plugin not installed

(9i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host04.usa.domain.com:3872 (25522944)... OK - plugin not installed

(9j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host04.usa.domain.com:3872 (25501416)... OK - plugin not installed

(9k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host04.usa.domain.com:3872 (25362898)... OK - plugin not installed

(9l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host04.usa.domain.com:3872 (25362890)... OK - plugin not installed

(9m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host04.usa.domain.com:3872 (25197712)... OK - plugin not installed

(10a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host03.domain.com:3872 (25501452)... OK - plugin not installed

(10b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host03.domain.com:3872 (25197692)... OK - plugin not installed

(10c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host03.domain.com:3872 (25501427)... OK - plugin not installed

(10d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host03.domain.com:3872 (25501430)... OK

(10e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host03.domain.com:3872 (25501408)... OK - plugin not installed

(10f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host03.domain.com:3872 (25162444)... OK - plugin not installed

(10g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host03.domain.com:3872 (25501436)... OK

(10h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host03.domain.com:3872 (25362875)... OK - plugin not installed

(10i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host03.domain.com:3872 (25522944)... OK - plugin not installed

(10j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host03.domain.com:3872 (25501416)... OK - plugin not installed

(10k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host03.domain.com:3872 (25362898)... OK - plugin not installed

(10l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host03.domain.com:3872 (25362890)... OK - plugin not installed

(10m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host03.domain.com:3872 (25197712)... OK - plugin not installed

(11a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host05.domain.com:3872 (25501452)... OK - plugin not installed

(11b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host05.domain.com:3872 (25197692)... OK - plugin not installed

(11c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host05.domain.com:3872 (25501427)... OK - plugin not installed

(11d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host05.domain.com:3872 (25501430)... OK

(11e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host05.domain.com:3872 (25501408)... OK - plugin not installed

(11f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host05.domain.com:3872 (25162444)... OK - plugin not installed

(11g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host05.domain.com:3872 (25501436)... OK

(11h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host05.domain.com:3872 (25362875)... OK - plugin not installed

(11i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host05.domain.com:3872 (25522944)... OK - plugin not installed

(11j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host05.domain.com:3872 (25501416)... OK - plugin not installed

(11k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host05.domain.com:3872 (25362898)... OK - plugin not installed

(11l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host05.domain.com:3872 (25362890)... OK - plugin not installed

(11m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host05.domain.com:3872 (25197712)... OK - plugin not installed

(12a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host06.domain.com:1830 (25501452)... OK - plugin not installed

(12b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host06.domain.com:1830 (25197692)... OK - plugin not installed

(12c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host06.domain.com:1830 (25501427)... OK - plugin not installed

(12d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host06.domain.com:1830 (25501430)... OK

(12e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host06.domain.com:1830 (25501408)... OK - plugin not installed

(12f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host06.domain.com:1830 (25162444)... OK - plugin not installed

(12g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host06.domain.com:1830 (25501436)... OK - plugin not installed

(12h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host06.domain.com:1830 (25362875)... OK - plugin not installed

(12i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host06.domain.com:1830 (25522944)... OK - plugin not installed

(12j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host06.domain.com:1830 (25501416)... OK - plugin not installed

(12k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host06.domain.com:1830 (25362898)... OK - plugin not installed

(12l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host06.domain.com:1830 (25362890)... OK - plugin not installed

(12m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host06.domain.com:1830 (25197712)... OK - plugin not installed

(13a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host07.domain.com:3872 (25501452)... OK - plugin not installed

(13b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host07.domain.com:3872 (25197692)... OK - plugin not installed

(13c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host07.domain.com:3872 (25501427)... OK - plugin not installed

(13d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host07.domain.com:3872 (25501430)... OK

(13e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host07.domain.com:3872 (25501408)... OK - plugin not installed

(13f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host07.domain.com:3872 (25162444)... OK - plugin not installed

(13g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host07.domain.com:3872 (25501436)... OK

(13h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host07.domain.com:3872 (25362875)... OK - plugin not installed

(13i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host07.domain.com:3872 (25522944)... OK - plugin not installed

(13j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host07.domain.com:3872 (25501416)... OK - plugin not installed

(13k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host07.domain.com:3872 (25362898)... OK - plugin not installed

(13l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host07.domain.com:3872 (25362890)... OK - plugin not installed

(13m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host07.domain.com:3872 (25197712)... OK - plugin not installed

(14a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host08.domain.com:3872 (25501452)... OK - plugin not installed

(14b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host08.domain.com:3872 (25197692)... OK - plugin not installed

(14c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host08.domain.com:3872 (25501427)... OK - plugin not installed

(14d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host08.domain.com:3872 (25501430)... OK

(14e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host08.domain.com:3872 (25501408)... OK - plugin not installed

(14f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host08.domain.com:3872 (25162444)... OK - plugin not installed

(14g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host08.domain.com:3872 (25501436)... OK

(14h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host08.domain.com:3872 (25362875)... OK - plugin not installed

(14i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host08.domain.com:3872 (25522944)... OK - plugin not installed

(14j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host08.domain.com:3872 (25501416)... OK - plugin not installed

(14k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host08.domain.com:3872 (25362898)... OK - plugin not installed

(14l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host08.domain.com:3872 (25362890)... OK - plugin not installed

(14m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host08.domain.com:3872 (25197712)... OK - plugin not installed

(15a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host09.domain.com:1830 (25501452)... OK - plugin not installed

(15b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host09.domain.com:1830 (25197692)... OK - plugin not installed

(15c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host09.domain.com:1830 (25501427)... OK - plugin not installed

(15d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host09.domain.com:1830 (25501430)... OK

(15e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host09.domain.com:1830 (25501408)... OK - plugin not installed

(15f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host09.domain.com:1830 (25162444)... OK - plugin not installed

(15g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host09.domain.com:1830 (25501436)... OK

(15h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host09.domain.com:1830 (25362875)... OK - plugin not installed

(15i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host09.domain.com:1830 (25522944)... OK - plugin not installed

(15j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host09.domain.com:1830 (25501416)... OK - plugin not installed

(15k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host09.domain.com:1830 (25362898)... OK - plugin not installed

(15l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host09.domain.com:1830 (25362890)... OK - plugin not installed

(15m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host09.domain.com:1830 (25197712)... OK - plugin not installed

(16a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host10.domain.com:3872 (25501452)... OK - plugin not installed

(16b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host10.domain.com:3872 (25197692)... OK - plugin not installed

(16c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host10.domain.com:3872 (25501427)... OK - plugin not installed

(16d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host10.domain.com:3872 (25501430)... OK

(16e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host10.domain.com:3872 (25501408)... OK - plugin not installed

(16f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host10.domain.com:3872 (25162444)... OK - plugin not installed

(16g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host10.domain.com:3872 (25501436)... OK

(16h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host10.domain.com:3872 (25362875)... OK - plugin not installed

(16i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host10.domain.com:3872 (25522944)... OK - plugin not installed

(16j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host10.domain.com:3872 (25501416)... OK - plugin not installed

(16k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host10.domain.com:3872 (25362898)... OK - plugin not installed

(16l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host10.domain.com:3872 (25362890)... OK - plugin not installed

(16m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host10.domain.com:3872 (25197712)... OK - plugin not installed

(17a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host11.domain.com:3872 (25501452)... OK - plugin not installed

(17b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host11.domain.com:3872 (25197692)... OK - plugin not installed

(17c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host11.domain.com:3872 (25501427)... OK - plugin not installed

(17d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host11.domain.com:3872 (25501430)... OK

(17e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host11.domain.com:3872 (25501408)... OK - plugin not installed

(17f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host11.domain.com:3872 (25162444)... OK - plugin not installed

(17g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host11.domain.com:3872 (25501436)... OK

(17h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host11.domain.com:3872 (25362875)... OK - plugin not installed

(17i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host11.domain.com:3872 (25522944)... OK - plugin not installed

(17j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host11.domain.com:3872 (25501416)... OK - plugin not installed

(17k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host11.domain.com:3872 (25362898)... OK - plugin not installed

(17l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host11.domain.com:3872 (25362890)... OK - plugin not installed

(17m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host11.domain.com:3872 (25197712)... OK - plugin not installed

(18a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host12.domain.com:3872 (25501452)... OK - plugin not installed

(18b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host12.domain.com:3872 (25197692)... OK - plugin not installed

(18c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host12.domain.com:3872 (25501427)... OK - plugin not installed

(18d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host12.domain.com:3872 (25501430)... OK

(18e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host12.domain.com:3872 (25501408)... OK - plugin not installed

(18f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host12.domain.com:3872 (25162444)... OK - plugin not installed

(18g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host12.domain.com:3872 (25501436)... OK

(18h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host12.domain.com:3872 (25362875)... OK - plugin not installed

(18i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host12.domain.com:3872 (25522944)... OK - plugin not installed

(18j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host12.domain.com:3872 (25501416)... OK - plugin not installed

(18k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host12.domain.com:3872 (25362898)... OK - plugin not installed

(18l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host12.domain.com:3872 (25362890)... OK - plugin not installed

(18m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host12.domain.com:3872 (25197712)... OK - plugin not installed

(19a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host13.domain.com:3872 (25501452)... OK - plugin not installed

(19b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host13.domain.com:3872 (25197692)... OK - plugin not installed

(19c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host13.domain.com:3872 (25501427)... OK - plugin not installed

(19d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host13.domain.com:3872 (25501430)... OK

(19e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host13.domain.com:3872 (25501408)... OK - plugin not installed

(19f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host13.domain.com:3872 (25162444)... OK - plugin not installed

(19g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host13.domain.com:3872 (25501436)... OK

(19h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host13.domain.com:3872 (25362875)... OK - plugin not installed

(19i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host13.domain.com:3872 (25522944)... OK - plugin not installed

(19j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host13.domain.com:3872 (25501416)... OK - plugin not installed

(19k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host13.domain.com:3872 (25362898)... OK - plugin not installed

(19l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host13.domain.com:3872 (25362890)... OK - plugin not installed

(19m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host13.domain.com:3872 (25197712)... OK - plugin not installed

(20a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host14.domain.com:3872 (25501452)... OK - plugin not installed

(20b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host14.domain.com:3872 (25197692)... OK - plugin not installed

(20c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host14.domain.com:3872 (25501427)... OK - plugin not installed

(20d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host14.domain.com:3872 (25501430)... OK

(20e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host14.domain.com:3872 (25501408)... OK - plugin not installed

(20f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host14.domain.com:3872 (25162444)... OK - plugin not installed

(20g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host14.domain.com:3872 (25501436)... OK

(20h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host14.domain.com:3872 (25362875)... OK - plugin not installed

(20i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host14.domain.com:3872 (25522944)... OK - plugin not installed

(20j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host14.domain.com:3872 (25501416)... OK - plugin not installed

(20k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host14.domain.com:3872 (25362898)... OK - plugin not installed

(20l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host14.domain.com:3872 (25362890)... OK - plugin not installed

(20m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host14.domain.com:3872 (25197712)... OK - plugin not installed

(21a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host15.domain.com:3872 (25501452)... OK - plugin not installed

(21b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host15.domain.com:3872 (25197692)... OK - plugin not installed

(21c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host15.domain.com:3872 (25501427)... OK - plugin not installed

(21d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host15.domain.com:3872 (25501430)... OK

(21e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host15.domain.com:3872 (25501408)... OK - plugin not installed

(21f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host15.domain.com:3872 (25162444)... OK - plugin not installed

(21g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host15.domain.com:3872 (25501436)... OK

(21h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host15.domain.com:3872 (25362875)... OK - plugin not installed

(21i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host15.domain.com:3872 (25522944)... OK - plugin not installed

(21j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host15.domain.com:3872 (25501416)... OK - plugin not installed

(21k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host15.domain.com:3872 (25362898)... OK - plugin not installed

(21l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host15.domain.com:3872 (25362890)... OK - plugin not installed

(21m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host15.domain.com:3872 (25197712)... OK - plugin not installed

(22a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host16.domain.com:3872 (25501452)... OK - plugin not installed

(22b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host16.domain.com:3872 (25197692)... OK - plugin not installed

(22c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host16.domain.com:3872 (25501427)... OK - plugin not installed

(22d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host16.domain.com:3872 (25501430)... OK

(22e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host16.domain.com:3872 (25501408)... OK - plugin not installed

(22f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host16.domain.com:3872 (25162444)... OK - plugin not installed

(22g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host16.domain.com:3872 (25501436)... OK

(22h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host16.domain.com:3872 (25362875)... OK - plugin not installed

(22i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host16.domain.com:3872 (25522944)... OK - plugin not installed

(22j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host16.domain.com:3872 (25501416)... OK - plugin not installed

(22k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host16.domain.com:3872 (25362898)... OK - plugin not installed

(22l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host16.domain.com:3872 (25362890)... OK - plugin not installed

(22m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host16.domain.com:3872 (25197712)... OK - plugin not installed

(23a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ omshost.domain.com:3872 (25501452)... OK - plugin not installed

(23b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ omshost.domain.com:3872 (25197692)... OK - plugin not installed

(23c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ omshost.domain.com:3872 (25501427)... OK - plugin not installed

(23d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ omshost.domain.com:3872 (25501430)... OK - plugin not installed

(23e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ omshost.domain.com:3872 (25501408)... OK - plugin not installed

(23f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ omshost.domain.com:3872 (25162444)... OK

(23g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ omshost.domain.com:3872 (25501436)... OK

(23h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ omshost.domain.com:3872 (25362875)... OK - plugin not installed

(23i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ omshost.domain.com:3872 (25522944)... OK - plugin not installed

(23j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ omshost.domain.com:3872 (25501416)... OK - plugin not installed

(23k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ omshost.domain.com:3872 (25362898)... OK - plugin not installed

(23l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ omshost.domain.com:3872 (25362890)... OK - plugin not installed

(23m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ omshost.domain.com:3872 (25197712)... OK - plugin not installed

(24a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host17.domain.com:3872 (25501452)... OK - plugin not installed

(24b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host17.domain.com:3872 (25197692)... OK - plugin not installed

(24c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host17.domain.com:3872 (25501427)... OK - plugin not installed

(24d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host17.domain.com:3872 (25501430)... OK

(24e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host17.domain.com:3872 (25501408)... OK - plugin not installed

(24f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host17.domain.com:3872 (25162444)... OK - plugin not installed

(24g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host17.domain.com:3872 (25501436)... OK

(24h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host17.domain.com:3872 (25362875)... OK - plugin not installed

(24i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host17.domain.com:3872 (25522944)... OK - plugin not installed

(24j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host17.domain.com:3872 (25501416)... OK - plugin not installed

(24k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host17.domain.com:3872 (25362898)... OK - plugin not installed

(24l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host17.domain.com:3872 (25362890)... OK - plugin not installed

(24m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host17.domain.com:3872 (25197712)... OK - plugin not installed

(25a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host18.domain.com:3872 (25501452)... OK - plugin not installed

(25b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host18.domain.com:3872 (25197692)... OK - plugin not installed

(25c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host18.domain.com:3872 (25501427)... OK - plugin not installed

(25d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host18.domain.com:3872 (25501430)... OK

(25e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host18.domain.com:3872 (25501408)... OK - plugin not installed

(25f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host18.domain.com:3872 (25162444)... OK - plugin not installed

(25g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host18.domain.com:3872 (25501436)... OK

(25h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host18.domain.com:3872 (25362875)... OK - plugin not installed

(25i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host18.domain.com:3872 (25522944)... OK - plugin not installed

(25j) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host18.domain.com:3872 (25501416)... OK - plugin not installed

(25k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host18.domain.com:3872 (25362898)... OK - plugin not installed

(25l) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host18.domain.com:3872 (25362890)... OK - plugin not installed

(25m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host18.domain.com:3872 (25197712)... OK - plugin not installed

Cleaning up temporary files... done
All tests succeeded.

Visit https://pardydba.wordpress.com/2016/10/28/securing-oracle-enterprise-manager-13cr2/ for more information.
Download the latest release from https://raw.githubusercontent.com/brianpardy/em13c/master/checksec13R2.sh
Download the latest beta release from https://raw.githubusercontent.com/brianpardy/em13c/beta/checksec13R2.sh

Example Output – create_user_for_checksec13R2.sh


Welcome to ./create_user_for_checksec13R2.sh, version 1.0, released 20170314.

Download the latest release of this script at any time from:
https://raw.githubusercontent.com/brianpardy/em13c/master/create_user_for_checksec13R2.sh

This script exists to supplement checksec13R2.sh and enable additional checks. When run, this
script will create a user named CHECKSEC in your EM13cR2 environment and give that user a
random password, which gets printed to the screen at the end of the script. The script then
grants CHECKSEC VIEW_ANY_TARGET and EM_ALL_OPERATOR privilege, and then prompts you to supply
the names of credentials existing in your EM13cR2 environment. The script validates the names of
credentials supplied, grants VIEW access to CHECKSEC for each credential, and assigns those
credentials as preferred credentials for CHECKSEC for each relevant target.

Providing credentials for the repository database enables the following additional checks in
checksec13R2.sh:
* Check for presence/absence of plugin bundle patches on all agents

Providing host credentials for every monitored host running an agent enables the following
additional checks in checksec13R2.sh:
* Check for presence/absence of the latest Java version on all agents

Login to EMCLI as SYSMAN before running this script. If you already have an CHECKSEC account,
running this script will delete and recreate it with a new password.

Continue? [y/n]
Continuing...

Synchronized successfully
User "CHECKSEC" deleted successfully

User "CHECKSEC" created successfully

Created user CHECKSEC with password: [redacted]

Now CHECKSEC needs preferred credentials for the repository DB and repository DB host.
Your repository DB target name is oemdb.domain.com
Enter the credential name for the repository DB Normal Database Credentials: DB-OEMDB-SYSTEM
Enter the credential name for the repository DB SYSDBA Database Credentials: DB-OEMDB-SYS
Enter the credential name for the repository DB Database Host Credentials: HOST-OMSHOST-ORACLE

Validating that supplied credentials exist.

Credentials "DB-OEMDB-SYSTEM:SYSMAN" tested successfully
Credentials "DB-OEMDB-SYS:SYSMAN" tested successfully
Credentials "HOST-OMSHOST-ORACLE:SYSMAN" tested successfully

Granting CHECKSEC GET_CREDENTIAL access to supplied credentials.
Privileges granted to user/role "CHECKSEC" successfully

Confirmed supplied credentials exist and granted to CHECKSEC.

The next section asks you to supply credentials for the account used to run the Oracle Management Agent.

You will receive a separate prompt for each agent. Enter 'done' (without quotes) to skip this step.

If you provide these credentials, checksec13R2.sh can report on the Java version used by your agents.

Generating a list of all agent targets.
Now loop through all agent targets and provide named credentials for the agent user account on each host.

Enter the credential name to login as the agent user for host1.domain.com:3872: HOST-HOST1-ORAAGENT
Credentials "HOST-HOST1-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host2.domain.com:3872: HOST-HOST2-ORAAGENT
Credentials "HOST-HOST2-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host3.domain.com:3872: HOST-HOST3-ORAAGENT
Credentials "HOST-HOST3-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host4.domain.com:1830: HOST-HOST4-ORAAGENT
Credentials "HOST-HOST4-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host5.domain.com:3872: HOST-HOST5-ORAAGENT
Credentials "HOST-HOST5-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host6.domain.com:1830: HOST-HOST6-ORAAGENT
Credentials "HOST-HOST6-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host7.domain.com:3872: HOST-HOST7-ORAAGENT
Credentials "HOST-HOST7-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host8.domain.com:3872: HOST-HOST8-ORAAGENT
Credentials "HOST-HOST8-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host9.domain.com:1830: HOST-HOST9-ORAAGENT
Credentials "HOST-HOST9-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host10.domain.com:3872: HOST-HOST10-ORAAGENT
Credentials "HOST-HOST10-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host11.domain.com:3872: HOST-HOST11-ORAAGENT
Credentials "HOST-HOST11-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host12.domain.com:3872: HOST-HOST12-ORAAGENT
Credentials "HOST-HOST12-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host13.domain.com:3872: HOST-HOST13-ORAAGENT
Credentials "HOST-HOST13-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host14.domain.com:3872: HOST-HOST14-ORAAGENT
Credentials "HOST-HOST14-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host15.domain.com:3872: HOST-HOST15-ORAAGENT
Credentials "HOST-HOST15-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host16.domain.com:3872: HOST-HOST16-ORAAGENT
Credentials "HOST-HOST16-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for omshost.domain.com:3872: HOST-OMSHOST-ORACLE
Credentials "HOST-OMSHOST-ORACLE:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host17.domain.com:3872: HOST-HOST17-ORAAGENT
Credentials "HOST-HOST17-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Enter the credential name to login as the agent user for host18.domain.com:3872: HOST-HOST18-ORAAGENT
Credentials "HOST-HOST18-ORAAGENT:SYSMAN" tested successfully
Privileges granted to user/role "CHECKSEC" successfully

Logging out of EMCLI
Logout successful

Logging in to EMCLI as CHECKSEC
Login successful

Setting preferred credentials DB-OEMDB-SYSTEM for CHECKSEC on oemdb.domain.com
Successfully set preferred credentials for target oemdb.domain.com:oracle_database.

Setting preferred credentials DB-OEMDB-SYS for CHECKSEC on oemdb.domain.com
Successfully set preferred credentials for target oemdb.domain.com:oracle_database.

Setting preferred credentials HOST-OMSHOST-ORACLE for CHECKSEC on oemdb.domain.com
Successfully set preferred credentials for target oemdb.domain.com:oracle_database.

Now assigning preferred credentials for agent targets.

Setting preferred credentials for CHECKSEC on host1.domain.com:3872
Successfully set preferred credentials for target host1.domain.com:host.

Setting preferred credentials for CHECKSEC on host2.domain.com:3872
Successfully set preferred credentials for target host2.domain.com:host.

Setting preferred credentials for CHECKSEC on host3.domain.com:3872
Successfully set preferred credentials for target host3.domain.com:host.

Setting preferred credentials for CHECKSEC on host4.domain.com:1830
Successfully set preferred credentials for target host4.domain.com:host.

Setting preferred credentials for CHECKSEC on host5.domain.com:3872
Successfully set preferred credentials for target host5.domain.com:host.

Setting preferred credentials for CHECKSEC on host6.domain.com:1830
Successfully set preferred credentials for target host6.domain.com:host.

Setting preferred credentials for CHECKSEC on host7.domain.com:3872
Successfully set preferred credentials for target host7.domain.com:host.

Setting preferred credentials for CHECKSEC on host8.domain.com:3872
Successfully set preferred credentials for target host8.domain.com:host.

Setting preferred credentials for CHECKSEC on host9.domain.com:1830
Successfully set preferred credentials for target host9.domain.com:host.

Setting preferred credentials for CHECKSEC on host10.domain.com:3872
Successfully set preferred credentials for target host10.domain.com:host.

Setting preferred credentials for CHECKSEC on host11.domain.com:3872
Successfully set preferred credentials for target host11.domain.com:host.

Setting preferred credentials for CHECKSEC on host12.domain.com:3872
Successfully set preferred credentials for target host12.domain.com:host.

Setting preferred credentials for CHECKSEC on host13.domain.com:3872
Successfully set preferred credentials for target host13.domain.com:host.

Setting preferred credentials for CHECKSEC on host14.domain.com:3872
Successfully set preferred credentials for target host14.domain.com:host.

Setting preferred credentials for CHECKSEC on host15.domain.com:3872
Successfully set preferred credentials for target host15.domain.com:host.

Setting preferred credentials for CHECKSEC on host16.domain.com:3872
Successfully set preferred credentials for target host16.domain.com:host.

Setting preferred credentials for CHECKSEC on omshost.domain.com:3872
Successfully set preferred credentials for target omshost.domain.com:host.

Setting preferred credentials for CHECKSEC on host17.domain.com:3872
Successfully set preferred credentials for target host17.domain.com:host.

Setting preferred credentials for CHECKSEC on host18.domain.com:3872
Successfully set preferred credentials for target host18.domain.com:host.

All finished. User CHECKSEC now logged in to EMCLI.

Now go run the checksec13R2.sh script.

As a reminder, user CHECKSEC has password [redacted].

Previous Versions

Fix: Plugin error when upgrading EM13cR1 agent to EM13cR2 (13.2) on Windows 2008 R2 x64

I ran into the following issue while attempting to upgrade to the Oracle management agent on my one Windows (2008 R2, x64) server to the 13.2.0.0.0 version distributed with Oracle Enterprise Manager 13.2. The agent upgrade repeatedly failed in the “Upgrading Management Agent” step with an error message:


Exit Code :0
The version is 13.1.0.0.0
Checking for the version 13.2.0.0.0
The agent is not upgraded successfully
[...]
Plugins upgrade failed.
Plugin upgrade failed.
Check the file E:/agent13c/agent_inst/install/plugins_upgrade.txt.status on agent for plugin upgrade status.
Check latest E:/agent13c/agent_inst/install/logs/agentplugindeploy_.log
Plugins upgrade failed.0

The log file referenced in the error message contains some, but not much, additional information:


The command executed for discovery at _2016_10_12_11_48_28 is : E:\agent13c\agent_inst\bin\emctl_upgrade.bat set_discovery_root plugin oracle.sysman.si E:\agent13c\agent_13.2.0.0.0\plugins\oracle.sysman.si.discovery.plugin_13.2.1.0.0 13.2.1.0.0
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
return value is : 65280
Install case : set_discovery_root failure...existing on error value 65280

I fixed this issue by installing the Microsoft Visual C++ 2010 Service Pack 1 Redistributable Package MFC Security Update package on the server. Finally, create a new directory called “prerelogs” inside of the …\agent_13.2.0.0.0\cfgtoollogs\ directory, or else my deployment hung forever during the “Performing prerequisite checks” step. Note that “prerelogs” does not contain the letter q, do not use ‘prereqlogs’. The agent deployment uses this directory to extract software for running the prerequisite checks; possibly setting SCRATCHPATH or another extra parameter would resolve this issue without manually creating the directory, but this worked for me.

After installation of the VC++ 2010 package, which did not require a reboot, followed by creating of the prerelogs directory, I repeated the agent upgrade and it completed successfully.

This issue may or may not occur on other Windows versions, I have not heard any other reports of problems. The first release of EM13c did not seem to require this package.

Script to automate lock down of all EM13c agents to TLSv1.2 with EMCLI

I could not find any obvious documentation about locking down Oracle Enterprise Manager 13c management agents to forbid TLSv1 and TLSv1.1, permitting only TLSv1.2, so I went looking and found the emdpropdefs.xml file in $AGENT_HOME/agent_13.1.0.0.0/sysman/admin/ that documents the existence of the minimumTLSVersion property in emd.properties:

name='minimumTLSVersion'
modifiable='true'
defaultValue='TLSv1'
description='The oldest version of the TLS protocol which this agent should support when accepting connections or initiating connections to the OMS. Currently supported values are "TLSv1", "TLSv1.1", and "TLSv1.2".'
valueType='String'
advanced='true'
migrate='source'
filename='emd.properties'
category='Runtime Settings'
internal='true'
restartRequired='true'

I tested this parameter on my OMS server agent, restarted the agent, and confirmed with my Securing Oracle Enterprise Manager 13c script that the agent no longer allowed connections using any protocol other than TLSv1.2. Next I wanted to automated this, to avoid the effort of manually changing this property on each agent and then restarting that agent, so I went directly to EMCLI which allows EM13c admins to (among many other things) set agent properties and restart agents. I then created a script to fetch a list of all agents, check for the TLS protocols each agent permits, and then apply the change and restart the agent for every agent that I had not already locked down. I have copied this script below.

Before using the script, you must login to EMCLI using “emcli login -username=yourusername” and provide your password. For security reasons I elected not to wrap the EMCLI login within this script; that way you do not have to trust my script to handle your password securely, as the script never sees your password. For the step to restart your agents to work correctly, you need to make sure that your EM13c user account has preferred host credentials set for your agent targets that can successfully login to the host server and restart the agent.

Here is a copy of the script, followed by the (anonymized) output from a sample run. Someday soon I will get set up on github to make it easier to retrieve my scripts, but for now you can copy and paste this. This script expects to find the emcli binary inside of the $MW_HOME/bin directory, so make sure you have $MW_HOME set before running it, or provide the full path to EMCLI within the script. It will also log you out of EMCLI when the script completes.


#!/bin/bash
#
# This script will retrieve a list of agents from your EM13c environment,
# determine if they allow connections using TLS protocol versions older
# than TLSv1.2, and then disable all protocols older than TLSv1.2.
#
# Finally it will restart each modified agent to apply the change.
#
# You need to login to EMCLI first before running this script.
#
# Released v0.1: Initial beta release 5 Oct 2016
#
#
# From: @BrianPardy on Twitter
# https://pardydba.wordpress.com/
#
# Known functional on Linux x86-64, may work on Solaris and AIX.

EMCLI=$MW_HOME/bin/emcli

if [[ -x "/usr/sfw/bin/gegrep" ]]; then
GREP=/usr/sfw/bin/gegrep
else
GREP=`which grep`
fi

OPENSSL=`which openssl`

if [[ -x "/usr/bin/openssl1" && -f "/etc/SuSE-release" ]]; then
OPENSSL=`which openssl1`
fi

OPENSSL_HAS_TLS1_2=`$OPENSSL s_client help 2>&1 | $GREP -c tls1_2`

$EMCLI sync
NOT_LOGGED_IN=$?

if [[ $NOT_LOGGED_IN > 0 ]]; then
echo "Login to EMCLI with \"$EMCLI login -username=USER\" then run this script again"
exit 1
fi

for agent in `$EMCLI get_targets -targets=oracle_emd | grep oracle_emd | awk '{print $4}'`
do
echo
if [[ $OPENSSL_HAS_TLS1_2 > 0 ]]; then
echo -n "Checking TLSv1 on $agent... "

OPENSSL_RETURN=`echo Q | $OPENSSL s_client -prexit -connect $agent -tls1 2>&1 | $GREP Cipher | $GREP -c 0000`

if [[ $OPENSSL_RETURN == 0 ]]; then
echo "allows TLSv1"
else
echo "already forbids TLSv1"
fi
fi

if [[ $OPENSSL_HAS_TLS1_2 > 0 ]]; then
echo -n "Checking TLSv1.1 on $agent... "

OPENSSL_TLS11_RETURN=`echo Q | $OPENSSL s_client -prexit -connect $agent -tls1_1 2>&1 | $GREP Cipher | $GREP -c 0000`

if [[ $OPENSSL_RETURN == 0 ]]; then
echo "allows TLSv1.1"
else
echo "already forbids TLSv1.1"
fi
fi

if [[ $OPENSSL_RETURN == 0 || $OPENSSL_TLS11_RETURN == 0 ]]; then
$EMCLI set_agent_property -agent_name=$agent -name=minimumTLSVersion -value=TLSv1.2 -new

echo
echo "Restarting $agent to apply changes"
$EMCLI restart_agent -agent_name=$agent -credential_setname="HostCreds"
RESTART_RETURN=$?

if [[ $RESTART_RETURN != 0 ]]; then
echo "Unable to restart agent: restart agent manually or set preferred host credentials for agent"
fi
fi
done

$EMCLI logout

exit 0

Sample (anonymized) output below. Note how the script cannot restart an agent lacking preferred host credentials. In this case, I assign preferred host credentials and then re-run the script to complete the process.


Synchronized successfully

Checking TLSv1 on server1.subdomain.domain.com:1830... already forbids TLSv1
Checking TLSv1.1 on server1.subdomain.domain.com:1830... already forbids TLSv1.1

Checking TLSv1 on server2.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server2.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server3.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server3.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server4.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server4.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server5.domain.com:1830... already forbids TLSv1
Checking TLSv1.1 on server5.domain.com:1830... already forbids TLSv1.1

Checking TLSv1 on server6.domain.com:1830... already forbids TLSv1
Checking TLSv1.1 on server6.domain.com:1830... already forbids TLSv1.1

Checking TLSv1 on server7.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server7.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server8.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server8.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server9.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server9.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server10.domain.com:1830... already forbids TLSv1
Checking TLSv1.1 on server10.domain.com:1830... already forbids TLSv1.1

Checking TLSv1 on server11.domain.com:1830... already forbids TLSv1
Checking TLSv1.1 on server11.domain.com:1830... already forbids TLSv1.1

Checking TLSv1 on server12.domain.com:1830... already forbids TLSv1
Checking TLSv1.1 on server12.domain.com:1830... already forbids TLSv1.1

Checking TLSv1 on omshost.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on omshost.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server13.domain.com:3872... allows TLSv1
Checking TLSv1.1 on server13.domain.com:3872... allows TLSv1.1
Agent Property minimumTLSVersion has been successfully updated to the value TLSv1.2.

Restarting server13.domain.com:3872 to apply changes
The Restart operation is in progress for the Agent: server13.domain.com:3872
The Agent "server13.domain.com:3872" has been restarted successfully.
---------------------
Operation Output
---------------------
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.Stopping agent ... stopped.Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.Starting agent ................ started.

Checking TLSv1 on server14.domain.com:1830... allows TLSv1
Checking TLSv1.1 on server14.domain.com:1830... allows TLSv1.1
Agent Property minimumTLSVersion has been successfully updated to the value TLSv1.2.

Restarting server14.domain.com:1830 to apply changes
The Restart operation is in progress for the Agent: server14.domain.com:1830
Unable to restart agent: restart agent manually or set preferred host credentials for agent

Checking TLSv1 on server15.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server15.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server16.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server16.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server17.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server17.domain.com:3872... already forbids TLSv1.1
Logout successful

Securing Oracle Enterprise Manager 13c

[20170418 NOTE: I have upgraded the patches referenced in this script to reflect the latest (20170418) PSU patch for EM13cR1. I no longer have an EM13cR1 environment available with which to test this script, so please feel free to report issues or to submit a git pull request. I have now placed this script on github: https://raw.githubusercontent.com/brianpardy/em13c/master/checksec13.sh.]
[20161013 NOTE: I have upgraded to EM13cR2, and this script still works as expected. If you attempt to run it on an EM13cR2 environment please take note that all of the patch recommendations listed apply to the older EM13cR1 release and will provide incorrect results on 13.2. The TLS, certificate, and cipher strength tests all function correctly on 13.2.]

Introduction

This post continues my series on securing Oracle Enterprise Manager environments with some updates relevant to EM13c. Oracle has made significant security improvements with Oracle Enterprise Manager 13c over the prior 12c version, first released in October 2011, more than four and a half years ago at this point. In the interest of security, I have to strongly recommend that any sites still using EM12c upgrade to (or perform a fresh installation of) EM13c as soon as possible. More recent versions of EM12c like 12.1.0.5 (June 2015) continue to use the same technology stack as the initial release, and the world of security has massively changed since then. Notably, EM13c uses Java 7, WebLogic 12.1.3, and disables SSLv3 out of the box.

Just to recap, back at the EM12c original release date:

  • Practically nobody had ever heard of Edward Snowden
  • The first release of Java 7 celebrated its three month birthday
  • Two months later, Oracle released WebLogic 12c; EM12c users remained on WebLogic 10.3.6
  • One month earlier, the public learned of the BEAST attack and people still believed that using RC4 (immune to BEAST) as a workaround improved security (spoiler warning: it did not)
  • We had three years to wait before the POODLE vulnerability caused vendors to recognize the need to disable SSLv3 (you DID disable SSLv3, right?)
  • Oracle still considered the MD5 hashing algorithm good enough to use in self-signed certificates produced by EM12c, despite flaws known to exist since 1996
  • Web browsers considered the SHA-1 hashing algorithm, now also deprecated due to brokenness, good enough to use

As the security world’s known unknowns collapsed around us, proactive EM12c administrators sought to make the best of their lot. Outside of Oracle, I and others poked at the software and wrote blog articles, while inside Oracle effort proceeded to support more recent Java releases that brought with them better cipher suites and hashing algorithms, as well as the usual security fixes. This process took some time for all involved and hit a few bumps along the way.

I do not intend in this post to review general day-to-day EM13c security design such as user roles or privileges, object level security within OEM, or integration with identity providers like LDAP; only the infrastructure level issues that tend to change in brief large bursts as new attacks come out. See this excellent list of EM13c blogs, links and videos that Philip Brown has provided for more details on these and other items.

On to EM13c

EM13c admins need to keep an eye on the same sorts of items as with EM12c. We really should read the documentation, even if only the Security Guide. I admit I often do not. It contains good information.

Patches

I consider it critical for admins to keep up with the OEM periodic patches, particularly security patches. The script below covers patches up to and including March 31, 2016. I plan to update again after the next set of Oracle security patches arrives, likely mid-April.

Certificates

The process for applying certificates on EM13c does not appear to have changed significantly from the prior version. I have confirmed that the new “omspatcher” tool that replaces opatchauto when applying a system patch to the OMS works perfectly fine with certificates on WebLogic that use the SHA-256 hashing algorithm. Given the upcoming deprecation of SHA-1 across all major browsers I do not see any valid reason not to use SHA-256 with new certificates.

Ciphersuites

Out of the box, my EM13c installation rejected weak ciphersuites and accepted the strong ones. Unfortunately it still accepted some that these versions of Java and OpenSSL consider as MEDIUM strength, so I want to disable those across the entire environment, leaving only the strongest ciphersuites available in this release and permitting other ciphersuites only where necessary.

[UPDATE 20160518: Please see MOS note 2138391.1 for the official procedure to disable weak cipher suites in EM13c.]

We will have to live with these unwanted ciphersuites enabled until Oracle provides a supported procedure to disable them across the entire stack. I have performed some preliminary tests and I find it very easy to get OEM into a situation where it cannot startup after manually editing config files that define enabled ciphersuites. The script below will identify ports permitting ciphersuites you may wish to disable when a supported method becomes available.

UPDATE 20160720: Take particular care of watching the ciphersuites accepted by your agents if you upgrade the JDK that the agents use. I just attempted a JDK update on an agent from the distributed version to 1.7.0-111, and that agent began to accept LOW and MEDIUM strength ciphersuites again, thus I have omitted JDK updates for agents from the check script.

Security Checkup

Below I provide an early version of the script I use to validate EM13c security configuration. I based this on my earlier EM12c script, linked above. The script will become more useful once I implement patch level checking after release of the first set of EM13c patches, but for the moment it will inspect your EM13c environment to identify relevant ports and confirm that your system will not respond to SSLv2 or SSLv3 requests, does respond to TLSv1 requests, supports HIGH, but not LOW or MEDIUM strength ciphersuites (as defined by the version of OpenSSL installed on your OMS host), and finally checks for the presence of demonstration-not-for-production-use certificates and self-signed certificates.

(A caveat on self-signed certificate checking: OpenSSL, not this script, performs the check, therefore if OpenSSL cannot validate your certificate to a trusted root, this script cannot either. If a well known certification authority has signed your certificates, OpenSSL should validate them successfully, but it may not do so if you use an internal certificate authority to sign certificates. In that case you should install a copy of your internal CA as a trusted root certificate in the system trust store so that OpenSSL can validate your EM13c certificates. I cannot document this process for every OS but Linux users should look to the documentation for the update-ca-certificates or update-ca-trust commands. If my script below incorrectly reports your certificate as self-signed, you can ignore the finding or address the issue at the OpenSSL level.)

EM13c TLS Security Checkup Script

[LATEST UPDATE: 20161004, adds 20160920 patches and fixes TLSv1 vs TLSv1.2 bugs, version 0.9]. Thank you to Bob Schuppin who reported a bug in the use of TLSv1 to check certificate and cipher suite usage in a TLSv1.2-only site. I have modified the relevant checks to use TLSv1.2 if supported by your OpenSSL version or to stick with TLSv1 if not supported.

[PRIOR UPDATE: 20160914 bugfix and enhancements, no new patch checks, version 0.8]. Thank you to Paige who reported a bug in the check of the SSL_CIPHER_SUITES parameter. I had a typo in the cipher suite names for the SSL_CIPHER_SUITES parameter, which I have now fixed. In researching this I realized that this parameter provides control over SSL/TLS authentication for clients, which I do not use in my environment. Instead I use native SQL*Net encryption, controlled by the various ENCRYPTION_(CLIENT|SERVER), ENCRYPTION_TYPES_(CLIENT|SERVER), CRYPTO_CHECKSUM_(CLIENT|SERVER), and CRYPTO_CHECKSUM_TYPES_(CLIENT|SERVER) parameters, which I have added into the script. The script will check to make sure that you do not permit MD5 as a SQL*Net checksum algorithm and that you do not permit DES, DES40, 3DES112, nor any of the RC4_ algorithms for SQL*Net encryption. Unfortunately due to bug 23587582, you will encounter problems promoting targets in OEM unless you allow use of the 3DES168 encryption algorithm and SHA1 hashing algorithm. Generally I would prefer to disable both of those for security reasons but I will permit them in this script as long as they remain necessary for full OEM functionality.

[PRIOR UPDATE: 20160819 for 20160816 bundle patches, version 0.7]. With this update, I have added support for TLSv1.1 and TLSv1.2 to the protocol checks. I have also added support for the optional SLES11 openssl1 package which provides a newer OpenSSL supporting TLSv1.1 and TLSv1.2 for systems on SLES11 like mine. The script will now dynamically determine (by parsing the “openssl s_client help” output) if your OpenSSL version supports TLSv1.2. If your OpenSSL version DOES support TLSv1.2, the script will now flag any support of TLSv1 or TLSv1.1 as a failure in your OEM stack. If your OpenSSL does NOT support TLSv1.2, the script will consider TLSv1 support in OEM as acceptable. If you notice problems with this new functionality please let me know.

Compatibility

Only tested on Linux x86-64, but may work on AIX and Solaris as the EM12c version I built this upon did work there. Planned future enhancements include checking that you have disabled non-encrypted HTTP access to EM13c components, upgraded Java to the most recent EM13c-supported release, and more.

You can download the latest version of the script from github: https://raw.githubusercontent.com/brianpardy/em13c/master/checksec13.sh.

EM13c TLS Security Checkup Script Sample Output


Performing EM13c security checkup version 0.9 on omshost.domain.com at Tue Oct 4 11:04:43 EDT 2016.

Using port definitions from configuration files
/etc/oragchomelist
/oracle/oem/gc_inst/em/EMGC_OMS1/emgc.properties
/oracle/oem/gc_inst/em/EMGC_OMS1/embip.properties

Agent port found at omshost.domain.com:3872
BIPublisher port found at omshost.domain.com:9803
BIPublisherOHS port found at omshost.domain.com:9851
NodeManager port found at omshost.domain.com:7403
OMSconsole port found at omshost.domain.com:7802
OMSproxy port found at omshost.domain.com:7301
OMSupload port found at omshost.domain.com:4903
WLSadmin found at omshost.domain.com:7102

Repository DB version=12.1.0.2.0 SID=oemdb host=omshost.domain.com

Using OPENSSL=/usr/bin/openssl1 (has TLS1_2=2)
Repository DB on OMS server, will check patches/parameters in /oracle/oem/product/12.1.0/db

(1) Checking SSL/TLS configuration (see notes 1602983.1, 1477287.1, 1905314.1)

(1a) Forbid SSLv2 connections
Confirming ssl2 disabled for Agent at omshost.domain.com:3872... OK
Confirming ssl2 disabled for BIPublisher at omshost.domain.com:9803... OK
Confirming ssl2 disabled for NodeManager at omshost.domain.com:7403... OK
Confirming ssl2 disabled for BIPublisherOHS at omshost.domain.com:9851... OK
Confirming ssl2 disabled for OMSconsole at omshost.domain.com:7802... OK
Confirming ssl2 disabled for OMSproxy at omshost.domain.com:7301... OK
Confirming ssl2 disabled for OMSupload at omshost.domain.com:4903... OK
Confirming ssl2 disabled for WLSadmin at omshost.domain.com:7102... OK

(1b) Forbid SSLv3 connections
Confirming ssl3 disabled for Agent at omshost.domain.com:3872... OK
Confirming ssl3 disabled for BIPublisher at omshost.domain.com:9803... OK
Confirming ssl3 disabled for NodeManager at omshost.domain.com:7403... OK
Confirming ssl3 disabled for BIPublisherOHS at omshost.domain.com:9851... OK
Confirming ssl3 disabled for OMSconsole at omshost.domain.com:7802... OK
Confirming ssl3 disabled for OMSproxy at omshost.domain.com:7301... OK
Confirming ssl3 disabled for OMSupload at omshost.domain.com:4903... OK
Confirming ssl3 disabled for WLSadmin at omshost.domain.com:7102... OK

(1c) Forbid TLSv1 connections
Confirming tls1 disabled for Agent at omshost.domain.com:3872... FAILED
Confirming tls1 disabled for BIPublisher at omshost.domain.com:9803... FAILED
Confirming tls1 disabled for NodeManager at omshost.domain.com:7403... FAILED
Confirming tls1 disabled for BIPublisherOHS at omshost.domain.com:9851... FAILED
Confirming tls1 disabled for OMSconsole at omshost.domain.com:7802... FAILED
Confirming tls1 disabled for OMSproxy at omshost.domain.com:7301... FAILED
Confirming tls1 disabled for OMSupload at omshost.domain.com:4903... FAILED
Confirming tls1 disabled for WLSadmin at omshost.domain.com:7102... FAILED

(1c) Forbid TLSv1.1 connections
Confirming tls1_1 disabled for Agent at omshost.domain.com:3872... FAILED
Confirming tls1_1 disabled for BIPublisher at omshost.domain.com:9803... FAILED
Confirming tls1_1 disabled for NodeManager at omshost.domain.com:7403... FAILED
Confirming tls1_1 disabled for BIPublisherOHS at omshost.domain.com:9851... FAILED
Confirming tls1_1 disabled for OMSconsole at omshost.domain.com:7802... FAILED
Confirming tls1_1 disabled for OMSproxy at omshost.domain.com:7301... FAILED
Confirming tls1_1 disabled for OMSupload at omshost.domain.com:4903... FAILED
Confirming tls1_1 disabled for WLSadmin at omshost.domain.com:7102... FAILED

(1c) Permit TLSv1.2 connections
Confirming tls1_2 available for Agent at omshost.domain.com:3872... OK
Confirming tls1_2 available for BIPublisher at omshost.domain.com:9803... OK
Confirming tls1_2 available for NodeManager at omshost.domain.com:7403... OK
Confirming tls1_2 available for BIPublisherOHS at omshost.domain.com:9851... OK
Confirming tls1_2 available for OMSconsole at omshost.domain.com:7802... OK
Confirming tls1_2 available for OMSproxy at omshost.domain.com:7301... OK
Confirming tls1_2 available for OMSupload at omshost.domain.com:4903... OK
Confirming tls1_2 available for WLSadmin at omshost.domain.com:7102... OK

(2) Checking supported ciphers at SSL/TLS endpoints (see notes 2138391.1, 1067411.1)
Checking LOW strength ciphers on Agent (omshost.domain.com:3872, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on Agent (omshost.domain.com:3872)... OK
Checking HIGH strength ciphers on Agent (omshost.domain.com:3872)... OK

Checking LOW strength ciphers on BIPublisher (omshost.domain.com:9803, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on BIPublisher (omshost.domain.com:9803)... OK
Checking HIGH strength ciphers on BIPublisher (omshost.domain.com:9803)... OK

Checking LOW strength ciphers on NodeManager (omshost.domain.com:7403, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on NodeManager (omshost.domain.com:7403)... OK
Checking HIGH strength ciphers on NodeManager (omshost.domain.com:7403)... OK

Checking LOW strength ciphers on BIPublisherOHS (omshost.domain.com:9851, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on BIPublisherOHS (omshost.domain.com:9851)... OK
Checking HIGH strength ciphers on BIPublisherOHS (omshost.domain.com:9851)... OK

Checking LOW strength ciphers on OMSconsole (omshost.domain.com:7802, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on OMSconsole (omshost.domain.com:7802)... OK
Checking HIGH strength ciphers on OMSconsole (omshost.domain.com:7802)... OK

Checking LOW strength ciphers on OMSproxy (omshost.domain.com:7301, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on OMSproxy (omshost.domain.com:7301)... OK
Checking HIGH strength ciphers on OMSproxy (omshost.domain.com:7301)... OK

Checking LOW strength ciphers on OMSupload (omshost.domain.com:4903, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on OMSupload (omshost.domain.com:4903)... OK
Checking HIGH strength ciphers on OMSupload (omshost.domain.com:4903)... OK

Checking LOW strength ciphers on WLSadmin (omshost.domain.com:7102, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on WLSadmin (omshost.domain.com:7102)... OK
Checking HIGH strength ciphers on WLSadmin (omshost.domain.com:7102)... OK

(3) Checking self-signed and demonstration certificates at SSL/TLS endpoints (see notes 1367988.1, 1399293.1, 1593183.1, 1527874.1, 123033.1, 1937457.1)
Checking certificate at Agent (omshost.domain.com:3872, protocol tls1_2)... FAILED - Found self-signed certificate
Checking demo certificate at Agent (omshost.domain.com:3872, protocol tls1_2)... OK
Checking certificate at BIPublisher (omshost.domain.com:9803, protocol tls1_2)... OK
Checking demo certificate at BIPublisher (omshost.domain.com:9803, protocol tls1_2)... OK
Checking certificate at NodeManager (omshost.domain.com:7403, protocol tls1_2)... OK
Checking demo certificate at NodeManager (omshost.domain.com:7403, protocol tls1_2)... OK
Checking certificate at BIPublisherOHS (omshost.domain.com:9851, protocol tls1_2)... OK
Checking demo certificate at BIPublisherOHS (omshost.domain.com:9851, protocol tls1_2)... OK
Checking certificate at OMSconsole (omshost.domain.com:7802, protocol tls1_2)... OK
Checking demo certificate at OMSconsole (omshost.domain.com:7802, protocol tls1_2)... OK
Checking certificate at OMSproxy (omshost.domain.com:7301, protocol tls1_2)... OK
Checking demo certificate at OMSproxy (omshost.domain.com:7301, protocol tls1_2)... OK
Checking certificate at OMSupload (omshost.domain.com:4903, protocol tls1_2)... OK
Checking demo certificate at OMSupload (omshost.domain.com:4903, protocol tls1_2)... OK
Checking certificate at WLSadmin (omshost.domain.com:7102, protocol tls1_2)... OK
Checking demo certificate at WLSadmin (omshost.domain.com:7102, protocol tls1_2)... OK

(4) Checking EM13c Oracle home patch levels against 20 Sep 2016 baseline (see notes 1664074.1, 1900943.1, 822485.1, 1470197.1, 1967243.1)

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) PSU 12.1.0.2.160719 (JUL2016) (23054246)... OK
Patch 23054246 : applied on Wed Jul 20 12:01:53 EDT 2016 Patch description: "Database Patch Set Update : 12.1.0.2.160719 (23054246)"

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) ORACLE JAVAVM COMPONENT 12.1.0.2.160719 DATABASE PSU (JUL2016) (23177536)... OK
Patch 23177536 : applied on Wed Jul 20 12:03:14 EDT 2016 21566993, 22670413, 19699946, 23177536, 22118835, 22118851, 19895326

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_TYPES_SERVER parameter (76629.1, 2167682.1)... OK
(AES128,AES256,AES192,3DES168)

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_SERVER parameter (76629.1, 2167682.1)... OK
requested

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_TYPES_CLIENT parameter (76629.1, 2167682.1)... OK
(AES128,AES256,AES192,3DES168)

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_CLIENT parameter (76629.1, 2167682.1)... OK
requested

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_TYPES_SERVER parameter (76629.1, 2167682.1)... OK
(SHA1)

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_SERVER parameter (76629.1, 2167682.1)... OK
requested

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_TYPES_CLIENT parameter (76629.1, 2167682.1)... OK
(SHA1)

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_CLIENT parameter (76629.1, 2167682.1)... OK
requested

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SSL_VERSION parameter (1545816.1)... OK
1.0

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SSL_CIPHER_SUITES parameter (1545816.1)... OK
(SSL_RSA_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_AES_256_CBC_SHA)

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) listener.ora SSL_VERSION parameter (1545816.1)... OK
1.0

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) listener.ora SSL_CIPHER_SUITES parameter (1545816.1)... OK
(SSL_RSA_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_AES_256_CBC_SHA)

(4c) *UPDATED* OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM-AGENT BUNDLE PATCH 13.1.0.0.160920 (24437699)... OK
Patch 24437699 : applied on Tue Sep 27 12:08:23 EDT 2016 24437699, 21779343, 22616051, 23759799, 22988508, 23089106, 23581450

(4c) *UPDATED* OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM DB PLUGIN BUNDLE PATCH 13.1.1.0.160920 MONITORING (24545984)... OK
Patch 24545984 : applied on Tue Sep 27 13:46:08 EDT 2016 22908077, 23294830, 22503390, 23075475, 23697777, 24545984, 24296310

(4c) *UPDATED* OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM DB PLUGIN BUNDLE PATCH 13.1.1.0.160920 DISCOVERY (24545989)... OK
Patch 24545989 : applied on Tue Sep 27 13:46:11 EDT 2016 23523964, 23294839, 24545989, 23226583, 24408840

(4c) *UPDATED* OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM FMW PLUGIN BUNDLE PATCH 13.1.1.0.160920 MONITORING (24658006)... OK
Patch 24658006 : applied on Tue Sep 27 13:46:13 EDT 2016 22834135, 23007497, 22447329, 22936491, 24658006, 23294872, 23306887

(4c) OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM SI PLUGIN BUNDLE PATCH 13.1.1.0.160719 MONITORING (23697783)... OK
Patch 23697783 : applied on Wed Jul 20 10:53:57 EDT 2016 22128210, 23338028, 23189991, 22823189, 21253819, 23697783, 23208587

(4c) OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM SI PLUGIN BUNDLE PATCH 13.1.1.0.160531 DISCOVERY (23294895)... OK
Patch 23294895 : applied on Thu Jun 16 11:28:18 EDT 2016 23197299, 23294895

(4c) OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM OH PLUGIN BUNDLE PATCH 13.1.1.0.160429 (23135564)... OK
Patch 23135564 : applied on Wed May 11 13:21:35 EDT 2016 22521822, 23135564

(4d) *UPDATED* OMS HOME (/oracle/oem/Middleware13cR1) ENTERPRISE MANAGER FOR OMS PLUGINS 13.1.1.0.160920 (24546113)... OK
oracle.sysman.emas.oms.plugin/13.1.1.0.0 Plugin 24546113 24437669 oracle.sysman.cfw.oms.plugin/13.1.1.0.0 Plugin 24546113 24437711 oracle.sysman.db.oms.plugin/13.1.1.0.0 Plugin 24546113 24437646 oracle.sysman.xa.oms.plugin/13.1.1.0.0 Plugin 24546113 24437656

(4d) (/oracle/oem/Middleware13cR1) WLS PATCH SET UPDATE 12.1.3.0.160719 (23094292)... OK
Patch 23094292 : applied on Wed Jul 20 12:27:53 EDT 2016

(4f) OMS HOME (/oracle/oem/Middleware13cR1) ENTERPRISE MANAGER BASE PLATFORM PATCH SET UPDATE 13.1.0.0.160719 (23134365)... OK
oracle.sysman.top.oms/13.1.0.0.0 Core 23134365 23134365

(5) Checking EM13c Java patch levels against 20 Sep 2016 baseline (see notes 1492980.1, 1616397.1)

(5a) WLS (/oracle/oem/Middleware13cR1/oracle_common/jdk) JAVA SE JDK VERSION 1.7.0-111 (13079846)... OK
1.7.0_111

Failed test count: 17 - Review output

sslcheck:Agent @ omshost.domain.com:3872:tls1 protocol connection allowed
sslcheck:BIPublisher @ omshost.domain.com:9803:tls1 protocol connection allowed
sslcheck:NodeManager @ omshost.domain.com:7403:tls1 protocol connection allowed
sslcheck:BIPublisherOHS @ omshost.domain.com:9851:tls1 protocol connection allowed
sslcheck:OMSconsole @ omshost.domain.com:7802:tls1 protocol connection allowed
sslcheck:OMSproxy @ omshost.domain.com:7301:tls1 protocol connection allowed
sslcheck:OMSupload @ omshost.domain.com:4903:tls1 protocol connection allowed
sslcheck:WLSadmin @ omshost.domain.com:7102:tls1 protocol connection allowed
sslcheck:Agent @ omshost.domain.com:3872:tls1_1 protocol connection allowed
sslcheck:BIPublisher @ omshost.domain.com:9803:tls1_1 protocol connection allowed
sslcheck:NodeManager @ omshost.domain.com:7403:tls1_1 protocol connection allowed
sslcheck:BIPublisherOHS @ omshost.domain.com:9851:tls1_1 protocol connection allowed
sslcheck:OMSconsole @ omshost.domain.com:7802:tls1_1 protocol connection allowed
sslcheck:OMSproxy @ omshost.domain.com:7301:tls1_1 protocol connection allowed
sslcheck:OMSupload @ omshost.domain.com:4903:tls1_1 protocol connection allowed
sslcheck:WLSadmin @ omshost.domain.com:7102:tls1_1 protocol connection allowed
certcheck:Agent @ omshost.domain.com:3872 found self-signed certificate

Visit https://pardydba.wordpress.com/2016/04/05/securing-oracle-enterprise-manager-13c/ for the latest version.

WORKAROUND: Unable to monitor Oracle XE 11gR2 with Oracle Enterprise Manager 13c

I have recently switched to using Oracle Enterprise Manager 13c (EM13c – 13.1), replacing my previous EM12c installation.  I elected to install a clean new environment instead of an upgrade, because my old install had been upgraded repeatedly going back to the initial release of EM12c and I wanted a fresh start.

I encountered only one difficult issue during the process. When I attempted to add one production Oracle XE 11gR2 database target, EM13c could not compute the target’s dynamic properties, leaving the target broken. Since you cannot submit jobs against a broken target, this prevented me from using EM13c to back up this database.  I had no comparable issues with XE as a target under EM12c.

The key metric errors that showed during this process included:
“Metric evaluation error start – Target {oracle_database.SID.domain.com} is broken: Dynamic Category property error,Get dynamic property error,No such metadata – No valid queryDescriptor or executionDescriptor found for target [oracle_database.SID.domain.com$30]”

and for the database system target:

“Metric evaluation error start – Received an exception when evaluating sev_eval_proc for:Target name = SID.domain.com_sys, metric_name = Response, metric_column = Status; Error msg = Target encountered metric erros; at least one member in in metric error”

I enabled debugging for the agent logs and attempted again to add the XE target.  Errors showing up in the logs included:

2016-01-15 12:10:05,905 [1806:4CE3192] DEBUG – Computing of dynamic property: [ComputeVC] is done (1 msec, error=true)

2016-01-15 12:10:06,452 [1806:F917F5F8] DEBUG – Computing of dynamic property: [GetDumpDestination] is done (0 msec, error=true)

2016-01-15 12:10:06,508 [1813:6EEEAC87] DEBUG – Computing of dynamic property: [DeduceAlertLogFile] is done (1 msec, error=true)

2016-01-15 12:11:18,779 [1830:CD3A325D] DEBUG – Error was added to oracle_database.SID.domain.com$23(0|MONITORED|false|true|<UF>): Invalid Input

2016-01-15 12:11:18,779 [1831:3657AE55] DEBUG – abandoning long op “CDProps:oracle_database.SID.domain.com:ComputeVC:GENERIC_TASK:Fri Jan 15 12:11:18 EST 2016”

2016-01-15 12:11:18,780 [1830:CD3A325D] DEBUG – Error during dynamic property ComputeVC calculation: critical=true, missingCatProps=[VersionCategory], missingProps=[VersionCategory] oracle.sysman.emSDK.agent.fetchlet.exception.FetchletException: Invalid Input

2016-01-15 12:35:38,038 INFO – Finished dynamic properties computation (total time 817 ms). Number of DP computed: 19. Number of DP with error: 3. Number of dynamic properties that were added: 132.

After reviewing the logs carefully (and posting this as a question in the MOS Oracle XE forum – https://community.oracle.com/thread/3892946) I eventually narrowed the issue down to a query that EM13c runs against DBA_REGISTRY_HISTORY in a target database when added. For database versions greater than 11.2 but less than 12.1.0.2, EM13c assumes that DBA_REGISTRY_HISTORY contains a BUNDLE_SERIES column.  This column does not exist in Oracle XE 11gR2, which reports a version string of 11.2.0.2.

This bug should eventually get a fix as EM13c gets patched, but in the meantime if you need to monitor an Oracle XE target with EM13c, the following workaround took care of the problem for me: create a new DBA_REGISTRY_HISTORY table containing a BUNDLE_SERIES column in your monitoring user’s schema in XE.  So, as user DBSNMP on XE, I ran:

SQL> create table dba_registry_history (ACTION_TIME TIMESTAMP(6), ACTION VARCHAR2(30), NAMESPACE VARCHAR2(30), VERSION VARCHAR2(30), ID NUMBER, BUNDLE_SERIES VARCHAR2(30), COMMENTS VARCHAR2(255));

Since one cannot patch XE, the real DBA_REGISTRY_HISTORY view has no rows and so you do not need to populate any data into this new table.

After adding the table, force a recalculation of dynamic properties by running the following against the EM13c management agent on the XE server:

$ emctl reload agent dynamicproperties SID.domain.com:oracle_database
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD recompute dynprops completed successfully

Once that completed successfully my XE target started to show the correct status in EM13c and I can submit jobs against the target.  All fixed.  I recommend deleting the DBSNMP.DBA_REGISTRY_HISTORY table once the bug gets fixed in OEM.

[EDIT 20160216: Oracle has documented this issue in MOS note EM13c: Database Target Status Shows “Dynamic Category property error” In 13c Cloud Control (Doc ID 2105001.1) and in bug 22592461 DATABASE TARGET STATUS SHOWS “DYNAMIC CATEGORY PROPERTY ERROR” IN 13C CONSOLE. Users on supported databases (e.g., not Oracle XE) should follow the resolution steps in that document instead to correct the real error.]

EM12c opatchauto, SHA256, and you

This post serves to document an issue I encountered after replacing expired SSL/TLS certificates on the server I use for Oracle Enterprise Manager 12c. To put it simply, using opatchauto to apply EM12c PSUs does not work if your WebLogic adminserver has a certificate installed that uses the SHA256 hashing algorithm.

[UPDATED 20151012: Please see this comment and this comment below, from Adam Robinson, who has provided a workaround that may work for you involving editing the opatchauto script to enable JSSE. As always, please consider workarounds requiring you to edit files as unsupported and at your own risk, but I would consider this fix superior to reverting back to the demo certificate every time you need to patch. You will need to repeat this fix every time you update OPatch in your OMS_HOME, though. Adam’s workaround does succeed in my environment.]

Error message

Expect to see the following error when running “opatchauto apply -analyze” or “opatchauto apply” against an installation with an SHA256-hashed certificate on the WLS adminserver:

oracle@omshost:/oracle/stage/21603255> opatchauto apply -analyze -property_file ~/property_file 
OPatch Automation Tool
Copyright (c) 2014, Oracle Corporation.  All rights reserved.


OPatchauto version : 11.1.0.12.3
OUI version        : 11.1.0.12.0
Running from       : /oracle/oem/Middleware12cR4/oms
Log file location  : /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatch/opatch2015-09-11_10-57-19AM_1.log

OPatchauto log file: /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatchauto/21603255/opatch_oms_2015-09-11_10-57-22AM_analyze.log



OPatchauto failed to establish JMX connection to weblogic server. This could be because of one (or) more of the following reasons:
1. Weblogic admin server URL that manages OMS application may not be right.
2. Weblogic admin server credentials (username, password) may not be right.
3. Virtual host configuration. If OMS, weblogic server are on virtual host configuration, Please make sure to add OPatchAuto.OMS_DISABLE_HOST_CHECK=true to command line and run again. (example: /oracle/oem/Middleware12cR4/oms/OPatch/opatchauto apply -analyze -property_file /home/oracle/property_file -invPtrLoc /oracle/oem/Middleware12cR4/oms/oraInst.loc  OPatchAuto.OMS_DISABLE_HOST_CHECK=true)

Please check above conditions and if error(s) still persist, Please contact Oracle support.


[ Error during Get weblogic Admin Server information Phase]. Detail: OPatchauto was not able to find right interview inputs.
OPatchauto failed: 
OPatchauto failed to establish JMX connection to weblogic server. This could be because of one (or) more of the following reasons:
1. Weblogic admin server URL that manages OMS application may not be right.
2. Weblogic admin server credentials (username, password) may not be right.
3. Virtual host configuration. If OMS, weblogic server are on virtual host configuration, Please make sure to add OPatchAuto.OMS_DISABLE_HOST_CHECK=true to command line and run again. (example: /oracle/oem/Middleware12cR4/oms/OPatch/opatchauto apply -analyze -property_file /home/oracle/property_file -invPtrLoc /oracle/oem/Middleware12cR4/oms/oraInst.loc  OPatchAuto.OMS_DISABLE_HOST_CHECK=true)

Please check above conditions and if error(s) still persist, Please contact Oracle support.

Log file location: /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatchauto/21603255/opatch_oms_2015-09-11_10-57-22AM_analyze.log

Recommended actions: Please correct the interview inputs and run opatchauto again.

OPatchauto failed with error code 231

Confirmation of the issue

To confirm this issue in your environment after receiving the preceding error message, check the hashing algorithm used on your adminserver certificate. I prefer to use the openssl commandline tool for this. If you don’t know the port used for your adminserver, you can retrieve it from the $EM_INSTANCE_BASE/em/EMGC_OMS1/emgc.properties file under AS_HTTPS_PORT. If your certificate does not show the usage of SHA256 (or another hash algorithm from the SHA-2 family) as in my example below, you may have a different problem.

oracle@omshost:~> openssl s_client -prexit -connect omshost.domain.com:7103 /dev/null | openssl x509 -text -in /dev/stdin | grep "Signature Algorithm" 2> /dev/null
        Signature Algorithm: sha256WithRSAEncryption
    Signature Algorithm: sha256WithRSAEncryption

Workaround

To work around this issue, you need to (temporarily!) replace the certificate on your WLS adminserver. Now, whenever I need to apply a PSU release, I resecure WLS using the default demonstration certificate, apply the PSU, then replace my original certificate.

oracle@omshost:/oracle/stage/21603255> emctl secure wls -use_demo_cert
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Securing WLS... Started.
Enter Enterprise Manager Root (SYSMAN) Password :
Securing WLS... Successful
Restart OMS using 'emctl stop oms -all' and 'emctl start oms'
oracle@omshost:/oracle/stage/21603255> emctl stop oms -all ; sleep 5 ; emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
Stopping BI Publisher Server...
BI Publisher Server Successfully Stopped
AdminServer Successfully Stopped
BI Publisher Server is Down
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
Starting BI Publisher Server ...
BI Publisher Server Successfully Started
BI Publisher Server is Up

[install the PSU according to the README instructions, including any post-installation steps]

oracle@omshost:/oracle/stage/21603255> emctl secure wls -wallet /oracle/oem/oemwallet
Oracle Enterprise Manager Cloud Control 12c Release 4  
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Securing WLS... Started.
Enter Enterprise Manager Root (SYSMAN) Password : 
Securing WLS... Successful
Restart OMS using 'emctl stop oms -all' and 'emctl start oms'
If there are multiple OMSs in this environment, perform this configuration on all of them.
oracle@omshost:/oracle/stage/21603255> emctl stop oms -all ; sleep 5 ; emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
Stopping BI Publisher Server...
BI Publisher Server Successfully Stopped
AdminServer Successfully Stopped
BI Publisher Server is Down
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
Starting BI Publisher Server ...
BI Publisher Server Successfully Started
BI Publisher Server is Up

I have not noticed any other EM12c issues using SHA256-hashed certificates. With this workaround, you can both continue to use quality certificates and keep your OMS patched.