EM13cR2 AWR Warehouse “Error communicating with agent” during transfer step with custom certificates

I have just noticed and resolved an issue in my EM13c R2 AWR Warehouse environment that I brought upon myself, hence a blog post for any others who might run into this, which also seems like a good time to release the scripts I use to generate and populate Oracle wallets for my EM13c agents.

After moving an AWRW source database from one EM13c managed server to a different EM13c managed server (same OS, same DB release), AWRW loads from that server began to fail. While debugging the issue, I first had to resolve an already-documented issue (see MOS note 2075341.1) where the source database had a NULL definition for the CAW_EXTR directory object, then fix up the data in the DBSNMP.CAW_EXTRACT_PROPERTIES table to reflect the CAW_EXTR directory. After resolving that, AWRW extracts ran successfully from the source database, but began to hang indefinitely during the CAW_RUN_ETL_NOW job in the transferAWR/transferFile job step, displaying only a cryptic error message:

A helpful error message

I ran through many debugging steps: changing preferred credentials, bouncing the agents, checking for firewalls blocking connectivity, none seemed to help. Eventually I realized the step I had missed in setting up the new managed server where the source database now runs: I had not generated an Oracle wallet for the agent on the new server, while I did have an Oracle wallet for the agent on the previous, now-retired server. This created an issue because I have secured the agent on my OMS host (where my AWRW repository database runs) with a custom third party certificate, and the new agent, lacking a wallet containing a trusted root certificate to which it could trace the repository agent’s certificate, could not initiate a connection from the AWRW source DB host agent to the AWRW repository DB host agent.

I generated a wallet for the new agent, added the trusted root certificate and a certificate for the host to the wallet, stopped the agent, deployed the wallet, and started the agent. After those steps, running the AWRW load from this source database completed successfully. I believe that the missing trusted root certificate prevented the creation of a secure channel between the two agents. I probably did not need to add the host certificate to resolve this problem, but consider it a good practice anyway.

If you read this far, you may find my create_agent_wallets.sh script useful to generate wallets and certificate signing requests for every agent in your environment. If you find the wallet creation script useful, you may also find my import_agent_wallets.sh script useful to populate those wallets with signed certificates received from your CA.

Advertisement

Securing Oracle Enterprise Manager 13cR2

IMPORTANT UPDATE 20190404: If you use, or have considered using, the EMCLI integration in this script, please take note of the comment posted by Christian Lehnert recently. Christian checked with Oracle ACS who reported that the repository views queried by the EMCLI integration in this script are licensed views and require the Lifecycle Management Pack. If you run the script without using the EMCLI integration, this code path is not reached, so you do not have any licensing implications. If however you do use the EMCLI integration by logging in to EMCLI before running the script, please take this information under advisement. I intend to modify the script going forward to avoid using these repository views, but that will have the side effect of dramatically slowing down the script in EMCLI mode as agent patch checks will have to rely on EM jobs instead of direct repository queries.

Oracle released Oracle Enterprise Manager 13cR2 at the beginning of October 2016. I have upgraded my production system to this new version, and here I provide a 13cR2-compatible version of my EM13c security checkup script. In addition to updating the script for EM13cR2, I have also updated it to take account of Oracle’s recommendation that single-instance non-RAC databases such as OEM repositories should now apply the DBBP Bundle Patch (previously known as the engineered systems bundle patch).

Latest Updates

Acknowledgements for previous release, November 28, 2017, version 2.21: This release includes many improvements provided by Jan Schnackenberg: combining the demo and self-signed certificate checks, adding a more robust multi-dot version string check, and many bugfixes that prevented the script from running correctly on AIX. This release includes the 20171031 bundle patches and latest OPatch, but please note the warning at the end of the script about open bugs with the latest OPatch release.  You may wish NOT to install OPatch 13.9.2.1.0 or the DB plugin bundle patch that requires it. Further, due to some changes in the EMCLI implementation to use “emcli list” instead of “emcli execute_sql”, if you use the optional EMCLI integration your EMCLI user will now require the ACCESS_EMCLI_SQL_LIST_VERB privilege. I have updated the create_user_for_checksec13R2.sh script to include this privilege for newly created CHECKSEC user accounts.
Latest release, Oct 18, 2018, version 2.40: This release covers the Oct 16, 2018 critical patch updates.

Download the latest release from https://raw.githubusercontent.com/brianpardy/em13c/master/checksec13R2.sh

EMCLI

If you have used this script for a while, you can download the latest release and just run it. It will continue to work the way it always has. If you would like to enable additional, optional functionality, enable the checksec13R2.sh EMCLI integration by logging in to EMCLI with an OEM administrator account before running checksec13R2.sh. The script will use EMCLI and attempt to check for plugin bundle patches on ALL of your OEM agents, not only the chained agent as it used to. It will also use EMCLI to attempt to validate the Java versions on all of your agents. This functionality requires that the EMCLI user account has access to run the execute_sql and execute_hostcmd, and also requires that the EMCLI user account has preferred credentials set for the repository database (normal and sysdba), repository database host, and for every host with a management agent.

To simplify the process, I have created a script to create a CHECKSEC user account in your OEM environment. The script will prompt you for the named credentials that the new account should use to access your repository database and each host. If you run this script after logging in to EMCLI as SYSMAN, it will create the new OEM user, grant acccess to all specified credentials, and grant EM_ALL_OPERATOR and VIEW_ANY_TARGET privileges so that the new account will have all the access needed to run all the optional checksec13R2.sh checks. I have included sample output from the user creation script at the end of this post. You can download the user creation script at create_user_for_checksec13R2.sh.

Download

You can access my EM13c script repository at https://github.com/brianpardy/em13c. To directly access the EM13cR2 security checkup script, use https://raw.githubusercontent.com/brianpardy/em13c/master/checksec13R2.sh.

Example Output – checksec13R2.sh


Performing EM13c R2 security checkup version 2.7 on omshost.domain.com at Mon May 1 15:38:41 EDT 2017.

Gathering info…
EM13c config… OK
Repos DB… 12.1.0.2.0 OK
OPatch-OMS… OK
OPatch-Agent… OK
OPatch-Repos DB… OK
OMSPatcher-OMS… OK
EMCLI login… OK
EMCLI-Agent list… OK
EMCLI-Agent patches… OK
EMCLI-Agent homes… OK

Using port definitions from configuration files
/etc/oragchomelist
/oracle/oem/gc_inst1/em/EMGC_OMS1/emgc.properties
/oracle/oem/gc_inst1/em/EMGC_OMS1/embip.properties
/oracle/oem/agent13cR1/agent_13.2.0.0.0/../agent_inst/sysman/emd/targets.xml

Agent port found at omshost.domain.com:3872
BIPublisher port found at omshost.domain.com:9803
BIPublisherOHS port found at omshost.domain.com:9852
NodeManager port found at omshost.domain.com:7403
OMSconsole port found at omshost.domain.com:7802
OMSproxy port found at omshost.domain.com:7301
OMSupload port found at omshost.domain.com:4903
WLSadmin found at omshost.domain.com:7102

Repository DB version=12.1.0.2.0 SID=oemdb host=omshost.domain.com
Repository DB target name=oemdb.domain.com

Using OPENSSL=/usr/bin/openssl1 (has TLS1_2=2)
Repository DB on OMS server, will check patches/parameters in /oracle/oem/product/12.1.0/db

(1) Checking SSL/TLS configuration (see notes 2138391.1, 2212006.1)

(1a) Forbid SSLv2 connections
Confirming ssl2 disabled for Agent at omshost.domain.com:3872… OK
Confirming ssl2 disabled for BIPublisher at omshost.domain.com:9803… OK
Confirming ssl2 disabled for NodeManager at omshost.domain.com:7403… OK
Confirming ssl2 disabled for BIPublisherOHS at omshost.domain.com:9852… OK
Confirming ssl2 disabled for OMSconsole at omshost.domain.com:7802… OK
Confirming ssl2 disabled for OMSproxy at omshost.domain.com:7301… OK
Confirming ssl2 disabled for OMSupload at omshost.domain.com:4903… OK
Confirming ssl2 disabled for WLSadmin at omshost.domain.com:7102… OK

Checking SSLv2 on all agents

Confirming ssl2 disabled for Agent at host01.domain.com:3872… OK
Confirming ssl2 disabled for Agent at host02.domain.com:3872… OK
Confirming ssl2 disabled for Agent at host04.usa.domain.com:3872… OK
Confirming ssl2 disabled for Agent at host03.domain.com:3872… OK
Confirming ssl2 disabled for Agent at host05.domain.com:3872… OK
Confirming ssl2 disabled for Agent at host06.domain.com:1830… OK
Confirming ssl2 disabled for Agent at host07.domain.com:3872… OK
Confirming ssl2 disabled for Agent at host08.domain.com:3872… OK
Confirming ssl2 disabled for Agent at host09.domain.com:1830… OK
Confirming ssl2 disabled for Agent at host10.domain.com:3872… OK
Confirming ssl2 disabled for Agent at host11.domain.com:3872… OK
Confirming ssl2 disabled for Agent at host12.domain.com:3872… OK
Confirming ssl2 disabled for Agent at host13.domain.com:3872… OK
Confirming ssl2 disabled for Agent at host14.domain.com:3872… OK
Confirming ssl2 disabled for Agent at host15.domain.com:3872… OK
Confirming ssl2 disabled for Agent at host16.domain.com:3872… OK
Confirming ssl2 disabled for Agent at omshost.domain.com:3872… OK
Confirming ssl2 disabled for Agent at host17.domain.com:3872… OK
Confirming ssl2 disabled for Agent at host18.domain.com:3872… OK

(1b) Forbid SSLv3 connections
Confirming ssl3 disabled for Agent at omshost.domain.com:3872… OK
Confirming ssl3 disabled for BIPublisher at omshost.domain.com:9803… OK
Confirming ssl3 disabled for NodeManager at omshost.domain.com:7403… OK
Confirming ssl3 disabled for BIPublisherOHS at omshost.domain.com:9852… OK
Confirming ssl3 disabled for OMSconsole at omshost.domain.com:7802… OK
Confirming ssl3 disabled for OMSproxy at omshost.domain.com:7301… OK
Confirming ssl3 disabled for OMSupload at omshost.domain.com:4903… OK
Confirming ssl3 disabled for WLSadmin at omshost.domain.com:7102… OK

Checking SSLv3 on all agents

Confirming ssl3 disabled for Agent at host01.domain.com:3872… OK
Confirming ssl3 disabled for Agent at host02.domain.com:3872… OK
Confirming ssl3 disabled for Agent at host04.usa.domain.com:3872… OK
Confirming ssl3 disabled for Agent at host03.domain.com:3872… OK
Confirming ssl3 disabled for Agent at host05.domain.com:3872… OK
Confirming ssl3 disabled for Agent at host06.domain.com:1830… OK
Confirming ssl3 disabled for Agent at host07.domain.com:3872… OK
Confirming ssl3 disabled for Agent at host08.domain.com:3872… OK
Confirming ssl3 disabled for Agent at host09.domain.com:1830… OK
Confirming ssl3 disabled for Agent at host10.domain.com:3872… OK
Confirming ssl3 disabled for Agent at host11.domain.com:3872… OK
Confirming ssl3 disabled for Agent at host12.domain.com:3872… OK
Confirming ssl3 disabled for Agent at host13.domain.com:3872… OK
Confirming ssl3 disabled for Agent at host14.domain.com:3872… OK
Confirming ssl3 disabled for Agent at host15.domain.com:3872… OK
Confirming ssl3 disabled for Agent at host16.domain.com:3872… OK
Confirming ssl3 disabled for Agent at omshost.domain.com:3872… OK
Confirming ssl3 disabled for Agent at host17.domain.com:3872… OK
Confirming ssl3 disabled for Agent at host18.domain.com:3872… OK

(1c) Forbid TLSv1 connections
Confirming tls1 disabled for Agent at omshost.domain.com:3872… OK
Confirming tls1 disabled for BIPublisher at omshost.domain.com:9803… OK
Confirming tls1 disabled for NodeManager at omshost.domain.com:7403… OK
Confirming tls1 disabled for BIPublisherOHS at omshost.domain.com:9852… OK
Confirming tls1 disabled for OMSconsole at omshost.domain.com:7802… OK
Confirming tls1 disabled for OMSproxy at omshost.domain.com:7301… OK
Confirming tls1 disabled for OMSupload at omshost.domain.com:4903… OK
Confirming tls1 disabled for WLSadmin at omshost.domain.com:7102… OK

Checking TLSv1 on all agents

Confirming tls1 disabled for Agent at host01.domain.com:3872… OK
Confirming tls1 disabled for Agent at host02.domain.com:3872… OK
Confirming tls1 disabled for Agent at host04.usa.domain.com:3872… OK
Confirming tls1 disabled for Agent at host03.domain.com:3872… OK
Confirming tls1 disabled for Agent at host05.domain.com:3872… OK
Confirming tls1 disabled for Agent at host06.domain.com:1830… OK
Confirming tls1 disabled for Agent at host07.domain.com:3872… OK
Confirming tls1 disabled for Agent at host08.domain.com:3872… OK
Confirming tls1 disabled for Agent at host09.domain.com:1830… OK
Confirming tls1 disabled for Agent at host10.domain.com:3872… OK
Confirming tls1 disabled for Agent at host11.domain.com:3872… OK
Confirming tls1 disabled for Agent at host12.domain.com:3872… OK
Confirming tls1 disabled for Agent at host13.domain.com:3872… OK
Confirming tls1 disabled for Agent at host14.domain.com:3872… OK
Confirming tls1 disabled for Agent at host15.domain.com:3872… OK
Confirming tls1 disabled for Agent at host16.domain.com:3872… OK
Confirming tls1 disabled for Agent at omshost.domain.com:3872… OK
Confirming tls1 disabled for Agent at host17.domain.com:3872… OK
Confirming tls1 disabled for Agent at host18.domain.com:3872… OK

(1d) Forbid TLSv1.1 connections
Confirming tls1_1 disabled for Agent at omshost.domain.com:3872… OK
Confirming tls1_1 disabled for BIPublisher at omshost.domain.com:9803… OK
Confirming tls1_1 disabled for NodeManager at omshost.domain.com:7403… OK
Confirming tls1_1 disabled for BIPublisherOHS at omshost.domain.com:9852… OK
Confirming tls1_1 disabled for OMSconsole at omshost.domain.com:7802… OK
Confirming tls1_1 disabled for OMSproxy at omshost.domain.com:7301… OK
Confirming tls1_1 disabled for OMSupload at omshost.domain.com:4903… OK
Confirming tls1_1 disabled for WLSadmin at omshost.domain.com:7102… OK

Checking TLSv1.1 on all agents

Confirming tls1_1 disabled for Agent at host01.domain.com:3872… OK
Confirming tls1_1 disabled for Agent at host02.domain.com:3872… OK
Confirming tls1_1 disabled for Agent at host04.usa.domain.com:3872… OK
Confirming tls1_1 disabled for Agent at host03.domain.com:3872… OK
Confirming tls1_1 disabled for Agent at host05.domain.com:3872… OK
Confirming tls1_1 disabled for Agent at host06.domain.com:1830… OK
Confirming tls1_1 disabled for Agent at host07.domain.com:3872… OK
Confirming tls1_1 disabled for Agent at host08.domain.com:3872… OK
Confirming tls1_1 disabled for Agent at host09.domain.com:1830… OK
Confirming tls1_1 disabled for Agent at host10.domain.com:3872… OK
Confirming tls1_1 disabled for Agent at host11.domain.com:3872… OK
Confirming tls1_1 disabled for Agent at host12.domain.com:3872… OK
Confirming tls1_1 disabled for Agent at host13.domain.com:3872… OK
Confirming tls1_1 disabled for Agent at host14.domain.com:3872… OK
Confirming tls1_1 disabled for Agent at host15.domain.com:3872… OK
Confirming tls1_1 disabled for Agent at host16.domain.com:3872… OK
Confirming tls1_1 disabled for Agent at omshost.domain.com:3872… OK
Confirming tls1_1 disabled for Agent at host17.domain.com:3872… OK
Confirming tls1_1 disabled for Agent at host18.domain.com:3872… OK

(1e) Permit TLSv1.2 connections
Confirming tls1_2 available for Agent at omshost.domain.com:3872… OK
Confirming tls1_2 available for BIPublisher at omshost.domain.com:9803… OK
Confirming tls1_2 available for NodeManager at omshost.domain.com:7403… OK
Confirming tls1_2 available for BIPublisherOHS at omshost.domain.com:9852… OK
Confirming tls1_2 available for OMSconsole at omshost.domain.com:7802… OK
Confirming tls1_2 available for OMSproxy at omshost.domain.com:7301… OK
Confirming tls1_2 available for OMSupload at omshost.domain.com:4903… OK
Confirming tls1_2 available for WLSadmin at omshost.domain.com:7102… OK

Checking TLSv1.2 on all agents

Confirming tls1_2 available for Agent at host01.domain.com:3872… OK
Confirming tls1_2 available for Agent at host02.domain.com:3872… OK
Confirming tls1_2 available for Agent at host04.usa.domain.com:3872… OK
Confirming tls1_2 available for Agent at host03.domain.com:3872… OK
Confirming tls1_2 available for Agent at host05.domain.com:3872… OK
Confirming tls1_2 available for Agent at host06.domain.com:1830… OK
Confirming tls1_2 available for Agent at host07.domain.com:3872… OK
Confirming tls1_2 available for Agent at host08.domain.com:3872… OK
Confirming tls1_2 available for Agent at host09.domain.com:1830… OK
Confirming tls1_2 available for Agent at host10.domain.com:3872… OK
Confirming tls1_2 available for Agent at host11.domain.com:3872… OK
Confirming tls1_2 available for Agent at host12.domain.com:3872… OK
Confirming tls1_2 available for Agent at host13.domain.com:3872… OK
Confirming tls1_2 available for Agent at host14.domain.com:3872… OK
Confirming tls1_2 available for Agent at host15.domain.com:3872… OK
Confirming tls1_2 available for Agent at host16.domain.com:3872… OK
Confirming tls1_2 available for Agent at omshost.domain.com:3872… OK
Confirming tls1_2 available for Agent at host17.domain.com:3872… OK
Confirming tls1_2 available for Agent at host18.domain.com:3872… OK

(2) Checking supported ciphers at SSL/TLS endpoints (see notes 2138391.1, 1067411.1)
(2a) Checking LOW strength ciphers on Agent (omshost.domain.com:3872, protocol tls1_2)… OK
(2a) Checking MEDIUM strength ciphers on Agent (omshost.domain.com:3872)… OK
(2a) Checking HIGH strength ciphers on Agent (omshost.domain.com:3872)… OK

(2b) Checking LOW strength ciphers on BIPublisher (omshost.domain.com:9803, protocol tls1_2)… OK
(2b) Checking MEDIUM strength ciphers on BIPublisher (omshost.domain.com:9803)… OK
(2b) Checking HIGH strength ciphers on BIPublisher (omshost.domain.com:9803)… OK

(2c) Checking LOW strength ciphers on NodeManager (omshost.domain.com:7403, protocol tls1_2)… OK
(2c) Checking MEDIUM strength ciphers on NodeManager (omshost.domain.com:7403)… OK
(2c) Checking HIGH strength ciphers on NodeManager (omshost.domain.com:7403)… OK

(2d) Checking LOW strength ciphers on BIPublisherOHS (omshost.domain.com:9852, protocol tls1_2)… OK
(2d) Checking MEDIUM strength ciphers on BIPublisherOHS (omshost.domain.com:9852)… OK
(2d) Checking HIGH strength ciphers on BIPublisherOHS (omshost.domain.com:9852)… OK

(2e) Checking LOW strength ciphers on OMSconsole (omshost.domain.com:7802, protocol tls1_2)… OK
(2e) Checking MEDIUM strength ciphers on OMSconsole (omshost.domain.com:7802)… OK
(2e) Checking HIGH strength ciphers on OMSconsole (omshost.domain.com:7802)… OK

(2f) Checking LOW strength ciphers on OMSproxy (omshost.domain.com:7301, protocol tls1_2)… OK
(2f) Checking MEDIUM strength ciphers on OMSproxy (omshost.domain.com:7301)… OK
(2f) Checking HIGH strength ciphers on OMSproxy (omshost.domain.com:7301)… OK

(2g) Checking LOW strength ciphers on OMSupload (omshost.domain.com:4903, protocol tls1_2)… OK
(2g) Checking MEDIUM strength ciphers on OMSupload (omshost.domain.com:4903)… OK
(2g) Checking HIGH strength ciphers on OMSupload (omshost.domain.com:4903)… OK

(2h) Checking LOW strength ciphers on WLSadmin (omshost.domain.com:7102, protocol tls1_2)… OK
(2h) Checking MEDIUM strength ciphers on WLSadmin (omshost.domain.com:7102)… OK
(2h) Checking HIGH strength ciphers on WLSadmin (omshost.domain.com:7102)… OK

Checking supported ciphers on all agents

(2i) Checking LOW strength ciphers on Agent (host01.domain.com:3872, protocol tls1_2)… OK
(2i) Checking MEDIUM strength ciphers on Agent (host01.domain.com:3872)… OK
(2i) Checking HIGH strength ciphers on Agent (host01.domain.com:3872)… OK

(2i) Checking LOW strength ciphers on Agent (host02.domain.com:3872, protocol tls1_2)… OK
(2i) Checking MEDIUM strength ciphers on Agent (host02.domain.com:3872)… OK
(2i) Checking HIGH strength ciphers on Agent (host02.domain.com:3872)… OK

(2i) Checking LOW strength ciphers on Agent (host04.usa.domain.com:3872, protocol tls1_2)… OK
(2i) Checking MEDIUM strength ciphers on Agent (host04.usa.domain.com:3872)… OK
(2i) Checking HIGH strength ciphers on Agent (host04.usa.domain.com:3872)… OK

(2i) Checking LOW strength ciphers on Agent (host03.domain.com:3872, protocol tls1_2)… OK
(2i) Checking MEDIUM strength ciphers on Agent (host03.domain.com:3872)… OK
(2i) Checking HIGH strength ciphers on Agent (host03.domain.com:3872)… OK

(2i) Checking LOW strength ciphers on Agent (host05.domain.com:3872, protocol tls1_2)… OK
(2i) Checking MEDIUM strength ciphers on Agent (host05.domain.com:3872)… OK
(2i) Checking HIGH strength ciphers on Agent (host05.domain.com:3872)… OK

(2i) Checking LOW strength ciphers on Agent (host06.domain.com:1830, protocol tls1_2)… OK
(2i) Checking MEDIUM strength ciphers on Agent (host06.domain.com:1830)… OK
(2i) Checking HIGH strength ciphers on Agent (host06.domain.com:1830)… OK

(2i) Checking LOW strength ciphers on Agent (host07.domain.com:3872, protocol tls1_2)… OK
(2i) Checking MEDIUM strength ciphers on Agent (host07.domain.com:3872)… OK
(2i) Checking HIGH strength ciphers on Agent (host07.domain.com:3872)… OK

(2i) Checking LOW strength ciphers on Agent (host08.domain.com:3872, protocol tls1_2)… OK
(2i) Checking MEDIUM strength ciphers on Agent (host08.domain.com:3872)… OK
(2i) Checking HIGH strength ciphers on Agent (host08.domain.com:3872)… OK

(2i) Checking LOW strength ciphers on Agent (host09.domain.com:1830, protocol tls1_2)… OK
(2i) Checking MEDIUM strength ciphers on Agent (host09.domain.com:1830)… OK
(2i) Checking HIGH strength ciphers on Agent (host09.domain.com:1830)… OK

(2i) Checking LOW strength ciphers on Agent (host10.domain.com:3872, protocol tls1_2)… OK
(2i) Checking MEDIUM strength ciphers on Agent (host10.domain.com:3872)… OK
(2i) Checking HIGH strength ciphers on Agent (host10.domain.com:3872)… OK

(2i) Checking LOW strength ciphers on Agent (host11.domain.com:3872, protocol tls1_2)… OK
(2i) Checking MEDIUM strength ciphers on Agent (host11.domain.com:3872)… OK
(2i) Checking HIGH strength ciphers on Agent (host11.domain.com:3872)… OK

(2i) Checking LOW strength ciphers on Agent (host12.domain.com:3872, protocol tls1_2)… OK
(2i) Checking MEDIUM strength ciphers on Agent (host12.domain.com:3872)… OK
(2i) Checking HIGH strength ciphers on Agent (host12.domain.com:3872)… OK

(2i) Checking LOW strength ciphers on Agent (host13.domain.com:3872, protocol tls1_2)… OK
(2i) Checking MEDIUM strength ciphers on Agent (host13.domain.com:3872)… OK
(2i) Checking HIGH strength ciphers on Agent (host13.domain.com:3872)… OK

(2i) Checking LOW strength ciphers on Agent (host14.domain.com:3872, protocol tls1_2)… OK
(2i) Checking MEDIUM strength ciphers on Agent (host14.domain.com:3872)… OK
(2i) Checking HIGH strength ciphers on Agent (host14.domain.com:3872)… OK

(2i) Checking LOW strength ciphers on Agent (host15.domain.com:3872, protocol tls1_2)… OK
(2i) Checking MEDIUM strength ciphers on Agent (host15.domain.com:3872)… OK
(2i) Checking HIGH strength ciphers on Agent (host15.domain.com:3872)… OK

(2i) Checking LOW strength ciphers on Agent (host16.domain.com:3872, protocol tls1_2)… OK
(2i) Checking MEDIUM strength ciphers on Agent (host16.domain.com:3872)… OK
(2i) Checking HIGH strength ciphers on Agent (host16.domain.com:3872)… OK

(2i) Checking LOW strength ciphers on Agent (omshost.domain.com:3872, protocol tls1_2)… OK
(2i) Checking MEDIUM strength ciphers on Agent (omshost.domain.com:3872)… OK
(2i) Checking HIGH strength ciphers on Agent (omshost.domain.com:3872)… OK

(2i) Checking LOW strength ciphers on Agent (host17.domain.com:3872, protocol tls1_2)… OK
(2i) Checking MEDIUM strength ciphers on Agent (host17.domain.com:3872)… OK
(2i) Checking HIGH strength ciphers on Agent (host17.domain.com:3872)… OK

(2i) Checking LOW strength ciphers on Agent (host18.domain.com:3872, protocol tls1_2)… OK
(2i) Checking MEDIUM strength ciphers on Agent (host18.domain.com:3872)… OK
(2i) Checking HIGH strength ciphers on Agent (host18.domain.com:3872)… OK

(3) Checking self-signed and demonstration certificates at SSL/TLS endpoints (see notes 2202569.1, 1367988.1, 1914184.1, 2213661.1, 2220788.1, 123033.1, 1937457.1)

(3a) Checking for self-signed certificates on OMS components
Checking certificate at Agent (omshost.domain.com:3872, protocol tls1_2)… OK
Checking certificate at BIPublisherOHS (omshost.domain.com:9852, protocol tls1_2)… OK
Checking certificate at BIPublisher (omshost.domain.com:9803, protocol tls1_2)… OK
Checking certificate at NodeManager (omshost.domain.com:7403, protocol tls1_2)… OK
Checking certificate at OMSconsole (omshost.domain.com:7802, protocol tls1_2)… OK
Checking certificate at OMSproxy (omshost.domain.com:7301, protocol tls1_2)… OK
Checking certificate at OMSupload (omshost.domain.com:4903, protocol tls1_2)… OK
Checking certificate at WLSadmin (omshost.domain.com:7102, protocol tls1_2)… OK

(3b) Checking for demonstration certificates on OMS components
Checking demo certificate at Agent (omshost.domain.com:3872, protocol tls1_2)… OK
Checking demo certificate at BIPublisherOHS (omshost.domain.com:9852, protocol tls1_2)… OK
Checking demo certificate at BIPublisher (omshost.domain.com:9803, protocol tls1_2)… OK
Checking demo certificate at NodeManager (omshost.domain.com:7403, protocol tls1_2)… OK
Checking demo certificate at OMSconsole (omshost.domain.com:7802, protocol tls1_2)… OK
Checking demo certificate at OMSproxy (omshost.domain.com:7301, protocol tls1_2)… OK
Checking demo certificate at OMSupload (omshost.domain.com:4903, protocol tls1_2)… OK
Checking demo certificate at WLSadmin (omshost.domain.com:7102, protocol tls1_2)… OK

(3c) Checking for self-signed certificates on all agents

Checking certificate at Agent (host01.domain.com:3872, protocol tls1_2)… OK
Checking certificate at Agent (host02.domain.com:3872, protocol tls1_2)… OK
Checking certificate at Agent (host04.usa.domain.com:3872, protocol tls1_2)… OK
Checking certificate at Agent (host03.domain.com:3872, protocol tls1_2)… OK
Checking certificate at Agent (host05.domain.com:3872, protocol tls1_2)… OK
Checking certificate at Agent (host06.domain.com:1830, protocol tls1_2)… OK
Checking certificate at Agent (host07.domain.com:3872, protocol tls1_2)… OK
Checking certificate at Agent (host08.domain.com:3872, protocol tls1_2)… OK
Checking certificate at Agent (host09.domain.com:1830, protocol tls1_2)… OK
Checking certificate at Agent (host10.domain.com:3872, protocol tls1_2)… OK
Checking certificate at Agent (host11.domain.com:3872, protocol tls1_2)… OK
Checking certificate at Agent (host12.domain.com:3872, protocol tls1_2)… OK
Checking certificate at Agent (host13.domain.com:3872, protocol tls1_2)… OK
Checking certificate at Agent (host14.domain.com:3872, protocol tls1_2)… OK
Checking certificate at Agent (host15.domain.com:3872, protocol tls1_2)… OK
Checking certificate at Agent (host16.domain.com:3872, protocol tls1_2)… OK
Checking certificate at Agent (omshost.domain.com:3872, protocol tls1_2)… OK
Checking certificate at Agent (host17.domain.com:3872, protocol tls1_2)… OK
Checking certificate at Agent (host18.domain.com:3872, protocol tls1_2)… OK

(3d) Checking for demonstration certificates on all agents

Checking demo certificate at Agent (host01.domain.com:3872, protocol tls1_2)… OK
Checking demo certificate at Agent (host02.domain.com:3872, protocol tls1_2)… OK
Checking demo certificate at Agent (host04.usa.domain.com:3872, protocol tls1_2)… OK
Checking demo certificate at Agent (host03.domain.com:3872, protocol tls1_2)… OK
Checking demo certificate at Agent (host05.domain.com:3872, protocol tls1_2)… OK
Checking demo certificate at Agent (host06.domain.com:1830, protocol tls1_2)… OK
Checking demo certificate at Agent (host07.domain.com:3872, protocol tls1_2)… OK
Checking demo certificate at Agent (host08.domain.com:3872, protocol tls1_2)… OK
Checking demo certificate at Agent (host09.domain.com:1830, protocol tls1_2)… OK
Checking demo certificate at Agent (host10.domain.com:3872, protocol tls1_2)… OK
Checking demo certificate at Agent (host11.domain.com:3872, protocol tls1_2)… OK
Checking demo certificate at Agent (host12.domain.com:3872, protocol tls1_2)… OK
Checking demo certificate at Agent (host13.domain.com:3872, protocol tls1_2)… OK
Checking demo certificate at Agent (host14.domain.com:3872, protocol tls1_2)… OK
Checking demo certificate at Agent (host15.domain.com:3872, protocol tls1_2)… OK
Checking demo certificate at Agent (host16.domain.com:3872, protocol tls1_2)… OK
Checking demo certificate at Agent (omshost.domain.com:3872, protocol tls1_2)… OK
Checking demo certificate at Agent (host17.domain.com:3872, protocol tls1_2)… OK
Checking demo certificate at Agent (host18.domain.com:3872, protocol tls1_2)… OK

(4) Checking EM13c Oracle home patch levels against 30 Apr 2017 baseline (see notes 1664074.1, 2219797.1, 822485.1, 1470197.1)

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) DATABASE BUNDLE PATCH: 12.1.0.2.170418 (APR2017) (25397136)… OK

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) Database PSU 12.1.0.2.170418, Oracle JavaVM Component (APR2017) (25437695)… OK

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) OCW Interim patch for 25481150 (25481150)… OK

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) EM QUERY WITH SQL_ID 4RQ83FNXTF39U PERFORMS POORLY ON ORACLE 12C RELATIVE TO 11G (20243268)… OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_TYPES_SERVER parameter (76629.1, 2167682.1)… OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_SERVER parameter (76629.1, 2167682.1)… OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_TYPES_CLIENT parameter (76629.1, 2167682.1)… OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_CLIENT parameter (76629.1, 2167682.1)… OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_TYPES_SERVER parameter (76629.1, 2167682.1)… OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_SERVER parameter (76629.1, 2167682.1)… OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_TYPES_CLIENT parameter (76629.1, 2167682.1)… OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_CLIENT parameter (76629.1, 2167682.1)… OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SSL_VERSION parameter (1545816.1)… OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SSL_CIPHER_SUITES parameter (1545816.1)… OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) listener.ora SSL_VERSION parameter (1545816.1)… OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) listener.ora SSL_CIPHER_SUITES parameter (1545816.1)… OK

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) APEX version… OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) ENTERPRISE MANAGER BASE PLATFORM – OMS 13.2.0.0.170418 PSU (25387277)… OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) TRACKING BUG TO REGISTER META VERSION FROM PS4 AND 13.1 BUNDLE PATCHES IN 13.2 (SYSTEM PATCH) (23603592)… OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) MERGE REQUEST ON TOP OF 12.1.3.0.0 FOR BUGS 24571979 24335626 (25322055)… OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) MERGE REQUEST ON TOP OF 12.1.3.0.0 FOR BUGS 22557350 19901079 20222451 (24329181)… OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) MERGE REQUEST ON TOP OF 12.1.3.0.0 FOR BUGS 19485414 20022048 (21849941)… OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) OPSS BUNDLE PATCH 12.1.3.0.170418 (22748215)… OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) ENTERPRISE MANAGER FOR OMS PLUGINS 13.2.0.0.170430 (Not used for 13.2.2 plugins) (25841652)… OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) WLS PATCH SET UPDATE 12.1.3.0.170418 (25388793)… OK

(4c) OMS HOME (/oracle/oem/Middleware13cR2) TOPLINK SECURITY PATCH UPDATE CPUJUL2016 (24327938)… OK

Using EMCLI to check for agent bundle patch on all agents

(4d) Agent host01.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170430 (25740081)… FAILED

(4d) Agent host02.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170430 (25740081)… OK

(4d) Agent host04.usa.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170430 (25740081)… OK

(4d) Agent host03.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170430 (25740081)… OK

(4d) Agent host05.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170430 (25740081)… OK

(4d) Agent host06.domain.com:1830 EM-AGENT BUNDLE PATCH 13.2.0.0.170430 (25740081)… OK

(4d) Agent host07.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170430 (25740081)… OK

(4d) Agent host08.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170430 (25740081)… OK

(4d) Agent host09.domain.com:1830 EM-AGENT BUNDLE PATCH 13.2.0.0.170430 (25740081)… OK

(4d) Agent host10.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170430 (25740081)… OK

(4d) Agent host11.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170430 (25740081)… OK

(4d) Agent host12.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170430 (25740081)… OK

(4d) Agent host13.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170430 (25740081)… OK

(4d) Agent host14.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170430 (25740081)… OK

(4d) Agent host15.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170430 (25740081)… FAILED

(4d) Agent host16.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170430 (25740081)… OK

(4d) Agent omshost.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170430 (25740081)… OK

(4d) Agent host17.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170430 (25740081)… OK

(4d) Agent host18.domain.com:3872 EM-AGENT BUNDLE PATCH 13.2.0.0.170430 (25740081)… OK

(5) Checking EM13cR2 Java patch levels against 30 Apr 2017 baseline (see notes 1506916.1, 2241373.1, 2241358.1)

(5a) Common Java (/oracle/oem/Middleware13cR2/oracle_common/jdk) JAVA SE JDK VERSION 1.7.0_141 (13079846)… OK

Using EMCLI to check Java patch levels on all agents

(5b) Agent host01.domain.com:3872 Java VERSION 1.7.0_141… OK

(5b) Agent host02.domain.com:3872 Java VERSION 1.7.0_141… OK

(5b) Agent host04.usa.domain.com:3872 Java VERSION 1.7.0_141… OK

(5b) Agent host03.domain.com:3872 Java VERSION 1.7.0_141… OK

(5b) Agent host05.domain.com:3872 Java VERSION 1.7.0_141… OK

(5b) Agent host06.domain.com:1830 Java VERSION 1.7.0_141… OK

(5b) Agent host07.domain.com:3872 Java VERSION 1.7.0_141… OK

(5b) Agent host08.domain.com:3872 Java VERSION 1.7.0_141… OK

(5b) Agent host09.domain.com:1830 Java VERSION 1.7.0_141… OK

(5b) Agent host10.domain.com:3872 Java VERSION 1.7.0_141… OK

(5b) Agent host11.domain.com:3872 Java VERSION 1.7.0_141… OK

(5b) Agent host12.domain.com:3872 Java VERSION 1.7.0_141… OK

(5b) Agent host13.domain.com:3872 Java VERSION 1.7.0_141… OK

(5b) Agent host14.domain.com:3872 Java VERSION 1.7.0_141… OK

(5b) Agent host15.domain.com:3872 Java VERSION 1.7.0_141… OK

(5b) Agent host16.domain.com:3872 Java VERSION 1.7.0_141… OK

(5b) Agent omshost.domain.com:3872 Java VERSION 1.7.0_141… OK

(5b) Agent host17.domain.com:3872 Java VERSION 1.7.0_141… OK

(5b) Agent host18.domain.com:3872 Java VERSION 1.7.0_141… OK

(6) Checking EM13cR2 OPatch/OMSPatcher patch levels against 30 Apr 2017 requirements (see patch 25197714 README, patches 6880880 and 19999993)

(6a) OMS OPatch (/oracle/oem/Middleware13cR2/OPatch) VERSION 13.9.1.3.0 or newer… OK

(6b) OMSPatcher (/oracle/oem/Middleware13cR2/OPatch) VERSION 13.8.0.0.2 or newer… OK

Checking OPatch patch levels on all agents

(6c) Agent host01.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0… OK

(6c) Agent host02.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0… OK

(6c) Agent host04.usa.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0… OK

(6c) Agent host03.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0… OK

(6c) Agent host05.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0… OK

(6c) Agent host06.domain.com:1830 ORACLE_HOME OPatch VERSION 13.9.1.3.0… OK

(6c) Agent host07.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0… OK

(6c) Agent host08.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0… OK

(6c) Agent host09.domain.com:1830 ORACLE_HOME OPatch VERSION 13.9.1.3.0… OK

(6c) Agent host10.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0… OK

(6c) Agent host11.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0… OK

(6c) Agent host12.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0… OK

(6c) Agent host13.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0… OK

(6c) Agent host14.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0… OK

(6c) Agent host15.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0… OK

(6c) Agent host16.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0… OK

(6c) Agent omshost.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0… OK

(6c) Agent host17.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0… OK

(6c) Agent host18.domain.com:3872 ORACLE_HOME OPatch VERSION 13.9.1.3.0… OK

(7) Agent plugin bundle patch checks on all agents…
(7a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host01.domain.com:3872 (25839989)… OK – plugin not installed

(7b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host01.domain.com:3872 (25197692)… OK – plugin not installed

(7c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host01.domain.com:3872 (25839746)… OK – plugin not installed

(7d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host01.domain.com:3872 (25501430)… OK

(7e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170331 MONITORING @ host01.domain.com:3872 (25682670)… OK – plugin not installed

(7f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host01.domain.com:3872 (25162444)… OK – plugin not installed

(7g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host01.domain.com:3872 (25501436)… OK

(7h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host01.domain.com:3872 (25362875)… OK – plugin not installed

(7i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host01.domain.com:3872 (25522944)… OK – plugin not installed

(7j) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170430 DISCOVERY @ host01.domain.com:3872 (25839874)… OK – plugin not installed

(7k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host01.domain.com:3872 (25501416)… OK – plugin not installed

(7l) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host01.domain.com:3872 (25362898)… OK – plugin not installed

(7m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host01.domain.com:3872 (25362890)… OK – plugin not installed

(7n) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host01.domain.com:3872 (25197712)… OK – plugin not installed

(8a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host02.domain.com:3872 (25839989)… OK – plugin not installed

(8b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host02.domain.com:3872 (25197692)… OK – plugin not installed

(8c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host02.domain.com:3872 (25839746)… OK – plugin not installed

(8d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host02.domain.com:3872 (25501430)… OK

(8e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170331 MONITORING @ host02.domain.com:3872 (25682670)… OK – plugin not installed

(8f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host02.domain.com:3872 (25162444)… OK – plugin not installed

(8g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host02.domain.com:3872 (25501436)… OK

(8h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host02.domain.com:3872 (25362875)… OK – plugin not installed

(8i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host02.domain.com:3872 (25522944)… OK – plugin not installed

(8j) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170430 DISCOVERY @ host02.domain.com:3872 (25839874)… OK – plugin not installed

(8k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host02.domain.com:3872 (25501416)… OK – plugin not installed

(8l) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host02.domain.com:3872 (25362898)… OK – plugin not installed

(8m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host02.domain.com:3872 (25362890)… OK – plugin not installed

(8n) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host02.domain.com:3872 (25197712)… OK – plugin not installed

(9a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host04.usa.domain.com:3872 (25839989)… OK – plugin not installed

(9b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host04.usa.domain.com:3872 (25197692)… OK – plugin not installed

(9c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host04.usa.domain.com:3872 (25839746)… OK – plugin not installed

(9d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host04.usa.domain.com:3872 (25501430)… OK – plugin not installed

(9e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170331 MONITORING @ host04.usa.domain.com:3872 (25682670)… OK – plugin not installed

(9f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host04.usa.domain.com:3872 (25162444)… OK – plugin not installed

(9g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host04.usa.domain.com:3872 (25501436)… OK – plugin not installed

(9h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host04.usa.domain.com:3872 (25362875)… OK – plugin not installed

(9i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host04.usa.domain.com:3872 (25522944)… OK – plugin not installed

(9j) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170430 DISCOVERY @ host04.usa.domain.com:3872 (25839874)… OK – plugin not installed

(9k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host04.usa.domain.com:3872 (25501416)… OK – plugin not installed

(9l) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host04.usa.domain.com:3872 (25362898)… OK – plugin not installed

(9m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host04.usa.domain.com:3872 (25362890)… OK – plugin not installed

(9n) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host04.usa.domain.com:3872 (25197712)… OK – plugin not installed

(10a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host03.domain.com:3872 (25839989)… OK – plugin not installed

(10b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host03.domain.com:3872 (25197692)… OK – plugin not installed

(10c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host03.domain.com:3872 (25839746)… OK – plugin not installed

(10d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host03.domain.com:3872 (25501430)… OK

(10e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170331 MONITORING @ host03.domain.com:3872 (25682670)… OK – plugin not installed

(10f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host03.domain.com:3872 (25162444)… OK – plugin not installed

(10g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host03.domain.com:3872 (25501436)… OK

(10h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host03.domain.com:3872 (25362875)… OK – plugin not installed

(10i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host03.domain.com:3872 (25522944)… OK – plugin not installed

(10j) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170430 DISCOVERY @ host03.domain.com:3872 (25839874)… OK – plugin not installed

(10k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host03.domain.com:3872 (25501416)… OK – plugin not installed

(10l) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host03.domain.com:3872 (25362898)… OK – plugin not installed

(10m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host03.domain.com:3872 (25362890)… OK – plugin not installed

(10n) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host03.domain.com:3872 (25197712)… OK – plugin not installed

(11a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host05.domain.com:3872 (25839989)… OK – plugin not installed

(11b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host05.domain.com:3872 (25197692)… OK – plugin not installed

(11c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host05.domain.com:3872 (25839746)… OK – plugin not installed

(11d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host05.domain.com:3872 (25501430)… OK

(11e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170331 MONITORING @ host05.domain.com:3872 (25682670)… OK – plugin not installed

(11f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host05.domain.com:3872 (25162444)… OK – plugin not installed

(11g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host05.domain.com:3872 (25501436)… OK

(11h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host05.domain.com:3872 (25362875)… OK – plugin not installed

(11i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host05.domain.com:3872 (25522944)… OK – plugin not installed

(11j) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170430 DISCOVERY @ host05.domain.com:3872 (25839874)… OK – plugin not installed

(11k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host05.domain.com:3872 (25501416)… OK – plugin not installed

(11l) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host05.domain.com:3872 (25362898)… OK – plugin not installed

(11m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host05.domain.com:3872 (25362890)… OK – plugin not installed

(11n) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host05.domain.com:3872 (25197712)… OK – plugin not installed

(12a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host06.domain.com:1830 (25839989)… OK – plugin not installed

(12b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host06.domain.com:1830 (25197692)… OK – plugin not installed

(12c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host06.domain.com:1830 (25839746)… OK – plugin not installed

(12d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host06.domain.com:1830 (25501430)… OK

(12e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170331 MONITORING @ host06.domain.com:1830 (25682670)… OK – plugin not installed

(12f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host06.domain.com:1830 (25162444)… OK – plugin not installed

(12g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host06.domain.com:1830 (25501436)… OK – plugin not installed

(12h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host06.domain.com:1830 (25362875)… OK – plugin not installed

(12i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host06.domain.com:1830 (25522944)… OK – plugin not installed

(12j) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170430 DISCOVERY @ host06.domain.com:1830 (25839874)… OK – plugin not installed

(12k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host06.domain.com:1830 (25501416)… OK – plugin not installed

(12l) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host06.domain.com:1830 (25362898)… OK – plugin not installed

(12m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host06.domain.com:1830 (25362890)… OK – plugin not installed

(12n) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host06.domain.com:1830 (25197712)… OK – plugin not installed

(13a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host07.domain.com:3872 (25839989)… OK – plugin not installed

(13b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host07.domain.com:3872 (25197692)… OK – plugin not installed

(13c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host07.domain.com:3872 (25839746)… OK – plugin not installed

(13d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host07.domain.com:3872 (25501430)… OK

(13e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170331 MONITORING @ host07.domain.com:3872 (25682670)… OK – plugin not installed

(13f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host07.domain.com:3872 (25162444)… OK – plugin not installed

(13g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host07.domain.com:3872 (25501436)… OK

(13h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host07.domain.com:3872 (25362875)… OK – plugin not installed

(13i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host07.domain.com:3872 (25522944)… OK – plugin not installed

(13j) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170430 DISCOVERY @ host07.domain.com:3872 (25839874)… OK – plugin not installed

(13k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host07.domain.com:3872 (25501416)… OK – plugin not installed

(13l) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host07.domain.com:3872 (25362898)… OK – plugin not installed

(13m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host07.domain.com:3872 (25362890)… OK – plugin not installed

(13n) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host07.domain.com:3872 (25197712)… OK – plugin not installed

(14a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host08.domain.com:3872 (25839989)… OK – plugin not installed

(14b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host08.domain.com:3872 (25197692)… OK – plugin not installed

(14c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host08.domain.com:3872 (25839746)… OK – plugin not installed

(14d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host08.domain.com:3872 (25501430)… OK

(14e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170331 MONITORING @ host08.domain.com:3872 (25682670)… OK – plugin not installed

(14f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host08.domain.com:3872 (25162444)… OK – plugin not installed

(14g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host08.domain.com:3872 (25501436)… OK

(14h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host08.domain.com:3872 (25362875)… OK – plugin not installed

(14i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host08.domain.com:3872 (25522944)… OK – plugin not installed

(14j) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170430 DISCOVERY @ host08.domain.com:3872 (25839874)… OK – plugin not installed

(14k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host08.domain.com:3872 (25501416)… OK – plugin not installed

(14l) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host08.domain.com:3872 (25362898)… OK – plugin not installed

(14m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host08.domain.com:3872 (25362890)… OK – plugin not installed

(14n) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host08.domain.com:3872 (25197712)… OK – plugin not installed

(15a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host09.domain.com:1830 (25839989)… OK – plugin not installed

(15b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host09.domain.com:1830 (25197692)… OK – plugin not installed

(15c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host09.domain.com:1830 (25839746)… OK – plugin not installed

(15d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host09.domain.com:1830 (25501430)… OK

(15e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170331 MONITORING @ host09.domain.com:1830 (25682670)… OK – plugin not installed

(15f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host09.domain.com:1830 (25162444)… OK – plugin not installed

(15g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host09.domain.com:1830 (25501436)… OK

(15h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host09.domain.com:1830 (25362875)… OK – plugin not installed

(15i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host09.domain.com:1830 (25522944)… OK – plugin not installed

(15j) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170430 DISCOVERY @ host09.domain.com:1830 (25839874)… OK – plugin not installed

(15k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host09.domain.com:1830 (25501416)… OK – plugin not installed

(15l) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host09.domain.com:1830 (25362898)… OK – plugin not installed

(15m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host09.domain.com:1830 (25362890)… OK – plugin not installed

(15n) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host09.domain.com:1830 (25197712)… OK – plugin not installed

(16a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host10.domain.com:3872 (25839989)… OK – plugin not installed

(16b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host10.domain.com:3872 (25197692)… OK – plugin not installed

(16c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host10.domain.com:3872 (25839746)… OK – plugin not installed

(16d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host10.domain.com:3872 (25501430)… OK

(16e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170331 MONITORING @ host10.domain.com:3872 (25682670)… OK – plugin not installed

(16f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host10.domain.com:3872 (25162444)… OK – plugin not installed

(16g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host10.domain.com:3872 (25501436)… OK

(16h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host10.domain.com:3872 (25362875)… OK – plugin not installed

(16i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host10.domain.com:3872 (25522944)… OK – plugin not installed

(16j) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170430 DISCOVERY @ host10.domain.com:3872 (25839874)… OK – plugin not installed

(16k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host10.domain.com:3872 (25501416)… OK – plugin not installed

(16l) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host10.domain.com:3872 (25362898)… OK – plugin not installed

(16m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host10.domain.com:3872 (25362890)… OK – plugin not installed

(16n) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host10.domain.com:3872 (25197712)… OK – plugin not installed

(17a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host11.domain.com:3872 (25839989)… OK – plugin not installed

(17b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host11.domain.com:3872 (25197692)… OK – plugin not installed

(17c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host11.domain.com:3872 (25839746)… OK – plugin not installed

(17d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host11.domain.com:3872 (25501430)… OK

(17e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170331 MONITORING @ host11.domain.com:3872 (25682670)… OK – plugin not installed

(17f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host11.domain.com:3872 (25162444)… OK – plugin not installed

(17g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host11.domain.com:3872 (25501436)… OK

(17h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host11.domain.com:3872 (25362875)… OK – plugin not installed

(17i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host11.domain.com:3872 (25522944)… OK – plugin not installed

(17j) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170430 DISCOVERY @ host11.domain.com:3872 (25839874)… OK – plugin not installed

(17k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host11.domain.com:3872 (25501416)… OK – plugin not installed

(17l) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host11.domain.com:3872 (25362898)… OK – plugin not installed

(17m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host11.domain.com:3872 (25362890)… OK – plugin not installed

(17n) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host11.domain.com:3872 (25197712)… OK – plugin not installed

(18a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host12.domain.com:3872 (25839989)… OK – plugin not installed

(18b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host12.domain.com:3872 (25197692)… OK – plugin not installed

(18c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host12.domain.com:3872 (25839746)… OK – plugin not installed

(18d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host12.domain.com:3872 (25501430)… OK

(18e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170331 MONITORING @ host12.domain.com:3872 (25682670)… OK – plugin not installed

(18f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host12.domain.com:3872 (25162444)… OK – plugin not installed

(18g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host12.domain.com:3872 (25501436)… OK

(18h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host12.domain.com:3872 (25362875)… OK – plugin not installed

(18i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host12.domain.com:3872 (25522944)… OK – plugin not installed

(18j) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170430 DISCOVERY @ host12.domain.com:3872 (25839874)… OK – plugin not installed

(18k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host12.domain.com:3872 (25501416)… OK – plugin not installed

(18l) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host12.domain.com:3872 (25362898)… OK – plugin not installed

(18m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host12.domain.com:3872 (25362890)… OK – plugin not installed

(18n) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host12.domain.com:3872 (25197712)… OK – plugin not installed

(19a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host13.domain.com:3872 (25839989)… OK – plugin not installed

(19b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host13.domain.com:3872 (25197692)… OK – plugin not installed

(19c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host13.domain.com:3872 (25839746)… OK – plugin not installed

(19d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host13.domain.com:3872 (25501430)… OK

(19e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170331 MONITORING @ host13.domain.com:3872 (25682670)… OK – plugin not installed

(19f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host13.domain.com:3872 (25162444)… OK – plugin not installed

(19g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host13.domain.com:3872 (25501436)… OK

(19h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host13.domain.com:3872 (25362875)… OK – plugin not installed

(19i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host13.domain.com:3872 (25522944)… OK – plugin not installed

(19j) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170430 DISCOVERY @ host13.domain.com:3872 (25839874)… OK – plugin not installed

(19k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host13.domain.com:3872 (25501416)… OK – plugin not installed

(19l) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host13.domain.com:3872 (25362898)… OK – plugin not installed

(19m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host13.domain.com:3872 (25362890)… OK – plugin not installed

(19n) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host13.domain.com:3872 (25197712)… OK – plugin not installed

(20a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host14.domain.com:3872 (25839989)… OK – plugin not installed

(20b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host14.domain.com:3872 (25197692)… OK – plugin not installed

(20c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host14.domain.com:3872 (25839746)… OK – plugin not installed

(20d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host14.domain.com:3872 (25501430)… OK

(20e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170331 MONITORING @ host14.domain.com:3872 (25682670)… OK – plugin not installed

(20f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host14.domain.com:3872 (25162444)… OK – plugin not installed

(20g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host14.domain.com:3872 (25501436)… OK

(20h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host14.domain.com:3872 (25362875)… OK – plugin not installed

(20i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host14.domain.com:3872 (25522944)… OK – plugin not installed

(20j) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170430 DISCOVERY @ host14.domain.com:3872 (25839874)… OK – plugin not installed

(20k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host14.domain.com:3872 (25501416)… OK – plugin not installed

(20l) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host14.domain.com:3872 (25362898)… OK – plugin not installed

(20m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host14.domain.com:3872 (25362890)… OK – plugin not installed

(20n) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host14.domain.com:3872 (25197712)… OK – plugin not installed

(21a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host15.domain.com:3872 (25839989)… OK – plugin not installed

(21b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host15.domain.com:3872 (25197692)… OK – plugin not installed

(21c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host15.domain.com:3872 (25839746)… OK – plugin not installed

(21d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host15.domain.com:3872 (25501430)… OK

(21e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170331 MONITORING @ host15.domain.com:3872 (25682670)… OK – plugin not installed

(21f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host15.domain.com:3872 (25162444)… OK – plugin not installed

(21g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host15.domain.com:3872 (25501436)… OK

(21h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host15.domain.com:3872 (25362875)… OK – plugin not installed

(21i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host15.domain.com:3872 (25522944)… OK – plugin not installed

(21j) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170430 DISCOVERY @ host15.domain.com:3872 (25839874)… OK – plugin not installed

(21k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host15.domain.com:3872 (25501416)… OK – plugin not installed

(21l) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host15.domain.com:3872 (25362898)… OK – plugin not installed

(21m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host15.domain.com:3872 (25362890)… OK – plugin not installed

(21n) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host15.domain.com:3872 (25197712)… OK – plugin not installed

(22a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host16.domain.com:3872 (25839989)… OK – plugin not installed

(22b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host16.domain.com:3872 (25197692)… OK – plugin not installed

(22c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host16.domain.com:3872 (25839746)… OK – plugin not installed

(22d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host16.domain.com:3872 (25501430)… OK

(22e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170331 MONITORING @ host16.domain.com:3872 (25682670)… OK – plugin not installed

(22f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host16.domain.com:3872 (25162444)… OK – plugin not installed

(22g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host16.domain.com:3872 (25501436)… OK

(22h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host16.domain.com:3872 (25362875)… OK – plugin not installed

(22i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host16.domain.com:3872 (25522944)… OK – plugin not installed

(22j) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170430 DISCOVERY @ host16.domain.com:3872 (25839874)… OK – plugin not installed

(22k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host16.domain.com:3872 (25501416)… OK – plugin not installed

(22l) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host16.domain.com:3872 (25362898)… OK – plugin not installed

(22m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host16.domain.com:3872 (25362890)… OK – plugin not installed

(22n) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host16.domain.com:3872 (25197712)… OK – plugin not installed

(23a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ omshost.domain.com:3872 (25839989)… OK – plugin not installed

(23b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ omshost.domain.com:3872 (25197692)… OK – plugin not installed

(23c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ omshost.domain.com:3872 (25839746)… OK – plugin not installed

(23d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ omshost.domain.com:3872 (25501430)… OK – plugin not installed

(23e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170331 MONITORING @ omshost.domain.com:3872 (25682670)… OK – plugin not installed

(23f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ omshost.domain.com:3872 (25162444)… OK

(23g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ omshost.domain.com:3872 (25501436)… OK

(23h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ omshost.domain.com:3872 (25362875)… OK – plugin not installed

(23i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ omshost.domain.com:3872 (25522944)… OK – plugin not installed

(23j) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170430 DISCOVERY @ omshost.domain.com:3872 (25839874)… OK – plugin not installed

(23k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ omshost.domain.com:3872 (25501416)… OK – plugin not installed

(23l) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ omshost.domain.com:3872 (25362898)… OK – plugin not installed

(23m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ omshost.domain.com:3872 (25362890)… OK – plugin not installed

(23n) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ omshost.domain.com:3872 (25197712)… OK – plugin not installed

(24a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host17.domain.com:3872 (25839989)… OK – plugin not installed

(24b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host17.domain.com:3872 (25197692)… OK – plugin not installed

(24c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host17.domain.com:3872 (25839746)… OK – plugin not installed

(24d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host17.domain.com:3872 (25501430)… OK

(24e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170331 MONITORING @ host17.domain.com:3872 (25682670)… OK – plugin not installed

(24f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host17.domain.com:3872 (25162444)… OK – plugin not installed

(24g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host17.domain.com:3872 (25501436)… OK

(24h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host17.domain.com:3872 (25362875)… OK – plugin not installed

(24i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host17.domain.com:3872 (25522944)… OK – plugin not installed

(24j) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170430 DISCOVERY @ host17.domain.com:3872 (25839874)… OK – plugin not installed

(24k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host17.domain.com:3872 (25501416)… OK – plugin not installed

(24l) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host17.domain.com:3872 (25362898)… OK – plugin not installed

(24m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host17.domain.com:3872 (25362890)… OK – plugin not installed

(24n) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host17.domain.com:3872 (25197712)… OK – plugin not installed

(25a) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host18.domain.com:3872 (25839989)… OK – plugin not installed

(25b) EM DB PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host18.domain.com:3872 (25197692)… OK – plugin not installed

(25c) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170430 MONITORING @ host18.domain.com:3872 (25839746)… OK – plugin not installed

(25d) EM FMW PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host18.domain.com:3872 (25501430)… OK

(25e) EM SI PLUGIN BUNDLE PATCH 13.2.1.0.170331 MONITORING @ host18.domain.com:3872 (25682670)… OK – plugin not installed

(25f) EM-BEACON BUNDLE PATCH 13.2.0.0.161231 @ host18.domain.com:3872 (25162444)… OK – plugin not installed

(25g) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 DISCOVERY @ host18.domain.com:3872 (25501436)… OK

(25h) EM EXADATA PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host18.domain.com:3872 (25362875)… OK – plugin not installed

(25i) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host18.domain.com:3872 (25522944)… OK – plugin not installed

(25j) EM FUSION APPS PLUGIN BUNDLE PATCH 13.2.1.0.170430 DISCOVERY @ host18.domain.com:3872 (25839874)… OK – plugin not installed

(25k) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170228 MONITORING @ host18.domain.com:3872 (25501416)… OK – plugin not installed

(25l) EM OVI PLUGIN BUNDLE PATCH 13.2.1.0.170131 DISCOVERY @ host18.domain.com:3872 (25362898)… OK – plugin not installed

(25m) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.170131 MONITORING @ host18.domain.com:3872 (25362890)… OK – plugin not installed

(25n) EM VIRTUALIZATION PLUGIN BUNDLE PATCH 13.2.1.0.161231 DISCOVERY @ host18.domain.com:3872 (25197712)… OK – plugin not installed

Cleaning up temporary files… done
Failed test count: 2 – Review output

emcliagentbundlecheck:25740081 missing on host01.domain.com:3872
emcliagentbundlecheck:25740081 missing on host15.domain.com:3872

Visit https://pardydba.wordpress.com/2016/10/28/securing-oracle-enterprise-manager-13cr2/ for more information.
Download the latest release from https://raw.githubusercontent.com/brianpardy/em13c/master/checksec13R2.sh
Download the latest beta release from https://raw.githubusercontent.com/brianpardy/em13c/beta/checksec13R2.sh

 

Example Output – create_user_for_checksec13R2.sh


Welcome to ./create_user_for_checksec13R2.sh, version 1.0, released 20170314.

Download the latest release of this script at any time from:
https://raw.githubusercontent.com/brianpardy/em13c/master/create_user_for_checksec13R2.sh

This script exists to supplement checksec13R2.sh and enable additional checks. When run, this
script will create a user named CHECKSEC in your EM13cR2 environment and give that user a
random password, which gets printed to the screen at the end of the script. The script then
grants CHECKSEC VIEW_ANY_TARGET and EM_ALL_OPERATOR privilege, and then prompts you to supply
the names of credentials existing in your EM13cR2 environment. The script validates the names of
credentials supplied, grants VIEW access to CHECKSEC for each credential, and assigns those
credentials as preferred credentials for CHECKSEC for each relevant target.

Providing credentials for the repository database enables the following additional checks in
checksec13R2.sh:
* Check for presence/absence of plugin bundle patches on all agents

Providing host credentials for every monitored host running an agent enables the following
additional checks in checksec13R2.sh:
* Check for presence/absence of the latest Java version on all agents

Login to EMCLI as SYSMAN before running this script. If you already have an CHECKSEC account,
running this script will delete and recreate it with a new password.

Continue? [y/n]
Continuing…

Synchronized successfully
User “CHECKSEC” deleted successfully

User “CHECKSEC” created successfully

Created user CHECKSEC with password: [redacted]

Now CHECKSEC needs preferred credentials for the repository DB and repository DB host.
Your repository DB target name is oemdb.domain.com
Enter the credential name for the repository DB Normal Database Credentials: DB-OEMDB-SYSTEM
Enter the credential name for the repository DB SYSDBA Database Credentials: DB-OEMDB-SYS
Enter the credential name for the repository DB Database Host Credentials: HOST-OMSHOST-ORACLE

Validating that supplied credentials exist.

Credentials “DB-OEMDB-SYSTEM:SYSMAN” tested successfully
Credentials “DB-OEMDB-SYS:SYSMAN” tested successfully
Credentials “HOST-OMSHOST-ORACLE:SYSMAN” tested successfully

Granting CHECKSEC GET_CREDENTIAL access to supplied credentials.
Privileges granted to user/role “CHECKSEC” successfully

Confirmed supplied credentials exist and granted to CHECKSEC.

The next section asks you to supply credentials for the account used to run the Oracle Management Agent.

You will receive a separate prompt for each agent. Enter ‘done’ (without quotes) to skip this step.

If you provide these credentials, checksec13R2.sh can report on the Java version used by your agents.

Generating a list of all agent targets.
Now loop through all agent targets and provide named credentials for the agent user account on each host.

Enter the credential name to login as the agent user for host1.domain.com:3872: HOST-HOST1-ORAAGENT
Credentials “HOST-HOST1-ORAAGENT:SYSMAN” tested successfully
Privileges granted to user/role “CHECKSEC” successfully

Enter the credential name to login as the agent user for host2.domain.com:3872: HOST-HOST2-ORAAGENT
Credentials “HOST-HOST2-ORAAGENT:SYSMAN” tested successfully
Privileges granted to user/role “CHECKSEC” successfully

Enter the credential name to login as the agent user for host3.domain.com:3872: HOST-HOST3-ORAAGENT
Credentials “HOST-HOST3-ORAAGENT:SYSMAN” tested successfully
Privileges granted to user/role “CHECKSEC” successfully

Enter the credential name to login as the agent user for host4.domain.com:1830: HOST-HOST4-ORAAGENT
Credentials “HOST-HOST4-ORAAGENT:SYSMAN” tested successfully
Privileges granted to user/role “CHECKSEC” successfully

Enter the credential name to login as the agent user for host5.domain.com:3872: HOST-HOST5-ORAAGENT
Credentials “HOST-HOST5-ORAAGENT:SYSMAN” tested successfully
Privileges granted to user/role “CHECKSEC” successfully

Enter the credential name to login as the agent user for host6.domain.com:1830: HOST-HOST6-ORAAGENT
Credentials “HOST-HOST6-ORAAGENT:SYSMAN” tested successfully
Privileges granted to user/role “CHECKSEC” successfully

Enter the credential name to login as the agent user for host7.domain.com:3872: HOST-HOST7-ORAAGENT
Credentials “HOST-HOST7-ORAAGENT:SYSMAN” tested successfully
Privileges granted to user/role “CHECKSEC” successfully

Enter the credential name to login as the agent user for host8.domain.com:3872: HOST-HOST8-ORAAGENT
Credentials “HOST-HOST8-ORAAGENT:SYSMAN” tested successfully
Privileges granted to user/role “CHECKSEC” successfully

Enter the credential name to login as the agent user for host9.domain.com:1830: HOST-HOST9-ORAAGENT
Credentials “HOST-HOST9-ORAAGENT:SYSMAN” tested successfully
Privileges granted to user/role “CHECKSEC” successfully

Enter the credential name to login as the agent user for host10.domain.com:3872: HOST-HOST10-ORAAGENT
Credentials “HOST-HOST10-ORAAGENT:SYSMAN” tested successfully
Privileges granted to user/role “CHECKSEC” successfully

Enter the credential name to login as the agent user for host11.domain.com:3872: HOST-HOST11-ORAAGENT
Credentials “HOST-HOST11-ORAAGENT:SYSMAN” tested successfully
Privileges granted to user/role “CHECKSEC” successfully

Enter the credential name to login as the agent user for host12.domain.com:3872: HOST-HOST12-ORAAGENT
Credentials “HOST-HOST12-ORAAGENT:SYSMAN” tested successfully
Privileges granted to user/role “CHECKSEC” successfully

Enter the credential name to login as the agent user for host13.domain.com:3872: HOST-HOST13-ORAAGENT
Credentials “HOST-HOST13-ORAAGENT:SYSMAN” tested successfully
Privileges granted to user/role “CHECKSEC” successfully

Enter the credential name to login as the agent user for host14.domain.com:3872: HOST-HOST14-ORAAGENT
Credentials “HOST-HOST14-ORAAGENT:SYSMAN” tested successfully
Privileges granted to user/role “CHECKSEC” successfully

Enter the credential name to login as the agent user for host15.domain.com:3872: HOST-HOST15-ORAAGENT
Credentials “HOST-HOST15-ORAAGENT:SYSMAN” tested successfully
Privileges granted to user/role “CHECKSEC” successfully

Enter the credential name to login as the agent user for host16.domain.com:3872: HOST-HOST16-ORAAGENT
Credentials “HOST-HOST16-ORAAGENT:SYSMAN” tested successfully
Privileges granted to user/role “CHECKSEC” successfully

Enter the credential name to login as the agent user for omshost.domain.com:3872: HOST-OMSHOST-ORACLE
Credentials “HOST-OMSHOST-ORACLE:SYSMAN” tested successfully
Privileges granted to user/role “CHECKSEC” successfully

Enter the credential name to login as the agent user for host17.domain.com:3872: HOST-HOST17-ORAAGENT
Credentials “HOST-HOST17-ORAAGENT:SYSMAN” tested successfully
Privileges granted to user/role “CHECKSEC” successfully

Enter the credential name to login as the agent user for host18.domain.com:3872: HOST-HOST18-ORAAGENT
Credentials “HOST-HOST18-ORAAGENT:SYSMAN” tested successfully
Privileges granted to user/role “CHECKSEC” successfully

Logging out of EMCLI
Logout successful

Logging in to EMCLI as CHECKSEC
Login successful

Setting preferred credentials DB-OEMDB-SYSTEM for CHECKSEC on oemdb.domain.com
Successfully set preferred credentials for target oemdb.domain.com:oracle_database.

Setting preferred credentials DB-OEMDB-SYS for CHECKSEC on oemdb.domain.com
Successfully set preferred credentials for target oemdb.domain.com:oracle_database.

Setting preferred credentials HOST-OMSHOST-ORACLE for CHECKSEC on oemdb.domain.com
Successfully set preferred credentials for target oemdb.domain.com:oracle_database.

Now assigning preferred credentials for agent targets.

Setting preferred credentials for CHECKSEC on host1.domain.com:3872
Successfully set preferred credentials for target host1.domain.com:host.

Setting preferred credentials for CHECKSEC on host2.domain.com:3872
Successfully set preferred credentials for target host2.domain.com:host.

Setting preferred credentials for CHECKSEC on host3.domain.com:3872
Successfully set preferred credentials for target host3.domain.com:host.

Setting preferred credentials for CHECKSEC on host4.domain.com:1830
Successfully set preferred credentials for target host4.domain.com:host.

Setting preferred credentials for CHECKSEC on host5.domain.com:3872
Successfully set preferred credentials for target host5.domain.com:host.

Setting preferred credentials for CHECKSEC on host6.domain.com:1830
Successfully set preferred credentials for target host6.domain.com:host.

Setting preferred credentials for CHECKSEC on host7.domain.com:3872
Successfully set preferred credentials for target host7.domain.com:host.

Setting preferred credentials for CHECKSEC on host8.domain.com:3872
Successfully set preferred credentials for target host8.domain.com:host.

Setting preferred credentials for CHECKSEC on host9.domain.com:1830
Successfully set preferred credentials for target host9.domain.com:host.

Setting preferred credentials for CHECKSEC on host10.domain.com:3872
Successfully set preferred credentials for target host10.domain.com:host.

Setting preferred credentials for CHECKSEC on host11.domain.com:3872
Successfully set preferred credentials for target host11.domain.com:host.

Setting preferred credentials for CHECKSEC on host12.domain.com:3872
Successfully set preferred credentials for target host12.domain.com:host.

Setting preferred credentials for CHECKSEC on host13.domain.com:3872
Successfully set preferred credentials for target host13.domain.com:host.

Setting preferred credentials for CHECKSEC on host14.domain.com:3872
Successfully set preferred credentials for target host14.domain.com:host.

Setting preferred credentials for CHECKSEC on host15.domain.com:3872
Successfully set preferred credentials for target host15.domain.com:host.

Setting preferred credentials for CHECKSEC on host16.domain.com:3872
Successfully set preferred credentials for target host16.domain.com:host.

Setting preferred credentials for CHECKSEC on omshost.domain.com:3872
Successfully set preferred credentials for target omshost.domain.com:host.

Setting preferred credentials for CHECKSEC on host17.domain.com:3872
Successfully set preferred credentials for target host17.domain.com:host.

Setting preferred credentials for CHECKSEC on host18.domain.com:3872
Successfully set preferred credentials for target host18.domain.com:host.

All finished. User CHECKSEC now logged in to EMCLI.

Now go run the checksec13R2.sh script.

As a reminder, user CHECKSEC has password [redacted].

Previous Versions

• Enterprise Manager 13c R1
• Enterprise Manager 12c R4

Fix: Plugin error when upgrading EM13cR1 agent to EM13cR2 (13.2) on Windows 2008 R2 x64

I ran into the following issue while attempting to upgrade to the Oracle management agent on my one Windows (2008 R2, x64) server to the 13.2.0.0.0 version distributed with Oracle Enterprise Manager 13.2. The agent upgrade repeatedly failed in the “Upgrading Management Agent” step with an error message:


Exit Code :0
The version is 13.1.0.0.0
Checking for the version 13.2.0.0.0
The agent is not upgraded successfully
[...]
Plugins upgrade failed.
Plugin upgrade failed.
Check the file E:/agent13c/agent_inst/install/plugins_upgrade.txt.status on agent for plugin upgrade status.
Check latest E:/agent13c/agent_inst/install/logs/agentplugindeploy_.log
Plugins upgrade failed.0


The log file referenced in the error message contains some, but not much, additional information:


The command executed for discovery at _2016_10_12_11_48_28 is : E:\agent13c\agent_inst\bin\emctl_upgrade.bat set_discovery_root plugin oracle.sysman.si E:\agent13c\agent_13.2.0.0.0\plugins\oracle.sysman.si.discovery.plugin_13.2.1.0.0 13.2.1.0.0
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
return value is : 65280
Install case : set_discovery_root failure...existing on error value 65280


I fixed this issue by installing the Microsoft Visual C++ 2010 Service Pack 1 Redistributable Package MFC Security Update package on the server. Finally, create a new directory called “prerelogs” inside of the …\agent_13.2.0.0.0\cfgtoollogs\ directory, or else my deployment hung forever during the “Performing prerequisite checks” step. Note that “prerelogs” does not contain the letter q, do not use ‘prereqlogs’. The agent deployment uses this directory to extract software for running the prerequisite checks; possibly setting SCRATCHPATH or another extra parameter would resolve this issue without manually creating the directory, but this worked for me.

After installation of the VC++ 2010 package, which did not require a reboot, followed by creating of the prerelogs directory, I repeated the agent upgrade and it completed successfully.

This issue may or may not occur on other Windows versions, I have not heard any other reports of problems. The first release of EM13c did not seem to require this package.

Script to automate lock down of all EM13c agents to TLSv1.2 with EMCLI

I could not find any obvious documentation about locking down Oracle Enterprise Manager 13c management agents to forbid TLSv1 and TLSv1.1, permitting only TLSv1.2, so I went looking and found the emdpropdefs.xml file in $AGENT_HOME/agent_13.1.0.0.0/sysman/admin/ that documents the existence of the minimumTLSVersion property in emd.properties:

name='minimumTLSVersion'
modifiable='true'
defaultValue='TLSv1'
description='The oldest version of the TLS protocol which this agent should support when accepting connections or initiating connections to the OMS. Currently supported values are "TLSv1", "TLSv1.1", and "TLSv1.2".'
valueType='String'
advanced='true'
migrate='source'
filename='emd.properties'
category='Runtime Settings'
internal='true'
restartRequired='true'

I tested this parameter on my OMS server agent, restarted the agent, and confirmed with my Securing Oracle Enterprise Manager 13c script that the agent no longer allowed connections using any protocol other than TLSv1.2. Next I wanted to automated this, to avoid the effort of manually changing this property on each agent and then restarting that agent, so I went directly to EMCLI which allows EM13c admins to (among many other things) set agent properties and restart agents. I then created a script to fetch a list of all agents, check for the TLS protocols each agent permits, and then apply the change and restart the agent for every agent that I had not already locked down. I have copied this script below.

Before using the script, you must login to EMCLI using “emcli login -username=yourusername” and provide your password. For security reasons I elected not to wrap the EMCLI login within this script; that way you do not have to trust my script to handle your password securely, as the script never sees your password. For the step to restart your agents to work correctly, you need to make sure that your EM13c user account has preferred host credentials set for your agent targets that can successfully login to the host server and restart the agent.

Here is a copy of the script, followed by the (anonymized) output from a sample run. Someday soon I will get set up on github to make it easier to retrieve my scripts, but for now you can copy and paste this. This script expects to find the emcli binary inside of the $MW_HOME/bin directory, so make sure you have $MW_HOME set before running it, or provide the full path to EMCLI within the script. It will also log you out of EMCLI when the script completes.


#!/bin/bash
#
# This script will retrieve a list of agents from your EM13c environment,
# determine if they allow connections using TLS protocol versions older
# than TLSv1.2, and then disable all protocols older than TLSv1.2.
#
# Finally it will restart each modified agent to apply the change.
#
# You need to login to EMCLI first before running this script.
#
# Released v0.1: Initial beta release 5 Oct 2016
#
#
# From: @BrianPardy on Twitter
# https://pardydba.wordpress.com/
#
# Known functional on Linux x86-64, may work on Solaris and AIX.

EMCLI=$MW_HOME/bin/emcli

if [[ -x "/usr/sfw/bin/gegrep" ]]; then
GREP=/usr/sfw/bin/gegrep
else
GREP=`which grep`
fi

OPENSSL=`which openssl`

if [[ -x "/usr/bin/openssl1" && -f "/etc/SuSE-release" ]]; then
OPENSSL=`which openssl1`
fi

OPENSSL_HAS_TLS1_2=`$OPENSSL s_client help 2>&1 | $GREP -c tls1_2`

$EMCLI sync
NOT_LOGGED_IN=$?

if [[ $NOT_LOGGED_IN > 0 ]]; then
echo "Login to EMCLI with \"$EMCLI login -username=USER\" then run this script again"
exit 1
fi

for agent in `$EMCLI get_targets -targets=oracle_emd | grep oracle_emd | awk '{print $4}'`
do
echo
if [[ $OPENSSL_HAS_TLS1_2 > 0 ]]; then
echo -n "Checking TLSv1 on $agent... "

OPENSSL_RETURN=`echo Q | $OPENSSL s_client -prexit -connect $agent -tls1 2>&1 | $GREP Cipher | $GREP -c 0000`

if [[ $OPENSSL_RETURN == 0 ]]; then
echo "allows TLSv1"
else
echo "already forbids TLSv1"
fi
fi

if [[ $OPENSSL_HAS_TLS1_2 > 0 ]]; then
echo -n "Checking TLSv1.1 on $agent... "

OPENSSL_TLS11_RETURN=`echo Q | $OPENSSL s_client -prexit -connect $agent -tls1_1 2>&1 | $GREP Cipher | $GREP -c 0000`

if [[ $OPENSSL_RETURN == 0 ]]; then
echo "allows TLSv1.1"
else
echo "already forbids TLSv1.1"
fi
fi

if [[ $OPENSSL_RETURN == 0 || $OPENSSL_TLS11_RETURN == 0 ]]; then
$EMCLI set_agent_property -agent_name=$agent -name=minimumTLSVersion -value=TLSv1.2 -new

echo
echo "Restarting $agent to apply changes"
$EMCLI restart_agent -agent_name=$agent -credential_setname="HostCreds"
RESTART_RETURN=$?

if [[ $RESTART_RETURN != 0 ]]; then
echo "Unable to restart agent: restart agent manually or set preferred host credentials for agent"
fi
fi
done

$EMCLI logout

exit 0


Sample (anonymized) output below. Note how the script cannot restart an agent lacking preferred host credentials. In this case, I assign preferred host credentials and then re-run the script to complete the process.


Synchronized successfully

Checking TLSv1 on server1.subdomain.domain.com:1830... already forbids TLSv1
Checking TLSv1.1 on server1.subdomain.domain.com:1830... already forbids TLSv1.1

Checking TLSv1 on server2.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server2.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server3.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server3.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server4.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server4.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server5.domain.com:1830... already forbids TLSv1
Checking TLSv1.1 on server5.domain.com:1830... already forbids TLSv1.1

Checking TLSv1 on server6.domain.com:1830... already forbids TLSv1
Checking TLSv1.1 on server6.domain.com:1830... already forbids TLSv1.1

Checking TLSv1 on server7.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server7.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server8.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server8.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server9.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server9.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server10.domain.com:1830... already forbids TLSv1
Checking TLSv1.1 on server10.domain.com:1830... already forbids TLSv1.1

Checking TLSv1 on server11.domain.com:1830... already forbids TLSv1
Checking TLSv1.1 on server11.domain.com:1830... already forbids TLSv1.1

Checking TLSv1 on server12.domain.com:1830... already forbids TLSv1
Checking TLSv1.1 on server12.domain.com:1830... already forbids TLSv1.1

Checking TLSv1 on omshost.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on omshost.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server13.domain.com:3872... allows TLSv1
Checking TLSv1.1 on server13.domain.com:3872... allows TLSv1.1
Agent Property minimumTLSVersion has been successfully updated to the value TLSv1.2.

Restarting server13.domain.com:3872 to apply changes
The Restart operation is in progress for the Agent: server13.domain.com:3872
The Agent "server13.domain.com:3872" has been restarted successfully.
---------------------
Operation Output
---------------------
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.Stopping agent ... stopped.Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.Starting agent ................ started.

Checking TLSv1 on server14.domain.com:1830... allows TLSv1
Checking TLSv1.1 on server14.domain.com:1830... allows TLSv1.1
Agent Property minimumTLSVersion has been successfully updated to the value TLSv1.2.

Restarting server14.domain.com:1830 to apply changes
The Restart operation is in progress for the Agent: server14.domain.com:1830
Unable to restart agent: restart agent manually or set preferred host credentials for agent

Checking TLSv1 on server15.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server15.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server16.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server16.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server17.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server17.domain.com:3872... already forbids TLSv1.1
Logout successful


Securing Oracle Enterprise Manager 13c

[20170822 NOTE: Oracle released the last set of bundle patches for the Oracle Enterprise Manager 13c version in April of 2017. See MOS note 2124038.1 for more details. You really should upgrade to EM13cR2 if you can. My security checkup script for EM13cR2 contains much added functionality and continues to receive updates. I do not expect to release any further updates to this script unless Oracle releases any further key patches or someone who uses it reports a fixable bug.]
[20170418 NOTE: I have upgraded the patches referenced in this script to reflect the latest (20170418) PSU patch for EM13cR1. I no longer have an EM13cR1 environment available with which to test this script, so please feel free to report issues or to submit a git pull request. I have now placed this script on github: https://raw.githubusercontent.com/brianpardy/em13c/master/checksec13.sh.]
[20161013 NOTE: I have upgraded to EM13cR2, and this script still works as expected. If you attempt to run it on an EM13cR2 environment please take note that all of the patch recommendations listed apply to the older EM13cR1 release and will provide incorrect results on 13.2. The TLS, certificate, and cipher strength tests all function correctly on 13.2.]

Download final version

You can download the final release of my EM13cR1 security checkup script at GitHub.

Introduction

This post continues my series on securing Oracle Enterprise Manager environments with some updates relevant to EM13c. Oracle has made significant security improvements with Oracle Enterprise Manager 13c over the prior 12c version, first released in October 2011, more than four and a half years ago at this point. In the interest of security, I have to strongly recommend that any sites still using EM12c upgrade to (or perform a fresh installation of) EM13c EM13cR2 as soon as possible. More recent versions of EM12c like 12.1.0.5 (June 2015) continue to use the same technology stack as the initial release, and the world of security has massively changed since then. Notably, EM13c uses Java 7, WebLogic 12.1.3, and disables SSLv3 out of the box.

Just to recap, back at the EM12c original release date:

• Practically nobody had ever heard of Edward Snowden
• The first release of Java 7 celebrated its three month birthday
• Two months later, Oracle released WebLogic 12c; EM12c users remained on WebLogic 10.3.6
• One month earlier, the public learned of the BEAST attack and people still believed that using RC4 (immune to BEAST) as a workaround improved security (spoiler warning: it did not)
• We had three years to wait before the POODLE vulnerability caused vendors to recognize the need to disable SSLv3 (you DID disable SSLv3, right?)
• Oracle still considered the MD5 hashing algorithm good enough to use in self-signed certificates produced by EM12c, despite flaws known to exist since 1996
• Web browsers considered the SHA-1 hashing algorithm, now also deprecated due to brokenness, good enough to use

As the security world’s known unknowns collapsed around us, proactive EM12c administrators sought to make the best of their lot. Outside of Oracle, I and others poked at the software and wrote blog articles, while inside Oracle effort proceeded to support more recent Java releases that brought with them better cipher suites and hashing algorithms, as well as the usual security fixes. This process took some time for all involved and hit a few bumps along the way.

• Replacement of EM12c certificates with custom certificates making use of the SHA-2 hashing algorithm appeared to work fine until we tried to apply patches: EM12c opatchauto, SHA256, and you
• Each time a new attack broke in the press, proactive EM12c administrators including myself looked for ways to mitigate it, as with my first fumbling efforts when the POODLE bug became public: How to unofficially disable SSL v3 in Oracle Enterprise Manager 12c to mitigate POODLE attack
• The difficulty of keeping track of now-monthly patching and rechecking known issues after patching led me to create an automated security check script for EM12c: EM12c R4 SSL Security Checkup Script
• The official documentation, though exhaustive, sometimes complicated security issues beyond the minimum necessary to achieve a task, so I produced a step-by-step guide to replacing as many certificates as possible with those sourced from third parties or internal sources: Step by step: Configuring third party SSL/TLS certificates in EM12c R4
• Sometimes bizarre bugs led to an inability to disable known weak cipher suites, requiring deep dives into patching and configuration: EM12c OHS, LOW strength ciphers, custom certificates, and patch 1994800 weirdness
• Sometimes we shoot ourselves in the foot by carelessly upgrading components: When proactive EM12c JDK upgrades bite back
• Applying your strong certificates to the full EM12c stack does not eliminate the severe security issues created by Oracle’s decision to include a widely known demo certificate authority with WebLogic: Insecure certificate warning still exists in EM12c WLS logs with third-party certificates applied

I do not intend in this post to review general day-to-day EM13c security design such as user roles or privileges, object level security within OEM, or integration with identity providers like LDAP; only the infrastructure level issues that tend to change in brief large bursts as new attacks come out. See this excellent list of EM13c blogs, links and videos that Philip Brown has provided for more details on these and other items.

On to EM13c

EM13c admins need to keep an eye on the same sorts of items as with EM12c. We really should read the documentation, even if only the Security Guide. I admit I often do not. It contains good information.

Patches

I consider it critical for admins to keep up with the OEM periodic patches, particularly security patches. The script below covers patches up to and including March 31, 2016. I plan to update again after the next set of Oracle security patches arrives, likely mid-April.

Certificates

The process for applying certificates on EM13c does not appear to have changed significantly from the prior version. I have confirmed that the new “omspatcher” tool that replaces opatchauto when applying a system patch to the OMS works perfectly fine with certificates on WebLogic that use the SHA-256 hashing algorithm. Given the upcoming deprecation of SHA-1 across all major browsers I do not see any valid reason not to use SHA-256 with new certificates.

Ciphersuites

Out of the box, my EM13c installation rejected weak ciphersuites and accepted the strong ones. Unfortunately it still accepted some that these versions of Java and OpenSSL consider as MEDIUM strength, so I want to disable those across the entire environment, leaving only the strongest ciphersuites available in this release and permitting other ciphersuites only where necessary.

[UPDATE 20160518: Please see MOS note 2138391.1 for the official procedure to disable weak cipher suites in EM13c.]

We will have to live with these unwanted ciphersuites enabled until Oracle provides a supported procedure to disable them across the entire stack. I have performed some preliminary tests and I find it very easy to get OEM into a situation where it cannot startup after manually editing config files that define enabled ciphersuites. The script below will identify ports permitting ciphersuites you may wish to disable when a supported method becomes available.

UPDATE 20160720: Take particular care of watching the ciphersuites accepted by your agents if you upgrade the JDK that the agents use. I just attempted a JDK update on an agent from the distributed version to 1.7.0-111, and that agent began to accept LOW and MEDIUM strength ciphersuites again, thus I have omitted JDK updates for agents from the check script.

Security Checkup

Below I provide an early version of the script I use to validate EM13c security configuration. I based this on my earlier EM12c script, linked above. The script will become more useful once I implement patch level checking after release of the first set of EM13c patches, but for the moment it will inspect your EM13c environment to identify relevant ports and confirm that your system will not respond to SSLv2 or SSLv3 requests, does respond to TLSv1 requests, supports HIGH, but not LOW or MEDIUM strength ciphersuites (as defined by the version of OpenSSL installed on your OMS host), and finally checks for the presence of demonstration-not-for-production-use certificates and self-signed certificates.

(A caveat on self-signed certificate checking: OpenSSL, not this script, performs the check, therefore if OpenSSL cannot validate your certificate to a trusted root, this script cannot either. If a well known certification authority has signed your certificates, OpenSSL should validate them successfully, but it may not do so if you use an internal certificate authority to sign certificates. In that case you should install a copy of your internal CA as a trusted root certificate in the system trust store so that OpenSSL can validate your EM13c certificates. I cannot document this process for every OS but Linux users should look to the documentation for the update-ca-certificates or update-ca-trust commands. If my script below incorrectly reports your certificate as self-signed, you can ignore the finding or address the issue at the OpenSSL level.)

EM13c TLS Security Checkup Script

[LATEST UPDATE: 20161004, adds 20160920 patches and fixes TLSv1 vs TLSv1.2 bugs, version 0.9]. Thank you to Bob Schuppin who reported a bug in the use of TLSv1 to check certificate and cipher suite usage in a TLSv1.2-only site. I have modified the relevant checks to use TLSv1.2 if supported by your OpenSSL version or to stick with TLSv1 if not supported.

[PRIOR UPDATE: 20160914 bugfix and enhancements, no new patch checks, version 0.8]. Thank you to Paige who reported a bug in the check of the SSL_CIPHER_SUITES parameter. I had a typo in the cipher suite names for the SSL_CIPHER_SUITES parameter, which I have now fixed. In researching this I realized that this parameter provides control over SSL/TLS authentication for clients, which I do not use in my environment. Instead I use native SQL*Net encryption, controlled by the various ENCRYPTION_(CLIENT|SERVER), ENCRYPTION_TYPES_(CLIENT|SERVER), CRYPTO_CHECKSUM_(CLIENT|SERVER), and CRYPTO_CHECKSUM_TYPES_(CLIENT|SERVER) parameters, which I have added into the script. The script will check to make sure that you do not permit MD5 as a SQL*Net checksum algorithm and that you do not permit DES, DES40, 3DES112, nor any of the RC4_ algorithms for SQL*Net encryption. Unfortunately due to bug 23587582, you will encounter problems promoting targets in OEM unless you allow use of the 3DES168 encryption algorithm and SHA1 hashing algorithm. Generally I would prefer to disable both of those for security reasons but I will permit them in this script as long as they remain necessary for full OEM functionality.

[PRIOR UPDATE: 20160819 for 20160816 bundle patches, version 0.7]. With this update, I have added support for TLSv1.1 and TLSv1.2 to the protocol checks. I have also added support for the optional SLES11 openssl1 package which provides a newer OpenSSL supporting TLSv1.1 and TLSv1.2 for systems on SLES11 like mine. The script will now dynamically determine (by parsing the “openssl s_client help” output) if your OpenSSL version supports TLSv1.2. If your OpenSSL version DOES support TLSv1.2, the script will now flag any support of TLSv1 or TLSv1.1 as a failure in your OEM stack. If your OpenSSL does NOT support TLSv1.2, the script will consider TLSv1 support in OEM as acceptable. If you notice problems with this new functionality please let me know.

Compatibility

Only tested on Linux x86-64, but may work on AIX and Solaris as the EM12c version I built this upon did work there. Planned future enhancements include checking that you have disabled non-encrypted HTTP access to EM13c components, upgraded Java to the most recent EM13c-supported release, and more.

You can download the latest version of the script from github: https://raw.githubusercontent.com/brianpardy/em13c/master/checksec13.sh.

EM13c TLS Security Checkup Script Sample Output


Performing EM13c security checkup version 0.9 on omshost.domain.com at Tue Oct 4 11:04:43 EDT 2016.

Using port definitions from configuration files
/etc/oragchomelist
/oracle/oem/gc_inst/em/EMGC_OMS1/emgc.properties
/oracle/oem/gc_inst/em/EMGC_OMS1/embip.properties

Agent port found at omshost.domain.com:3872
BIPublisher port found at omshost.domain.com:9803
BIPublisherOHS port found at omshost.domain.com:9851
NodeManager port found at omshost.domain.com:7403
OMSconsole port found at omshost.domain.com:7802
OMSproxy port found at omshost.domain.com:7301
OMSupload port found at omshost.domain.com:4903
WLSadmin found at omshost.domain.com:7102

Repository DB version=12.1.0.2.0 SID=oemdb host=omshost.domain.com

Using OPENSSL=/usr/bin/openssl1 (has TLS1_2=2)
Repository DB on OMS server, will check patches/parameters in /oracle/oem/product/12.1.0/db

(1) Checking SSL/TLS configuration (see notes 1602983.1, 1477287.1, 1905314.1)

(1a) Forbid SSLv2 connections
Confirming ssl2 disabled for Agent at omshost.domain.com:3872... OK
Confirming ssl2 disabled for BIPublisher at omshost.domain.com:9803... OK
Confirming ssl2 disabled for NodeManager at omshost.domain.com:7403... OK
Confirming ssl2 disabled for BIPublisherOHS at omshost.domain.com:9851... OK
Confirming ssl2 disabled for OMSconsole at omshost.domain.com:7802... OK
Confirming ssl2 disabled for OMSproxy at omshost.domain.com:7301... OK
Confirming ssl2 disabled for OMSupload at omshost.domain.com:4903... OK
Confirming ssl2 disabled for WLSadmin at omshost.domain.com:7102... OK

(1b) Forbid SSLv3 connections
Confirming ssl3 disabled for Agent at omshost.domain.com:3872... OK
Confirming ssl3 disabled for BIPublisher at omshost.domain.com:9803... OK
Confirming ssl3 disabled for NodeManager at omshost.domain.com:7403... OK
Confirming ssl3 disabled for BIPublisherOHS at omshost.domain.com:9851... OK
Confirming ssl3 disabled for OMSconsole at omshost.domain.com:7802... OK
Confirming ssl3 disabled for OMSproxy at omshost.domain.com:7301... OK
Confirming ssl3 disabled for OMSupload at omshost.domain.com:4903... OK
Confirming ssl3 disabled for WLSadmin at omshost.domain.com:7102... OK

(1c) Forbid TLSv1 connections
Confirming tls1 disabled for Agent at omshost.domain.com:3872... FAILED
Confirming tls1 disabled for BIPublisher at omshost.domain.com:9803... FAILED
Confirming tls1 disabled for NodeManager at omshost.domain.com:7403... FAILED
Confirming tls1 disabled for BIPublisherOHS at omshost.domain.com:9851... FAILED
Confirming tls1 disabled for OMSconsole at omshost.domain.com:7802... FAILED
Confirming tls1 disabled for OMSproxy at omshost.domain.com:7301... FAILED
Confirming tls1 disabled for OMSupload at omshost.domain.com:4903... FAILED
Confirming tls1 disabled for WLSadmin at omshost.domain.com:7102... FAILED

(1c) Forbid TLSv1.1 connections
Confirming tls1_1 disabled for Agent at omshost.domain.com:3872... FAILED
Confirming tls1_1 disabled for BIPublisher at omshost.domain.com:9803... FAILED
Confirming tls1_1 disabled for NodeManager at omshost.domain.com:7403... FAILED
Confirming tls1_1 disabled for BIPublisherOHS at omshost.domain.com:9851... FAILED
Confirming tls1_1 disabled for OMSconsole at omshost.domain.com:7802... FAILED
Confirming tls1_1 disabled for OMSproxy at omshost.domain.com:7301... FAILED
Confirming tls1_1 disabled for OMSupload at omshost.domain.com:4903... FAILED
Confirming tls1_1 disabled for WLSadmin at omshost.domain.com:7102... FAILED

(1c) Permit TLSv1.2 connections
Confirming tls1_2 available for Agent at omshost.domain.com:3872... OK
Confirming tls1_2 available for BIPublisher at omshost.domain.com:9803... OK
Confirming tls1_2 available for NodeManager at omshost.domain.com:7403... OK
Confirming tls1_2 available for BIPublisherOHS at omshost.domain.com:9851... OK
Confirming tls1_2 available for OMSconsole at omshost.domain.com:7802... OK
Confirming tls1_2 available for OMSproxy at omshost.domain.com:7301... OK
Confirming tls1_2 available for OMSupload at omshost.domain.com:4903... OK
Confirming tls1_2 available for WLSadmin at omshost.domain.com:7102... OK

(2) Checking supported ciphers at SSL/TLS endpoints (see notes 2138391.1, 1067411.1)
Checking LOW strength ciphers on Agent (omshost.domain.com:3872, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on Agent (omshost.domain.com:3872)... OK
Checking HIGH strength ciphers on Agent (omshost.domain.com:3872)... OK

Checking LOW strength ciphers on BIPublisher (omshost.domain.com:9803, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on BIPublisher (omshost.domain.com:9803)... OK
Checking HIGH strength ciphers on BIPublisher (omshost.domain.com:9803)... OK

Checking LOW strength ciphers on NodeManager (omshost.domain.com:7403, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on NodeManager (omshost.domain.com:7403)... OK
Checking HIGH strength ciphers on NodeManager (omshost.domain.com:7403)... OK

Checking LOW strength ciphers on BIPublisherOHS (omshost.domain.com:9851, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on BIPublisherOHS (omshost.domain.com:9851)... OK
Checking HIGH strength ciphers on BIPublisherOHS (omshost.domain.com:9851)... OK

Checking LOW strength ciphers on OMSconsole (omshost.domain.com:7802, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on OMSconsole (omshost.domain.com:7802)... OK
Checking HIGH strength ciphers on OMSconsole (omshost.domain.com:7802)... OK

Checking LOW strength ciphers on OMSproxy (omshost.domain.com:7301, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on OMSproxy (omshost.domain.com:7301)... OK
Checking HIGH strength ciphers on OMSproxy (omshost.domain.com:7301)... OK

Checking LOW strength ciphers on OMSupload (omshost.domain.com:4903, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on OMSupload (omshost.domain.com:4903)... OK
Checking HIGH strength ciphers on OMSupload (omshost.domain.com:4903)... OK

Checking LOW strength ciphers on WLSadmin (omshost.domain.com:7102, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on WLSadmin (omshost.domain.com:7102)... OK
Checking HIGH strength ciphers on WLSadmin (omshost.domain.com:7102)... OK

(3) Checking self-signed and demonstration certificates at SSL/TLS endpoints (see notes 1367988.1, 1399293.1, 1593183.1, 1527874.1, 123033.1, 1937457.1)
Checking certificate at Agent (omshost.domain.com:3872, protocol tls1_2)... FAILED - Found self-signed certificate
Checking demo certificate at Agent (omshost.domain.com:3872, protocol tls1_2)... OK
Checking certificate at BIPublisher (omshost.domain.com:9803, protocol tls1_2)... OK
Checking demo certificate at BIPublisher (omshost.domain.com:9803, protocol tls1_2)... OK
Checking certificate at NodeManager (omshost.domain.com:7403, protocol tls1_2)... OK
Checking demo certificate at NodeManager (omshost.domain.com:7403, protocol tls1_2)... OK
Checking certificate at BIPublisherOHS (omshost.domain.com:9851, protocol tls1_2)... OK
Checking demo certificate at BIPublisherOHS (omshost.domain.com:9851, protocol tls1_2)... OK
Checking certificate at OMSconsole (omshost.domain.com:7802, protocol tls1_2)... OK
Checking demo certificate at OMSconsole (omshost.domain.com:7802, protocol tls1_2)... OK
Checking certificate at OMSproxy (omshost.domain.com:7301, protocol tls1_2)... OK
Checking demo certificate at OMSproxy (omshost.domain.com:7301, protocol tls1_2)... OK
Checking certificate at OMSupload (omshost.domain.com:4903, protocol tls1_2)... OK
Checking demo certificate at OMSupload (omshost.domain.com:4903, protocol tls1_2)... OK
Checking certificate at WLSadmin (omshost.domain.com:7102, protocol tls1_2)... OK
Checking demo certificate at WLSadmin (omshost.domain.com:7102, protocol tls1_2)... OK

(4) Checking EM13c Oracle home patch levels against 20 Sep 2016 baseline (see notes 1664074.1, 1900943.1, 822485.1, 1470197.1, 1967243.1)

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) PSU 12.1.0.2.160719 (JUL2016) (23054246)... OK
Patch 23054246 : applied on Wed Jul 20 12:01:53 EDT 2016 Patch description: "Database Patch Set Update : 12.1.0.2.160719 (23054246)"

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) ORACLE JAVAVM COMPONENT 12.1.0.2.160719 DATABASE PSU (JUL2016) (23177536)... OK
Patch 23177536 : applied on Wed Jul 20 12:03:14 EDT 2016 21566993, 22670413, 19699946, 23177536, 22118835, 22118851, 19895326

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_TYPES_SERVER parameter (76629.1, 2167682.1)... OK
(AES128,AES256,AES192,3DES168)

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_SERVER parameter (76629.1, 2167682.1)... OK
requested

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_TYPES_CLIENT parameter (76629.1, 2167682.1)... OK
(AES128,AES256,AES192,3DES168)

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_CLIENT parameter (76629.1, 2167682.1)... OK
requested

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_TYPES_SERVER parameter (76629.1, 2167682.1)... OK
(SHA1)

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_SERVER parameter (76629.1, 2167682.1)... OK
requested

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_TYPES_CLIENT parameter (76629.1, 2167682.1)... OK
(SHA1)

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_CLIENT parameter (76629.1, 2167682.1)... OK
requested

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SSL_VERSION parameter (1545816.1)... OK
1.0

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SSL_CIPHER_SUITES parameter (1545816.1)... OK
(SSL_RSA_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_AES_256_CBC_SHA)

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) listener.ora SSL_VERSION parameter (1545816.1)... OK
1.0

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) listener.ora SSL_CIPHER_SUITES parameter (1545816.1)... OK
(SSL_RSA_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_AES_256_CBC_SHA)

(4c) *UPDATED* OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM-AGENT BUNDLE PATCH 13.1.0.0.160920 (24437699)... OK
Patch 24437699 : applied on Tue Sep 27 12:08:23 EDT 2016 24437699, 21779343, 22616051, 23759799, 22988508, 23089106, 23581450

(4c) *UPDATED* OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM DB PLUGIN BUNDLE PATCH 13.1.1.0.160920 MONITORING (24545984)... OK
Patch 24545984 : applied on Tue Sep 27 13:46:08 EDT 2016 22908077, 23294830, 22503390, 23075475, 23697777, 24545984, 24296310

(4c) *UPDATED* OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM DB PLUGIN BUNDLE PATCH 13.1.1.0.160920 DISCOVERY (24545989)... OK
Patch 24545989 : applied on Tue Sep 27 13:46:11 EDT 2016 23523964, 23294839, 24545989, 23226583, 24408840

(4c) *UPDATED* OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM FMW PLUGIN BUNDLE PATCH 13.1.1.0.160920 MONITORING (24658006)... OK
Patch 24658006 : applied on Tue Sep 27 13:46:13 EDT 2016 22834135, 23007497, 22447329, 22936491, 24658006, 23294872, 23306887

(4c) OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM SI PLUGIN BUNDLE PATCH 13.1.1.0.160719 MONITORING (23697783)... OK
Patch 23697783 : applied on Wed Jul 20 10:53:57 EDT 2016 22128210, 23338028, 23189991, 22823189, 21253819, 23697783, 23208587

(4c) OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM SI PLUGIN BUNDLE PATCH 13.1.1.0.160531 DISCOVERY (23294895)... OK
Patch 23294895 : applied on Thu Jun 16 11:28:18 EDT 2016 23197299, 23294895

(4c) OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM OH PLUGIN BUNDLE PATCH 13.1.1.0.160429 (23135564)... OK
Patch 23135564 : applied on Wed May 11 13:21:35 EDT 2016 22521822, 23135564

(4d) *UPDATED* OMS HOME (/oracle/oem/Middleware13cR1) ENTERPRISE MANAGER FOR OMS PLUGINS 13.1.1.0.160920 (24546113)... OK
oracle.sysman.emas.oms.plugin/13.1.1.0.0 Plugin 24546113 24437669 oracle.sysman.cfw.oms.plugin/13.1.1.0.0 Plugin 24546113 24437711 oracle.sysman.db.oms.plugin/13.1.1.0.0 Plugin 24546113 24437646 oracle.sysman.xa.oms.plugin/13.1.1.0.0 Plugin 24546113 24437656

(4d) (/oracle/oem/Middleware13cR1) WLS PATCH SET UPDATE 12.1.3.0.160719 (23094292)... OK
Patch 23094292 : applied on Wed Jul 20 12:27:53 EDT 2016

(4f) OMS HOME (/oracle/oem/Middleware13cR1) ENTERPRISE MANAGER BASE PLATFORM PATCH SET UPDATE 13.1.0.0.160719 (23134365)... OK
oracle.sysman.top.oms/13.1.0.0.0 Core 23134365 23134365

(5) Checking EM13c Java patch levels against 20 Sep 2016 baseline (see notes 1492980.1, 1616397.1)

(5a) WLS (/oracle/oem/Middleware13cR1/oracle_common/jdk) JAVA SE JDK VERSION 1.7.0-111 (13079846)... OK
1.7.0_111

Failed test count: 17 - Review output

sslcheck:Agent @ omshost.domain.com:3872:tls1 protocol connection allowed
sslcheck:BIPublisher @ omshost.domain.com:9803:tls1 protocol connection allowed
sslcheck:NodeManager @ omshost.domain.com:7403:tls1 protocol connection allowed
sslcheck:BIPublisherOHS @ omshost.domain.com:9851:tls1 protocol connection allowed
sslcheck:OMSconsole @ omshost.domain.com:7802:tls1 protocol connection allowed
sslcheck:OMSproxy @ omshost.domain.com:7301:tls1 protocol connection allowed
sslcheck:OMSupload @ omshost.domain.com:4903:tls1 protocol connection allowed
sslcheck:WLSadmin @ omshost.domain.com:7102:tls1 protocol connection allowed
sslcheck:Agent @ omshost.domain.com:3872:tls1_1 protocol connection allowed
sslcheck:BIPublisher @ omshost.domain.com:9803:tls1_1 protocol connection allowed
sslcheck:NodeManager @ omshost.domain.com:7403:tls1_1 protocol connection allowed
sslcheck:BIPublisherOHS @ omshost.domain.com:9851:tls1_1 protocol connection allowed
sslcheck:OMSconsole @ omshost.domain.com:7802:tls1_1 protocol connection allowed
sslcheck:OMSproxy @ omshost.domain.com:7301:tls1_1 protocol connection allowed
sslcheck:OMSupload @ omshost.domain.com:4903:tls1_1 protocol connection allowed
sslcheck:WLSadmin @ omshost.domain.com:7102:tls1_1 protocol connection allowed
certcheck:Agent @ omshost.domain.com:3872 found self-signed certificate

Visit https://pardydba.wordpress.com/2016/04/05/securing-oracle-enterprise-manager-13c/ for the latest version.


WORKAROUND: Unable to monitor Oracle XE 11gR2 with Oracle Enterprise Manager 13c

I have recently switched to using Oracle Enterprise Manager 13c (EM13c – 13.1), replacing my previous EM12c installation.  I elected to install a clean new environment instead of an upgrade, because my old install had been upgraded repeatedly going back to the initial release of EM12c and I wanted a fresh start.

I encountered only one difficult issue during the process. When I attempted to add one production Oracle XE 11gR2 database target, EM13c could not compute the target’s dynamic properties, leaving the target broken. Since you cannot submit jobs against a broken target, this prevented me from using EM13c to back up this database.  I had no comparable issues with XE as a target under EM12c.

The key metric errors that showed during this process included:
“Metric evaluation error start – Target {oracle_database.SID.domain.com} is broken: Dynamic Category property error,Get dynamic property error,No such metadata – No valid queryDescriptor or executionDescriptor found for target [oracle_database.SID.domain.com$30]”

and for the database system target:

“Metric evaluation error start – Received an exception when evaluating sev_eval_proc for:Target name = SID.domain.com_sys, metric_name = Response, metric_column = Status; Error msg = Target encountered metric erros; at least one member in in metric error”

I enabled debugging for the agent logs and attempted again to add the XE target.  Errors showing up in the logs included:

2016-01-15 12:10:05,905 [1806:4CE3192] DEBUG – Computing of dynamic property: [ComputeVC] is done (1 msec, error=true)

2016-01-15 12:10:06,452 [1806:F917F5F8] DEBUG – Computing of dynamic property: [GetDumpDestination] is done (0 msec, error=true)

2016-01-15 12:10:06,508 [1813:6EEEAC87] DEBUG – Computing of dynamic property: [DeduceAlertLogFile] is done (1 msec, error=true)

2016-01-15 12:11:18,779 [1830:CD3A325D] DEBUG – Error was added to oracle_database.SID.domain.com$23(0|MONITORED|false|true|<UF>): Invalid Input

2016-01-15 12:11:18,779 [1831:3657AE55] DEBUG – abandoning long op “CDProps:oracle_database.SID.domain.com:ComputeVC:GENERIC_TASK:Fri Jan 15 12:11:18 EST 2016”

2016-01-15 12:11:18,780 [1830:CD3A325D] DEBUG – Error during dynamic property ComputeVC calculation: critical=true, missingCatProps=[VersionCategory], missingProps=[VersionCategory] oracle.sysman.emSDK.agent.fetchlet.exception.FetchletException: Invalid Input

2016-01-15 12:35:38,038 INFO – Finished dynamic properties computation (total time 817 ms). Number of DP computed: 19. Number of DP with error: 3. Number of dynamic properties that were added: 132.

After reviewing the logs carefully (and posting this as a question in the MOS Oracle XE forum – https://community.oracle.com/thread/3892946) I eventually narrowed the issue down to a query that EM13c runs against DBA_REGISTRY_HISTORY in a target database when added. For database versions greater than 11.2 but less than 12.1.0.2, EM13c assumes that DBA_REGISTRY_HISTORY contains a BUNDLE_SERIES column.  This column does not exist in Oracle XE 11gR2, which reports a version string of 11.2.0.2.

This bug should eventually get a fix as EM13c gets patched, but in the meantime if you need to monitor an Oracle XE target with EM13c, the following workaround took care of the problem for me: create a new DBA_REGISTRY_HISTORY table containing a BUNDLE_SERIES column in your monitoring user’s schema in XE.  So, as user DBSNMP on XE, I ran:

SQL> create table dba_registry_history (ACTION_TIME TIMESTAMP(6), ACTION VARCHAR2(30), NAMESPACE VARCHAR2(30), VERSION VARCHAR2(30), ID NUMBER, BUNDLE_SERIES VARCHAR2(30), COMMENTS VARCHAR2(255));

Since one cannot patch XE, the real DBA_REGISTRY_HISTORY view has no rows and so you do not need to populate any data into this new table.

After adding the table, force a recalculation of dynamic properties by running the following against the EM13c management agent on the XE server:

$ emctl reload agent dynamicproperties SID.domain.com:oracle_database
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD recompute dynprops completed successfully

Once that completed successfully my XE target started to show the correct status in EM13c and I can submit jobs against the target.  All fixed.  I recommend deleting the DBSNMP.DBA_REGISTRY_HISTORY table once the bug gets fixed in OEM.

[EDIT 20160216: Oracle has documented this issue in MOS note EM13c: Database Target Status Shows “Dynamic Category property error” In 13c Cloud Control (Doc ID 2105001.1) and in bug 22592461 DATABASE TARGET STATUS SHOWS “DYNAMIC CATEGORY PROPERTY ERROR” IN 13C CONSOLE. Users on supported databases (e.g., not Oracle XE) should follow the resolution steps in that document instead to correct the real error.]

EM12c opatchauto, SHA256, and you

This post serves to document an issue I encountered after replacing expired SSL/TLS certificates on the server I use for Oracle Enterprise Manager 12c. To put it simply, using opatchauto to apply EM12c PSUs does not work if your WebLogic adminserver has a certificate installed that uses the SHA256 hashing algorithm.

[UPDATED 20151012: Please see this comment and this comment below, from Adam Robinson, who has provided a workaround that may work for you involving editing the opatchauto script to enable JSSE. As always, please consider workarounds requiring you to edit files as unsupported and at your own risk, but I would consider this fix superior to reverting back to the demo certificate every time you need to patch. You will need to repeat this fix every time you update OPatch in your OMS_HOME, though. Adam’s workaround does succeed in my environment.]

Error message

Expect to see the following error when running “opatchauto apply -analyze” or “opatchauto apply” against an installation with an SHA256-hashed certificate on the WLS adminserver:

oracle@omshost:/oracle/stage/21603255> opatchauto apply -analyze -property_file ~/property_file 
OPatch Automation Tool
Copyright (c) 2014, Oracle Corporation.  All rights reserved.


OPatchauto version : 11.1.0.12.3
OUI version        : 11.1.0.12.0
Running from       : /oracle/oem/Middleware12cR4/oms
Log file location  : /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatch/opatch2015-09-11_10-57-19AM_1.log

OPatchauto log file: /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatchauto/21603255/opatch_oms_2015-09-11_10-57-22AM_analyze.log



OPatchauto failed to establish JMX connection to weblogic server. This could be because of one (or) more of the following reasons:
1. Weblogic admin server URL that manages OMS application may not be right.
2. Weblogic admin server credentials (username, password) may not be right.
3. Virtual host configuration. If OMS, weblogic server are on virtual host configuration, Please make sure to add OPatchAuto.OMS_DISABLE_HOST_CHECK=true to command line and run again. (example: /oracle/oem/Middleware12cR4/oms/OPatch/opatchauto apply -analyze -property_file /home/oracle/property_file -invPtrLoc /oracle/oem/Middleware12cR4/oms/oraInst.loc  OPatchAuto.OMS_DISABLE_HOST_CHECK=true)

Please check above conditions and if error(s) still persist, Please contact Oracle support.


[ Error during Get weblogic Admin Server information Phase]. Detail: OPatchauto was not able to find right interview inputs.
OPatchauto failed: 
OPatchauto failed to establish JMX connection to weblogic server. This could be because of one (or) more of the following reasons:
1. Weblogic admin server URL that manages OMS application may not be right.
2. Weblogic admin server credentials (username, password) may not be right.
3. Virtual host configuration. If OMS, weblogic server are on virtual host configuration, Please make sure to add OPatchAuto.OMS_DISABLE_HOST_CHECK=true to command line and run again. (example: /oracle/oem/Middleware12cR4/oms/OPatch/opatchauto apply -analyze -property_file /home/oracle/property_file -invPtrLoc /oracle/oem/Middleware12cR4/oms/oraInst.loc  OPatchAuto.OMS_DISABLE_HOST_CHECK=true)

Please check above conditions and if error(s) still persist, Please contact Oracle support.

Log file location: /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatchauto/21603255/opatch_oms_2015-09-11_10-57-22AM_analyze.log

Recommended actions: Please correct the interview inputs and run opatchauto again.

OPatchauto failed with error code 231

Confirmation of the issue

To confirm this issue in your environment after receiving the preceding error message, check the hashing algorithm used on your adminserver certificate. I prefer to use the openssl commandline tool for this. If you don’t know the port used for your adminserver, you can retrieve it from the $EM_INSTANCE_BASE/em/EMGC_OMS1/emgc.properties file under AS_HTTPS_PORT. If your certificate does not show the usage of SHA256 (or another hash algorithm from the SHA-2 family) as in my example below, you may have a different problem.

oracle@omshost:~> openssl s_client -prexit -connect omshost.domain.com:7103 /dev/null | openssl x509 -text -in /dev/stdin | grep "Signature Algorithm" 2> /dev/null
        Signature Algorithm: sha256WithRSAEncryption
    Signature Algorithm: sha256WithRSAEncryption

Workaround

To work around this issue, you need to (temporarily!) replace the certificate on your WLS adminserver. Now, whenever I need to apply a PSU release, I resecure WLS using the default demonstration certificate, apply the PSU, then replace my original certificate.

oracle@omshost:/oracle/stage/21603255> emctl secure wls -use_demo_cert
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Securing WLS... Started.
Enter Enterprise Manager Root (SYSMAN) Password :
Securing WLS... Successful
Restart OMS using 'emctl stop oms -all' and 'emctl start oms'
oracle@omshost:/oracle/stage/21603255> emctl stop oms -all ; sleep 5 ; emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
Stopping BI Publisher Server...
BI Publisher Server Successfully Stopped
AdminServer Successfully Stopped
BI Publisher Server is Down
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
Starting BI Publisher Server ...
BI Publisher Server Successfully Started
BI Publisher Server is Up

[install the PSU according to the README instructions, including any post-installation steps]

oracle@omshost:/oracle/stage/21603255> emctl secure wls -wallet /oracle/oem/oemwallet
Oracle Enterprise Manager Cloud Control 12c Release 4  
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Securing WLS... Started.
Enter Enterprise Manager Root (SYSMAN) Password : 
Securing WLS... Successful
Restart OMS using 'emctl stop oms -all' and 'emctl start oms'
If there are multiple OMSs in this environment, perform this configuration on all of them.
oracle@omshost:/oracle/stage/21603255> emctl stop oms -all ; sleep 5 ; emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
Stopping BI Publisher Server...
BI Publisher Server Successfully Stopped
AdminServer Successfully Stopped
BI Publisher Server is Down
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
Starting BI Publisher Server ...
BI Publisher Server Successfully Started
BI Publisher Server is Up

I have not noticed any other EM12c issues using SHA256-hashed certificates. With this workaround, you can both continue to use quality certificates and keep your OMS patched.

EM12c OHS, LOW strength ciphers, custom certificates, and patch 19948000 weirdness

This post documents an unusual issue I encountered with the Oracle HTTP Server (OHS) installation in my Oracle Enterprise Manager 12c R4 (12.1.0.4) environment after following MOS note 1984662.1 and applying patch 19948000 (CPUJAN2015) to my OHS home.  It also contains a workaround I found that you should consider UNSUPPORTED, UNOFFICIAL, and NOT RECOMMENDED, only for use if absolutely necessary to meet auditor requirements.  If you do not have to follow the steps I describe below, I suggest waiting for new patches and further guidance from Oracle Support. If this change breaks your system and eats all the food in the break room refrigerator, I warned you not to do it.

Like other security-conscious EM12c admins, I want to keep my installation secure, and so I watch closely when security patches become available for EM12c or its various components. Thus, when I noticed patch 19948000’s availability for OHS, which disables SSLv3, I installed it on my system, and confirmed through testing that OHS no longer permitted SSLv3 connections (test for yourself with: openssl s_client -connect host.domain.com:port -ssl3, or try my EM12c SSL security checkup script that I have blogged about previously).

As I proceeded with further hardening of my EM12c system, specifically an attempt to disable LOW and MEDIUM strength cipher suite usage as per MOS note 1477287.1, I noticed that after following the directions provided, all of my EM12c endpoints correctly rejected LOW and MEDIUM strength ciphers, with one exception.  The OMS HTTPS upload port, inexplicably, continued to permit LOW strength connections. It refused MEDIUM strength ciphers, but had no problem accepting a LOW strength DES-CBC-SHA connection over TLSv1:

$ openssl s_client -connect omshost.domain.com:4902 -cipher LOW
[...]
SSL handshake has read 4109 bytes and written 385 bytes
---
New, TLSv1/SSLv3, Cipher is DES-CBC-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DES-CBC-SHA
    Session-ID: 37BF30668DCAD2CC5D0BAC4142CC1FA1
    Session-ID-ctx:
    Master-Key: [redacted]
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1429290250
    Timeout   : 300 (sec)
---

This confused me greatly, as I had edited all configuration files as instructed, and none of my other OHS listen ports accepted this LOW strength cipher connection.  I spent quite a bit of time trying to diagnose and resolve the issue with no luck, until I eventually stumbled upon an odd fix.  If I remove or comment out the “IfDefine SSL” directives from my $GC_INSTANCE_HOME/WebTierIH1/config/OHS/ohs1/httpd_em.conf file, then suddenly OHS would refuse LOW strength cipher connections on this port, with no apparent ill effect on the other listening ports.

$ openssl s_client -connect omshost.domain.com:4902 -cipher LOW 
CONNECTED(00000003)
2282780:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:769:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 67 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
---

I have noted these IfDefine SSL directives with “HERE” in the excerpt below from my httpd_em.conf file.

##
## CAUTION: Edit only the .template version of this file!
##
##     The command
##         emctl secure [lock|unlock]
##     will replace httpd_em.conf (discarding your changes) 
##     using the httpd_em.conf.template file.
##
## This file contains virtual hosts and other directives
## required for the "Enterprise Manager Central Console"
## to function correctly.
##

#UseWebCacheIp On

<IfDefine SSL>      #### HERE
    Listen 4902
    <VirtualHost *:4902>
        <Location /empbs/upload>
            Order allow,deny
            Allow from all
        </Location>
        <Location /empbs/jobrecv>
            Order allow,deny
            Allow from all
        </Location>
        <Location /em>
            Order allow,deny
            Allow from all
        </Location>
        <Location /agent_download>
            Order allow,deny
            Allow from all
        </Location>
        <Location /xmlpserver>
            Order allow,deny
            Allow from all
        </Location>

        #DocumentRoot &ORACLE_HOME&/Apache/Apache/htdocs
        ServerName omshost.domain.com
        #Port 4902
        Timeout 900

        LogFormat "%h %l %u %t \"%r\" %>s %b [ecid: %{ECID-Context}i] [User-Agent: %{User-Agent}i]" common
        SetEnvIf Request_URI "\.(bmp|jpg|png|gif|css|js$)" no-log
        SetEnvIf Request_URI "/em/dynamicImage/*"  no-log
        CustomLog "|${ORACLE_HOME}/ohs/bin/odl_rotatelogs /oracle/oem/gc_inst1/WebTierIH1/diagnostics/logs/OHS/ohs1/em_upload_https_access_log 10M 100M" common env=!no-log

        ErrorLog "|${ORACLE_HOME}/ohs/bin/odl_rotatelogs /oracle/oem/gc_inst1/WebTierIH1/diagnostics/logs/OHS/ohs1/em_upload_https_error_log 10M 100M"
        SSLEngine on
        SSLCipherSuite HIGH
        SSLWallet file:/oracle/oem/gc_inst1/WebTierIH1/config/OHS/ohs1/keystores/upload
        SSLProtocol TLSv1

        <Files ~ "\.(cgi|shtml)$">
            SSLOptions +StdEnvVars
        </Files>
        #<Directory &ORACLE_HOME&/Apache/Apache/cgi-bin>
        #    SSLOptions +StdEnvVars
        #</Directory>
        SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown
    </VirtualHost>
</IfDefine>     #### HERE
[remainder of file removed]

If I leave the IfDefine SSL statements in there, my OMS upload port accepts the weak DES-CBC-SHA cipher along with HIGH strength ciphers.  If I remove the IfDefine SSL, my OMS upload port refuses DES-CBC-SHA along with all other LOW/MEDIUM strength ciphers.

This makes no sense, given what I know of OHS and Apache-like products and the way that the handle the SSLCipherSuite configuration directive.

I raised this issue on Twitter and heard back from Andrew Bulloch at Oracle, who graciously spent quite a bit of time attempting to reproduce the issue on his side and working with me to identify the situations in which this behavior occurs.  After much testing, it appears that this behavior only occurs in the following situation:

1. The OHS installed with EM12c R4 has patch 19948000 installed, AND
2. The administrator has installed a third party SSL certificate, replacing the demo certificate used by default, AND
3. The OHS httpd_em.conf contains the “IfDefine SSL” directive.

If I remove my custom certificate, returning the OMS to the demo certificate, the issue disappears, then returns if I reinstall the custom certificate.

If I remove patch 19948000, the issue disappears, and does not return whether I use a custom certificate or a demo certificate.

If I remove the IfDefine SSL directive, the issue disappears, and does not return whether I use a custom certificate, a demo certificate, or whether or not I have patch 19948000 installed.

I attempted to replicate this behavior with an SSL certificate that did not come from a true certificate authority, by using OpenSSL to create a CA, create a cert, sign it, then install it into OHS per the documentation in MOS note 1399293.1, but I could not reproduce it, possibly due to the fact that I used a certificate signed directly by a root CA (as with the demo certificate) instead of a certificate signed by an intermediate chain certificate signed by a root CA, as with the paid-for commercial certificate that revealed the issue. I have not had a chance to test that configuration.

Unfortunately, removing patch 19948000 means that OHS cannot refuse SSLv3 connections, and removing the custom certificate reverts the system back to the demo certificate that I do not wish to use, both of which will represent audit findings in regulated sites.

Due to this issue, I have edited my EM12c security checkup script to remove my recommendation to install patch 19948000, although I still have it installed.  For security reasons, I will leave my system in the workaround state I have described here, as I want SSLv3 disabled, and I want LOW strength cipher suites disabled, and I want to use a custom SSL certificate, but I accept the risk that I may have to undo this setup at any time to receive support or to successfully apply later patches.  You will have to make your own decisions based on your site’s audit requirements and the availability of personnel to validate your configuration and handle future patching.

I would be very interested if anyone else reading this has encountered this issue, as I do not know if my installation somehow uniquely surfaces this behavior or if the certificate vendor that we used has some strange settings on their certificates that cause confusion for OHS.

EM12c R4 SSL Security Checkup Script

[Final update: I have migrated to EM13c and no longer have an EM12c installation available on which to further develop this script.  Please stay tuned for something similar for EM13c once patches become available.]

[LATEST SCRIPT UPDATE: 20151204, VERSION 1.11, covers 20151130 patch release]

Download the script here.

With all the recent news on companies getting hacked and attacks on encryption techniques, you need to act proactively to secure your Oracle Enterprise Manager Cloud Control 12c environment. Do not wait for your employer’s auditor to come around and send you a report of all the flaws in your system.

To put it in very simple terms, if you do not do the following across EVERY EM12c component, you should consider your setup vulnerable:

• Disable SSLv2 and SSLv3
• Enable TLSv1
• Disable weak ciphersuites such as those using the MD5 or RC4 algorithms, or those previously designed for export outside the USA back in the 1990s, or those that do not use enough key bits for encryption.
• Eliminate the use of self-signed and demonstration certificates.
• Stay current on EM12c base releases (currently EM12c R5 but I have not yet upgraded)
• Stay current on PSU updates to EM12c (PSU5 as of October 2015)
• Stay current on monthly system patch bundles
• Stay current on quarterly critical patch update alerts for all EM12c components – note that you have to pay attention to, for example, Oracle HTTP Server (OHS) critical patch updates, as EM12c distributes and relies on OHS. See MOS note 1664074.1 for a good, but incomplete list of patches needed.
• Stay current on repository database patch set updates
• Stay current on EM12c Java versions [EDIT: 20150415: Added Java check to script] [EDIT: 20150818: Java 1.6_101 caused the Node Manager to fail to start on my system.  Therefore I have kept the Java version check at 1.6_95.]
  

Yes, this takes a lot of work.  Yes, the documentation sometimes leaves the process as clear as mud.  Yes, you can contact Oracle support for assistance.

Yes, you do need to deal with EVERY endpoint for the SSL configuration.  That includes:

• OMS console
• OMS upload port
• OMS console proxy port
• Management agents
• EM Node Manager
• WebLogic Server administration console
• OHS administration port
• OPMN port
• BI Publisher

In the meantime, though, you need to have a good idea of where your system has flaws so that you know where to spend your time fixing it. To help with this, I have created a script that will examine your EM12c environment, find all the ports in use, check for SSLv2, SSLv3, and TLSv1, validate the cipher suites in use, check to make sure you have current patches installed, check for the usage of self-signed certificates on SSL/TLS endpoints, and check for current Java JDK versions in EM12c components. [EDIT: 20150311: Added self-signed certificate check]. [EDIT: 20150313: Added patch check for repository databases on same host as OMS server. I have only tested this on an 11.2.0.4 repository, but I believe it will work for the 12.1.0.2 repository just recently re-certified. If it fails for you please let me know.] [EDIT: 20150415: Added check for Java JDK versions.] [EDIT: 20150630: Added check for SSL_VERSION and SSL_CIPHER_SUITES parameters in repository database sqlnet.ora and listener.ora.]

This script does not require any arguments or configuration. I have tested it ONLY on EM12c R4 and on Linux x86-64 and only on single-host OMS environments.  To run this script, copy it from the end of this post (or from the pastebin link above, and execute it as the Oracle software owner on your OMS host, with your environment fully up and running. [EDIT: 20150311: Updated script incorporating feedback from Dave Corsar and opa tropa to support Solaris and AIX.]

The script will not make any changes to your system.  Mostly it crawls your configuration files to identify ports, then tests them with the openssl s_client command and various command line arguments to identify protocol and cipher suite usage, and whether or not it can find self-signed certificates.  At the end it runs OPatch checks for current needed security and functionality patches.

As of the version 1.1 release, I will mark newly checked patches with “*NEW*” in the script output and updated patches with “*UPDATED*”. For example, when a new PSU patch comes out, I will mark it as an update, but I will mark new (or previously not checked) patches as new. [EDIT: 20150415: This paragraph added.]

Example output from my fully patched and secured system [EDIT: 20150311: Unfortunately I still have self-signed certificates for OPMN and the OHS administration port, so my sample output now includes some failed checks]:

Performing EM12cR4 security checkup version 1.11 on omshost.domain.com at Fri Dec  4 14:17:40 EST 2015.

Using port definitions from configuration files 
	/etc/oragchomelist
	/oracle/oem/gc_inst1/em/EMGC_OMS1/emgc.properties
	/oracle/oem/gc_inst1/em/EMGC_OMS1/embip.properties
	/oracle/oem/gc_inst1/WebTierIH1/config/OPMN/opmn/ports.prop
	/oracle/oem/gc_inst1/WebTierIH1/config/OHS/ohs1/admin.conf

	Agent port found at omshost.domain.com:3872
	BIPublisher port found at omshost.domain.com:9702
	NodeManager port found at omshost.domain.com:7404
	OHSadmin port found at omshost.domain.com:9999
	OMSconsole port found at omshost.domain.com:7803
	OMSproxy port found at omshost.domain.com:7302
	OMSupload port found at omshost.domain.com:4902
	OPMN port found at omshost.domain.com:6701
	WLSadmin found at omshost.domain.com:7103

	Repository DB version=11.2.0.4.0 SID=emrep host=omshost.domain.com
	Repository DB on OMS server, will check patches/parameters in /oracle/oem/product/11.2.0/dbhome_2

(1) Checking SSL/TLS configuration (see notes 1602983.1, 1477287.1, 1905314.1)

	(1a) Forbid SSLv2 connections
	Confirming ssl2 disabled for Agent at omshost.domain.com:3872... OK
	Confirming ssl2 disabled for BIPublisher at omshost.domain.com:9702... OK
	Confirming ssl2 disabled for NodeManager at omshost.domain.com:7404... OK
	Confirming ssl2 disabled for OHSadmin at omshost.domain.com:9999... OK
	Confirming ssl2 disabled for OMSconsole at omshost.domain.com:7803... OK
	Confirming ssl2 disabled for OMSproxy at omshost.domain.com:7302... OK
	Confirming ssl2 disabled for OMSupload at omshost.domain.com:4902... OK
	Confirming ssl2 disabled for OPMN at omshost.domain.com:6701... OK
	Confirming ssl2 disabled for WLSadmin at omshost.domain.com:7103... OK

	(1b) Forbid SSLv3 connections
	Confirming ssl3 disabled for Agent at omshost.domain.com:3872... OK
	Confirming ssl3 disabled for BIPublisher at omshost.domain.com:9702... OK
	Confirming ssl3 disabled for NodeManager at omshost.domain.com:7404... OK
	Confirming ssl3 disabled for OHSadmin at omshost.domain.com:9999... OK
	Confirming ssl3 disabled for OMSconsole at omshost.domain.com:7803... OK
	Confirming ssl3 disabled for OMSproxy at omshost.domain.com:7302... OK
	Confirming ssl3 disabled for OMSupload at omshost.domain.com:4902... OK
	Confirming ssl3 disabled for OPMN at omshost.domain.com:6701... OK
	Confirming ssl3 disabled for WLSadmin at omshost.domain.com:7103... OK

	(1c) Permit TLSv1 connections
	Confirming tls1 available for Agent at omshost.domain.com:3872... OK
	Confirming tls1 available for BIPublisher at omshost.domain.com:9702... OK
	Confirming tls1 available for NodeManager at omshost.domain.com:7404... OK
	Confirming tls1 available for OHSadmin at omshost.domain.com:9999... OK
	Confirming tls1 available for OMSconsole at omshost.domain.com:7803... OK
	Confirming tls1 available for OMSproxy at omshost.domain.com:7302... OK
	Confirming tls1 available for OMSupload at omshost.domain.com:4902... OK
	Confirming tls1 available for OPMN at omshost.domain.com:6701... OK
	Confirming tls1 available for WLSadmin at omshost.domain.com:7103... OK

(2) Checking supported ciphers at SSL/TLS endpoints (see notes 1477287.1, 1905314.1, 1067411.1)
	Checking LOW strength ciphers on Agent (omshost.domain.com:3872)...	OK
	Checking MEDIUM strength ciphers on Agent (omshost.domain.com:3872)...	OK
	Checking HIGH strength ciphers on Agent (omshost.domain.com:3872)...	OK

	Checking LOW strength ciphers on BIPublisher (omshost.domain.com:9702)...	OK
	Checking MEDIUM strength ciphers on BIPublisher (omshost.domain.com:9702)...	OK
	Checking HIGH strength ciphers on BIPublisher (omshost.domain.com:9702)...	OK

	Checking LOW strength ciphers on NodeManager (omshost.domain.com:7404)...	OK
	Checking MEDIUM strength ciphers on NodeManager (omshost.domain.com:7404)...	OK
	Checking HIGH strength ciphers on NodeManager (omshost.domain.com:7404)...	OK

	Checking LOW strength ciphers on OHSadmin (omshost.domain.com:9999)...	OK
	Checking MEDIUM strength ciphers on OHSadmin (omshost.domain.com:9999)...	OK
	Checking HIGH strength ciphers on OHSadmin (omshost.domain.com:9999)...	OK

	Checking LOW strength ciphers on OMSconsole (omshost.domain.com:7803)...	OK
	Checking MEDIUM strength ciphers on OMSconsole (omshost.domain.com:7803)...	OK
	Checking HIGH strength ciphers on OMSconsole (omshost.domain.com:7803)...	OK

	Checking LOW strength ciphers on OMSproxy (omshost.domain.com:7302)...	OK
	Checking MEDIUM strength ciphers on OMSproxy (omshost.domain.com:7302)...	OK
	Checking HIGH strength ciphers on OMSproxy (omshost.domain.com:7302)...	OK

	Checking LOW strength ciphers on OMSupload (omshost.domain.com:4902)...	OK
	Checking MEDIUM strength ciphers on OMSupload (omshost.domain.com:4902)...	OK
	Checking HIGH strength ciphers on OMSupload (omshost.domain.com:4902)...	OK

	Checking LOW strength ciphers on OPMN (omshost.domain.com:6701)...	OK
	Checking MEDIUM strength ciphers on OPMN (omshost.domain.com:6701)...	OK
	Checking HIGH strength ciphers on OPMN (omshost.domain.com:6701)...	OK

	Checking LOW strength ciphers on WLSadmin (omshost.domain.com:7103)...	OK
	Checking MEDIUM strength ciphers on WLSadmin (omshost.domain.com:7103)...	OK
	Checking HIGH strength ciphers on WLSadmin (omshost.domain.com:7103)...	OK


(3) Checking self-signed and demonstration certificates at SSL/TLS endpoints (see notes 1367988.1, 1399293.1, 1593183.1, 1527874.1, 123033.1, 1937457.1)
	Checking certificate at Agent (omshost.domain.com:3872)... OK
	Checking certificate at Agent (omshost.domain.com:3872)... OK
	Checking certificate at BIPublisher (omshost.domain.com:9702)... OK
	Checking certificate at BIPublisher (omshost.domain.com:9702)... OK
	Checking certificate at NodeManager (omshost.domain.com:7404)... OK
	Checking certificate at NodeManager (omshost.domain.com:7404)... OK
	Checking certificate at OHSadmin (omshost.domain.com:9999)... FAILED - Found self-signed certificate
	Checking certificate at OHSadmin (omshost.domain.com:9999)... OK
	Checking certificate at OMSconsole (omshost.domain.com:7803)... OK
	Checking certificate at OMSconsole (omshost.domain.com:7803)... OK
	Checking certificate at OMSproxy (omshost.domain.com:7302)... OK
	Checking certificate at OMSproxy (omshost.domain.com:7302)... OK
	Checking certificate at OMSupload (omshost.domain.com:4902)... OK
	Checking certificate at OMSupload (omshost.domain.com:4902)... OK
	Checking certificate at OPMN (omshost.domain.com:6701)... FAILED - Found self-signed certificate
	Checking certificate at OPMN (omshost.domain.com:6701)... OK
	Checking certificate at WLSadmin (omshost.domain.com:7103)... OK
	Checking certificate at WLSadmin (omshost.domain.com:7103)... OK

(4) Checking EM12c Oracle home patch levels against 30 Nov 2015 baseline (see notes 1664074.1, 1900943.1, 822485.1, 1470197.1, 1967243.1)

	(4a) OMS (/oracle/oem/Middleware12cR4/oms) ENTERPRISE MANAGER BASE PLATFORM - OMS 12.1.0.4.5 PSU Patch (21462217)... OK
Patch 21462217 : applied on Tue Oct 20 12:13:32 EDT 2015 19055251, 19586898, 20260177, 19323634, 21462217, 19941819, 18725891

	(4a) OMS HOME (/oracle/oem/agent12c/core/12.1.0.4.0) JDBC Merge Patch (18502187)... OK
Patch 18502187 : applied on Thu Oct 22 10:29:36 EDT 2015

	(4b) BI Publisher (/oracle/oem/Middleware12cR4/Oracle_BI1) CPUJAN2015 Patch (19822893)... OK
19822893 19822893 Patch 19822893 : applied on Wed Feb 25 09:16:21 EST 2015

	(4b) BI Publisher (/oracle/oem/Middleware12cR4/Oracle_BI1) Merge Patch (20444447)... OK
Patch 20444447 : applied on Wed Feb 25 09:21:03 EST 2015

	(4c) AS Common (/oracle/oem/Middleware12cR4/oracle_common) CVE-2015-0426 Oracle Help Patch (20075252)... OK
Patch 20075252 : applied on Thu Jan 22 14:39:21 EST 2015

	(4c) AS Common (/oracle/oem/Middleware12cR4/oracle_common) WEBCENTER PORTAL BUNDLE PATCH 11.1.1.7.1 (16761779)... OK
Patch 16761779 : applied on Wed Apr 15 12:18:20 EDT 2015

	(4c) AS Common (/oracle/oem/Middleware12cR4/oracle_common) CVE-2015-4742 MERGE REQUEST ON TOP OF 11.1.1.7.1 FOR BUGS 20747356 18274008 (21068288)... OK
Patch 21068288 : applied on Thu Sep 17 09:52:53 EDT 2015

	(4d) WebLogic Server (/oracle/oem/Middleware12cR4/wlserver_10.3) 10.3.6.0.12 EJUW Patch (20780171)... 	OK
CR(s)..................... 20780171 Jar....................... BUG20780171_1036012.jar Destination............... $WLS_INSTALL_DIR$/bugsfixed/20780171-WLS-10.3.6.0.12_PSU_WebServices-ClientSide-Configuration-README.txt

	(4d) WebLogic Server (/oracle/oem/Middleware12cR4/wlserver_10.3) SU Patch [GDFA]: WEBLOGIC.STORE.PERSISTENTSTOREEXCEPTION: [STORE:280040] OCCURS EASILEY (16420963)... 	OK
CR(s)..................... 16420963 Jar....................... BUG16420963_1036.jar

	(4e) WebTier (/oracle/oem/Middleware12cR4/Oracle_WT) OHS SECURITY PATCH UPDATE 11.1.1.7.0 CPUOCT2015 Patch (21640624)... OK
Patch 21640624 : applied on Mon Oct 26 13:59:17 EDT 2015

	(4e) WebTier (/oracle/oem/Middleware12cR4/Oracle_WT) CVE-2014-4212 OPMN Patch (19345576)... OK
Patch 19345576 : applied on Thu Jan 22 13:02:25 EST 2015

	(4e) WebTier (/oracle/oem/Middleware12cR4/Oracle_WT) CVE 2015-2658 MERGE REQUEST ON TOP OF 11.1.1.7.0 FOR BUGS 16370190 20310323 20715657 (20807683)... OK
Patch 20807683 : applied on Wed Jul 15 12:22:04 EDT 2015

	(4e) WebTier (/oracle/oem/Middleware12cR4/Oracle_WT) CVE-2013-0169,CVE-2011-3389 OSS SECURITY PATCH UPDATE 11.1.1.7.0 CPUOCT2013 (17337741)... OK
Patch 17337741 : applied on Wed Apr 15 10:36:26 EDT 2015

	(4e) WebTier (/oracle/oem/Middleware12cR4/Oracle_WT) WLSPLUGINS (OHS) SECURITY PATCH UPDATE 11.1.1.7.0 CPUJUL2014 (18423831)... OK
Patch 18423831 : applied on Wed Apr 15 12:45:02 EDT 2015

	(4f) *UPDATED* OMS (/oracle/oem/Middleware12cR4/oms) DB PLUGIN BUNDLE PATCH 12.1.0.7.10 (22062307)... OK
22062307;EM DB PLUGIN BUNDLE PATCH 12.1.0.7.10 21744966,21745018,21972104,22062375,22062307

	(4g) *UPDATED* OMS (/oracle/oem/Middleware12cR4/oms) FMW PLUGIN BUNDLE PATCH 12.1.0.7.10 (22062375)... OK
22062375;EM FMW PLUGIN BUNDLE PATCH 12.1.0.7.10 21744966,21745018,21972104,22062375,22062307

	(4h) OMS (/oracle/oem/Middleware12cR4/oms) MOS PLUGIN BUNDLE PATCH 12.1.0.6.8 (21745018)... OK
21745018;EM MOS PLUGIN BUNDLE PATCH 12.1.0.6.8 21744966,21745018,21972104,22062375,22062307

	(4i) *UPDATED* OMS (/oracle/oem/Middleware12cR4/oms) EXADATA PLUGIN BUNDLE PATCH 12.1.0.6.11 (21744966)... OK
21744966;EM EXADATA PLUGIN BUNDLE PATCH 12.1.0.6.11 21744966,21745018,21972104,22062375,22062307

	(4j) *UPDATED* OMS (/oracle/oem/Middleware12cR4/oms) CFW PLUGIN BUNDLE PATCH 12.1.0.2.4 (21972104)... OK
21972104;EM CFW Plugin Bundle Patch 12.1.0.2.4 21744966,21745018,21972104,22062375,22062307

	(4k) *UPDATED* OMS CHAINED AGENT HOME (/oracle/oem/agent12c/core/12.1.0.4.0) EM-AGENT BUNDLE PATCH 12.1.0.4.14 (21913823)... OK
Patch 21913823 : applied on Fri Dec 04 09:16:23 EST 2015 17438375, 18936726, 21913823, 20496804, 21325110, 20701411, 21565489

	(4k) OMS CHAINED AGENT HOME (/oracle/oem/agent12c/core/12.1.0.4.0) Merge Patch (18502187)... OK
Patch 18502187 : applied on Fri Apr 03 09:45:56 EDT 2015

	(4k) OMS CHAINED AGENT HOME (/oracle/oem/agent12c/core/12.1.0.4.0) JDBC Security Patch (18721761)... OK
Patch 18721761 : applied on Fri Apr 03 09:45:52 EDT 2015

	(4k) OMS CHAINED AGENT HOME (/oracle/oem/agent12c/core/12.1.0.4.0) CVE 2012-3137 EM Agent only: Instant Client Security Patch (20114054)... OK
Patch 20114054 : applied on Fri May 01 10:01:01 EDT 2015 20114054

	(4l) *UPDATED* OMS CHAINED AGENT DB PLUGIN (/oracle/oem/agent12c/core/12.1.0.4.0/../../plugins/oracle.sysman.db.agent.plugin_12.1.0.7.0) DB PLUGIN BUNDLE 12.1.0.7.10 AGENT-SIDE MONITORING (22140476)... OK
Patch 22140476 : applied on Fri Dec 04 11:54:20 EST 2015 15837598, 21907123, 21460951, 20765041, 20844888, 22140476, 21806804

	(4l) OMS CHAINED AGENT DB PLUGIN (/oracle/oem/agent12c/core/12.1.0.4.0/../../plugins/oracle.sysman.db.discovery.plugin_12.1.0.7.0) DB PLUGIN BUNDLE 12.1.0.7.4 AGENT-SIDE DISCOVERY (21065239)... OK
Patch 21065239 : applied on Thu Jun 04 11:15:02 EDT 2015 18413892, 21065239

	(4m) *UPDATED* OMS CHAINED AGENT FMW PLUGIN (/oracle/oem/agent12c/core/12.1.0.4.0/../../plugins/oracle.sysman.emas.agent.plugin_12.1.0.7.0) FMW PLUGIN BUNDLE 12.1.0.7.9 AGENT-SIDE MONITORING (21941290)... OK
Patch 21941290 : applied on Fri Dec 04 12:01:35 EST 2015 20644295, 21894243, 20677020, 21888856, 21527296, 21941290, 21415166

	(4m) OMS CHAINED AGENT FMW PLUGIN (/oracle/oem/agent12c/core/12.1.0.4.0/../../plugins/oracle.sysman.emas.discovery.plugin_12.1.0.7.0) FMW PLUGIN BUNDLE 12.1.0.7.7 AGENT-SIDE DISCOVERY (21611921)... OK
Patch 21611921 : applied on Tue Sep 01 13:34:27 EDT 2015 21611921, 20644315, 20677038, 21199835, 21229841, 21610843

	(4n) *UPDATED* OMS CHAINED AGENT BEACON PLUGIN (/oracle/oem/agent12c/core/12.1.0.4.0/../../plugins/oracle.sysman.beacon.agent.plugin_12.1.0.4.0) EM-BEACON BUNDLE PATCH 12.1.0.4.2 (21928148)... OK
Patch 21928148 : applied on Fri Dec 04 12:35:11 EST 2015 21928008, 21928148, 20466772, 20397739

	(4o) OMS CHAINED AGENT EM-OH BUNDLE PATCH 12.1.0.4.1 (20855134)... OK
Patch 20855134 : applied on Thu Apr 30 15:54:47 EDT 2015 15985793, 20855134

	(4p) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/11.2.0/dbhome_2) PSU 11.2.0.4.8 (OCT2015) (21352635)... OK
Patch 21352635 : applied on Thu Oct 22 09:39:55 EDT 2015 Patch description: "Database Patch Set Update : 11.2.0.4.8 (21352635)"

	(4p) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/11.2.0/dbhome_2) ORACLE JAVAVM COMPONENT 11.2.0.4.5 DATABASE PSU (OCT2015) (21555791)... OK
Patch 21555791 : applied on Thu Oct 22 09:41:22 EDT 2015

	(4q) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/11.2.0/dbhome_2) sqlnet.ora SSL_VERSION parameter (1545816.1)... OK
1.0

	(4q) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/11.2.0/dbhome_2) sqlnet.ora SSL_CIPHER_SUITES parameter (1545816.1)... OK
(SSL_RSA_WITH_AES128_CBC_SHA,SSL_RSA_WITH_AES256_CBC_SHA)

	(4q) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/11.2.0/dbhome_2) listener.ora SSL_VERSION parameter (1545816.1)... OK
1.0

	(4q) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/11.2.0/dbhome_2) listener.ora SSL_CIPHER_SUITES parameter (1545816.1)... OK
(SSL_RSA_WITH_AES128_CBC_SHA,SSL_RSA_WITH_AES256_CBC_SHA)


(5) Checking EM12c Java versions against baseline (see notes 1506916.1, 1492980.1)

	(5a) MW (/oracle/oem/Middleware12cR4/jdk16/jdk) Java version 1.6.0_95 (9553040)... 	OK
1.6.0_95

	(5b) WebTier (/oracle/oem/Middleware12cR4/Oracle_WT/jdk) Java version 1.6.0_95 (9553040)... 	OK
1.6.0_95

Failed test count: 2 - Review output

certcheck:OHSadmin @ omshost.domain.com:9999 found self-signed certificate
certcheck:OPMN @ omshost.domain.com:6701 found self-signed certificate

Visit https://pardydba.wordpress.com/2015/03/09/em12c-r4-ssl-security-checkup-script/ for the latest version.

Body of script:

#!/bin/bash
#
# This script should examine your EM12c R4 environment, identify the ports
# each component uses, and check for SSLv2/SSLv3 usage, as well as make
# sure that weak cipher suites get rejected.  It also contains a patch
# check currently comparing against the latest recommended patches
# and flags the use of self-signed certificates.  Further checks include
# EM12c Java JDK version.
#
# Added in v1.0:   Repository database patch check
# Added in v1.1:   EM12c Java JDK version check
# Change in v1.2:  Removed patch 19948000 recommendation for OHS.
# Change in v1.3:  Update for 30 Apr 2015 patches, add EM-OH plugin home
#                  restored GDFA/16420963 for WLS
#                  added 20114054 for Agent - only applicable for Linux x86-64
# Change in v1.4:  Add datestamp/hostname to output header
#		   Update for 31 May 2015 patches, add EM-DB-DISC plugin home
# Change in v1.5:  Add repo DB check for SSL_VERSION and SSL_CIPHER_SUITES
#                  Add VERBOSE_CHECKSEC variable:
#                   Set to 0 for quiet run.
#                   Set to 1 to see failed check summary after run.
#                   Set to 2 for failed check summary and patch details.
# Change in v1.6:  Add PSU4 for EM12cR4, complete VERBOSE_CHECKSEC work
#                  Add 14 July 2015 patches
# Change in v1.7:  Update for 31 Jul 2015 patches
# Change in v1.8:  Update for 31 Aug 2015 patches
# Change in v1.9:  Add 17714229 for OMS home
#                  Add 21068288 CVE-2015-4742 for oracle_common home
#                  Add check for usage of demonstration SSL certificates
# Change in v1.10: Update for 1 Oct 2015 patches, PSU5, CPUOCT2015
#		   Added 18502187 for OMS home
# Change in v1.11: Update for 30 Nov 2015 patches
#
# From: @BrianPardy on Twitter
#
# Known functional on Linux x86-64, Solaris, AIX.
#
# Run this script as the Oracle EM12c software owner, with your environment
# fully up and running.
#
# Thanks to Dave Corsar, who tested on Solaris and let me know the 
# changes needed to make an earlier version work on Solaris.
#
# Thanks to opa tropa who confirmed AIX functionality and noted the 
# use of GNU extensions to grep, which I have since removed.
# 
# Dedicated to our two Lhasa Apsos:
#   Lucy (6/13/1998 - 3/13/2015)
#   Ethel (6/13/1998 - 7/31/2015)
#
# 

SCRIPTNAME=`basename $0`
PATCHDATE="30 Nov 2015"
OMSHOST=`hostname -f`
VERSION="1.11"
FAIL_COUNT=0
FAIL_TESTS=""

RUN_DB_CHECK=0
VERBOSE_CHECKSEC=2

HOST_OS=`uname -s`
HOST_ARCH=`uname -m`

ORAGCHOMELIST="/etc/oragchomelist"
ORATAB="/etc/oratab"

if [[ ! -r $ORAGCHOMELIST ]]; then			# Solaris
	ORAGCHOMELIST="/var/opt/oracle/oragchomelist"
fi

if [[ ! -r $ORATAB ]]; then 				# Solaris
	ORATAB="/var/opt/oracle/oratab"
fi

if [[ -x "/usr/sfw/bin/gegrep" ]]; then
	GREP=/usr/sfw/bin/gegrep
else
	GREP=`which grep`
fi

OMS_HOME=`$GREP -i oms $ORAGCHOMELIST | xargs ls -d 2>/dev/null`

OPATCH="$OMS_HOME/OPatch/opatch"
OPATCHAUTO="$OMS_HOME/OPatch/opatchauto"
OMSORAINST="$OMS_HOME/oraInst.loc"
ORAINVENTORY=`head -n 1 $OMSORAINST | awk -F= '{print $2}'`

MW_HOME=`dirname $OMS_HOME`
BIP_HOME=`$GREP -vi REMOVED $ORAINVENTORY/ContentsXML/inventory.xml | $GREP "HOME NAME=\"Oracle_BI" | awk '{print $3}' | sed -e 's/LOC=\"//' | sed -e 's/"//'`
COMMON_HOME=`$GREP -vi REMOVED $ORAINVENTORY/ContentsXML/inventory.xml | $GREP "HOME NAME=\"common" | awk '{print $3}' | sed -e 's/LOC=\"//' | sed -e 's/"//'`
WEBTIER_HOME=`$GREP -vi REMOVED $ORAINVENTORY/ContentsXML/inventory.xml | $GREP "HOME NAME=\"webtier" | awk '{print $3}' | sed -e 's/LOC=\"//' | sed -e 's/"//'`
AGENT_HOME=`$GREP -vi REMOVED $ORAINVENTORY/ContentsXML/inventory.xml | $GREP "HOME NAME=\"agent12c" | awk '{print $3}' | sed -e 's/LOC=\"//' | sed -e 's/"//'`
AGENT_DB_PLUGIN_HOME="$AGENT_HOME/../../plugins/oracle.sysman.db.agent.plugin_12.1.0.7.0"
AGENT_DB_PLUGIN_DISC_HOME="$AGENT_HOME/../../plugins/oracle.sysman.db.discovery.plugin_12.1.0.7.0"
AGENT_FMW_PLUGIN_HOME="$AGENT_HOME/../../plugins/oracle.sysman.emas.agent.plugin_12.1.0.7.0"
AGENT_FMW_PLUGIN_DISC_HOME="$AGENT_HOME/../../plugins/oracle.sysman.emas.discovery.plugin_12.1.0.7.0"
AGENT_BEACON_PLUGIN_HOME="$AGENT_HOME/../../plugins/oracle.sysman.beacon.agent.plugin_12.1.0.4.0"
AGENT_OH_PLUGIN_HOME="$AGENT_HOME/../../plugins/oracle.sysman.oh.agent.plugin_12.1.0.4.0"

EM_INSTANCE_BASE=`$GREP GCDomain $MW_HOME/domain-registry.xml | sed -e 's/.*=//' | sed -e 's/\/user_projects.*$//' | sed -e 's/"//'`
WL_HOME=`$GREP wlserver $MW_HOME/domain-registry.xml | sed -e 's/.*=//' | sed -e 's/\/samples.*$//' | sed -e 's/"//' | uniq`

EMGC_PROPS="$EM_INSTANCE_BASE/em/EMGC_OMS1/emgc.properties"
EMBIP_PROPS="$EM_INSTANCE_BASE/em/EMGC_OMS1/embip.properties"
OPMN_PROPS="$EM_INSTANCE_BASE/WebTierIH1/config/OPMN/opmn/ports.prop"
OHS_ADMIN_CONF="$EM_INSTANCE_BASE/WebTierIH1/config/OHS/ohs1/admin.conf"

PORT_UPL=`$GREP EM_UPLOAD_HTTPS_PORT $EMGC_PROPS | awk -F= '{print $2}'`
PORT_OMS=`$GREP EM_CONSOLE_HTTPS_PORT $EMGC_PROPS | awk -F= '{print $2}'`
PORT_OMS_JAVA=`$GREP MS_HTTPS_PORT $EMGC_PROPS | awk -F= '{print $2}'`
PORT_NODEMANAGER=`$GREP EM_NODEMGR_PORT $EMGC_PROPS | awk -F= '{print $2}'`
PORT_BIP=`$GREP BIP_HTTPS_PORT $EMBIP_PROPS | awk -F= '{print $2}'`
PORT_ADMINSERVER=`$GREP AS_HTTPS_PORT $EMGC_PROPS | awk -F= '{print $2}'`
PORT_OPMN=`$GREP '/opmn/remote_port' $OPMN_PROPS | awk -F= '{print $2}'`
PORT_OHS_ADMIN=`$GREP Listen $OHS_ADMIN_CONF | awk '{print $2}'`
PORT_AGENT=`$AGENT_HOME/bin/emctl status agent | $GREP 'Agent URL' | sed -e 's/\/emd\/main\///' | sed -e 's/^.*://' | uniq`

REPOS_DB_CONNDESC=`$GREP EM_REPOS_CONNECTDESCRIPTOR $EMGC_PROPS | sed -e 's/EM_REPOS_CONNECTDESCRIPTOR=//' | sed -e 's/\\\\//g'`
REPOS_DB_HOST=`echo $REPOS_DB_CONNDESC | sed -e 's/^.*HOST=//' | sed -e 's/).*$//'`
REPOS_DB_SID=`echo $REPOS_DB_CONNDESC | sed -e 's/^.*SID=//' | sed -e 's/).*$//'`

if [[ "$REPOS_DB_HOST" == "$OMSHOST" ]]; then
	REPOS_DB_HOME=`$GREP "$REPOS_DB_SID:" $ORATAB | awk -F: '{print $2}'`
	REPOS_DB_VERSION=`$REPOS_DB_HOME/OPatch/opatch lsinventory -oh $REPOS_DB_HOME | $GREP 'Oracle Database' | awk '{print $4}'`

	if [[ "$REPOS_DB_VERSION" == "11.2.0.4.0" ]]; then
		RUN_DB_CHECK=1
	fi

	if [[ "$REPOS_DB_VERSION" == "12.1.0.2.0" ]]; then
		RUN_DB_CHECK=1
	fi

	if [[ "$RUN_DB_CHECK" -eq 0 ]]; then
		echo -e "\tSkipping local repository DB patch check, only 11.2.0.4 or 12.1.0.2 supported by this script for now"
	fi
fi


sslcheck () {
	OPENSSL_CHECK_COMPONENT=$1
	OPENSSL_CHECK_HOST=$2
	OPENSSL_CHECK_PORT=$3
	OPENSSL_CHECK_PROTO=$4

	OPENSSL_RETURN=`echo Q | openssl s_client -prexit -connect $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT -$OPENSSL_CHECK_PROTO 2>&1 | $GREP Cipher | $GREP -c 0000`
	
	

	if [[ $OPENSSL_CHECK_PROTO == "tls1" ]]; then
		echo -en "\tConfirming $OPENSSL_CHECK_PROTO available for $OPENSSL_CHECK_COMPONENT at $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT... "
		if [[ $OPENSSL_RETURN -eq "0" ]]; then
			echo OK
		else
			echo FAILED
			FAIL_COUNT=$((FAIL_COUNT+1))
			FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OPENSSL_CHECK_COMPONENT @ $OPENSSL_CHECK_HOST:${OPENSSL_CHECK_PORT}:$OPENSSL_CHECK_PROTO protocol connection failed"
		fi
	fi

	if [[ $OPENSSL_CHECK_PROTO == "ssl2" || $OPENSSL_CHECK_PROTO == "ssl3" ]]; then
		echo -en "\tConfirming $OPENSSL_CHECK_PROTO disabled for $OPENSSL_CHECK_COMPONENT at $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT... "
		if [[ $OPENSSL_RETURN -ne "0" ]]; then
			echo OK
		else
			echo FAILED
			FAIL_COUNT=$((FAIL_COUNT+1))
			FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OPENSSL_CHECK_COMPONENT @ $OPENSSL_CHECK_HOST:${OPENSSL_CHECK_PORT}:$OPENSSL_CHECK_PROTO protocol connection succeeded"
		fi
	fi
}

opatchcheck () {
	OPATCH_CHECK_COMPONENT=$1
	OPATCH_CHECK_OH=$2
	OPATCH_CHECK_PATCH=$3

	if [[ "$OPATCH_CHECK_COMPONENT" == "ReposDBHome" ]]; then
		OPATCH_RET=`$OPATCH_CHECK_OH/OPatch/opatch lsinv -oh $OPATCH_CHECK_OH | $GREP $OPATCH_CHECK_PATCH`
	else
		OPATCH_RET=`$OPATCH lsinv -oh $OPATCH_CHECK_OH | $GREP $OPATCH_CHECK_PATCH`
	fi

	if [[ -z "$OPATCH_RET" ]]; then
		echo FAILED
		FAIL_COUNT=$((FAIL_COUNT+1))
		FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OPATCH_CHECK_COMPONENT @ ${OPATCH_CHECK_OH}:Patch $OPATCH_CHECK_PATCH not found"
	else
		echo OK
	fi

	test $VERBOSE_CHECKSEC -ge 2 && echo $OPATCH_RET

}

opatchautocheck () {
	OPATCHAUTO_CHECK_COMPONENT=$1
	OPATCHAUTO_CHECK_OH=$2
	OPATCHAUTO_CHECK_PATCH=$3

	OPATCHAUTO_RET=`$OPATCHAUTO lspatches -oh $OPATCHAUTO_CHECK_OH | $GREP $OPATCHAUTO_CHECK_PATCH`

	if [[ -z "$OPATCHAUTO_RET" ]]; then
		echo FAILED
		FAIL_COUNT=$((FAIL_COUNT+1))
		FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OPATCHAUTO_CHECK_COMPONENT @ ${OPATCHAUTO_CHECK_OH}:Patch $OPATCHAUTO_CHECK_PATCH not found"
	else
		echo OK
	fi

	test $VERBOSE_CHECKSEC -ge 2 && echo $OPATCHAUTO_RET

}

certcheck () {
	CERTCHECK_CHECK_COMPONENT=$1
	CERTCHECK_CHECK_HOST=$2
	CERTCHECK_CHECK_PORT=$3

	echo -ne "\tChecking certificate at $CERTCHECK_CHECK_COMPONENT ($CERTCHECK_CHECK_HOST:$CERTCHECK_CHECK_PORT)... "

	OPENSSL_SELFSIGNED_COUNT=`echo Q | openssl s_client -prexit -connect $CERTCHECK_CHECK_HOST:$CERTCHECK_CHECK_PORT 2>&1 | $GREP -ci "self signed certificate"`

	if [[ $OPENSSL_SELFSIGNED_COUNT -eq "0" ]]; then
		echo OK
	else
		echo FAILED - Found self-signed certificate
		FAIL_COUNT=$((FAIL_COUNT+1))
		FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$CERTCHECK_CHECK_COMPONENT @ ${CERTCHECK_CHECK_HOST}:${CERTCHECK_CHECK_PORT} found self-signed certificate"
	fi
}

democertcheck () {
	DEMOCERTCHECK_CHECK_COMPONENT=$1
	DEMOCERTCHECK_CHECK_HOST=$2
	DEMOCERTCHECK_CHECK_PORT=$3

	echo -ne "\tChecking certificate at $DEMOCERTCHECK_CHECK_COMPONENT ($DEMOCERTCHECK_CHECK_HOST:$DEMOCERTCHECK_CHECK_PORT)... "

	OPENSSL_DEMO_COUNT=`echo Q | openssl s_client -prexit -connect $DEMOCERTCHECK_CHECK_HOST:$DEMOCERTCHECK_CHECK_PORT 2>&1 | $GREP -ci "issuer=/C=US/ST=MyState/L=MyTown/O=MyOrganization/OU=FOR TESTING ONLY/CN=CertGenCAB"`

	if [[ $OPENSSL_DEMO_COUNT -eq "0" ]]; then
		echo OK
	else
		echo FAILED - Found demonstration certificate
		FAIL_COUNT=$((FAIL_COUNT+1))
		FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$DEMOCERTCHECK_CHECK_COMPONENT @ ${DEMOCERTCHECK_CHECK_HOST}:${DEMOCERTCHECK_CHECK_PORT} found demonstration certificate"
	fi
}


ciphercheck () {
	OPENSSL_CHECK_COMPONENT=$1
	OPENSSL_CHECK_HOST=$2
	OPENSSL_CHECK_PORT=$3

	echo -ne "\tChecking LOW strength ciphers on $OPENSSL_CHECK_COMPONENT ($OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT)..."

	OPENSSL_LOW_RETURN=`echo Q | openssl s_client -prexit -connect $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT -tls1 -cipher LOW 2>&1 | $GREP Cipher | uniq | $GREP -c 0000`

	if [[ $OPENSSL_LOW_RETURN -eq "0" ]]; then
		echo -e "\tFAILED - PERMITS LOW STRENGTH CIPHER CONNECTIONS"
		FAIL_COUNT=$((FAIL_COUNT+1))
		FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OPENSSL_CHECK_COMPONENT @ $OPENSSL_CHECK_HOST:${OPENSSL_CHECK_PORT}:Permits LOW strength ciphers"
	else
		echo -e "\tOK"
	fi


	echo -ne "\tChecking MEDIUM strength ciphers on $OPENSSL_CHECK_COMPONENT ($OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT)..."

	OPENSSL_MEDIUM_RETURN=`echo Q | openssl s_client -prexit -connect $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT -tls1 -cipher MEDIUM 2>&1 | $GREP Cipher | uniq | $GREP -c 0000`

	if [[ $OPENSSL_MEDIUM_RETURN -eq "0" ]]; then
		echo -e "\tFAILED - PERMITS MEDIUM STRENGTH CIPHER CONNECTIONS"
		FAIL_COUNT=$((FAIL_COUNT+1))
		FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OPENSSL_CHECK_COMPONENT @ $OPENSSL_CHECK_HOST:${OPENSSL_CHECK_PORT}:Permits MEDIUM strength ciphers"
	else
		echo -e "\tOK"
	fi



	echo -ne "\tChecking HIGH strength ciphers on $OPENSSL_CHECK_COMPONENT ($OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT)..."

	OPENSSL_HIGH_RETURN=`echo Q | openssl s_client -prexit -connect $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT -tls1 -cipher HIGH 2>&1 | $GREP Cipher | uniq | $GREP -c 0000`

	if [[ $OPENSSL_HIGH_RETURN -eq "0" ]]; then
		echo -e "\tOK"
	else
		echo -e "\tFAILED - CANNOT CONNECT WITH HIGH STRENGTH CIPHER"
		FAIL_COUNT=$((FAIL_COUNT+1))
		FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OPENSSL_CHECK_COMPONENT @ $OPENSSL_CHECK_HOST:${OPENSSL_CHECK_PORT}:Rejects HIGH strength ciphers"
	fi
	echo
}

wlspatchcheck () {
	WLSDIR=$1
	WLSPATCH=$2

	WLSCHECK_RETURN=`( cd $MW_HOME/utils/bsu && $MW_HOME/utils/bsu/bsu.sh -report ) | $GREP $WLSPATCH`
	WLSCHECK_COUNT=`echo $WLSCHECK_RETURN | wc -l`

	if [[ $WLSCHECK_COUNT -ge "1" ]]; then
		echo -e "\tOK"
	else
		echo -e "\tFAILED - PATCH NOT FOUND"
		FAIL_COUNT=$((FAIL_COUNT+1))
		FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$WLSDIR:Patch $WLSPATCH not found"
	fi

	test $VERBOSE_CHECKSEC -ge 2 && echo $WLSCHECK_RETURN
	
}

javacheck () {
	WHICH_JAVA=$1
	JAVA_DIR=$2

	JAVACHECK_RETURN=`$JAVA_DIR/bin/java -version 2>&1 | $GREP version | awk '{print $3}' | sed -e 's/"//g'`

	if [[ "$JAVACHECK_RETURN" == "1.6.0_95" ]]; then
		echo -e "\tOK"
	else
		#echo -e "\tFAILED - Found version $JAVACHECK_RETURN"
		echo -e "\tFAILED"
		FAIL_COUNT=$((FAIL_COUNT+1))
		FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$WHICH_JAVA Java in ${JAVA_DIR}:Found incorrect version $JAVACHECK_RETURN"
	fi
	test $VERBOSE_CHECKSEC -ge 2 && echo $JAVACHECK_RETURN
}

paramcheck () {
	WHICH_PARAM=$1
	WHICH_ORACLE_HOME=$2
	WHICH_FILE=$3

	PARAMCHECK_RETURN=`$GREP $WHICH_PARAM $WHICH_ORACLE_HOME/network/admin/$WHICH_FILE | awk -F= '{print $2}' | sed -e 's/\s//g'`
	if [[ "$WHICH_PARAM" == "SSL_VERSION" ]]; then
		if [[ "$PARAMCHECK_RETURN" == "1.0" ]]; then
			echo -e "OK"
		else
			echo -e "FAILED - Found $WHICH_PARAM = $PARAMCHECK_RETURN"
			FAIL_COUNT=$((FAIL_COUNT+1))
			FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$WHICH_PARAM in $WHICH_FILE for home ${WHICH_ORACLE_HOME}:incorrect parameter value"
		fi
		test $VERBOSE_CHECKSEC -ge 2 && echo $PARAMCHECK_RETURN
	fi

	if [[ "$WHICH_PARAM" == "SSL_CIPHER_SUITES" ]]; then
		if [[ "$PARAMCHECK_RETURN" == "(SSL_RSA_WITH_AES128_CBC_SHA,SSL_RSA_WITH_AES256_CBC_SHA)" ]]; then
			echo -e "OK"
		else
			echo -e "FAILED - Found $WHICH_PARAM = $PARAMCHECK_RETURN"
			FAIL_COUNT=$((FAIL_COUNT+1))
			FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$WHICH_PARAM in $WHICH_FILE for home ${WHICH_ORACLE_HOME}:incorrect parameter value"
		fi
		test $VERBOSE_CHECKSEC -ge 2 && echo $PARAMCHECK_RETURN
	fi
}


### MAIN SCRIPT HERE


echo -e "Performing EM12cR4 security checkup version $VERSION on $OMSHOST at `date`.\n"

echo "Using port definitions from configuration files "
echo -e "\t/etc/oragchomelist"
echo -e "\t$EMGC_PROPS"
echo -e "\t$EMBIP_PROPS"
echo -e "\t$OPMN_PROPS"
echo -e "\t$OHS_ADMIN_CONF"
echo
echo -e "\tAgent port found at $OMSHOST:$PORT_AGENT"
echo -e "\tBIPublisher port found at $OMSHOST:$PORT_BIP"
echo -e "\tNodeManager port found at $OMSHOST:$PORT_NODEMANAGER"
echo -e "\tOHSadmin port found at $OMSHOST:$PORT_OHS_ADMIN"
echo -e "\tOMSconsole port found at $OMSHOST:$PORT_OMS"
echo -e "\tOMSproxy port found at $OMSHOST:$PORT_OMS_JAVA"
echo -e "\tOMSupload port found at $OMSHOST:$PORT_UPL"
echo -e "\tOPMN port found at $OMSHOST:$PORT_OPMN"
echo -e "\tWLSadmin found at $OMSHOST:$PORT_ADMINSERVER"
echo
echo -e "\tRepository DB version=$REPOS_DB_VERSION SID=$REPOS_DB_SID host=$REPOS_DB_HOST"

if [[ $RUN_DB_CHECK -eq "1" ]]; then
	echo -e "\tRepository DB on OMS server, will check patches/parameters in $REPOS_DB_HOME"
fi


echo -e "\n(1) Checking SSL/TLS configuration (see notes 1602983.1, 1477287.1, 1905314.1)"

echo -e "\n\t(1a) Forbid SSLv2 connections"
sslcheck Agent $OMSHOST $PORT_AGENT ssl2
sslcheck BIPublisher $OMSHOST $PORT_BIP ssl2
sslcheck NodeManager $OMSHOST $PORT_NODEMANAGER ssl2
sslcheck OHSadmin $OMSHOST $PORT_OHS_ADMIN ssl2
sslcheck OMSconsole $OMSHOST $PORT_OMS ssl2
sslcheck OMSproxy $OMSHOST $PORT_OMS_JAVA ssl2
sslcheck OMSupload $OMSHOST $PORT_UPL ssl2
sslcheck OPMN $OMSHOST $PORT_OPMN ssl2
sslcheck WLSadmin $OMSHOST $PORT_ADMINSERVER ssl2

echo -e "\n\t(1b) Forbid SSLv3 connections"
sslcheck Agent $OMSHOST $PORT_AGENT ssl3
sslcheck BIPublisher $OMSHOST $PORT_BIP ssl3
sslcheck NodeManager $OMSHOST $PORT_NODEMANAGER ssl3
sslcheck OHSadmin $OMSHOST $PORT_OHS_ADMIN ssl3
sslcheck OMSconsole $OMSHOST $PORT_OMS ssl3
sslcheck OMSproxy $OMSHOST $PORT_OMS_JAVA ssl3
sslcheck OMSupload $OMSHOST $PORT_UPL ssl3
sslcheck OPMN $OMSHOST $PORT_OPMN ssl3
sslcheck WLSadmin $OMSHOST $PORT_ADMINSERVER ssl3

echo -e "\n\t(1c) Permit TLSv1 connections"
sslcheck Agent $OMSHOST $PORT_AGENT tls1
sslcheck BIPublisher $OMSHOST $PORT_BIP tls1
sslcheck NodeManager $OMSHOST $PORT_NODEMANAGER tls1
sslcheck OHSadmin $OMSHOST $PORT_OHS_ADMIN tls1
sslcheck OMSconsole $OMSHOST $PORT_OMS tls1
sslcheck OMSproxy $OMSHOST $PORT_OMS_JAVA tls1
sslcheck OMSupload $OMSHOST $PORT_UPL tls1
sslcheck OPMN $OMSHOST $PORT_OPMN tls1
sslcheck WLSadmin $OMSHOST $PORT_ADMINSERVER tls1

echo -e "\n(2) Checking supported ciphers at SSL/TLS endpoints (see notes 1477287.1, 1905314.1, 1067411.1)"
ciphercheck Agent $OMSHOST $PORT_AGENT
ciphercheck BIPublisher $OMSHOST $PORT_BIP
ciphercheck NodeManager $OMSHOST $PORT_NODEMANAGER
ciphercheck OHSadmin $OMSHOST $PORT_OHS_ADMIN
ciphercheck OMSconsole $OMSHOST $PORT_OMS
ciphercheck OMSproxy $OMSHOST $PORT_OMS_JAVA
ciphercheck OMSupload $OMSHOST $PORT_UPL
ciphercheck OPMN $OMSHOST $PORT_OPMN
ciphercheck WLSadmin $OMSHOST $PORT_ADMINSERVER

echo -e "\n(3) Checking self-signed and demonstration certificates at SSL/TLS endpoints (see notes 1367988.1, 1399293.1, 1593183.1, 1527874.1, 123033.1, 1937457.1)"
certcheck Agent $OMSHOST $PORT_AGENT
democertcheck Agent $OMSHOST $PORT_AGENT
certcheck BIPublisher $OMSHOST $PORT_BIP
democertcheck BIPublisher $OMSHOST $PORT_BIP
certcheck NodeManager $OMSHOST $PORT_NODEMANAGER
democertcheck NodeManager $OMSHOST $PORT_NODEMANAGER
certcheck OHSadmin $OMSHOST $PORT_OHS_ADMIN
democertcheck OHSadmin $OMSHOST $PORT_OHS_ADMIN
certcheck OMSconsole $OMSHOST $PORT_OMS
democertcheck OMSconsole $OMSHOST $PORT_OMS
certcheck OMSproxy $OMSHOST $PORT_OMS_JAVA
democertcheck OMSproxy $OMSHOST $PORT_OMS_JAVA
certcheck OMSupload $OMSHOST $PORT_UPL
democertcheck OMSupload $OMSHOST $PORT_UPL
certcheck OPMN $OMSHOST $PORT_OPMN
democertcheck OPMN $OMSHOST $PORT_OPMN
certcheck WLSadmin $OMSHOST $PORT_ADMINSERVER
democertcheck WLSadmin $OMSHOST $PORT_ADMINSERVER


echo -e "\n(4) Checking EM12c Oracle home patch levels against $PATCHDATE baseline (see notes 1664074.1, 1900943.1, 822485.1, 1470197.1, 1967243.1)"

#echo -ne "\n\t(4a) OMS ($OMS_HOME) PSU2 Patch 19830994... "
#opatchcheck OMS $OMS_HOME 19830994

#echo -ne "\n\t(4a) OMS ($OMS_HOME) ENTERPRISE MANAGER BASE PLATFORM - OMS 12.1.0.4.3 PSU Patch (20392036)... "
#opatchcheck OMS $OMS_HOME 20392036

#echo -ne "\n\t(4a) OMS ($OMS_HOME) ENTERPRISE MANAGER BASE PLATFORM - OMS 12.1.0.4.4 PSU Patch (20870437)... "
#opatchcheck OMS $OMS_HOME 20870437

echo -ne "\n\t(4a) OMS ($OMS_HOME) ENTERPRISE MANAGER BASE PLATFORM - OMS 12.1.0.4.5 PSU Patch (21462217)... "
opatchcheck OMS $OMS_HOME 21462217

echo -ne "\n\t(4a) OMS HOME ($AGENT_HOME) JDBC Merge Patch (18502187)... "
opatchcheck OMS $OMS_HOME 18502187

#echo -ne "\n\t(4a) OMS ($OMS_HOME) DO NOT CREATE INCIDENT WHEN A COMMAND IS OVER RUN IN JOB WORKER (17714229)... "
#opatchcheck OMS $OMS_HOME 17714229

echo -ne "\n\t(4b) BI Publisher ($BIP_HOME) CPUJAN2015 Patch (19822893)... "
opatchcheck BIP $BIP_HOME 19822893

echo -ne "\n\t(4b) BI Publisher ($BIP_HOME) Merge Patch (20444447)... "
opatchcheck BIP $BIP_HOME 20444447

#echo -ne "\n\t(4b) BI Publisher ($BIP_HOME) ORACLE BI PUBLISHER PATCH BUG FOR PRIVATE EMCC PS3 MANDATORY INSTALL PATCH (17888172)... "
#opatchcheck BIP $BIP_HOME 17888172

echo -ne "\n\t(4c) AS Common ($COMMON_HOME) CVE-2015-0426 Oracle Help Patch (20075252)... "
opatchcheck COMMON $COMMON_HOME 20075252

#echo -ne "\n\t(4c) AS Common ($COMMON_HOME) ADF MERGE REQUEST ON TOP OF 11.1.1.7.1 FOR BUGS 20465665 18820382 20645397 (20747356)... "
#opatchcheck COMMON $COMMON_HOME 20747356

echo -ne "\n\t(4c) AS Common ($COMMON_HOME) WEBCENTER PORTAL BUNDLE PATCH 11.1.1.7.1 (16761779)... "
opatchcheck COMMON $COMMON_HOME 16761779

# Replaced 20747356, commented out above
echo -ne "\n\t(4c) AS Common ($COMMON_HOME) CVE-2015-4742 MERGE REQUEST ON TOP OF 11.1.1.7.1 FOR BUGS 20747356 18274008 (21068288)... "
opatchcheck COMMON $COMMON_HOME 21068288


#echo -ne "\n\t(4d) WebLogic Server ($WL_HOME) 10.3.6.0.10 12UV Patch (19637463)... "
#wlspatchcheck $WL_HOME 19637463

#echo -ne "\n\t(4d) WebLogic Server ($WL_HOME) 10.3.6.0.11 YUIS Patch (20181997)... "
#wlspatchcheck $WL_HOME 20181997

echo -ne "\n\t(4d) WebLogic Server ($WL_HOME) 10.3.6.0.12 EJUW Patch (20780171)... "
wlspatchcheck $WL_HOME 20780171

echo -ne "\n\t(4d) WebLogic Server ($WL_HOME) SU Patch [GDFA]: WEBLOGIC.STORE.PERSISTENTSTOREEXCEPTION: [STORE:280040] OCCURS EASILEY (16420963)... "
wlspatchcheck $WL_HOME 16420963

# Commented this patch out 4/17/2015, as Oracle no longer recommends it for EM12c installations.
# This patch still appears in note 1664074.1 for EM12c.
# Per personal communication w/Oracle I do NOT recommend using it.
#echo -ne "\n\t(4e) WebTier ($WEBTIER_HOME) CPUJAN2015 Patch (19948000)... "
#opatchcheck WebTier $WEBTIER_HOME 19948000

echo -ne "\n\t(4e) WebTier ($WEBTIER_HOME) OHS SECURITY PATCH UPDATE 11.1.1.7.0 CPUOCT2015 Patch (21640624)... "
opatchcheck WebTier $WEBTIER_HOME 21640624

echo -ne "\n\t(4e) WebTier ($WEBTIER_HOME) CVE-2014-4212 OPMN Patch (19345576)... "
opatchcheck WebTier $WEBTIER_HOME 19345576

#echo -ne "\n\t(4e) WebTier ($WEBTIER_HOME) CVE-2013-3836 PLACEHOLDER FOR SECURITY PATCH FOR WEBCACHE 11.1.1.7.0 WITH OCT2013 CPU (17306880)... "
#opatchcheck WebTier $WEBTIER_HOME 17306880

echo -ne "\n\t(4e) WebTier ($WEBTIER_HOME) CVE 2015-2658 MERGE REQUEST ON TOP OF 11.1.1.7.0 FOR BUGS 16370190 20310323 20715657 (20807683)... "
opatchcheck WebTier $WEBTIER_HOME 20807683

echo -ne "\n\t(4e) WebTier ($WEBTIER_HOME) CVE-2013-0169,CVE-2011-3389 OSS SECURITY PATCH UPDATE 11.1.1.7.0 CPUOCT2013 (17337741)... "
opatchcheck WebTier $WEBTIER_HOME 17337741

echo -ne "\n\t(4e) WebTier ($WEBTIER_HOME) WLSPLUGINS (OHS) SECURITY PATCH UPDATE 11.1.1.7.0 CPUJUL2014 (18423831)... "
opatchcheck WebTier $WEBTIER_HOME 18423831

#echo -ne "\n\t(4f) OMS ($OMS_HOME) DB PLUGIN BUNDLE 12.1.0.7.2 (20613714)... "
#opatchautocheck OMS $OMS_HOME 20613714

#echo -ne "\n\t(4f) OMS ($OMS_HOME) DB PLUGIN BUNDLE PATCH 12.1.0.7.3 (20804122)... "
#opatchautocheck OMS $OMS_HOME 20804122

#echo -ne "\n\t(4f) OMS ($OMS_HOME) DB PLUGIN BUNDLE PATCH 12.1.0.7.4 (20950048)... "
#opatchautocheck OMS $OMS_HOME 20950048

#echo -ne "\n\t(4f) OMS ($OMS_HOME) DB PLUGIN BUNDLE PATCH 12.1.0.7.5 (21167937)... "
#opatchautocheck OMS $OMS_HOME 21167937

#echo -ne "\n\t(4f) OMS ($OMS_HOME) DB PLUGIN BUNDLE PATCH 12.1.0.7.6 (21324654)... "
#opatchautocheck OMS $OMS_HOME 21324654

#echo -ne "\n\t(4f) OMS ($OMS_HOME) DB PLUGIN BUNDLE PATCH 12.1.0.7.7 (21506301)... "
#opatchautocheck OMS $OMS_HOME 21506301

#echo -ne "\n\t(4f) OMS ($OMS_HOME) DB PLUGIN BUNDLE PATCH 12.1.0.7.8 (21744938)... "
#opatchautocheck OMS $OMS_HOME 21744938

echo -ne "\n\t(4f) *UPDATED* OMS ($OMS_HOME) DB PLUGIN BUNDLE PATCH 12.1.0.7.10 (22062307)... "
opatchautocheck OMS $OMS_HOME 22062307

#echo -ne "\n\t(4g) OMS ($OMS_HOME) FMW PLUGIN BUNDLE 12.1.0.7.2 (20613870)... "
#opatchautocheck OMS $OMS_HOME 20613870

#echo -ne "\n\t(4g) OMS ($OMS_HOME) FMW PLUGIN BUNDLE PATCH 12.1.0.7.3 (20804213)... "
#opatchautocheck OMS $OMS_HOME 20804213

#echo -ne "\n\t(4g) OMS ($OMS_HOME) FMW PLUGIN BUNDLE PATCH 12.1.0.7.4 (20950040)... "
#opatchautocheck OMS $OMS_HOME 20950040

#echo -ne "\n\t(4g) OMS ($OMS_HOME) FMW PLUGIN BUNDLE PATCH 12.1.0.7.5 (21167980)... "
#opatchautocheck OMS $OMS_HOME 21167980

#echo -ne "\n\t(4g) OMS ($OMS_HOME) FMW PLUGIN BUNDLE PATCH 12.1.0.7.6 (21324861)... "
#opatchautocheck OMS $OMS_HOME 21324861

#echo -ne "\n\t(4g) OMS ($OMS_HOME) FMW PLUGIN BUNDLE PATCH 12.1.0.7.7 (21506335)... "
#opatchautocheck OMS $OMS_HOME 21506335

#echo -ne "\n\t(4g) OMS ($OMS_HOME) FMW PLUGIN BUNDLE PATCH 12.1.0.7.8 (21744989)... "
#opatchautocheck OMS $OMS_HOME 21744989

echo -ne "\n\t(4g) *UPDATED* OMS ($OMS_HOME) FMW PLUGIN BUNDLE PATCH 12.1.0.7.10 (22062375)... "
opatchautocheck OMS $OMS_HOME 22062375

#echo -ne "\n\t(4h) OMS ($OMS_HOME) MOS PLUGIN BUNDLE PATCH 12.1.0.6.4 (20613886)... "
#opatchautocheck OMS $OMS_HOME 20613886

#echo -ne "\n\t(4h) OMS ($OMS_HOME) MOS PLUGIN BUNDLE PATCH 12.1.0.6.5 (20822914)... "
#opatchautocheck OMS $OMS_HOME 20822914

#echo -ne "\n\t(4h) OMS ($OMS_HOME) MOS PLUGIN BUNDLE PATCH 12.1.0.6.6 (21167991)... "
#opatchautocheck OMS $OMS_HOME 21167991

#echo -ne "\n\t(4h) OMS ($OMS_HOME) MOS PLUGIN BUNDLE PATCH 12.1.0.6.7 (21506428)... "
#opatchautocheck OMS $OMS_HOME 21506428

echo -ne "\n\t(4h) OMS ($OMS_HOME) MOS PLUGIN BUNDLE PATCH 12.1.0.6.8 (21745018)... "
opatchautocheck OMS $OMS_HOME 21745018

#echo -ne "\n\t(4i) OMS ($OMS_HOME) EXADATA PLUGIN BUNDLE 12.1.0.6.6 (20613853)... "
#opatchautocheck OMS $OMS_HOME 20613853

#echo -ne "\n\t(4i) OMS ($OMS_HOME) EXADATA PLUGIN BUNDLE PATCH 12.1.0.6.7 (20822866)... "
#opatchautocheck OMS $OMS_HOME 20822866

#echo -ne "\n\t(4i) OMS ($OMS_HOME) EXADATA PLUGIN BUNDLE PATCH 12.1.0.6.8 (20962507)... "
#opatchautocheck OMS $OMS_HOME 20962507

#echo -ne "\n\t(4i) OMS ($OMS_HOME) EXADATA PLUGIN BUNDLE PATCH 12.1.0.6.9 (21167953)... "
#opatchautocheck OMS $OMS_HOME 21167953

#echo -ne "\n\t(4i) OMS ($OMS_HOME) EXADATA PLUGIN BUNDLE PATCH 12.1.0.6.10 (21324852)... "
#opatchautocheck OMS $OMS_HOME 21324852

echo -ne "\n\t(4i) *UPDATED* OMS ($OMS_HOME) EXADATA PLUGIN BUNDLE PATCH 12.1.0.6.11 (21744966)... "
opatchautocheck OMS $OMS_HOME 21744966

#echo -ne "\n\t(4j) OMS CHAINED AGENT HOME ($AGENT_HOME) EM-AGENT BUNDLE 12.1.0.4.7 (20613931)... "
#opatchcheck Agent $AGENT_HOME 20613931

#echo -ne "\n\t(4j) OMS ($OMS_HOME) CFW PLUGIN BUNDLE PATCH 12.1.0.2.1 (20385040)... "
#opatchautocheck OMS $OMS_HOME 20385040

#echo -ne "\n\t(4j) OMS ($OMS_HOME) CFW PLUGIN BUNDLE PATCH 12.1.0.2.2 (21167573)... "
#opatchautocheck OMS $OMS_HOME 21167573

#echo -ne "\n\t(4j) OMS ($OMS_HOME) CFW PLUGIN BUNDLE PATCH 12.1.0.2.3 (21324632)... "
#opatchautocheck OMS $OMS_HOME 21324632

echo -ne "\n\t(4j) *UPDATED* OMS ($OMS_HOME) CFW PLUGIN BUNDLE PATCH 12.1.0.2.4 (21972104)... "
opatchautocheck OMS $OMS_HOME 21972104

#echo -ne "\n\t(4k) OMS CHAINED AGENT HOME ($AGENT_HOME) EM-AGENT BUNDLE PATCH 12.1.0.4.9 (20950034)... "
#opatchcheck Agent $AGENT_HOME 20950034

#echo -ne "\n\t(4k) OMS CHAINED AGENT HOME ($AGENT_HOME) EM-AGENT BUNDLE PATCH 12.1.0.4.10 (21168025)... "
#opatchcheck Agent $AGENT_HOME 21168025

#echo -ne "\n\t(4k) OMS CHAINED AGENT HOME ($AGENT_HOME) EM-AGENT BUNDLE PATCH 12.1.0.4.11 (21325110)... "
#opatchcheck Agent $AGENT_HOME 21325110

#echo -ne "\n\t(4k) OMS CHAINED AGENT HOME ($AGENT_HOME) EM-AGENT BUNDLE PATCH 12.1.0.4.12 (21506284)... "
#opatchcheck Agent $AGENT_HOME 21506284

#echo -ne "\n\t(4k) OMS CHAINED AGENT HOME ($AGENT_HOME) EM-AGENT BUNDLE PATCH 12.1.0.4.13 (21759280)... "
#opatchcheck Agent $AGENT_HOME 21759280

echo -ne "\n\t(4k) *UPDATED* OMS CHAINED AGENT HOME ($AGENT_HOME) EM-AGENT BUNDLE PATCH 12.1.0.4.14 (21913823)... "
opatchcheck Agent $AGENT_HOME 21913823

echo -ne "\n\t(4k) OMS CHAINED AGENT HOME ($AGENT_HOME) Merge Patch (18502187)... "
opatchcheck Agent $AGENT_HOME 18502187

echo -ne "\n\t(4k) OMS CHAINED AGENT HOME ($AGENT_HOME) JDBC Security Patch (18721761)... "
opatchcheck Agent $AGENT_HOME 18721761

if [[ "$HOST_OS" == "Linux" && "$HOST_ARCH" == "x86_64" ]]; then
	echo -ne "\n\t(4k) OMS CHAINED AGENT HOME ($AGENT_HOME) CVE 2012-3137 EM Agent only: Instant Client Security Patch (20114054)... "
	opatchcheck Agent $AGENT_HOME 20114054
fi

#echo -ne "\n\t(4k) OMS CHAINED AGENT DB PLUGIN ($AGENT_DB_PLUGIN_HOME) DB PLUGIN BUNDLE 12.1.0.7.2 AGENT-SIDE 20676926... "
#opatchcheck AgentDBPlugin $AGENT_DB_PLUGIN_HOME 20676926

#echo -ne "\n\t(4l) OMS CHAINED AGENT DB PLUGIN ($AGENT_DB_PLUGIN_HOME) DB PLUGIN BUNDLE 12.1.0.7.4 AGENT-SIDE MONITORING (21065223)... "
#opatchcheck AgentDBPlugin $AGENT_DB_PLUGIN_HOME 21065223

#echo -ne "\n\t(4l) OMS CHAINED AGENT DB PLUGIN ($AGENT_DB_PLUGIN_HOME) DB PLUGIN BUNDLE 12.1.0.7.5 AGENT-SIDE MONITORING (21229731)... "
#opatchcheck AgentDBPlugin $AGENT_DB_PLUGIN_HOME 21229731

#echo -ne "\n\t(4l) OMS CHAINED AGENT DB PLUGIN ($AGENT_DB_PLUGIN_HOME) DB PLUGIN BUNDLE 12.1.0.7.6 AGENT-SIDE MONITORING (21415075)... "
#opatchcheck AgentDBPlugin $AGENT_DB_PLUGIN_HOME 21415075

#echo -ne "\n\t(4l) OMS CHAINED AGENT DB PLUGIN ($AGENT_DB_PLUGIN_HOME) DB PLUGIN BUNDLE 12.1.0.7.7 AGENT-SIDE MONITORING (21603371)... "
#opatchcheck AgentDBPlugin $AGENT_DB_PLUGIN_HOME 21603371

#echo -ne "\n\t(4l) OMS CHAINED AGENT DB PLUGIN ($AGENT_DB_PLUGIN_HOME) DB PLUGIN BUNDLE 12.1.0.7.8 AGENT-SIDE MONITORING (21806804)... "
#opatchcheck AgentDBPlugin $AGENT_DB_PLUGIN_HOME 21806804

echo -ne "\n\t(4l) *UPDATED* OMS CHAINED AGENT DB PLUGIN ($AGENT_DB_PLUGIN_HOME) DB PLUGIN BUNDLE 12.1.0.7.10 AGENT-SIDE MONITORING (22140476)... "
opatchcheck AgentDBPlugin $AGENT_DB_PLUGIN_HOME 22140476

echo -ne "\n\t(4l) OMS CHAINED AGENT DB PLUGIN ($AGENT_DB_PLUGIN_DISC_HOME) DB PLUGIN BUNDLE 12.1.0.7.4 AGENT-SIDE DISCOVERY (21065239)... "
opatchcheck AgentDBPlugin $AGENT_DB_PLUGIN_DISC_HOME 21065239

#echo -ne "\n\t(4l) OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_HOME) FMW PLUGIN BUNDLE 12.1.0.7.2 AGENT-SIDE MONITORING (20677020)... "
#opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_HOME 20677020

#echo -ne "\n\t(4m) OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_HOME) FMW PLUGIN BUNDLE 12.1.0.7.4 AGENT-SIDE MONITORING (21065760)... "
#opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_HOME 21065760

#echo -ne "\n\t(4m) OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_HOME) FMW PLUGIN BUNDLE 12.1.0.7.5 AGENT-SIDE MONITORING (21229821)... "
#opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_HOME 21229821

#echo -ne "\n\t(4m) OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_HOME) FMW PLUGIN BUNDLE 12.1.0.7.6 AGENT-SIDE MONITORING (21415166)... "
#opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_HOME 21415166

#echo -ne "\n\t(4m) OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_HOME) FMW PLUGIN BUNDLE 12.1.0.7.7 AGENT-SIDE MONITORING (21603497)... "
#opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_HOME 21603497

#echo -ne "\n\t(4m) OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_HOME) FMW PLUGIN BUNDLE 12.1.0.7.8 AGENT-SIDE MONITORING (21806984)... "
#opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_HOME 21806984

#echo -ne "\n\t(4m) OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_HOME) FMW PLUGIN BUNDLE 12.1.0.7.8 AGENT-SIDE MONITORING (21806984)... "
#opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_HOME 21806984

echo -ne "\n\t(4m) *UPDATED* OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_HOME) FMW PLUGIN BUNDLE 12.1.0.7.9 AGENT-SIDE MONITORING (21941290)... "
opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_HOME 21941290

#echo -ne "\n\t(4m) OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_DISC_HOME) FMW PLUGIN BUNDLE 12.1.0.7.2 AGENT-SIDE DISCOVERY (20677038)... "
#opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_DISC_HOME 20677038

#echo -ne "\n\t(4m) OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_DISC_HOME) FMW PLUGIN BUNDLE 12.1.0.7.5 AGENT-SIDE DISCOVERY (21229841)... "
#opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_DISC_HOME 21229841

echo -ne "\n\t(4m) OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_DISC_HOME) FMW PLUGIN BUNDLE 12.1.0.7.7 AGENT-SIDE DISCOVERY (21611921)... "
opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_DISC_HOME 21611921

#echo -ne "\n\t(4n) OMS CHAINED AGENT BEACON PLUGIN ($AGENT_BEACON_PLUGIN_HOME) EM-BEACON BUNDLE PATCH 12.1.0.4.1 (20466772)... "
#opatchcheck AgentBeaconPlugin $AGENT_BEACON_PLUGIN_HOME 20466772

echo -ne "\n\t(4n) *UPDATED* OMS CHAINED AGENT BEACON PLUGIN ($AGENT_BEACON_PLUGIN_HOME) EM-BEACON BUNDLE PATCH 12.1.0.4.2 (21928148)... "
opatchcheck AgentBeaconPlugin $AGENT_BEACON_PLUGIN_HOME 21928148

echo -ne "\n\t(4o) OMS CHAINED AGENT EM-OH BUNDLE PATCH 12.1.0.4.1 (20855134)... "
opatchcheck AgentOHPlugin $AGENT_OH_PLUGIN_HOME 20855134


if [[ $RUN_DB_CHECK -eq 1 ]]; then

#	if [[ "$REPOS_DB_VERSION" == "11.2.0.4.0" ]]; then
#		echo -ne "\n\t(4m) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) PSU 11.2.0.4.5 19769489... "
#		opatchcheck ReposDBHome $REPOS_DB_HOME 19769489
#
#		echo -ne "\n\t(4m) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) ORACLE JAVAVM COMPONENT 11.2.0.4.2 DATABASE PSU (JAN2015) 19877440... "
#		opatchcheck ReposDBHome $REPOS_DB_HOME 19877440
#	fi

	if [[ "$REPOS_DB_VERSION" == "11.2.0.4.0" ]]; then
		#echo -ne "\n\t(4p) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) PSU 11.2.0.4.6 (APR2015) (20299013)... "
		#opatchcheck ReposDBHome $REPOS_DB_HOME 20299013

		echo -ne "\n\t(4p) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) PSU 11.2.0.4.8 (OCT2015) (21352635)... "
		opatchcheck ReposDBHome $REPOS_DB_HOME 21352635

		#echo -ne "\n\t(4p) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) ORACLE JAVAVM COMPONENT 11.2.0.4.3 DATABASE PSU (APR2015) (20406239)... "
		#opatchcheck ReposDBHome $REPOS_DB_HOME 20406239

		echo -ne "\n\t(4p) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) ORACLE JAVAVM COMPONENT 11.2.0.4.5 DATABASE PSU (OCT2015) (21555791)... "
		opatchcheck ReposDBHome $REPOS_DB_HOME 21555791
	fi

#	if [[ "$REPOS_DB_VERSION" == "12.1.0.2.0" ]]; then
#		echo -ne "\n\t(4m) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) Required Patch 20243268... "
#		opatchcheck ReposDBHome $REPOS_DB_HOME 20243268
#
#		echo -ne "\n\t(4m) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) PSU 12.1.0.2.2 19769480... "
#		opatchcheck ReposDBHome $REPOS_DB_HOME 19769480
#
#		echo -ne "\n\t(4m) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) ORACLE JAVAVM COMPONENT 12.1.0.2.2 ORACLE JAVAVM COMPONENT 12.1.0.2.2 DATABASE PSU (JAN2015) 19877336... "
#		opatchcheck ReposDBHome $REPOS_DB_HOME 19877336
#	fi

	if [[ "$REPOS_DB_VERSION" == "12.1.0.2.0" ]]; then
		echo -ne "\n\t(4p) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) Required Patch (20243268)... "
		opatchcheck ReposDBHome $REPOS_DB_HOME 20243268

		#echo -ne "\n\t(4p) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) PSU 12.1.0.2.3 (APR2015) (20299023)... "
		#opatchcheck ReposDBHome $REPOS_DB_HOME 20299023

		echo -ne "\n\t(4p) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) PSU 12.1.0.2.5 (OCT2015) (21359755)... "
		opatchcheck ReposDBHome $REPOS_DB_HOME 21359755

		#echo -ne "\n\t(4p) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) ORACLE JAVAVM COMPONENT 12.1.0.2.3 DATABASE PSU (APR2015) (20415564)... "
		#opatchcheck ReposDBHome $REPOS_DB_HOME 20415564

		echo -ne "\n\t(4p) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) ORACLE JAVAVM COMPONENT 12.1.0.2.5 DATABASE PSU (OCT2015) (21555660)... "
		opatchcheck ReposDBHome $REPOS_DB_HOME 21555660
	fi

	echo -ne "\n\t(4q) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) sqlnet.ora SSL_VERSION parameter (1545816.1)... "
	paramcheck SSL_VERSION $REPOS_DB_HOME sqlnet.ora

	echo -ne "\n\t(4q) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) sqlnet.ora SSL_CIPHER_SUITES parameter (1545816.1)... "
	paramcheck SSL_CIPHER_SUITES $REPOS_DB_HOME sqlnet.ora

	echo -ne "\n\t(4q) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) listener.ora SSL_VERSION parameter (1545816.1)... "
	paramcheck SSL_VERSION $REPOS_DB_HOME listener.ora

	echo -ne "\n\t(4q) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) listener.ora SSL_CIPHER_SUITES parameter (1545816.1)... "
	paramcheck SSL_CIPHER_SUITES $REPOS_DB_HOME listener.ora
fi

echo

echo -e "\n(5) Checking EM12c Java versions against baseline (see notes 1506916.1, 1492980.1)"

echo -ne "\n\t(5a) MW ($MW_HOME/jdk16/jdk) Java version 1.6.0_95 (9553040)... "
javacheck MW $MW_HOME/jdk16/jdk 1.6.0_95

echo -ne "\n\t(5b) WebTier ($WEBTIER_HOME/jdk) Java version 1.6.0_95 (9553040)... "
javacheck WebTier $WEBTIER_HOME/jdk 1.6.0_95

echo

if [[ $FAIL_COUNT -gt "0" ]]; then
	echo "Failed test count: $FAIL_COUNT - Review output"
	test $VERBOSE_CHECKSEC -ge 1 && echo -e $FAIL_TESTS
else
	echo "All tests succeeded."
fi

echo
echo "Visit https://pardydba.wordpress.com/2015/03/09/em12c-r4-ssl-security-checkup-script/ for the latest version."
echo

exit

If you try this script, please leave me a comment.  If you can share any changes you’ve made that allow it to run on other operating systems, I and others will appreciate it. I spent a lot of time making it so the user does not have to specify any directory locations or port settings, so if you have code changes to offer please let me know.  If enough people use this I may learn how to put it on github or something.

Good luck and happy compliance audits!

Further Reading

• Patching 101 – The User Friendly Guide To Understanding EM Patches
• Step by step: Configuring third party SSL/TLS certificates in EM12c R4

• How to unofficially disable SSL v3 in Oracle Enterprise Manager 12c to mitigate POODLE attack

Step by step: Configuring third party SSL/TLS certificates in EM12c R4

[EDIT 20170227: The process for configuring third party certificates for EM13c works about the same as for EM12c. If you have access to Oracle support, I suggest you review notes 2220788.1 and 2213661.1 for the most up-to-date documentation directly from Oracle.]

By default, when an administrator configures Oracle Enterprise Manager 12c to use SSL, the system will use a default self-signed certificate, provided for demo purposes only.  The documentation states repeatedly that users should not use these certificates in a production environment, as they represent a security risk. This blog post documents, step by step, a process to replace these demo certificates with custom third party certificates, across the OMS console, OMS upload port, agents, and WebLogic Server. I will follow this process on a single-OMS configuration; if you have more than one OMS please consult the documentation for more details, as your process will vary and the steps I have provided may break your system.

I have tested these instructions on Linux x86-64 (SLES11 SP3) with EM12c R4 PSU2 (12.1.0.4).

Official Documentation

The official documentation for this process resides in the following My Oracle Support notes:

• Using ORAPKI Utility to Create a Wallet with Third Party Trusted Certificate and Import into OMS (Doc ID 1367988.1)
• EM 12c Cloud Control How to Create a Wallet With Third Party Trusted Certificate that Can Be Imported into the OMS For SSL Comunication ? (Doc ID 1399293.1)
• 12c Cloud Control: Steps to Import Third Party Trusted SSL Certificate into 12c Cloud Control Agent URL (Doc ID 1593183.1)
• 12c Cloud Control: Steps to Create and Import Third Party / Self-Signed SSL Certificates for WebLogic Server in an Enterprise Manager Installation (Doc ID 1527874.1)
• How to Create a Java Keystore via Keytool in FMW 11g/12c (Doc ID 1230333.1)

Why Should I Do This?

You may not fully understand the mechanics of SSL/TLS certificates and the chain of trust. I cannot fully explain this complex topic in a blog post, but if you need a reason to make this change other than demands from your organizational security/compliance team, please take Oracle’s word for it, and notice the text that appears in your GCDomain.log file when you run your system with the provided default demo certificates.

Read that again if you didn’t catch it the first time through: “The system is vulnerable to security attacks, since it trusts certificates signed by the demo trusted CA.” This text comes from code in WebLogic, not from me. Here Oracle tells you very explicitly that your system currently contains a severe vulnerability.

You will also notice that when using the EM12c console, or accessing an agent URL, or accessing the WebLogic Server administration console may show warnings in your browser about untrusted certificates. Once you replace your certificates as described in the documentation above or my steps below, you will no longer have those issues.

Using 3rd Party SSL/TLS Certificates With EM12c

Overview

You will follow 7 high level steps to complete the process of securing your EM12c environment with custom third party SSL/TLS certificates.

1. Create an Oracle wallet for the OMS.
2. Secure the OMS console using the OMS wallet.
3. Secure the OMS upload port using the OMS wallet.
4. Re-secure all agents.
5. Create Oracle wallets for agents.
6. Configure the agents to use their wallets.
7. Secure WebLogic with the OMS wallet.

Create an Oracle wallet for the OMS

First we follow steps 1a through 1h from document 1367988.1.  All these steps occur on the OMS host.

Disable shell history (optional but recommended)

While following these steps, you will repeatedly have to type passphrases on the command line. To avoid having these passphrases stored in your Oracle user’s shell history, disable history saving.  In the bash shell that I use, I accomplish this by unsetting the HISTFILE variable. You may need to use another mechanism in another shell.

$ unset HISTFILE

Use the correct ORAPKI command

You should use the ORAPKI command from your middleware home’s oracle_common/bin directory.  I will refer to this as $MW_HOME/oracle_common/bin/orapki in the following instructions.

Create an Oracle wallet

The documentation specified that we should create an auto-login wallet, but in my single-OMS setup, I believe that I will achieve better security with an auto-login-local wallet, as the auto-login feature will only function on this specific host. You will need to select a base directory for your OMS wallet.  I used $ORACLE_BASE/oemwallet. ORAPKI will prompt you for a password. Use a secure one, and note it down somewhere safe. You will use it many times during this process.

$ mkdir $ORACLE_BASE/oemwallet
$ $MW_HOME/oracle_common/bin/orapki wallet create -wallet $ORACLE_BASE/oemwallet -auto_login_local
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Enter password:
 Enter password again:

Get in the habit of displaying the wallet contents after each operation to confirm that everything worked.

$ $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/oemwallet
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Requested Certificates:
 User Certificates:
 Trusted Certificates:
 Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
 Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US

Create a key within the wallet.  Make sure you replace omshost.domain.com with the fully qualified domain name of your OMS host. I highly recommend using a 2048 bit keysize, as shown below. Include the wallet password you specified earlier on the commandline as the -pwd argument, contained in single quotes. Display the wallet again afterward.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -dn "CN=omshost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US" -keysize 2048 -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

$ $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/oemwallet
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Requested Certificates:
 Subject: CN=omshost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US
 User Certificates:
 Trusted Certificates:
 Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US

Export a certificate signing request based on this key. Make sure the -dn you specify exactly matches the -dn specified earlier. Provide a filename in the -request argument in which to store the certificate signing request (CSR).

$ $MW_HOME/oracle_common/bin/orapki wallet export -wallet $ORACLE_BASE/oemwallet -dn "CN=omshost.domain.com, OU=EM,O=Organization,L=City,ST=State,C=US" -request ~/EM12cCSR.txt
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

Submit this CSR file to your signing authority. Inform them that you MUST have a single-host certificate with your OMS host’s fully qualified domain name in the CN field. Subject Alternate Name (SAN) certificates or wildcard certificates will not work at all. Your signing authority should then provide you with a root certificate, an intermediate certificate, and a user certificate.

Import the root, intermediate, and user certificates into the OMS wallet. Note that you must import the root and intermediate certificates using -trusted_cert, and the user certificate using -user_cert.  I used DigiCert, and I can confirm that their certificates function correctly in EM12c and recommend their service.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -trusted_cert -cert ~/TrustedRoot.cer -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -trusted_cert -cert ~/DigiCertCA2.cer -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -user_cert -cert ~/omshost.domain.com.cer -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

Display the wallet contents after this operation.

$  $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/oemwallet
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Requested Certificates:
 User Certificates:
 Subject: CN=omshost.domain.com,OU=[REDACTED],O=[REDACTED],L=City,ST=State,C=US
 Trusted Certificates:
 Subject: CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US
 Subject: CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US
 Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
 Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US

Concatenate the root and intermediate certificates into a single file, which you will use later.

$  cat ~/DigiCertCA2.cer ~/TrustedRoot.cer > $ORACLE_BASE/trusted_certs.txt

You have completed configuration of your OMS wallet.

Secure the OMS console

Now, using emctl from the $OMS_HOME, tell EM12c to secure the OMS console using the certificate contained in your wallet. The system will prompt you for the SYSMAN password and inform you to restart the entire OMS once complete.

$ $OMS_HOME/bin/emctl secure console -wallet /oracle/oem/oemwallet
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Securing Console... Started.
 Enter Enterprise Manager Root (SYSMAN) Password :
 Securing Console... Successful
 Restart OMS
$ $OMS_HOME/bin/emctl stop oms -all ; sleep 5 ; $OMS_HOME/bin/emctl start oms

Now access your OMS console with your favorite browser and confirm that your new certificate appears.  Your certificate should show a trusted path back to a root certificate, and your browser should produce no warnings.

At this point, you have secured communication between your browser and the EM12c OMS console with your custom certificate.  You still have more work to do though. Your agents upload monitoring data to the OMS upload port, and it still uses the demo certificate. Fix that in the next step.

Secure the OMS upload port

Secure the OMS upload port. Expect to receive email or pager alerts after this step, as once you restart the OMS, none of your agents can communicate with it, as they expect to see the demo certificates on the upload port. You will need to provide the SYSMAN password as well as an agent registration password.

$ $OMS_HOME/bin/emctl secure oms -wallet $ORACLE_BASE/oemwallet -trust_certs_loc $ORACLE_BASE/trusted_certs.txt
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Securing OMS... Started.
 Enter Enterprise Manager Root (SYSMAN) Password :
 Enter Agent Registration Password :
 Securing OMS... Successful
 Restart OMS
$ $OMS_HOME/bin/emctl stop oms -all ; sleep 5 ; $OMS_HOME/bin/emctl start oms

Re-secure all agents

Now you must re-secure all of your agents so that they can resume uploading data to the OMS console and monitoring your systems. Execute the following steps on every agent, using emctl from the agent home.  You will need to provide an agent registration password to complete this process.

$ $AGENT_HOME/bin/emctl secure agent
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Agent successfully stopped... Done.
Securing agent... Started.
Enter Agent Registration Password : 
Agent successfully restarted... Done.
Securing agent... Successful.
$ $AGENT_HOME/bin/emctl upload agent
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

It may take a little while for the OMS to process the new agents and their uploads, but once you have run this process on every agent they should all communicate successfully with the OMS and appear as OK from the agent management screen.

Create Oracle wallets for agents

Next we secure the agent URLs. The OMS connects to the agents at this URL to submit management requests. At the moment, the agents still use self-signed certificates to secure this URL.  For this process we create an Oracle wallet, on the OMS host, using the same ORAPKI command as for the OMS wallet. We will generate a certificate signing request from each agent wallet, submit those CSRs to a certificate authority, and import the received certificates.

As with the OMS, the agents must use single-host certificates, not wildcard or subject alternate name (SAN) certificates.  To determine the correct fully qualified domain name for each agent, execute emctl status agent from the agent home.

$  $AGENT_HOME/bin/emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 12.1.0.4.0
OMS Version : 12.1.0.4.0
Protocol Version : 12.1.0.1.0
Agent Home : /oraagent/agent12c/agent_inst
Agent Log Directory : /oraagent/agent12c/agent_inst/sysman/log
Agent Binaries : /oraagent/agent12c/core/12.1.0.4.0
Agent Process ID : 12480
Parent Process ID : 12359
Agent URL : https://agenthost.domain.com:3872/emd/main/

Repeat these steps for every agent.

Create a directory to store the agent wallet, and an agent wallet. This time do NOT use -auto_login_local, use only -auto_login, as you will distribute these wallets to the agent hosts after generating them on the OMS host.  Use a strong password, and save it for later, as you will reuse it many times.

$ mkdir $ORACLE_BASE/agentwallets
$ mkdir $ORACLE_BASE/agentwallets/agenthost.domain.com
$ $MW_HOME/oracle_common/bin/orapki wallet create -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -auto_login
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Enter password:
 Enter password again:

Create the certificate, then a certificate signing request, saving it to file ~/agenthost.domain.com.csr. Again, I recommend a 2048 bit certificate.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -dn "CN=agenthost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US" -keysize 2048 -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
$ $MW_HOME/oracle_common/bin/orapki wallet export -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -dn "CN=agenthost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US" -request ~/agenthost.domain.com.csr
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

As before, submit this certificate signing request to your certificate authority, and receive back three files containing a root certificate, an intermediate certificate, and a user certificate. Import these into the agent wallet, and display the wallet afterwards to confirm everything imported successfully.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -trusted_cert -cert ~/TrustedRoot.crt -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -trusted_cert -cert ~/DigiCertCA.crt -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -user_cert -cert ~/agenthost.domain.com.crt -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
$ $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Requested Certificates:
 User Certificates:
 Subject: CN=agenthost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US
 Trusted Certificates:
 Subject: CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US
 Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
 Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=DigiCert High Assurance CA-3,OU=www.digicert.com,O=DigiCert Inc,C=US

You have finished creating this agent’s wallet.  Repeat this for every agent.

Configure the agents to use their wallets

Inside the agent wallets you’ve just created, you will find a cwallet.sso file. Take this file from each agent’s wallet and copy it to the agent server. Stop the agent, then place the file into $AGENT_INSTANCE_DIR/sysman/config/server/ and set the permissions to 640, then start the agent.

$ $AGENT_HOME/bin/emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Stopping agent ..... stopped.
$ cp cwallet.sso $AGENT_INSTANCE_DIR/sysman/config/server
$ chmod 640 $AGENT_INSTANCE_DIR/sysman/config/server
$ $AGENT_HOME/bin/emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Starting agent ............. started.

Next, visit the agent URL in your favorite web browser and examine the certificate it uses.  You should now see that it uses the 3rd party SSL/TLS certificate that you installed.

Secure WebLogic with the OMS wallet

Now the OMS (both console and upload ports) and agents will use your new certificates. This leaves WebLogic as the one remaining component needing your new certificates. Please note in following the below directions that securing WebLogic with a wallet only works as of EM12c R3, earlier versions must use a Java keystore. See note 1527874.1 for more information.

[NOTE: 20150910: If you secure WebLogic with a certificate that uses the SHA256 hashing algorithm, future attempts to apply EM12c PSU patches using ‘opatchauto’ will fail. Some piece of opatchauto does not support SHA256 usage in certificates. If you run into this issue, revert your WLS to the demonstration certificate using emctl secure wls -use_demo_cert, then apply the PSU, then resecure WLS using these steps with your desired certificate. I intend to write a full blog post about this later.]

First import the root and intermediate certificates to the keystore on the OMS host’s agent. Use the default password welcome for the agent keystore, and alias names rootcacert and intercacert.

$ $AGENT_HOME/bin/emctl secure add_trust_cert_to_jks -trust_certs_loc ~/TrustedRoot. crt -alias rootcacert -password welcome
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.

 Message : Certificate was added to keystore
 ExitStatus: SUCCESS
$ $AGENT_HOME/bin/emctl secure add_trust_cert_to_jks -trust_certs_loc ~/DigiCertCA. crt -alias intercacert -password welcome
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.

 Message : Certificate was added to keystore
 ExitStatus: SUCCESS

Back up some WLS configuration files, just in case, before securing WLS with your certificate.  If you have problems in this step, make sure you have stopped all WLS processes, then restore these files from backup.

$ mkdir ~/wlscertbak
$ cp -a $EM_INSTANCE_BASE/em/EMGC_OMS1/emgc.properties ~/wlscertbak/
$ cp -a $EM_INSTANCE_BASE/NodeManager/emnodemanager/nodemanager.properties ~/wlscertbak/
$ cp -a $EM_INSTANCE_BASE/WebTierIH1/config/OHS/ohs1/keystores/proxy ~/wlscertbak/
$ cp -a $EM_INSTANCE_BASE/user_projects/domains/GCDomain/config/config.xml ~/wlscertbak/

Stop the OMS.

$ $OMS_HOME/bin/emctl stop oms
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Stopping WebTier...
 WebTier Successfully Stopped
 Stopping Oracle Management Server...
 Oracle Management Server Successfully Stopped
 Oracle Management Server is Down

Secure WLS using the OMS wallet created earlier. You will need to provide the SYSMAN password.

$ $OMS_HOME/bin/emctl secure wls -wallet $ORACLE_BASE/oemwallet
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Securing WLS... Started.
 Enter Enterprise Manager Root (SYSMAN) Password :
 Securing WLS... Successful
 Restart OMS using 'emctl stop oms -all' and 'emctl start oms'
 If there are multiple OMSs in this environment, perform this configuration on all of them.

Stop the entire WLS stack, then start the OMS.

$ $OMS_HOME/bin/emctl stop oms -all
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Stopping WebTier...
 WebTier Successfully Stopped
 Stopping Oracle Management Server...
 Oracle Management Server Already Stopped
 Oracle Management Server is Down
 Stopping BI Publisher Server...
 BI Publisher Server Successfully Stopped
 AdminServer Successfully Stopped
 BI Publisher Server is Down
$ $OMS_HOME/bin/emctl start oms
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Starting Oracle Management Server...
 Starting WebTier...
 WebTier Successfully Started
 Oracle Management Server Successfully Started
 Oracle Management Server is Up
 Starting BI Publisher Server ...
 BI Publisher Server Successfully Started
 BI Publisher Server is Up

Visit the WebLogic Server administration console and you should now see that it presents your custom SSL/TLS certificate and no longer uses the demo certificate.

Conclusion

If you have successfully followed these steps, your system should now use your custom SSL/TLS certificates everywhere, and you should no longer experience certificate warnings in your browsers.

See Also

• EM12c R4 SSL Security Checkup Script

When proactive EM12c JDK upgrades bite back

You probably will not encounter this issue, but I will post this anyway to get the error message and resolution indexed by Google.

While attempting to apply patch 19513382 (EM agent bundle patch 12.1.0.4.3) to my EM12cR4 agents, I ran into multiple problems.  Initially it would not apply to any of my agents.  Bug 20134182 and the resolution described in MOS note 1952355.1 resolved the first problem (OPatch reporting that identical patches 18721761 and 18502187 already exist), but that left me with one agent I could not upgrade. Attempts to run patch plan validation within EM12c produced the following error:

PatchList : 19513382
PatchLocList : /tmp/p19513382_600000000009641_2000_0/oraagent
TargetName : [redacted]:[port]
----------------------------------------
[11_12_2014_10_00_40] Command Arguments:
/oraagent/agent12c/core/12.1.0.4.0/OPatch/opatch checkComponents -phbasedir /tmp/p19513382_600000000009641_2000_0/oraagent/19513382 -oh /oraagent/agent12c/core/12.1.0.4.0 -invPtrloc /oraagent/agent12c/core/12.1.0.4.0/oraInst.loc
 
OPatch cannot continue because it would not be able to load OUI platform dependent library from the directory "/oraagent/agent12c/core/12.1.0.4.0/oui/lib/linux64". The directory does not exist in the Oracle home.
This could be due to the following reasons.

(1) Incompatible usage of java with OUI (32/64 bit).
(2) Wrong 32-bit Oracle Home installation in 64-bit environment (or) vice-versa.
Please contact Oracle support for more details.
 
OPatch failed with error code 1
 
PREREQ_CONTEXT_HOST_NAME:[redacted]
REREQ_CONTEXT_HOME_LOCATION:/oraagent/agent12c/core/12.1.0.4.0
PREREQ_NAME: Checking if the patches are applicable.
PREREQ_DESC: Checking if the patches are applicable on the Management Agent.
PREREQ_TYPE:APPLICABILITY
PREREQ_STATUS:FAILED
PREREQ_MESG: None of these patches are applicable on the Management Agent.
PREREQ_MESG_PATCH:19513382
PREREQ_REMEDY:MANUAL
PREREQ_REMEDY_DETAILS: Remove patch(es) 19513382 from this patch plan.

I already know from the previously referenced MOS note that OPatch 11.1.0.12.3 contains bugs, so as a first debugging step I attempted to rollback the OPatch upgrade by restoring the backup copies of OPatch found in $AGENT_HOME/OPatch/backup/.  I received the same error message with OPatch 11.1.0.10.4 and 11.1.0.11.0.  I also received a similar error even if I simply tried to run “opatch lsinv” from the command line with the older versions. So OPatch did not cause this issue.

Since the error message mentions 32-bit and 64-bit incompatibility, I needed to consider the environment.  This server runs Linux x86-64 (SLES 10 SP4), but must use a 32-bit EM agent based on the certification matrix and MOS note 1488161.1. I next checked to find my last successful patch run, which happened only a month ago, so a recent change has to have caused this problem. Going through my notes, the only recent change on this server involved upgrading the JDK used by the EM agent per MOS note 1944044.1.

Luckily I still had the old JDK available for comparison.

> java -version
java version "1.6.0_43"
Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
Java HotSpot(TM) Server VM (build 20.14-b01, mixed mode)
> file `which java`
/oraagent/agent12c/core/12.1.0.4.0/jdk/bin/java: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), not stripped

Looking at the new JDK:

> ./java -version
java version "1.6.0_85"
Java(TM) SE Runtime Environment (build 1.6.0_85-b13)
Java HotSpot(TM) 64-Bit Server VM (build 20.85-b01, mixed mode)
> file java
java: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), for GNU/Linux 2.4.0, dynamically linked (uses shared libs), not stripped

There I have my problem. In upgrading the JDK, I had installed the 64-bit version of Java 1.6u85, not the 32-bit version, based on the fact that the server runs a 64-bit OS. I had not considered that a 64-bit JDK would not remain compatible with the 32-bit agent on this 64-bit system.

Surprisingly, everything about the agent seems to have worked fine, despite the 64-bit JDK.  Nothing broke until I attempted to use OPatch.

To resolve the issue, I stopped the agent and moved the original 32 bit 1.6u43 JDK back to where it belongs, followed note 1952355.1 to work around the known bugs when using OPatch 11.1.0.12.3 to apply 19513382, then successfully applied the patch.  After that I downloaded the correct 32-bit version of the 1.6u85 JDK, installed it per 1944044.1, and now OPatch works as expected.

How to unofficially disable SSL v3 in Oracle Enterprise Manager 12c to mitigate POODLE attack

With the recent POODLE vulnerability, server operators must now (finally) disable SSL version 3.0 and move up to TLS 1.0 at the minimum, if not TLS 1.2.

UPDATE: Many thanks to Courtney Llamas who provided me with a link to the section of the documentation that describes the right way to do this.  If you want to disable SSLv3 in EM12c, follow the instructions in section 2.3.2.4 of chapter 2 of the Oracle Enterprise Manager Cloud Control Security Guide.  You will need to re-secure your OMS during the process and this will require that you have access to the SYSMAN password and an agent registration password. I can confirm that the steps in this document work and do disable SSLv3. Make sure you follow the steps in the document to secure the management agents, too.

[EDIT: 20150312: Please note that you MUST install the 12.1.0.4.6 Agent bundle patch 20423395 to allow the agent-side “allowTLSOnly” property to function correctly. If you do not install this patch on your 12.1.0.4 agent, the agent will continue to permit SSLv3 connections.]

Configuring AWR Warehouse (AWRW) in EM12c

Oracle Enterprise Manager 12cR4 introduces the new “AWR Warehouse (AWRW)” feature, permitting administrators to consolidate AWR statistics from many individual databases managed by OEM into a single AWRW repository database.

As with all Oracle features, you must pay attention to licensing here.  I will not discuss licensing other than to point you to the relevant documents which you must read and understand yourself: Oracle Database Licensing Information 12c Release 1.

Documentation

At the moment official documentation appears limited to MOS note 1907335.1 and one section of the Oracle Database 2-Day Database + Performance Tuning Guide.  Get familiar with it.

Prerequisites

Repository Database

You must use Enterprise Edition for the AWRW repository database.  You must use version 12.1.0.2 or higher, or version 11.2.0.4 with patch 18547891 applied. Oracle recommends you use a database not used for any other purpose. I strongly agree with that recommendation.  Do not use your OEM repository. Note that I had to enable the diagnostic and tuning packs on the AWRW repository database by setting the control_management_pack_access initialization parameter to “DIAG+TUNING” before EM12c would allow me to select it for the repository.  I cannot reiterate enough how much I wish Oracle would explicitly state that users may enable management packs on their limited-use repository databases that support EM12c, RMAN catalogs and AWRW, but only a sucker expects license clarity out of Oracle.

I have selected 11.2.0.4 with patch 18547891 for my AWRW repository.

Oracle Enterprise Manager

You must use Oracle Enterprise Manager 12cR4 (12.1.0.4), and your OMS must have at least the August 31, 2014 bundle patch (19391521, or a later bundle patch) applied.  Your agents must run version 12.1.0.4.2 or later (requiring patch 19051570).

Licensing

Double check your licensing one more time.  Do not use features you have not licensed or you will pay a lot of money once you get audited, and you will get audited.

Configuration

For the purposes of this post I will skip the database installation and configuration steps.  If you have not yet gained proficiency with base installation and configuration tasks, you should probably gain some experience there before diving in to the AWR Warehouse.  Install a database of the appropriate version and register it with EM12c.

Planning

Think about your architecture.  With the recent release of AWRW functionality, some rough edges still exist.  These will probably get cleaned up over the next few releases but they took me by surprise and I have not seen them documented anywhere.

Oracle Enterprise Manager Agent Considerations

Do you use a separate dedicated user account on your servers to run the OEM Agent?  I do. Your AGENT_INSTANCE_DIR will get used by AWRW as a place to hold Data Pump output containing each source database’s AWR data.  I had to make this directory group writable by the dba group.  You also need to make sure the volume where this directory resides has enough free space to hold AWR extracts, which end up quite large on a busy system.  You may need to add more space if you keep your agent on a dedicated filesystem, as I do.

Do you run multiple instances under isolated accounts that don’t share a group membership?  You will probably need to create a group they all share that can write to the AGENT_INSTANCE_DIR.

Preferred Credential Considerations

AWRW strongly depends on the preferred credentials set for a database instance by the user that adds the database to AWRW.  If you already heavily use preferred credentials and want to use a different preferred database login for AWRW extraction compared to your usual DBA activities, you may elect to create a dedicated EM12c administrator to maintain AWRW to avoid conflicts.

The AWRW extraction user in the target database must have the DBA role and must also have an explicit execute grant on package SYS.DBMS_SWRF_INTERNAL.  I have chosen to use the SYSTEM account, to match my other preferred credential usage, but a more secure setup would use an account dedicated to this task.

Space Considerations

Take a look at how much space AWR consumes in your SYSAUX tablespaces already.  Your AWRW repository will need at least this much space, multiplied by however long you plan to keep these AWR snapshots around.  This will get very large, very quickly.

Added 20140912: I highly recommend that you disable data file autogrowth on your AWRW repository database.  I experienced repeated hangs until I determined that my jobs continually got stuck when SYSTEM or SYSAUX nearly filled and they sat there waiting on data file operations I/O as the system failed to resize the data files or identify a deadlock.  Do not rely on data file autogrowth, at least when using an 11.2.0.4 AWRW repository.

Initialize The AWR Warehouse

To begin configuring the AWR Warehouse, you must login using an EM12c super administrator account, like SYSMAN.  Once logged in, go to the Databases target list screen.  Unfortunately for this purpose you must use the “Database Load Map” form of the screen and not the infinitely more useful “Deprecated Search List (with Metrics)” that I have up on screen 99.9% of the time. Click the Targets menu, select Databases from the submenu that appears, and then if you do not see a “Performance” menu, enable the “Database Load Map” radio button.

Click the Performance menu and select the “AWR Warehouse” item.

This button makes things happen

At this point, if you used a super administrator account, you should see a summary screen that briefly describes the feature with a link to click to begin configuration.  If you don’t, log out and come back with the SYSMAN account.

Click Configure to continue

The next screen offers a search box to select the database to use as your AWRW repository and the usual credential selector.  Select the correct database, choose a database credential (I first selected SYSTEM, which failed, so use SYS as SYSDBA) and provide host credentials.

Rough edge: no warning that you must use SYSDBA

Once you click Next, the tool will pop up a dialog box warning you to make sure that your repository database has the necessary patch installed, and then asks you to select how long the system should keep AWRW data.  You can also select a staging location for AWR data extract storage prior to data loading.

Diamonds and AWR Warehouses are forever

Click Submit on this screen and OEM will submit a job to initialize the AWRW repository.  To find this job later, if needed, go to the advanced job activity page and search for jobs of type “dbSetupCAW”.  The job should complete successfully if you have done everything correct so far.  On my system it only took six seconds, so just wait a moment and then reload the page, which should now look like this.

That was easy

Click on the database icon at the upper left to switch away from the repository configuration tab to the database selection tab.

No data yet

As of this point you no longer need to use the SYSMAN account.  I switched back to my regular account, then returned to this screen.

Click the Add button to begin adding your first database(s). OEM will prompt you with the usual target selection screen.  Choose one or more databases and then click the Select button.  AWRW will NOT prompt you for credentials at this time.  Instead it will silently use the database host and normal database user preferred credentials you have established for the database target.  Another rough edge I expect to work better in future versions.  AWRW will perform some initial validations of those credentials to make sure that the database user has the DBA role and the previously mentioned execute grant on SYS.DBMS_SWRF_INTERNAL.  If you have missed any of these steps OEM will tell you about it and prevent you from adding the database.  Again, later I expect this to include an automated setup that will fix those issues.

I can’t show you the name

At this point you can just walk away and within about 24 hours you should have AWR data loaded into the warehouse.  If you feel impatient, click on one of the lines for a database to select it, then choose “Upload Snapshots Now” from the Actions menu.  This will submit a job to extract and load the AWR data, which you can find later under the job type “Run AWR Extract, Transfer and Load”.  In the background, this job extracts AWR data to the AGENT_INSTANCE_DIR on the target database’s server, compresses the data, transfers it to the staging area defined during AWRW repository setup, then loads the transferred data into the consolidated AWR Warehouse repository.

One database in there. So many to go.

Summary

The size of and load on your selected database, along with the amount of AWR history you keep online, will influence how long each load takes.  My first load appeared to hang, with the AWRW repository database full of sessions waiting on enq: HW contention and buffer busy waits.  I ended up doing a shutdown abort and following the workaround instructions in MOS note 1912230.1.  I do not know if I truly needed to do this or not, but the symptoms sounded similar.  I’ve also noticed that some limits appear to exist.  I keep 42 days worth of hourly snapshots in each AWR, and my initial load only picked up 20 days / 500 snapshots.  This may represent rate-limiting as a way to trickle-load the AWRW, or it may mean AWRW does not yet play nicely with large AWR data.  Time will tell, and I fully expect future versions to shake out any bugs and to hold the DBA’s hand a bit more through the process.

I hope to cover using AWRW for performance tuning in a later post and I look forward to comments.

More Information

See these other fine posts for more information.

• AWR Warehouse in EM12c Rel. 4
• AWR Warehouse in EM12c, Rel 4, Part II
• AWR Warehouse Webinar from ODTUG
• AWR Warehouse, Status

How to migrate EM12c R3 OMS and repository to a new host

(EDIT 20130917: If you simply need to change the IP address of your OEM server, please review MOS note 1562029.1.  The procedure in that note may allow you to change your OEM server’s IP address without following the lengthy process I describe below.)

In order to save power in our data center, I need to migrate my EM12c R3 environment from the host where it currently runs to a new host.  I have a simple configuration, with a single OMS, no load balancer, and the repository database runs on the same host as EM12c R3 itself.  I also have BI Publisher installed and integrated with EM12c, and a few third party plugins as I’ve detailed elsewhere on this blog.  If you use an OS other than Linux x86-64 I suggest you research thoroughly as this procedure may or may not apply to your environment.  Further, if you have a multi-OMS setup or use a load balancer, you must read the documentation and adapt the process accordingly to match your system’s needs.  Note that I wrote this as I did the migration, live, on my production system, so I have text in a few places showing where I would have done things differently if I knew what to expect in the first place.  It all ended up working, but it could have been simpler.

Oracle documents the procedure for this migration in the EM12c Administrator’s Guide, Part VII, section 29, “Backing Up and Recovering Enterprise Manager“.  As a first step, my system administrator installed SLES 11 SP3 on the new server and created an account for me along with the ‘oracle’ account for EM12c. I have a 70GB volume to use for the database and OEM binaries, a 1GB volume for the DB control files and a 2GB volume for redo logs supplemented with a 15GB FRA volume to support flashback.  Due to our tape backup strategy I use the FRA only for flashback, which we don’t wish to backup, and use a separate volume for RMAN backupsets.  To avoid a backup/restore cycle, the volumes holding the database datafiles will just be moved over to the new host on the storage side.

First I will relocate the management repository database to the new host, then complete the process by relocating the OMS.

Relocating the Management Repository Database

I run Oracle Database 11.2.0.3, Enterprise Edition, plus PSU Jul 2013.  Rather than installing the database software from scratch and patching it, I will clone the existing Oracle home to the new server.  Unfortunately I cannot use EM12c to do the cloning, as cloning via EM12c requires a management agent on the new host.  The software-only install of EM12c that I will run later installs a management agent as part of the process and I do not wish these two to conflict, so I do not want to install an agent on the new host at this time.

I will clone the database home according to the procedure in Appendix B of the 11gR2 database documentation.  You should review the documentation for full details.

Cloning the Database Home

Stop the OMS, database and management agent before cloning the existing Oracle home.

oracle$ $OMS_HOME/bin/emctl stop oms -all ; $AGENT_HOME/bin/emctl stop agent ; $ORACLE_HOME/bin/dbshut $ORACLE_HOME

Create a zip file of the existing database home.  Run this step as root (or using sudo) to make sure that you get all the files.

oracle$ sudo zip -r dbhome_1.zip /oracle/oem/product/11.2.0/dbhome_1

Now I will start the original database back up so that OEM continues running while I prepare the cloned Oracle home.  I will perform this migration over a few days, as I have time, so I need to keep OEM up and running as much as possible to support and manage my other databases.

oracle$ $ORACLE_HOME/bin/dbstart $ORACLE_HOME ; sleep 10 ; $OMS_HOME/bin/emctl start oms ; sleep 10 ; $AGENT_HOME/bin/emctl start agent

Copy this zip file to the new host.

oracle$  scp dbhome_1.zip oracle@newhost:/oracle/oem

On the new host, extract this zip file to the target directory.

oracle@newhost$ unzip -d / dbhome_1.zip

Remove all “*.ora” files from the extracted $ORACLE_HOME/network/admin directory.

oracle@newhost$  rm /oracle/oem/product/11.2.0/dbhome_1/network/admin/*.ora

Execute clone.pl from $ORACLE_HOME/clone/bin.

oracle@newhost$ export ORACLE_HOME=/oracle/oem/product/11.2.0/dbhome_1
oracle@newhost$ $ORACLE_HOME/perl/bin/perl clone.pl ORACLE_BASE="/oracle/oem" ORACLE_HOME="/oracle/oem/product/11.2.0/dbhome_1" OSDBA_GROUP=dba OSOPER_GROUP=oper -defaultHomeName

Unfortunately this creates an oraInventory directory in the oracle user’s home directory.  I prefer to keep oraInventory under ORACLE_BASE, so I moved it and edited the generated files to change the path from /home/oracle/oraInventory to /oracle/oem/oraInventory.  Most likely some environment variable, or a previously existing /etc/oraInst.loc would have prevented this optional step.

oracle@newhost$ cp -a ~/oraInventory /oracle/oem
oracle@newhost$ cd /oracle/oem/oraInventory
oracle@newhost$ perl -pi.bak -e 's#/home/oracle#/oracle/oem#' oraInst.loc orainstRoot.sh

Complete the cloning steps by running the orainstRoot.sh and root.sh scripts.

oracle@newhost$ sudo /oracle/oem/oraInventory/orainstRoot.sh
Changing permissions of /oracle/oem/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /oracle/oem/oraInventory to dba.
The execution of the script is complete.
oracle@newhost$ sudo /oracle/oem/product/11.2.0/dbhome_1/root.sh
Check /oracle/oem/product/11.2.0/dbhome_1/install/root_newhost_2013-08-27_13-04-51.log for the output of root script

I do not want to use netca to configure the listener, so I will just copy the $ORACLE_HOME/network/admin/*.ora files back over from the original server to the new server, and edit them accordingly.

oracle$ scp *.ora oracle@newhost:/oracle/oem/product/11.2.0/dbhome_1/network/admin/ 

oracle@newhost$ cd $ORACLE_HOME/network/admin
oracle@newhost$ perl -pi.bak -e 's#oldhost#newhost#' *.ora

This completes the database cloning.

Start Management Repository Database On New Host

At this point you will probably use RMAN to create a backup of your original repository database, then restore that backup onto the new host.  Instead, I will cheat a bit, shut down OEM and the database, and ask my sysadmin to move the repository database’s datafile LUN over to the new host and mount it at the same location.

Before moving the LUN, create directories that the database needs for a successful startup.  These include the admin/SID/adump directory, and in my case, the /oracle/mirror/SID/cntrl and /oracle/mirror/SID/log directories where I keep the multiplexed copies of my redo logs and controlfiles.

oracle@newhost$ mkdir -p /oracle/oem/admin/emrep/adump
oracle@newhost$ mkdir -p /oracle/mirror/emrep/cntrl ; mkdir -p /oracle/mirror/emrep/log

As a sanity check, you should try starting up the listener on the new server and starting the database in NOMOUNT mode before proceeding.  This will help catch any issues that may exist in your environment before you start the outage on your original server.  Investigate and resolve any issues found before proceeding.

Shutdown the OMS, agent and database on the original server.

oracle$ $OMS_HOME/bin/emctl stop oms -all ; $AGENT_HOME/bin/emctl stop agent ; $ORACLE_HOME/bin/dbshut $ORACLE_HOME

Copy the controlfiles and redo logs from the original server to the new server.

oracle$ scp /oracle/oem/cntrl/control01.ctl oracle@newhost:/oracle/oem/cntrl/control01.ctl
oracle$ scp /oracle/mirror/emrep/cntrl/control02.ctl oracle@newhost:/oracle/mirror/emrep/cntrl/control02.ctl
oracle$ scp /oracle/oem/log/redo* oracle@newhost:/oracle/oem/log
oracle$ scp /oracle/mirror/emrep/log/redo* oracle@newhost:/oracle/mirror/emrep/log

Back on the new server, start up the listener, then the database.  I probably should have disabled flashback first.

oracle@newhost$ lsnrctl start LISTENER
oracle@newhost$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Wed Aug 28 10:09:01 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup;
ORACLE instance started.

Total System Global Area 9620525056 bytes
Fixed Size                  2236488 bytes
Variable Size            6241128376 bytes
Database Buffers         3355443200 bytes
Redo Buffers               21716992 bytes

Database mounted.
ORA-38760: This database instance failed to turn on flashback database
SQL> select open_mode from v$database;

OPEN_MODE
--------------------
MOUNTED

SQL> alter database flashback off;

Database altered.

SQL> alter database open;

Database altered.

Reconfigure Existing OMS For New Repository Database

Start the OMS and agent on the original server.  OMS startup will fail, as you have not yet reconfigured the repository.

oracle$ $OMS_HOME/bin/emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server is not functioning because of the following reason:
Failed to connect to repository database. OMS will be automatically restarted once it identifies that database and listener are up.
Check EM Server log file for details: /oracle/oem/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
oracle$ $AGENT_HOME/bin/emctl start agent

Reconfigure the OMS repository database connection.  Provide SYSMAN’s password when prompted.

oracle$ $OMS_HOME/bin/emctl config oms -store_repos_details -repos_conndesc "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=newhost)(PORT=1521)))(CONNECT_DATA=(SID=emrep)))" -repos_user sysman
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Enter Repository User's Password : 
Successfully updated datasources and stored repository details in Credential Store.
If there are multiple OMSs in this environment, run this store_repos_details command on all of them.
And finally, restart all the OMSs using 'emctl stop oms -all' and 'emctl start oms'.
It is also necessary to restart the BI Publisher Managed Server.

Stop, then restart the OMS.

oracle$ $OMS_HOME/bin/emctl stop oms -all ; sleep 5 ; $OMS_HOME/bin/emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
AdminServer Successfully Stopped
Oracle Management Server is Down
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up

Login to OEM and confirm proper operation of the system.  I had a lot of alerts for failed backup jobs since my repository database hosts my RMAN catalog.  These can wait for now.  Also expect your repository target to show as down, since you have not yet updated the monitoring configuration.  Reconfigure it now, providing the SYSMAN password when prompted.

oracle$ $OMS_HOME/bin/emctl config emrep -conn_desc "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=newhost)(PORT=1521)))(CONNECT_DATA=(SID=emrep)))"
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Please enter repository password:                                    Enter password :                                               Login successful
Target "Management Services and Repository:oracle_emrep" modified successfully
Command completed successfully!

At this point you have successfully moved your repository database.  Don’t worry about any errors for now, though if you rely on an RMAN catalog and stored scripts for your backups, and these all live in your OEM repository database, you should go through now and update the monitoring configuration for the repository database and listener so that backups of your other databases do not fail.  I had to edit the recovery catalog and specify the host, port, and SID manually, since for some reason when I told it to use the repository database it kept trying to use the old hostname.  I will fix this after I complete the rest of the migration.

IMPORTANT NOTE: Since you have not yet migrated the repository database target to an agent local to that machine, backups of your repository database may not run.  Monitor your archived log directory on this system until you complete the rest of the migration, and manually run backups when necessary.

Installing OMS On A New Host

To install the OMS on a new host, perform a software-only installation from the same EM12c R3 installer that was used to install on the original host.  You will need to identify and retrieve all of the plugins that you have installed on the current OMS, as well as any patches that are currently installed on the OMS.  You must also make sure to use the same directory layout as on the original OMS.

Identifying Installed Patches

oracle$ $OMS_HOME/OPatch/opatch lsinv -oh $OMS_HOME
[...]
Interim patches (1) :

Patch  13983293     : applied on Thu Jul 11 09:56:16 EDT 2013
Unique Patch ID:  14779750
   Created on 25 Apr 2012, 02:18:06 hrs PST8PDT
   Bugs fixed:
     13587457, 13425845, 11822929

This patch gets installed by the EM12c R3 installer, so no need to bother with it any further.  If you have other patches installed, go fetch them, and install them after you have completed the plugin installation (see below).

Identifying Installed Plugins

Identify all plugins installed on your system using the query provided in the documentation, run as SYSMAN against your repository database.

SELECT epv.display_name, epv.plugin_id, epv.version, epv.rev_version,decode(su.aru_file, null, 'Media/External', 'https://updates.oracle.com/Orion/Services/download/'||aru_file||'?aru='||aru_id||chr(38)||'patch_file='||aru_file) URL
FROM em_plugin_version epv, em_current_deployed_plugin ecp, em_su_entities su
WHERE epv.plugin_type NOT IN ('BUILT_IN_TARGET_TYPE', 'INSTALL_HOME')
AND ecp.dest_type='2'
AND epv.plugin_version_id = ecp.plugin_version_id
AND su.entity_id = epv.su_entity_id;

Oracle-provided plugins will show a URL from which you must download the plugin.  Third-party plugins will not; you will need to make sure you have the appropriate downloaded plugin install .opar file from when you initially installed it.  Gather up all of these plugin files into a single directory on your NEW OMS host, changing the “.zip” filename extension to “.opar” for the Oracle-provided plugins.  You need EVERY plugin returned by this query or else your installation will NOT work.  I placed mine in /oracle/oem/migration/plugins.

You also need to copy over the three .zip files containing the OEM 12cR3 distribution: V38641-01.zip, V38642-01.zip and V38643-01.zip.  Save them into a convenient staging area on the new server (I use /oracle/oem/stage).

Perform Software-Only Installation Of EM12c R3

Go to the staging area on the new server and extract the three .zip files containing the EM12c R3 distribution, then start the installer.

oracle@newhost$ unzip V38641-01.zip ; unzip V38642-01.zip ; unzip V38643-01.zip 
[...]
oracle@newhost$ ./runInstaller

You can follow my previous post about upgrading EM12c R2 to R3 for more information about the installation process, just make sure you run it as a software only install and use the exact same path names as configured on the original OMS.  In my case this means a middleware home of /oracle/oem/Middleware12cR3 and an agent base directory of /oracle/oem/agent12c.

While the software installation proceeds, you should run an exportconfig on your current OMS to produce the configuration backup file you will need to use to reconfigure the new one.  Enter the SYSMAN password when prompted.

oracle$ $OMS_HOME/bin/emctl exportconfig oms
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password : 
ExportConfig started...
Machine is Admin Server host. Performing Admin Server backup...
Exporting emoms properties...
Exporting secure properties...

Export has determined that the OMS is not fronted 
by an SLB. The local hostname was NOT exported. 
The exported data can be imported on any host but 
resecure of all agents will be required. Please 
see the EM Advanced Configuration Guide for more 
details.

Exporting configuration for pluggable modules...
Preparing archive file...
Backup has been written to file: /oracle/oem/gc_inst/em/EMGC_OMS1/sysman/backup/opf_ADMIN_20130828_120424.bka

The export file contains sensitive data. 
 You must keep it secure.

ExportConfig completed successfully!

Copy that backup file to the new server.

oracle$  scp /oracle/oem/gc_inst/em/EMGC_OMS1/sysman/backup/opf_ADMIN_20130828_120424.bka oracle@newhost:/oracle/oem

Once the software-only install finishes, it will prompt you to run allroot.sh.  Do so.

oracle@newhost$ sudo /oracle/oem/Middleware12cR3/oms/allroot.sh 

Starting to execute allroot.sh ......... 

Starting to execute /oracle/oem/Middleware12cR3/oms/root.sh ......
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /oracle/oem/Middleware12cR3/oms

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: 
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: 
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: 

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
/etc exist

Creating /etc/oragchomelist file...
/oracle/oem/Middleware12cR3/oms
Finished execution of  /oracle/oem/Middleware12cR3/oms/root.sh ......

Starting to execute /oracle/oem/agent12c/core/12.1.0.3.0/root.sh ......
Finished product-specific root actions.
/etc exist
/oracle/oem/agent12c/core/12.1.0.3.0
Finished execution of  /oracle/oem/agent12c/core/12.1.0.3.0/root.sh ......

After running allroot.sh, you need to run the PluginInstall.sh script with the path where you saved the .opar files.  Make sure you select every plugin listed when you ran the query to retrieve the plugin list earlier, then hit install.

oracle@newhost$ /oracle/oem/Middleware12cR3/oms/sysman/install/PluginInstall.sh -pluginLocation /oracle/oem/migration/plugins

This must match the list you generated previously

Prepare the Software Library

Go to the original server, and copy the contents of the software library to the new server.

oracle$ scp -r /oracle/oem/software_library/ oracle@newhost:/oracle/oem

Recreate the OMS with OMSCA

Shut everything down on your old server.

oracle$ $OMS_HOME/bin/emctl stop oms -all ; sleep 5 ; $AGENT_HOME/bin/emctl stop agent

Run OMSCA using the exportconfig backup file you generated earlier.  Enter the administration server, node manager, repository database user and agent registration passwords when prompted.

oracle@newhost$ $OMS_HOME/bin/omsca recover -as -ms -nostart -backup_file /oracle/oem/opf_ADMIN_20130828_120424.bka
Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.3.0
Copyright (c) 1996, 2013, Oracle. All rights reserved.

OS check passed.
OMS version check passed.
Performing Admin Server Recovery...
Retrieved Admin Server template.
Source Instance Host name where configuration is exported : [deleted]
Populated install params from backup...
Enter Administration Server user password:
Confirm Password:
Enter Node Manager Password:
Confirm Password:
Enter Repository database user password:
Enter Agent Registration password:
Confirm Password:
Doing pre requisite checks ......
Pre requisite checks completed successfully

Checking Plugin software bits
Proceed to recovery
Setting up domain from template...
Setup EM infrastructure succeeded!
Admin Server recovered from backup.
Now performing cleanup of OMS EMGC_OMS1...
Now launching DeleteOMS...
OMS Deleted successfully

Delete finished successfully
Now launching AddOMS...
Infrastructure setup of EM completed successfully.

Doing pre deployment operations ......
Pre deployment of EM completed successfully.

Deploying EM ......
Deployment of EM completed successfully.

Configuring webtier ......
Configuring webTier completed successfully.

Importing OMS configuration from recovery file...

If you have software library configured 
please make sure it is functional and accessible 
from this OMS by visiting:
 Setup->Provisioning and Patching->Software Library

Securing OMS ......
Adapter already exists: emgc_USER
Adapter already exists: emgc_GROUP
Post "Deploy and Repos Setup" operations completed successfully.

Performing Post deploy operations ....
Total 0 errors, 78 warnings. 0 entities imported.
pluginID:oracle.sysman.core
Done with csg import
pluginID:oracle.sysman.core
Done with csg import
No logging has been configured and default agent logging support is unavailable.
Post deploy operations completed successfully.

EM configuration completed successfully.
EM URL is:https://newhost:7803/em

Add OMS finished successfully
Recovery of server EMGC_OMS1 completed successfully
OMSCA Recover completed successfully

Start the OMS on the new server.

oracle@newhost$ $OMS_HOME/bin/emctl start oms

Configure the central agent on the new server, then run the root.sh script.

oracle@newhost$ /oracle/oem/agent12c/core/12.1.0.3.0/sysman/install/agentDeploy.sh AGENT_BASE_DIR=/oracle/oem/agent12c AGENT_INSTANCE_HOME=/oracle/oem/agent12c/agent_inst AGENT_PORT=3872 -configOnly OMS_HOST=newhost EM_UPLOAD_PORT=4902 AGENT_REGISTRATION_PASSWORD=password
[...]
oracle@newhost$ sudo /oracle/oem/agent12c/core/12.1.0.3.0/root.sh

Relocate the oracle_emrep target to the new OMS host.

oracle@newhost$ $OMS_HOME/bin/emcli login -username=sysman
Enter password : 

Login successful
oracle@newhost$ $OMS_HOME/bin/emcli sync
Synchronized successfully
oracle@newhost$ $OMS_HOME/bin/emctl config emrep -agent newhost:3872
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Please enter repository password: 
Enter password :                                                               
Login successful
Moved all targets from oldhost:3872 to newhost:3872
Command completed successfully!
Enter password :                                                               
Login successful
Moved all targets from oldhost:3872 to newhost:3872
Command completed successfully!

Step through each of your existing agents to re-secure them against the new OMS.  Provide the OMS HTTP port (not HTTPS) in this command, and enter the agent registration password when prompted.

$ $AGENT_INSTANCE_DIR/bin/emctl secure agent -emdWalletSrcUrl "http://newhost:4890/em"
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Agent successfully stopped...   Done.
Securing agent...   Started.
Enter Agent Registration Password : 
Agent successfully restarted...   Done.
EMD gensudoprops completed successfully
Securing agent...   Successful.

Start the agent on the old OMS server.  You should not need to do this, but I could not update the WebLogic Domain monitoring configuration without doing so first.  Also re-secure this agent to point to the new OMS.

oracle$ $AGENT_HOME/bin/emctl start agent
oracle$ $AGENT_INSTANCE_DIR/bin/emctl secure agent -emdWalletSrcUrl "http://newhost:4890/em"

Login to the OEM GUI running on the new server and navigate to the WebLogic Domain target for the Cloud Control domain.  In the Target Setup -> Monitoring Credentials section, update the Administration server host value to the new server name, then hit OK.  Then execute a Refresh WebLogic Domain, selecting Add/Update Targets, to move all WebLogic targets to the new central agent.

I use third-party plugins to monitor VMWare targets, NetApp storage and MySQL servers.  I had many of them set up to run from the OMS agent (except for the VMWare ones, since Blue Medora helpfully advised not to use the OMS agent for this — great advice).  I now need to relocate each of these targets to the new central agent using emcli.  You won’t need to do this step unless you also have things set up this way.  If I had to do this again, I would not use the OMS agent for these targets, since I would not need to change anything if I just had these on some other agent.

oracle@newhost$ ./emcli relocate_targets -src_agent=oldhost:3872 -dest_agent=newhost:3872 -copy_from_src -target_name=nameoftarget -target_type=typeoftarget

Final Cleanup Steps

By now you have completed the bulk of the work necessary to migrate your EM12c stack to a new server.  Only a few steps remain.  If you use any utility scripts on the old server, go ahead and copy those over now.  I have scripts to automate starting/stopping the OMS and agent, so I’ve copied those over.  Also make sure the oracle user on the new server has all the environment variables set up in their shell initialization files.

oracle$ scp ~/bin/CCstart ~/bin/CCstop oracle@newhost:bin/

The GCDomain Oracle WebLogic Domain target did not get moved to my new agent.  If this happened to you, go to the target home page and select the Modify Agents menu item.  Click Continue, then find GCDomain in the list, scroll to the right, and assign the new OMS server’s agent as the monitoring agent for this target, then click the Modify Agents button.

Reinstall BI Publisher

Since I had BI Publisher installed on the old server, I need to install it again on the new one.  Retrieve the 11.1.1.6.0 BI Publisher installation files used previously, and copy them to your staging area.  Run the “runInstaller” program from bishiphome/Disk1, and perform a software-only installation with the middleware home set to your EM12c installation middleware home, and leave the Oracle home as Oracle_BI1.

Instead of running the configureBIP script as you normally would to integrate BI Publisher with EM12c, just go to the WebLogic administration console after the software-only install completes, and navigate to the BIP server configuration page.  Lock the configuration for editing, and edit the configuration to change the listen address to reference the new server’s hostname and change the machine to the machine name where the admin server runs (in my case it showed up as EMGC_MACHINE2).  Save and activate the changes, then start the BIP server.

After the server has started, return to the WebLogic Domain page and re-run the Refresh WebLogic Domain step, again with Add/Update targets, to move BIP to your new OMS agent.

I actually had to do the Refresh WebLogic Domain step here twice.  I may have simply not waited long enough after starting BIP before I ran it, but I do not know for sure.

Update EM Console Service

I have only one target showing down at this point, the EM Console Service.  Go to the target, and click on the Monitoring Configuration tab.  Click on Service Tests and Beacons.  Select the EM Console Service Test, and click the Edit button.  Make sure you have the “Access Login page” step selected, and click Edit.  Change the URL to reflect your new OEM server, and save the changes.

Remove Previous OMS Server From OEM

Stop the agent on your original OMS server.

oracle$ $AGENT_HOME/bin/emctl stop agent

Remove the host target where your original OMS ran.  Then remove the agent target.

One Last Bounce

Finally, bounce the whole thing one last time, then start it back up.  All green.

Conclusion

I would prefer a simpler process to migrate the EM12c stack to a new server, but this works.  If you find yourself in a similar position to mine, I hope this helps you.  I’ve spent a lot of time working in EM12c so I feel capable to diagnose and resolve issues encountered during the process, but if you run into problems do not hesitate to contact Oracle Support and file a service requests.  If you want your system to stay supportable, stick with the experts and just use blogs as a guide to get started.  Good luck.

How to connect to the default EM12c R3 self-signed WebLogic SSL port with WLST

After upgrading to Oracle Enterprise Manager 12c R3, I decided it was time to get roles configured properly for BI Publisher so that I can use it under my regular account rather than only permitting SYSMAN to access it.  Adeesh Fulay (@AdeeshF) helpfully provided me with a link to the documentation about setting up BI Publisher for EM12c.  The first step to perform the configuration involves connecting to the secured WebLogic adminserver via wlst.sh, but I immediately encountered an error:

wls:/offline> connect('weblogic', 'password', 't3s://host.domain.com:7103')
Connecting to t3s://host.domain.com:7103 with userid weblogic ...
<Jul 19, 2013 9:41:15 AM EDT> <Warning> <Security> <BEA-090542> <Certificate chain received from host.domain.com - x.x.x.x was not trusted causing SSL handshake failure. Check the certificate chain to determine if it should be trusted or not. If it should be trusted, then update the client trusted CA configuration to trust the CA certificate that signed the peer certificate chain. If you are connecting to a WLS server that is using demo certificates (the default WLS server behavior), and you want this client to trust demo certificates, then specify -Dweblogic.security.TrustKeyStore=DemoTrust on the command line for this client.> 
Traceback (innermost last):
  File "<console>", line 1, in ?
  File "<iostream>", line 22, in connect
  File "<iostream>", line 648, in raiseWLSTException
WLSTException: Error occured while performing connect : Error getting the initial context. There is no server running at t3s://host.domain.com:7103 
Use dumpStack() to view the full stacktrace

I could not find any obvious reference in the documentation on how to add the “-Dweblogic.security.TrustKeyStore=DemoTrust” options on the command line.  I attempted to just run wlst.sh with that parameter but I also received an error.

After a little searching I found a fix and figured I would post it.

In the documentation for the WebLogic 10.3.6 Oracle WebLogic Scripting Tool, section “Invoking WLST”, an example is included where it shows how to provide a different command line option to the WLST tool, by setting the environment variable CONFIG_JVM_ARGS. (EDITED 20130719: Adeesh has let me know that the preferred environment variable to use for this string is WLST_PROPERTIES, not CONFIG_JVM_ARGS.  Both work at the moment, but the documentation will be updated to refer to WLST_PROPERTIES so I advise you to use that one.)

I tried that before making my wlst.sh call, and everything worked successfully:

oracle@host:~> export WLST_PROPERTIES=-Dweblogic.security.TrustKeyStore=DemoTrust
oracle@host:~> /oracle/oem/Middleware12cR3/oracle_common/common/bin/wlst.sh 
[...]
Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

wls:/offline> connect('weblogic', 'password', 't3s://host.domain.com:7103')
[...]
Successfully connected to Admin Server 'EMGC_ADMINSERVER' that belongs to domain 'GCDomain'.wls:/GCDomain/serverConfig>

So if you are having trouble connecting to your WebLogic admin server using the default self-signed certificate via wlst.sh, this environment variable is the answer.  I was now able to proceed with granting my account access to BI Publisher, and now I am able to access BI Publisher features as needed without using the SYSMAN account.

wls:/GCDomain/serverConfig> grantAppRole(appStripe="obi",appRoleName="EMBIPViewer",principalClass="weblogic.security.principal.WLSUserImpl",principalName="USERNAME")    
Location changed to domainRuntime tree. This is a read-only tree with DomainMBean as the root. 
For more help, use help(domainRuntime)

wls:/GCDomain/serverConfig> grantAppRole(appStripe="obi",appRoleName="EMBIPAdministrator",principalClass="weblogic.security.principal.WLSUserImply", principalName="USERNAME")                                                
Already in Domain Runtime Tree

wls:/GCDomain/serverConfig> grantAppRole(appStripe="obi",appRoleName="EMBIPScheduler",principalClass="weblogic.security.principal.WLSUserImply", principalName="USERNAME")
Already in Domain Runtime Tree

wls:/GCDomain/serverConfig> grantAppRole(appStripe="obi",appRoleName="EMBIPAuthor",principalClass="weblogic.security.principal.WLSUserImply", principalName="USERNAME")
Already in Domain Runtime Tree

wls:/GCDomain/serverConfig> exit()

Exiting WebLogic Scripting Tool.

My Production EM12c Upgrade From R2 (12.1.0.2) to R3 (12.1.0.3)

This post covers my production upgrade from EM12c R2 to EM12c R3 on Linux x86-64 (SLES11 SP2).  To stress test the upgrade and keep it interesting, this system also has BI Publisher integrated into OEM, and also has the plugin from NetApp (version 12.1.0.1.0), the VMware plugin from BlueMedora (version 12.1.0.5.0), and the MySQL plugin from Pythian (version 12.1.0.1.1).  The repository database is running on version 11.2.0.3.

I’m feeling lucky today.  So this is just going straight into production.  Famous last words…

Preparation

1. Go to edelivery and search for the Oracle Enterprise Manager product pack and platform Linux x86-64
2. Follow the link titled “Oracle Enterprise Manager Cloud Control 12c Release 3 (12.1.0.3) Media Pack for Linux x86-64“
3. Download the three files needed for EM12c R3: V38641-01.zip, V38642-01.zip, and V38643-01.zip
4. Download patch 13349651 for WebLogic, you will need it during the post-upgrade steps
5. View the digest and run md5sum against each downloaded file to confirm that the files downloaded correctly
6. Transfer the EM12c R3 files to a staging area on your Oracle Enterprise Manager server and unzip all three of them
7. Delete the three downloaded .zip files if you are short on space (but don’t just “rm *.zip” or you’ll remove the necessary WT.zip file at the top level of your staging directory)
8. Review the upgrade guide
   
9. Create backups of your current OMS home, agent home, software library and Oracle inventory
10. Create a backup of your current repository database that you can restore from if necessary
11. If you use a dedicated filesystem for your Oracle Management Agents, make sure that dedicated filesystem has enough free space.  I used a small, 2GB filesystem, and this was barely large enough, except for the one sandbox server where it was too small and I was not able to complete the agent upgrade without adding space
12. Stop the BI Publisher WebLogic Server (if you have it installed) via the WebLogic Admin Console
13. Make sure your repository database does not have snapshots created on any tables by running “select master, log_table from all_mview_logs where log_owner = ‘SYSMAN'” — if any snapshots are found, follow the instructions in the upgrade guide to drop them
14. If your repository database is version 11.1.0.7 or 11.2.0.1, follow the steps in the upgrade guide to apply the prerequisite patches needed to proceed
15. Copy the emkey from the existing OMS to the existing management repository by running “$OMS_HOME/bin/emctl config emkey -copy_to_repos” and enter the SYSMAN password when prompted
16. Confirm the emkey was copied by running “$OMS_HOME/bin/emctl status emkey” and enter the SYSMAN password when prompted
17. Stop every OMS in your environment by running “$OMS_HOME/bin/emctl stop oms -all”
18. Stop the management agent monitoring the management services and repository target by running “$AGENT_HOME/bin/emctl stop agent”

Running The Upgrade

Due to some issues I had with the upgrade from EM12c R1 to R2, I highly recommend that you do NOT use Cygwin to ssh to your OMS host and display the installer over ssh using Cygwin’s X server.  Use VNC instead.  I’m not going to try Cygwin this time through.

First, I start vncserver on the OMS host.  Then I connect to it to using TightVNC from my desktop machine.

1. Navigate to the staging directory where you unzipped the EM12c R3 distribution files and run “./runInstaller” as your oracle software owner
2. As an SAP customer, we are not allowed to use OCM, so I skipped the steps involving entering my My Oracle Support credentials
3. I also skipped the search for updates as this is a new enough release at the moment there should not be any necessary
4. The prerequisite checks run.  I received a warning about libstdc++43-4.3 not being found, but libstdc++43-devel-4.3.3-x86_64 fulfills this need, so I click ignore and then Next to continue
5. Only the one system upgrade is supported when upgrading from EM12cR2, so I selected “Upgrade an existing system”, “One system upgrade”, and my existing middleware home.  Click Next
6. This out of place upgrade will go into a new middleware home.  I am using /oracle/oem/Middleware12cR3.  Click Next and the installer will confirm that you have enough free space available
7. Here I had to pause and request more space from my storage admin, as the installer wants at least 14.0GB of free space.  Once that was done, I proceeded
8. Enter the SYS and SYSMAN passwords and check the box to confirm that you have backed up the repository (you should have your OMS, agent, etc all backed up as well), then click Next
9. The installer will check various parameters on your repository database and offer the chance to fix them if any need to be changed.  I accept the fixes and click Yes
10. The installer checks some additional settings and notes that they should be reviewed after the installation or fixed now.  I explicitly granted execute on DBMS_RANDOM to DBSNMP and then clicked OK
11. The installer lists the plugin versions that will change and the plugins that will migrate. Confirm this all looks right and then click Next
12. The installer lists additional plugins you can choose to deploy at install time.  I do not use any of these so I left them all unchecked and clicked Next
13. The installer requests the password for your WebLogic adminserver and confirmation of the hostname, port and username.  Provide the password and click Next.  You may be able to change the OMS instance directory here but I do not suggest doing so
14. You now have a chance to review your settings, then click Install to proceed with the upgrade
15. Installation proceeds
16. You are then prompted to run the allroot.sh file.  Login to the server and execute it as root or via sudo
17. Once the install/upgrade is complete, the installer will display an installation summary.  Review it, save the URLs it gives you for the OMS and adminserver, then click Close

Overall, the upgrade installation steps took 1 hour and 15 minutes in my environment.  This was on a physical server with 126GB RAM, 16 dual core processors and 200 managed targets.  This does not include the post-upgrade steps shown below.

Post Upgrade Steps

Now that the upgrade is complete, return to the upgrade guide to complete post upgrade steps.  Your environment may differ from mine, but these are the steps I had to follow.

1. Start your central agent by running “$AGENT_HOME/bin/emctl start agent”.  At this point the load on my system went up very high and began responding very slowly.  I walked away for 10 minutes to let things settle down
2. Open your web browser and go to the URL provided at the end of the installation and login as SYSMAN.  When I first tried to do so using Firefox I received an error indicating an invalid certificate.  I had to delete the old certificates and authorities from my previous installation and restart Firefox before it would allow me in.  MSIE worked fine though
3. Update the central agent (the management agent installed on the OMS host) by clicking on the Setup menu, then Manage Cloud Control, then Upgrade Agents.  Click the Add button and select your central agent.  I choose “Override preferred credentials” since I have not configured sudo.  Click Submit to continue, then OK when warned that you may have to run root.sh manually
4. My first upgrade attempt on the central agent failed due to a prerequisite check for package libstdc++-43.  The easy thing to do here is expand the Additional Inputs region and provide “-ignorePrereqs” as an additional parameter, but I chose to complete this agent upgrade using emcli and describe that process in the next two steps
5. First run “$OMS_HOME/bin/emcli login -username=sysman” and enter the SYSMAN password when prompted.  Then run “$OMS_HOME/bin/emcli sync”
6. Upgrade the agent by running “$OMS_HOME/bin/emcli upgrade_agents -additional_parameters=”-ignorePrereqs” -agents=”hostname.domain.com:3872″
7. Wait while the upgrade proceeds.  You can view upgrade progress by running “$OMS_HOME/bin/emcli get_agent_upgrade_status”, or in the GUI by clicking on “Agent Upgrade Results” in the “Upgrade Agents” page
8. Click Done once the central agent upgrade completes.  Go to the new agent home and run root.sh as root or via sudo
9. Then repeat this process for the rest of your agents.  Try to install them first WITHOUT using the “-ignorePrereqs” flag, because if there are missing prerequisites you need to identify the issue and find out if it is something that it is appropriate to ignore, as the libstdc++ version was in my case.  Execute root.sh for each agent afterwards, unless you have sudo configured in which case it will happen automatically
10. Two of my agents that run on different platforms could not be upgraded right away.  The new versions of the agent software needed to be downloaded from Self Update.  I am skipping them for now
11. Next, apply patch 13349651 to WebLogic, following the instructions in the README file.  I attempted to do so, but the patch was already installed so I skipped this step
12. There are a few other optional, post-upgrade steps like deleting obsolete targets.  These are documented in the upgrade guide and I will not note them here
13. As a final step, make sure you update your Oracle user’s environment variables to reflect the new middleware home, OMS home, agent home, and so on

Conclusion

At this point my EM12cR3 production upgrade is complete!  Everything I have checked so far appears fully functional.  The only problem I had was a small filesystem for the management agent on my sandbox server causing the agent upgrade to run out of space, forcing manual intervention to resolve.  Don’t be stingy with space like I am and you should be fine.

I haven’t taken any time to investigate the new features yet, but I will be now.

(EDITED TO ADD: I forgot to mention the steps to get BI Publisher working again.  Please refer to the EM12cR3 Advanced Installation and Configuration Guide, chapter 15.  Essentially you will need to perform a software-only installation into the new middleware home, then execute the configureBIP script with an -upgrade flag to complete the BIP setup.)

Using EM12c Compliance Rules, Standards, and Frameworks

I recently reviewed SAP note 740897 and discovered that the application-specific full use license SAP customers receive when they purchase the Oracle database through SAP includes the Database Lifecycle Management Pack.  This means I can make use of, among other things, the compliance checking capabilities provided by Oracle Enterprise Manager 12c.

Many of the posts I put up here serve as “how to” documents, explaining how I do something so that others can decide how they would like to do something.  This post is slightly different.  I will be describing how I currently use the compliance rules, but in addition to simply providing a “how to”, this is more of a plea for anyone who finds this to tell me how this can be done more easily and efficiently.  The compliance functionality in EM12c appears to be much more configurable than that provided by EM11g, but one key piece that existed in EM11g appears to be gone. That key piece is the ability to ignore/suppress a particular key value from a compliance check. I would love to have someone tell me that I’m just not finding that function in EM12c.

As I recall, in EM11g, when you had compliance checks enabled you could ignore a single key value.  As an example, perhaps you had the rule to flag users with access to select from DBA_* views. That is great, except that my account has the DBA role, so my account appeared as a violation.  But I had the ability to ignore any violations on that rule where the key value was my account name.  This does not seem to be the case with EM12c.  Hence this post, where I describe how I’m achieving similar functionality in a very different way, hoping someone else knows a better way to do it.

Getting Started

The first step to using the EM12c compliance functionality for your databases is to have a license for the Database Lifecycle Management Pack.  If you don’t have one already, contact your Oracle sales representative.  Note that if you purchased your licenses before Oracle 11g was released, you may have a license to some retired management packs such as the Configuration Management Pack, Change Management Pack, or the Provisioning and Patch Automation Pack.  These three legacy packs combined seem to provide most/all of the functionality included in the Database Lifecycle Management Pack and according to the EM12c documentation grant you a license to use the functionality provided by the Database Lifecycle Management Pack.  Don’t take my word for it, review the Oracle Enterprise Manager Licensing Information document, particularly sections 2.3, 2.6, 2.7 and 2.8, then consult with your sales contact if you have questions.

Once you have confirmed your entitlement to use this feature, enable the Database Lifecycle Management Pack in EM12c as follows:

1. Login to EM12c as the repository owner (SYSMAN)
2. Navigate to the Management Pack Access screen via the Setup menu, then the Management Packs submenu
3. If not selected already, select the “Target Based” Pack Access radio button
4. If not selected already, select “Database” from the search drop-down
5. Click the Go button
6. Check the box in the Database Lifecycle Management Pack column for each database where you have this pack licensed and then click the Apply button

Management Pack Access screen

This setup step enables the compliance functionality, but to make use of it you will need to first enable collection of some additional information about your databases, then “attach” your database targets to a “compliance standard”.

Collecting Data Needed For Compliance Monitoring

Presumably to reduce load on systems where people don’t use the compliance functionality, EM12c does not collect the information needed to make full use of the compliance standards out of the box.  You need to enable this collection.  To do so:

1. Click on the Enterprise menu, then the Monitoring submenu, then Monitoring Templates
2. Check the box next to “Display Oracle Certified templates”
3. Click the Go button
4. Select the radio button next to “Oracle Certified-Enable Database Security Configuration Metrics”
5. Click the Apply button
6. On the next page, click the Add button to select the database targets for which you will use the compliance functionality
7. Click the OK button
8. Repeat these steps for the “Oracle Certified-Enable Listener Security Configuration Metrics” and your listener targets if you intend to monitor listener compliance

Applying out-of-box templates to enable security configuration metrics

Compliance Frameworks vs Compliance Standards vs Compliance Rules

EM12c uses a three-tier approach to compliance monitoring.  For a full understanding of how this works you should read the Oracle Enterprise Manager Cloud Control Oracle Database Compliance Standards documentation, but to summarize it briefly a compliance rule checks a particular compliance item (like permissions on a certain file, or a specific database role), while a compliance standard groups multiple compliance rules into a set to which you then attach the targets you want to have monitored.  A compliance framework then groups multiple compliance standards into a superset for reporting/auditing purposes.  This gives you a single view of your overall compliance when you have multiple compliance standards applying to different target types, as a compliance standard only applies to one target type — that is, you use a separate compliance standard for your listeners than for your databases, but you then include both standards in your compliance framework for a view of your entire environment.  EM12c comes with a large number of pre-built compliance rules, standards and frameworks which you can use as-is if you wish, but read on to find out why I prefer to customize them.

Working With Compliance Standards

To get started with compliance standards, click the Enterprise menu, then the Compliance submenu, and then click on Library.  This will take you to a screen with tabs to move between compliance frameworks, standards and rules.  For your first foray into compliance checking, start with one of the simpler Oracle-provided templates, like the “Storage Best Practices for Oracle Database” applicable to Database Instance targets.  To find it, click on the Compliance Standards tab, then the little triangle next to the word “Search” at the top of the screen.  Type “Storage Best Practices” into the Compliance Standard field, and select Database Instance from the Applicable To drop down, then click the Search button.  Once you see that standard on your screen, click on that row of the table (NOT the name of the standard), then click the “Associate Targets” button.  This will bring up a screen where you can then click the ‘Add’ button to select one or more of your database instances to attach to the standard.  After adding a target, click the OK button.  One more pop up window will appear asking you to confirm that you are ready to deploy the association, go ahead and click Yes on this screen.

Searching for a compliance standard and associating targets

You now have at least one target associated to a compliance standard.  So what now?

Viewing Compliance Results

Once you have a target associated to a compliance standard, the main Enterprise Summary page will show an overview of the compliance check results along with a list of your least compliant targets.

Compliance region on Enterprise Summary page

The Compliance Summary region also has a Compliance Frameworks tab which provides another way of viewing the same information — further down I will cover how to set up a framework.

Compliance Summary region, Compliance Framework tab on Enterprise Summary page

For another view, you can also use the Compliance Dashboard, through the Enterprise Menu, Compliance sub-menu, and then clicking on Dashboard.

Compliance Dashboard

Compliance violations are grouped into minor warnings, warnings, and critical violations, based on the configuration of each compliance rule contained in a standard. Depending on your needs, you can change the significance of a violation as appropriate for your environment.  I will cover this later as well.

To get some more information about the specific violations Enterprise Manager has found, click on the name of your compliance standard from one of those screens and you will see some more details about what is contained in the compliance standard and the status of your targets.  For further detail, click on the name of a compliance rule on the left-hand side.  Pardon the blurred text in these images, I have already customized some rules and standards and included my employer name, which I highly recommend doing to distinguish your customizations from the out-of-the-box configuration.

View of compliance standard check details

Drill down into compliance rule details

This page shows that of the three database instances I have associated with this compliance standard, I have only one violation, and that violation is a minor warning associated with the “Non-System Data Segments in System Tablespaces” compliance rule.  Because SAP requires that users create some particular segments in the SYSTEM tablespace, this is a good one to work through as an example to show how to customize compliance monitoring to fit your environment.

Customizing Compliance Monitoring

There are a few different ways to customize your compliance monitoring beyond the high-level decision of which specific targets you associate to each specific standard.  One way is to create your own compliance standards, selecting and excluding the compliance rules that are not relevant in your environment — this way, for example, you can complete disable the check for “Non-System Data Segments in System Tablespaces” if you choose to (I wouldn’t, but you might want to).  Another way is to customize the specific compliance rules contained in your compliance standards.  I do both.

I highly recommend not attempting to edit any of the Oracle-provided compliance frameworks, standards, or rules.  The “Create Like” button in the compliance library will be very helpful to you here.

The “Create Like…” button is your friend

First create your own compliance standard by selecting an existing one (I’ll continue to demonstrate this with the “Storage Best Practices for Oracle Database” standard) and clicking on the “Create Like…” button.  EM will prompt you to provide a name for the new standard.  For simplicity I prefer to use some indicator like my employer’s name followed by the name of the original standard.  Click Continue once you have named your new standard and you will proceed to the compliance standard editing page.

Here you specify the rules to include or exclude from your compliance standard

From this page you can add or remove compliance rules from your newly-created compliance standard.  To remove a rule, right-click on it in the region on the left and choose “Remove Rule Reference”, then click OK.

You can remove individual rules or groups of rules from this screen

The rules in the predefined standards are grouped into “rule folders”.  Instead of removing a single rule, you can remove an entire rule folder if you wish by right-clicking and selecting “Remove Rule Folder” and then clicking OK.  You can also create a new rule folder by right-clicking on the name of the compliance standard on the left and selecting “Create Rule Folder”, providing a name, then clicking OK.

Add or remove rule folders to group compliance rules

The compliance standard we’re working with has only a few rules.  If you wish, you can add one of the many other rules that are contained in other compliance standards.  Right-click on the compliance standard name or a rule folder, and select “Add Rules”.  A screen will appear allowing you to select one or more rules to add to the standard.  You can scroll through to select your rules or search by name or keyword.  Once you click OK, the selected rule(s) will be added to your compliance standard.

Select as many rules to add to your standard as you wish

The compliance standard editing screen is also where you can change the importance of a compliance rule violation.  To change the importance of the “Insufficient Redo Log Size” rule from “Normal” to “High”, click on that rule, then the drop-down box next to “Importance” and select a new value.

I guess “Low”, “Normal” and “High” correspond to “Minor Warning”, “Warning” and “Critical”

Finally, click the Save button to save your new compliance standard.  At this point your new standard will not have any targets associated with it, so you should click on it and then on the “Associate Targets” button to do so.  You may also wish to remove the association of those targets with the original standard you used to create this new standard.  Once you finish in this screen, you can return to the Enterprise Summary or Compliance Dashboard, refresh the page, and you should see the results of the checks run by this new rule.

Changing A Compliance Rule

That is all useful, but what if you want to change the actual details behind a rule?  I want to get eliminate the complaints about non-system data segments in the system tablespace so that I don’t see any more violations for the SAP-required segments I have in there, but I don’t want to remove the entire rule because I do want to be notified if other segments show up in there that I wasn’t aware of.  The solution is create a new rule based on the rule you want to change, edit it (finally we get to write some SQL) and then remove the old rule from your compliance standard and replace it with the new rule.

Go back to the Compliance Dashboard and click the Compliance Standard Rules tab.  Open up the search widget and search for “Non-System Data Segments” for target type “Database Instance”.  Click on the offending rule and then the “Create Like” button.

The lock icon shows that you can’t edit the default rules but you can duplicate them

Provide a title for your new rule following whatever scheme you like.  I will call it “DEMO Non-System Data Segments in System Tablespaces”.  Click Continue and you will see the edit screen for Compliance Standard Rules.

You can change the text here if you wish, or add keywords

Click Next to go to step 2 where you can edit the rule SQL.

Finally, SQL!

This screen allows you to edit the rule SQL.  If you aren’t familiar with the EM12c repository, this can be difficult.  I recommend pulling up a SQL*Plus window connected to your repository database as SYSMAN, then copy/pasting the SQL text into the query window so that you can see the results that it returns.  In my case I want to exclude violations for the “SAPUSER” table that SAP requires us to create in the SYSTEM tablespace, so I just add the text “and OBJECT_NAME not like ‘%SAPUSER%’” to the end of the SELECT statement.

Anything you can do in SQL, you can do here

Click Next once you have edited the SQL to your liking.  This will bring you to a new screen where you specify the key values and violation conditions.  This is one of the clunky parts of working with compliance rules, because the predefined violation condition is lost when you “Create Like” on a built in rule.

What now?

If you just proceed with finishing the rule from here, you’ll have a problem.  Every single segment in the SYSTEM and SYSAUX tablespaces will be flagged as a violation.  You need a where clause.  But what should it be?  What was it in the original rule?  Here I typically open up a second browser window, navigate to the original rule in the Compliance Library, click the “Show Details” button and then scroll down to the bottom, which brings up the following screen:

At least there’s a way to get the configuration of the original rule

The lucky part here is that, even though the area is grayed out, you can select and copy the text from the original rule’s where clause, then paste that into your new rule’s where clause, as shown below.  I’ve also checked the “Key” checkboxes for TABLESPACE_NAME, OBJECT_OWNER, and OBJECT_TYPE, because I suspect (but haven’t yet confirmed) that these key values determine how many individual violation events you will receive.

You can always re-edit this later if you don’t get it perfectly right the first time

Once you click Next on that screen you’ll be presented with step 4, where you can test your new compliance rule against a specific target.  You can type in the target’s name or click the magnifying glass to select the target, as with the other target selection screens in EM12c.  Click Run Test after you have selected and target and confirm that the results you see are the results you wanted.

Run tests against all your targets one at a time to see what will happen when your rule goes live

If you are satisfied with the test results, click Next.  Otherwise click Back and try again with your SQL code and where clause.  Once you click Next you will see step 5, which is just a summary page displaying your rule’s details.  Click Finish when you are done.

All done, can I go home now?

Now that you clicked Finish, your new compliance standard rule is saved in the repository and available for use.  You will need to attach it to a compliance standard, as described above, before it will do anything useful, and you probably want to detach the original rule that you used as the source to create this one.

Repeat these steps for every rule you wish to edit.  This is the part I referred to at the beginning of the post where I hoped someone can suggest a better way.  As I recall, in EM Grid Control 11g, an admin could simply select a specific compliance violation and choose to suppress it for that key value with a couple of clicks, as compared to this long process needed to duplicate and edit a rule.  EM12c compliance rules are very customizable, just not quite as easy to work with — sort of like incident rules and notifications.  You need to learn a new way of doing things, but it can do a lot.

Creating A Compliance Framework

Finally, you should create a custom compliance framework.  This follows essentially the same process as creating a standard and attaching rules, but instead you create a framework and attach standards.  Go to the Compliance Frameworks tab on the Compliance Library page and click “Create”.  Give your framework a name and click Continue, and the Compliance Framework edit screen should look familiar.

Where have I seen this before?

Right-click on the compliance framework’s name in the left bar, and select “Add Standards”.  A screen will pop up from which you can select the standards you created previously, just like when you add a rule.  You can also add standard subgroups, which work much like rule folders.  Click on your new standards and then OK.

Easy enough, right?

Click Save and you’ll be returned to the framework tab.  At this point your new framework is in “Development” state, and you will NOT see it in the Enterprise Summary page.  Click on the framework, then click “Edit”.  Change the Compliance Framework State to Production and click Save.

Finally done!

You’re done!  You now have a custom compliance framework, one or more custom compliance standards within that framework, and several rules in your standards, including some you have edited to meet your needs.  Go back to the Enterprise Summary page, wait a minute or two, click the refresh button and then admire your work.

Time for a cold beer…

Conclusion

The compliance functions in EM12c are extremely customizable and capable.  There are a some rough spots where I prefer EM11g’s functionality, and a couple spots where I need to open another browser window or SQL*Plus connection to get things set up the way I want, but that’s a small inconvenience compared to their power.

So now that you have these compliance evaluations staring you in the face every time you visit the Enterprise Summary page, get to work fixing those violations!

(EDITED: 20130903, typos fixed)

Stale EM12c patch recommendations? Get patch 14822626

I don’t use the automated patching functionality provided by EM12c.  I do, however, get value out of the patch recommendations since they serve as a good reminder when I’ve missed a patch that should be applied to one of my targets.  For this reason I was disappointed when, after upgrading from EM12cR1 to EM12cR2, the patch recommendations it gave me became stale and stopped getting updated when I loaded the em_catalog.zip file.

If you DO use the automated patching functionality, you have probably already followed all of the advice and installed the required patches documented in MOS note 427577.1, “Enterprise Manager patches required for setting up Provisioning, Patching and Cloning (Deployment Procedures)”.  In that case you already have this patch installed and don’t need to read any further, but if not, read on.

After upgrading to EM12cR2, I also upgraded several databases from 10gR2 to 11gR2.  Months passed, and yet the patch recommendations EM12c gave me continued to refer to 10gR2 patches which I knew weren’t applicable as I was running 11gR2.  I tried several things, like setting EM12c to offline mode, to online mode, loading em_catalog.zip, re-running the various “Refresh From My Oracle Support” jobs, all without ever receiving fresh patch recommendations.

So to sort this out, I did what I usually do, and asked about it on Twitter.  Big thanks to Sudip Datta, Vice President of Product Management at Oracle, who pointed me to bug 14822626 and its associated patch.  The bug does not appear to be public, but MOS note 1522918.1, “12C – Patch Recommendations Not Updating After Upgrade To 12.1.0.2 Cloud Control – ‘…Patch Recommendations Computation is disabled … skipping …'” documents the problem as a known issue after upgrading from 12.1.0.1 to 12.1.0.2 and clearly matches the behavior I saw.

As soon as I applied patch 14822626, the old stale patch recommendations were cleared out, and once I loaded the current em_catalog.zip file, I had accurate patch recommendations for my environment that I can now use to make decisions going forward.

Thank you, Sudip!

Using EM12c to set up a Data Guard physical standby database

This post will cover using EM12cR2 to create a Data Guard configuration including one primary database and one physical standby server, making use of the Data Guard broker.  The application software we use does not support logical standby databases so I have not attempted to do so and will not document that here.

Prerequisites

As this post focuses specifically on creating a new Data Guard configuration, I will assume you have an existing functional EM12c environment in place along with two servers to use for Data Guard and an existing database which you wish to protect running on one of those servers.

Both servers should exist as promoted targets within EM12c.  The existing database (along with its listener and ORACLE_HOME) should exist as promoted targets within EM12c. The standby server should have a software-only installation of the same version and patch level of the Oracle Database (Enterprise Edition only) as exists on the primary server. For simplicity I suggest using the same filesystem paths on both servers, although Data Guard does allow rewrite rules if necessary.

Note that Data Guard is a feature included with an Enterprise Edition license but running a physical standby will require a license for the database software on the standby server.  Contact your sales representative for full details.  For the purposes of this post I will assume you are using a copy of the database downloaded from OTN for prototyping purposes.

Configure the database as you wish.  One point I recommend is to make sure that your redo logs are appropriately sized and that you have enough of them, as adding or resizing redo logs after Data Guard is operational requires some special care.

Adding A Physical Standby

Now that your environment is set up with a working EM12c installation and one active database that you wish to protect with a Data Guard physical standby, you can proceed. Start by going to the database home page and select ‘Add Standby Database’ from the drop-down menu under ‘Availability’.

Step 1

On the next page, select ‘Create a new physical standby database’ and click continue.

On the next page you select a method to instantiate the physical standby database.  Select ‘Online Backup’ and ‘Use Recovery Manager (RMAN) to copy database files’, then click Next.

On the next page you specify the degree of parallelism for the RMAN backup, provide operating system credentials for the user owning the Oracle installation on the primary server (oradgd, in my case) and define the locations of the standby redo logs.  The degree of parallelism is up to you and depends on how many CPUs you have and how quickly you need the backup to run.  I specify new named credentials here and save them as preferred database host credentials.  I recommend clicking the ‘Test’ button to validate the supplied credentials.  I do not use Oracle-Managed Files so I have unchecked the box to use them for standby redo log files, which allows me to specify a location for the standby redo logs if I do not like the default.  I left these locations at their default.  After making your entries on this page, click Next.

On the next page you will specify configuration details for the standby database.  All entries on this page relate to the STANDBY server, not the primary.  Enter the hostname of the standby server and the path to the Oracle home installation you will use for the standby database.  Enter credentials for the Oracle home’s software owner, and again I recommend saving them as preferred credentials and clicking the Test button.  The instance name you provide must not already exist on the standby server.  I used the same instance name on the standby as on the primary.  Click Next after entering all required information.

The next page allows you to select the file locations for the standby database and define the listener parameters.  I want to keep things simple with my standby using the same file paths as the primary so I select the radio button labeled “Keep file names and locations the same as the primary database“.  If you wish, you can click the ‘Customize’ button and specify alternate file locations for data files, control files, temp files, directories and external files, but keeping everything identical between the two servers will simplify things greatly. I will also use the default listener name and port.  Click Next once you have made your selections here.

On this page you specify some final parameters for the standby database such as the DB_UNIQUE_NAME, the name EM12c will use for the standby database target, the location for archived redo log files received from the primary, the size of your FRA and the deletion policy for archived redo log files.  For the best monitoring experience, check the ‘Use SYSDBA monitoring credentials’ box.  I also suggest you leave the option checked to use the Data Guard Broker. Click Next once you have made your selections here.

The final page you see here will show a review of the settings you have selected through the process.  You can see here that I am setting up a standby on Oracle Enterprise Linux while my primary runs on SUSE; this is not a problem for Data Guard. Double check everything to make sure it is all correct, and once you are satisfied, click the Finish button.

As soon as you click Finish on the previous screen, EM12c will start setting up your standby database.  You should quickly see a screen like the one below showing that a job has been submitted to create the Data Guard configuration.

If you click on the ‘View Job’ text, you will see the execution log from the job run.

To monitor the job as it proceeds, you can click on the ‘Expand All’ text and then set an auto-refresh interval from the drop-down at the top right. Depending on the size of your database and your server performance, and assuming everything went well, you should soon see that the job has completed successfully.

Validating Data Guard Configuration

Once you see the setup job has succeeded, your Data Guard physical standby is now up and running, actively processing redo from your source database.  You can verify this by returning to the primary database’s home page and clicking the ‘Availability’ menu, which now has additional options such as ‘Data Guard Administration’, ‘Data Guard Performance’ and ‘Verify Data Guard Configuration’.  Click on ‘Data Guard Administration’

The Data Guard administration page shows a summary of your setup.  You can see the host running the primary database, the status of your standby(s) and various metrics like the current and last-applied archived log numbers.  The various links on this page can then be used to change the protection mode, enable/disable fast start failover and so on.  You can also use the ‘Failover’ and ‘Switchover’ buttons to initiate a role transition.  Read the documentation so that you understand the difference and know which to use in which situations.

To help convince yourself that all is working properly, set the auto-refresh interval to 30 seconds and leave this page up.  Open a sqlplus session on your primary database as sysdba and run “alter system switch logfile”.  You should see the log numbers increment once the refresh interval has passed, as shown below.

As a final test, attempt a switchover operation.  This will leave your current primary database running as a standby, while your current standby database takes over the primary role.  Click on the ‘Switchover’ button. Here you are prompted for credentials on the standby database, which is why I suggested saving them as preferred credentials during the setup process.  If you did not do so then, provide appropriate credentials now, then click Continue.

Next you’ll be prompted for credentials for the primary server.  Provide those credentials, if necessary, and then click Continue.

Next you will have one final screen to click through to start the switchover process.  There is a checkbox to choose whether or not you want to swap monitoring settings between the primary and standby databases.  I check the box as this is a good thing, but as the text says you have the option to NOT swap the monitoring settings and instead use your own detailed monitoring templates for each system and apply them after the switchover.  I prefer to keep it simple. Once you are ready to go, click Yes, but be aware this will disconnect any sessions active in your primary database.

You will see a progress screen as the switchover occurs.

Once the switchover completes EM12c will return you to the Data Guard Administration page, where you should see that your primary and standby servers have switched roles.

Conclusion

If you have been following along, you now have a functional Data Guard system with a physical standby and have successfully completed one switchover operation.  You can repeat this process to add another physical standby database on a third server if you wish.  As you look around you’ll also notice a few other changes, such as the additional targets that EM12c added for the standby database, or that the Databases list view has some extra text added that indicates which instance is running as primary and which is running as a standby.  Now it’s time to research your needs for Data Guard and get all the remaining bits configured to best support your users. Good luck!

First Thoughts: BlueMedora’s Oracle Enterprise Manager Plugin for VMware

This post reviews the Oracle Enterprise Manager Plugin for VMware from BlueMedora.  This commercial (for-fee) plugin integrates into OEM 12c to provide visibility into your VMware environment for Cloud Control users.  I have only had the plugin installed for two days so this will serve as more of a “first thoughts” report than as a full review of the product and all of its capabilities.

Prerequisites

The plugin, license and support are available directly from BlueMedora.  They are also currently offering a fully-functional 30-day free evaluation copy.  You will need to meet the following prerequisites in your environment:

1. An active, functional installation of Enterprise Manager 12c (I used 12.1.0.2.0 + PSU2, but any EM12c release should work)
2. A functional emcli installation.  I run emcli out of the $OMS_HOME on my OMS server.
3. An OEM user account with appropriate permissions to import and deploy plugins (I simply used SYSMAN)
4. An Oracle Management Agent installed on some server other than your OMS server to host the plugin (per BlueMedora’s recommendation, which I followed)
5. A login to your VMware environment with read-only access to vCenter, your cluster, datastores, ESX hosts, and virtual machines.  If you use an account with permission to start/stop VMs that functionality is available for use in EM12c via the plugin, but I am using a read-only account and have not tested that part of the product
6. At install time you will also need the hostname of your vCenter server and the SDK port if it has been changed from the default 443.

Installation and Configuration

After downloading the plugin from BlueMedora, installation is as simple as any other plugin you may already be using.  Their support team will walk you through the install and configuration (over a webinar in my case), but the steps are just what you would expect: copy the plugin .opar file to a location accessible by your OMS and import it using the emcli import_plugin verb, then after the import completes, deploy the plugin to your OMS and finally deploy the plugin to a management agent.

Configuring the plugin (adding targets) is also simple.  For the first step you will run an  “Add Target Manually” step with the “VMware vSphere” target type to let the plugin know about your vSphere environment.  Here is where you will provide your VMware login credentials, hostname and port (if non-default) in order to begin monitoring.  After adding the vSphere target, access it through the All Targets view, and go to the “All Metrics” link to validate that your VMware login credentials worked and metrics are being collected.  As an aside, a particularly interesting feature here that I have not seen in other plugins is that there is a metric group called “Collection Trigger” that, when clicked, triggers a collection.  This is handy and I would like to see this implemented elsewhere; I find it much easier than going over to an agent and running emctl to force a metric collection.

Once that is done you will see a new auto-discovery module called “vSphere Discovery Module”, and after configuring that discovery module with the name of your vSphere target you will run auto-discovery on the management agent to which you have deployed the plugin.  Auto-discovery identified our VMware cluster, ESX hosts (hypervisors), datastores and all of our virtual machines.  From the auto-discovery results you then promote any targets you wish to monitor through Enterprise Manager.  You may not want to promote all of your VMs (for management reasons, or to comply with your license terms) but you should promote all clusters, hypervisors and datastores, for the best overall view of your environment.

As with the other plugins I’ve used, after promoting targets you’ll need to wait a little while before metrics are collected and screens populate with data.  In my case I had useful data appearing on screen within about 10 minutes after promoting the first VM.

What’s In It For Me?

For a good high-level overview of what the plugin can do, take a look at the white paper from BlueMedora, the product overview PDF and the product datasheet.  I’ll only cover some of the additional items that these sources do not go into in detail.

Visuals

The plugin provides several nice visualizations.  On top of the “CPU and memory usage over time” graphs you would expect to find, I particularly like the bar graphs that group your hypervisor CPU and memory load into quantiles, making it easy to see, for example, that one host has 75-100% CPU usage while the other hosts are all in the 0-25% bucket, indicating that you may want to allocate your VM load a bit more evenly.  The datastore visualization showing the fill percentage on each datastore is also nice. Samples of these can be found in the product overview PDF.

Another useful visual is the integrated view of an Oracle database along with the VM on which it runs, the datastore(s) assigned to that VM, and the hypervisor on which the VM is running.  You can quickly see if, for example, the hypervisor is under memory pressure even if the VM does not appear to be, along with the executions and IOPS per second and average active sessions metric for the database.

Centralized Data

I find it hard to click through the vSphere client to find information I’m looking for.  I’m always in the wrong inventory view, or seeing only a subset of the data since I have a host selected rather than the entire cluster, and so on.  This plugin provides an easy to use centralized view of information across the VMware environment.  By going to the “Virtual Machines” view from the main vSphere target I can see a data grid showing each VM’s power status, provisioned disk, consumed CPU and memory, guest memory usage percentage, Even better, the grid includes a column indicating whether or not VMware tools are installed and running. There’s also an uptime column but I’m not sure how to parse it.  I think it represents the VM’s uptime on the specific hypervisor currently running it, but I’ll be asking support to clarify that for me.

New Job Types

These will only be useful to you if you want to automate your VMware environment from within OEM, and if you grant permissions beyond read-only to your monitoring user.  I do not expect to make use of this feature. But if you choose to do so, the plugin adds several new job types you can use for OEM jobs:

• vSphere Hypervisor Enter Maintenance
• vSphere Hypervisor Exit Maintenance
• vSphere VM Power Off
• vSphere VM Power On
• vSphere VM Reset
• vSphere VM Restart Guest
• vSphere VM Shutdown Guest
• vSphere VM Suspend

Metrics/Alerts

What use is an OEM plugin without metrics and alerting?  Very little.  This plugin provides a ton of metrics for your VMware environment.  The list is too long to include here, but see this on pastebin for a quick view produced from the MGMT_METRICS table.  You can also set warning and critical thresholds for many (although not all) metrics, and those alerts will go through the normal EM12c event framework to create incidents and/or notifications if configured to do so through incident rules. You can also view the same metric-over-time graphs as you can with the out-of-the-box EM metrics.

Other Items

The overview states that the plugin includes some integration with BI Publisher for reports.  I do not have BI Publisher installed with my EM12c environment so I can’t speak to this feature.

Disclaimer

I am not employed by BlueMedora, VMware or Oracle and neither I nor my employer received any consideration or compensation from those vendors.

Why your EM12cR2 FMW stack probably needs patch 13490778 to avoid OHS down/up events

MOS note 1496775.1 describes a situation with EM12cR2 where OEM will falsely report the Oracle HTTP Server instance (ohs1) as down, even though it is up.  This is due to some changes in FMW 11.1.1.6.  If you don’t have any incident rules or notifications set up that would catch this event, it’s easy to miss it and not know that it is happening.  I had run into this note a couple times before but ignored it, since I had never seen any open events complaining about OHS being down so I figured I just wasn’t hitting the bug.

This morning I caught one of the events.  I found myself wondering how often this had been happening — was it an issue once every couple days, every few hours, or what?

SQL> col msg format a45
SQL> select msg, count(*) from sysman.mgmt$events
  2  where closed_date >= sysdate - 1 and msg like '%HTTP Server instance%'
  3  group by msg;

MSG                                             COUNT(*)
--------------------------------------------- ----------
CLEARED - The HTTP Server instance is up             430
The HTTP Server instance is down                     430

Turns out it had been happening a LOT.  If you’ve followed Oracle’s recommendations and set up target lifecycle status priorities (see my post on doing so) you’ve probably set your OEM targets up with “MissionCritical” priority.  That means your OMS has been burning a lot of CPU to process all these up/down events on a mission critical target with high priority, potentially delaying processing of other events elsewhere in your events.

Applying patch 13490778, with ORACLE_HOME set to $MW_HOME/oracle_common should resolve this issue.  For best results, stop all OEM components prior to patch application and restart them when complete.

To convince yourself that applying the patch helped, re-run that query about 15 minutes after applying the patch and you should see the count decrease.

EM12cR2 PSU1 (12.1.0.2.1) Patch 14840279 Now Available

I just noticed that the first PSU for EM12cR2 is out. It’s under patch number 14840279, and gives us a new version for the EM12cR2 setup: 12.1.0.2.1.

I’ve applied this patch without any trouble. I did so on top of the Dec 2012 performance bundle (patch 14807119). The PSU is a superset of the performance bundle so some patches were rolled back, but everything applied cleanly and my OMS came up fine. It also includes the fix for the EM_JOB_METRICS issue I posted about before so if you aren’t comfortable applying one off patches and have been tolerating the increased redo while waiting for a bundled patch, this PSU is for you.

The only minor issue I had was in the post-patch application step when running post_deploy.sh. The file wasn’t executable so I simply had to chmod +x post_deploy.sh before running ./post_deploy.sh.

EM12cR2 ORA-20233: Agent targets cannot be directly blacked out

Here’s an annoying change Oracle implemented in EM12cR2. You can no longer directly black out an agent target through the OEM GUI interface. Attempting to do so will result in an unhelpful error message on the GUI stating simply: Error editing the blackout “Blackout-Jan 14 2013 10:13:59 AM”. This worked in EM12cR1 and I made use of this functionality very frequently, and I’ve only now dug in a bit to figure out why.

Apparently Oracle has decided to remove the ability to black out an agent target. You couldn’t tell that from the error on the screen, but OEM follows up that error by generating an incident and a problem, and picking through the incident packaging directory you can see the actual error message:

[2013-01-14T10:12:43.245-05:00] [EMGC_OMS1] [ERROR] [EM-02916] [oracle.sysman.eml.admin.rep.blackout.BlackoutEvents] [host: omshost.domain.com] [nwaddr: x.x.x.x] [tid: EMUI_10_12_43_/console/admin/rep/blackout/blackoutConfig] [userId: USERNAME] [ecid: 004oo4JeaWFDwW95zf8DyW0007vt0004WR,0:1] [APP: emgc] [LOG_FILE: /oracle/oem/Middleware12cR2/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/sysman/log/emoms.log] [URI: /em/console/admin/rep/blackout/blackoutConfig] Submit of Blackout Failed.[[
java.sql.SQLException: ORA-20233: Agent targets cannot be directly blacked out
ORA-06512: at “SYSMAN.MGMT_BLACKOUT_ENGINE”, line 2368
ORA-06512: at “SYSMAN.MGMT_BLACKOUT_ENGINE”, line 2395
ORA-06512: at “SYSMAN.MGMT_BLACKOUT”, line 39
ORA-06512: at “SYSMAN.MGMT_BLACKOUT_UI”, line 1403
ORA-06512: at line 1

It seems like it was a late decision to remove this functionality since it’s still a bit rough around the edges. If you’re logged in as SYSMAN and attempt to create a blackout against an agent target, the “Search and Select: Targets” screen will not even contain ‘Agent’ in the drop down selection list for target type. But if you follow Oracle’s security recommendations and login as a regular user account rather than SYSMAN, you will still see ‘Agent’ available in the selection list, but attempting to use it throws the error and incident described above.

The workaround appears to be to select a HOST target to blackout, rather than an agent. This seems to replicate the functionality (black out everything on that host) that was achieved previously by blacking out an agent.

EM12c “Supplied date is in the future” event post-DST fixed w/agent clearstate and restart

Since about an hour before the transition from daylight savings time back to standard time, I’ve noticed some events showing up in the incident manager that make it appear as though my agents are time traveling.

The events are showing up as metric evaluation errors on database instance and agent targets stating, for example: “Row(8): Supplied date is in the future : now = Mon Nov 05 08:42:41 EST 2012 supplied value = Mon Nov 05 09:38:42 EST 2012”. Sometimes the times reported are exactly one hour apart: “Row(1): Supplied date is in the future : now = Sun Nov 04 01:59:52 EDT 2012 supplied value = Sun Nov 04 01:59:52 EST 2012”. I see these from the repository database instance, several monitored database instances, and four different agents.

I’m seeing cases where a monitored database instance has this event and the monitoring agent does NOT, and I’m also seeing cases where both a monitored database and the agent performing the monitoring both show the event.

It seems that stopping the agent, running an emctl clearstate agent, then starting up the agent on each affected server will, after a few minutes, cause these events to clear. I only noticed this right when it started since I was logged in to take down SAP instances that can’t handle the time change, I wasn’t expecting to hit a related problem with EM12c.

emctl stop agent ; emctl clearstate agent ; emctl start agent

If you’re impatient, you can force collection of the EMDStatus metric collection to speed things up in clearing the events:

emctl control agent runCollection agenthost.domain.com:3872:oracle_emd EMDStatusCollection

Final (!) update/fix for EM12c increased redo from EM_JOB_METRICS after upgrade to R2 (bug/patch 14726136)

Update 13 Nov 2012: Patch 14726136 has been obsoleted. The note on MOS indicated that the patch caused some metrics to be calculated incorrectly. The patch has been replaced by patch 14833587. I have applied the new patch and all appears well — jobs are running and redo generation on the repository database remains where it was before the upgrade to EM12c R2.

Following up again on the EM12cR2 upgrade issue from BP1 that causes significantly increased redo logging on the repository database due to heavy insert activity in EM_JOB_METRICS. I first covered this here, with a followup here, a partial workaround here, and then a warning here.

Patch 14726136 has been re-released on MOS. The initial release caused problems for me as it prevented all OEM scheduled jobs from running, eventually causing me to rollback the patch. I am very pleased to report that the new update of the patch (from Oct 30th) applies cleanly and all of my jobs are now running on-schedule and completing successfully. EM_JOB_METRICS is showing no more than 11 inserts per second, and more than a minute is passing between the batches of inserts. My redo volume is already down significantly.

Big thanks to Oracle support and development and the EM12c team!