WORKAROUND: Unable to monitor Oracle XE 11gR2 with Oracle Enterprise Manager 13c

I have recently switched to using Oracle Enterprise Manager 13c (EM13c – 13.1), replacing my previous EM12c installation.  I elected to install a clean new environment instead of an upgrade, because my old install had been upgraded repeatedly going back to the initial release of EM12c and I wanted a fresh start.

I encountered only one difficult issue during the process. When I attempted to add one production Oracle XE 11gR2 database target, EM13c could not compute the target’s dynamic properties, leaving the target broken. Since you cannot submit jobs against a broken target, this prevented me from using EM13c to back up this database.  I had no comparable issues with XE as a target under EM12c.

The key metric errors that showed during this process included:
“Metric evaluation error start – Target {oracle_database.SID.domain.com} is broken: Dynamic Category property error,Get dynamic property error,No such metadata – No valid queryDescriptor or executionDescriptor found for target [oracle_database.SID.domain.com$30]”

and for the database system target:

“Metric evaluation error start – Received an exception when evaluating sev_eval_proc for:Target name = SID.domain.com_sys, metric_name = Response, metric_column = Status; Error msg = Target encountered metric erros; at least one member in in metric error”

I enabled debugging for the agent logs and attempted again to add the XE target.  Errors showing up in the logs included:

2016-01-15 12:10:05,905 [1806:4CE3192] DEBUG – Computing of dynamic property: [ComputeVC] is done (1 msec, error=true)

2016-01-15 12:10:06,452 [1806:F917F5F8] DEBUG – Computing of dynamic property: [GetDumpDestination] is done (0 msec, error=true)

2016-01-15 12:10:06,508 [1813:6EEEAC87] DEBUG – Computing of dynamic property: [DeduceAlertLogFile] is done (1 msec, error=true)

2016-01-15 12:11:18,779 [1830:CD3A325D] DEBUG – Error was added to oracle_database.SID.domain.com$23(0|MONITORED|false|true|<UF>): Invalid Input

2016-01-15 12:11:18,779 [1831:3657AE55] DEBUG – abandoning long op “CDProps:oracle_database.SID.domain.com:ComputeVC:GENERIC_TASK:Fri Jan 15 12:11:18 EST 2016”

2016-01-15 12:11:18,780 [1830:CD3A325D] DEBUG – Error during dynamic property ComputeVC calculation: critical=true, missingCatProps=[VersionCategory], missingProps=[VersionCategory] oracle.sysman.emSDK.agent.fetchlet.exception.FetchletException: Invalid Input

2016-01-15 12:35:38,038 INFO – Finished dynamic properties computation (total time 817 ms). Number of DP computed: 19. Number of DP with error: 3. Number of dynamic properties that were added: 132.

After reviewing the logs carefully (and posting this as a question in the MOS Oracle XE forum – https://community.oracle.com/thread/3892946) I eventually narrowed the issue down to a query that EM13c runs against DBA_REGISTRY_HISTORY in a target database when added. For database versions greater than 11.2 but less than 12.1.0.2, EM13c assumes that DBA_REGISTRY_HISTORY contains a BUNDLE_SERIES column.  This column does not exist in Oracle XE 11gR2, which reports a version string of 11.2.0.2.

This bug should eventually get a fix as EM13c gets patched, but in the meantime if you need to monitor an Oracle XE target with EM13c, the following workaround took care of the problem for me: create a new DBA_REGISTRY_HISTORY table containing a BUNDLE_SERIES column in your monitoring user’s schema in XE.  So, as user DBSNMP on XE, I ran:

SQL> create table dba_registry_history (ACTION_TIME TIMESTAMP(6), ACTION VARCHAR2(30), NAMESPACE VARCHAR2(30), VERSION VARCHAR2(30), ID NUMBER, BUNDLE_SERIES VARCHAR2(30), COMMENTS VARCHAR2(255));

Since one cannot patch XE, the real DBA_REGISTRY_HISTORY view has no rows and so you do not need to populate any data into this new table.

After adding the table, force a recalculation of dynamic properties by running the following against the EM13c management agent on the XE server:

$ emctl reload agent dynamicproperties SID.domain.com:oracle_database
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD recompute dynprops completed successfully

Once that completed successfully my XE target started to show the correct status in EM13c and I can submit jobs against the target.  All fixed.  I recommend deleting the DBSNMP.DBA_REGISTRY_HISTORY table once the bug gets fixed in OEM.

EM12c opatchauto, SHA256, and you

This post serves to document an issue I encountered after replacing expired SSL/TLS certificates on the server I use for Oracle Enterprise Manager 12c. To put it simply, using opatchauto to apply EM12c PSUs does not work if your WebLogic adminserver has a certificate installed that uses the SHA256 hashing algorithm.

[UPDATED 20151012: Please see this comment and this comment below, from Adam Robinson, who has provided a workaround that may work for you involving editing the opatchauto script to enable JSSE. As always, please consider workarounds requiring you to edit files as unsupported and at your own risk, but I would consider this fix superior to reverting back to the demo certificate every time you need to patch. You will need to repeat this fix every time you update OPatch in your OMS_HOME, though. Adam’s workaround does succeed in my environment.]

Error message

Expect to see the following error when running “opatchauto apply -analyze” or “opatchauto apply” against an installation with an SHA256-hashed certificate on the WLS adminserver:

oracle@omshost:/oracle/stage/21603255> opatchauto apply -analyze -property_file ~/property_file 
OPatch Automation Tool
Copyright (c) 2014, Oracle Corporation.  All rights reserved.


OPatchauto version : 11.1.0.12.3
OUI version        : 11.1.0.12.0
Running from       : /oracle/oem/Middleware12cR4/oms
Log file location  : /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatch/opatch2015-09-11_10-57-19AM_1.log

OPatchauto log file: /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatchauto/21603255/opatch_oms_2015-09-11_10-57-22AM_analyze.log



OPatchauto failed to establish JMX connection to weblogic server. This could be because of one (or) more of the following reasons:
1. Weblogic admin server URL that manages OMS application may not be right.
2. Weblogic admin server credentials (username, password) may not be right.
3. Virtual host configuration. If OMS, weblogic server are on virtual host configuration, Please make sure to add OPatchAuto.OMS_DISABLE_HOST_CHECK=true to command line and run again. (example: /oracle/oem/Middleware12cR4/oms/OPatch/opatchauto apply -analyze -property_file /home/oracle/property_file -invPtrLoc /oracle/oem/Middleware12cR4/oms/oraInst.loc  OPatchAuto.OMS_DISABLE_HOST_CHECK=true)

Please check above conditions and if error(s) still persist, Please contact Oracle support.


[ Error during Get weblogic Admin Server information Phase]. Detail: OPatchauto was not able to find right interview inputs.
OPatchauto failed: 
OPatchauto failed to establish JMX connection to weblogic server. This could be because of one (or) more of the following reasons:
1. Weblogic admin server URL that manages OMS application may not be right.
2. Weblogic admin server credentials (username, password) may not be right.
3. Virtual host configuration. If OMS, weblogic server are on virtual host configuration, Please make sure to add OPatchAuto.OMS_DISABLE_HOST_CHECK=true to command line and run again. (example: /oracle/oem/Middleware12cR4/oms/OPatch/opatchauto apply -analyze -property_file /home/oracle/property_file -invPtrLoc /oracle/oem/Middleware12cR4/oms/oraInst.loc  OPatchAuto.OMS_DISABLE_HOST_CHECK=true)

Please check above conditions and if error(s) still persist, Please contact Oracle support.

Log file location: /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatchauto/21603255/opatch_oms_2015-09-11_10-57-22AM_analyze.log

Recommended actions: Please correct the interview inputs and run opatchauto again.

OPatchauto failed with error code 231

Confirmation of the issue

To confirm this issue in your environment after receiving the preceding error message, check the hashing algorithm used on your adminserver certificate. I prefer to use the openssl commandline tool for this. If you don’t know the port used for your adminserver, you can retrieve it from the $EM_INSTANCE_BASE/em/EMGC_OMS1/emgc.properties file under AS_HTTPS_PORT. If your certificate does not show the usage of SHA256 (or another hash algorithm from the SHA-2 family) as in my example below, you may have a different problem.

oracle@omshost:~> openssl s_client -prexit -connect omshost.domain.com:7103 /dev/null | openssl x509 -text -in /dev/stdin | grep "Signature Algorithm" 2> /dev/null
        Signature Algorithm: sha256WithRSAEncryption
    Signature Algorithm: sha256WithRSAEncryption

Workaround

To work around this issue, you need to (temporarily!) replace the certificate on your WLS adminserver. Now, whenever I need to apply a PSU release, I resecure WLS using the default demonstration certificate, apply the PSU, then replace my original certificate.

oracle@omshost:/oracle/stage/21603255> emctl secure wls -use_demo_cert
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Securing WLS... Started.
Enter Enterprise Manager Root (SYSMAN) Password :
Securing WLS... Successful
Restart OMS using 'emctl stop oms -all' and 'emctl start oms'
oracle@omshost:/oracle/stage/21603255> emctl stop oms -all ; sleep 5 ; emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
Stopping BI Publisher Server...
BI Publisher Server Successfully Stopped
AdminServer Successfully Stopped
BI Publisher Server is Down
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
Starting BI Publisher Server ...
BI Publisher Server Successfully Started
BI Publisher Server is Up

[install the PSU according to the README instructions, including any post-installation steps]

oracle@omshost:/oracle/stage/21603255> emctl secure wls -wallet /oracle/oem/oemwallet
Oracle Enterprise Manager Cloud Control 12c Release 4  
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Securing WLS... Started.
Enter Enterprise Manager Root (SYSMAN) Password : 
Securing WLS... Successful
Restart OMS using 'emctl stop oms -all' and 'emctl start oms'
If there are multiple OMSs in this environment, perform this configuration on all of them.
oracle@omshost:/oracle/stage/21603255> emctl stop oms -all ; sleep 5 ; emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
Stopping BI Publisher Server...
BI Publisher Server Successfully Stopped
AdminServer Successfully Stopped
BI Publisher Server is Down
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
Starting BI Publisher Server ...
BI Publisher Server Successfully Started
BI Publisher Server is Up

I have not noticed any other EM12c issues using SHA256-hashed certificates. With this workaround, you can both continue to use quality certificates and keep your OMS patched.

EM12c OHS, LOW strength ciphers, custom certificates, and patch 19948000 weirdness

This post documents an unusual issue I encountered with the Oracle HTTP Server (OHS) installation in my Oracle Enterprise Manager 12c R4 (12.1.0.4) environment after following MOS note 1984662.1 and applying patch 19948000 (CPUJAN2015) to my OHS home.  It also contains a workaround I found that you should consider UNSUPPORTED, UNOFFICIAL, and NOT RECOMMENDED, only for use if absolutely necessary to meet auditor requirements.  If you do not have to follow the steps I describe below, I suggest waiting for new patches and further guidance from Oracle Support. If this change breaks your system and eats all the food in the break room refrigerator, I warned you not to do it.

Like other security-conscious EM12c admins, I want to keep my installation secure, and so I watch closely when security patches become available for EM12c or its various components. Thus, when I noticed patch 19948000’s availability for OHS, which disables SSLv3, I installed it on my system, and confirmed through testing that OHS no longer permitted SSLv3 connections (test for yourself with: openssl s_client -connect host.domain.com:port -ssl3, or try my EM12c SSL security checkup script that I have blogged about previously).

As I proceeded with further hardening of my EM12c system, specifically an attempt to disable LOW and MEDIUM strength cipher suite usage as per MOS note 1477287.1, I noticed that after following the directions provided, all of my EM12c endpoints correctly rejected LOW and MEDIUM strength ciphers, with one exception.  The OMS HTTPS upload port, inexplicably, continued to permit LOW strength connections. It refused MEDIUM strength ciphers, but had no problem accepting a LOW strength DES-CBC-SHA connection over TLSv1:

$ openssl s_client -connect omshost.domain.com:4902 -cipher LOW
[...]
SSL handshake has read 4109 bytes and written 385 bytes
---
New, TLSv1/SSLv3, Cipher is DES-CBC-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DES-CBC-SHA
    Session-ID: 37BF30668DCAD2CC5D0BAC4142CC1FA1
    Session-ID-ctx:
    Master-Key: [redacted]
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1429290250
    Timeout   : 300 (sec)
---

This confused me greatly, as I had edited all configuration files as instructed, and none of my other OHS listen ports accepted this LOW strength cipher connection.  I spent quite a bit of time trying to diagnose and resolve the issue with no luck, until I eventually stumbled upon an odd fix.  If I remove or comment out the “IfDefine SSL” directives from my $GC_INSTANCE_HOME/WebTierIH1/config/OHS/ohs1/httpd_em.conf file, then suddenly OHS would refuse LOW strength cipher connections on this port, with no apparent ill effect on the other listening ports.

$ openssl s_client -connect omshost.domain.com:4902 -cipher LOW 
CONNECTED(00000003)
2282780:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:769:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 67 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
---

I have noted these IfDefine SSL directives with “HERE” in the excerpt below from my httpd_em.conf file.

##
## CAUTION: Edit only the .template version of this file!
##
##     The command
##         emctl secure [lock|unlock]
##     will replace httpd_em.conf (discarding your changes) 
##     using the httpd_em.conf.template file.
##
## This file contains virtual hosts and other directives
## required for the "Enterprise Manager Central Console"
## to function correctly.
##

#UseWebCacheIp On

<IfDefine SSL>      #### HERE
    Listen 4902
    <VirtualHost *:4902>
        <Location /empbs/upload>
            Order allow,deny
            Allow from all
        </Location>
        <Location /empbs/jobrecv>
            Order allow,deny
            Allow from all
        </Location>
        <Location /em>
            Order allow,deny
            Allow from all
        </Location>
        <Location /agent_download>
            Order allow,deny
            Allow from all
        </Location>
        <Location /xmlpserver>
            Order allow,deny
            Allow from all
        </Location>

        #DocumentRoot &ORACLE_HOME&/Apache/Apache/htdocs
        ServerName omshost.domain.com
        #Port 4902
        Timeout 900

        LogFormat "%h %l %u %t \"%r\" %>s %b [ecid: %{ECID-Context}i] [User-Agent: %{User-Agent}i]" common
        SetEnvIf Request_URI "\.(bmp|jpg|png|gif|css|js$)" no-log
        SetEnvIf Request_URI "/em/dynamicImage/*"  no-log
        CustomLog "|${ORACLE_HOME}/ohs/bin/odl_rotatelogs /oracle/oem/gc_inst1/WebTierIH1/diagnostics/logs/OHS/ohs1/em_upload_https_access_log 10M 100M" common env=!no-log

        ErrorLog "|${ORACLE_HOME}/ohs/bin/odl_rotatelogs /oracle/oem/gc_inst1/WebTierIH1/diagnostics/logs/OHS/ohs1/em_upload_https_error_log 10M 100M"
        SSLEngine on
        SSLCipherSuite HIGH
        SSLWallet file:/oracle/oem/gc_inst1/WebTierIH1/config/OHS/ohs1/keystores/upload
        SSLProtocol TLSv1

        <Files ~ "\.(cgi|shtml)$">
            SSLOptions +StdEnvVars
        </Files>
        #<Directory &ORACLE_HOME&/Apache/Apache/cgi-bin>
        #    SSLOptions +StdEnvVars
        #</Directory>
        SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown
    </VirtualHost>
</IfDefine>     #### HERE
[remainder of file removed]

If I leave the IfDefine SSL statements in there, my OMS upload port accepts the weak DES-CBC-SHA cipher along with HIGH strength ciphers.  If I remove the IfDefine SSL, my OMS upload port refuses DES-CBC-SHA along with all other LOW/MEDIUM strength ciphers.

This makes no sense, given what I know of OHS and Apache-like products and the way that the handle the SSLCipherSuite configuration directive.

I raised this issue on Twitter and heard back from Andrew Bulloch at Oracle, who graciously spent quite a bit of time attempting to reproduce the issue on his side and working with me to identify the situations in which this behavior occurs.  After much testing, it appears that this behavior only occurs in the following situation:

  1. The OHS installed with EM12c R4 has patch 19948000 installed, AND
  2. The administrator has installed a third party SSL certificate, replacing the demo certificate used by default, AND
  3. The OHS httpd_em.conf contains the “IfDefine SSL” directive.

If I remove my custom certificate, returning the OMS to the demo certificate, the issue disappears, then returns if I reinstall the custom certificate.

If I remove patch 19948000, the issue disappears, and does not return whether I use a custom certificate or a demo certificate.

If I remove the IfDefine SSL directive, the issue disappears, and does not return whether I use a custom certificate, a demo certificate, or whether or not I have patch 19948000 installed.

I attempted to replicate this behavior with an SSL certificate that did not come from a true certificate authority, by using OpenSSL to create a CA, create a cert, sign it, then install it into OHS per the documentation in MOS note 1399293.1, but I could not reproduce it, possibly due to the fact that I used a certificate signed directly by a root CA (as with the demo certificate) instead of a certificate signed by an intermediate chain certificate signed by a root CA, as with the paid-for commercial certificate that revealed the issue. I have not had a chance to test that configuration.

Unfortunately, removing patch 19948000 means that OHS cannot refuse SSLv3 connections, and removing the custom certificate reverts the system back to the demo certificate that I do not wish to use, both of which will represent audit findings in regulated sites.

Due to this issue, I have edited my EM12c security checkup script to remove my recommendation to install patch 19948000, although I still have it installed.  For security reasons, I will leave my system in the workaround state I have described here, as I want SSLv3 disabled, and I want LOW strength cipher suites disabled, and I want to use a custom SSL certificate, but I accept the risk that I may have to undo this setup at any time to receive support or to successfully apply later patches.  You will have to make your own decisions based on your site’s audit requirements and the availability of personnel to validate your configuration and handle future patching.

I would be very interested if anyone else reading this has encountered this issue, as I do not know if my installation somehow uniquely surfaces this behavior or if the certificate vendor that we used has some strange settings on their certificates that cause confusion for OHS.

EM12c R4 SSL Security Checkup Script

[LATEST SCRIPT UPDATE: 20151204, VERSION 1.11, covers 20151130 patch release

Download the script here.

With all the recent news on companies getting hacked and attacks on encryption techniques, you need to act proactively to secure your Oracle Enterprise Manager Cloud Control 12c environment. Do not wait for your employer’s auditor to come around and send you a report of all the flaws in your system.

To put it in very simple terms, if you do not do the following across EVERY EM12c component, you should consider your setup vulnerable:

  • Disable SSLv2 and SSLv3
  • Enable TLSv1
  • Disable weak ciphersuites such as those using the MD5 or RC4 algorithms, or those previously designed for export outside the USA back in the 1990s, or those that do not use enough key bits for encryption.
  • Eliminate the use of self-signed and demonstration certificates.
  • Stay current on EM12c base releases (currently EM12c R5 but I have not yet upgraded)
  • Stay current on PSU updates to EM12c (PSU5 as of October 2015)
  • Stay current on monthly system patch bundles
  • Stay current on quarterly critical patch update alerts for all EM12c components – note that you have to pay attention to, for example, Oracle HTTP Server (OHS) critical patch updates, as EM12c distributes and relies on OHS. See MOS note 1664074.1 for a good, but incomplete list of patches needed.
  • Stay current on repository database patch set updates
  • Stay current on EM12c Java versions [EDIT: 20150415: Added Java check to script] [EDIT: 20150818: Java 1.6_101 caused the Node Manager to fail to start on my system.  Therefore I have kept the Java version check at 1.6_95.]

Yes, this takes a lot of work.  Yes, the documentation sometimes leaves the process as clear as mud.  Yes, you can contact Oracle support for assistance.

Yes, you do need to deal with EVERY endpoint for the SSL configuration.  That includes:

  • OMS console
  • OMS upload port
  • OMS console proxy port
  • Management agents
  • EM Node Manager
  • WebLogic Server administration console
  • OHS administration port
  • OPMN port
  • BI Publisher

In the meantime, though, you need to have a good idea of where your system has flaws so that you know where to spend your time fixing it. To help with this, I have created a script that will examine your EM12c environment, find all the ports in use, check for SSLv2, SSLv3, and TLSv1, validate the cipher suites in use, check to make sure you have current patches installed, check for the usage of self-signed certificates on SSL/TLS endpoints, and check for current Java JDK versions in EM12c components. [EDIT: 20150311: Added self-signed certificate check]. [EDIT: 20150313: Added patch check for repository databases on same host as OMS server. I have only tested this on an 11.2.0.4 repository, but I believe it will work for the 12.1.0.2 repository just recently re-certified. If it fails for you please let me know.] [EDIT: 20150415: Added check for Java JDK versions.] [EDIT: 20150630: Added check for SSL_VERSION and SSL_CIPHER_SUITES parameters in repository database sqlnet.ora and listener.ora.]

This script does not require any arguments or configuration. I have tested it ONLY on EM12c R4 and on Linux x86-64 and only on single-host OMS environments.  To run this script, copy it from the end of this post (or from the pastebin link above, and execute it as the Oracle software owner on your OMS host, with your environment fully up and running. [EDIT: 20150311: Updated script incorporating feedback from Dave Corsar and opa tropa to support Solaris and AIX.]

The script will not make any changes to your system.  Mostly it crawls your configuration files to identify ports, then tests them with the openssl s_client command and various command line arguments to identify protocol and cipher suite usage, and whether or not it can find self-signed certificates.  At the end it runs OPatch checks for current needed security and functionality patches.

As of the version 1.1 release, I will mark newly checked patches with “*NEW*” in the script output and updated patches with “*UPDATED*”. For example, when a new PSU patch comes out, I will mark it as an update, but I will mark new (or previously not checked) patches as new. [EDIT: 20150415: This paragraph added.]

Example output from my fully patched and secured system [EDIT: 20150311: Unfortunately I still have self-signed certificates for OPMN and the OHS administration port, so my sample output now includes some failed checks]:

Performing EM12cR4 security checkup version 1.11 on omshost.domain.com at Fri Dec  4 14:17:40 EST 2015.

Using port definitions from configuration files 
	/etc/oragchomelist
	/oracle/oem/gc_inst1/em/EMGC_OMS1/emgc.properties
	/oracle/oem/gc_inst1/em/EMGC_OMS1/embip.properties
	/oracle/oem/gc_inst1/WebTierIH1/config/OPMN/opmn/ports.prop
	/oracle/oem/gc_inst1/WebTierIH1/config/OHS/ohs1/admin.conf

	Agent port found at omshost.domain.com:3872
	BIPublisher port found at omshost.domain.com:9702
	NodeManager port found at omshost.domain.com:7404
	OHSadmin port found at omshost.domain.com:9999
	OMSconsole port found at omshost.domain.com:7803
	OMSproxy port found at omshost.domain.com:7302
	OMSupload port found at omshost.domain.com:4902
	OPMN port found at omshost.domain.com:6701
	WLSadmin found at omshost.domain.com:7103

	Repository DB version=11.2.0.4.0 SID=emrep host=omshost.domain.com
	Repository DB on OMS server, will check patches/parameters in /oracle/oem/product/11.2.0/dbhome_2

(1) Checking SSL/TLS configuration (see notes 1602983.1, 1477287.1, 1905314.1)

	(1a) Forbid SSLv2 connections
	Confirming ssl2 disabled for Agent at omshost.domain.com:3872... OK
	Confirming ssl2 disabled for BIPublisher at omshost.domain.com:9702... OK
	Confirming ssl2 disabled for NodeManager at omshost.domain.com:7404... OK
	Confirming ssl2 disabled for OHSadmin at omshost.domain.com:9999... OK
	Confirming ssl2 disabled for OMSconsole at omshost.domain.com:7803... OK
	Confirming ssl2 disabled for OMSproxy at omshost.domain.com:7302... OK
	Confirming ssl2 disabled for OMSupload at omshost.domain.com:4902... OK
	Confirming ssl2 disabled for OPMN at omshost.domain.com:6701... OK
	Confirming ssl2 disabled for WLSadmin at omshost.domain.com:7103... OK

	(1b) Forbid SSLv3 connections
	Confirming ssl3 disabled for Agent at omshost.domain.com:3872... OK
	Confirming ssl3 disabled for BIPublisher at omshost.domain.com:9702... OK
	Confirming ssl3 disabled for NodeManager at omshost.domain.com:7404... OK
	Confirming ssl3 disabled for OHSadmin at omshost.domain.com:9999... OK
	Confirming ssl3 disabled for OMSconsole at omshost.domain.com:7803... OK
	Confirming ssl3 disabled for OMSproxy at omshost.domain.com:7302... OK
	Confirming ssl3 disabled for OMSupload at omshost.domain.com:4902... OK
	Confirming ssl3 disabled for OPMN at omshost.domain.com:6701... OK
	Confirming ssl3 disabled for WLSadmin at omshost.domain.com:7103... OK

	(1c) Permit TLSv1 connections
	Confirming tls1 available for Agent at omshost.domain.com:3872... OK
	Confirming tls1 available for BIPublisher at omshost.domain.com:9702... OK
	Confirming tls1 available for NodeManager at omshost.domain.com:7404... OK
	Confirming tls1 available for OHSadmin at omshost.domain.com:9999... OK
	Confirming tls1 available for OMSconsole at omshost.domain.com:7803... OK
	Confirming tls1 available for OMSproxy at omshost.domain.com:7302... OK
	Confirming tls1 available for OMSupload at omshost.domain.com:4902... OK
	Confirming tls1 available for OPMN at omshost.domain.com:6701... OK
	Confirming tls1 available for WLSadmin at omshost.domain.com:7103... OK

(2) Checking supported ciphers at SSL/TLS endpoints (see notes 1477287.1, 1905314.1, 1067411.1)
	Checking LOW strength ciphers on Agent (omshost.domain.com:3872)...	OK
	Checking MEDIUM strength ciphers on Agent (omshost.domain.com:3872)...	OK
	Checking HIGH strength ciphers on Agent (omshost.domain.com:3872)...	OK

	Checking LOW strength ciphers on BIPublisher (omshost.domain.com:9702)...	OK
	Checking MEDIUM strength ciphers on BIPublisher (omshost.domain.com:9702)...	OK
	Checking HIGH strength ciphers on BIPublisher (omshost.domain.com:9702)...	OK

	Checking LOW strength ciphers on NodeManager (omshost.domain.com:7404)...	OK
	Checking MEDIUM strength ciphers on NodeManager (omshost.domain.com:7404)...	OK
	Checking HIGH strength ciphers on NodeManager (omshost.domain.com:7404)...	OK

	Checking LOW strength ciphers on OHSadmin (omshost.domain.com:9999)...	OK
	Checking MEDIUM strength ciphers on OHSadmin (omshost.domain.com:9999)...	OK
	Checking HIGH strength ciphers on OHSadmin (omshost.domain.com:9999)...	OK

	Checking LOW strength ciphers on OMSconsole (omshost.domain.com:7803)...	OK
	Checking MEDIUM strength ciphers on OMSconsole (omshost.domain.com:7803)...	OK
	Checking HIGH strength ciphers on OMSconsole (omshost.domain.com:7803)...	OK

	Checking LOW strength ciphers on OMSproxy (omshost.domain.com:7302)...	OK
	Checking MEDIUM strength ciphers on OMSproxy (omshost.domain.com:7302)...	OK
	Checking HIGH strength ciphers on OMSproxy (omshost.domain.com:7302)...	OK

	Checking LOW strength ciphers on OMSupload (omshost.domain.com:4902)...	OK
	Checking MEDIUM strength ciphers on OMSupload (omshost.domain.com:4902)...	OK
	Checking HIGH strength ciphers on OMSupload (omshost.domain.com:4902)...	OK

	Checking LOW strength ciphers on OPMN (omshost.domain.com:6701)...	OK
	Checking MEDIUM strength ciphers on OPMN (omshost.domain.com:6701)...	OK
	Checking HIGH strength ciphers on OPMN (omshost.domain.com:6701)...	OK

	Checking LOW strength ciphers on WLSadmin (omshost.domain.com:7103)...	OK
	Checking MEDIUM strength ciphers on WLSadmin (omshost.domain.com:7103)...	OK
	Checking HIGH strength ciphers on WLSadmin (omshost.domain.com:7103)...	OK


(3) Checking self-signed and demonstration certificates at SSL/TLS endpoints (see notes 1367988.1, 1399293.1, 1593183.1, 1527874.1, 123033.1, 1937457.1)
	Checking certificate at Agent (omshost.domain.com:3872)... OK
	Checking certificate at Agent (omshost.domain.com:3872)... OK
	Checking certificate at BIPublisher (omshost.domain.com:9702)... OK
	Checking certificate at BIPublisher (omshost.domain.com:9702)... OK
	Checking certificate at NodeManager (omshost.domain.com:7404)... OK
	Checking certificate at NodeManager (omshost.domain.com:7404)... OK
	Checking certificate at OHSadmin (omshost.domain.com:9999)... FAILED - Found self-signed certificate
	Checking certificate at OHSadmin (omshost.domain.com:9999)... OK
	Checking certificate at OMSconsole (omshost.domain.com:7803)... OK
	Checking certificate at OMSconsole (omshost.domain.com:7803)... OK
	Checking certificate at OMSproxy (omshost.domain.com:7302)... OK
	Checking certificate at OMSproxy (omshost.domain.com:7302)... OK
	Checking certificate at OMSupload (omshost.domain.com:4902)... OK
	Checking certificate at OMSupload (omshost.domain.com:4902)... OK
	Checking certificate at OPMN (omshost.domain.com:6701)... FAILED - Found self-signed certificate
	Checking certificate at OPMN (omshost.domain.com:6701)... OK
	Checking certificate at WLSadmin (omshost.domain.com:7103)... OK
	Checking certificate at WLSadmin (omshost.domain.com:7103)... OK

(4) Checking EM12c Oracle home patch levels against 30 Nov 2015 baseline (see notes 1664074.1, 1900943.1, 822485.1, 1470197.1, 1967243.1)

	(4a) OMS (/oracle/oem/Middleware12cR4/oms) ENTERPRISE MANAGER BASE PLATFORM - OMS 12.1.0.4.5 PSU Patch (21462217)... OK
Patch 21462217 : applied on Tue Oct 20 12:13:32 EDT 2015 19055251, 19586898, 20260177, 19323634, 21462217, 19941819, 18725891

	(4a) OMS HOME (/oracle/oem/agent12c/core/12.1.0.4.0) JDBC Merge Patch (18502187)... OK
Patch 18502187 : applied on Thu Oct 22 10:29:36 EDT 2015

	(4b) BI Publisher (/oracle/oem/Middleware12cR4/Oracle_BI1) CPUJAN2015 Patch (19822893)... OK
19822893 19822893 Patch 19822893 : applied on Wed Feb 25 09:16:21 EST 2015

	(4b) BI Publisher (/oracle/oem/Middleware12cR4/Oracle_BI1) Merge Patch (20444447)... OK
Patch 20444447 : applied on Wed Feb 25 09:21:03 EST 2015

	(4c) AS Common (/oracle/oem/Middleware12cR4/oracle_common) CVE-2015-0426 Oracle Help Patch (20075252)... OK
Patch 20075252 : applied on Thu Jan 22 14:39:21 EST 2015

	(4c) AS Common (/oracle/oem/Middleware12cR4/oracle_common) WEBCENTER PORTAL BUNDLE PATCH 11.1.1.7.1 (16761779)... OK
Patch 16761779 : applied on Wed Apr 15 12:18:20 EDT 2015

	(4c) AS Common (/oracle/oem/Middleware12cR4/oracle_common) CVE-2015-4742 MERGE REQUEST ON TOP OF 11.1.1.7.1 FOR BUGS 20747356 18274008 (21068288)... OK
Patch 21068288 : applied on Thu Sep 17 09:52:53 EDT 2015

	(4d) WebLogic Server (/oracle/oem/Middleware12cR4/wlserver_10.3) 10.3.6.0.12 EJUW Patch (20780171)... 	OK
CR(s)..................... 20780171 Jar....................... BUG20780171_1036012.jar Destination............... $WLS_INSTALL_DIR$/bugsfixed/20780171-WLS-10.3.6.0.12_PSU_WebServices-ClientSide-Configuration-README.txt

	(4d) WebLogic Server (/oracle/oem/Middleware12cR4/wlserver_10.3) SU Patch [GDFA]: WEBLOGIC.STORE.PERSISTENTSTOREEXCEPTION: [STORE:280040] OCCURS EASILEY (16420963)... 	OK
CR(s)..................... 16420963 Jar....................... BUG16420963_1036.jar

	(4e) WebTier (/oracle/oem/Middleware12cR4/Oracle_WT) OHS SECURITY PATCH UPDATE 11.1.1.7.0 CPUOCT2015 Patch (21640624)... OK
Patch 21640624 : applied on Mon Oct 26 13:59:17 EDT 2015

	(4e) WebTier (/oracle/oem/Middleware12cR4/Oracle_WT) CVE-2014-4212 OPMN Patch (19345576)... OK
Patch 19345576 : applied on Thu Jan 22 13:02:25 EST 2015

	(4e) WebTier (/oracle/oem/Middleware12cR4/Oracle_WT) CVE 2015-2658 MERGE REQUEST ON TOP OF 11.1.1.7.0 FOR BUGS 16370190 20310323 20715657 (20807683)... OK
Patch 20807683 : applied on Wed Jul 15 12:22:04 EDT 2015

	(4e) WebTier (/oracle/oem/Middleware12cR4/Oracle_WT) CVE-2013-0169,CVE-2011-3389 OSS SECURITY PATCH UPDATE 11.1.1.7.0 CPUOCT2013 (17337741)... OK
Patch 17337741 : applied on Wed Apr 15 10:36:26 EDT 2015

	(4e) WebTier (/oracle/oem/Middleware12cR4/Oracle_WT) WLSPLUGINS (OHS) SECURITY PATCH UPDATE 11.1.1.7.0 CPUJUL2014 (18423831)... OK
Patch 18423831 : applied on Wed Apr 15 12:45:02 EDT 2015

	(4f) *UPDATED* OMS (/oracle/oem/Middleware12cR4/oms) DB PLUGIN BUNDLE PATCH 12.1.0.7.10 (22062307)... OK
22062307;EM DB PLUGIN BUNDLE PATCH 12.1.0.7.10 21744966,21745018,21972104,22062375,22062307

	(4g) *UPDATED* OMS (/oracle/oem/Middleware12cR4/oms) FMW PLUGIN BUNDLE PATCH 12.1.0.7.10 (22062375)... OK
22062375;EM FMW PLUGIN BUNDLE PATCH 12.1.0.7.10 21744966,21745018,21972104,22062375,22062307

	(4h) OMS (/oracle/oem/Middleware12cR4/oms) MOS PLUGIN BUNDLE PATCH 12.1.0.6.8 (21745018)... OK
21745018;EM MOS PLUGIN BUNDLE PATCH 12.1.0.6.8 21744966,21745018,21972104,22062375,22062307

	(4i) *UPDATED* OMS (/oracle/oem/Middleware12cR4/oms) EXADATA PLUGIN BUNDLE PATCH 12.1.0.6.11 (21744966)... OK
21744966;EM EXADATA PLUGIN BUNDLE PATCH 12.1.0.6.11 21744966,21745018,21972104,22062375,22062307

	(4j) *UPDATED* OMS (/oracle/oem/Middleware12cR4/oms) CFW PLUGIN BUNDLE PATCH 12.1.0.2.4 (21972104)... OK
21972104;EM CFW Plugin Bundle Patch 12.1.0.2.4 21744966,21745018,21972104,22062375,22062307

	(4k) *UPDATED* OMS CHAINED AGENT HOME (/oracle/oem/agent12c/core/12.1.0.4.0) EM-AGENT BUNDLE PATCH 12.1.0.4.14 (21913823)... OK
Patch 21913823 : applied on Fri Dec 04 09:16:23 EST 2015 17438375, 18936726, 21913823, 20496804, 21325110, 20701411, 21565489

	(4k) OMS CHAINED AGENT HOME (/oracle/oem/agent12c/core/12.1.0.4.0) Merge Patch (18502187)... OK
Patch 18502187 : applied on Fri Apr 03 09:45:56 EDT 2015

	(4k) OMS CHAINED AGENT HOME (/oracle/oem/agent12c/core/12.1.0.4.0) JDBC Security Patch (18721761)... OK
Patch 18721761 : applied on Fri Apr 03 09:45:52 EDT 2015

	(4k) OMS CHAINED AGENT HOME (/oracle/oem/agent12c/core/12.1.0.4.0) CVE 2012-3137 EM Agent only: Instant Client Security Patch (20114054)... OK
Patch 20114054 : applied on Fri May 01 10:01:01 EDT 2015 20114054

	(4l) *UPDATED* OMS CHAINED AGENT DB PLUGIN (/oracle/oem/agent12c/core/12.1.0.4.0/../../plugins/oracle.sysman.db.agent.plugin_12.1.0.7.0) DB PLUGIN BUNDLE 12.1.0.7.10 AGENT-SIDE MONITORING (22140476)... OK
Patch 22140476 : applied on Fri Dec 04 11:54:20 EST 2015 15837598, 21907123, 21460951, 20765041, 20844888, 22140476, 21806804

	(4l) OMS CHAINED AGENT DB PLUGIN (/oracle/oem/agent12c/core/12.1.0.4.0/../../plugins/oracle.sysman.db.discovery.plugin_12.1.0.7.0) DB PLUGIN BUNDLE 12.1.0.7.4 AGENT-SIDE DISCOVERY (21065239)... OK
Patch 21065239 : applied on Thu Jun 04 11:15:02 EDT 2015 18413892, 21065239

	(4m) *UPDATED* OMS CHAINED AGENT FMW PLUGIN (/oracle/oem/agent12c/core/12.1.0.4.0/../../plugins/oracle.sysman.emas.agent.plugin_12.1.0.7.0) FMW PLUGIN BUNDLE 12.1.0.7.9 AGENT-SIDE MONITORING (21941290)... OK
Patch 21941290 : applied on Fri Dec 04 12:01:35 EST 2015 20644295, 21894243, 20677020, 21888856, 21527296, 21941290, 21415166

	(4m) OMS CHAINED AGENT FMW PLUGIN (/oracle/oem/agent12c/core/12.1.0.4.0/../../plugins/oracle.sysman.emas.discovery.plugin_12.1.0.7.0) FMW PLUGIN BUNDLE 12.1.0.7.7 AGENT-SIDE DISCOVERY (21611921)... OK
Patch 21611921 : applied on Tue Sep 01 13:34:27 EDT 2015 21611921, 20644315, 20677038, 21199835, 21229841, 21610843

	(4n) *UPDATED* OMS CHAINED AGENT BEACON PLUGIN (/oracle/oem/agent12c/core/12.1.0.4.0/../../plugins/oracle.sysman.beacon.agent.plugin_12.1.0.4.0) EM-BEACON BUNDLE PATCH 12.1.0.4.2 (21928148)... OK
Patch 21928148 : applied on Fri Dec 04 12:35:11 EST 2015 21928008, 21928148, 20466772, 20397739

	(4o) OMS CHAINED AGENT EM-OH BUNDLE PATCH 12.1.0.4.1 (20855134)... OK
Patch 20855134 : applied on Thu Apr 30 15:54:47 EDT 2015 15985793, 20855134

	(4p) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/11.2.0/dbhome_2) PSU 11.2.0.4.8 (OCT2015) (21352635)... OK
Patch 21352635 : applied on Thu Oct 22 09:39:55 EDT 2015 Patch description: "Database Patch Set Update : 11.2.0.4.8 (21352635)"

	(4p) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/11.2.0/dbhome_2) ORACLE JAVAVM COMPONENT 11.2.0.4.5 DATABASE PSU (OCT2015) (21555791)... OK
Patch 21555791 : applied on Thu Oct 22 09:41:22 EDT 2015

	(4q) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/11.2.0/dbhome_2) sqlnet.ora SSL_VERSION parameter (1545816.1)... OK
1.0

	(4q) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/11.2.0/dbhome_2) sqlnet.ora SSL_CIPHER_SUITES parameter (1545816.1)... OK
(SSL_RSA_WITH_AES128_CBC_SHA,SSL_RSA_WITH_AES256_CBC_SHA)

	(4q) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/11.2.0/dbhome_2) listener.ora SSL_VERSION parameter (1545816.1)... OK
1.0

	(4q) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/11.2.0/dbhome_2) listener.ora SSL_CIPHER_SUITES parameter (1545816.1)... OK
(SSL_RSA_WITH_AES128_CBC_SHA,SSL_RSA_WITH_AES256_CBC_SHA)


(5) Checking EM12c Java versions against baseline (see notes 1506916.1, 1492980.1)

	(5a) MW (/oracle/oem/Middleware12cR4/jdk16/jdk) Java version 1.6.0_95 (9553040)... 	OK
1.6.0_95

	(5b) WebTier (/oracle/oem/Middleware12cR4/Oracle_WT/jdk) Java version 1.6.0_95 (9553040)... 	OK
1.6.0_95

Failed test count: 2 - Review output

certcheck:OHSadmin @ omshost.domain.com:9999 found self-signed certificate
certcheck:OPMN @ omshost.domain.com:6701 found self-signed certificate

Visit https://pardydba.wordpress.com/2015/03/09/em12c-r4-ssl-security-checkup-script/ for the latest version.


Body of script:

#!/bin/bash
#
# This script should examine your EM12c R4 environment, identify the ports
# each component uses, and check for SSLv2/SSLv3 usage, as well as make
# sure that weak cipher suites get rejected.  It also contains a patch
# check currently comparing against the latest recommended patches
# and flags the use of self-signed certificates.  Further checks include
# EM12c Java JDK version.
#
# Added in v1.0:   Repository database patch check
# Added in v1.1:   EM12c Java JDK version check
# Change in v1.2:  Removed patch 19948000 recommendation for OHS.
# Change in v1.3:  Update for 30 Apr 2015 patches, add EM-OH plugin home
#                  restored GDFA/16420963 for WLS
#                  added 20114054 for Agent - only applicable for Linux x86-64
# Change in v1.4:  Add datestamp/hostname to output header
#		   Update for 31 May 2015 patches, add EM-DB-DISC plugin home
# Change in v1.5:  Add repo DB check for SSL_VERSION and SSL_CIPHER_SUITES
#                  Add VERBOSE_CHECKSEC variable:
#                   Set to 0 for quiet run.
#                   Set to 1 to see failed check summary after run.
#                   Set to 2 for failed check summary and patch details.
# Change in v1.6:  Add PSU4 for EM12cR4, complete VERBOSE_CHECKSEC work
#                  Add 14 July 2015 patches
# Change in v1.7:  Update for 31 Jul 2015 patches
# Change in v1.8:  Update for 31 Aug 2015 patches
# Change in v1.9:  Add 17714229 for OMS home
#                  Add 21068288 CVE-2015-4742 for oracle_common home
#                  Add check for usage of demonstration SSL certificates
# Change in v1.10: Update for 1 Oct 2015 patches, PSU5, CPUOCT2015
#		   Added 18502187 for OMS home
# Change in v1.11: Update for 30 Nov 2015 patches
#
# From: @BrianPardy on Twitter
#
# Known functional on Linux x86-64, Solaris, AIX.
#
# Run this script as the Oracle EM12c software owner, with your environment
# fully up and running.
#
# Thanks to Dave Corsar, who tested on Solaris and let me know the 
# changes needed to make an earlier version work on Solaris.
#
# Thanks to opa tropa who confirmed AIX functionality and noted the 
# use of GNU extensions to grep, which I have since removed.
# 
# Dedicated to our two Lhasa Apsos:
#   Lucy (6/13/1998 - 3/13/2015)
#   Ethel (6/13/1998 - 7/31/2015)
#
# 

SCRIPTNAME=`basename $0`
PATCHDATE="30 Nov 2015"
OMSHOST=`hostname -f`
VERSION="1.11"
FAIL_COUNT=0
FAIL_TESTS=""

RUN_DB_CHECK=0
VERBOSE_CHECKSEC=2

HOST_OS=`uname -s`
HOST_ARCH=`uname -m`

ORAGCHOMELIST="/etc/oragchomelist"
ORATAB="/etc/oratab"

if [[ ! -r $ORAGCHOMELIST ]]; then			# Solaris
	ORAGCHOMELIST="/var/opt/oracle/oragchomelist"
fi

if [[ ! -r $ORATAB ]]; then 				# Solaris
	ORATAB="/var/opt/oracle/oratab"
fi

if [[ -x "/usr/sfw/bin/gegrep" ]]; then
	GREP=/usr/sfw/bin/gegrep
else
	GREP=`which grep`
fi

OMS_HOME=`$GREP -i oms $ORAGCHOMELIST | xargs ls -d 2>/dev/null`

OPATCH="$OMS_HOME/OPatch/opatch"
OPATCHAUTO="$OMS_HOME/OPatch/opatchauto"
OMSORAINST="$OMS_HOME/oraInst.loc"
ORAINVENTORY=`head -n 1 $OMSORAINST | awk -F= '{print $2}'`

MW_HOME=`dirname $OMS_HOME`
BIP_HOME=`$GREP -vi REMOVED $ORAINVENTORY/ContentsXML/inventory.xml | $GREP "HOME NAME=\"Oracle_BI" | awk '{print $3}' | sed -e 's/LOC=\"//' | sed -e 's/"//'`
COMMON_HOME=`$GREP -vi REMOVED $ORAINVENTORY/ContentsXML/inventory.xml | $GREP "HOME NAME=\"common" | awk '{print $3}' | sed -e 's/LOC=\"//' | sed -e 's/"//'`
WEBTIER_HOME=`$GREP -vi REMOVED $ORAINVENTORY/ContentsXML/inventory.xml | $GREP "HOME NAME=\"webtier" | awk '{print $3}' | sed -e 's/LOC=\"//' | sed -e 's/"//'`
AGENT_HOME=`$GREP -vi REMOVED $ORAINVENTORY/ContentsXML/inventory.xml | $GREP "HOME NAME=\"agent12c" | awk '{print $3}' | sed -e 's/LOC=\"//' | sed -e 's/"//'`
AGENT_DB_PLUGIN_HOME="$AGENT_HOME/../../plugins/oracle.sysman.db.agent.plugin_12.1.0.7.0"
AGENT_DB_PLUGIN_DISC_HOME="$AGENT_HOME/../../plugins/oracle.sysman.db.discovery.plugin_12.1.0.7.0"
AGENT_FMW_PLUGIN_HOME="$AGENT_HOME/../../plugins/oracle.sysman.emas.agent.plugin_12.1.0.7.0"
AGENT_FMW_PLUGIN_DISC_HOME="$AGENT_HOME/../../plugins/oracle.sysman.emas.discovery.plugin_12.1.0.7.0"
AGENT_BEACON_PLUGIN_HOME="$AGENT_HOME/../../plugins/oracle.sysman.beacon.agent.plugin_12.1.0.4.0"
AGENT_OH_PLUGIN_HOME="$AGENT_HOME/../../plugins/oracle.sysman.oh.agent.plugin_12.1.0.4.0"

EM_INSTANCE_BASE=`$GREP GCDomain $MW_HOME/domain-registry.xml | sed -e 's/.*=//' | sed -e 's/\/user_projects.*$//' | sed -e 's/"//'`
WL_HOME=`$GREP wlserver $MW_HOME/domain-registry.xml | sed -e 's/.*=//' | sed -e 's/\/samples.*$//' | sed -e 's/"//' | uniq`

EMGC_PROPS="$EM_INSTANCE_BASE/em/EMGC_OMS1/emgc.properties"
EMBIP_PROPS="$EM_INSTANCE_BASE/em/EMGC_OMS1/embip.properties"
OPMN_PROPS="$EM_INSTANCE_BASE/WebTierIH1/config/OPMN/opmn/ports.prop"
OHS_ADMIN_CONF="$EM_INSTANCE_BASE/WebTierIH1/config/OHS/ohs1/admin.conf"

PORT_UPL=`$GREP EM_UPLOAD_HTTPS_PORT $EMGC_PROPS | awk -F= '{print $2}'`
PORT_OMS=`$GREP EM_CONSOLE_HTTPS_PORT $EMGC_PROPS | awk -F= '{print $2}'`
PORT_OMS_JAVA=`$GREP MS_HTTPS_PORT $EMGC_PROPS | awk -F= '{print $2}'`
PORT_NODEMANAGER=`$GREP EM_NODEMGR_PORT $EMGC_PROPS | awk -F= '{print $2}'`
PORT_BIP=`$GREP BIP_HTTPS_PORT $EMBIP_PROPS | awk -F= '{print $2}'`
PORT_ADMINSERVER=`$GREP AS_HTTPS_PORT $EMGC_PROPS | awk -F= '{print $2}'`
PORT_OPMN=`$GREP '/opmn/remote_port' $OPMN_PROPS | awk -F= '{print $2}'`
PORT_OHS_ADMIN=`$GREP Listen $OHS_ADMIN_CONF | awk '{print $2}'`
PORT_AGENT=`$AGENT_HOME/bin/emctl status agent | $GREP 'Agent URL' | sed -e 's/\/emd\/main\///' | sed -e 's/^.*://' | uniq`

REPOS_DB_CONNDESC=`$GREP EM_REPOS_CONNECTDESCRIPTOR $EMGC_PROPS | sed -e 's/EM_REPOS_CONNECTDESCRIPTOR=//' | sed -e 's/\\\\//g'`
REPOS_DB_HOST=`echo $REPOS_DB_CONNDESC | sed -e 's/^.*HOST=//' | sed -e 's/).*$//'`
REPOS_DB_SID=`echo $REPOS_DB_CONNDESC | sed -e 's/^.*SID=//' | sed -e 's/).*$//'`

if [[ "$REPOS_DB_HOST" == "$OMSHOST" ]]; then
	REPOS_DB_HOME=`$GREP "$REPOS_DB_SID:" $ORATAB | awk -F: '{print $2}'`
	REPOS_DB_VERSION=`$REPOS_DB_HOME/OPatch/opatch lsinventory -oh $REPOS_DB_HOME | $GREP 'Oracle Database' | awk '{print $4}'`

	if [[ "$REPOS_DB_VERSION" == "11.2.0.4.0" ]]; then
		RUN_DB_CHECK=1
	fi

	if [[ "$REPOS_DB_VERSION" == "12.1.0.2.0" ]]; then
		RUN_DB_CHECK=1
	fi

	if [[ "$RUN_DB_CHECK" -eq 0 ]]; then
		echo -e "\tSkipping local repository DB patch check, only 11.2.0.4 or 12.1.0.2 supported by this script for now"
	fi
fi


sslcheck () {
	OPENSSL_CHECK_COMPONENT=$1
	OPENSSL_CHECK_HOST=$2
	OPENSSL_CHECK_PORT=$3
	OPENSSL_CHECK_PROTO=$4

	OPENSSL_RETURN=`echo Q | openssl s_client -prexit -connect $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT -$OPENSSL_CHECK_PROTO 2>&1 | $GREP Cipher | $GREP -c 0000`
	
	

	if [[ $OPENSSL_CHECK_PROTO == "tls1" ]]; then
		echo -en "\tConfirming $OPENSSL_CHECK_PROTO available for $OPENSSL_CHECK_COMPONENT at $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT... "
		if [[ $OPENSSL_RETURN -eq "0" ]]; then
			echo OK
		else
			echo FAILED
			FAIL_COUNT=$((FAIL_COUNT+1))
			FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OPENSSL_CHECK_COMPONENT @ $OPENSSL_CHECK_HOST:${OPENSSL_CHECK_PORT}:$OPENSSL_CHECK_PROTO protocol connection failed"
		fi
	fi

	if [[ $OPENSSL_CHECK_PROTO == "ssl2" || $OPENSSL_CHECK_PROTO == "ssl3" ]]; then
		echo -en "\tConfirming $OPENSSL_CHECK_PROTO disabled for $OPENSSL_CHECK_COMPONENT at $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT... "
		if [[ $OPENSSL_RETURN -ne "0" ]]; then
			echo OK
		else
			echo FAILED
			FAIL_COUNT=$((FAIL_COUNT+1))
			FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OPENSSL_CHECK_COMPONENT @ $OPENSSL_CHECK_HOST:${OPENSSL_CHECK_PORT}:$OPENSSL_CHECK_PROTO protocol connection succeeded"
		fi
	fi
}

opatchcheck () {
	OPATCH_CHECK_COMPONENT=$1
	OPATCH_CHECK_OH=$2
	OPATCH_CHECK_PATCH=$3

	if [[ "$OPATCH_CHECK_COMPONENT" == "ReposDBHome" ]]; then
		OPATCH_RET=`$OPATCH_CHECK_OH/OPatch/opatch lsinv -oh $OPATCH_CHECK_OH | $GREP $OPATCH_CHECK_PATCH`
	else
		OPATCH_RET=`$OPATCH lsinv -oh $OPATCH_CHECK_OH | $GREP $OPATCH_CHECK_PATCH`
	fi

	if [[ -z "$OPATCH_RET" ]]; then
		echo FAILED
		FAIL_COUNT=$((FAIL_COUNT+1))
		FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OPATCH_CHECK_COMPONENT @ ${OPATCH_CHECK_OH}:Patch $OPATCH_CHECK_PATCH not found"
	else
		echo OK
	fi

	test $VERBOSE_CHECKSEC -ge 2 && echo $OPATCH_RET

}

opatchautocheck () {
	OPATCHAUTO_CHECK_COMPONENT=$1
	OPATCHAUTO_CHECK_OH=$2
	OPATCHAUTO_CHECK_PATCH=$3

	OPATCHAUTO_RET=`$OPATCHAUTO lspatches -oh $OPATCHAUTO_CHECK_OH | $GREP $OPATCHAUTO_CHECK_PATCH`

	if [[ -z "$OPATCHAUTO_RET" ]]; then
		echo FAILED
		FAIL_COUNT=$((FAIL_COUNT+1))
		FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OPATCHAUTO_CHECK_COMPONENT @ ${OPATCHAUTO_CHECK_OH}:Patch $OPATCHAUTO_CHECK_PATCH not found"
	else
		echo OK
	fi

	test $VERBOSE_CHECKSEC -ge 2 && echo $OPATCHAUTO_RET

}

certcheck () {
	CERTCHECK_CHECK_COMPONENT=$1
	CERTCHECK_CHECK_HOST=$2
	CERTCHECK_CHECK_PORT=$3

	echo -ne "\tChecking certificate at $CERTCHECK_CHECK_COMPONENT ($CERTCHECK_CHECK_HOST:$CERTCHECK_CHECK_PORT)... "

	OPENSSL_SELFSIGNED_COUNT=`echo Q | openssl s_client -prexit -connect $CERTCHECK_CHECK_HOST:$CERTCHECK_CHECK_PORT 2>&1 | $GREP -ci "self signed certificate"`

	if [[ $OPENSSL_SELFSIGNED_COUNT -eq "0" ]]; then
		echo OK
	else
		echo FAILED - Found self-signed certificate
		FAIL_COUNT=$((FAIL_COUNT+1))
		FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$CERTCHECK_CHECK_COMPONENT @ ${CERTCHECK_CHECK_HOST}:${CERTCHECK_CHECK_PORT} found self-signed certificate"
	fi
}

democertcheck () {
	DEMOCERTCHECK_CHECK_COMPONENT=$1
	DEMOCERTCHECK_CHECK_HOST=$2
	DEMOCERTCHECK_CHECK_PORT=$3

	echo -ne "\tChecking certificate at $DEMOCERTCHECK_CHECK_COMPONENT ($DEMOCERTCHECK_CHECK_HOST:$DEMOCERTCHECK_CHECK_PORT)... "

	OPENSSL_DEMO_COUNT=`echo Q | openssl s_client -prexit -connect $DEMOCERTCHECK_CHECK_HOST:$DEMOCERTCHECK_CHECK_PORT 2>&1 | $GREP -ci "issuer=/C=US/ST=MyState/L=MyTown/O=MyOrganization/OU=FOR TESTING ONLY/CN=CertGenCAB"`

	if [[ $OPENSSL_DEMO_COUNT -eq "0" ]]; then
		echo OK
	else
		echo FAILED - Found demonstration certificate
		FAIL_COUNT=$((FAIL_COUNT+1))
		FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$DEMOCERTCHECK_CHECK_COMPONENT @ ${DEMOCERTCHECK_CHECK_HOST}:${DEMOCERTCHECK_CHECK_PORT} found demonstration certificate"
	fi
}


ciphercheck () {
	OPENSSL_CHECK_COMPONENT=$1
	OPENSSL_CHECK_HOST=$2
	OPENSSL_CHECK_PORT=$3

	echo -ne "\tChecking LOW strength ciphers on $OPENSSL_CHECK_COMPONENT ($OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT)..."

	OPENSSL_LOW_RETURN=`echo Q | openssl s_client -prexit -connect $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT -tls1 -cipher LOW 2>&1 | $GREP Cipher | uniq | $GREP -c 0000`

	if [[ $OPENSSL_LOW_RETURN -eq "0" ]]; then
		echo -e "\tFAILED - PERMITS LOW STRENGTH CIPHER CONNECTIONS"
		FAIL_COUNT=$((FAIL_COUNT+1))
		FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OPENSSL_CHECK_COMPONENT @ $OPENSSL_CHECK_HOST:${OPENSSL_CHECK_PORT}:Permits LOW strength ciphers"
	else
		echo -e "\tOK"
	fi


	echo -ne "\tChecking MEDIUM strength ciphers on $OPENSSL_CHECK_COMPONENT ($OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT)..."

	OPENSSL_MEDIUM_RETURN=`echo Q | openssl s_client -prexit -connect $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT -tls1 -cipher MEDIUM 2>&1 | $GREP Cipher | uniq | $GREP -c 0000`

	if [[ $OPENSSL_MEDIUM_RETURN -eq "0" ]]; then
		echo -e "\tFAILED - PERMITS MEDIUM STRENGTH CIPHER CONNECTIONS"
		FAIL_COUNT=$((FAIL_COUNT+1))
		FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OPENSSL_CHECK_COMPONENT @ $OPENSSL_CHECK_HOST:${OPENSSL_CHECK_PORT}:Permits MEDIUM strength ciphers"
	else
		echo -e "\tOK"
	fi



	echo -ne "\tChecking HIGH strength ciphers on $OPENSSL_CHECK_COMPONENT ($OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT)..."

	OPENSSL_HIGH_RETURN=`echo Q | openssl s_client -prexit -connect $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT -tls1 -cipher HIGH 2>&1 | $GREP Cipher | uniq | $GREP -c 0000`

	if [[ $OPENSSL_HIGH_RETURN -eq "0" ]]; then
		echo -e "\tOK"
	else
		echo -e "\tFAILED - CANNOT CONNECT WITH HIGH STRENGTH CIPHER"
		FAIL_COUNT=$((FAIL_COUNT+1))
		FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OPENSSL_CHECK_COMPONENT @ $OPENSSL_CHECK_HOST:${OPENSSL_CHECK_PORT}:Rejects HIGH strength ciphers"
	fi
	echo
}

wlspatchcheck () {
	WLSDIR=$1
	WLSPATCH=$2

	WLSCHECK_RETURN=`( cd $MW_HOME/utils/bsu && $MW_HOME/utils/bsu/bsu.sh -report ) | $GREP $WLSPATCH`
	WLSCHECK_COUNT=`echo $WLSCHECK_RETURN | wc -l`

	if [[ $WLSCHECK_COUNT -ge "1" ]]; then
		echo -e "\tOK"
	else
		echo -e "\tFAILED - PATCH NOT FOUND"
		FAIL_COUNT=$((FAIL_COUNT+1))
		FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$WLSDIR:Patch $WLSPATCH not found"
	fi

	test $VERBOSE_CHECKSEC -ge 2 && echo $WLSCHECK_RETURN
	
}

javacheck () {
	WHICH_JAVA=$1
	JAVA_DIR=$2

	JAVACHECK_RETURN=`$JAVA_DIR/bin/java -version 2>&1 | $GREP version | awk '{print $3}' | sed -e 's/"//g'`

	if [[ "$JAVACHECK_RETURN" == "1.6.0_95" ]]; then
		echo -e "\tOK"
	else
		#echo -e "\tFAILED - Found version $JAVACHECK_RETURN"
		echo -e "\tFAILED"
		FAIL_COUNT=$((FAIL_COUNT+1))
		FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$WHICH_JAVA Java in ${JAVA_DIR}:Found incorrect version $JAVACHECK_RETURN"
	fi
	test $VERBOSE_CHECKSEC -ge 2 && echo $JAVACHECK_RETURN
}

paramcheck () {
	WHICH_PARAM=$1
	WHICH_ORACLE_HOME=$2
	WHICH_FILE=$3

	PARAMCHECK_RETURN=`$GREP $WHICH_PARAM $WHICH_ORACLE_HOME/network/admin/$WHICH_FILE | awk -F= '{print $2}' | sed -e 's/\s//g'`
	if [[ "$WHICH_PARAM" == "SSL_VERSION" ]]; then
		if [[ "$PARAMCHECK_RETURN" == "1.0" ]]; then
			echo -e "OK"
		else
			echo -e "FAILED - Found $WHICH_PARAM = $PARAMCHECK_RETURN"
			FAIL_COUNT=$((FAIL_COUNT+1))
			FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$WHICH_PARAM in $WHICH_FILE for home ${WHICH_ORACLE_HOME}:incorrect parameter value"
		fi
		test $VERBOSE_CHECKSEC -ge 2 && echo $PARAMCHECK_RETURN
	fi

	if [[ "$WHICH_PARAM" == "SSL_CIPHER_SUITES" ]]; then
		if [[ "$PARAMCHECK_RETURN" == "(SSL_RSA_WITH_AES128_CBC_SHA,SSL_RSA_WITH_AES256_CBC_SHA)" ]]; then
			echo -e "OK"
		else
			echo -e "FAILED - Found $WHICH_PARAM = $PARAMCHECK_RETURN"
			FAIL_COUNT=$((FAIL_COUNT+1))
			FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$WHICH_PARAM in $WHICH_FILE for home ${WHICH_ORACLE_HOME}:incorrect parameter value"
		fi
		test $VERBOSE_CHECKSEC -ge 2 && echo $PARAMCHECK_RETURN
	fi
}


### MAIN SCRIPT HERE


echo -e "Performing EM12cR4 security checkup version $VERSION on $OMSHOST at `date`.\n"

echo "Using port definitions from configuration files "
echo -e "\t/etc/oragchomelist"
echo -e "\t$EMGC_PROPS"
echo -e "\t$EMBIP_PROPS"
echo -e "\t$OPMN_PROPS"
echo -e "\t$OHS_ADMIN_CONF"
echo
echo -e "\tAgent port found at $OMSHOST:$PORT_AGENT"
echo -e "\tBIPublisher port found at $OMSHOST:$PORT_BIP"
echo -e "\tNodeManager port found at $OMSHOST:$PORT_NODEMANAGER"
echo -e "\tOHSadmin port found at $OMSHOST:$PORT_OHS_ADMIN"
echo -e "\tOMSconsole port found at $OMSHOST:$PORT_OMS"
echo -e "\tOMSproxy port found at $OMSHOST:$PORT_OMS_JAVA"
echo -e "\tOMSupload port found at $OMSHOST:$PORT_UPL"
echo -e "\tOPMN port found at $OMSHOST:$PORT_OPMN"
echo -e "\tWLSadmin found at $OMSHOST:$PORT_ADMINSERVER"
echo
echo -e "\tRepository DB version=$REPOS_DB_VERSION SID=$REPOS_DB_SID host=$REPOS_DB_HOST"

if [[ $RUN_DB_CHECK -eq "1" ]]; then
	echo -e "\tRepository DB on OMS server, will check patches/parameters in $REPOS_DB_HOME"
fi


echo -e "\n(1) Checking SSL/TLS configuration (see notes 1602983.1, 1477287.1, 1905314.1)"

echo -e "\n\t(1a) Forbid SSLv2 connections"
sslcheck Agent $OMSHOST $PORT_AGENT ssl2
sslcheck BIPublisher $OMSHOST $PORT_BIP ssl2
sslcheck NodeManager $OMSHOST $PORT_NODEMANAGER ssl2
sslcheck OHSadmin $OMSHOST $PORT_OHS_ADMIN ssl2
sslcheck OMSconsole $OMSHOST $PORT_OMS ssl2
sslcheck OMSproxy $OMSHOST $PORT_OMS_JAVA ssl2
sslcheck OMSupload $OMSHOST $PORT_UPL ssl2
sslcheck OPMN $OMSHOST $PORT_OPMN ssl2
sslcheck WLSadmin $OMSHOST $PORT_ADMINSERVER ssl2

echo -e "\n\t(1b) Forbid SSLv3 connections"
sslcheck Agent $OMSHOST $PORT_AGENT ssl3
sslcheck BIPublisher $OMSHOST $PORT_BIP ssl3
sslcheck NodeManager $OMSHOST $PORT_NODEMANAGER ssl3
sslcheck OHSadmin $OMSHOST $PORT_OHS_ADMIN ssl3
sslcheck OMSconsole $OMSHOST $PORT_OMS ssl3
sslcheck OMSproxy $OMSHOST $PORT_OMS_JAVA ssl3
sslcheck OMSupload $OMSHOST $PORT_UPL ssl3
sslcheck OPMN $OMSHOST $PORT_OPMN ssl3
sslcheck WLSadmin $OMSHOST $PORT_ADMINSERVER ssl3

echo -e "\n\t(1c) Permit TLSv1 connections"
sslcheck Agent $OMSHOST $PORT_AGENT tls1
sslcheck BIPublisher $OMSHOST $PORT_BIP tls1
sslcheck NodeManager $OMSHOST $PORT_NODEMANAGER tls1
sslcheck OHSadmin $OMSHOST $PORT_OHS_ADMIN tls1
sslcheck OMSconsole $OMSHOST $PORT_OMS tls1
sslcheck OMSproxy $OMSHOST $PORT_OMS_JAVA tls1
sslcheck OMSupload $OMSHOST $PORT_UPL tls1
sslcheck OPMN $OMSHOST $PORT_OPMN tls1
sslcheck WLSadmin $OMSHOST $PORT_ADMINSERVER tls1

echo -e "\n(2) Checking supported ciphers at SSL/TLS endpoints (see notes 1477287.1, 1905314.1, 1067411.1)"
ciphercheck Agent $OMSHOST $PORT_AGENT
ciphercheck BIPublisher $OMSHOST $PORT_BIP
ciphercheck NodeManager $OMSHOST $PORT_NODEMANAGER
ciphercheck OHSadmin $OMSHOST $PORT_OHS_ADMIN
ciphercheck OMSconsole $OMSHOST $PORT_OMS
ciphercheck OMSproxy $OMSHOST $PORT_OMS_JAVA
ciphercheck OMSupload $OMSHOST $PORT_UPL
ciphercheck OPMN $OMSHOST $PORT_OPMN
ciphercheck WLSadmin $OMSHOST $PORT_ADMINSERVER

echo -e "\n(3) Checking self-signed and demonstration certificates at SSL/TLS endpoints (see notes 1367988.1, 1399293.1, 1593183.1, 1527874.1, 123033.1, 1937457.1)"
certcheck Agent $OMSHOST $PORT_AGENT
democertcheck Agent $OMSHOST $PORT_AGENT
certcheck BIPublisher $OMSHOST $PORT_BIP
democertcheck BIPublisher $OMSHOST $PORT_BIP
certcheck NodeManager $OMSHOST $PORT_NODEMANAGER
democertcheck NodeManager $OMSHOST $PORT_NODEMANAGER
certcheck OHSadmin $OMSHOST $PORT_OHS_ADMIN
democertcheck OHSadmin $OMSHOST $PORT_OHS_ADMIN
certcheck OMSconsole $OMSHOST $PORT_OMS
democertcheck OMSconsole $OMSHOST $PORT_OMS
certcheck OMSproxy $OMSHOST $PORT_OMS_JAVA
democertcheck OMSproxy $OMSHOST $PORT_OMS_JAVA
certcheck OMSupload $OMSHOST $PORT_UPL
democertcheck OMSupload $OMSHOST $PORT_UPL
certcheck OPMN $OMSHOST $PORT_OPMN
democertcheck OPMN $OMSHOST $PORT_OPMN
certcheck WLSadmin $OMSHOST $PORT_ADMINSERVER
democertcheck WLSadmin $OMSHOST $PORT_ADMINSERVER


echo -e "\n(4) Checking EM12c Oracle home patch levels against $PATCHDATE baseline (see notes 1664074.1, 1900943.1, 822485.1, 1470197.1, 1967243.1)"

#echo -ne "\n\t(4a) OMS ($OMS_HOME) PSU2 Patch 19830994... "
#opatchcheck OMS $OMS_HOME 19830994

#echo -ne "\n\t(4a) OMS ($OMS_HOME) ENTERPRISE MANAGER BASE PLATFORM - OMS 12.1.0.4.3 PSU Patch (20392036)... "
#opatchcheck OMS $OMS_HOME 20392036

#echo -ne "\n\t(4a) OMS ($OMS_HOME) ENTERPRISE MANAGER BASE PLATFORM - OMS 12.1.0.4.4 PSU Patch (20870437)... "
#opatchcheck OMS $OMS_HOME 20870437

echo -ne "\n\t(4a) OMS ($OMS_HOME) ENTERPRISE MANAGER BASE PLATFORM - OMS 12.1.0.4.5 PSU Patch (21462217)... "
opatchcheck OMS $OMS_HOME 21462217

echo -ne "\n\t(4a) OMS HOME ($AGENT_HOME) JDBC Merge Patch (18502187)... "
opatchcheck OMS $OMS_HOME 18502187

#echo -ne "\n\t(4a) OMS ($OMS_HOME) DO NOT CREATE INCIDENT WHEN A COMMAND IS OVER RUN IN JOB WORKER (17714229)... "
#opatchcheck OMS $OMS_HOME 17714229

echo -ne "\n\t(4b) BI Publisher ($BIP_HOME) CPUJAN2015 Patch (19822893)... "
opatchcheck BIP $BIP_HOME 19822893

echo -ne "\n\t(4b) BI Publisher ($BIP_HOME) Merge Patch (20444447)... "
opatchcheck BIP $BIP_HOME 20444447

#echo -ne "\n\t(4b) BI Publisher ($BIP_HOME) ORACLE BI PUBLISHER PATCH BUG FOR PRIVATE EMCC PS3 MANDATORY INSTALL PATCH (17888172)... "
#opatchcheck BIP $BIP_HOME 17888172

echo -ne "\n\t(4c) AS Common ($COMMON_HOME) CVE-2015-0426 Oracle Help Patch (20075252)... "
opatchcheck COMMON $COMMON_HOME 20075252

#echo -ne "\n\t(4c) AS Common ($COMMON_HOME) ADF MERGE REQUEST ON TOP OF 11.1.1.7.1 FOR BUGS 20465665 18820382 20645397 (20747356)... "
#opatchcheck COMMON $COMMON_HOME 20747356

echo -ne "\n\t(4c) AS Common ($COMMON_HOME) WEBCENTER PORTAL BUNDLE PATCH 11.1.1.7.1 (16761779)... "
opatchcheck COMMON $COMMON_HOME 16761779

# Replaced 20747356, commented out above
echo -ne "\n\t(4c) AS Common ($COMMON_HOME) CVE-2015-4742 MERGE REQUEST ON TOP OF 11.1.1.7.1 FOR BUGS 20747356 18274008 (21068288)... "
opatchcheck COMMON $COMMON_HOME 21068288


#echo -ne "\n\t(4d) WebLogic Server ($WL_HOME) 10.3.6.0.10 12UV Patch (19637463)... "
#wlspatchcheck $WL_HOME 19637463

#echo -ne "\n\t(4d) WebLogic Server ($WL_HOME) 10.3.6.0.11 YUIS Patch (20181997)... "
#wlspatchcheck $WL_HOME 20181997

echo -ne "\n\t(4d) WebLogic Server ($WL_HOME) 10.3.6.0.12 EJUW Patch (20780171)... "
wlspatchcheck $WL_HOME 20780171

echo -ne "\n\t(4d) WebLogic Server ($WL_HOME) SU Patch [GDFA]: WEBLOGIC.STORE.PERSISTENTSTOREEXCEPTION: [STORE:280040] OCCURS EASILEY (16420963)... "
wlspatchcheck $WL_HOME 16420963

# Commented this patch out 4/17/2015, as Oracle no longer recommends it for EM12c installations.
# This patch still appears in note 1664074.1 for EM12c.
# Per personal communication w/Oracle I do NOT recommend using it.
#echo -ne "\n\t(4e) WebTier ($WEBTIER_HOME) CPUJAN2015 Patch (19948000)... "
#opatchcheck WebTier $WEBTIER_HOME 19948000

echo -ne "\n\t(4e) WebTier ($WEBTIER_HOME) OHS SECURITY PATCH UPDATE 11.1.1.7.0 CPUOCT2015 Patch (21640624)... "
opatchcheck WebTier $WEBTIER_HOME 21640624

echo -ne "\n\t(4e) WebTier ($WEBTIER_HOME) CVE-2014-4212 OPMN Patch (19345576)... "
opatchcheck WebTier $WEBTIER_HOME 19345576

#echo -ne "\n\t(4e) WebTier ($WEBTIER_HOME) CVE-2013-3836 PLACEHOLDER FOR SECURITY PATCH FOR WEBCACHE 11.1.1.7.0 WITH OCT2013 CPU (17306880)... "
#opatchcheck WebTier $WEBTIER_HOME 17306880

echo -ne "\n\t(4e) WebTier ($WEBTIER_HOME) CVE 2015-2658 MERGE REQUEST ON TOP OF 11.1.1.7.0 FOR BUGS 16370190 20310323 20715657 (20807683)... "
opatchcheck WebTier $WEBTIER_HOME 20807683

echo -ne "\n\t(4e) WebTier ($WEBTIER_HOME) CVE-2013-0169,CVE-2011-3389 OSS SECURITY PATCH UPDATE 11.1.1.7.0 CPUOCT2013 (17337741)... "
opatchcheck WebTier $WEBTIER_HOME 17337741

echo -ne "\n\t(4e) WebTier ($WEBTIER_HOME) WLSPLUGINS (OHS) SECURITY PATCH UPDATE 11.1.1.7.0 CPUJUL2014 (18423831)... "
opatchcheck WebTier $WEBTIER_HOME 18423831

#echo -ne "\n\t(4f) OMS ($OMS_HOME) DB PLUGIN BUNDLE 12.1.0.7.2 (20613714)... "
#opatchautocheck OMS $OMS_HOME 20613714

#echo -ne "\n\t(4f) OMS ($OMS_HOME) DB PLUGIN BUNDLE PATCH 12.1.0.7.3 (20804122)... "
#opatchautocheck OMS $OMS_HOME 20804122

#echo -ne "\n\t(4f) OMS ($OMS_HOME) DB PLUGIN BUNDLE PATCH 12.1.0.7.4 (20950048)... "
#opatchautocheck OMS $OMS_HOME 20950048

#echo -ne "\n\t(4f) OMS ($OMS_HOME) DB PLUGIN BUNDLE PATCH 12.1.0.7.5 (21167937)... "
#opatchautocheck OMS $OMS_HOME 21167937

#echo -ne "\n\t(4f) OMS ($OMS_HOME) DB PLUGIN BUNDLE PATCH 12.1.0.7.6 (21324654)... "
#opatchautocheck OMS $OMS_HOME 21324654

#echo -ne "\n\t(4f) OMS ($OMS_HOME) DB PLUGIN BUNDLE PATCH 12.1.0.7.7 (21506301)... "
#opatchautocheck OMS $OMS_HOME 21506301

#echo -ne "\n\t(4f) OMS ($OMS_HOME) DB PLUGIN BUNDLE PATCH 12.1.0.7.8 (21744938)... "
#opatchautocheck OMS $OMS_HOME 21744938

echo -ne "\n\t(4f) *UPDATED* OMS ($OMS_HOME) DB PLUGIN BUNDLE PATCH 12.1.0.7.10 (22062307)... "
opatchautocheck OMS $OMS_HOME 22062307

#echo -ne "\n\t(4g) OMS ($OMS_HOME) FMW PLUGIN BUNDLE 12.1.0.7.2 (20613870)... "
#opatchautocheck OMS $OMS_HOME 20613870

#echo -ne "\n\t(4g) OMS ($OMS_HOME) FMW PLUGIN BUNDLE PATCH 12.1.0.7.3 (20804213)... "
#opatchautocheck OMS $OMS_HOME 20804213

#echo -ne "\n\t(4g) OMS ($OMS_HOME) FMW PLUGIN BUNDLE PATCH 12.1.0.7.4 (20950040)... "
#opatchautocheck OMS $OMS_HOME 20950040

#echo -ne "\n\t(4g) OMS ($OMS_HOME) FMW PLUGIN BUNDLE PATCH 12.1.0.7.5 (21167980)... "
#opatchautocheck OMS $OMS_HOME 21167980

#echo -ne "\n\t(4g) OMS ($OMS_HOME) FMW PLUGIN BUNDLE PATCH 12.1.0.7.6 (21324861)... "
#opatchautocheck OMS $OMS_HOME 21324861

#echo -ne "\n\t(4g) OMS ($OMS_HOME) FMW PLUGIN BUNDLE PATCH 12.1.0.7.7 (21506335)... "
#opatchautocheck OMS $OMS_HOME 21506335

#echo -ne "\n\t(4g) OMS ($OMS_HOME) FMW PLUGIN BUNDLE PATCH 12.1.0.7.8 (21744989)... "
#opatchautocheck OMS $OMS_HOME 21744989

echo -ne "\n\t(4g) *UPDATED* OMS ($OMS_HOME) FMW PLUGIN BUNDLE PATCH 12.1.0.7.10 (22062375)... "
opatchautocheck OMS $OMS_HOME 22062375

#echo -ne "\n\t(4h) OMS ($OMS_HOME) MOS PLUGIN BUNDLE PATCH 12.1.0.6.4 (20613886)... "
#opatchautocheck OMS $OMS_HOME 20613886

#echo -ne "\n\t(4h) OMS ($OMS_HOME) MOS PLUGIN BUNDLE PATCH 12.1.0.6.5 (20822914)... "
#opatchautocheck OMS $OMS_HOME 20822914

#echo -ne "\n\t(4h) OMS ($OMS_HOME) MOS PLUGIN BUNDLE PATCH 12.1.0.6.6 (21167991)... "
#opatchautocheck OMS $OMS_HOME 21167991

#echo -ne "\n\t(4h) OMS ($OMS_HOME) MOS PLUGIN BUNDLE PATCH 12.1.0.6.7 (21506428)... "
#opatchautocheck OMS $OMS_HOME 21506428

echo -ne "\n\t(4h) OMS ($OMS_HOME) MOS PLUGIN BUNDLE PATCH 12.1.0.6.8 (21745018)... "
opatchautocheck OMS $OMS_HOME 21745018

#echo -ne "\n\t(4i) OMS ($OMS_HOME) EXADATA PLUGIN BUNDLE 12.1.0.6.6 (20613853)... "
#opatchautocheck OMS $OMS_HOME 20613853

#echo -ne "\n\t(4i) OMS ($OMS_HOME) EXADATA PLUGIN BUNDLE PATCH 12.1.0.6.7 (20822866)... "
#opatchautocheck OMS $OMS_HOME 20822866

#echo -ne "\n\t(4i) OMS ($OMS_HOME) EXADATA PLUGIN BUNDLE PATCH 12.1.0.6.8 (20962507)... "
#opatchautocheck OMS $OMS_HOME 20962507

#echo -ne "\n\t(4i) OMS ($OMS_HOME) EXADATA PLUGIN BUNDLE PATCH 12.1.0.6.9 (21167953)... "
#opatchautocheck OMS $OMS_HOME 21167953

#echo -ne "\n\t(4i) OMS ($OMS_HOME) EXADATA PLUGIN BUNDLE PATCH 12.1.0.6.10 (21324852)... "
#opatchautocheck OMS $OMS_HOME 21324852

echo -ne "\n\t(4i) *UPDATED* OMS ($OMS_HOME) EXADATA PLUGIN BUNDLE PATCH 12.1.0.6.11 (21744966)... "
opatchautocheck OMS $OMS_HOME 21744966

#echo -ne "\n\t(4j) OMS CHAINED AGENT HOME ($AGENT_HOME) EM-AGENT BUNDLE 12.1.0.4.7 (20613931)... "
#opatchcheck Agent $AGENT_HOME 20613931

#echo -ne "\n\t(4j) OMS ($OMS_HOME) CFW PLUGIN BUNDLE PATCH 12.1.0.2.1 (20385040)... "
#opatchautocheck OMS $OMS_HOME 20385040

#echo -ne "\n\t(4j) OMS ($OMS_HOME) CFW PLUGIN BUNDLE PATCH 12.1.0.2.2 (21167573)... "
#opatchautocheck OMS $OMS_HOME 21167573

#echo -ne "\n\t(4j) OMS ($OMS_HOME) CFW PLUGIN BUNDLE PATCH 12.1.0.2.3 (21324632)... "
#opatchautocheck OMS $OMS_HOME 21324632

echo -ne "\n\t(4j) *UPDATED* OMS ($OMS_HOME) CFW PLUGIN BUNDLE PATCH 12.1.0.2.4 (21972104)... "
opatchautocheck OMS $OMS_HOME 21972104

#echo -ne "\n\t(4k) OMS CHAINED AGENT HOME ($AGENT_HOME) EM-AGENT BUNDLE PATCH 12.1.0.4.9 (20950034)... "
#opatchcheck Agent $AGENT_HOME 20950034

#echo -ne "\n\t(4k) OMS CHAINED AGENT HOME ($AGENT_HOME) EM-AGENT BUNDLE PATCH 12.1.0.4.10 (21168025)... "
#opatchcheck Agent $AGENT_HOME 21168025

#echo -ne "\n\t(4k) OMS CHAINED AGENT HOME ($AGENT_HOME) EM-AGENT BUNDLE PATCH 12.1.0.4.11 (21325110)... "
#opatchcheck Agent $AGENT_HOME 21325110

#echo -ne "\n\t(4k) OMS CHAINED AGENT HOME ($AGENT_HOME) EM-AGENT BUNDLE PATCH 12.1.0.4.12 (21506284)... "
#opatchcheck Agent $AGENT_HOME 21506284

#echo -ne "\n\t(4k) OMS CHAINED AGENT HOME ($AGENT_HOME) EM-AGENT BUNDLE PATCH 12.1.0.4.13 (21759280)... "
#opatchcheck Agent $AGENT_HOME 21759280

echo -ne "\n\t(4k) *UPDATED* OMS CHAINED AGENT HOME ($AGENT_HOME) EM-AGENT BUNDLE PATCH 12.1.0.4.14 (21913823)... "
opatchcheck Agent $AGENT_HOME 21913823

echo -ne "\n\t(4k) OMS CHAINED AGENT HOME ($AGENT_HOME) Merge Patch (18502187)... "
opatchcheck Agent $AGENT_HOME 18502187

echo -ne "\n\t(4k) OMS CHAINED AGENT HOME ($AGENT_HOME) JDBC Security Patch (18721761)... "
opatchcheck Agent $AGENT_HOME 18721761

if [[ "$HOST_OS" == "Linux" && "$HOST_ARCH" == "x86_64" ]]; then
	echo -ne "\n\t(4k) OMS CHAINED AGENT HOME ($AGENT_HOME) CVE 2012-3137 EM Agent only: Instant Client Security Patch (20114054)... "
	opatchcheck Agent $AGENT_HOME 20114054
fi

#echo -ne "\n\t(4k) OMS CHAINED AGENT DB PLUGIN ($AGENT_DB_PLUGIN_HOME) DB PLUGIN BUNDLE 12.1.0.7.2 AGENT-SIDE 20676926... "
#opatchcheck AgentDBPlugin $AGENT_DB_PLUGIN_HOME 20676926

#echo -ne "\n\t(4l) OMS CHAINED AGENT DB PLUGIN ($AGENT_DB_PLUGIN_HOME) DB PLUGIN BUNDLE 12.1.0.7.4 AGENT-SIDE MONITORING (21065223)... "
#opatchcheck AgentDBPlugin $AGENT_DB_PLUGIN_HOME 21065223

#echo -ne "\n\t(4l) OMS CHAINED AGENT DB PLUGIN ($AGENT_DB_PLUGIN_HOME) DB PLUGIN BUNDLE 12.1.0.7.5 AGENT-SIDE MONITORING (21229731)... "
#opatchcheck AgentDBPlugin $AGENT_DB_PLUGIN_HOME 21229731

#echo -ne "\n\t(4l) OMS CHAINED AGENT DB PLUGIN ($AGENT_DB_PLUGIN_HOME) DB PLUGIN BUNDLE 12.1.0.7.6 AGENT-SIDE MONITORING (21415075)... "
#opatchcheck AgentDBPlugin $AGENT_DB_PLUGIN_HOME 21415075

#echo -ne "\n\t(4l) OMS CHAINED AGENT DB PLUGIN ($AGENT_DB_PLUGIN_HOME) DB PLUGIN BUNDLE 12.1.0.7.7 AGENT-SIDE MONITORING (21603371)... "
#opatchcheck AgentDBPlugin $AGENT_DB_PLUGIN_HOME 21603371

#echo -ne "\n\t(4l) OMS CHAINED AGENT DB PLUGIN ($AGENT_DB_PLUGIN_HOME) DB PLUGIN BUNDLE 12.1.0.7.8 AGENT-SIDE MONITORING (21806804)... "
#opatchcheck AgentDBPlugin $AGENT_DB_PLUGIN_HOME 21806804

echo -ne "\n\t(4l) *UPDATED* OMS CHAINED AGENT DB PLUGIN ($AGENT_DB_PLUGIN_HOME) DB PLUGIN BUNDLE 12.1.0.7.10 AGENT-SIDE MONITORING (22140476)... "
opatchcheck AgentDBPlugin $AGENT_DB_PLUGIN_HOME 22140476

echo -ne "\n\t(4l) OMS CHAINED AGENT DB PLUGIN ($AGENT_DB_PLUGIN_DISC_HOME) DB PLUGIN BUNDLE 12.1.0.7.4 AGENT-SIDE DISCOVERY (21065239)... "
opatchcheck AgentDBPlugin $AGENT_DB_PLUGIN_DISC_HOME 21065239

#echo -ne "\n\t(4l) OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_HOME) FMW PLUGIN BUNDLE 12.1.0.7.2 AGENT-SIDE MONITORING (20677020)... "
#opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_HOME 20677020

#echo -ne "\n\t(4m) OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_HOME) FMW PLUGIN BUNDLE 12.1.0.7.4 AGENT-SIDE MONITORING (21065760)... "
#opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_HOME 21065760

#echo -ne "\n\t(4m) OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_HOME) FMW PLUGIN BUNDLE 12.1.0.7.5 AGENT-SIDE MONITORING (21229821)... "
#opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_HOME 21229821

#echo -ne "\n\t(4m) OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_HOME) FMW PLUGIN BUNDLE 12.1.0.7.6 AGENT-SIDE MONITORING (21415166)... "
#opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_HOME 21415166

#echo -ne "\n\t(4m) OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_HOME) FMW PLUGIN BUNDLE 12.1.0.7.7 AGENT-SIDE MONITORING (21603497)... "
#opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_HOME 21603497

#echo -ne "\n\t(4m) OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_HOME) FMW PLUGIN BUNDLE 12.1.0.7.8 AGENT-SIDE MONITORING (21806984)... "
#opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_HOME 21806984

#echo -ne "\n\t(4m) OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_HOME) FMW PLUGIN BUNDLE 12.1.0.7.8 AGENT-SIDE MONITORING (21806984)... "
#opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_HOME 21806984

echo -ne "\n\t(4m) *UPDATED* OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_HOME) FMW PLUGIN BUNDLE 12.1.0.7.9 AGENT-SIDE MONITORING (21941290)... "
opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_HOME 21941290

#echo -ne "\n\t(4m) OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_DISC_HOME) FMW PLUGIN BUNDLE 12.1.0.7.2 AGENT-SIDE DISCOVERY (20677038)... "
#opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_DISC_HOME 20677038

#echo -ne "\n\t(4m) OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_DISC_HOME) FMW PLUGIN BUNDLE 12.1.0.7.5 AGENT-SIDE DISCOVERY (21229841)... "
#opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_DISC_HOME 21229841

echo -ne "\n\t(4m) OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_DISC_HOME) FMW PLUGIN BUNDLE 12.1.0.7.7 AGENT-SIDE DISCOVERY (21611921)... "
opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_DISC_HOME 21611921

#echo -ne "\n\t(4n) OMS CHAINED AGENT BEACON PLUGIN ($AGENT_BEACON_PLUGIN_HOME) EM-BEACON BUNDLE PATCH 12.1.0.4.1 (20466772)... "
#opatchcheck AgentBeaconPlugin $AGENT_BEACON_PLUGIN_HOME 20466772

echo -ne "\n\t(4n) *UPDATED* OMS CHAINED AGENT BEACON PLUGIN ($AGENT_BEACON_PLUGIN_HOME) EM-BEACON BUNDLE PATCH 12.1.0.4.2 (21928148)... "
opatchcheck AgentBeaconPlugin $AGENT_BEACON_PLUGIN_HOME 21928148

echo -ne "\n\t(4o) OMS CHAINED AGENT EM-OH BUNDLE PATCH 12.1.0.4.1 (20855134)... "
opatchcheck AgentOHPlugin $AGENT_OH_PLUGIN_HOME 20855134


if [[ $RUN_DB_CHECK -eq 1 ]]; then

#	if [[ "$REPOS_DB_VERSION" == "11.2.0.4.0" ]]; then
#		echo -ne "\n\t(4m) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) PSU 11.2.0.4.5 19769489... "
#		opatchcheck ReposDBHome $REPOS_DB_HOME 19769489
#
#		echo -ne "\n\t(4m) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) ORACLE JAVAVM COMPONENT 11.2.0.4.2 DATABASE PSU (JAN2015) 19877440... "
#		opatchcheck ReposDBHome $REPOS_DB_HOME 19877440
#	fi

	if [[ "$REPOS_DB_VERSION" == "11.2.0.4.0" ]]; then
		#echo -ne "\n\t(4p) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) PSU 11.2.0.4.6 (APR2015) (20299013)... "
		#opatchcheck ReposDBHome $REPOS_DB_HOME 20299013

		echo -ne "\n\t(4p) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) PSU 11.2.0.4.8 (OCT2015) (21352635)... "
		opatchcheck ReposDBHome $REPOS_DB_HOME 21352635

		#echo -ne "\n\t(4p) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) ORACLE JAVAVM COMPONENT 11.2.0.4.3 DATABASE PSU (APR2015) (20406239)... "
		#opatchcheck ReposDBHome $REPOS_DB_HOME 20406239

		echo -ne "\n\t(4p) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) ORACLE JAVAVM COMPONENT 11.2.0.4.5 DATABASE PSU (OCT2015) (21555791)... "
		opatchcheck ReposDBHome $REPOS_DB_HOME 21555791
	fi

#	if [[ "$REPOS_DB_VERSION" == "12.1.0.2.0" ]]; then
#		echo -ne "\n\t(4m) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) Required Patch 20243268... "
#		opatchcheck ReposDBHome $REPOS_DB_HOME 20243268
#
#		echo -ne "\n\t(4m) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) PSU 12.1.0.2.2 19769480... "
#		opatchcheck ReposDBHome $REPOS_DB_HOME 19769480
#
#		echo -ne "\n\t(4m) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) ORACLE JAVAVM COMPONENT 12.1.0.2.2 ORACLE JAVAVM COMPONENT 12.1.0.2.2 DATABASE PSU (JAN2015) 19877336... "
#		opatchcheck ReposDBHome $REPOS_DB_HOME 19877336
#	fi

	if [[ "$REPOS_DB_VERSION" == "12.1.0.2.0" ]]; then
		echo -ne "\n\t(4p) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) Required Patch (20243268)... "
		opatchcheck ReposDBHome $REPOS_DB_HOME 20243268

		#echo -ne "\n\t(4p) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) PSU 12.1.0.2.3 (APR2015) (20299023)... "
		#opatchcheck ReposDBHome $REPOS_DB_HOME 20299023

		echo -ne "\n\t(4p) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) PSU 12.1.0.2.5 (OCT2015) (21359755)... "
		opatchcheck ReposDBHome $REPOS_DB_HOME 21359755

		#echo -ne "\n\t(4p) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) ORACLE JAVAVM COMPONENT 12.1.0.2.3 DATABASE PSU (APR2015) (20415564)... "
		#opatchcheck ReposDBHome $REPOS_DB_HOME 20415564

		echo -ne "\n\t(4p) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) ORACLE JAVAVM COMPONENT 12.1.0.2.5 DATABASE PSU (OCT2015) (21555660)... "
		opatchcheck ReposDBHome $REPOS_DB_HOME 21555660
	fi

	echo -ne "\n\t(4q) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) sqlnet.ora SSL_VERSION parameter (1545816.1)... "
	paramcheck SSL_VERSION $REPOS_DB_HOME sqlnet.ora

	echo -ne "\n\t(4q) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) sqlnet.ora SSL_CIPHER_SUITES parameter (1545816.1)... "
	paramcheck SSL_CIPHER_SUITES $REPOS_DB_HOME sqlnet.ora

	echo -ne "\n\t(4q) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) listener.ora SSL_VERSION parameter (1545816.1)... "
	paramcheck SSL_VERSION $REPOS_DB_HOME listener.ora

	echo -ne "\n\t(4q) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) listener.ora SSL_CIPHER_SUITES parameter (1545816.1)... "
	paramcheck SSL_CIPHER_SUITES $REPOS_DB_HOME listener.ora
fi

echo

echo -e "\n(5) Checking EM12c Java versions against baseline (see notes 1506916.1, 1492980.1)"

echo -ne "\n\t(5a) MW ($MW_HOME/jdk16/jdk) Java version 1.6.0_95 (9553040)... "
javacheck MW $MW_HOME/jdk16/jdk 1.6.0_95

echo -ne "\n\t(5b) WebTier ($WEBTIER_HOME/jdk) Java version 1.6.0_95 (9553040)... "
javacheck WebTier $WEBTIER_HOME/jdk 1.6.0_95

echo

if [[ $FAIL_COUNT -gt "0" ]]; then
	echo "Failed test count: $FAIL_COUNT - Review output"
	test $VERBOSE_CHECKSEC -ge 1 && echo -e $FAIL_TESTS
else
	echo "All tests succeeded."
fi

echo
echo "Visit https://pardydba.wordpress.com/2015/03/09/em12c-r4-ssl-security-checkup-script/ for the latest version."
echo

exit

If you try this script, please leave me a comment.  If you can share any changes you’ve made that allow it to run on other operating systems, I and others will appreciate it. I spent a lot of time making it so the user does not have to specify any directory locations or port settings, so if you have code changes to offer please let me know.  If enough people use this I may learn how to put it on github or something.

Good luck and happy compliance audits!

Further Reading

Step by step: Configuring third party SSL/TLS certificates in EM12c R4

By default, when an administrator configures Oracle Enterprise Manager 12c to use SSL, the system will use a default self-signed certificate, provided for demo purposes only.  The documentation states repeatedly that users should not use these certificates in a production environment, as they represent a security risk. This blog post documents, step by step, a process to replace these demo certificates with custom third party certificates, across the OMS console, OMS upload port, agents, and WebLogic Server. I will follow this process on a single-OMS configuration; if you have more than one OMS please consult the documentation for more details, as your process will vary and the steps I have provided may break your system.

I have tested these instructions on Linux x86-64 (SLES11 SP3) with EM12c R4 PSU2 (12.1.0.4).

Official Documentation

The official documentation for this process resides in the following My Oracle Support notes:

  • Using ORAPKI Utility to Create a Wallet with Third Party Trusted Certificate and Import into OMS (Doc ID 1367988.1)
  • EM 12c Cloud Control How to Create a Wallet With Third Party Trusted Certificate that Can Be Imported into the OMS For SSL Comunication ? (Doc ID 1399293.1)
  • 12c Cloud Control: Steps to Import Third Party Trusted SSL Certificate into 12c Cloud Control Agent URL (Doc ID 1593183.1)
  • 12c Cloud Control: Steps to Create and Import Third Party / Self-Signed SSL Certificates for WebLogic Server in an Enterprise Manager Installation (Doc ID 1527874.1)
  • How to Create a Java Keystore via Keytool in FMW 11g/12c (Doc ID 1230333.1)

Why Should I Do This?

You may not fully understand the mechanics of SSL/TLS certificates and the chain of trust. I cannot fully explain this complex topic in a blog post, but if you need a reason to make this change other than demands from your organizational security/compliance team, please take Oracle’s word for it, and notice this text that appears in your GCDomain.log file when you run your system with the provided default demo certificates:

####<Feb 12, 2015 2:58:07 PM EST> <Alert> <Security> <[hostname redacted]> <EMGC_ADMINSERVER> <VDE Replication Thread> <<WLS Kernel>> <> <> <1423771087209> <BEA-090152> <Demo trusted CA certificate is being used in production mode: [
[
 Version: V3
 Subject: CN=CACERT, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
 Signature Algorithm: MD5withRSA, OID = 1.2.840.113549.1.1.4

 Key: Sun RSA public key, 512 bits
 modulus: 9550192877869244258838480703390456015046425375252278279190673063544122510925482179963329236052146047356415957587628011282484772458983977898996276815440753
 public exponent: 65537
 Validity: [From: Thu Mar 21 15:12:27 EST 2002,
 To: Tue Mar 22 16:12:27 EDT 2022]
 Issuer: CN=CACERT, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
 SerialNumber: [ 33f10648 fcde0deb 4199921f d64537f4]

Certificate Extensions: 1
[1]: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
 Key_CertSign
]

] 
 Algorithm: [MD5withRSA]
 Signature:
0000: 9D 26 4C 29 C8 91 C3 A7 06 C3 24 6F AE B4 F8 82 .&L)......$o.... 
0010: 80 4D AA CB 7C 79 46 84 81 C4 66 95 F4 1E D8 C4 .M...yF...f..... 
0020: E9 B7 D9 7C E2 23 33 A4 B7 21 E0 AA 54 2B 4A FF .....#3..!..T+J. 
0030: CB 21 20 88 81 21 DB AC 90 54 D8 7D 79 63 23 3C .! ..!...T..yc#<

] The system is vulnerable to security attacks, since it trusts certificates signed by the demo trusted CA.>

Read that again if you didn’t catch it the first time through: “The system is vulnerable to security attacks, since it trusts certificates signed by the demo trusted CA.” This text comes from code in WebLogic, not from me. Here Oracle tells you very explicitly that your system currently contains a severe vulnerability.

You will also notice that when using the EM12c console, or accessing an agent URL, or accessing the WebLogic Server administration console may show warnings in your browser about untrusted certificates. Once you replace your certificates as described in the documentation above or my steps below, you will no longer have those issues.

Using 3rd Party SSL/TLS Certificates With EM12c

Overview

You will follow 7 high level steps to complete the process of securing your EM12c environment with custom third party SSL/TLS certificates.

  1. Create an Oracle wallet for the OMS.
  2. Secure the OMS console using the OMS wallet.
  3. Secure the OMS upload port using the OMS wallet.
  4. Re-secure all agents.
  5. Create Oracle wallets for agents.
  6. Configure the agents to use their wallets.
  7. Secure WebLogic with the OMS wallet.

Create an Oracle wallet for the OMS

First we follow steps 1a through 1h from document 1367988.1.  All these steps occur on the OMS host.

Disable shell history (optional but recommended)

While following these steps, you will repeatedly have to type passphrases on the command line. To avoid having these passphrases stored in your Oracle user’s shell history, disable history saving.  In the bash shell that I use, I accomplish this by unsetting the HISTFILE variable. You may need to use another mechanism in another shell.

$ unset HISTFILE

Use the correct ORAPKI command

You should use the ORAPKI command from your middleware home’s oracle_common/bin directory.  I will refer to this as $MW_HOME/oracle_common/bin/orapki in the following instructions.

Create an Oracle wallet

The documentation specified that we should create an auto-login wallet, but in my single-OMS setup, I believe that I will achieve better security with an auto-login-local wallet, as the auto-login feature will only function on this specific host. You will need to select a base directory for your OMS wallet.  I used $ORACLE_BASE/oemwallet. ORAPKI will prompt you for a password. Use a secure one, and note it down somewhere safe. You will use it many times during this process.

$ mkdir $ORACLE_BASE/oemwallet
$ $MW_HOME/oracle_common/bin/orapki wallet create -wallet $ORACLE_BASE/oemwallet -auto_login_local
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Enter password:
 Enter password again:

Get in the habit of displaying the wallet contents after each operation to confirm that everything worked.

$ $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/oemwallet
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Requested Certificates:
 User Certificates:
 Trusted Certificates:
 Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
 Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US

Create a key within the wallet.  Make sure you replace omshost.domain.com with the fully qualified domain name of your OMS host. I highly recommend using a 2048 bit keysize, as shown below. Include the wallet password you specified earlier on the commandline as the -pwd argument, contained in single quotes. Display the wallet again afterward.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -dn "CN=omshost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US" -keysize 2048 -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

$ $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/oemwallet
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Requested Certificates:
 Subject: CN=omshost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US
 User Certificates:
 Trusted Certificates:
 Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US

Export a certificate signing request based on this key. Make sure the -dn you specify exactly matches the -dn specified earlier. Provide a filename in the -request argument in which to store the certificate signing request (CSR).

$ $MW_HOME/oracle_common/bin/orapki wallet export -wallet $ORACLE_BASE/oemwallet -dn "CN=omshost.domain.com, OU=EM,O=Organization,L=City,ST=State,C=US" -request ~/EM12cCSR.txt
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

Submit this CSR file to your signing authority. Inform them that you MUST have a single-host certificate with your OMS host’s fully qualified domain name in the CN field. Subject Alternate Name (SAN) certificates or wildcard certificates will not work at all. Your signing authority should then provide you with a root certificate, an intermediate certificate, and a user certificate.

Import the root, intermediate, and user certificates into the OMS wallet. Note that you must import the root and intermediate certificates using -trusted_cert, and the user certificate using -user_cert.  I used DigiCert, and I can confirm that their certificates function correctly in EM12c and recommend their service.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -trusted_cert -cert ~/TrustedRoot.cer -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -trusted_cert -cert ~/DigiCertCA2.cer -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -user_cert -cert ~/omshost.domain.com.cer -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

Display the wallet contents after this operation.

$  $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/oemwallet
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Requested Certificates:
 User Certificates:
 Subject: CN=omshost.domain.com,OU=[REDACTED],O=[REDACTED],L=City,ST=State,C=US
 Trusted Certificates:
 Subject: CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US
 Subject: CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US
 Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
 Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US

Concatenate the root and intermediate certificates into a single file, which you will use later.

$  cat ~/DigiCertCA2.cer ~/TrustedRoot.cer > $ORACLE_BASE/trusted_certs.txt

You have completed configuration of your OMS wallet.

Secure the OMS console

Now, using emctl from the $OMS_HOME, tell EM12c to secure the OMS console using the certificate contained in your wallet. The system will prompt you for the SYSMAN password and inform you to restart the entire OMS once complete.

$ $OMS_HOME/bin/emctl secure console -wallet /oracle/oem/oemwallet
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Securing Console... Started.
 Enter Enterprise Manager Root (SYSMAN) Password :
 Securing Console... Successful
 Restart OMS
$ $OMS_HOME/bin/emctl stop oms -all ; sleep 5 ; $OMS_HOME/bin/emctl start oms

Now access your OMS console with your favorite browser and confirm that your new certificate appears.  Your certificate should show a trusted path back to a root certificate, and your browser should produce no warnings.

At this point, you have secured communication between your browser and the EM12c OMS console with your custom certificate.  You still have more work to do though. Your agents upload monitoring data to the OMS upload port, and it still uses the demo certificate. Fix that in the next step.

Secure the OMS upload port

Secure the OMS upload port. Expect to receive email or pager alerts after this step, as once you restart the OMS, none of your agents can communicate with it, as they expect to see the demo certificates on the upload port. You will need to provide the SYSMAN password as well as an agent registration password.

$ $OMS_HOME/bin/emctl secure oms -wallet $ORACLE_BASE/oemwallet -trust_certs_loc $ORACLE_BASE/trusted_certs.txt
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Securing OMS... Started.
 Enter Enterprise Manager Root (SYSMAN) Password :
 Enter Agent Registration Password :
 Securing OMS... Successful
 Restart OMS
$ $OMS_HOME/bin/emctl stop oms -all ; sleep 5 ; $OMS_HOME/bin/emctl start oms

Re-secure all agents

Now you must re-secure all of your agents so that they can resume uploading data to the OMS console and monitoring your systems. Execute the following steps on every agent, using emctl from the agent home.  You will need to provide an agent registration password to complete this process.

$ $AGENT_HOME/bin/emctl secure agent
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Agent successfully stopped... Done.
Securing agent... Started.
Enter Agent Registration Password : 
Agent successfully restarted... Done.
Securing agent... Successful.
$ $AGENT_HOME/bin/emctl upload agent
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

It may take a little while for the OMS to process the new agents and their uploads, but once you have run this process on every agent they should all communicate successfully with the OMS and appear as OK from the agent management screen.

Create Oracle wallets for agents

Next we secure the agent URLs. The OMS connects to the agents at this URL to submit management requests. At the moment, the agents still use self-signed certificates to secure this URL.  For this process we create an Oracle wallet, on the OMS host, using the same ORAPKI command as for the OMS wallet. We will generate a certificate signing request from each agent wallet, submit those CSRs to a certificate authority, and import the received certificates.

As with the OMS, the agents must use single-host certificates, not wildcard or subject alternate name (SAN) certificates.  To determine the correct fully qualified domain name for each agent, execute emctl status agent from the agent home.

$  $AGENT_HOME/bin/emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 12.1.0.4.0
OMS Version : 12.1.0.4.0
Protocol Version : 12.1.0.1.0
Agent Home : /oraagent/agent12c/agent_inst
Agent Log Directory : /oraagent/agent12c/agent_inst/sysman/log
Agent Binaries : /oraagent/agent12c/core/12.1.0.4.0
Agent Process ID : 12480
Parent Process ID : 12359
Agent URL : https://agenthost.domain.com:3872/emd/main/    <---- use this hostname
Local Agent URL in NAT : https://agenthost.domain.com:3872/emd/main/
Repository URL : https://omshost.domain.com:4902/empbs/upload
Started at : 2015-03-03 12:41:45
Started by user : oraagent
Operating System : Linux version 3.0.101-0.40-default (amd64)
Last Reload : (none)
Last successful upload : 2015-03-03 13:41:48
Last attempted upload : 2015-03-03 13:41:48
Total Megabytes of XML files uploaded so far : 0.4
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0
Available disk space on upload filesystem : 56.64%
Collection Status : Collections enabled
Heartbeat Status : Ok
Last attempted heartbeat to OMS : 2015-03-03 13:48:56
Last successful heartbeat to OMS : 2015-03-03 13:48:56
Next scheduled heartbeat to OMS : 2015-03-03 13:49:56

---------------------------------------------------------------
Agent is Running and Ready

Repeat these steps for every agent.

Create a directory to store the agent wallet, and an agent wallet. This time do NOT use -auto_login_local, use only -auto_login, as you will distribute these wallets to the agent hosts after generating them on the OMS host.  Use a strong password, and save it for later, as you will reuse it many times.

$ mkdir $ORACLE_BASE/agentwallets
$ mkdir $ORACLE_BASE/agentwallets/agenthost.domain.com
$ $MW_HOME/oracle_common/bin/orapki wallet create -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -auto_login
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Enter password:
 Enter password again:

Create the certificate, then a certificate signing request, saving it to file ~/agenthost.domain.com.csr. Again, I recommend a 2048 bit certificate.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -dn "CN=agenthost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US" -keysize 2048 -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
$ $MW_HOME/oracle_common/bin/orapki wallet export -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -dn "CN=agenthost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US" -request ~/agenthost.domain.com.csr
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

As before, submit this certificate signing request to your certificate authority, and receive back three files containing a root certificate, an intermediate certificate, and a user certificate. Import these into the agent wallet, and display the wallet afterwards to confirm everything imported successfully.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -trusted_cert -cert ~/TrustedRoot.crt -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -trusted_cert -cert ~/DigiCertCA.crt -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -user_cert -cert ~/agenthost.domain.com.crt -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
$ $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Requested Certificates:
 User Certificates:
 Subject: CN=agenthost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US
 Trusted Certificates:
 Subject: CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US
 Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
 Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=DigiCert High Assurance CA-3,OU=www.digicert.com,O=DigiCert Inc,C=US

You have finished creating this agent’s wallet.  Repeat this for every agent.

Configure the agents to use their wallets

Inside the agent wallets you’ve just created, you will find a cwallet.sso file. Take this file from each agent’s wallet and copy it to the agent server. Stop the agent, then place the file into $AGENT_INSTANCE_DIR/sysman/config/server/ and set the permissions to 640, then start the agent.

$ $AGENT_HOME/bin/emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Stopping agent ..... stopped.
$ cp cwallet.sso $AGENT_INSTANCE_DIR/sysman/config/server
$ chmod 640 $AGENT_INSTANCE_DIR/sysman/config/server
$ $AGENT_HOME/bin/emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Starting agent ............. started.

Next, visit the agent URL in your favorite web browser and examine the certificate it uses.  You should now see that it uses the 3rd party SSL/TLS certificate that you installed.

Secure WebLogic with the OMS wallet

Now the OMS (both console and upload ports) and agents will use your new certificates. This leaves WebLogic as the one remaining component needing your new certificates. Please note in following the below directions that securing WebLogic with a wallet only works as of EM12c R3, earlier versions must use a Java keystore. See note 1527874.1 for more information.

[NOTE: 20150910: If you secure WebLogic with a certificate that uses the SHA256 hashing algorithm, future attempts to apply EM12c PSU patches using ‘opatchauto’ will fail. Some piece of opatchauto does not support SHA256 usage in certificates. If you run into this issue, revert your WLS to the demonstration certificate using emctl secure wls -use_demo_cert, then apply the PSU, then resecure WLS using these steps with your desired certificate. I intend to write a full blog post about this later.]

First import the root and intermediate certificates to the keystore on the OMS host’s agent. Use the default password welcome for the agent keystore, and alias names rootcacert and intercacert.

$ $AGENT_HOME/bin/emctl secure add_trust_cert_to_jks -trust_certs_loc ~/TrustedRoot. crt -alias rootcacert -password welcome
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.

 Message : Certificate was added to keystore
 ExitStatus: SUCCESS
$ $AGENT_HOME/bin/emctl secure add_trust_cert_to_jks -trust_certs_loc ~/DigiCertCA. crt -alias intercacert -password welcome
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.

 Message : Certificate was added to keystore
 ExitStatus: SUCCESS

Back up some WLS configuration files, just in case, before securing WLS with your certificate.  If you have problems in this step, make sure you have stopped all WLS processes, then restore these files from backup.

$ mkdir ~/wlscertbak
$ cp -a $EM_INSTANCE_BASE/em/EMGC_OMS1/emgc.properties ~/wlscertbak/
$ cp -a $EM_INSTANCE_BASE/NodeManager/emnodemanager/nodemanager.properties ~/wlscertbak/
$ cp -a $EM_INSTANCE_BASE/WebTierIH1/config/OHS/ohs1/keystores/proxy ~/wlscertbak/
$ cp -a $EM_INSTANCE_BASE/user_projects/domains/GCDomain/config/config.xml ~/wlscertbak/

Stop the OMS.

$ $OMS_HOME/bin/emctl stop oms
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Stopping WebTier...
 WebTier Successfully Stopped
 Stopping Oracle Management Server...
 Oracle Management Server Successfully Stopped
 Oracle Management Server is Down

Secure WLS using the OMS wallet created earlier. You will need to provide the SYSMAN password.

$ $OMS_HOME/bin/emctl secure wls -wallet $ORACLE_BASE/oemwallet
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Securing WLS... Started.
 Enter Enterprise Manager Root (SYSMAN) Password :
 Securing WLS... Successful
 Restart OMS using 'emctl stop oms -all' and 'emctl start oms'
 If there are multiple OMSs in this environment, perform this configuration on all of them.

Stop the entire WLS stack, then start the OMS.

$ $OMS_HOME/bin/emctl stop oms -all
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Stopping WebTier...
 WebTier Successfully Stopped
 Stopping Oracle Management Server...
 Oracle Management Server Already Stopped
 Oracle Management Server is Down
 Stopping BI Publisher Server...
 BI Publisher Server Successfully Stopped
 AdminServer Successfully Stopped
 BI Publisher Server is Down
$ $OMS_HOME/bin/emctl start oms
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Starting Oracle Management Server...
 Starting WebTier...
 WebTier Successfully Started
 Oracle Management Server Successfully Started
 Oracle Management Server is Up
 Starting BI Publisher Server ...
 BI Publisher Server Successfully Started
 BI Publisher Server is Up

Visit the WebLogic Server administration console and you should now see that it presents your custom SSL/TLS certificate and no longer uses the demo certificate.

Conclusion

If you have successfully followed these steps, your system should now use your custom SSL/TLS certificates everywhere, and you should no longer experience certificate warnings in your browsers.

See Also

When proactive EM12c JDK upgrades bite back

You probably will not encounter this issue, but I will post this anyway to get the error message and resolution indexed by Google.

While attempting to apply patch 19513382 (EM agent bundle patch 12.1.0.4.3) to my EM12cR4 agents, I ran into multiple problems.  Initially it would not apply to any of my agents.  Bug 20134182 and the resolution described in MOS note 1952355.1 resolved the first problem (OPatch reporting that identical patches 18721761 and 18502187 already exist), but that left me with one agent I could not upgrade. Attempts to run patch plan validation within EM12c produced the following error:

PatchList : 19513382
PatchLocList : /tmp/p19513382_600000000009641_2000_0/oraagent
TargetName : [redacted]:[port]
----------------------------------------
[11_12_2014_10_00_40] Command Arguments:
/oraagent/agent12c/core/12.1.0.4.0/OPatch/opatch checkComponents -phbasedir /tmp/p19513382_600000000009641_2000_0/oraagent/19513382 -oh /oraagent/agent12c/core/12.1.0.4.0 -invPtrloc /oraagent/agent12c/core/12.1.0.4.0/oraInst.loc
 
OPatch cannot continue because it would not be able to load OUI platform dependent library from the directory "/oraagent/agent12c/core/12.1.0.4.0/oui/lib/linux64". The directory does not exist in the Oracle home.
This could be due to the following reasons.

(1) Incompatible usage of java with OUI (32/64 bit).
(2) Wrong 32-bit Oracle Home installation in 64-bit environment (or) vice-versa.
Please contact Oracle support for more details.
 
OPatch failed with error code 1
 
PREREQ_CONTEXT_HOST_NAME:[redacted]
REREQ_CONTEXT_HOME_LOCATION:/oraagent/agent12c/core/12.1.0.4.0
PREREQ_NAME: Checking if the patches are applicable.
PREREQ_DESC: Checking if the patches are applicable on the Management Agent.
PREREQ_TYPE:APPLICABILITY
PREREQ_STATUS:FAILED
PREREQ_MESG: None of these patches are applicable on the Management Agent.
PREREQ_MESG_PATCH:19513382
PREREQ_REMEDY:MANUAL
PREREQ_REMEDY_DETAILS: Remove patch(es) 19513382 from this patch plan.

I already know from the previously referenced MOS note that OPatch 11.1.0.12.3 contains bugs, so as a first debugging step I attempted to rollback the OPatch upgrade by restoring the backup copies of OPatch found in $AGENT_HOME/OPatch/backup/.  I received the same error message with OPatch 11.1.0.10.4 and 11.1.0.11.0.  I also received a similar error even if I simply tried to run “opatch lsinv” from the command line with the older versions. So OPatch did not cause this issue.

Since the error message mentions 32-bit and 64-bit incompatibility, I needed to consider the environment.  This server runs Linux x86-64 (SLES 10 SP4), but must use a 32-bit EM agent based on the certification matrix and MOS note 1488161.1. I next checked to find my last successful patch run, which happened only a month ago, so a recent change has to have caused this problem. Going through my notes, the only recent change on this server involved upgrading the JDK used by the EM agent per MOS note 1944044.1.

Luckily I still had the old JDK available for comparison.

> java -version
java version "1.6.0_43"
Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
Java HotSpot(TM) Server VM (build 20.14-b01, mixed mode)
> file `which java`
/oraagent/agent12c/core/12.1.0.4.0/jdk/bin/java: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), not stripped

Looking at the new JDK:

> ./java -version
java version "1.6.0_85"
Java(TM) SE Runtime Environment (build 1.6.0_85-b13)
Java HotSpot(TM) 64-Bit Server VM (build 20.85-b01, mixed mode)
> file java
java: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), for GNU/Linux 2.4.0, dynamically linked (uses shared libs), not stripped

There I have my problem. In upgrading the JDK, I had installed the 64-bit version of Java 1.6u85, not the 32-bit version, based on the fact that the server runs a 64-bit OS. I had not considered that a 64-bit JDK would not remain compatible with the 32-bit agent on this 64-bit system.

Surprisingly, everything about the agent seems to have worked fine, despite the 64-bit JDK.  Nothing broke until I attempted to use OPatch.

To resolve the issue, I stopped the agent and moved the original 32 bit 1.6u43 JDK back to where it belongs, followed note 1952355.1 to work around the known bugs when using OPatch 11.1.0.12.3 to apply 19513382, then successfully applied the patch.  After that I downloaded the correct 32-bit version of the 1.6u85 JDK, installed it per 1944044.1, and now OPatch works as expected.

How to unofficially disable SSL v3 in Oracle Enterprise Manager 12c to mitigate POODLE attack

With the recent POODLE vulnerability, server operators must now (finally) disable SSL version 3.0 and move up to TLS 1.0 at the minimum, if not TLS 1.2.

UPDATE: Many thanks to Courtney Llamas who provided me with a link to the section of the documentation that describes the right way to do this.  If you want to disable SSLv3 in EM12c, follow the instructions in section 2.3.2.4 of chapter 2 of the Oracle Enterprise Manager Cloud Control Security Guide.  You will need to re-secure your OMS during the process and this will require that you have access to the SYSMAN password and an agent registration password. I can confirm that the steps in this document work and do disable SSLv3. Make sure you follow the steps in the document to secure the management agents, too.

[EDIT: 20150312: Please note that you MUST install the 12.1.0.4.6 Agent bundle patch 20423395 to allow the agent-side “allowTLSOnly” property to function correctly. If you do not install this patch on your 12.1.0.4 agent, the agent will continue to permit SSLv3 connections.]

Continue reading