[20170822 NOTE: Oracle released the last set of bundle patches for the Oracle Enterprise Manager 13c version in April of 2017. See MOS note 2124038.1 for more details. You really should upgrade to EM13cR2 if you can. My security checkup script for EM13cR2 contains much added functionality and continues to receive updates. I do not expect to release any further updates to this script unless Oracle releases any further key patches or someone who uses it reports a fixable bug.]
[20170418 NOTE: I have upgraded the patches referenced in this script to reflect the latest (20170418) PSU patch for EM13cR1. I no longer have an EM13cR1 environment available with which to test this script, so please feel free to report issues or to submit a git pull request. I have now placed this script on github: https://raw.githubusercontent.com/brianpardy/em13c/master/checksec13.sh.]
[20161013 NOTE: I have upgraded to EM13cR2, and this script still works as expected. If you attempt to run it on an EM13cR2 environment please take note that all of the patch recommendations listed apply to the older EM13cR1 release and will provide incorrect results on 13.2. The TLS, certificate, and cipher strength tests all function correctly on 13.2.]
Download final version
You can download the final release of my EM13cR1 security checkup script at GitHub.
Introduction
This post continues my series on securing Oracle Enterprise Manager environments with some updates relevant to EM13c. Oracle has made significant security improvements with Oracle Enterprise Manager 13c over the prior 12c version, first released in October 2011, more than four and a half years ago at this point. In the interest of security, I have to strongly recommend that any sites still using EM12c upgrade to (or perform a fresh installation of) EM13c EM13cR2 as soon as possible. More recent versions of EM12c like 12.1.0.5 (June 2015) continue to use the same technology stack as the initial release, and the world of security has massively changed since then. Notably, EM13c uses Java 7, WebLogic 12.1.3, and disables SSLv3 out of the box.
Just to recap, back at the EM12c original release date:
- Practically nobody had ever heard of Edward Snowden
- The first release of Java 7 celebrated its three month birthday
- Two months later, Oracle released WebLogic 12c; EM12c users remained on WebLogic 10.3.6
- One month earlier, the public learned of the BEAST attack and people still believed that using RC4 (immune to BEAST) as a workaround improved security (spoiler warning: it did not)
- We had three years to wait before the POODLE vulnerability caused vendors to recognize the need to disable SSLv3 (you DID disable SSLv3, right?)
- Oracle still considered the MD5 hashing algorithm good enough to use in self-signed certificates produced by EM12c, despite flaws known to exist since 1996
- Web browsers considered the SHA-1 hashing algorithm, now also deprecated due to brokenness, good enough to use
As the security world’s known unknowns collapsed around us, proactive EM12c administrators sought to make the best of their lot. Outside of Oracle, I and others poked at the software and wrote blog articles, while inside Oracle effort proceeded to support more recent Java releases that brought with them better cipher suites and hashing algorithms, as well as the usual security fixes. This process took some time for all involved and hit a few bumps along the way.
- Replacement of EM12c certificates with custom certificates making use of the SHA-2 hashing algorithm appeared to work fine until we tried to apply patches: EM12c opatchauto, SHA256, and you
- Each time a new attack broke in the press, proactive EM12c administrators including myself looked for ways to mitigate it, as with my first fumbling efforts when the POODLE bug became public: How to unofficially disable SSL v3 in Oracle Enterprise Manager 12c to mitigate POODLE attack
- The difficulty of keeping track of now-monthly patching and rechecking known issues after patching led me to create an automated security check script for EM12c: EM12c R4 SSL Security Checkup Script
- The official documentation, though exhaustive, sometimes complicated security issues beyond the minimum necessary to achieve a task, so I produced a step-by-step guide to replacing as many certificates as possible with those sourced from third parties or internal sources: Step by step: Configuring third party SSL/TLS certificates in EM12c R4
- Sometimes bizarre bugs led to an inability to disable known weak cipher suites, requiring deep dives into patching and configuration: EM12c OHS, LOW strength ciphers, custom certificates, and patch 1994800 weirdness
- Sometimes we shoot ourselves in the foot by carelessly upgrading components: When proactive EM12c JDK upgrades bite back
- Applying your strong certificates to the full EM12c stack does not eliminate the severe security issues created by Oracle’s decision to include a widely known demo certificate authority with WebLogic: Insecure certificate warning still exists in EM12c WLS logs with third-party certificates applied
I do not intend in this post to review general day-to-day EM13c security design such as user roles or privileges, object level security within OEM, or integration with identity providers like LDAP; only the infrastructure level issues that tend to change in brief large bursts as new attacks come out. See this excellent list of EM13c blogs, links and videos that Philip Brown has provided for more details on these and other items.
On to EM13c
EM13c admins need to keep an eye on the same sorts of items as with EM12c. We really should read the documentation, even if only the Security Guide. I admit I often do not. It contains good information.
Patches
I consider it critical for admins to keep up with the OEM periodic patches, particularly security patches. The script below covers patches up to and including March 31, 2016. I plan to update again after the next set of Oracle security patches arrives, likely mid-April.
Certificates
The process for applying certificates on EM13c does not appear to have changed significantly from the prior version. I have confirmed that the new “omspatcher” tool that replaces opatchauto when applying a system patch to the OMS works perfectly fine with certificates on WebLogic that use the SHA-256 hashing algorithm. Given the upcoming deprecation of SHA-1 across all major browsers I do not see any valid reason not to use SHA-256 with new certificates.
Ciphersuites
Out of the box, my EM13c installation rejected weak ciphersuites and accepted the strong ones. Unfortunately it still accepted some that these versions of Java and OpenSSL consider as MEDIUM strength, so I want to disable those across the entire environment, leaving only the strongest ciphersuites available in this release and permitting other ciphersuites only where necessary.
[UPDATE 20160518: Please see MOS note 2138391.1 for the official procedure to disable weak cipher suites in EM13c.]
We will have to live with these unwanted ciphersuites enabled until Oracle provides a supported procedure to disable them across the entire stack. I have performed some preliminary tests and I find it very easy to get OEM into a situation where it cannot startup after manually editing config files that define enabled ciphersuites. The script below will identify ports permitting ciphersuites you may wish to disable when a supported method becomes available.
UPDATE 20160720: Take particular care of watching the ciphersuites accepted by your agents if you upgrade the JDK that the agents use. I just attempted a JDK update on an agent from the distributed version to 1.7.0-111, and that agent began to accept LOW and MEDIUM strength ciphersuites again, thus I have omitted JDK updates for agents from the check script.
Security Checkup
Below I provide an early version of the script I use to validate EM13c security configuration. I based this on my earlier EM12c script, linked above. The script will become more useful once I implement patch level checking after release of the first set of EM13c patches, but for the moment it will inspect your EM13c environment to identify relevant ports and confirm that your system will not respond to SSLv2 or SSLv3 requests, does respond to TLSv1 requests, supports HIGH, but not LOW or MEDIUM strength ciphersuites (as defined by the version of OpenSSL installed on your OMS host), and finally checks for the presence of demonstration-not-for-production-use certificates and self-signed certificates.
(A caveat on self-signed certificate checking: OpenSSL, not this script, performs the check, therefore if OpenSSL cannot validate your certificate to a trusted root, this script cannot either. If a well known certification authority has signed your certificates, OpenSSL should validate them successfully, but it may not do so if you use an internal certificate authority to sign certificates. In that case you should install a copy of your internal CA as a trusted root certificate in the system trust store so that OpenSSL can validate your EM13c certificates. I cannot document this process for every OS but Linux users should look to the documentation for the update-ca-certificates or update-ca-trust commands. If my script below incorrectly reports your certificate as self-signed, you can ignore the finding or address the issue at the OpenSSL level.)
EM13c TLS Security Checkup Script
[LATEST UPDATE: 20161004, adds 20160920 patches and fixes TLSv1 vs TLSv1.2 bugs, version 0.9]. Thank you to Bob Schuppin who reported a bug in the use of TLSv1 to check certificate and cipher suite usage in a TLSv1.2-only site. I have modified the relevant checks to use TLSv1.2 if supported by your OpenSSL version or to stick with TLSv1 if not supported.
[PRIOR UPDATE: 20160914 bugfix and enhancements, no new patch checks, version 0.8]. Thank you to Paige who reported a bug in the check of the SSL_CIPHER_SUITES parameter. I had a typo in the cipher suite names for the SSL_CIPHER_SUITES parameter, which I have now fixed. In researching this I realized that this parameter provides control over SSL/TLS authentication for clients, which I do not use in my environment. Instead I use native SQL*Net encryption, controlled by the various ENCRYPTION_(CLIENT|SERVER), ENCRYPTION_TYPES_(CLIENT|SERVER), CRYPTO_CHECKSUM_(CLIENT|SERVER), and CRYPTO_CHECKSUM_TYPES_(CLIENT|SERVER) parameters, which I have added into the script. The script will check to make sure that you do not permit MD5 as a SQL*Net checksum algorithm and that you do not permit DES, DES40, 3DES112, nor any of the RC4_ algorithms for SQL*Net encryption. Unfortunately due to bug 23587582, you will encounter problems promoting targets in OEM unless you allow use of the 3DES168 encryption algorithm and SHA1 hashing algorithm. Generally I would prefer to disable both of those for security reasons but I will permit them in this script as long as they remain necessary for full OEM functionality.
[PRIOR UPDATE: 20160819 for 20160816 bundle patches, version 0.7]. With this update, I have added support for TLSv1.1 and TLSv1.2 to the protocol checks. I have also added support for the optional SLES11 openssl1 package which provides a newer OpenSSL supporting TLSv1.1 and TLSv1.2 for systems on SLES11 like mine. The script will now dynamically determine (by parsing the “openssl s_client help” output) if your OpenSSL version supports TLSv1.2. If your OpenSSL version DOES support TLSv1.2, the script will now flag any support of TLSv1 or TLSv1.1 as a failure in your OEM stack. If your OpenSSL does NOT support TLSv1.2, the script will consider TLSv1 support in OEM as acceptable. If you notice problems with this new functionality please let me know.
Compatibility
Only tested on Linux x86-64, but may work on AIX and Solaris as the EM12c version I built this upon did work there. Planned future enhancements include checking that you have disabled non-encrypted HTTP access to EM13c components, upgraded Java to the most recent EM13c-supported release, and more.
You can download the latest version of the script from github: https://raw.githubusercontent.com/brianpardy/em13c/master/checksec13.sh.
EM13c TLS Security Checkup Script Sample Output
Performing EM13c security checkup version 0.9 on omshost.domain.com at Tue Oct 4 11:04:43 EDT 2016.
Using port definitions from configuration files
/etc/oragchomelist
/oracle/oem/gc_inst/em/EMGC_OMS1/emgc.properties
/oracle/oem/gc_inst/em/EMGC_OMS1/embip.properties
Agent port found at omshost.domain.com:3872
BIPublisher port found at omshost.domain.com:9803
BIPublisherOHS port found at omshost.domain.com:9851
NodeManager port found at omshost.domain.com:7403
OMSconsole port found at omshost.domain.com:7802
OMSproxy port found at omshost.domain.com:7301
OMSupload port found at omshost.domain.com:4903
WLSadmin found at omshost.domain.com:7102
Repository DB version=12.1.0.2.0 SID=oemdb host=omshost.domain.com
Using OPENSSL=/usr/bin/openssl1 (has TLS1_2=2)
Repository DB on OMS server, will check patches/parameters in /oracle/oem/product/12.1.0/db
(1) Checking SSL/TLS configuration (see notes 1602983.1, 1477287.1, 1905314.1)
(1a) Forbid SSLv2 connections
Confirming ssl2 disabled for Agent at omshost.domain.com:3872... OK
Confirming ssl2 disabled for BIPublisher at omshost.domain.com:9803... OK
Confirming ssl2 disabled for NodeManager at omshost.domain.com:7403... OK
Confirming ssl2 disabled for BIPublisherOHS at omshost.domain.com:9851... OK
Confirming ssl2 disabled for OMSconsole at omshost.domain.com:7802... OK
Confirming ssl2 disabled for OMSproxy at omshost.domain.com:7301... OK
Confirming ssl2 disabled for OMSupload at omshost.domain.com:4903... OK
Confirming ssl2 disabled for WLSadmin at omshost.domain.com:7102... OK
(1b) Forbid SSLv3 connections
Confirming ssl3 disabled for Agent at omshost.domain.com:3872... OK
Confirming ssl3 disabled for BIPublisher at omshost.domain.com:9803... OK
Confirming ssl3 disabled for NodeManager at omshost.domain.com:7403... OK
Confirming ssl3 disabled for BIPublisherOHS at omshost.domain.com:9851... OK
Confirming ssl3 disabled for OMSconsole at omshost.domain.com:7802... OK
Confirming ssl3 disabled for OMSproxy at omshost.domain.com:7301... OK
Confirming ssl3 disabled for OMSupload at omshost.domain.com:4903... OK
Confirming ssl3 disabled for WLSadmin at omshost.domain.com:7102... OK
(1c) Forbid TLSv1 connections
Confirming tls1 disabled for Agent at omshost.domain.com:3872... FAILED
Confirming tls1 disabled for BIPublisher at omshost.domain.com:9803... FAILED
Confirming tls1 disabled for NodeManager at omshost.domain.com:7403... FAILED
Confirming tls1 disabled for BIPublisherOHS at omshost.domain.com:9851... FAILED
Confirming tls1 disabled for OMSconsole at omshost.domain.com:7802... FAILED
Confirming tls1 disabled for OMSproxy at omshost.domain.com:7301... FAILED
Confirming tls1 disabled for OMSupload at omshost.domain.com:4903... FAILED
Confirming tls1 disabled for WLSadmin at omshost.domain.com:7102... FAILED
(1c) Forbid TLSv1.1 connections
Confirming tls1_1 disabled for Agent at omshost.domain.com:3872... FAILED
Confirming tls1_1 disabled for BIPublisher at omshost.domain.com:9803... FAILED
Confirming tls1_1 disabled for NodeManager at omshost.domain.com:7403... FAILED
Confirming tls1_1 disabled for BIPublisherOHS at omshost.domain.com:9851... FAILED
Confirming tls1_1 disabled for OMSconsole at omshost.domain.com:7802... FAILED
Confirming tls1_1 disabled for OMSproxy at omshost.domain.com:7301... FAILED
Confirming tls1_1 disabled for OMSupload at omshost.domain.com:4903... FAILED
Confirming tls1_1 disabled for WLSadmin at omshost.domain.com:7102... FAILED
(1c) Permit TLSv1.2 connections
Confirming tls1_2 available for Agent at omshost.domain.com:3872... OK
Confirming tls1_2 available for BIPublisher at omshost.domain.com:9803... OK
Confirming tls1_2 available for NodeManager at omshost.domain.com:7403... OK
Confirming tls1_2 available for BIPublisherOHS at omshost.domain.com:9851... OK
Confirming tls1_2 available for OMSconsole at omshost.domain.com:7802... OK
Confirming tls1_2 available for OMSproxy at omshost.domain.com:7301... OK
Confirming tls1_2 available for OMSupload at omshost.domain.com:4903... OK
Confirming tls1_2 available for WLSadmin at omshost.domain.com:7102... OK
(2) Checking supported ciphers at SSL/TLS endpoints (see notes 2138391.1, 1067411.1)
Checking LOW strength ciphers on Agent (omshost.domain.com:3872, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on Agent (omshost.domain.com:3872)... OK
Checking HIGH strength ciphers on Agent (omshost.domain.com:3872)... OK
Checking LOW strength ciphers on BIPublisher (omshost.domain.com:9803, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on BIPublisher (omshost.domain.com:9803)... OK
Checking HIGH strength ciphers on BIPublisher (omshost.domain.com:9803)... OK
Checking LOW strength ciphers on NodeManager (omshost.domain.com:7403, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on NodeManager (omshost.domain.com:7403)... OK
Checking HIGH strength ciphers on NodeManager (omshost.domain.com:7403)... OK
Checking LOW strength ciphers on BIPublisherOHS (omshost.domain.com:9851, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on BIPublisherOHS (omshost.domain.com:9851)... OK
Checking HIGH strength ciphers on BIPublisherOHS (omshost.domain.com:9851)... OK
Checking LOW strength ciphers on OMSconsole (omshost.domain.com:7802, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on OMSconsole (omshost.domain.com:7802)... OK
Checking HIGH strength ciphers on OMSconsole (omshost.domain.com:7802)... OK
Checking LOW strength ciphers on OMSproxy (omshost.domain.com:7301, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on OMSproxy (omshost.domain.com:7301)... OK
Checking HIGH strength ciphers on OMSproxy (omshost.domain.com:7301)... OK
Checking LOW strength ciphers on OMSupload (omshost.domain.com:4903, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on OMSupload (omshost.domain.com:4903)... OK
Checking HIGH strength ciphers on OMSupload (omshost.domain.com:4903)... OK
Checking LOW strength ciphers on WLSadmin (omshost.domain.com:7102, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on WLSadmin (omshost.domain.com:7102)... OK
Checking HIGH strength ciphers on WLSadmin (omshost.domain.com:7102)... OK
(3) Checking self-signed and demonstration certificates at SSL/TLS endpoints (see notes 1367988.1, 1399293.1, 1593183.1, 1527874.1, 123033.1, 1937457.1)
Checking certificate at Agent (omshost.domain.com:3872, protocol tls1_2)... FAILED - Found self-signed certificate
Checking demo certificate at Agent (omshost.domain.com:3872, protocol tls1_2)... OK
Checking certificate at BIPublisher (omshost.domain.com:9803, protocol tls1_2)... OK
Checking demo certificate at BIPublisher (omshost.domain.com:9803, protocol tls1_2)... OK
Checking certificate at NodeManager (omshost.domain.com:7403, protocol tls1_2)... OK
Checking demo certificate at NodeManager (omshost.domain.com:7403, protocol tls1_2)... OK
Checking certificate at BIPublisherOHS (omshost.domain.com:9851, protocol tls1_2)... OK
Checking demo certificate at BIPublisherOHS (omshost.domain.com:9851, protocol tls1_2)... OK
Checking certificate at OMSconsole (omshost.domain.com:7802, protocol tls1_2)... OK
Checking demo certificate at OMSconsole (omshost.domain.com:7802, protocol tls1_2)... OK
Checking certificate at OMSproxy (omshost.domain.com:7301, protocol tls1_2)... OK
Checking demo certificate at OMSproxy (omshost.domain.com:7301, protocol tls1_2)... OK
Checking certificate at OMSupload (omshost.domain.com:4903, protocol tls1_2)... OK
Checking demo certificate at OMSupload (omshost.domain.com:4903, protocol tls1_2)... OK
Checking certificate at WLSadmin (omshost.domain.com:7102, protocol tls1_2)... OK
Checking demo certificate at WLSadmin (omshost.domain.com:7102, protocol tls1_2)... OK
(4) Checking EM13c Oracle home patch levels against 20 Sep 2016 baseline (see notes 1664074.1, 1900943.1, 822485.1, 1470197.1, 1967243.1)
(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) PSU 12.1.0.2.160719 (JUL2016) (23054246)... OK
Patch 23054246 : applied on Wed Jul 20 12:01:53 EDT 2016 Patch description: "Database Patch Set Update : 12.1.0.2.160719 (23054246)"
(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) ORACLE JAVAVM COMPONENT 12.1.0.2.160719 DATABASE PSU (JUL2016) (23177536)... OK
Patch 23177536 : applied on Wed Jul 20 12:03:14 EDT 2016 21566993, 22670413, 19699946, 23177536, 22118835, 22118851, 19895326
(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_TYPES_SERVER parameter (76629.1, 2167682.1)... OK
(AES128,AES256,AES192,3DES168)
(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_SERVER parameter (76629.1, 2167682.1)... OK
requested
(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_TYPES_CLIENT parameter (76629.1, 2167682.1)... OK
(AES128,AES256,AES192,3DES168)
(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_CLIENT parameter (76629.1, 2167682.1)... OK
requested
(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_TYPES_SERVER parameter (76629.1, 2167682.1)... OK
(SHA1)
(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_SERVER parameter (76629.1, 2167682.1)... OK
requested
(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_TYPES_CLIENT parameter (76629.1, 2167682.1)... OK
(SHA1)
(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_CLIENT parameter (76629.1, 2167682.1)... OK
requested
(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SSL_VERSION parameter (1545816.1)... OK
1.0
(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SSL_CIPHER_SUITES parameter (1545816.1)... OK
(SSL_RSA_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_AES_256_CBC_SHA)
(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) listener.ora SSL_VERSION parameter (1545816.1)... OK
1.0
(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) listener.ora SSL_CIPHER_SUITES parameter (1545816.1)... OK
(SSL_RSA_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_AES_256_CBC_SHA)
(4c) *UPDATED* OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM-AGENT BUNDLE PATCH 13.1.0.0.160920 (24437699)... OK
Patch 24437699 : applied on Tue Sep 27 12:08:23 EDT 2016 24437699, 21779343, 22616051, 23759799, 22988508, 23089106, 23581450
(4c) *UPDATED* OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM DB PLUGIN BUNDLE PATCH 13.1.1.0.160920 MONITORING (24545984)... OK
Patch 24545984 : applied on Tue Sep 27 13:46:08 EDT 2016 22908077, 23294830, 22503390, 23075475, 23697777, 24545984, 24296310
(4c) *UPDATED* OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM DB PLUGIN BUNDLE PATCH 13.1.1.0.160920 DISCOVERY (24545989)... OK
Patch 24545989 : applied on Tue Sep 27 13:46:11 EDT 2016 23523964, 23294839, 24545989, 23226583, 24408840
(4c) *UPDATED* OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM FMW PLUGIN BUNDLE PATCH 13.1.1.0.160920 MONITORING (24658006)... OK
Patch 24658006 : applied on Tue Sep 27 13:46:13 EDT 2016 22834135, 23007497, 22447329, 22936491, 24658006, 23294872, 23306887
(4c) OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM SI PLUGIN BUNDLE PATCH 13.1.1.0.160719 MONITORING (23697783)... OK
Patch 23697783 : applied on Wed Jul 20 10:53:57 EDT 2016 22128210, 23338028, 23189991, 22823189, 21253819, 23697783, 23208587
(4c) OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM SI PLUGIN BUNDLE PATCH 13.1.1.0.160531 DISCOVERY (23294895)... OK
Patch 23294895 : applied on Thu Jun 16 11:28:18 EDT 2016 23197299, 23294895
(4c) OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM OH PLUGIN BUNDLE PATCH 13.1.1.0.160429 (23135564)... OK
Patch 23135564 : applied on Wed May 11 13:21:35 EDT 2016 22521822, 23135564
(4d) *UPDATED* OMS HOME (/oracle/oem/Middleware13cR1) ENTERPRISE MANAGER FOR OMS PLUGINS 13.1.1.0.160920 (24546113)... OK
oracle.sysman.emas.oms.plugin/13.1.1.0.0 Plugin 24546113 24437669 oracle.sysman.cfw.oms.plugin/13.1.1.0.0 Plugin 24546113 24437711 oracle.sysman.db.oms.plugin/13.1.1.0.0 Plugin 24546113 24437646 oracle.sysman.xa.oms.plugin/13.1.1.0.0 Plugin 24546113 24437656
(4d) (/oracle/oem/Middleware13cR1) WLS PATCH SET UPDATE 12.1.3.0.160719 (23094292)... OK
Patch 23094292 : applied on Wed Jul 20 12:27:53 EDT 2016
(4f) OMS HOME (/oracle/oem/Middleware13cR1) ENTERPRISE MANAGER BASE PLATFORM PATCH SET UPDATE 13.1.0.0.160719 (23134365)... OK
oracle.sysman.top.oms/13.1.0.0.0 Core 23134365 23134365
(5) Checking EM13c Java patch levels against 20 Sep 2016 baseline (see notes 1492980.1, 1616397.1)
(5a) WLS (/oracle/oem/Middleware13cR1/oracle_common/jdk) JAVA SE JDK VERSION 1.7.0-111 (13079846)... OK
1.7.0_111
Failed test count: 17 - Review output
sslcheck:Agent @ omshost.domain.com:3872:tls1 protocol connection allowed
sslcheck:BIPublisher @ omshost.domain.com:9803:tls1 protocol connection allowed
sslcheck:NodeManager @ omshost.domain.com:7403:tls1 protocol connection allowed
sslcheck:BIPublisherOHS @ omshost.domain.com:9851:tls1 protocol connection allowed
sslcheck:OMSconsole @ omshost.domain.com:7802:tls1 protocol connection allowed
sslcheck:OMSproxy @ omshost.domain.com:7301:tls1 protocol connection allowed
sslcheck:OMSupload @ omshost.domain.com:4903:tls1 protocol connection allowed
sslcheck:WLSadmin @ omshost.domain.com:7102:tls1 protocol connection allowed
sslcheck:Agent @ omshost.domain.com:3872:tls1_1 protocol connection allowed
sslcheck:BIPublisher @ omshost.domain.com:9803:tls1_1 protocol connection allowed
sslcheck:NodeManager @ omshost.domain.com:7403:tls1_1 protocol connection allowed
sslcheck:BIPublisherOHS @ omshost.domain.com:9851:tls1_1 protocol connection allowed
sslcheck:OMSconsole @ omshost.domain.com:7802:tls1_1 protocol connection allowed
sslcheck:OMSproxy @ omshost.domain.com:7301:tls1_1 protocol connection allowed
sslcheck:OMSupload @ omshost.domain.com:4903:tls1_1 protocol connection allowed
sslcheck:WLSadmin @ omshost.domain.com:7102:tls1_1 protocol connection allowed
certcheck:Agent @ omshost.domain.com:3872 found self-signed certificate
Visit https://pardydba.wordpress.com/2016/04/05/securing-oracle-enterprise-manager-13c/ for the latest version.
Pardy DBA 
Please advise or clarify…. what is the name of the Cipher Suite that should be checked for in the SQLNET and LISTENER.ORA files? You have it listed as SSL_RSA_WITH_AES128_CBC_SHA, but the note you referenced has it listed as SSL_RSA_WITH_AES_128_CBC_SHA
Hello Paige,
Thank you for catching this! The note is correct: SSL_RSA_WITH_AES_128_CBC_SHA is the correct cipher suite name (along with SSL_RSA_WITH_AES_256_CBC_SHA), and not the incorrect spelling this script currently checks in the SSL_CIPHER_SUITES parameter. I will update the script as soon as I have a chance and credit you with the catch.
I appreciate your pointing this out. Looking at this more carefully, I realize now that SSL_CIPHER_SUITES serves to select cipher suites available for SSL client authentication, which I do not use in my environment. I do use SQL*Net encryption without SSL client authentication, which uses the SQLNET.ENCRYPTION_TYPES_(SERVER|CLIENT), SQLNET.CRYPTO_CHECKSUM_TYPES_(SERVER|CLIENT), SQLNET.ENCRYPTION_(SERVER|CLIENT), and SQLNET.CRYPTO_CHECKSUM_(SERVER|CLIENT) parameters in sqlnet.ora, and these are the ones I should be validating in the script, not SSL_CIPHER_SUITES.
Here are the parameters I have set on my OEM 13c repository database sqlnet.ora to support SQL*Net encryption:
SQLNET.ENCRYPTION_TYPES_SERVER=(AES128,AES256,AES192,3DES168)
SQLNET.ENCRYPTION_SERVER=requested
SQLNET.CRYPTO_CHECKSUM_SERVER=requested
SQLNET.CRYPTO_CHECKSUM_TYPES_SERVER=(SHA1)
SQLNET.ENCRYPTION_TYPES_CLIENT=(AES128,AES256,AES192,3DES168)
SQLNET.ENCRYPTION_CLIENT=requested
SQLNET.CRYPTO_CHECKSUM_CLIENT=requested
SQLNET.CRYPTO_CHECKSUM_TYPES_CLIENT=(SHA1)
It would be more secure to use “required” here instead of “requested”, if it works in your environment. I unfortunately must support a few older Oracle versions that cannot communicate with my OMS if I specify that encryption and crypto checksums are “required”.
Thanks very much for the quick response! I will take into consideration your new observations.
Hi Paige,
I’ve just pushed out an updated version of the script (v0.8) that resolves this typo and adds checks for the SQL*Net encryption parameters I listed in my previous reply. It will note an error on those parameters if you have enabled the MD5 hashing algorithm or any of the very weak encryption algorithms (DES, DES40, 3DES112, or any of the RC4_* algorithms). Thank you again for catching this problem!
-Brian
Hi Brian, I finished securing my new 13c environment for TLSv1.2 only but I ended up being unable to disable NodeManager for TLSv1 and TLSV1.1 (FAILED). I had Oracle support perplexed and ready to file a bug. How did you accomplish this? Also in your script, I had to change protocol for “checking HIGH strength ciphers on $openssl_check_component from -tls1 to -tls1.2 because I’m only interested in Protocol TLSv1.2
Hi Bob, I’m at OpenWorld away from my systems right now but I think I can answer this — I know I need to, but I haven’t yet attempted to disable TLSv1 and TLSv1.1on any of my EM13c components. So I haven’t run into this issue, but once I’m back in the office I plan to give it a try. I’m not even sure if there’s a MOS note covering NodeManager for disabling older TLS versions. I will follow up with you here and probably with another post once I try it out and see how things go.
Thanks for pointing out the -tls1 issue, I will get that resolved ASAP. My OMS is on an older SLES release with an ancient OpenSSL that only knows TLSv1 and I’ve only recently started updating the script for an OpenSSL compatible with the more recent ciphers while trying to keep backwards compatibility for others stuck on OpenSSL 0.9.8 like I have been.
Cheers!
Hi Bob,
I believe I have resolved this tls1 vs tls1_2 issue in the cipher strength check with my latest version. It should now use tls1_2 if supported by your OpenSSL install, and fall back to tls1 if not. I’ve also gone through the steps to disable TLSv1 and TLSv1.1 across the stack, and I see the same behavior as you are seeing – NodeManager still allows connections with TLSv1 or TLSv1.1, even though the OMS console, upload port, BIP, and so on all accept only TLSv1.2. I would be very interested to hear the final report on the SR that you filed and whether or not they can provide a fix that limits TLS versions permitted by NodeManager.
-Brian
Hi Brian, oracle was able to reproduce the issue with nodemanager communication for tls1.2 only but still working for tls1,tls1.1. They have raised a bug. It does work fine for weblogic but when the rest of em13c stack is added – does not work.
Try this for NodeManager, it just worked for me. Edit the startNodeManager.sh script inside of …/user_projects/domains/GCDomain/bin, and change the final line that sets the JAVA_OPTIONS variable by adding “-Dhttps.protocols=TLSv1.2 -Dweblogic.security.SSL.minimumProtocolVersion=TLSv1.2″. I do not know for sure that this is the supported way to do so, but as long as I can still apply the next set of monthly patches I will consider this good enough. The end of my startNodeManager.sh script now looks like:
#JAVA_OPTIONS=”-Dohs.product.home=/oracle/oem/Middleware13cR1/ohs ${JAVA_OPTIONS}”
JAVA_OPTIONS=”-Dohs.product.home=/oracle/oem/Middleware13cR1/ohs ${JAVA_OPTIONS} -Dhttps.protocols=TLSv1.2 -Dweblogic.security.SSL.minimumProtocolVersion=TLSv1.2 ”
export JAVA_OPTIONS
# start node manager
${WL_HOME}/server/bin/startNodeManager.sh
Just another hopefully final followup on the NodeManager TLSv1 / TLSv1.1 / TLSv1.2 issue.
I have upgraded my OEM environment to EM13cR2 (the just released 13.2). After I finished, NodeManager was back to accepting TLSv1 and TLSv1.1 connections, which I did not want. I made the same modifications to the NodeManager startup script as I had with the first release of EM13c (the “-Dhttps.protocols=TLSv1.2 -Dweblogic.security.SSL.minimumProtocolVersion=TLSv1.2” settings I mentioned in a comment), and that did not resolve the issue for me.
However, as soon as I stopped OEM and applied the WLS 12.1.3 160719 PSU patch and restarted the system, NodeManager came up in TLSv1.2-only mode. So I believe this patch (and probably any future WLS 12.1.3 PSU) is a prerequisite for running NodeManager with TLSv1.2 only.
Hi Bob, so how did you fix your OEM using Protocol TLSv1.2 ?
Pingback: Script to automate lock down of all EM13c agents to TLSv1.2 with EMCLI | Pardy DBA
Very extensive article, thanks for sharing.
Pingback: Securing Oracle Enterprise Manager 13cR2 | Pardy DBA