This post serves to document an issue I encountered after replacing expired SSL/TLS certificates on the server I use for Oracle Enterprise Manager 12c. To put it simply, using opatchauto to apply EM12c PSUs does not work if your WebLogic adminserver has a certificate installed that uses the SHA256 hashing algorithm.
[UPDATED 20151012: Please see this comment and this comment below, from Adam Robinson, who has provided a workaround that may work for you involving editing the opatchauto script to enable JSSE. As always, please consider workarounds requiring you to edit files as unsupported and at your own risk, but I would consider this fix superior to reverting back to the demo certificate every time you need to patch. You will need to repeat this fix every time you update OPatch in your OMS_HOME, though. Adam’s workaround does succeed in my environment.]
Error message
Expect to see the following error when running “opatchauto apply -analyze” or “opatchauto apply” against an installation with an SHA256-hashed certificate on the WLS adminserver:
oracle@omshost:/oracle/stage/21603255> opatchauto apply -analyze -property_file ~/property_file OPatch Automation Tool Copyright (c) 2014, Oracle Corporation. All rights reserved. OPatchauto version : 11.1.0.12.3 OUI version : 11.1.0.12.0 Running from : /oracle/oem/Middleware12cR4/oms Log file location : /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatch/opatch2015-09-11_10-57-19AM_1.log OPatchauto log file: /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatchauto/21603255/opatch_oms_2015-09-11_10-57-22AM_analyze.log OPatchauto failed to establish JMX connection to weblogic server. This could be because of one (or) more of the following reasons: 1. Weblogic admin server URL that manages OMS application may not be right. 2. Weblogic admin server credentials (username, password) may not be right. 3. Virtual host configuration. If OMS, weblogic server are on virtual host configuration, Please make sure to add OPatchAuto.OMS_DISABLE_HOST_CHECK=true to command line and run again. (example: /oracle/oem/Middleware12cR4/oms/OPatch/opatchauto apply -analyze -property_file /home/oracle/property_file -invPtrLoc /oracle/oem/Middleware12cR4/oms/oraInst.loc OPatchAuto.OMS_DISABLE_HOST_CHECK=true) Please check above conditions and if error(s) still persist, Please contact Oracle support. [ Error during Get weblogic Admin Server information Phase]. Detail: OPatchauto was not able to find right interview inputs. OPatchauto failed: OPatchauto failed to establish JMX connection to weblogic server. This could be because of one (or) more of the following reasons: 1. Weblogic admin server URL that manages OMS application may not be right. 2. Weblogic admin server credentials (username, password) may not be right. 3. Virtual host configuration. If OMS, weblogic server are on virtual host configuration, Please make sure to add OPatchAuto.OMS_DISABLE_HOST_CHECK=true to command line and run again. (example: /oracle/oem/Middleware12cR4/oms/OPatch/opatchauto apply -analyze -property_file /home/oracle/property_file -invPtrLoc /oracle/oem/Middleware12cR4/oms/oraInst.loc OPatchAuto.OMS_DISABLE_HOST_CHECK=true) Please check above conditions and if error(s) still persist, Please contact Oracle support. Log file location: /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatchauto/21603255/opatch_oms_2015-09-11_10-57-22AM_analyze.log Recommended actions: Please correct the interview inputs and run opatchauto again. OPatchauto failed with error code 231
Confirmation of the issue
To confirm this issue in your environment after receiving the preceding error message, check the hashing algorithm used on your adminserver certificate. I prefer to use the openssl commandline tool for this. If you don’t know the port used for your adminserver, you can retrieve it from the $EM_INSTANCE_BASE/em/EMGC_OMS1/emgc.properties file under AS_HTTPS_PORT. If your certificate does not show the usage of SHA256 (or another hash algorithm from the SHA-2 family) as in my example below, you may have a different problem.
oracle@omshost:~> openssl s_client -prexit -connect omshost.domain.com:7103 /dev/null | openssl x509 -text -in /dev/stdin | grep "Signature Algorithm" 2> /dev/null
Signature Algorithm: sha256WithRSAEncryption
Signature Algorithm: sha256WithRSAEncryption
Workaround
To work around this issue, you need to (temporarily!) replace the certificate on your WLS adminserver. Now, whenever I need to apply a PSU release, I resecure WLS using the default demonstration certificate, apply the PSU, then replace my original certificate.
oracle@omshost:/oracle/stage/21603255> emctl secure wls -use_demo_cert Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Securing WLS... Started. Enter Enterprise Manager Root (SYSMAN) Password : Securing WLS... Successful Restart OMS using 'emctl stop oms -all' and 'emctl start oms' oracle@omshost:/oracle/stage/21603255> emctl stop oms -all ; sleep 5 ; emctl start oms Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Stopping WebTier... WebTier Successfully Stopped Stopping Oracle Management Server... Oracle Management Server Successfully Stopped Oracle Management Server is Down Stopping BI Publisher Server... BI Publisher Server Successfully Stopped AdminServer Successfully Stopped BI Publisher Server is Down Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Starting Oracle Management Server... Starting WebTier... WebTier Successfully Started Oracle Management Server Successfully Started Oracle Management Server is Up Starting BI Publisher Server ... BI Publisher Server Successfully Started BI Publisher Server is Up [install the PSU according to the README instructions, including any post-installation steps] oracle@omshost:/oracle/stage/21603255> emctl secure wls -wallet /oracle/oem/oemwallet Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Securing WLS... Started. Enter Enterprise Manager Root (SYSMAN) Password : Securing WLS... Successful Restart OMS using 'emctl stop oms -all' and 'emctl start oms' If there are multiple OMSs in this environment, perform this configuration on all of them. oracle@omshost:/oracle/stage/21603255> emctl stop oms -all ; sleep 5 ; emctl start oms Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Stopping WebTier... WebTier Successfully Stopped Stopping Oracle Management Server... Oracle Management Server Successfully Stopped Oracle Management Server is Down Stopping BI Publisher Server... BI Publisher Server Successfully Stopped AdminServer Successfully Stopped BI Publisher Server is Down Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Starting Oracle Management Server... Starting WebTier... WebTier Successfully Started Oracle Management Server Successfully Started Oracle Management Server is Up Starting BI Publisher Server ... BI Publisher Server Successfully Started BI Publisher Server is Up
I have not noticed any other EM12c issues using SHA256-hashed certificates. With this workaround, you can both continue to use quality certificates and keep your OMS patched.
Pardy DBA 
Ran into this exact issue myself setting up a new EM12cR5. Have a ticket open right now, but its not really going anywhere.
Are you sure things are working correctly with your SHA256 cert? In addition to the patching issue, I was seeing stuff like this in EMGC_OMS1.out:
I patched WebLogic to 10.3.0.6.12 and Patch 12932224 as was recommended by some knowledge documents for similar issues with WebLogic, but the problem persisted. I decided to edit startEMServer.sh based upon Doc ID 1938799.1 for remediating POODLE. Instead of just adding:
JAVA_OPTIONS=”${JAVA_OPTIONS} -Dweblogic.security.SSL.protocolVersion=TLS1″
in the aforementioned places I added:
JAVA_OPTIONS=”${JAVA_OPTIONS} -Dweblogic.security.SSL.protocolVersion=TLS1 -Dweblogic.ssl.JSSEEnabled=true ”
This stopped the errors I was seeing in EMGC_OMS1.out
Hi Adam,
Apologies for the delay getting this comment approved, it got filed in with spam comments which I only check occasionally.
As far as I can tell, my SHA256 cert is working correctly. I do have the WLS 10.3.0.6.12 patch in, but I have not run across any references to patch 12932224 and haven’t yet installed it. I’ll try taking a look at it.
I see JAVA_OPTIONS does include -Dweblogic.ssl.JSSEEnabled=true, but only for EMGC_JVMDMANAGER. It looks like HTML filtering (or something else) ate the warnings from your EMGC_OMS1.out file that you had tried to post. Could you please post the warnings or errors that you saw? I would be interested in taking a look through my logs to see if I have similar entries noted. Although I am still on EM12c R4 so you may be seeing something that only exists in the newer code base.
Thanks!
-Brian
Hi Brian,
The two errors I noticed (in addition to the “Ignoring the trusted CA certificate” error) were:
BEA-141151
The admin server could not be reached at https://oms1-p.redacted.com:7101.
BEA-150018
This server is being started in managed server independence mode in the absence of the admin server.
The logs had a bunch of angle brackets so that probably caused the issue.
I found bugs 20949233 and 20554421 and they describe what I am seeing pretty much exactly. However they attribute the issue to a 2048 bit key size, not that the cert is SHA-2.
Hi Adam,
I can confirm that I see both BEA-141151 and BEA-150018 errors in my EMGC_OMS1.out when running with a 2048 bit SHA-2 cert, and that they do not appear when running with the demo cert. Good find! I also now see a MOS doc referring to patch 12932224 that you mentioned previously (MOS PowerView had hidden it from me). I think you/they may be on to something about the key size rather than SHA-2, that should be easy enough to test. I’m going to try some of these other patches and workarounds on my side when I have the chance, though I don’t expect results better than you have already seen. I will follow up if I find something that tolerates 2048 bit SHA-2, to get all the bases covered.
Thanks!
Hi Brian,
I work at a University and we’re a member of the InCommon Federation which provides certificate signing services for us. Unfortunately, they will not sign a cert with less than a 2048 bit key so I cannot try making a 1024 bit SHA-2 cert 😦
We do have an internal CA for Active Directory and I am going to try signing a 1024 bit key with SHA-2 on there. That will at least be more secure than the demo cert and I think my browser should still trust it as my machine is joined to the domain. I’ll let you know how that goes.
Hi Adam,
I won’t argue with anyone that wants to avoid 1024 bit keys 🙂 I’m unfortunately just using a manual OpenSSL CA for now, as our commercial cert provider strongly prefers wildcard/subject-alternate-name certs, which EM12c won’t accept, and they balked at providing a dozen site-specific certificates for my OMS and agents. Perhaps I can test a bit more easily than you can. I’ll definitely update on this thread when I have more information.
Have a great day/weekend!
-Brian
I’ve just tried a 1024-bit certificate signed with SHA-256 by my root CA – I still see the same errors in EMGC_OMS1.out, and opatchauto still fails while trying to run ‘opatchauto apply -analyze’ against a recent system patch. With EM12cR4 and the set of patches I’m running, it looks like 1024- vs 2048-bit is not at the root of this issue.
I think I got it to work. In $OMS_HOME/OPatch/scripts/oms/opatchauto I edited the line near the end that starts with $JAVA and added “-Dweblogic.security.SSL.enableJSSE=true”. There must be some old component of WebLogic bundled with OPatch as the newer “-Dweblogic.security.SSL.enableJSSE=true” doesn’t work.
Looks like you need BOTH “-Dweblogic.security.SSL.enableJSSE=true” -Dweblogic.ssl.JSSEEnabled=true” for it to work.
Thanks Adam, this is great! I’ve just added both defines to opatchauto’s Java invocation on my system, and confirmed that the “opatchauto apply -analyze” step succeeds for me now where it did not before. I don’t have any pending patches to apply so I can’t test that out right now, but I am very confident this will take care of it. I’m going to update this post to point to your workaround, with the usual “unsupported, at your own risk” disclaimer. I will also ping an EM12c product manager or two via Twitter to make sure they’re aware of this. Thank you again – this is much better than reverting to the demo cert.
-Brian
Hi Brian,
The fix didn’t work on my second host. So there must be something else in conjunction with that fix on my working host/your working host.
Revising my update to say this “may work” 🙂
I’d be happy to be someone to bounce ideas off of if you’d like, but please don’t feel compelled to debug it here unless you wish. Is your second host a separate EM12c system, or a secondary OMS in the same system? I wonder if it’s possibly a cert chain validation issue if you’re not using the same wallet on each system. I only have one OMS, so I don’t know the process for multi-OMS sites so I’m not sure where it could go off the rails. You may have to enable SSL debugging to see what’s up there. Good luck!
I figured out my issue, on my second host I was editing the debug that echo’s the command instead of the command itself ;-).
Pingback: Securing Oracle Enterprise Manager 13c | Pardy DBA