This post serves to document an issue I encountered after replacing expired SSL/TLS certificates on the server I use for Oracle Enterprise Manager 12c. To put it simply, using opatchauto to apply EM12c PSUs does not work if your WebLogic adminserver has a certificate installed that uses the SHA256 hashing algorithm.
[UPDATED 20151012: Please see this comment and this comment below, from Adam Robinson, who has provided a workaround that may work for you involving editing the opatchauto script to enable JSSE. As always, please consider workarounds requiring you to edit files as unsupported and at your own risk, but I would consider this fix superior to reverting back to the demo certificate every time you need to patch. You will need to repeat this fix every time you update OPatch in your OMS_HOME, though. Adam’s workaround does succeed in my environment.]
Error message
Expect to see the following error when running “opatchauto apply -analyze” or “opatchauto apply” against an installation with an SHA256-hashed certificate on the WLS adminserver:
oracle@omshost:/oracle/stage/21603255> opatchauto apply -analyze -property_file ~/property_file OPatch Automation Tool Copyright (c) 2014, Oracle Corporation. All rights reserved. OPatchauto version : 11.1.0.12.3 OUI version : 11.1.0.12.0 Running from : /oracle/oem/Middleware12cR4/oms Log file location : /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatch/opatch2015-09-11_10-57-19AM_1.log OPatchauto log file: /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatchauto/21603255/opatch_oms_2015-09-11_10-57-22AM_analyze.log OPatchauto failed to establish JMX connection to weblogic server. This could be because of one (or) more of the following reasons: 1. Weblogic admin server URL that manages OMS application may not be right. 2. Weblogic admin server credentials (username, password) may not be right. 3. Virtual host configuration. If OMS, weblogic server are on virtual host configuration, Please make sure to add OPatchAuto.OMS_DISABLE_HOST_CHECK=true to command line and run again. (example: /oracle/oem/Middleware12cR4/oms/OPatch/opatchauto apply -analyze -property_file /home/oracle/property_file -invPtrLoc /oracle/oem/Middleware12cR4/oms/oraInst.loc OPatchAuto.OMS_DISABLE_HOST_CHECK=true) Please check above conditions and if error(s) still persist, Please contact Oracle support. [ Error during Get weblogic Admin Server information Phase]. Detail: OPatchauto was not able to find right interview inputs. OPatchauto failed: OPatchauto failed to establish JMX connection to weblogic server. This could be because of one (or) more of the following reasons: 1. Weblogic admin server URL that manages OMS application may not be right. 2. Weblogic admin server credentials (username, password) may not be right. 3. Virtual host configuration. If OMS, weblogic server are on virtual host configuration, Please make sure to add OPatchAuto.OMS_DISABLE_HOST_CHECK=true to command line and run again. (example: /oracle/oem/Middleware12cR4/oms/OPatch/opatchauto apply -analyze -property_file /home/oracle/property_file -invPtrLoc /oracle/oem/Middleware12cR4/oms/oraInst.loc OPatchAuto.OMS_DISABLE_HOST_CHECK=true) Please check above conditions and if error(s) still persist, Please contact Oracle support. Log file location: /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatchauto/21603255/opatch_oms_2015-09-11_10-57-22AM_analyze.log Recommended actions: Please correct the interview inputs and run opatchauto again. OPatchauto failed with error code 231
Confirmation of the issue
To confirm this issue in your environment after receiving the preceding error message, check the hashing algorithm used on your adminserver certificate. I prefer to use the openssl commandline tool for this. If you don’t know the port used for your adminserver, you can retrieve it from the $EM_INSTANCE_BASE/em/EMGC_OMS1/emgc.properties file under AS_HTTPS_PORT. If your certificate does not show the usage of SHA256 (or another hash algorithm from the SHA-2 family) as in my example below, you may have a different problem.
oracle@omshost:~> openssl s_client -prexit -connect omshost.domain.com:7103 /dev/null | openssl x509 -text -in /dev/stdin | grep "Signature Algorithm" 2> /dev/null
Signature Algorithm: sha256WithRSAEncryption
Signature Algorithm: sha256WithRSAEncryption
Workaround
To work around this issue, you need to (temporarily!) replace the certificate on your WLS adminserver. Now, whenever I need to apply a PSU release, I resecure WLS using the default demonstration certificate, apply the PSU, then replace my original certificate.
oracle@omshost:/oracle/stage/21603255> emctl secure wls -use_demo_cert Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Securing WLS... Started. Enter Enterprise Manager Root (SYSMAN) Password : Securing WLS... Successful Restart OMS using 'emctl stop oms -all' and 'emctl start oms' oracle@omshost:/oracle/stage/21603255> emctl stop oms -all ; sleep 5 ; emctl start oms Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Stopping WebTier... WebTier Successfully Stopped Stopping Oracle Management Server... Oracle Management Server Successfully Stopped Oracle Management Server is Down Stopping BI Publisher Server... BI Publisher Server Successfully Stopped AdminServer Successfully Stopped BI Publisher Server is Down Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Starting Oracle Management Server... Starting WebTier... WebTier Successfully Started Oracle Management Server Successfully Started Oracle Management Server is Up Starting BI Publisher Server ... BI Publisher Server Successfully Started BI Publisher Server is Up [install the PSU according to the README instructions, including any post-installation steps] oracle@omshost:/oracle/stage/21603255> emctl secure wls -wallet /oracle/oem/oemwallet Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Securing WLS... Started. Enter Enterprise Manager Root (SYSMAN) Password : Securing WLS... Successful Restart OMS using 'emctl stop oms -all' and 'emctl start oms' If there are multiple OMSs in this environment, perform this configuration on all of them. oracle@omshost:/oracle/stage/21603255> emctl stop oms -all ; sleep 5 ; emctl start oms Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Stopping WebTier... WebTier Successfully Stopped Stopping Oracle Management Server... Oracle Management Server Successfully Stopped Oracle Management Server is Down Stopping BI Publisher Server... BI Publisher Server Successfully Stopped AdminServer Successfully Stopped BI Publisher Server is Down Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Starting Oracle Management Server... Starting WebTier... WebTier Successfully Started Oracle Management Server Successfully Started Oracle Management Server is Up Starting BI Publisher Server ... BI Publisher Server Successfully Started BI Publisher Server is Up
I have not noticed any other EM12c issues using SHA256-hashed certificates. With this workaround, you can both continue to use quality certificates and keep your OMS patched.
Pardy DBA 