EM12c opatchauto, SHA256, and you

This post serves to document an issue I encountered after replacing expired SSL/TLS certificates on the server I use for Oracle Enterprise Manager 12c. To put it simply, using opatchauto to apply EM12c PSUs does not work if your WebLogic adminserver has a certificate installed that uses the SHA256 hashing algorithm.

[UPDATED 20151012: Please see this comment and this comment below, from Adam Robinson, who has provided a workaround that may work for you involving editing the opatchauto script to enable JSSE. As always, please consider workarounds requiring you to edit files as unsupported and at your own risk, but I would consider this fix superior to reverting back to the demo certificate every time you need to patch. You will need to repeat this fix every time you update OPatch in your OMS_HOME, though. Adam’s workaround does succeed in my environment.]

Error message

Expect to see the following error when running “opatchauto apply -analyze” or “opatchauto apply” against an installation with an SHA256-hashed certificate on the WLS adminserver:

oracle@omshost:/oracle/stage/21603255> opatchauto apply -analyze -property_file ~/property_file 
OPatch Automation Tool
Copyright (c) 2014, Oracle Corporation.  All rights reserved.


OPatchauto version : 11.1.0.12.3
OUI version        : 11.1.0.12.0
Running from       : /oracle/oem/Middleware12cR4/oms
Log file location  : /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatch/opatch2015-09-11_10-57-19AM_1.log

OPatchauto log file: /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatchauto/21603255/opatch_oms_2015-09-11_10-57-22AM_analyze.log



OPatchauto failed to establish JMX connection to weblogic server. This could be because of one (or) more of the following reasons:
1. Weblogic admin server URL that manages OMS application may not be right.
2. Weblogic admin server credentials (username, password) may not be right.
3. Virtual host configuration. If OMS, weblogic server are on virtual host configuration, Please make sure to add OPatchAuto.OMS_DISABLE_HOST_CHECK=true to command line and run again. (example: /oracle/oem/Middleware12cR4/oms/OPatch/opatchauto apply -analyze -property_file /home/oracle/property_file -invPtrLoc /oracle/oem/Middleware12cR4/oms/oraInst.loc  OPatchAuto.OMS_DISABLE_HOST_CHECK=true)

Please check above conditions and if error(s) still persist, Please contact Oracle support.


[ Error during Get weblogic Admin Server information Phase]. Detail: OPatchauto was not able to find right interview inputs.
OPatchauto failed: 
OPatchauto failed to establish JMX connection to weblogic server. This could be because of one (or) more of the following reasons:
1. Weblogic admin server URL that manages OMS application may not be right.
2. Weblogic admin server credentials (username, password) may not be right.
3. Virtual host configuration. If OMS, weblogic server are on virtual host configuration, Please make sure to add OPatchAuto.OMS_DISABLE_HOST_CHECK=true to command line and run again. (example: /oracle/oem/Middleware12cR4/oms/OPatch/opatchauto apply -analyze -property_file /home/oracle/property_file -invPtrLoc /oracle/oem/Middleware12cR4/oms/oraInst.loc  OPatchAuto.OMS_DISABLE_HOST_CHECK=true)

Please check above conditions and if error(s) still persist, Please contact Oracle support.

Log file location: /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatchauto/21603255/opatch_oms_2015-09-11_10-57-22AM_analyze.log

Recommended actions: Please correct the interview inputs and run opatchauto again.

OPatchauto failed with error code 231

Confirmation of the issue

To confirm this issue in your environment after receiving the preceding error message, check the hashing algorithm used on your adminserver certificate. I prefer to use the openssl commandline tool for this. If you don’t know the port used for your adminserver, you can retrieve it from the $EM_INSTANCE_BASE/em/EMGC_OMS1/emgc.properties file under AS_HTTPS_PORT. If your certificate does not show the usage of SHA256 (or another hash algorithm from the SHA-2 family) as in my example below, you may have a different problem.

oracle@omshost:~> openssl s_client -prexit -connect omshost.domain.com:7103 /dev/null | openssl x509 -text -in /dev/stdin | grep "Signature Algorithm" 2> /dev/null
        Signature Algorithm: sha256WithRSAEncryption
    Signature Algorithm: sha256WithRSAEncryption

Workaround

To work around this issue, you need to (temporarily!) replace the certificate on your WLS adminserver. Now, whenever I need to apply a PSU release, I resecure WLS using the default demonstration certificate, apply the PSU, then replace my original certificate.

oracle@omshost:/oracle/stage/21603255> emctl secure wls -use_demo_cert
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Securing WLS... Started.
Enter Enterprise Manager Root (SYSMAN) Password :
Securing WLS... Successful
Restart OMS using 'emctl stop oms -all' and 'emctl start oms'
oracle@omshost:/oracle/stage/21603255> emctl stop oms -all ; sleep 5 ; emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
Stopping BI Publisher Server...
BI Publisher Server Successfully Stopped
AdminServer Successfully Stopped
BI Publisher Server is Down
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
Starting BI Publisher Server ...
BI Publisher Server Successfully Started
BI Publisher Server is Up

[install the PSU according to the README instructions, including any post-installation steps]

oracle@omshost:/oracle/stage/21603255> emctl secure wls -wallet /oracle/oem/oemwallet
Oracle Enterprise Manager Cloud Control 12c Release 4  
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Securing WLS... Started.
Enter Enterprise Manager Root (SYSMAN) Password : 
Securing WLS... Successful
Restart OMS using 'emctl stop oms -all' and 'emctl start oms'
If there are multiple OMSs in this environment, perform this configuration on all of them.
oracle@omshost:/oracle/stage/21603255> emctl stop oms -all ; sleep 5 ; emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
Stopping BI Publisher Server...
BI Publisher Server Successfully Stopped
AdminServer Successfully Stopped
BI Publisher Server is Down
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
Starting BI Publisher Server ...
BI Publisher Server Successfully Started
BI Publisher Server is Up

I have not noticed any other EM12c issues using SHA256-hashed certificates. With this workaround, you can both continue to use quality certificates and keep your OMS patched.

Advertisements

14 thoughts on “EM12c opatchauto, SHA256, and you

  1. Adam Robinson

    Ran into this exact issue myself setting up a new EM12cR5. Have a ticket open right now, but its not really going anywhere.

    Are you sure things are working correctly with your SHA256 cert? In addition to the patching issue, I was seeing stuff like this in EMGC_OMS1.out:

    I patched WebLogic to 10.3.0.6.12 and Patch 12932224 as was recommended by some knowledge documents for similar issues with WebLogic, but the problem persisted. I decided to edit startEMServer.sh based upon Doc ID 1938799.1 for remediating POODLE. Instead of just adding:
    JAVA_OPTIONS=”${JAVA_OPTIONS} -Dweblogic.security.SSL.protocolVersion=TLS1″
    in the aforementioned places I added:
    JAVA_OPTIONS=”${JAVA_OPTIONS} -Dweblogic.security.SSL.protocolVersion=TLS1 -Dweblogic.ssl.JSSEEnabled=true ”

    This stopped the errors I was seeing in EMGC_OMS1.out

    Reply
    1. Brian Pardy Post author

      Hi Adam,

      Apologies for the delay getting this comment approved, it got filed in with spam comments which I only check occasionally.

      As far as I can tell, my SHA256 cert is working correctly. I do have the WLS 10.3.0.6.12 patch in, but I have not run across any references to patch 12932224 and haven’t yet installed it. I’ll try taking a look at it.

      I see JAVA_OPTIONS does include -Dweblogic.ssl.JSSEEnabled=true, but only for EMGC_JVMDMANAGER. It looks like HTML filtering (or something else) ate the warnings from your EMGC_OMS1.out file that you had tried to post. Could you please post the warnings or errors that you saw? I would be interested in taking a look through my logs to see if I have similar entries noted. Although I am still on EM12c R4 so you may be seeing something that only exists in the newer code base.

      Thanks!
      -Brian

      Reply
      1. Adam Robinson

        Hi Brian,
        The two errors I noticed (in addition to the “Ignoring the trusted CA certificate” error) were:
        BEA-141151
        The admin server could not be reached at https://oms1-p.redacted.com:7101.
        BEA-150018
        This server is being started in managed server independence mode in the absence of the admin server.

        The logs had a bunch of angle brackets so that probably caused the issue.

        I found bugs 20949233 and 20554421 and they describe what I am seeing pretty much exactly. However they attribute the issue to a 2048 bit key size, not that the cert is SHA-2.

      2. Brian Pardy Post author

        Hi Adam,

        I can confirm that I see both BEA-141151 and BEA-150018 errors in my EMGC_OMS1.out when running with a 2048 bit SHA-2 cert, and that they do not appear when running with the demo cert. Good find! I also now see a MOS doc referring to patch 12932224 that you mentioned previously (MOS PowerView had hidden it from me). I think you/they may be on to something about the key size rather than SHA-2, that should be easy enough to test. I’m going to try some of these other patches and workarounds on my side when I have the chance, though I don’t expect results better than you have already seen. I will follow up if I find something that tolerates 2048 bit SHA-2, to get all the bases covered.

        Thanks!

  2. Adam Robinson

    Hi Brian,
    I work at a University and we’re a member of the InCommon Federation which provides certificate signing services for us. Unfortunately, they will not sign a cert with less than a 2048 bit key so I cannot try making a 1024 bit SHA-2 cert 😦

    We do have an internal CA for Active Directory and I am going to try signing a 1024 bit key with SHA-2 on there. That will at least be more secure than the demo cert and I think my browser should still trust it as my machine is joined to the domain. I’ll let you know how that goes.

    Reply
    1. Brian Pardy Post author

      Hi Adam,

      I won’t argue with anyone that wants to avoid 1024 bit keys 🙂 I’m unfortunately just using a manual OpenSSL CA for now, as our commercial cert provider strongly prefers wildcard/subject-alternate-name certs, which EM12c won’t accept, and they balked at providing a dozen site-specific certificates for my OMS and agents. Perhaps I can test a bit more easily than you can. I’ll definitely update on this thread when I have more information.

      Have a great day/weekend!
      -Brian

      Reply
    2. Brian Pardy Post author

      I’ve just tried a 1024-bit certificate signed with SHA-256 by my root CA – I still see the same errors in EMGC_OMS1.out, and opatchauto still fails while trying to run ‘opatchauto apply -analyze’ against a recent system patch. With EM12cR4 and the set of patches I’m running, it looks like 1024- vs 2048-bit is not at the root of this issue.

      Reply
      1. Adam Robinson

        I think I got it to work. In $OMS_HOME/OPatch/scripts/oms/opatchauto I edited the line near the end that starts with $JAVA and added “-Dweblogic.security.SSL.enableJSSE=true”. There must be some old component of WebLogic bundled with OPatch as the newer “-Dweblogic.security.SSL.enableJSSE=true” doesn’t work.

  3. Adam Robinson

    Looks like you need BOTH “-Dweblogic.security.SSL.enableJSSE=true” -Dweblogic.ssl.JSSEEnabled=true” for it to work.

    Reply
    1. Brian Pardy Post author

      Thanks Adam, this is great! I’ve just added both defines to opatchauto’s Java invocation on my system, and confirmed that the “opatchauto apply -analyze” step succeeds for me now where it did not before. I don’t have any pending patches to apply so I can’t test that out right now, but I am very confident this will take care of it. I’m going to update this post to point to your workaround, with the usual “unsupported, at your own risk” disclaimer. I will also ping an EM12c product manager or two via Twitter to make sure they’re aware of this. Thank you again – this is much better than reverting to the demo cert.

      -Brian

      Reply
      1. Adam Robinson

        Hi Brian,
        The fix didn’t work on my second host. So there must be something else in conjunction with that fix on my working host/your working host.

      2. Brian Pardy Post author

        Revising my update to say this “may work” 🙂

        I’d be happy to be someone to bounce ideas off of if you’d like, but please don’t feel compelled to debug it here unless you wish. Is your second host a separate EM12c system, or a secondary OMS in the same system? I wonder if it’s possibly a cert chain validation issue if you’re not using the same wallet on each system. I only have one OMS, so I don’t know the process for multi-OMS sites so I’m not sure where it could go off the rails. You may have to enable SSL debugging to see what’s up there. Good luck!

  4. Adam Robinson

    I figured out my issue, on my second host I was editing the debug that echo’s the command instead of the command itself ;-).

    Reply
  5. Pingback: Securing Oracle Enterprise Manager 13c | Pardy DBA

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s