Tag Archives: bug

Fix: Plugin error when upgrading EM13cR1 agent to EM13cR2 (13.2) on Windows 2008 R2 x64

I ran into the following issue while attempting to upgrade to the Oracle management agent on my one Windows (2008 R2, x64) server to the 13.2.0.0.0 version distributed with Oracle Enterprise Manager 13.2. The agent upgrade repeatedly failed in the “Upgrading Management Agent” step with an error message:


Exit Code :0
The version is 13.1.0.0.0
Checking for the version 13.2.0.0.0
The agent is not upgraded successfully
[...]
Plugins upgrade failed.
Plugin upgrade failed.
Check the file E:/agent13c/agent_inst/install/plugins_upgrade.txt.status on agent for plugin upgrade status.
Check latest E:/agent13c/agent_inst/install/logs/agentplugindeploy_.log
Plugins upgrade failed.0

The log file referenced in the error message contains some, but not much, additional information:


The command executed for discovery at _2016_10_12_11_48_28 is : E:\agent13c\agent_inst\bin\emctl_upgrade.bat set_discovery_root plugin oracle.sysman.si E:\agent13c\agent_13.2.0.0.0\plugins\oracle.sysman.si.discovery.plugin_13.2.1.0.0 13.2.1.0.0
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
return value is : 65280
Install case : set_discovery_root failure...existing on error value 65280

I fixed this issue by installing the Microsoft Visual C++ 2010 Service Pack 1 Redistributable Package MFC Security Update package on the server. Finally, create a new directory called “prerelogs” inside of the …\agent_13.2.0.0.0\cfgtoollogs\ directory, or else my deployment hung forever during the “Performing prerequisite checks” step. Note that “prerelogs” does not contain the letter q, do not use ‘prereqlogs’. The agent deployment uses this directory to extract software for running the prerequisite checks; possibly setting SCRATCHPATH or another extra parameter would resolve this issue without manually creating the directory, but this worked for me.

After installation of the VC++ 2010 package, which did not require a reboot, followed by creating of the prerelogs directory, I repeated the agent upgrade and it completed successfully.

This issue may or may not occur on other Windows versions, I have not heard any other reports of problems. The first release of EM13c did not seem to require this package.

Advertisements

Oracle PSU 12.1.0.2.160719 (patch 23054246) for Linux x86-64 requires libodbcinst

Oracle recently released patch 23054246 (DATABASE PATCH SET UPDATE 12.1.0.2.160719) for database 12.1.0.2, containing security updates from the July 2016 critical patch update advisory.

[EDIT 20160726: Oracle has documented this issue in MOS note 2163593.1]

This patch appears to have introduced a dependency on libodbcinst. During my first attempt to install this patch, I received errors while linking libsqora. The error appears as follows in OPatch logs:


[Jul 20, 2016 11:22:57 AM] The following warnings have occurred during OPatch execution:
[Jul 20, 2016 11:22:57 AM] 1) OUI-67200:Make failed to invoke "/usr/bin/make -f ins_odbc.mk isqora ORACLE_HOME=/oracle/oem/product/12.1.0/awrdb"....'/usr/lib64/gcc/x86_64-suse-linux/4.3/../../../../x86_64-suse-linux/bin/ld: cannot find -lodbcinst
collect2: ld returned 1 exit status
make: *** [/oracle/oem/product/12.1.0/awrdb/odbc/lib/libsqora.so.12.1] Error 1
'
[Jul 20, 2016 11:22:57 AM] 2) OUI-67124:Re-link fails on target "isqora".
[Jul 20, 2016 11:22:57 AM] 3) OUI-67200:Make failed to invoke "/usr/bin/make -f ins_odbc.mk isqora ORACLE_HOME=/oracle/oem/product/12.1.0/awrdb"....'/usr/lib64/gcc/x86_64-suse-linux/4.3/../../../../x86_64-suse-linux/bin/ld: cannot find -lodbcinst
collect2: ld returned 1 exit status
make: *** [/oracle/oem/product/12.1.0/awrdb/odbc/lib/libsqora.so.12.1] Error 1
'
[Jul 20, 2016 11:22:57 AM] 4) OUI-67124:
NApply was not able to restore the home. Please invoke the following scripts:
- restore.[sh,bat]
- make.txt (Unix only)
to restore the ORACLE_HOME. They are located under
"/oracle/oem/product/12.1.0/awrdb/.patch_storage/NApply/2016-07-20_11-20-22AM"

After installing the unixODBC package on my SLES11 system, this error went away.

[Update: see also Brian Peasland’s blog post “July 2016 PSU fails to make isqora” for a different workaround to this issue that does not involving installing any additional packages.]

At the time of release, Oracle’s installation requirements for database 12.1.0.2 listed the unixODBC package as an optional dependency, required only “[i]f you intend to use ODBC”. This no longer seems to hold true. At the moment Oracle has not made it clear whether or not patch 23054246 contains a bug that introduces the libodbcinst dependency or if the database platform will require this library in all cases going forward.

If you have attempted patch application without libodbcinst available, the opatch apply step will fail and you will have to manually revert the patch, following the instructions that OPatch provides and/or contact Oracle Support for guidance. In my case, I followed the instructions to revert, installed unixODBC, then attempted again to apply the patch, at which point it completed successfully as expected. If you have not yet attempted to apply this patch, I highly recommend installing unixODBC first. I have already seen two others report on Twitter that they encountered this issue but none have yet confirmed to me that installing unixODBC resolved the problem. I believe it will.

UPDATE: See also “BUG 24332805 – OUI-67124:RE-LINK FAILS ON TARGET “ISQORA” DURING JUL 2016 PSU APPLY” once made public.

WORKAROUND: Unable to monitor Oracle XE 11gR2 with Oracle Enterprise Manager 13c

I have recently switched to using Oracle Enterprise Manager 13c (EM13c – 13.1), replacing my previous EM12c installation.  I elected to install a clean new environment instead of an upgrade, because my old install had been upgraded repeatedly going back to the initial release of EM12c and I wanted a fresh start.

I encountered only one difficult issue during the process. When I attempted to add one production Oracle XE 11gR2 database target, EM13c could not compute the target’s dynamic properties, leaving the target broken. Since you cannot submit jobs against a broken target, this prevented me from using EM13c to back up this database.  I had no comparable issues with XE as a target under EM12c.

The key metric errors that showed during this process included:
“Metric evaluation error start – Target {oracle_database.SID.domain.com} is broken: Dynamic Category property error,Get dynamic property error,No such metadata – No valid queryDescriptor or executionDescriptor found for target [oracle_database.SID.domain.com$30]”

and for the database system target:

“Metric evaluation error start – Received an exception when evaluating sev_eval_proc for:Target name = SID.domain.com_sys, metric_name = Response, metric_column = Status; Error msg = Target encountered metric erros; at least one member in in metric error”

I enabled debugging for the agent logs and attempted again to add the XE target.  Errors showing up in the logs included:

2016-01-15 12:10:05,905 [1806:4CE3192] DEBUG – Computing of dynamic property: [ComputeVC] is done (1 msec, error=true)

2016-01-15 12:10:06,452 [1806:F917F5F8] DEBUG – Computing of dynamic property: [GetDumpDestination] is done (0 msec, error=true)

2016-01-15 12:10:06,508 [1813:6EEEAC87] DEBUG – Computing of dynamic property: [DeduceAlertLogFile] is done (1 msec, error=true)

2016-01-15 12:11:18,779 [1830:CD3A325D] DEBUG – Error was added to oracle_database.SID.domain.com$23(0|MONITORED|false|true|<UF>): Invalid Input

2016-01-15 12:11:18,779 [1831:3657AE55] DEBUG – abandoning long op “CDProps:oracle_database.SID.domain.com:ComputeVC:GENERIC_TASK:Fri Jan 15 12:11:18 EST 2016”

2016-01-15 12:11:18,780 [1830:CD3A325D] DEBUG – Error during dynamic property ComputeVC calculation: critical=true, missingCatProps=[VersionCategory], missingProps=[VersionCategory] oracle.sysman.emSDK.agent.fetchlet.exception.FetchletException: Invalid Input

2016-01-15 12:35:38,038 INFO – Finished dynamic properties computation (total time 817 ms). Number of DP computed: 19. Number of DP with error: 3. Number of dynamic properties that were added: 132.

After reviewing the logs carefully (and posting this as a question in the MOS Oracle XE forum – https://community.oracle.com/thread/3892946) I eventually narrowed the issue down to a query that EM13c runs against DBA_REGISTRY_HISTORY in a target database when added. For database versions greater than 11.2 but less than 12.1.0.2, EM13c assumes that DBA_REGISTRY_HISTORY contains a BUNDLE_SERIES column.  This column does not exist in Oracle XE 11gR2, which reports a version string of 11.2.0.2.

This bug should eventually get a fix as EM13c gets patched, but in the meantime if you need to monitor an Oracle XE target with EM13c, the following workaround took care of the problem for me: create a new DBA_REGISTRY_HISTORY table containing a BUNDLE_SERIES column in your monitoring user’s schema in XE.  So, as user DBSNMP on XE, I ran:

SQL> create table dba_registry_history (ACTION_TIME TIMESTAMP(6), ACTION VARCHAR2(30), NAMESPACE VARCHAR2(30), VERSION VARCHAR2(30), ID NUMBER, BUNDLE_SERIES VARCHAR2(30), COMMENTS VARCHAR2(255));

Since one cannot patch XE, the real DBA_REGISTRY_HISTORY view has no rows and so you do not need to populate any data into this new table.

After adding the table, force a recalculation of dynamic properties by running the following against the EM13c management agent on the XE server:

$ emctl reload agent dynamicproperties SID.domain.com:oracle_database
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD recompute dynprops completed successfully

Once that completed successfully my XE target started to show the correct status in EM13c and I can submit jobs against the target.  All fixed.  I recommend deleting the DBSNMP.DBA_REGISTRY_HISTORY table once the bug gets fixed in OEM.

[EDIT 20160216: Oracle has documented this issue in MOS note EM13c: Database Target Status Shows “Dynamic Category property error” In 13c Cloud Control (Doc ID 2105001.1) and in bug 22592461 DATABASE TARGET STATUS SHOWS “DYNAMIC CATEGORY PROPERTY ERROR” IN 13C CONSOLE. Users on supported databases (e.g., not Oracle XE) should follow the resolution steps in that document instead to correct the real error.]

EM12c opatchauto, SHA256, and you

This post serves to document an issue I encountered after replacing expired SSL/TLS certificates on the server I use for Oracle Enterprise Manager 12c. To put it simply, using opatchauto to apply EM12c PSUs does not work if your WebLogic adminserver has a certificate installed that uses the SHA256 hashing algorithm.

[UPDATED 20151012: Please see this comment and this comment below, from Adam Robinson, who has provided a workaround that may work for you involving editing the opatchauto script to enable JSSE. As always, please consider workarounds requiring you to edit files as unsupported and at your own risk, but I would consider this fix superior to reverting back to the demo certificate every time you need to patch. You will need to repeat this fix every time you update OPatch in your OMS_HOME, though. Adam’s workaround does succeed in my environment.]

Error message

Expect to see the following error when running “opatchauto apply -analyze” or “opatchauto apply” against an installation with an SHA256-hashed certificate on the WLS adminserver:

oracle@omshost:/oracle/stage/21603255> opatchauto apply -analyze -property_file ~/property_file 
OPatch Automation Tool
Copyright (c) 2014, Oracle Corporation.  All rights reserved.


OPatchauto version : 11.1.0.12.3
OUI version        : 11.1.0.12.0
Running from       : /oracle/oem/Middleware12cR4/oms
Log file location  : /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatch/opatch2015-09-11_10-57-19AM_1.log

OPatchauto log file: /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatchauto/21603255/opatch_oms_2015-09-11_10-57-22AM_analyze.log



OPatchauto failed to establish JMX connection to weblogic server. This could be because of one (or) more of the following reasons:
1. Weblogic admin server URL that manages OMS application may not be right.
2. Weblogic admin server credentials (username, password) may not be right.
3. Virtual host configuration. If OMS, weblogic server are on virtual host configuration, Please make sure to add OPatchAuto.OMS_DISABLE_HOST_CHECK=true to command line and run again. (example: /oracle/oem/Middleware12cR4/oms/OPatch/opatchauto apply -analyze -property_file /home/oracle/property_file -invPtrLoc /oracle/oem/Middleware12cR4/oms/oraInst.loc  OPatchAuto.OMS_DISABLE_HOST_CHECK=true)

Please check above conditions and if error(s) still persist, Please contact Oracle support.


[ Error during Get weblogic Admin Server information Phase]. Detail: OPatchauto was not able to find right interview inputs.
OPatchauto failed: 
OPatchauto failed to establish JMX connection to weblogic server. This could be because of one (or) more of the following reasons:
1. Weblogic admin server URL that manages OMS application may not be right.
2. Weblogic admin server credentials (username, password) may not be right.
3. Virtual host configuration. If OMS, weblogic server are on virtual host configuration, Please make sure to add OPatchAuto.OMS_DISABLE_HOST_CHECK=true to command line and run again. (example: /oracle/oem/Middleware12cR4/oms/OPatch/opatchauto apply -analyze -property_file /home/oracle/property_file -invPtrLoc /oracle/oem/Middleware12cR4/oms/oraInst.loc  OPatchAuto.OMS_DISABLE_HOST_CHECK=true)

Please check above conditions and if error(s) still persist, Please contact Oracle support.

Log file location: /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatchauto/21603255/opatch_oms_2015-09-11_10-57-22AM_analyze.log

Recommended actions: Please correct the interview inputs and run opatchauto again.

OPatchauto failed with error code 231

Confirmation of the issue

To confirm this issue in your environment after receiving the preceding error message, check the hashing algorithm used on your adminserver certificate. I prefer to use the openssl commandline tool for this. If you don’t know the port used for your adminserver, you can retrieve it from the $EM_INSTANCE_BASE/em/EMGC_OMS1/emgc.properties file under AS_HTTPS_PORT. If your certificate does not show the usage of SHA256 (or another hash algorithm from the SHA-2 family) as in my example below, you may have a different problem.

oracle@omshost:~> openssl s_client -prexit -connect omshost.domain.com:7103 /dev/null | openssl x509 -text -in /dev/stdin | grep "Signature Algorithm" 2> /dev/null
        Signature Algorithm: sha256WithRSAEncryption
    Signature Algorithm: sha256WithRSAEncryption

Workaround

To work around this issue, you need to (temporarily!) replace the certificate on your WLS adminserver. Now, whenever I need to apply a PSU release, I resecure WLS using the default demonstration certificate, apply the PSU, then replace my original certificate.

oracle@omshost:/oracle/stage/21603255> emctl secure wls -use_demo_cert
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Securing WLS... Started.
Enter Enterprise Manager Root (SYSMAN) Password :
Securing WLS... Successful
Restart OMS using 'emctl stop oms -all' and 'emctl start oms'
oracle@omshost:/oracle/stage/21603255> emctl stop oms -all ; sleep 5 ; emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
Stopping BI Publisher Server...
BI Publisher Server Successfully Stopped
AdminServer Successfully Stopped
BI Publisher Server is Down
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
Starting BI Publisher Server ...
BI Publisher Server Successfully Started
BI Publisher Server is Up

[install the PSU according to the README instructions, including any post-installation steps]

oracle@omshost:/oracle/stage/21603255> emctl secure wls -wallet /oracle/oem/oemwallet
Oracle Enterprise Manager Cloud Control 12c Release 4  
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Securing WLS... Started.
Enter Enterprise Manager Root (SYSMAN) Password : 
Securing WLS... Successful
Restart OMS using 'emctl stop oms -all' and 'emctl start oms'
If there are multiple OMSs in this environment, perform this configuration on all of them.
oracle@omshost:/oracle/stage/21603255> emctl stop oms -all ; sleep 5 ; emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
Stopping BI Publisher Server...
BI Publisher Server Successfully Stopped
AdminServer Successfully Stopped
BI Publisher Server is Down
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
Starting BI Publisher Server ...
BI Publisher Server Successfully Started
BI Publisher Server is Up

I have not noticed any other EM12c issues using SHA256-hashed certificates. With this workaround, you can both continue to use quality certificates and keep your OMS patched.

EM12c OHS, LOW strength ciphers, custom certificates, and patch 19948000 weirdness

This post documents an unusual issue I encountered with the Oracle HTTP Server (OHS) installation in my Oracle Enterprise Manager 12c R4 (12.1.0.4) environment after following MOS note 1984662.1 and applying patch 19948000 (CPUJAN2015) to my OHS home.  It also contains a workaround I found that you should consider UNSUPPORTED, UNOFFICIAL, and NOT RECOMMENDED, only for use if absolutely necessary to meet auditor requirements.  If you do not have to follow the steps I describe below, I suggest waiting for new patches and further guidance from Oracle Support. If this change breaks your system and eats all the food in the break room refrigerator, I warned you not to do it.

Like other security-conscious EM12c admins, I want to keep my installation secure, and so I watch closely when security patches become available for EM12c or its various components. Thus, when I noticed patch 19948000’s availability for OHS, which disables SSLv3, I installed it on my system, and confirmed through testing that OHS no longer permitted SSLv3 connections (test for yourself with: openssl s_client -connect host.domain.com:port -ssl3, or try my EM12c SSL security checkup script that I have blogged about previously).

As I proceeded with further hardening of my EM12c system, specifically an attempt to disable LOW and MEDIUM strength cipher suite usage as per MOS note 1477287.1, I noticed that after following the directions provided, all of my EM12c endpoints correctly rejected LOW and MEDIUM strength ciphers, with one exception.  The OMS HTTPS upload port, inexplicably, continued to permit LOW strength connections. It refused MEDIUM strength ciphers, but had no problem accepting a LOW strength DES-CBC-SHA connection over TLSv1:

$ openssl s_client -connect omshost.domain.com:4902 -cipher LOW
[...]
SSL handshake has read 4109 bytes and written 385 bytes
---
New, TLSv1/SSLv3, Cipher is DES-CBC-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DES-CBC-SHA
    Session-ID: 37BF30668DCAD2CC5D0BAC4142CC1FA1
    Session-ID-ctx:
    Master-Key: [redacted]
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1429290250
    Timeout   : 300 (sec)
---

This confused me greatly, as I had edited all configuration files as instructed, and none of my other OHS listen ports accepted this LOW strength cipher connection.  I spent quite a bit of time trying to diagnose and resolve the issue with no luck, until I eventually stumbled upon an odd fix.  If I remove or comment out the “IfDefine SSL” directives from my $GC_INSTANCE_HOME/WebTierIH1/config/OHS/ohs1/httpd_em.conf file, then suddenly OHS would refuse LOW strength cipher connections on this port, with no apparent ill effect on the other listening ports.

$ openssl s_client -connect omshost.domain.com:4902 -cipher LOW 
CONNECTED(00000003)
2282780:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:769:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 67 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
---

I have noted these IfDefine SSL directives with “HERE” in the excerpt below from my httpd_em.conf file.

##
## CAUTION: Edit only the .template version of this file!
##
##     The command
##         emctl secure [lock|unlock]
##     will replace httpd_em.conf (discarding your changes) 
##     using the httpd_em.conf.template file.
##
## This file contains virtual hosts and other directives
## required for the "Enterprise Manager Central Console"
## to function correctly.
##

#UseWebCacheIp On

<IfDefine SSL>      #### HERE
    Listen 4902
    <VirtualHost *:4902>
        <Location /empbs/upload>
            Order allow,deny
            Allow from all
        </Location>
        <Location /empbs/jobrecv>
            Order allow,deny
            Allow from all
        </Location>
        <Location /em>
            Order allow,deny
            Allow from all
        </Location>
        <Location /agent_download>
            Order allow,deny
            Allow from all
        </Location>
        <Location /xmlpserver>
            Order allow,deny
            Allow from all
        </Location>

        #DocumentRoot &ORACLE_HOME&/Apache/Apache/htdocs
        ServerName omshost.domain.com
        #Port 4902
        Timeout 900

        LogFormat "%h %l %u %t \"%r\" %>s %b [ecid: %{ECID-Context}i] [User-Agent: %{User-Agent}i]" common
        SetEnvIf Request_URI "\.(bmp|jpg|png|gif|css|js$)" no-log
        SetEnvIf Request_URI "/em/dynamicImage/*"  no-log
        CustomLog "|${ORACLE_HOME}/ohs/bin/odl_rotatelogs /oracle/oem/gc_inst1/WebTierIH1/diagnostics/logs/OHS/ohs1/em_upload_https_access_log 10M 100M" common env=!no-log

        ErrorLog "|${ORACLE_HOME}/ohs/bin/odl_rotatelogs /oracle/oem/gc_inst1/WebTierIH1/diagnostics/logs/OHS/ohs1/em_upload_https_error_log 10M 100M"
        SSLEngine on
        SSLCipherSuite HIGH
        SSLWallet file:/oracle/oem/gc_inst1/WebTierIH1/config/OHS/ohs1/keystores/upload
        SSLProtocol TLSv1

        <Files ~ "\.(cgi|shtml)$">
            SSLOptions +StdEnvVars
        </Files>
        #<Directory &ORACLE_HOME&/Apache/Apache/cgi-bin>
        #    SSLOptions +StdEnvVars
        #</Directory>
        SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown
    </VirtualHost>
</IfDefine>     #### HERE
[remainder of file removed]

If I leave the IfDefine SSL statements in there, my OMS upload port accepts the weak DES-CBC-SHA cipher along with HIGH strength ciphers.  If I remove the IfDefine SSL, my OMS upload port refuses DES-CBC-SHA along with all other LOW/MEDIUM strength ciphers.

This makes no sense, given what I know of OHS and Apache-like products and the way that the handle the SSLCipherSuite configuration directive.

I raised this issue on Twitter and heard back from Andrew Bulloch at Oracle, who graciously spent quite a bit of time attempting to reproduce the issue on his side and working with me to identify the situations in which this behavior occurs.  After much testing, it appears that this behavior only occurs in the following situation:

  1. The OHS installed with EM12c R4 has patch 19948000 installed, AND
  2. The administrator has installed a third party SSL certificate, replacing the demo certificate used by default, AND
  3. The OHS httpd_em.conf contains the “IfDefine SSL” directive.

If I remove my custom certificate, returning the OMS to the demo certificate, the issue disappears, then returns if I reinstall the custom certificate.

If I remove patch 19948000, the issue disappears, and does not return whether I use a custom certificate or a demo certificate.

If I remove the IfDefine SSL directive, the issue disappears, and does not return whether I use a custom certificate, a demo certificate, or whether or not I have patch 19948000 installed.

I attempted to replicate this behavior with an SSL certificate that did not come from a true certificate authority, by using OpenSSL to create a CA, create a cert, sign it, then install it into OHS per the documentation in MOS note 1399293.1, but I could not reproduce it, possibly due to the fact that I used a certificate signed directly by a root CA (as with the demo certificate) instead of a certificate signed by an intermediate chain certificate signed by a root CA, as with the paid-for commercial certificate that revealed the issue. I have not had a chance to test that configuration.

Unfortunately, removing patch 19948000 means that OHS cannot refuse SSLv3 connections, and removing the custom certificate reverts the system back to the demo certificate that I do not wish to use, both of which will represent audit findings in regulated sites.

Due to this issue, I have edited my EM12c security checkup script to remove my recommendation to install patch 19948000, although I still have it installed.  For security reasons, I will leave my system in the workaround state I have described here, as I want SSLv3 disabled, and I want LOW strength cipher suites disabled, and I want to use a custom SSL certificate, but I accept the risk that I may have to undo this setup at any time to receive support or to successfully apply later patches.  You will have to make your own decisions based on your site’s audit requirements and the availability of personnel to validate your configuration and handle future patching.

I would be very interested if anyone else reading this has encountered this issue, as I do not know if my installation somehow uniquely surfaces this behavior or if the certificate vendor that we used has some strange settings on their certificates that cause confusion for OHS.

When proactive EM12c JDK upgrades bite back

You probably will not encounter this issue, but I will post this anyway to get the error message and resolution indexed by Google.

While attempting to apply patch 19513382 (EM agent bundle patch 12.1.0.4.3) to my EM12cR4 agents, I ran into multiple problems.  Initially it would not apply to any of my agents.  Bug 20134182 and the resolution described in MOS note 1952355.1 resolved the first problem (OPatch reporting that identical patches 18721761 and 18502187 already exist), but that left me with one agent I could not upgrade. Attempts to run patch plan validation within EM12c produced the following error:

PatchList : 19513382
PatchLocList : /tmp/p19513382_600000000009641_2000_0/oraagent
TargetName : [redacted]:[port]
----------------------------------------
[11_12_2014_10_00_40] Command Arguments:
/oraagent/agent12c/core/12.1.0.4.0/OPatch/opatch checkComponents -phbasedir /tmp/p19513382_600000000009641_2000_0/oraagent/19513382 -oh /oraagent/agent12c/core/12.1.0.4.0 -invPtrloc /oraagent/agent12c/core/12.1.0.4.0/oraInst.loc
 
OPatch cannot continue because it would not be able to load OUI platform dependent library from the directory "/oraagent/agent12c/core/12.1.0.4.0/oui/lib/linux64". The directory does not exist in the Oracle home.
This could be due to the following reasons.

(1) Incompatible usage of java with OUI (32/64 bit).
(2) Wrong 32-bit Oracle Home installation in 64-bit environment (or) vice-versa.
Please contact Oracle support for more details.
 
OPatch failed with error code 1
 
PREREQ_CONTEXT_HOST_NAME:[redacted]
REREQ_CONTEXT_HOME_LOCATION:/oraagent/agent12c/core/12.1.0.4.0
PREREQ_NAME: Checking if the patches are applicable.
PREREQ_DESC: Checking if the patches are applicable on the Management Agent.
PREREQ_TYPE:APPLICABILITY
PREREQ_STATUS:FAILED
PREREQ_MESG: None of these patches are applicable on the Management Agent.
PREREQ_MESG_PATCH:19513382
PREREQ_REMEDY:MANUAL
PREREQ_REMEDY_DETAILS: Remove patch(es) 19513382 from this patch plan.

I already know from the previously referenced MOS note that OPatch 11.1.0.12.3 contains bugs, so as a first debugging step I attempted to rollback the OPatch upgrade by restoring the backup copies of OPatch found in $AGENT_HOME/OPatch/backup/.  I received the same error message with OPatch 11.1.0.10.4 and 11.1.0.11.0.  I also received a similar error even if I simply tried to run “opatch lsinv” from the command line with the older versions. So OPatch did not cause this issue.

Since the error message mentions 32-bit and 64-bit incompatibility, I needed to consider the environment.  This server runs Linux x86-64 (SLES 10 SP4), but must use a 32-bit EM agent based on the certification matrix and MOS note 1488161.1. I next checked to find my last successful patch run, which happened only a month ago, so a recent change has to have caused this problem. Going through my notes, the only recent change on this server involved upgrading the JDK used by the EM agent per MOS note 1944044.1.

Luckily I still had the old JDK available for comparison.

> java -version
java version "1.6.0_43"
Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
Java HotSpot(TM) Server VM (build 20.14-b01, mixed mode)
> file `which java`
/oraagent/agent12c/core/12.1.0.4.0/jdk/bin/java: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), not stripped

Looking at the new JDK:

> ./java -version
java version "1.6.0_85"
Java(TM) SE Runtime Environment (build 1.6.0_85-b13)
Java HotSpot(TM) 64-Bit Server VM (build 20.85-b01, mixed mode)
> file java
java: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), for GNU/Linux 2.4.0, dynamically linked (uses shared libs), not stripped

There I have my problem. In upgrading the JDK, I had installed the 64-bit version of Java 1.6u85, not the 32-bit version, based on the fact that the server runs a 64-bit OS. I had not considered that a 64-bit JDK would not remain compatible with the 32-bit agent on this 64-bit system.

Surprisingly, everything about the agent seems to have worked fine, despite the 64-bit JDK.  Nothing broke until I attempted to use OPatch.

To resolve the issue, I stopped the agent and moved the original 32 bit 1.6u43 JDK back to where it belongs, followed note 1952355.1 to work around the known bugs when using OPatch 11.1.0.12.3 to apply 19513382, then successfully applied the patch.  After that I downloaded the correct 32-bit version of the 1.6u85 JDK, installed it per 1944044.1, and now OPatch works as expected.

Stale EM12c patch recommendations? Get patch 14822626

I don’t use the automated patching functionality provided by EM12c.  I do, however, get value out of the patch recommendations since they serve as a good reminder when I’ve missed a patch that should be applied to one of my targets.  For this reason I was disappointed when, after upgrading from EM12cR1 to EM12cR2, the patch recommendations it gave me became stale and stopped getting updated when I loaded the em_catalog.zip file.

If you DO use the automated patching functionality, you have probably already followed all of the advice and installed the required patches documented in MOS note 427577.1, “Enterprise Manager patches required for setting up Provisioning, Patching and Cloning (Deployment Procedures)”.  In that case you already have this patch installed and don’t need to read any further, but if not, read on.

After upgrading to EM12cR2, I also upgraded several databases from 10gR2 to 11gR2.  Months passed, and yet the patch recommendations EM12c gave me continued to refer to 10gR2 patches which I knew weren’t applicable as I was running 11gR2.  I tried several things, like setting EM12c to offline mode, to online mode, loading em_catalog.zip, re-running the various “Refresh From My Oracle Support” jobs, all without ever receiving fresh patch recommendations.

So to sort this out, I did what I usually do, and asked about it on Twitter.  Big thanks to Sudip Datta, Vice President of Product Management at Oracle, who pointed me to bug 14822626 and its associated patch.  The bug does not appear to be public, but MOS note 1522918.1, “12C – Patch Recommendations Not Updating After Upgrade To 12.1.0.2 Cloud Control – ‘…Patch Recommendations Computation is disabled … skipping …'” documents the problem as a known issue after upgrading from 12.1.0.1 to 12.1.0.2 and clearly matches the behavior I saw.

As soon as I applied patch 14822626, the old stale patch recommendations were cleared out, and once I loaded the current em_catalog.zip file, I had accurate patch recommendations for my environment that I can now use to make decisions going forward.

Thank you, Sudip!