Step by step: Configuring third party SSL/TLS certificates in EM12c R4

[EDIT 20170227: The process for configuring third party certificates for EM13c works about the same as for EM12c. If you have access to Oracle support, I suggest you review notes 2220788.1 and 2213661.1 for the most up-to-date documentation directly from Oracle.]

By default, when an administrator configures Oracle Enterprise Manager 12c to use SSL, the system will use a default self-signed certificate, provided for demo purposes only.  The documentation states repeatedly that users should not use these certificates in a production environment, as they represent a security risk. This blog post documents, step by step, a process to replace these demo certificates with custom third party certificates, across the OMS console, OMS upload port, agents, and WebLogic Server. I will follow this process on a single-OMS configuration; if you have more than one OMS please consult the documentation for more details, as your process will vary and the steps I have provided may break your system.

I have tested these instructions on Linux x86-64 (SLES11 SP3) with EM12c R4 PSU2 (12.1.0.4).

Official Documentation

The official documentation for this process resides in the following My Oracle Support notes:

• Using ORAPKI Utility to Create a Wallet with Third Party Trusted Certificate and Import into OMS (Doc ID 1367988.1)
• EM 12c Cloud Control How to Create a Wallet With Third Party Trusted Certificate that Can Be Imported into the OMS For SSL Comunication ? (Doc ID 1399293.1)
• 12c Cloud Control: Steps to Import Third Party Trusted SSL Certificate into 12c Cloud Control Agent URL (Doc ID 1593183.1)
• 12c Cloud Control: Steps to Create and Import Third Party / Self-Signed SSL Certificates for WebLogic Server in an Enterprise Manager Installation (Doc ID 1527874.1)
• How to Create a Java Keystore via Keytool in FMW 11g/12c (Doc ID 1230333.1)

Why Should I Do This?

You may not fully understand the mechanics of SSL/TLS certificates and the chain of trust. I cannot fully explain this complex topic in a blog post, but if you need a reason to make this change other than demands from your organizational security/compliance team, please take Oracle’s word for it, and notice the text that appears in your GCDomain.log file when you run your system with the provided default demo certificates.

Read that again if you didn’t catch it the first time through: “The system is vulnerable to security attacks, since it trusts certificates signed by the demo trusted CA.” This text comes from code in WebLogic, not from me. Here Oracle tells you very explicitly that your system currently contains a severe vulnerability.

You will also notice that when using the EM12c console, or accessing an agent URL, or accessing the WebLogic Server administration console may show warnings in your browser about untrusted certificates. Once you replace your certificates as described in the documentation above or my steps below, you will no longer have those issues.

Using 3rd Party SSL/TLS Certificates With EM12c

Overview

You will follow 7 high level steps to complete the process of securing your EM12c environment with custom third party SSL/TLS certificates.

1. Create an Oracle wallet for the OMS.
2. Secure the OMS console using the OMS wallet.
3. Secure the OMS upload port using the OMS wallet.
4. Re-secure all agents.
5. Create Oracle wallets for agents.
6. Configure the agents to use their wallets.
7. Secure WebLogic with the OMS wallet.

Create an Oracle wallet for the OMS

First we follow steps 1a through 1h from document 1367988.1.  All these steps occur on the OMS host.

Disable shell history (optional but recommended)

While following these steps, you will repeatedly have to type passphrases on the command line. To avoid having these passphrases stored in your Oracle user’s shell history, disable history saving.  In the bash shell that I use, I accomplish this by unsetting the HISTFILE variable. You may need to use another mechanism in another shell.

$ unset HISTFILE

Use the correct ORAPKI command

You should use the ORAPKI command from your middleware home’s oracle_common/bin directory.  I will refer to this as $MW_HOME/oracle_common/bin/orapki in the following instructions.

Create an Oracle wallet

The documentation specified that we should create an auto-login wallet, but in my single-OMS setup, I believe that I will achieve better security with an auto-login-local wallet, as the auto-login feature will only function on this specific host. You will need to select a base directory for your OMS wallet.  I used $ORACLE_BASE/oemwallet. ORAPKI will prompt you for a password. Use a secure one, and note it down somewhere safe. You will use it many times during this process.

$ mkdir $ORACLE_BASE/oemwallet
$ $MW_HOME/oracle_common/bin/orapki wallet create -wallet $ORACLE_BASE/oemwallet -auto_login_local
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Enter password:
 Enter password again:

Get in the habit of displaying the wallet contents after each operation to confirm that everything worked.

$ $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/oemwallet
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Requested Certificates:
 User Certificates:
 Trusted Certificates:
 Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
 Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US

Create a key within the wallet.  Make sure you replace omshost.domain.com with the fully qualified domain name of your OMS host. I highly recommend using a 2048 bit keysize, as shown below. Include the wallet password you specified earlier on the commandline as the -pwd argument, contained in single quotes. Display the wallet again afterward.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -dn "CN=omshost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US" -keysize 2048 -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

$ $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/oemwallet
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Requested Certificates:
 Subject: CN=omshost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US
 User Certificates:
 Trusted Certificates:
 Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US

Export a certificate signing request based on this key. Make sure the -dn you specify exactly matches the -dn specified earlier. Provide a filename in the -request argument in which to store the certificate signing request (CSR).

$ $MW_HOME/oracle_common/bin/orapki wallet export -wallet $ORACLE_BASE/oemwallet -dn "CN=omshost.domain.com, OU=EM,O=Organization,L=City,ST=State,C=US" -request ~/EM12cCSR.txt
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

Submit this CSR file to your signing authority. Inform them that you MUST have a single-host certificate with your OMS host’s fully qualified domain name in the CN field. Subject Alternate Name (SAN) certificates or wildcard certificates will not work at all. Your signing authority should then provide you with a root certificate, an intermediate certificate, and a user certificate.

Import the root, intermediate, and user certificates into the OMS wallet. Note that you must import the root and intermediate certificates using -trusted_cert, and the user certificate using -user_cert.  I used DigiCert, and I can confirm that their certificates function correctly in EM12c and recommend their service.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -trusted_cert -cert ~/TrustedRoot.cer -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -trusted_cert -cert ~/DigiCertCA2.cer -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -user_cert -cert ~/omshost.domain.com.cer -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

Display the wallet contents after this operation.

$  $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/oemwallet
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Requested Certificates:
 User Certificates:
 Subject: CN=omshost.domain.com,OU=[REDACTED],O=[REDACTED],L=City,ST=State,C=US
 Trusted Certificates:
 Subject: CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US
 Subject: CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US
 Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
 Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US

Concatenate the root and intermediate certificates into a single file, which you will use later.

$  cat ~/DigiCertCA2.cer ~/TrustedRoot.cer > $ORACLE_BASE/trusted_certs.txt

You have completed configuration of your OMS wallet.

Secure the OMS console

Now, using emctl from the $OMS_HOME, tell EM12c to secure the OMS console using the certificate contained in your wallet. The system will prompt you for the SYSMAN password and inform you to restart the entire OMS once complete.

$ $OMS_HOME/bin/emctl secure console -wallet /oracle/oem/oemwallet
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Securing Console... Started.
 Enter Enterprise Manager Root (SYSMAN) Password :
 Securing Console... Successful
 Restart OMS
$ $OMS_HOME/bin/emctl stop oms -all ; sleep 5 ; $OMS_HOME/bin/emctl start oms

Now access your OMS console with your favorite browser and confirm that your new certificate appears.  Your certificate should show a trusted path back to a root certificate, and your browser should produce no warnings.

At this point, you have secured communication between your browser and the EM12c OMS console with your custom certificate.  You still have more work to do though. Your agents upload monitoring data to the OMS upload port, and it still uses the demo certificate. Fix that in the next step.

Secure the OMS upload port

Secure the OMS upload port. Expect to receive email or pager alerts after this step, as once you restart the OMS, none of your agents can communicate with it, as they expect to see the demo certificates on the upload port. You will need to provide the SYSMAN password as well as an agent registration password.

$ $OMS_HOME/bin/emctl secure oms -wallet $ORACLE_BASE/oemwallet -trust_certs_loc $ORACLE_BASE/trusted_certs.txt
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Securing OMS... Started.
 Enter Enterprise Manager Root (SYSMAN) Password :
 Enter Agent Registration Password :
 Securing OMS... Successful
 Restart OMS
$ $OMS_HOME/bin/emctl stop oms -all ; sleep 5 ; $OMS_HOME/bin/emctl start oms

Re-secure all agents

Now you must re-secure all of your agents so that they can resume uploading data to the OMS console and monitoring your systems. Execute the following steps on every agent, using emctl from the agent home.  You will need to provide an agent registration password to complete this process.

$ $AGENT_HOME/bin/emctl secure agent
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Agent successfully stopped... Done.
Securing agent... Started.
Enter Agent Registration Password : 
Agent successfully restarted... Done.
Securing agent... Successful.
$ $AGENT_HOME/bin/emctl upload agent
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

It may take a little while for the OMS to process the new agents and their uploads, but once you have run this process on every agent they should all communicate successfully with the OMS and appear as OK from the agent management screen.

Create Oracle wallets for agents

Next we secure the agent URLs. The OMS connects to the agents at this URL to submit management requests. At the moment, the agents still use self-signed certificates to secure this URL.  For this process we create an Oracle wallet, on the OMS host, using the same ORAPKI command as for the OMS wallet. We will generate a certificate signing request from each agent wallet, submit those CSRs to a certificate authority, and import the received certificates.

As with the OMS, the agents must use single-host certificates, not wildcard or subject alternate name (SAN) certificates.  To determine the correct fully qualified domain name for each agent, execute emctl status agent from the agent home.

$  $AGENT_HOME/bin/emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 12.1.0.4.0
OMS Version : 12.1.0.4.0
Protocol Version : 12.1.0.1.0
Agent Home : /oraagent/agent12c/agent_inst
Agent Log Directory : /oraagent/agent12c/agent_inst/sysman/log
Agent Binaries : /oraagent/agent12c/core/12.1.0.4.0
Agent Process ID : 12480
Parent Process ID : 12359
Agent URL : https://agenthost.domain.com:3872/emd/main/

Repeat these steps for every agent.

Create a directory to store the agent wallet, and an agent wallet. This time do NOT use -auto_login_local, use only -auto_login, as you will distribute these wallets to the agent hosts after generating them on the OMS host.  Use a strong password, and save it for later, as you will reuse it many times.

$ mkdir $ORACLE_BASE/agentwallets
$ mkdir $ORACLE_BASE/agentwallets/agenthost.domain.com
$ $MW_HOME/oracle_common/bin/orapki wallet create -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -auto_login
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Enter password:
 Enter password again:

Create the certificate, then a certificate signing request, saving it to file ~/agenthost.domain.com.csr. Again, I recommend a 2048 bit certificate.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -dn "CN=agenthost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US" -keysize 2048 -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
$ $MW_HOME/oracle_common/bin/orapki wallet export -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -dn "CN=agenthost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US" -request ~/agenthost.domain.com.csr
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

As before, submit this certificate signing request to your certificate authority, and receive back three files containing a root certificate, an intermediate certificate, and a user certificate. Import these into the agent wallet, and display the wallet afterwards to confirm everything imported successfully.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -trusted_cert -cert ~/TrustedRoot.crt -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -trusted_cert -cert ~/DigiCertCA.crt -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -user_cert -cert ~/agenthost.domain.com.crt -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
$ $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Requested Certificates:
 User Certificates:
 Subject: CN=agenthost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US
 Trusted Certificates:
 Subject: CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US
 Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
 Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=DigiCert High Assurance CA-3,OU=www.digicert.com,O=DigiCert Inc,C=US

You have finished creating this agent’s wallet.  Repeat this for every agent.

Configure the agents to use their wallets

Inside the agent wallets you’ve just created, you will find a cwallet.sso file. Take this file from each agent’s wallet and copy it to the agent server. Stop the agent, then place the file into $AGENT_INSTANCE_DIR/sysman/config/server/ and set the permissions to 640, then start the agent.

$ $AGENT_HOME/bin/emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Stopping agent ..... stopped.
$ cp cwallet.sso $AGENT_INSTANCE_DIR/sysman/config/server
$ chmod 640 $AGENT_INSTANCE_DIR/sysman/config/server
$ $AGENT_HOME/bin/emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Starting agent ............. started.

Next, visit the agent URL in your favorite web browser and examine the certificate it uses.  You should now see that it uses the 3rd party SSL/TLS certificate that you installed.

Secure WebLogic with the OMS wallet

Now the OMS (both console and upload ports) and agents will use your new certificates. This leaves WebLogic as the one remaining component needing your new certificates. Please note in following the below directions that securing WebLogic with a wallet only works as of EM12c R3, earlier versions must use a Java keystore. See note 1527874.1 for more information.

[NOTE: 20150910: If you secure WebLogic with a certificate that uses the SHA256 hashing algorithm, future attempts to apply EM12c PSU patches using ‘opatchauto’ will fail. Some piece of opatchauto does not support SHA256 usage in certificates. If you run into this issue, revert your WLS to the demonstration certificate using emctl secure wls -use_demo_cert, then apply the PSU, then resecure WLS using these steps with your desired certificate. I intend to write a full blog post about this later.]

First import the root and intermediate certificates to the keystore on the OMS host’s agent. Use the default password welcome for the agent keystore, and alias names rootcacert and intercacert.

$ $AGENT_HOME/bin/emctl secure add_trust_cert_to_jks -trust_certs_loc ~/TrustedRoot. crt -alias rootcacert -password welcome
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.

 Message : Certificate was added to keystore
 ExitStatus: SUCCESS
$ $AGENT_HOME/bin/emctl secure add_trust_cert_to_jks -trust_certs_loc ~/DigiCertCA. crt -alias intercacert -password welcome
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.

 Message : Certificate was added to keystore
 ExitStatus: SUCCESS

Back up some WLS configuration files, just in case, before securing WLS with your certificate.  If you have problems in this step, make sure you have stopped all WLS processes, then restore these files from backup.

$ mkdir ~/wlscertbak
$ cp -a $EM_INSTANCE_BASE/em/EMGC_OMS1/emgc.properties ~/wlscertbak/
$ cp -a $EM_INSTANCE_BASE/NodeManager/emnodemanager/nodemanager.properties ~/wlscertbak/
$ cp -a $EM_INSTANCE_BASE/WebTierIH1/config/OHS/ohs1/keystores/proxy ~/wlscertbak/
$ cp -a $EM_INSTANCE_BASE/user_projects/domains/GCDomain/config/config.xml ~/wlscertbak/

Stop the OMS.

$ $OMS_HOME/bin/emctl stop oms
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Stopping WebTier...
 WebTier Successfully Stopped
 Stopping Oracle Management Server...
 Oracle Management Server Successfully Stopped
 Oracle Management Server is Down

Secure WLS using the OMS wallet created earlier. You will need to provide the SYSMAN password.

$ $OMS_HOME/bin/emctl secure wls -wallet $ORACLE_BASE/oemwallet
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Securing WLS... Started.
 Enter Enterprise Manager Root (SYSMAN) Password :
 Securing WLS... Successful
 Restart OMS using 'emctl stop oms -all' and 'emctl start oms'
 If there are multiple OMSs in this environment, perform this configuration on all of them.

Stop the entire WLS stack, then start the OMS.

$ $OMS_HOME/bin/emctl stop oms -all
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Stopping WebTier...
 WebTier Successfully Stopped
 Stopping Oracle Management Server...
 Oracle Management Server Already Stopped
 Oracle Management Server is Down
 Stopping BI Publisher Server...
 BI Publisher Server Successfully Stopped
 AdminServer Successfully Stopped
 BI Publisher Server is Down
$ $OMS_HOME/bin/emctl start oms
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Starting Oracle Management Server...
 Starting WebTier...
 WebTier Successfully Started
 Oracle Management Server Successfully Started
 Oracle Management Server is Up
 Starting BI Publisher Server ...
 BI Publisher Server Successfully Started
 BI Publisher Server is Up

Visit the WebLogic Server administration console and you should now see that it presents your custom SSL/TLS certificate and no longer uses the demo certificate.

Conclusion

If you have successfully followed these steps, your system should now use your custom SSL/TLS certificates everywhere, and you should no longer experience certificate warnings in your browsers.

See Also

• EM12c R4 SSL Security Checkup Script