[EDIT 20170227: The process for configuring third party certificates for EM13c works about the same as for EM12c. If you have access to Oracle support, I suggest you review notes 2220788.1 and 2213661.1 for the most up-to-date documentation directly from Oracle.]
By default, when an administrator configures Oracle Enterprise Manager 12c to use SSL, the system will use a default self-signed certificate, provided for demo purposes only. The documentation states repeatedly that users should not use these certificates in a production environment, as they represent a security risk. This blog post documents, step by step, a process to replace these demo certificates with custom third party certificates, across the OMS console, OMS upload port, agents, and WebLogic Server. I will follow this process on a single-OMS configuration; if you have more than one OMS please consult the documentation for more details, as your process will vary and the steps I have provided may break your system.
I have tested these instructions on Linux x86-64 (SLES11 SP3) with EM12c R4 PSU2 (12.1.0.4).
Official Documentation
The official documentation for this process resides in the following My Oracle Support notes:
- Using ORAPKI Utility to Create a Wallet with Third Party Trusted Certificate and Import into OMS (Doc ID 1367988.1)
- EM 12c Cloud Control How to Create a Wallet With Third Party Trusted Certificate that Can Be Imported into the OMS For SSL Comunication ? (Doc ID 1399293.1)
- 12c Cloud Control: Steps to Import Third Party Trusted SSL Certificate into 12c Cloud Control Agent URL (Doc ID 1593183.1)
- 12c Cloud Control: Steps to Create and Import Third Party / Self-Signed SSL Certificates for WebLogic Server in an Enterprise Manager Installation (Doc ID 1527874.1)
- How to Create a Java Keystore via Keytool in FMW 11g/12c (Doc ID 1230333.1)
Why Should I Do This?
You may not fully understand the mechanics of SSL/TLS certificates and the chain of trust. I cannot fully explain this complex topic in a blog post, but if you need a reason to make this change other than demands from your organizational security/compliance team, please take Oracle’s word for it, and notice the text that appears in your GCDomain.log file when you run your system with the provided default demo certificates.
Read that again if you didn’t catch it the first time through: “The system is vulnerable to security attacks, since it trusts certificates signed by the demo trusted CA.” This text comes from code in WebLogic, not from me. Here Oracle tells you very explicitly that your system currently contains a severe vulnerability.
You will also notice that when using the EM12c console, or accessing an agent URL, or accessing the WebLogic Server administration console may show warnings in your browser about untrusted certificates. Once you replace your certificates as described in the documentation above or my steps below, you will no longer have those issues.
Using 3rd Party SSL/TLS Certificates With EM12c
Overview
You will follow 7 high level steps to complete the process of securing your EM12c environment with custom third party SSL/TLS certificates.
- Create an Oracle wallet for the OMS.
- Secure the OMS console using the OMS wallet.
- Secure the OMS upload port using the OMS wallet.
- Re-secure all agents.
- Create Oracle wallets for agents.
- Configure the agents to use their wallets.
- Secure WebLogic with the OMS wallet.
Create an Oracle wallet for the OMS
First we follow steps 1a through 1h from document 1367988.1. All these steps occur on the OMS host.
Disable shell history (optional but recommended)
While following these steps, you will repeatedly have to type passphrases on the command line. To avoid having these passphrases stored in your Oracle user’s shell history, disable history saving. In the bash shell that I use, I accomplish this by unsetting the HISTFILE variable. You may need to use another mechanism in another shell.
$ unset HISTFILE
Use the correct ORAPKI command
You should use the ORAPKI command from your middleware home’s oracle_common/bin directory. I will refer to this as $MW_HOME/oracle_common/bin/orapki in the following instructions.
Create an Oracle wallet
The documentation specified that we should create an auto-login wallet, but in my single-OMS setup, I believe that I will achieve better security with an auto-login-local wallet, as the auto-login feature will only function on this specific host. You will need to select a base directory for your OMS wallet. I used $ORACLE_BASE/oemwallet. ORAPKI will prompt you for a password. Use a secure one, and note it down somewhere safe. You will use it many times during this process.
$ mkdir $ORACLE_BASE/oemwallet $ $MW_HOME/oracle_common/bin/orapki wallet create -wallet $ORACLE_BASE/oemwallet -auto_login_local Oracle PKI Tool : Version 11.1.1.7.0 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved. Enter password: Enter password again:
Get in the habit of displaying the wallet contents after each operation to confirm that everything worked.
$ $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/oemwallet Oracle PKI Tool : Version 11.1.1.7.0 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved. Requested Certificates: User Certificates: Trusted Certificates: Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
Create a key within the wallet. Make sure you replace omshost.domain.com with the fully qualified domain name of your OMS host. I highly recommend using a 2048 bit keysize, as shown below. Include the wallet password you specified earlier on the commandline as the -pwd argument, contained in single quotes. Display the wallet again afterward.
$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -dn "CN=omshost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US" -keysize 2048 -pwd '[REDACTED]' Oracle PKI Tool : Version 11.1.1.7.0 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved. $ $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/oemwallet Oracle PKI Tool : Version 11.1.1.7.0 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved. Requested Certificates: Subject: CN=omshost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US User Certificates: Trusted Certificates: Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
Export a certificate signing request based on this key. Make sure the -dn you specify exactly matches the -dn specified earlier. Provide a filename in the -request argument in which to store the certificate signing request (CSR).
$ $MW_HOME/oracle_common/bin/orapki wallet export -wallet $ORACLE_BASE/oemwallet -dn "CN=omshost.domain.com, OU=EM,O=Organization,L=City,ST=State,C=US" -request ~/EM12cCSR.txt Oracle PKI Tool : Version 11.1.1.7.0 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
Submit this CSR file to your signing authority. Inform them that you MUST have a single-host certificate with your OMS host’s fully qualified domain name in the CN field. Subject Alternate Name (SAN) certificates or wildcard certificates will not work at all. Your signing authority should then provide you with a root certificate, an intermediate certificate, and a user certificate.
Import the root, intermediate, and user certificates into the OMS wallet. Note that you must import the root and intermediate certificates using -trusted_cert, and the user certificate using -user_cert. I used DigiCert, and I can confirm that their certificates function correctly in EM12c and recommend their service.
$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -trusted_cert -cert ~/TrustedRoot.cer -pwd '[REDACTED]' Oracle PKI Tool : Version 11.1.1.7.0 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved. $ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -trusted_cert -cert ~/DigiCertCA2.cer -pwd '[REDACTED]' Oracle PKI Tool : Version 11.1.1.7.0 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved. $ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -user_cert -cert ~/omshost.domain.com.cer -pwd '[REDACTED]' Oracle PKI Tool : Version 11.1.1.7.0 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
Display the wallet contents after this operation.
$ $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/oemwallet Oracle PKI Tool : Version 11.1.1.7.0 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved. Requested Certificates: User Certificates: Subject: CN=omshost.domain.com,OU=[REDACTED],O=[REDACTED],L=City,ST=State,C=US Trusted Certificates: Subject: CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US Subject: CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
Concatenate the root and intermediate certificates into a single file, which you will use later.
$ cat ~/DigiCertCA2.cer ~/TrustedRoot.cer > $ORACLE_BASE/trusted_certs.txt
You have completed configuration of your OMS wallet.
Secure the OMS console
Now, using emctl from the $OMS_HOME, tell EM12c to secure the OMS console using the certificate contained in your wallet. The system will prompt you for the SYSMAN password and inform you to restart the entire OMS once complete.
$ $OMS_HOME/bin/emctl secure console -wallet /oracle/oem/oemwallet Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Securing Console... Started. Enter Enterprise Manager Root (SYSMAN) Password : Securing Console... Successful Restart OMS $ $OMS_HOME/bin/emctl stop oms -all ; sleep 5 ; $OMS_HOME/bin/emctl start oms
Now access your OMS console with your favorite browser and confirm that your new certificate appears. Your certificate should show a trusted path back to a root certificate, and your browser should produce no warnings.
At this point, you have secured communication between your browser and the EM12c OMS console with your custom certificate. You still have more work to do though. Your agents upload monitoring data to the OMS upload port, and it still uses the demo certificate. Fix that in the next step.
Secure the OMS upload port
Secure the OMS upload port. Expect to receive email or pager alerts after this step, as once you restart the OMS, none of your agents can communicate with it, as they expect to see the demo certificates on the upload port. You will need to provide the SYSMAN password as well as an agent registration password.
$ $OMS_HOME/bin/emctl secure oms -wallet $ORACLE_BASE/oemwallet -trust_certs_loc $ORACLE_BASE/trusted_certs.txt Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Securing OMS... Started. Enter Enterprise Manager Root (SYSMAN) Password : Enter Agent Registration Password : Securing OMS... Successful Restart OMS $ $OMS_HOME/bin/emctl stop oms -all ; sleep 5 ; $OMS_HOME/bin/emctl start oms
Re-secure all agents
Now you must re-secure all of your agents so that they can resume uploading data to the OMS console and monitoring your systems. Execute the following steps on every agent, using emctl from the agent home. You will need to provide an agent registration password to complete this process.
$ $AGENT_HOME/bin/emctl secure agent Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Agent successfully stopped... Done. Securing agent... Started. Enter Agent Registration Password : Agent successfully restarted... Done. Securing agent... Successful. $ $AGENT_HOME/bin/emctl upload agent Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. --------------------------------------------------------------- EMD upload completed successfully
It may take a little while for the OMS to process the new agents and their uploads, but once you have run this process on every agent they should all communicate successfully with the OMS and appear as OK from the agent management screen.
Create Oracle wallets for agents
Next we secure the agent URLs. The OMS connects to the agents at this URL to submit management requests. At the moment, the agents still use self-signed certificates to secure this URL. For this process we create an Oracle wallet, on the OMS host, using the same ORAPKI command as for the OMS wallet. We will generate a certificate signing request from each agent wallet, submit those CSRs to a certificate authority, and import the received certificates.
As with the OMS, the agents must use single-host certificates, not wildcard or subject alternate name (SAN) certificates. To determine the correct fully qualified domain name for each agent, execute emctl status agent from the agent home.
$ $AGENT_HOME/bin/emctl status agent Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Agent Version : 12.1.0.4.0 OMS Version : 12.1.0.4.0 Protocol Version : 12.1.0.1.0 Agent Home : /oraagent/agent12c/agent_inst Agent Log Directory : /oraagent/agent12c/agent_inst/sysman/log Agent Binaries : /oraagent/agent12c/core/12.1.0.4.0 Agent Process ID : 12480 Parent Process ID : 12359 Agent URL : https://agenthost.domain.com:3872/emd/main/
Repeat these steps for every agent.
Create a directory to store the agent wallet, and an agent wallet. This time do NOT use -auto_login_local, use only -auto_login, as you will distribute these wallets to the agent hosts after generating them on the OMS host. Use a strong password, and save it for later, as you will reuse it many times.
$ mkdir $ORACLE_BASE/agentwallets $ mkdir $ORACLE_BASE/agentwallets/agenthost.domain.com $ $MW_HOME/oracle_common/bin/orapki wallet create -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -auto_login Oracle PKI Tool : Version 11.1.1.7.0 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved. Enter password: Enter password again:
Create the certificate, then a certificate signing request, saving it to file ~/agenthost.domain.com.csr. Again, I recommend a 2048 bit certificate.
$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -dn "CN=agenthost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US" -keysize 2048 -pwd '[REDACTED]' Oracle PKI Tool : Version 11.1.1.7.0 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved. $ $MW_HOME/oracle_common/bin/orapki wallet export -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -dn "CN=agenthost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US" -request ~/agenthost.domain.com.csr Oracle PKI Tool : Version 11.1.1.7.0 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
As before, submit this certificate signing request to your certificate authority, and receive back three files containing a root certificate, an intermediate certificate, and a user certificate. Import these into the agent wallet, and display the wallet afterwards to confirm everything imported successfully.
$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -trusted_cert -cert ~/TrustedRoot.crt -pwd '[REDACTED]' Oracle PKI Tool : Version 11.1.1.7.0 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved. $ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -trusted_cert -cert ~/DigiCertCA.crt -pwd '[REDACTED]' Oracle PKI Tool : Version 11.1.1.7.0 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved. $ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -user_cert -cert ~/agenthost.domain.com.crt -pwd '[REDACTED]' Oracle PKI Tool : Version 11.1.1.7.0 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved. $ $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com Oracle PKI Tool : Version 11.1.1.7.0 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved. Requested Certificates: User Certificates: Subject: CN=agenthost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US Trusted Certificates: Subject: CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US Subject: CN=DigiCert High Assurance CA-3,OU=www.digicert.com,O=DigiCert Inc,C=US
You have finished creating this agent’s wallet. Repeat this for every agent.
Configure the agents to use their wallets
Inside the agent wallets you’ve just created, you will find a cwallet.sso file. Take this file from each agent’s wallet and copy it to the agent server. Stop the agent, then place the file into $AGENT_INSTANCE_DIR/sysman/config/server/ and set the permissions to 640, then start the agent.
$ $AGENT_HOME/bin/emctl stop agent Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Stopping agent ..... stopped. $ cp cwallet.sso $AGENT_INSTANCE_DIR/sysman/config/server $ chmod 640 $AGENT_INSTANCE_DIR/sysman/config/server $ $AGENT_HOME/bin/emctl start agent Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Starting agent ............. started.
Next, visit the agent URL in your favorite web browser and examine the certificate it uses. You should now see that it uses the 3rd party SSL/TLS certificate that you installed.
Secure WebLogic with the OMS wallet
Now the OMS (both console and upload ports) and agents will use your new certificates. This leaves WebLogic as the one remaining component needing your new certificates. Please note in following the below directions that securing WebLogic with a wallet only works as of EM12c R3, earlier versions must use a Java keystore. See note 1527874.1 for more information.
[NOTE: 20150910: If you secure WebLogic with a certificate that uses the SHA256 hashing algorithm, future attempts to apply EM12c PSU patches using ‘opatchauto’ will fail. Some piece of opatchauto does not support SHA256 usage in certificates. If you run into this issue, revert your WLS to the demonstration certificate using emctl secure wls -use_demo_cert, then apply the PSU, then resecure WLS using these steps with your desired certificate. I intend to write a full blog post about this later.]
First import the root and intermediate certificates to the keystore on the OMS host’s agent. Use the default password welcome for the agent keystore, and alias names rootcacert and intercacert.
$ $AGENT_HOME/bin/emctl secure add_trust_cert_to_jks -trust_certs_loc ~/TrustedRoot. crt -alias rootcacert -password welcome Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Message : Certificate was added to keystore ExitStatus: SUCCESS $ $AGENT_HOME/bin/emctl secure add_trust_cert_to_jks -trust_certs_loc ~/DigiCertCA. crt -alias intercacert -password welcome Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Message : Certificate was added to keystore ExitStatus: SUCCESS
Back up some WLS configuration files, just in case, before securing WLS with your certificate. If you have problems in this step, make sure you have stopped all WLS processes, then restore these files from backup.
$ mkdir ~/wlscertbak $ cp -a $EM_INSTANCE_BASE/em/EMGC_OMS1/emgc.properties ~/wlscertbak/ $ cp -a $EM_INSTANCE_BASE/NodeManager/emnodemanager/nodemanager.properties ~/wlscertbak/ $ cp -a $EM_INSTANCE_BASE/WebTierIH1/config/OHS/ohs1/keystores/proxy ~/wlscertbak/ $ cp -a $EM_INSTANCE_BASE/user_projects/domains/GCDomain/config/config.xml ~/wlscertbak/
Stop the OMS.
$ $OMS_HOME/bin/emctl stop oms Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Stopping WebTier... WebTier Successfully Stopped Stopping Oracle Management Server... Oracle Management Server Successfully Stopped Oracle Management Server is Down
Secure WLS using the OMS wallet created earlier. You will need to provide the SYSMAN password.
$ $OMS_HOME/bin/emctl secure wls -wallet $ORACLE_BASE/oemwallet Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Securing WLS... Started. Enter Enterprise Manager Root (SYSMAN) Password : Securing WLS... Successful Restart OMS using 'emctl stop oms -all' and 'emctl start oms' If there are multiple OMSs in this environment, perform this configuration on all of them.
Stop the entire WLS stack, then start the OMS.
$ $OMS_HOME/bin/emctl stop oms -all Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Stopping WebTier... WebTier Successfully Stopped Stopping Oracle Management Server... Oracle Management Server Already Stopped Oracle Management Server is Down Stopping BI Publisher Server... BI Publisher Server Successfully Stopped AdminServer Successfully Stopped BI Publisher Server is Down $ $OMS_HOME/bin/emctl start oms Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Starting Oracle Management Server... Starting WebTier... WebTier Successfully Started Oracle Management Server Successfully Started Oracle Management Server is Up Starting BI Publisher Server ... BI Publisher Server Successfully Started BI Publisher Server is Up
Visit the WebLogic Server administration console and you should now see that it presents your custom SSL/TLS certificate and no longer uses the demo certificate.
Conclusion
If you have successfully followed these steps, your system should now use your custom SSL/TLS certificates everywhere, and you should no longer experience certificate warnings in your browsers.
Pardy DBA 