Step by step: Configuring third party SSL/TLS certificates in EM12c R4

[EDIT 20170227: The process for configuring third party certificates for EM13c works about the same as for EM12c. If you have access to Oracle support, I suggest you review notes 2220788.1 and 2213661.1 for the most up-to-date documentation directly from Oracle.]

By default, when an administrator configures Oracle Enterprise Manager 12c to use SSL, the system will use a default self-signed certificate, provided for demo purposes only.  The documentation states repeatedly that users should not use these certificates in a production environment, as they represent a security risk. This blog post documents, step by step, a process to replace these demo certificates with custom third party certificates, across the OMS console, OMS upload port, agents, and WebLogic Server. I will follow this process on a single-OMS configuration; if you have more than one OMS please consult the documentation for more details, as your process will vary and the steps I have provided may break your system.

I have tested these instructions on Linux x86-64 (SLES11 SP3) with EM12c R4 PSU2 (12.1.0.4).

Official Documentation

The official documentation for this process resides in the following My Oracle Support notes:

  • Using ORAPKI Utility to Create a Wallet with Third Party Trusted Certificate and Import into OMS (Doc ID 1367988.1)
  • EM 12c Cloud Control How to Create a Wallet With Third Party Trusted Certificate that Can Be Imported into the OMS For SSL Comunication ? (Doc ID 1399293.1)
  • 12c Cloud Control: Steps to Import Third Party Trusted SSL Certificate into 12c Cloud Control Agent URL (Doc ID 1593183.1)
  • 12c Cloud Control: Steps to Create and Import Third Party / Self-Signed SSL Certificates for WebLogic Server in an Enterprise Manager Installation (Doc ID 1527874.1)
  • How to Create a Java Keystore via Keytool in FMW 11g/12c (Doc ID 1230333.1)

Why Should I Do This?

You may not fully understand the mechanics of SSL/TLS certificates and the chain of trust. I cannot fully explain this complex topic in a blog post, but if you need a reason to make this change other than demands from your organizational security/compliance team, please take Oracle’s word for it, and notice this text that appears in your GCDomain.log file when you run your system with the provided default demo certificates:

####   <[hostname redacted]>   <> <> <>

Read that again if you didn’t catch it the first time through: “The system is vulnerable to security attacks, since it trusts certificates signed by the demo trusted CA.” This text comes from code in WebLogic, not from me. Here Oracle tells you very explicitly that your system currently contains a severe vulnerability.

You will also notice that when using the EM12c console, or accessing an agent URL, or accessing the WebLogic Server administration console may show warnings in your browser about untrusted certificates. Once you replace your certificates as described in the documentation above or my steps below, you will no longer have those issues.

Using 3rd Party SSL/TLS Certificates With EM12c

Overview

You will follow 7 high level steps to complete the process of securing your EM12c environment with custom third party SSL/TLS certificates.

  1. Create an Oracle wallet for the OMS.
  2. Secure the OMS console using the OMS wallet.
  3. Secure the OMS upload port using the OMS wallet.
  4. Re-secure all agents.
  5. Create Oracle wallets for agents.
  6. Configure the agents to use their wallets.
  7. Secure WebLogic with the OMS wallet.

Create an Oracle wallet for the OMS

First we follow steps 1a through 1h from document 1367988.1.  All these steps occur on the OMS host.

Disable shell history (optional but recommended)

While following these steps, you will repeatedly have to type passphrases on the command line. To avoid having these passphrases stored in your Oracle user’s shell history, disable history saving.  In the bash shell that I use, I accomplish this by unsetting the HISTFILE variable. You may need to use another mechanism in another shell.

$ unset HISTFILE

Use the correct ORAPKI command

You should use the ORAPKI command from your middleware home’s oracle_common/bin directory.  I will refer to this as $MW_HOME/oracle_common/bin/orapki in the following instructions.

Create an Oracle wallet

The documentation specified that we should create an auto-login wallet, but in my single-OMS setup, I believe that I will achieve better security with an auto-login-local wallet, as the auto-login feature will only function on this specific host. You will need to select a base directory for your OMS wallet.  I used $ORACLE_BASE/oemwallet. ORAPKI will prompt you for a password. Use a secure one, and note it down somewhere safe. You will use it many times during this process.

$ mkdir $ORACLE_BASE/oemwallet
$ $MW_HOME/oracle_common/bin/orapki wallet create -wallet $ORACLE_BASE/oemwallet -auto_login_local
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Enter password:
 Enter password again:

Get in the habit of displaying the wallet contents after each operation to confirm that everything worked.

$ $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/oemwallet
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Requested Certificates:
 User Certificates:
 Trusted Certificates:
 Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
 Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US

Create a key within the wallet.  Make sure you replace omshost.domain.com with the fully qualified domain name of your OMS host. I highly recommend using a 2048 bit keysize, as shown below. Include the wallet password you specified earlier on the commandline as the -pwd argument, contained in single quotes. Display the wallet again afterward.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -dn "CN=omshost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US" -keysize 2048 -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

$ $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/oemwallet
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Requested Certificates:
 Subject: CN=omshost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US
 User Certificates:
 Trusted Certificates:
 Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US

Export a certificate signing request based on this key. Make sure the -dn you specify exactly matches the -dn specified earlier. Provide a filename in the -request argument in which to store the certificate signing request (CSR).

$ $MW_HOME/oracle_common/bin/orapki wallet export -wallet $ORACLE_BASE/oemwallet -dn "CN=omshost.domain.com, OU=EM,O=Organization,L=City,ST=State,C=US" -request ~/EM12cCSR.txt
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

Submit this CSR file to your signing authority. Inform them that you MUST have a single-host certificate with your OMS host’s fully qualified domain name in the CN field. Subject Alternate Name (SAN) certificates or wildcard certificates will not work at all. Your signing authority should then provide you with a root certificate, an intermediate certificate, and a user certificate.

Import the root, intermediate, and user certificates into the OMS wallet. Note that you must import the root and intermediate certificates using -trusted_cert, and the user certificate using -user_cert.  I used DigiCert, and I can confirm that their certificates function correctly in EM12c and recommend their service.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -trusted_cert -cert ~/TrustedRoot.cer -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -trusted_cert -cert ~/DigiCertCA2.cer -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -user_cert -cert ~/omshost.domain.com.cer -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

Display the wallet contents after this operation.

$  $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/oemwallet
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Requested Certificates:
 User Certificates:
 Subject: CN=omshost.domain.com,OU=[REDACTED],O=[REDACTED],L=City,ST=State,C=US
 Trusted Certificates:
 Subject: CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US
 Subject: CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US
 Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
 Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US

Concatenate the root and intermediate certificates into a single file, which you will use later.

$  cat ~/DigiCertCA2.cer ~/TrustedRoot.cer > $ORACLE_BASE/trusted_certs.txt

You have completed configuration of your OMS wallet.

Secure the OMS console

Now, using emctl from the $OMS_HOME, tell EM12c to secure the OMS console using the certificate contained in your wallet. The system will prompt you for the SYSMAN password and inform you to restart the entire OMS once complete.

$ $OMS_HOME/bin/emctl secure console -wallet /oracle/oem/oemwallet
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Securing Console... Started.
 Enter Enterprise Manager Root (SYSMAN) Password :
 Securing Console... Successful
 Restart OMS
$ $OMS_HOME/bin/emctl stop oms -all ; sleep 5 ; $OMS_HOME/bin/emctl start oms

Now access your OMS console with your favorite browser and confirm that your new certificate appears.  Your certificate should show a trusted path back to a root certificate, and your browser should produce no warnings.

At this point, you have secured communication between your browser and the EM12c OMS console with your custom certificate.  You still have more work to do though. Your agents upload monitoring data to the OMS upload port, and it still uses the demo certificate. Fix that in the next step.

Secure the OMS upload port

Secure the OMS upload port. Expect to receive email or pager alerts after this step, as once you restart the OMS, none of your agents can communicate with it, as they expect to see the demo certificates on the upload port. You will need to provide the SYSMAN password as well as an agent registration password.

$ $OMS_HOME/bin/emctl secure oms -wallet $ORACLE_BASE/oemwallet -trust_certs_loc $ORACLE_BASE/trusted_certs.txt
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Securing OMS... Started.
 Enter Enterprise Manager Root (SYSMAN) Password :
 Enter Agent Registration Password :
 Securing OMS... Successful
 Restart OMS
$ $OMS_HOME/bin/emctl stop oms -all ; sleep 5 ; $OMS_HOME/bin/emctl start oms

Re-secure all agents

Now you must re-secure all of your agents so that they can resume uploading data to the OMS console and monitoring your systems. Execute the following steps on every agent, using emctl from the agent home.  You will need to provide an agent registration password to complete this process.

$ $AGENT_HOME/bin/emctl secure agent
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Agent successfully stopped... Done.
Securing agent... Started.
Enter Agent Registration Password : 
Agent successfully restarted... Done.
Securing agent... Successful.
$ $AGENT_HOME/bin/emctl upload agent
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

It may take a little while for the OMS to process the new agents and their uploads, but once you have run this process on every agent they should all communicate successfully with the OMS and appear as OK from the agent management screen.

Create Oracle wallets for agents

Next we secure the agent URLs. The OMS connects to the agents at this URL to submit management requests. At the moment, the agents still use self-signed certificates to secure this URL.  For this process we create an Oracle wallet, on the OMS host, using the same ORAPKI command as for the OMS wallet. We will generate a certificate signing request from each agent wallet, submit those CSRs to a certificate authority, and import the received certificates.

As with the OMS, the agents must use single-host certificates, not wildcard or subject alternate name (SAN) certificates.  To determine the correct fully qualified domain name for each agent, execute emctl status agent from the agent home.

$  $AGENT_HOME/bin/emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 12.1.0.4.0
OMS Version : 12.1.0.4.0
Protocol Version : 12.1.0.1.0
Agent Home : /oraagent/agent12c/agent_inst
Agent Log Directory : /oraagent/agent12c/agent_inst/sysman/log
Agent Binaries : /oraagent/agent12c/core/12.1.0.4.0
Agent Process ID : 12480
Parent Process ID : 12359
Agent URL : https://agenthost.domain.com:3872/emd/main/

Repeat these steps for every agent.

Create a directory to store the agent wallet, and an agent wallet. This time do NOT use -auto_login_local, use only -auto_login, as you will distribute these wallets to the agent hosts after generating them on the OMS host.  Use a strong password, and save it for later, as you will reuse it many times.

$ mkdir $ORACLE_BASE/agentwallets
$ mkdir $ORACLE_BASE/agentwallets/agenthost.domain.com
$ $MW_HOME/oracle_common/bin/orapki wallet create -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -auto_login
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Enter password:
 Enter password again:

Create the certificate, then a certificate signing request, saving it to file ~/agenthost.domain.com.csr. Again, I recommend a 2048 bit certificate.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -dn "CN=agenthost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US" -keysize 2048 -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
$ $MW_HOME/oracle_common/bin/orapki wallet export -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -dn "CN=agenthost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US" -request ~/agenthost.domain.com.csr
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

As before, submit this certificate signing request to your certificate authority, and receive back three files containing a root certificate, an intermediate certificate, and a user certificate. Import these into the agent wallet, and display the wallet afterwards to confirm everything imported successfully.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -trusted_cert -cert ~/TrustedRoot.crt -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -trusted_cert -cert ~/DigiCertCA.crt -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -user_cert -cert ~/agenthost.domain.com.crt -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
$ $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Requested Certificates:
 User Certificates:
 Subject: CN=agenthost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US
 Trusted Certificates:
 Subject: CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US
 Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
 Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=DigiCert High Assurance CA-3,OU=www.digicert.com,O=DigiCert Inc,C=US

You have finished creating this agent’s wallet.  Repeat this for every agent.

Configure the agents to use their wallets

Inside the agent wallets you’ve just created, you will find a cwallet.sso file. Take this file from each agent’s wallet and copy it to the agent server. Stop the agent, then place the file into $AGENT_INSTANCE_DIR/sysman/config/server/ and set the permissions to 640, then start the agent.

$ $AGENT_HOME/bin/emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Stopping agent ..... stopped.
$ cp cwallet.sso $AGENT_INSTANCE_DIR/sysman/config/server
$ chmod 640 $AGENT_INSTANCE_DIR/sysman/config/server
$ $AGENT_HOME/bin/emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Starting agent ............. started.

Next, visit the agent URL in your favorite web browser and examine the certificate it uses.  You should now see that it uses the 3rd party SSL/TLS certificate that you installed.

Secure WebLogic with the OMS wallet

Now the OMS (both console and upload ports) and agents will use your new certificates. This leaves WebLogic as the one remaining component needing your new certificates. Please note in following the below directions that securing WebLogic with a wallet only works as of EM12c R3, earlier versions must use a Java keystore. See note 1527874.1 for more information.

[NOTE: 20150910: If you secure WebLogic with a certificate that uses the SHA256 hashing algorithm, future attempts to apply EM12c PSU patches using ‘opatchauto’ will fail. Some piece of opatchauto does not support SHA256 usage in certificates. If you run into this issue, revert your WLS to the demonstration certificate using emctl secure wls -use_demo_cert, then apply the PSU, then resecure WLS using these steps with your desired certificate. I intend to write a full blog post about this later.]

First import the root and intermediate certificates to the keystore on the OMS host’s agent. Use the default password welcome for the agent keystore, and alias names rootcacert and intercacert.

$ $AGENT_HOME/bin/emctl secure add_trust_cert_to_jks -trust_certs_loc ~/TrustedRoot. crt -alias rootcacert -password welcome
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.

 Message : Certificate was added to keystore
 ExitStatus: SUCCESS
$ $AGENT_HOME/bin/emctl secure add_trust_cert_to_jks -trust_certs_loc ~/DigiCertCA. crt -alias intercacert -password welcome
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.

 Message : Certificate was added to keystore
 ExitStatus: SUCCESS

Back up some WLS configuration files, just in case, before securing WLS with your certificate.  If you have problems in this step, make sure you have stopped all WLS processes, then restore these files from backup.

$ mkdir ~/wlscertbak
$ cp -a $EM_INSTANCE_BASE/em/EMGC_OMS1/emgc.properties ~/wlscertbak/
$ cp -a $EM_INSTANCE_BASE/NodeManager/emnodemanager/nodemanager.properties ~/wlscertbak/
$ cp -a $EM_INSTANCE_BASE/WebTierIH1/config/OHS/ohs1/keystores/proxy ~/wlscertbak/
$ cp -a $EM_INSTANCE_BASE/user_projects/domains/GCDomain/config/config.xml ~/wlscertbak/

Stop the OMS.

$ $OMS_HOME/bin/emctl stop oms
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Stopping WebTier...
 WebTier Successfully Stopped
 Stopping Oracle Management Server...
 Oracle Management Server Successfully Stopped
 Oracle Management Server is Down

Secure WLS using the OMS wallet created earlier. You will need to provide the SYSMAN password.

$ $OMS_HOME/bin/emctl secure wls -wallet $ORACLE_BASE/oemwallet
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Securing WLS... Started.
 Enter Enterprise Manager Root (SYSMAN) Password :
 Securing WLS... Successful
 Restart OMS using 'emctl stop oms -all' and 'emctl start oms'
 If there are multiple OMSs in this environment, perform this configuration on all of them.

Stop the entire WLS stack, then start the OMS.

$ $OMS_HOME/bin/emctl stop oms -all
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Stopping WebTier...
 WebTier Successfully Stopped
 Stopping Oracle Management Server...
 Oracle Management Server Already Stopped
 Oracle Management Server is Down
 Stopping BI Publisher Server...
 BI Publisher Server Successfully Stopped
 AdminServer Successfully Stopped
 BI Publisher Server is Down
$ $OMS_HOME/bin/emctl start oms
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Starting Oracle Management Server...
 Starting WebTier...
 WebTier Successfully Started
 Oracle Management Server Successfully Started
 Oracle Management Server is Up
 Starting BI Publisher Server ...
 BI Publisher Server Successfully Started
 BI Publisher Server is Up

Visit the WebLogic Server administration console and you should now see that it presents your custom SSL/TLS certificate and no longer uses the demo certificate.

Conclusion

If you have successfully followed these steps, your system should now use your custom SSL/TLS certificates everywhere, and you should no longer experience certificate warnings in your browsers.

See Also

Advertisements

18 thoughts on “Step by step: Configuring third party SSL/TLS certificates in EM12c R4

  1. Pingback: EM12c R4 SSL Security Checkup Script | Pardy DBA

  2. Karl Brown

    Thanks for the great page!

    Caution when attempting to Secure the OMS console using the the Oracle Wallet from the 12c DB Client instead of the Oracle Wallet supplied with OMS.

    Received the following error message.
    2015-09-30 04:25:09,643 [main] ERROR oms.SecureConsoleCmds main.187 – Failed to secure the Console: oracle.security.crypto.core.CipherException: Invalid padding string (or incorrect password)
    java.io.IOException: oracle.security.crypto.core.CipherException: Invalid padding string (or incorrect password)
    at oracle.security.pki.OracleWallet.open(Unknown Source)

    Opened the same wallet in version 11.1.1.7.0. Saved it again. Command ran successfully.

    Thanks
    Karl

    Reply
  3. Karl Brown

    Another tip – on the section “Re-secure all agents”

    This step can be done as a batch job from the OEM UI.

    1) Login as Sysman
    2) Navigate to Setup->Manage Cloud Control->Agents
    3) Highlight all your agents.
    4) Click Secure
    5) Enter the agent registration password.
    6) Click Submit.
    7) Have a coffee while sitting back and watching the pretty graphs of the job securing all your agents.

    Cheers
    Karl

    Reply
  4. George Tarabuta

    Hi, one more thing: I have created the wallet with auto_login_local and the crashed with the error:
    oracle.security.crypto.core.CipherException: Invalid padding string (or incorrect password).
    Changed the wallet to simple auto_login (not local) and it works fine . . .
    George

    Reply
    1. Brian Pardy Post author

      Hello Prasad,

      Do you mean recreating the certificate signing request that comes out of the Oracle wallet to send to either a third party certificate authority or to your internal certificate authority?

      If so you can repeat the “orapki wallet export” step above under “Export a certificate signing request”. That will reproduce the CSR for an existing wallet record and you can have it re-signed if needed.

      If your question was about some other step, please clarify if you can.

      -Brian

      Reply
      1. Prasad

        Thank you for the Prompt response Brian,

        This helps.

        Also, if we intend to make use of a self signed certificate, do we need to concatenate the only certificate content to “trusted_certs.txt” ? is that how it works..?

        Thanks again,
        Prasad

      2. Brian Pardy Post author

        Hi Prasad,

        I think you are correct. If you use a self-signed certificate, that same certificate is used to authenticate itself as the root, so yes, I believe that you will need to concatenate only that one certificate’s content into the trusted_certs.txt file. Also, two steps before that, when you add the certificate to the wallet using -trusted_cert, you will not have an intermediate certificate so you should only need to import that certificate once with -trusted_cert and then once again with -user_cert.

        Good luck, I hope this helps!
        -Brian

  5. tenzil

    Nice instructions. I had just secured my test server last month, but came across something unexpected. Looking the the OMS logs, I noticed that warning messages about using the DemoTrust. So I rolled back everything and tried again.

    When I secure WLS, I can confirm in the weblogic console that the admin server and oms are using custom identity & trust stores. However, after securing the OMS, I still see the warnings in the log. secure.log shows that my trusted ca certs were imported into its stores (in the wlserver_10.3 directory because the log shows that’s what it’s using).

    Any ideas?

    Thanks,
    Ten

    Reply
    1. Brian Pardy Post author

      Hi there,
      Bonus points to you for noticing this! I have upgraded to EM13c since making this post, but I can absolutely confirm that I noticed the same thing in the logs when I initially prepared this procedure.
      I cannot say for sure if this leaves your EM12c installation vulnerable to still have the widely known demonstration CA certificates trusted by WLS even after applying your own custom certificates. I am inclined to believe the warning message in the logs is true and that the system remains vulnerable.
      I filed a security service request at the time on this exact issue you noticed, pointing it out and providing a set of keytool commands to remove the demo CA certs from this file, resulting in a system that does NOT print that warning in the logs on startup. I was advised, after the SR was sent to security, that this was the intended, expected behavior, and that removing the demo certs was not supported. I let it go at that and elected not to publicize the issue (other than hinting at it in this blog post), but I still consider this a potential security issue. It is likely in my opinion that it will not be resolved with a patch unless someone can demonstrate a proof of concept exploit.
      In the meantime, if your environment has extreme security requirements, I don’t think it would be a bad idea to script something up to remove these demo certs after applying your own certs, perhaps coupled with another script to restore the demo CA certs prior to patching or upgrading your EM12c in case any of those steps rely (unwisely) on those certs.
      When I have a chance, probably Monday, I will reply again and post the keytool commands I used to remove the demo CAs on my system. I do suggest taking care here as it is possible to get your EM12c into a confused setup where you need to restore the various DemoTrust|DemoIdentity files from backup copies to allow the whole thing to start up successfully, so don’t experiment on production without a backup like I did at the time 🙂
      Thanks for your post and good luck!
      -Brian

      Reply
  6. tenzil

    Brian,

    Thanks for replying so fast. I had the same thoughts of removing the demo CAs. BTW, is this fixed in EM 13?

    Ten

    Reply
    1. Brian Pardy Post author

      I haven’t really checked into securing EM13c beyond the manual yet. There’s some hope with the newer WLS release that this has changed but I need to have a look at it. Thanks again for posting.
      -Brian

      Reply
  7. Pingback: Securing Oracle Enterprise Manager 13c | Pardy DBA

  8. vadim roytman

    PKI-04006: No matching private key in the wallet

    I follow all steps in this article, but getting this error message on this particular step:

    orapki wallet add -wallet /app04/oracle/OEM13cR2/WALLET_12302016 -user_cert -cert /app04/oracle/OEM13cR2/WALLET_12302016/user_1230.txt -pwd redacted

    Any help will be much appreciated

    Reply
    1. Brian Pardy Post author

      Hello Vadim,

      That error should mean that the wallet you are using as the target when adding the user cert is not the same wallet you used when you first created the key and then exported it to create the CSR. I have seen that when I tried to add a signed certificate to the incorrect wallet.

      Try displaying the contents of the WALLET_12302016 wallet and confirm that you see the subject DN in the “Requested certificates” section of the orapki display output, I think you may just be pointing at the wrong wallet.

      Good luck!

      Reply
  9. Manga

    Hello Brian, thank you for your detailed notes. SSL is totally new to me and I am tasked with fixing some issues resulting from security scan violations from outdated SSL certificates from a our grid control server. I have been reading online and need some clarifications. I am assuming SSL was setup before I joined the project and the certs are now outdated. I will be using federal issued certificates, I just need cloud control (13c) specific instructions on how to replace the new certs with what is currently in Production now and causing security scan violations. Please help me narrow this down. Basically, I am confused and do not know what portion of your instructions above will be valid for my scenario. (instructions on how to replace outdated certs with new issued certs)

    Reply
    1. Brian Pardy Post author

      Hello Manga,

      Thanks for commenting. It sounds like you will need to either renew your existing certificate, or just create a new one. First I think you need to identify the exact issue reported from your security scans: they may be reporting that your certificates have simply expired, or they may be reporting that your certificates are using outdated algorithms like MD5 or SHA-1. You should also determine the exact version of Cloud Control that you are running (EM13c 13.1 R1, or EM13c 13.2 R2) so that you can find the correct documentation to follow.

      Once you know your EM13c version and the exact problem with your current certificates, you can proceed from there. Hopefully your site has support from Oracle and you can refer to MOS notes for more information. I suggest reviewing MOS note 2202569.1 as it contains a lot of useful information. If your OMS is up and running, you can run “emctl secdiag openurl -url https://omshostname.domainnname:4903/empbs/upload” (replace omshostname and domainname with the correct hostname and domain name for your site), and that will display your installed certificates. From that output you should be able to find the “Valid from” and “Valid till” lines to see the date your current certificate expires to confirm that it actually has expired.

      If it is correct that your OMS certificate has expired, the following steps should allow you to request and install a new certificate that is not expired. (You would need to run different steps if the security issue reported is MD5 or SHA-1 usage.) First, take a backup of the entire OMS server just in case you have problems. If it is on a virtual machine, a snapshot works very well.

      1 – Create a new Oracle wallet on your OMS server (this is under the “create an Oracle wallet for the OMS” section in this blog post)

      2 – Create a new key in that wallet (in the same section of the blog post, in the “orapki wallet add” command). When you specify the “-dn” parameter to orapki, you should use the same values that were used for your current certificate, which should be visible in the output from the “oemctl secdiag openurl” command I mentioned before. The CN= should match the hostname of your OMS server, the ST= and C= fields should match your current state and country, and the OU= field should match your current OU= (organization unit) field.

      3 – Export a “signing request” (in the blog post in the “orapki wallet export” command) for your new key. This will create a new file using the filename specified with the “-request” parameter.

      4 – Send the file produced in step 3 to your signing authority. I do not know how your federal issuer receives certificate signing requests, but they should have some documentation available for you.

      5 – Wait for the signing authority to send you a signed certificate. They will probably send you back multiple files that include your new certificate and two additional files for their root certificate and their chain certificate. Some signers will only send you your certificate and their root certificate because they do not use chain certificates, your federal issuer should be able to tell you what they will provide.

      6 – Once you have received those files, follow the “orapki wallet add” commands in the “Import the root, intermediate, and user certificates into the OMS wallet” section. You will add your certificate with the -user_cert parameter, and the signer’s root+chain certificates with the -trusted_cert parameter. Also make sure to follow the step at the end of this section to create the trusted_certs.txt file.

      Now you should have a new wallet with a new certificate. Make sure you have good backups of your server before attempting the final two steps, because you may need to restore from backup if the secure steps fail.

      7 – Follow the “Secure the OMS console” section that has you run “emctl secure console -wallet” and provide the path to the new wallet where you imported the signed certificates.

      8 – Follow the “Secure the OMS upload port” section that has you run “emctl secure oms -wallet” and provide the path to the new wallet and the trusted_certs.txt file created in step 6.

      You are probably all done at this point. Since you are re-using the same federal issuer, you probably will NOT need to re-secure your agents. But, if your federal issuer has changed root or chain certificates since your old certificates were installed, you MAY need to go through the “Re-secure all agents” step in the blog post.

      After all this, have your security team re-run their scans and your EM13c environment should now pass, hopefully.

      If you feel uncomfortable with these steps, I suggest that you file a service request with Oracle if your site has support, and request their help to guide you through the process of renewing your expired certificate.

      Good luck!
      -Brian

      Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s