(EDIT 20130917: If you simply need to change the IP address of your OEM server, please review MOS note 1562029.1. The procedure in that note may allow you to change your OEM server’s IP address without following the lengthy process I describe below.)
In order to save power in our data center, I need to migrate my EM12c R3 environment from the host where it currently runs to a new host. I have a simple configuration, with a single OMS, no load balancer, and the repository database runs on the same host as EM12c R3 itself. I also have BI Publisher installed and integrated with EM12c, and a few third party plugins as I’ve detailed elsewhere on this blog. If you use an OS other than Linux x86-64 I suggest you research thoroughly as this procedure may or may not apply to your environment. Further, if you have a multi-OMS setup or use a load balancer, you must read the documentation and adapt the process accordingly to match your system’s needs. Note that I wrote this as I did the migration, live, on my production system, so I have text in a few places showing where I would have done things differently if I knew what to expect in the first place. It all ended up working, but it could have been simpler.
Oracle documents the procedure for this migration in the EM12c Administrator’s Guide, Part VII, section 29, “Backing Up and Recovering Enterprise Manager“. As a first step, my system administrator installed SLES 11 SP3 on the new server and created an account for me along with the ‘oracle’ account for EM12c. I have a 70GB volume to use for the database and OEM binaries, a 1GB volume for the DB control files and a 2GB volume for redo logs supplemented with a 15GB FRA volume to support flashback. Due to our tape backup strategy I use the FRA only for flashback, which we don’t wish to backup, and use a separate volume for RMAN backupsets. To avoid a backup/restore cycle, the volumes holding the database datafiles will just be moved over to the new host on the storage side.
First I will relocate the management repository database to the new host, then complete the process by relocating the OMS.
Relocating the Management Repository Database
I run Oracle Database 11.2.0.3, Enterprise Edition, plus PSU Jul 2013. Rather than installing the database software from scratch and patching it, I will clone the existing Oracle home to the new server. Unfortunately I cannot use EM12c to do the cloning, as cloning via EM12c requires a management agent on the new host. The software-only install of EM12c that I will run later installs a management agent as part of the process and I do not wish these two to conflict, so I do not want to install an agent on the new host at this time.
I will clone the database home according to the procedure in Appendix B of the 11gR2 database documentation. You should review the documentation for full details.
Cloning the Database Home
Stop the OMS, database and management agent before cloning the existing Oracle home.
oracle$ $OMS_HOME/bin/emctl stop oms -all ; $AGENT_HOME/bin/emctl stop agent ; $ORACLE_HOME/bin/dbshut $ORACLE_HOME
Create a zip file of the existing database home. Run this step as root (or using sudo) to make sure that you get all the files.
oracle$ sudo zip -r dbhome_1.zip /oracle/oem/product/11.2.0/dbhome_1
Now I will start the original database back up so that OEM continues running while I prepare the cloned Oracle home. I will perform this migration over a few days, as I have time, so I need to keep OEM up and running as much as possible to support and manage my other databases.
oracle$ $ORACLE_HOME/bin/dbstart $ORACLE_HOME ; sleep 10 ; $OMS_HOME/bin/emctl start oms ; sleep 10 ; $AGENT_HOME/bin/emctl start agent
Copy this zip file to the new host.
oracle$ scp dbhome_1.zip oracle@newhost:/oracle/oem
On the new host, extract this zip file to the target directory.
oracle@newhost$ unzip -d / dbhome_1.zip
Remove all “*.ora” files from the extracted $ORACLE_HOME/network/admin directory.
oracle@newhost$ rm /oracle/oem/product/11.2.0/dbhome_1/network/admin/*.ora
Execute clone.pl from $ORACLE_HOME/clone/bin.
oracle@newhost$ export ORACLE_HOME=/oracle/oem/product/11.2.0/dbhome_1 oracle@newhost$ $ORACLE_HOME/perl/bin/perl clone.pl ORACLE_BASE="/oracle/oem" ORACLE_HOME="/oracle/oem/product/11.2.0/dbhome_1" OSDBA_GROUP=dba OSOPER_GROUP=oper -defaultHomeName
Unfortunately this creates an oraInventory directory in the oracle user’s home directory. I prefer to keep oraInventory under ORACLE_BASE, so I moved it and edited the generated files to change the path from /home/oracle/oraInventory to /oracle/oem/oraInventory. Most likely some environment variable, or a previously existing /etc/oraInst.loc would have prevented this optional step.
oracle@newhost$ cp -a ~/oraInventory /oracle/oem oracle@newhost$ cd /oracle/oem/oraInventory oracle@newhost$ perl -pi.bak -e 's#/home/oracle#/oracle/oem#' oraInst.loc orainstRoot.sh
Complete the cloning steps by running the orainstRoot.sh and root.sh scripts.
oracle@newhost$ sudo /oracle/oem/oraInventory/orainstRoot.sh Changing permissions of /oracle/oem/oraInventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /oracle/oem/oraInventory to dba. The execution of the script is complete. oracle@newhost$ sudo /oracle/oem/product/11.2.0/dbhome_1/root.sh Check /oracle/oem/product/11.2.0/dbhome_1/install/root_newhost_2013-08-27_13-04-51.log for the output of root script
I do not want to use netca to configure the listener, so I will just copy the $ORACLE_HOME/network/admin/*.ora files back over from the original server to the new server, and edit them accordingly.
oracle$ scp *.ora oracle@newhost:/oracle/oem/product/11.2.0/dbhome_1/network/admin/ oracle@newhost$ cd $ORACLE_HOME/network/admin oracle@newhost$ perl -pi.bak -e 's#oldhost#newhost#' *.ora
This completes the database cloning.
Start Management Repository Database On New Host
At this point you will probably use RMAN to create a backup of your original repository database, then restore that backup onto the new host. Instead, I will cheat a bit, shut down OEM and the database, and ask my sysadmin to move the repository database’s datafile LUN over to the new host and mount it at the same location.
Before moving the LUN, create directories that the database needs for a successful startup. These include the admin/SID/adump directory, and in my case, the /oracle/mirror/SID/cntrl and /oracle/mirror/SID/log directories where I keep the multiplexed copies of my redo logs and controlfiles.
oracle@newhost$ mkdir -p /oracle/oem/admin/emrep/adump oracle@newhost$ mkdir -p /oracle/mirror/emrep/cntrl ; mkdir -p /oracle/mirror/emrep/log
As a sanity check, you should try starting up the listener on the new server and starting the database in NOMOUNT mode before proceeding. This will help catch any issues that may exist in your environment before you start the outage on your original server. Investigate and resolve any issues found before proceeding.
Shutdown the OMS, agent and database on the original server.
oracle$ $OMS_HOME/bin/emctl stop oms -all ; $AGENT_HOME/bin/emctl stop agent ; $ORACLE_HOME/bin/dbshut $ORACLE_HOME
Copy the controlfiles and redo logs from the original server to the new server.
oracle$ scp /oracle/oem/cntrl/control01.ctl oracle@newhost:/oracle/oem/cntrl/control01.ctl oracle$ scp /oracle/mirror/emrep/cntrl/control02.ctl oracle@newhost:/oracle/mirror/emrep/cntrl/control02.ctl oracle$ scp /oracle/oem/log/redo* oracle@newhost:/oracle/oem/log oracle$ scp /oracle/mirror/emrep/log/redo* oracle@newhost:/oracle/mirror/emrep/log
Back on the new server, start up the listener, then the database. I probably should have disabled flashback first.
oracle@newhost$ lsnrctl start LISTENER oracle@newhost$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.3.0 Production on Wed Aug 28 10:09:01 2013 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to an idle instance. SQL> startup; ORACLE instance started. Total System Global Area 9620525056 bytes Fixed Size 2236488 bytes Variable Size 6241128376 bytes Database Buffers 3355443200 bytes Redo Buffers 21716992 bytes Database mounted. ORA-38760: This database instance failed to turn on flashback database SQL> select open_mode from v$database; OPEN_MODE -------------------- MOUNTED SQL> alter database flashback off; Database altered. SQL> alter database open; Database altered.
Reconfigure Existing OMS For New Repository Database
Start the OMS and agent on the original server. OMS startup will fail, as you have not yet reconfigured the repository.
oracle$ $OMS_HOME/bin/emctl start oms Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. Starting Oracle Management Server... Starting WebTier... WebTier Successfully Started Oracle Management Server is not functioning because of the following reason: Failed to connect to repository database. OMS will be automatically restarted once it identifies that database and listener are up. Check EM Server log file for details: /oracle/oem/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out oracle$ $AGENT_HOME/bin/emctl start agent
Reconfigure the OMS repository database connection. Provide SYSMAN’s password when prompted.
oracle$ $OMS_HOME/bin/emctl config oms -store_repos_details -repos_conndesc "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=newhost)(PORT=1521)))(CONNECT_DATA=(SID=emrep)))" -repos_user sysman Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. Enter Repository User's Password : Successfully updated datasources and stored repository details in Credential Store. If there are multiple OMSs in this environment, run this store_repos_details command on all of them. And finally, restart all the OMSs using 'emctl stop oms -all' and 'emctl start oms'. It is also necessary to restart the BI Publisher Managed Server.
Stop, then restart the OMS.
oracle$ $OMS_HOME/bin/emctl stop oms -all ; sleep 5 ; $OMS_HOME/bin/emctl start oms Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. Stopping WebTier... WebTier Successfully Stopped Stopping Oracle Management Server... Oracle Management Server Successfully Stopped AdminServer Successfully Stopped Oracle Management Server is Down Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. Starting Oracle Management Server... Starting WebTier... WebTier Successfully Started Oracle Management Server Successfully Started Oracle Management Server is Up
Login to OEM and confirm proper operation of the system. I had a lot of alerts for failed backup jobs since my repository database hosts my RMAN catalog. These can wait for now. Also expect your repository target to show as down, since you have not yet updated the monitoring configuration. Reconfigure it now, providing the SYSMAN password when prompted.
oracle$ $OMS_HOME/bin/emctl config emrep -conn_desc "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=newhost)(PORT=1521)))(CONNECT_DATA=(SID=emrep)))" Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. Please enter repository password: Enter password : Login successful Target "Management Services and Repository:oracle_emrep" modified successfully Command completed successfully!
At this point you have successfully moved your repository database. Don’t worry about any errors for now, though if you rely on an RMAN catalog and stored scripts for your backups, and these all live in your OEM repository database, you should go through now and update the monitoring configuration for the repository database and listener so that backups of your other databases do not fail. I had to edit the recovery catalog and specify the host, port, and SID manually, since for some reason when I told it to use the repository database it kept trying to use the old hostname. I will fix this after I complete the rest of the migration.
IMPORTANT NOTE: Since you have not yet migrated the repository database target to an agent local to that machine, backups of your repository database may not run. Monitor your archived log directory on this system until you complete the rest of the migration, and manually run backups when necessary.
Installing OMS On A New Host
To install the OMS on a new host, perform a software-only installation from the same EM12c R3 installer that was used to install on the original host. You will need to identify and retrieve all of the plugins that you have installed on the current OMS, as well as any patches that are currently installed on the OMS. You must also make sure to use the same directory layout as on the original OMS.
Identifying Installed Patches
oracle$ $OMS_HOME/OPatch/opatch lsinv -oh $OMS_HOME
[...]
Interim patches (1) :
Patch 13983293 : applied on Thu Jul 11 09:56:16 EDT 2013
Unique Patch ID: 14779750
Created on 25 Apr 2012, 02:18:06 hrs PST8PDT
Bugs fixed:
13587457, 13425845, 11822929
This patch gets installed by the EM12c R3 installer, so no need to bother with it any further. If you have other patches installed, go fetch them, and install them after you have completed the plugin installation (see below).
Identifying Installed Plugins
Identify all plugins installed on your system using the query provided in the documentation, run as SYSMAN against your repository database.
SELECT epv.display_name, epv.plugin_id, epv.version, epv.rev_version,decode(su.aru_file, null, 'Media/External', 'https://updates.oracle.com/Orion/Services/download/'||aru_file||'?aru='||aru_id||chr(38)||'patch_file='||aru_file) URL
FROM em_plugin_version epv, em_current_deployed_plugin ecp, em_su_entities su
WHERE epv.plugin_type NOT IN ('BUILT_IN_TARGET_TYPE', 'INSTALL_HOME')
AND ecp.dest_type='2'
AND epv.plugin_version_id = ecp.plugin_version_id
AND su.entity_id = epv.su_entity_id;
Oracle-provided plugins will show a URL from which you must download the plugin. Third-party plugins will not; you will need to make sure you have the appropriate downloaded plugin install .opar file from when you initially installed it. Gather up all of these plugin files into a single directory on your NEW OMS host, changing the “.zip” filename extension to “.opar” for the Oracle-provided plugins. You need EVERY plugin returned by this query or else your installation will NOT work. I placed mine in /oracle/oem/migration/plugins.
You also need to copy over the three .zip files containing the OEM 12cR3 distribution: V38641-01.zip, V38642-01.zip and V38643-01.zip. Save them into a convenient staging area on the new server (I use /oracle/oem/stage).
Perform Software-Only Installation Of EM12c R3
Go to the staging area on the new server and extract the three .zip files containing the EM12c R3 distribution, then start the installer.
oracle@newhost$ unzip V38641-01.zip ; unzip V38642-01.zip ; unzip V38643-01.zip [...] oracle@newhost$ ./runInstaller
You can follow my previous post about upgrading EM12c R2 to R3 for more information about the installation process, just make sure you run it as a software only install and use the exact same path names as configured on the original OMS. In my case this means a middleware home of /oracle/oem/Middleware12cR3 and an agent base directory of /oracle/oem/agent12c.
While the software installation proceeds, you should run an exportconfig on your current OMS to produce the configuration backup file you will need to use to reconfigure the new one. Enter the SYSMAN password when prompted.
oracle$ $OMS_HOME/bin/emctl exportconfig oms Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root (SYSMAN) Password : ExportConfig started... Machine is Admin Server host. Performing Admin Server backup... Exporting emoms properties... Exporting secure properties... Export has determined that the OMS is not fronted by an SLB. The local hostname was NOT exported. The exported data can be imported on any host but resecure of all agents will be required. Please see the EM Advanced Configuration Guide for more details. Exporting configuration for pluggable modules... Preparing archive file... Backup has been written to file: /oracle/oem/gc_inst/em/EMGC_OMS1/sysman/backup/opf_ADMIN_20130828_120424.bka The export file contains sensitive data. You must keep it secure. ExportConfig completed successfully!
Copy that backup file to the new server.
oracle$ scp /oracle/oem/gc_inst/em/EMGC_OMS1/sysman/backup/opf_ADMIN_20130828_120424.bka oracle@newhost:/oracle/oem
Once the software-only install finishes, it will prompt you to run allroot.sh. Do so.
oracle@newhost$ sudo /oracle/oem/Middleware12cR3/oms/allroot.sh
Starting to execute allroot.sh .........
Starting to execute /oracle/oem/Middleware12cR3/oms/root.sh ......
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /oracle/oem/Middleware12cR3/oms
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
/etc exist
Creating /etc/oragchomelist file...
/oracle/oem/Middleware12cR3/oms
Finished execution of /oracle/oem/Middleware12cR3/oms/root.sh ......
Starting to execute /oracle/oem/agent12c/core/12.1.0.3.0/root.sh ......
Finished product-specific root actions.
/etc exist
/oracle/oem/agent12c/core/12.1.0.3.0
Finished execution of /oracle/oem/agent12c/core/12.1.0.3.0/root.sh ......
After running allroot.sh, you need to run the PluginInstall.sh script with the path where you saved the .opar files. Make sure you select every plugin listed when you ran the query to retrieve the plugin list earlier, then hit install.
oracle@newhost$ /oracle/oem/Middleware12cR3/oms/sysman/install/PluginInstall.sh -pluginLocation /oracle/oem/migration/plugins
This must match the list you generated previously
Prepare the Software Library
Go to the original server, and copy the contents of the software library to the new server.
oracle$ scp -r /oracle/oem/software_library/ oracle@newhost:/oracle/oem
Recreate the OMS with OMSCA
Shut everything down on your old server.
oracle$ $OMS_HOME/bin/emctl stop oms -all ; sleep 5 ; $AGENT_HOME/bin/emctl stop agent
Run OMSCA using the exportconfig backup file you generated earlier. Enter the administration server, node manager, repository database user and agent registration passwords when prompted.
oracle@newhost$ $OMS_HOME/bin/omsca recover -as -ms -nostart -backup_file /oracle/oem/opf_ADMIN_20130828_120424.bka Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.3.0 Copyright (c) 1996, 2013, Oracle. All rights reserved. OS check passed. OMS version check passed. Performing Admin Server Recovery... Retrieved Admin Server template. Source Instance Host name where configuration is exported : [deleted] Populated install params from backup... Enter Administration Server user password: Confirm Password: Enter Node Manager Password: Confirm Password: Enter Repository database user password: Enter Agent Registration password: Confirm Password: Doing pre requisite checks ...... Pre requisite checks completed successfully Checking Plugin software bits Proceed to recovery Setting up domain from template... Setup EM infrastructure succeeded! Admin Server recovered from backup. Now performing cleanup of OMS EMGC_OMS1... Now launching DeleteOMS... OMS Deleted successfully Delete finished successfully Now launching AddOMS... Infrastructure setup of EM completed successfully. Doing pre deployment operations ...... Pre deployment of EM completed successfully. Deploying EM ...... Deployment of EM completed successfully. Configuring webtier ...... Configuring webTier completed successfully. Importing OMS configuration from recovery file... If you have software library configured please make sure it is functional and accessible from this OMS by visiting: Setup->Provisioning and Patching->Software Library Securing OMS ...... Adapter already exists: emgc_USER Adapter already exists: emgc_GROUP Post "Deploy and Repos Setup" operations completed successfully. Performing Post deploy operations .... Total 0 errors, 78 warnings. 0 entities imported. pluginID:oracle.sysman.core Done with csg import pluginID:oracle.sysman.core Done with csg import No logging has been configured and default agent logging support is unavailable. Post deploy operations completed successfully. EM configuration completed successfully. EM URL is:https://newhost:7803/em Add OMS finished successfully Recovery of server EMGC_OMS1 completed successfully OMSCA Recover completed successfully
Start the OMS on the new server.
oracle@newhost$ $OMS_HOME/bin/emctl start oms
Configure the central agent on the new server, then run the root.sh script.
oracle@newhost$ /oracle/oem/agent12c/core/12.1.0.3.0/sysman/install/agentDeploy.sh AGENT_BASE_DIR=/oracle/oem/agent12c AGENT_INSTANCE_HOME=/oracle/oem/agent12c/agent_inst AGENT_PORT=3872 -configOnly OMS_HOST=newhost EM_UPLOAD_PORT=4902 AGENT_REGISTRATION_PASSWORD=password [...] oracle@newhost$ sudo /oracle/oem/agent12c/core/12.1.0.3.0/root.sh
Relocate the oracle_emrep target to the new OMS host.
oracle@newhost$ $OMS_HOME/bin/emcli login -username=sysman Enter password : Login successful oracle@newhost$ $OMS_HOME/bin/emcli sync Synchronized successfully oracle@newhost$ $OMS_HOME/bin/emctl config emrep -agent newhost:3872 Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. Please enter repository password: Enter password : Login successful Moved all targets from oldhost:3872 to newhost:3872 Command completed successfully! Enter password : Login successful Moved all targets from oldhost:3872 to newhost:3872 Command completed successfully!
Step through each of your existing agents to re-secure them against the new OMS. Provide the OMS HTTP port (not HTTPS) in this command, and enter the agent registration password when prompted.
$ $AGENT_INSTANCE_DIR/bin/emctl secure agent -emdWalletSrcUrl "http://newhost:4890/em" Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. Agent successfully stopped... Done. Securing agent... Started. Enter Agent Registration Password : Agent successfully restarted... Done. EMD gensudoprops completed successfully Securing agent... Successful.
Start the agent on the old OMS server. You should not need to do this, but I could not update the WebLogic Domain monitoring configuration without doing so first. Also re-secure this agent to point to the new OMS.
oracle$ $AGENT_HOME/bin/emctl start agent oracle$ $AGENT_INSTANCE_DIR/bin/emctl secure agent -emdWalletSrcUrl "http://newhost:4890/em"
Login to the OEM GUI running on the new server and navigate to the WebLogic Domain target for the Cloud Control domain. In the Target Setup -> Monitoring Credentials section, update the Administration server host value to the new server name, then hit OK. Then execute a Refresh WebLogic Domain, selecting Add/Update Targets, to move all WebLogic targets to the new central agent.
I use third-party plugins to monitor VMWare targets, NetApp storage and MySQL servers. I had many of them set up to run from the OMS agent (except for the VMWare ones, since Blue Medora helpfully advised not to use the OMS agent for this — great advice). I now need to relocate each of these targets to the new central agent using emcli. You won’t need to do this step unless you also have things set up this way. If I had to do this again, I would not use the OMS agent for these targets, since I would not need to change anything if I just had these on some other agent.
oracle@newhost$ ./emcli relocate_targets -src_agent=oldhost:3872 -dest_agent=newhost:3872 -copy_from_src -target_name=nameoftarget -target_type=typeoftarget
Final Cleanup Steps
By now you have completed the bulk of the work necessary to migrate your EM12c stack to a new server. Only a few steps remain. If you use any utility scripts on the old server, go ahead and copy those over now. I have scripts to automate starting/stopping the OMS and agent, so I’ve copied those over. Also make sure the oracle user on the new server has all the environment variables set up in their shell initialization files.
oracle$ scp ~/bin/CCstart ~/bin/CCstop oracle@newhost:bin/
The GCDomain Oracle WebLogic Domain target did not get moved to my new agent. If this happened to you, go to the target home page and select the Modify Agents menu item. Click Continue, then find GCDomain in the list, scroll to the right, and assign the new OMS server’s agent as the monitoring agent for this target, then click the Modify Agents button.
Reinstall BI Publisher
Since I had BI Publisher installed on the old server, I need to install it again on the new one. Retrieve the 11.1.1.6.0 BI Publisher installation files used previously, and copy them to your staging area. Run the “runInstaller” program from bishiphome/Disk1, and perform a software-only installation with the middleware home set to your EM12c installation middleware home, and leave the Oracle home as Oracle_BI1.
Instead of running the configureBIP script as you normally would to integrate BI Publisher with EM12c, just go to the WebLogic administration console after the software-only install completes, and navigate to the BIP server configuration page. Lock the configuration for editing, and edit the configuration to change the listen address to reference the new server’s hostname and change the machine to the machine name where the admin server runs (in my case it showed up as EMGC_MACHINE2). Save and activate the changes, then start the BIP server.
After the server has started, return to the WebLogic Domain page and re-run the Refresh WebLogic Domain step, again with Add/Update targets, to move BIP to your new OMS agent.
I actually had to do the Refresh WebLogic Domain step here twice. I may have simply not waited long enough after starting BIP before I ran it, but I do not know for sure.
Update EM Console Service
I have only one target showing down at this point, the EM Console Service. Go to the target, and click on the Monitoring Configuration tab. Click on Service Tests and Beacons. Select the EM Console Service Test, and click the Edit button. Make sure you have the “Access Login page” step selected, and click Edit. Change the URL to reflect your new OEM server, and save the changes.
Remove Previous OMS Server From OEM
Stop the agent on your original OMS server.
oracle$ $AGENT_HOME/bin/emctl stop agent
Remove the host target where your original OMS ran. Then remove the agent target.
One Last Bounce
Finally, bounce the whole thing one last time, then start it back up. All green.
Conclusion
I would prefer a simpler process to migrate the EM12c stack to a new server, but this works. If you find yourself in a similar position to mine, I hope this helps you. I’ve spent a lot of time working in EM12c so I feel capable to diagnose and resolve issues encountered during the process, but if you run into problems do not hesitate to contact Oracle Support and file a service requests. If you want your system to stay supportable, stick with the experts and just use blogs as a guide to get started. Good luck.
Pardy DBA 
Pingback: How to migrate EM12c R3 OMS and repository to a...
how are you using BI publisher with OEM. are you writing your own scripts on management views to get the required data
Hi Mahesh,
Thank you for reading. Unfortunately I have not had the time or need yet to write any BI Publisher reports. I have attended a few webinars, and you may want to check out the recordings of the webinars that Blue Medora’s Kyle Wassink provided. He gave a brief introduction and demonstration of some sample reports. I believe recordings are available from http://www.bluemedora.com/webinar-oracle-enterprise-manager-12c-training-bi-publisher-advanced-data-modeler and http://bluemedora.com/webinar-oracle-enterprise-manager-12c-training-bi-publisher-desktop-application .
Reblogged this on Thoughts from James H. Lui and commented:
Very useful, even if you’re moving parts of OEM 12c from one host to another, or modifying the filesystem mounts.
Pingback: Move EM 12c OMS and Repository to a new host (1/2) | Qite
Thank you for such detailed procedure. My issue is simpler (I guess), but I am new to EM 12C. We need to rebuild the server to use bigger disks in RAID 0. This will require to re-install the operating system on the new drives. Can I take a full backup of my /u01 filesystem where all 12C and it’s database installed, then restore it after OS is installed (name and IP will not change)? Is there any files under / that I need to backup too? or I have to re-install the EM 12C?
Hi Sam,
The exact answer here would depend on which version of EM12c you are using: R1, R2 or R3. The process has changed a little bit with each release. It also depends on whether or not you need your EM12c deployment to be supported by Oracle or not. If this is a test deployment, I think you would probably be fine just backing up /u01 and restoring it onto a new OS with the same hostname and IP. There are definitely a few files that get written under / that you will want to deal with, such as the /etc/init.d/gcstartup init script used to start/stop EM12c at boot/shutdown and probably also your /etc/oraInst.loc and perhaps your oraInventory depending on where it is stored.
If this is a production deployment, I hate to say it, but I think the right thing to do is to export your OMS configuration and then do a software-only installation followed by restoring from the OMS configuration export. For an idea of how to handle this, take a look at http://docs.oracle.com/cd/E24628_01/doc.121/e24473/ha_backup_recover.htm#BABFJFJH , specifically section 31.6.4.3 (“Single OMS, No SLB, OMS Restored on a Different Host using the Original Hostname”). Note that this section refers to section 31.6.3.1 which does clearly say that if you are recovering to the same host, you can just use a filesystem backup instead of re-installing. I would probably consider this to be “the same host” but I am not sure what Oracle support would say.
Good luck! My guess is just backing up /u01 and then restoring it will work, as long as you include copies of /etc/oraInst.loc and the startup scripts from /etc/init.d, but I would always work with Oracle support for something mission critical.
Excellent article at a much needed time. we are in the process of migrating our OMS to a new host in another datacentre. this will be very useful.
wanted to confirm one thing, if we want to repoint agents from one host to another host do we need to unstall the agents in the old host and reinstall them or is there a simple way to do it.
please suggest.
Your articles are of great help.
Hi Mahesh,
Thank you very much!
No, you do not need to uninstall the agents and reinstall them, as long as your OMS has been migrated successfully. You should be able to point your existing agents to the new hostname using something like:
$AGENT_INSTANCE_DIR/bin/emctl secure agent -emdWalletSrcUrl “http://newhost:4890/em”
That was one of the final steps in the migration that I did and that part worked just fine for me. You will have to configure an agent registration password in the OMS and provide that password during the re-secure step.
This will ONLY work if you have migrated your existing OMS and repository to the new host. If instead the new host in the other data center is running a new, fresh install of EM12c that does not have targets/agents configured, this will NOT work (in that case you need to remove the agent and deploy it again from the new OMS).
Good luck!
Thanks Brian. appreciate it.
Very usefull information !
Thank for sharing your work , is of great help
Hi Brian,
Thanks for such an exhaustive writeup. I am doing a similar activity, but the target OMS_HOME is not same as the source. Is there any way out for the same?? please help out.
Thanks,
Rajesh
Hi Rajesh,
I am not sure I understand your question completely. Do you mean that the path to the OMS_HOME is in a different location on the target server than the path to the OMS_HOME on the source server?
Unfortunately I have never tried to migrate a server that way, and the documentation says several times that the OMS_HOME location is fixed and cannot be changed. My suggestion would be to create symlinks on the target server so that a path matching the original OMS_HOME path on the source server can be used to find the same directory.
For example, if your source OMS was installed in /u01/app/oracle/Middleware, but your target server’s OMS needs to sit in /usr/local/oracle/Middleware, I would create directories /u01/app on the target server and add a symlink /u01/app/oracle that points to /usr/local/oracle.
When performing the software-only install on the target server before migrating, you should already have those symlinks in place and when the installer asks you for directory paths to use, enter the original directory path as used on the source OMS. If you have already done the software-only install on the target server with a different path, I would remove it and then start over with the original path.
If this does not work, I think you will need to file a service request with Oracle for guidance from support.
Good luck!
Hello,
thank for the great documentation in the first place.
Could it be possible that there is a bug or something missing in the script to identify the installed plugins. When I try to run this script I always getting an error message about a missing right paranthesis. Unfortunately my SQL knowledge is allmost not existing
Hi Martin,
I’m glad to help! I did just copy and paste that SQL script from my blog into a query as SYSMAN against my EM12c repository, and it ran successfully on my system.
However, I looked at the PDF documentation found at https://docs.oracle.com/cd/E24628_01/install.121/e24089.pdf and if I copy/paste the script from there, I receive the same error as you:
ERROR at line 4:ORA-00907: missing right parenthesis
I compared the two scripts, and it looks like the version in the PDF document contains an extra line break that should not be there. The portion of the SQL that reads:
decode(su.aru_file, null, 'Media/External', 'https://updates.oracle.com/Orion/Services/download/'||aru_file||'?aru='||aru_id||chr(38)||'patch_file='||aru_file) URLThis should all go on one line when you run the query. There should NOT be a line break/newline character where I see one in the PDF:
decode(su.aru_file, null, 'Media/External','https://updates.oracle.com/Orion/Services/download/'||aru_file||'?aru='||aru_**BAD LINE BREAK**
id||chr(38)||'patch_file='||aru_file) URL
I hope this helps!
Good luck!
-Brian
Thank you Brian for this detailed article. We are planning on doing an EM12c migration soon. While many of your steps will not apply as the new hosts we are migrating to will use a different OS and database version (moving from Solaris and 11g to RHEL and 12c), many of the steps you undertook and documented are still very useful.
Hi Brian
Thank you for the well written article. Like Omer (Sept 17/15 posting), I am migrating EM12c from a Solaris host to RHEL host. Your steps and comments will be helpful. Quick question: can the OMS configuration backup file (generated by exportconfig utility) be imported into an OMS on a different o/s?
Joanne
Hi Joanne,
I’m glad this could be helpful for you. Unfortunately I do not know whether or not you can transfer the exportconfig OMS configuration backup file to an OMS on a different operating system and then import it successfully. I suspect that this will not work, based on the idea that adding a secondary OMS to an existing installation can only work if the second OMS host runs the same OS as the first OMS. I believe that process uses an exportconfig to accomplish its task.
I’ve just put out a question on Twitter to see if I can get any clarity on a cross-platform exportconfig/importconfig OMS migration, I will reply again if I hear anything back. Your best bet is probably to file an SR with Oracle Support so that they can guide you through the process if there is one, unless you have the option of discarding your historical data and starting with a fresh OEM installation on the RHEL host.
Great question, I wish I could answer it.
-Brian
Hello Brain,
Nicely articulated, appreciated all hard work done to create the steps!
Anyone some migration from Solaris 11 to rhel 7 for 13c oms/repository?
I have a requirement to move the database and oms from Solaris 11 to rhel 7 both are running same host .
Would you please share any document or a thought?
Just reading and grasping all the details took me so much effort, so i could imagine what it took you to write such a detailed procedure. Big thanks for the share, much appreciated.
Foued
Hi Mr. Pardy,
These steps helped a lot. Thanks for a great article.
Regards,
Sethu
Hi Mr. Pardy,
I was able to migrate my OEM 12c R3 from HostA to HostB with the help of your article above. Very informative. But I am having an issue, the “Oracle Fusion Middleware Farm” and “Oracle Weblogic Domain” targets are still showing as running on HostA. I tried all the steps as explained above. The Configured agent shows that of HostB. Can you please help me?
Thanks,
Sethu
Hello Sethu,
The step where you run “Refresh WebLogic Domain” should have resolved that for you, but when I did this migration I had to run the Refresh WebLogic Domain step more than once before it actually identified all of my changes. Did you make sure to follow this step?
“Login to the OEM GUI running on the new server and navigate to the WebLogic Domain target for the Cloud Control domain. In the Target Setup -> Monitoring Credentials section, update the Administration server host value to the new server name, then hit OK. Then execute a Refresh WebLogic Domain, selecting Add/Update Targets, to move all WebLogic targets to the new central agent.”
If you did try this and it did not work, I would suggest stopping your EM12c OMS, starting it up again, and then repeating the Refresh WebLogic Domain, making sure that you modify the administration server host value in the WLS domain’s Monitoring Credentials section. If this still doesn’t work, there was one other thing I did that seemed to help:
“The GCDomain Oracle WebLogic Domain target did not get moved to my new agent. If this happened to you, go to the target home page and select the Modify Agents menu item. Click Continue, then find GCDomain in the list, scroll to the right, and assign the new OMS server’s agent as the monitoring agent for this target, then click the Modify Agents button.”
It seems like there may have been some bugs left over that made the process of moving the WebLogic monitoring to the new agent more difficult than the rest of the process.
Good luck!
-Brian
Hi Brian,
That was a prompt reply. I really appreciate for the help with the article as well as support with a long reply. So kind of you.
As explained, I already have done the below steps multiple times
1. “Login to the OEM GUI running on the new server and navigate to the WebLogic Domain target for the Cloud Control domain. In the Target Setup -> Monitoring Credentials section, update the Administration server host value to the new server name, then hit OK. Then execute a Refresh WebLogic Domain, selecting Add/Update Targets, to move all WebLogic targets to the new central agent.”
2. Go to the target home page and select the Modify Agents menu item. Click Continue, then find GCDomain in the list, scroll to the right, and assign the new OMS server’s agent as the monitoring agent for this target, then click the Modify Agents button.”
The agent gets changed to HostB but the host column still shows HostA and there is no way to update that. This is true for “Oracle Fusion Middleware Farm” and “Oracle Weblogic Domain”
But nothing seems to help. Have opened a ticket to MOS also but it is taking a long time for resolution.
Thanks,
Sethu
Unfortunately I no longer have an EM12c system available, only EM13c, so I cannot test the situation that you have encountered. I think that a service request with MOS is probably your best chance for a fix, or you could possibly try posting this situation in the OTN forum for Enterprise Manager. I’m sorry I do not have any other useful suggestions.
Hi Brian,
Very useful article for migrating OMS. What is the reason behind using http port to secure the agents? Can’t we use htpps port? For us the http communication is closed. Also can we use agent deploy or mass gold image deployment from OEM console?
Regards,
Kotesh
Hello Kotesh,
That is a good question. You MAY be able to use the HTTPS port, but the official documentation from Oracle instructs us to use the HTTP port. As I recall, when I performed this migration before, I switched from a third-party SSL certificate on the OMS to a certificate that I generated myself. Before the agents would recognize the new certificate, I had to make a copy of my root certificate available to the agents. I wanted to keep the process in this post simple, so I left that part out since it would not apply to every migration.
I believe that if you are not making that kind of certificate change during a migration that securing the agents with the HTTPS port instead of HTTP will work but I cannot be sure of that.
My apologies for the delay in getting back to you!
-Brian
Hi Brian Pardy,
This is a nice document. I am new to this and have a question on the db clone instruction above. On the new server, does Oracle software only need to be installed first before doing the cloing?
Thanks.
BeeSeng Lee
Hello,
Thank you very much, I hope the document is useful for you. You are correct that the Oracle software needs to be installed first on the new server before cloning. It needs to be the same release of Oracle with the same patch level. I found a couple other setup things that I needed to do before cloning that may or may not apply to your environment. Specifically, make sure to create all directory paths to hold your data files, control files, audit, diagnostic_dest, and redo logs on the new server before cloning, or you may experience failures during the clone step (such as when Oracle attempts to create a control file or a data file in a directory that does not exist). Everything went much better for me when I set up the new server with all of the same directory paths as the old server.
Good luck!
-Brian
Hi Brian,
Great article….
I have a similar requirement of migrating OEM12cR4 from one host with RHEL5 to another host with RHEL6.7, OEM Repository Database is in a different machine. In our case can we simply start with software only install and export existing OMS and import in new host ?
What all steps need to be followed..
Thanks in advance for your guidance.
Regards,
Mahesh Atnoor
Hello Mahesh,
Thank you very much! I believe that with your configuration you should be able to start from the section I have labeled “Installing OMS On A New Host” and then follow all steps from there. Make sure that you have collected the correct versions of all of the plugins running on your RHEL5 OMS server so that you can install them when you run PluginInstall.sh on the new RHEL6.7 OMS server.
All of my systems run with the repository database on the same server as the OMS, so I have not tested the configuration you run, but I do think it will work.
If your new OMS server does not use the same hostname as your old OMS server, you will have to re-secure all of your agents to point at the relocated OMS after the migration completes. I did not include that step in this post, because I kept the same hostname. See, for example, the end of section 19.6.4.2 in the EM12cR5 documentation https://docs.oracle.com/cd/E24628_01/install.121/e24089/ha_backup_recover.htm#EMADV10752 for instructions on re-securing agents to a new OMS hostname.
Good luck!
Hi Brian,
Thanks for the prompt reply.
Yes, in our case both the host names are different.
Will follow this article for our migration… Thanks for the help.
Regards,
Mahesh Atnoor
Hi Brian,
Like all of the previous comments, you write up for the migration is great. I am getting ready to procede with my migration. I have two questions. The first, I have two OMS’s which are behind a Secure Load Balancer(SLB). Are there any particular issues/command/concerns I should have about this?
The second question is that we have applied third party certs for the 2 current OMS’s, do I need to apply the certs to the “new” host?
Thanks Dan
Hi Dan,
First, thank you for the kind words! Second, I haven’t had a chance to work with a multiple OMS setup, so i can only venture a few guesses based on documentation and other blogs I’ve read, not actual experience.
But with that said, there are definitely some specific changes needed for a 2-OMS setup. The exact path to take depends on many things: if the repository DB is moving, if the primary OMS is moving, if host names are changing — OEM has some really good documentation for each different scenario in the advanced install/configuration guide (see https://docs.oracle.com/cd/E24628_01/install.121/e24089/ha_backup_recover.htm#EMADV11339 for 12cR5, but use the docs for your exact release if you don’t run 12cR5).
I would plan to reapply those third party certs on any new OMS hosts, especially if you do a software only install on a clean system and not restoring a filesystem backup.
As far as other advice, I would just say take it slowly, have backups, and don’t be surprised to see that old hostname in unexpected places years from now, I did.
-Brian
Brian,
Again, thank you. The repository DB is not moving, the primary OMS is moving, and with new hostnames.
I will read through the documentation.
This should be fun.
Dan Clark
Brian,
I got the new OMS host installed with the same EM software and the plugins as the original host. I am now trying to install the two patches I need to get the new host up to date. I am trying to install the patch via opatchauto apply -standby and I am getting prompted for the weblogic admin URL, username and password. I thought I could by pass this. Do I need to to provide this? Is there a way around that?
Again, thanks for your help.
Dan Clark
Hi Dan,
I think I’m a bit out of my element here, having never used opatchauto with the standby flag. As far as I know opatchauto always needs to contact the admin server, but it wouldn’t yet be available on the new host and it probably shouldn’t contact the old host’s admin server.
Do the patches you need to install happen to have opatch napply install documentation? Maybe you can get around this by avoiding opatchauto in favor of good old opatch. I would also make sure that you have the latest versions of opatch and opatchauto just in case, or at least the minimum releases noted in their docs.
This might be worth asking on the OTN/MOSC forums to get some more input.
Good luck!
Please help with my problem:
I try to do omsca recover but I get error message: failed to connect to repository database.
I know the problem. The config. assistant tries to connect to the source server database referred in the backup file. But I have already cloned the database to the target server.
So how can i force the assistant to take the target server database instead to avoid the error message ?
Hello Helmut,
If I understand correctly, it sounds like you are trying to move both your repository database and your OMS at the same time. I do not believe that can be done successfully. I have always first moved the repository database, then reconfigured the OMS to use the new repository database. It often causes big problems when trying to move both at the same time.
The omsca recover command supports the “-REPOS_CONN_STR” commandline argument. See section 19.7.2 in https://docs.oracle.com/cd/E24628_01/install.121/e24089/ha_backup_recover.htm#EMADV10764 — the -REPOS_CONN_STR argument may give you exactly what you need to finish restoring your OMS. I have never tried that though.
If you are NOT moving your OMS, and are only moving the repository database, then instead of running “omsca recover”, try running this instead (using the correct values for your new host, new port, and new SID) :
$OMS_HOME/bin/emctl config oms -store_repos_details -repos_conndesc “(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=newhost)(PORT=1521)))(CONNECT_DATA=(SID=emrep)))” -repos_user sysman
If you ARE moving your OMS, and also moving the repository database, and the -REPOS_CONN_STR commandline argument did not work, I think the best thing to do would be to start the repository database again on the original source database server, then proceed with the omsca recover command. Hopefully your OMS will be able to connect to the old repository database and get itself back up and running. Once the OMS is running again, then go ahead and shut it down, clone the database again to the new target server, then use the “emctl config oms -store_repos_details” command to point the OMS to the new database target server.
If this OMS monitors your production environment I would also suggest filing a service request with Oracle to get their assistance, especially if you do not have good backups of the OMS server filesystem and repository database.
Good luck!
Hi Brian,
we are using OEM12cR5 (Single OMS, No SLB) Our DB repository database and OMS are on different servers. We are planning to migrate our OMS to the server where repository database is currently installed.
1. Do we need to perform any additional steps other than those mentioned in this blog? We will be starting from installing new OMS software only on the new host (db server) and will skip the DB migration part.
2. Do we need to re-secure agents for all targets (more than 300) configured in our environment since the OMS host name will be changed (it will now be same as DB server name) or do we need to re-secure just the agent on the new OMS host?
Hello Tim,
First I apologize for not seeing and approving your comment until now, it got caught in the spam filter and I only saw it today.
To answer your questions as best I can:
1) I believe you are correct. You should be able to completely skip the DB migration steps and perform only the steps starting from “Installing OMS on a new host”. I would suggest you also review the “OMS Recovery Scenarios” section in the EM12cR5 documentation, especially section 19.6.4.2 since it covers your exact scenario with the single OMS and no SLB. Make sure you run an “emctl exportconfig oms” on your original OMS server before shutting it down. My post here was for 12cR3 but I do not think very much changed between R3 and R5 so the general process should work the same.
2) Unfortunately I do believe you will have to re-secure all of your agents, or else they will continue trying to connect to the old OMS hostname. If you are not already using secure/HTTPS for agent to OMS communication you may be able to use some DNS or /etc/hosts tricks to avoid having to immediately re-secure all of your agents, but I would definitely plan to re-secure them and use secure/HTTPS agent to OMS communication. Do make sure that you configure an agent registration password in the OMS so that you will have the details you need to re-secure the agents.
Again I apologize for not seeing this until today. Good luck!