Finding the AWR Warehouse link

Configuring AWR Warehouse (AWRW) in EM12c

Oracle Enterprise Manager 12cR4 introduces the new “AWR Warehouse (AWRW)” feature, permitting administrators to consolidate AWR statistics from many individual databases managed by OEM into a single AWRW repository database.

As with all Oracle features, you must pay attention to licensing here.  I will not discuss licensing other than to point you to the relevant documents which you must read and understand yourself: Oracle Database Licensing Information 12c Release 1.

Documentation

At the moment official documentation appears limited to MOS note 1907335.1 and one section of the Oracle Database 2-Day Database + Performance Tuning Guide.  Get familiar with it.

Prerequisites

Repository Database

You must use Enterprise Edition for the AWRW repository database.  You must use version 12.1.0.2 or higher, or version 11.2.0.4 with patch 18547891 applied. Oracle recommends you use a database not used for any other purpose. I strongly agree with that recommendation.  Do not use your OEM repository. Note that I had to enable the diagnostic and tuning packs on the AWRW repository database by setting the control_management_pack_access initialization parameter to “DIAG+TUNING” before EM12c would allow me to select it for the repository.  I cannot reiterate enough how much I wish Oracle would explicitly state that users may enable management packs on their limited-use repository databases that support EM12c, RMAN catalogs and AWRW, but only a sucker expects license clarity out of Oracle.

I have selected 11.2.0.4 with patch 18547891 for my AWRW repository.

Oracle Enterprise Manager

You must use Oracle Enterprise Manager 12cR4 (12.1.0.4), and your OMS must have at least the August 31, 2014 bundle patch (19391521, or a later bundle patch) applied.  Your agents must run version 12.1.0.4.2 or later (requiring patch 19051570).

Licensing

Double check your licensing one more time.  Do not use features you have not licensed or you will pay a lot of money once you get audited, and you will get audited.

Configuration

For the purposes of this post I will skip the database installation and configuration steps.  If you have not yet gained proficiency with base installation and configuration tasks, you should probably gain some experience there before diving in to the AWR Warehouse.  Install a database of the appropriate version and register it with EM12c.

Planning

Think about your architecture.  With the recent release of AWRW functionality, some rough edges still exist.  These will probably get cleaned up over the next few releases but they took me by surprise and I have not seen them documented anywhere.

Oracle Enterprise Manager Agent Considerations

Do you use a separate dedicated user account on your servers to run the OEM Agent?  I do. Your AGENT_INSTANCE_DIR will get used by AWRW as a place to hold Data Pump output containing each source database’s AWR data.  I had to make this directory group writable by the dba group.  You also need to make sure the volume where this directory resides has enough free space to hold AWR extracts, which end up quite large on a busy system.  You may need to add more space if you keep your agent on a dedicated filesystem, as I do.

Do you run multiple instances under isolated accounts that don’t share a group membership?  You will probably need to create a group they all share that can write to the AGENT_INSTANCE_DIR.

Preferred Credential Considerations

AWRW strongly depends on the preferred credentials set for a database instance by the user that adds the database to AWRW.  If you already heavily use preferred credentials and want to use a different preferred database login for AWRW extraction compared to your usual DBA activities, you may elect to create a dedicated EM12c administrator to maintain AWRW to avoid conflicts.

The AWRW extraction user in the target database must have the DBA role and must also have an explicit execute grant on package SYS.DBMS_SWRF_INTERNAL.  I have chosen to use the SYSTEM account, to match my other preferred credential usage, but a more secure setup would use an account dedicated to this task.

Space Considerations

Take a look at how much space AWR consumes in your SYSAUX tablespaces already.  Your AWRW repository will need at least this much space, multiplied by however long you plan to keep these AWR snapshots around.  This will get very large, very quickly.

Added 20140912: I highly recommend that you disable data file autogrowth on your AWRW repository database.  I experienced repeated hangs until I determined that my jobs continually got stuck when SYSTEM or SYSAUX nearly filled and they sat there waiting on data file operations I/O as the system failed to resize the data files or identify a deadlock.  Do not rely on data file autogrowth, at least when using an 11.2.0.4 AWRW repository.

Initialize The AWR Warehouse

To begin configuring the AWR Warehouse, you must login using an EM12c super administrator account, like SYSMAN.  Once logged in, go to the Databases target list screen.  Unfortunately for this purpose you must use the “Database Load Map” form of the screen and not the infinitely more useful “Deprecated Search List (with Metrics)” that I have up on screen 99.9% of the time. Click the Targets menu, select Databases from the submenu that appears, and then if you do not see a “Performance” menu, enable the “Database Load Map” radio button.

Click the Performance menu and select the “AWR Warehouse” item.

Finding the AWR Warehouse link

This button makes things happen

At this point, if you used a super administrator account, you should see a summary screen that briefly describes the feature with a link to click to begin configuration.  If you don’t, log out and come back with the SYSMAN account.

Begin AWRW Configuration

Click Configure to continue

The next screen offers a search box to select the database to use as your AWRW repository and the usual credential selector.  Select the correct database, choose a database credential (I first selected SYSTEM, which failed, so use SYS as SYSDBA) and provide host credentials.

Database Selection

Rough edge: no warning that you must use SYSDBA

Once you click Next, the tool will pop up a dialog box warning you to make sure that your repository database has the necessary patch installed, and then asks you to select how long the system should keep AWRW data.  You can also select a staging location for AWR data extract storage prior to data loading.

Repository Configuration (Continued)

Diamonds and AWR Warehouses are forever

Click Submit on this screen and OEM will submit a job to initialize the AWRW repository.  To find this job later, if needed, go to the advanced job activity page and search for jobs of type “dbSetupCAW”.  The job should complete successfully if you have done everything correct so far.  On my system it only took six seconds, so just wait a moment and then reload the page, which should now look like this.

Repository Ready

That was easy

Click on the database icon at the upper left to switch away from the repository configuration tab to the database selection tab.

Database Selection

No data yet

As of this point you no longer need to use the SYSMAN account.  I switched back to my regular account, then returned to this screen.

Click the Add button to begin adding your first database(s). OEM will prompt you with the usual target selection screen.  Choose one or more databases and then click the Select button.  AWRW will NOT prompt you for credentials at this time.  Instead it will silently use the database host and normal database user preferred credentials you have established for the database target.  Another rough edge I expect to work better in future versions.  AWRW will perform some initial validations of those credentials to make sure that the database user has the DBA role and the previously mentioned execute grant on SYS.DBMS_SWRF_INTERNAL.  If you have missed any of these steps OEM will tell you about it and prevent you from adding the database.  Again, later I expect this to include an automated setup that will fix those issues.

First Target DB

I can’t show you the name

At this point you can just walk away and within about 24 hours you should have AWR data loaded into the warehouse.  If you feel impatient, click on one of the lines for a database to select it, then choose “Upload Snapshots Now” from the Actions menu.  This will submit a job to extract and load the AWR data, which you can find later under the job type “Run AWR Extract, Transfer and Load”.  In the background, this job extracts AWR data to the AGENT_INSTANCE_DIR on the target database’s server, compresses the data, transfers it to the staging area defined during AWRW repository setup, then loads the transferred data into the consolidated AWR Warehouse repository.

Loaded

One database in there. So many to go.

Summary

The size of and load on your selected database, along with the amount of AWR history you keep online, will influence how long each load takes.  My first load appeared to hang, with the AWRW repository database full of sessions waiting on enq: HW contention and buffer busy waits.  I ended up doing a shutdown abort and following the workaround instructions in MOS note 1912230.1.  I do not know if I truly needed to do this or not, but the symptoms sounded similar.  I’ve also noticed that some limits appear to exist.  I keep 42 days worth of hourly snapshots in each AWR, and my initial load only picked up 20 days / 500 snapshots.  This may represent rate-limiting as a way to trickle-load the AWRW, or it may mean AWRW does not yet play nicely with large AWR data.  Time will tell, and I fully expect future versions to shake out any bugs and to hold the DBA’s hand a bit more through the process.

I hope to cover using AWRW for performance tuning in a later post and I look forward to comments.

More Information

See these other fine posts for more information.

Walking through a simple substitution cipher

While reading The Security Dialogue, I noticed the code contest and decided to give it a shot.  Here I present a way, one way of many, to solve it.  I enjoy solving newspaper cryptograms but I don’t claim to have any real cryptanalytic experience of any kind so take everything with a big grain of salt.

Given the following ciphertext, and assuming a simple substitution cipher:

jdc9)c9)4ds)9sz21x)z2xs)z214s94!))ud25vx)-25)es4)4dc9)8ced4)q1x)zq1)stqcv)ts)q4)9z8c6s1vfc1e[etqcv!z2t)wc894@)-25)7cvv)es4)q)}+&)ecw4)zq8x)42)Gtq=21!z2t!))jdq1f9)w28)3vq-c1e!

How to crack it?

First make some assumptions. At some point if you don’t get the unencrypted cleartext you may need to revisit these assumptions, but you have to start somewhere.  Knowing your target makes breaking codes much, much easier.  Sometimes you will gain more by spending a few hours researching rather than staring at the cipher.

I made the following assumptions:

  • Scriven truthfully relayed that he used a substitution cipher
  • The message consists of one or more grammatically proper sentences in English.

Start by counting the frequency of each symbol in the ciphertext. You can do that manually with a message this short but I wrote some basic Perl code to do it.  Run the code, paste in the ciphertext and hit ctrl-D to end, and it prints the character frequencies:

#!/usr/bin/perl 

until(eof(STDIN)) { $ch = getc(STDIN) }
  continue { $ch !~ m/\n/ && ($c{$ch} = defined($c{$ch}) ? $c{$ch}+1 : 1) }

print "$_\t$c{$_}\n" foreach (reverse sort {$c{$a} <=> $c{$b}} (keys %c));
)       31
c       12
4       11
2       11
q       10
s       9
1       9
z       8
9       8
v       7
e       7
t       6
d       6
8       5
!       5
x       5
5       3
-       3
w       3
f       2
j       2
}       1
G       1
&       1
+       1
3       1
6       1
[       1
@       1
u       1
7       1
=       1

The ‘)’ character appears 19 more times than any other symbol in the message and seems distributed throughout the message in a way that it could represent a blank space between words. I will assume for now that ‘)’ = ‘ ‘. Having ciphertext broken up into words makes the rest of the work infinitely easier, so rewrite the message with this change.

jdc9 c9 4ds 9sz21x z2xs z214s94!  ud25vx -25 es4 4dc9 8ced4 q1x zq1 stqcv ts q4 9z8c6s1vfc1e[etqcv!z2t wc894@ -25 7cvv es4 q }+& ecw4 zq8x 42 Gtq=21!z2t!   jdq1f9 w28 3vq-c1e!

Of interest when you do this, ‘))’ appears twice, both times preceded by ‘!’. Going from the assumption that ‘)’ = ‘ ‘, this could indicate what we in the US currently call “French spacing”, or using two spaces after the end of a sentence instead of just one. Though considered deprecated in American English style guides, many people still use it (including me), and autocorrect on mobile devices even takes advantage of that to turn a double tap on the space bar into a period followed by a space and then a capital letter. This adds strength to the assumption and indicates we likely have three sentences. I don’t yet have a reason for why the assumed sentence-terminator ‘!’ sometimes appears in a word, but I will go with it for now.

With the ciphertext letter frequencies in hand, now you need English text letter and word frequencies. You can use ETAOIN SHRDLU as a mnemonic for the most frequently used letters in descending order if you want to keep things simple.

Look at the (assumed) words in the ciphertext. Make lists of all the words with only one letter, only two letters, only three letters, only four letters.  Note any that appear twice or more, and any repeated strings. I made this list by hand but you can write code to do it.

1 letter words: q
2 letter words: c9 ts q4 42
3 letter words: 4ds -25 es4 q1x zq1 -25 es4 w28
4 letter words: jdc9 z2xs 4dc9 7cvv ecw4 zq8x

No repeated 2 letter words
Repeated 3 letter words: -25 es4
No repeated 4 letter words

Repeated digrams (2 letters): c9 jd 4d z2 25 1x zq q1 21     
Repeated trigrams (3 letters): -25 es4 dc9 c1e z2t
Repeated fourgrams (4 letters): !z2t

Repeated letters: vv

Notice the single one letter word: q. In English this can only mean one of the words “I” or “a”. The frequency of ‘q’ in the ciphertext also indicates a possible vowel.

Look for repeated digraphs, pairs or triplets of symbols that appear next to each other frequently. I already noticed ‘!))’ which may mean ‘.  ‘, but I also see ‘c9′ three times. Twice it ends a four letter word, once it stands alone as a two letter word. The ciphertext starts with “jdc9 c9 4ds”, or a four letter word followed by a two letter word made up from the last two letters of the preceding word. In English, “This is” or “What at” or “That at” or even “Shit it” all fit that pattern and can fit grammatically at the start of a sentence. The ‘d’ in the third (three letter) word yields the cleartext ‘h’ in each case, as the second letter of “this”, “what” and “shit”.  Many three letter words have ‘h’ as their second letter and can fit in the sentence I have so far: “This is why”, “This is the”, “What at the”.  I will throw out “That at” for now because I don’t like to see both ‘j’ and ‘9’ meaning ‘t’, unless he decided to sneakily use different symbols for the upper and lowercase versions of the same letter.

So assume for now with some confidence:

')' = ' '
'!' = '.'
'd' = 'h'

For clarity, when I rewrite the text with my substitutions, I will use capital letters for cleartext and lowercase letters for ciphertext (though the ciphertext contains a single capital ‘G’ that I will ignore for the moment).  Rewrite the text with the three substitutions so far:

jHc9 c9 4Hs 9sz21x z2xs z214s94.  uH25vx -25 es4 4Hc9 8ceH4 q1x zq1 
stqcv ts q4 9z8c6s1vfc1e[etqcv.z2t wc894@ -25 7cvv es4 q }+& ecw4 
zq8x 42 Gtq=21.z2t.   jHq1f9 w28 3vq-c1e.

I wrote some simple Perl code to handle rewriting the ciphertext pasted into it, configurable by adding new substitutions to the code.  I will use this going forward instead of substituting manually.

#!/usr/bin/perl

$subst{')'} = ' ';
$subst{'!'} = '.';
$subst{'d'} = 'H';
# add more substitutions here following the same pattern

until(eof(STDIN)) { $ch = getc(STDIN) }
  continue { print defined($subst{$ch}) ? $subst{$ch} : $ch }

Time now to make some guesses.  Earlier I suspected the first two words may encode “What at” or “This is”, and I also know that ‘q’ must represent ‘a’ or ‘I’, so let’s have a look at the ciphertext with those changes. As a simple substitution cipher, no cleartext character can come from two different ciphertext characters, so assume ‘q’ means ‘I’ if ‘c’ means ‘a’, and vice versa (since both ‘c’ and ‘q’ cannot map the same letter).

All use the previous substitutions:

')' = ' '
'!' = '.'
'd' = 'h'

"What at":

'j' = 'W'
'c' = 'A'
'9' = 'T'
'q' = 'I'

WHAT AT 4Hs Tsz21x z2xs z214sT4.  uH25vx -25 es4 4HAT 8AeH4 I1x zI1 
stIAv ts I4 Tz8A6s1vfA1e[etIAv.z2t wA8T4@ -25 7Avv es4 I }+& eAw4 
zI8x 42 GtI=21.z2t.  WHI1fT w28 3vI-A1e.

"Shit it":

'j' = 'S'
'c' = 'I'
'9' = 'T'
'q' = 'A'

SHIT IT 4Hs Tsz21x z2xs z214sT4.  uH25vx -25 es4 4HIT 8IeH4 A1x zA1 
stAIv ts A4 Tz8I6s1vfI1e[etAIv.z2t wI8T4@ -25 7Ivv es4 A }+& eIw4 
zA8x 42 GtA=21.z2t.  SHA1fT w28 3vA-I1e.

"This is":

'j' = 'T'
'c' = 'I'
'9' = 'S'
'q' = 'A'

THIS IS 4Hs Ssz21x z2xs z214sS4.  uH25vx -25 es4 4HIS 8IeH4 A1x zA1 
stAIv ts A4 Sz8I6s1vfI1e[etAIv.z2t wI8S4@ -25 7Ivv es4 A }+& eIw4 
zA8x 42 GtA=21.z2t.  THA1fS w28 3vA-I1e.

Each of these seems like a start on a solution.  Where to go from here to give some weight to one choice or the other?  All three could produce a grammatical sentence given the first two words, though I’ve lost faith in “Shit it” at this point if I ever had any.

Take a look at the words where you almost have all of the letters translated, but not quite.  I see the original word “4dc9″ which we have translated as either “-hat” or “-his”, and we have the original word “jdq1f9″ which we have translated as either “tha–s” or “whi–t”.  That second one seems like a good candidate.  Now I need a word list. I will use a classic English word list from Donald E. Knuth.  You must use a word list appropriate for the cleartext you expect to find.  This would not help me for French text, nor would it help for government or corporate information which might contain many acronyms.

I have two possible six letter words identified: “tha–s” and “whi–t”.  Check the word list for words that match each pattern.  The following Perl command line will do it, assuming you have a word list file named ‘wordlist.txt’.

$ perl -ne 'print if m/^tha[a-z]{2}s$/' wordlist.txt
thanks
$ perl -ne 'print if m/^whi[a-z]{2}t$/' wordlist.txt
whilst

So only one word fits for each possibility.  I will go out on a limb and assume he used the word “Thanks” rather than “Whilst”.  I follow him on Twitter and I’ve seen him say “thanks”, but never “whilst”. Speakers of American English simply don’t use “whilst” very often.  Let’s take a look at the text if we assume the word “jdq1f9″ means “Thanks”.  We get two more letters, ‘1’=’n’ and ‘f’=’k’.

')' = ' '
'!' = '.'
'd' = 'h'
'j' = 'T'
'c' = 'I'
'9' = 'S'
'q' = 'A'
'1' = 'N'
'f' = 'K'

THIS IS 4Hs Ssz2Nx z2xs z2N4sS4.  uH25vx -25 es4 4HIS 8IeH4 ANx zAN 
stAIv ts A4 Sz8I6sNvKINe[etAIv.z2t wI8S4@ -25 7Ivv es4 A }+& eIw4 
zA8x 42 GtA=2N.z2t.  THANKS w28 3vA-INe.

Looking better here.  Three words possibly done and nothing else looks too wrong.  I want to get that third word, after “This is”.  So what three letter words match the pattern “-h-“?

$ perl -ne 'print if m/^[a-z]h[a-z]$/i' wordlist.txt
aha
chi
ohm
oho
phi
rho
she
shh
shy
the
tho
thy
who
why

Which of those words make sense in a sentence following “This is”?  Only “the”, “who” and “why”.  I lean towards “who” and “why”, but if a capital ‘T’ at the beginning of a sentence has a different symbol from a lowercase ‘t’ in the middle of the sentence, “the” may do it.  This gives me a few more combinations to test:

Using the previous substitutions:

')' = ' '
'!' = '.'
'd' = 'h'
'j' = 'T'
'c' = 'I'
'9' = 'S'
'q' = 'A'
'1' = 'N'
'f' = 'K'

"This is who":

'4' = 'W'
's' = 'O'

THIS IS WHO SOz2Nx z2xO z2NWOSW.  uH25vx -25 eOW WHIS 8IeHW ANx zAN 
OtAIv tO AW Sz8I6ONvKINe[etAIv.z2t wI8SW@ -25 7Ivv eOW A }+& eIwW 
zA8x W2 GtA=2N.z2t.  THANKS w28 3vA-INe.

"This is why":

'4' = 'W'
's' = 'Y'

THIS IS WHY SYz2Nx z2xY z2NWYSW.  uH25vx -25 eYW WHIS 8IeHW ANx zAN 
YtAIv tY AW Sz8I6YNvKINe[etAIv.z2t wI8SW@ -25 7Ivv eYW A }+& eIwW 
zA8x W2 GtA=2N.z2t.  THANKS w28 3vA-INe.

"This is the":

'4' = 'T' (lowercase t!)
's' = 'E'

THIS IS THE SEz2Nx z2xE z2NTEST.  uH25vx -25 eET THIS 8IeHT ANx zAN 
EtAIv tE AT Sz8I6ENvKINe[etAIv.z2t wI8ST@ -25 7Ivv eET A }+& eIwT 
zA8x T2 GtA=2N.z2t.  THANKS w28 3vA-INe.

The last one gives me words 3 (“the”), 10 (“this”) and 16 (“at”).  My word list does not contain “whis” so I will throw out the two previous tries and continue from here.

Word 6 (“z214s94″) looks interesting with the pattern “–ntest”.  Only one word fits that pattern: “contest”.  It doesn’t surprise me one bit to find the word “contest” in the cleartext.  Assign ‘z’=’C’ and ‘2’=’O’.

Using the previous substitutions:

')' = ' '
'!' = '.'
'd' = 'h'
'j' = 'T'
'c' = 'I'
'9' = 'S'
'q' = 'A'
'1' = 'N'
'f' = 'K'
'4' = 'T'
's' = 'E'

Add in "contest":

'z' = 'C'
'2' = 'O'

THIS IS THE SECONx COxE CONTEST.  uHO5vx -O5 eET THIS 8IeHT ANx CAN 
EtAIv tE AT SC8I6ENvKINe[etAIv.COt wI8ST@ -O5 7Ivv eET A }+& eIwT 
CA8x TO GtA=ON.COt.  THANKS wO8 3vA-INe.

That gave me words 5 (“contest”), 13 (“can”) and 26 (“to”).  Now I want to take a look at the last word, “3vq-c1e” which I so far have matching the pattern “–a-in-“. The final encrypted ‘e’ also serves as the first letter in the three letter word “-et” (word 21).

53 words match the “–a-in-” pattern.  Of those 53, 44 of them (83%) end with “ing”, and would yield “get” for word 21.  I’ll take a leap here and assign ‘e’=’G’.

Looking at that first sentence, if ‘x’=’D’ then “This is the second code contest.” That makes perfect sense.

Using the previous substitutions:

')' = ' '
'!' = '.'
'd' = 'h'
'j' = 'T'
'c' = 'I'
'9' = 'S'
'q' = 'A'
'1' = 'N'
'f' = 'K'
'4' = 'T'
's' = 'E'
'z' = 'C'
'2' = 'O'

Add our new letters:

'e' = 'G'
'x' = 'D'

THIS IS THE SECOND CODE CONTEST.  uHO5vD -O5 GET THIS 8IGHT AND CAN 
EtAIv tE AT SC8I6ENvKING[GtAIv.COt wI8ST@ -O5 7Ivv GET A }+& GIwT 
CA8D TO GtA=ON.COt.  THANKS wO8 3vA-ING.

That gives me words 4 (“second”), 5 (“code”), 9 (“get”), 12 (“and”), 21 (“get”).

Word 17 (“9z8c6s1vfc1e[etqcv!z2t”), or “sc-i-en-king-g-ai-.co-” looks suspiciously like our host’s email address that he provided in the contest description.  Let’s substitute the letters to complete that.

Using the previous substitutions:

')' = ' '
'!' = '.'
'd' = 'h'
'j' = 'T'
'c' = 'I'
'9' = 'S'
'q' = 'A'
'1' = 'N'
'f' = 'K'
'4' = 'T'
's' = 'E'
'z' = 'C'
'2' = 'O'
'e' = 'G'
'x' = 'D'

Completing his email address:

'8' = 'R'
'6' = 'V'
'v' = 'L'
'[' = '@'
't' = 'M'

THIS IS THE SECOND CODE CONTEST.  uHO5LD -O5 GET THIS RIGHT AND CAN 
EMAIL ME AT SCRIVENLKING@GMAIL.COM wIRST@ -O5 7ILL GET A }+& GIwT 
CARD TO GMA=ON.COM.  THANKS wOR 3LA-ING.

Definitely on the right track here.  I can feel that Amazon.com gift card for $25.  The message even seems to mention it “-ill get a — gi-t card to -ma-on.com”.  From here, one only needs to plug in the letters and symbols that make sense and finish stepping through the process.

New: The Data Driven Drinker

I have not blogged much recently. My database work has been distracted by Java programming and a crash course in DevOps, leaving me with little worth posting that a tweet couldn’t exhaustively cover.

That changes now. Time for a new blog series that I will call The Data Driven Drinker. I will acquire, imbibe and comment on alcoholic beverages and I hope you will join me. Expect a focus on scotch whisky, on local (Vermont, USA) products and on everything else interesting I taste.

Improving security in your web browsers: Firefox

(Update 20140730: If you disable RC4 ciphers as noted in the about:config section, Google’s YouTube product may no longer function.  As of roughly March or April 2014, Google has forced HTTPS on YouTube while at the same time they have not deployed any alternative ciphers for RC4 on googlevideo.com.  Respond to this issue as you choose; I generally do not use YouTube.  You may create another profile that permits RC4, or enable it on an as-needed basis, remembering to disable it later.  Perhaps an enterprising individual will create a plugin to do just that. Credit to @mincina for bringing this to my attention.)

(Update 20140110: Removed recommendation to enable security.ssl.enable_false_start, as it appears to be unsafe.)

(Update 20140107: Added recommendation to disable security.ssl3.rsa_des_ede3_sha.)

(Update 20131212:  I have revised my recommendations.  I now recommend Adblock Edge instead of Adblock Plus, and Disconnect instead of Ghostery, with the reasons noted inline below.

At this time Firefox has released version 26 which includes click-to-play functionality for Java and other plugin content.  This change will greatly enhance your security.  Upgrade now!)

Your web browsers implement poor security by default.  They do this, in large part, for interoperability reasons; if your just-downloaded new browser can’t connect to the sites you like to use, you either won’t use the browser or you’ll complain to the developers, and they don’t want to spend the time walking you through how to disable the specific security settings keeping you from using some random website that hasn’t upgraded their SSL implementation since 2002.

With effort and testing, you can significantly improve your security.  Don’t hold me responsible if this breaks your favorite site or eats all the food in your fridge, but if you want to step up and accept that security and convenience don’t go together, consider trying some or all of these steps to secure your Firefox browser.  I have Windows in front of me at the moment, but if you use a real operating system you can figure out how to perform the appropriate changes there.  Consider the fact that using Windows represents a greater security threat than almost anything else you can do.

Do note that even if you follow every suggestion I make on this page, you have not guaranteed security for yourself.  These steps cannot protect you from foolish decisions.  If, after doing all of this, you then proceed to visit some shady site and download a cracked version of some commercial software product, then execute it, you will get hacked, you will get compromised, you will get malware.

Why Security?

Only you know the adversaries you may have.  The malware spewed across the internet presents a risk to us all and these steps can help protect you from it.  But beyond that point, if you want to protect yourself from a determined adversary, then please only consider the steps I describe as a start.  If you work with confidential corporate documents, or if you work to promote human rights in repressive countries, or if you write news articles disclosing secret government projects, or if you run a hidden site selling drugs for bitcoins, you have a threat model much more complex than the average user.

Security Defined

One could write a book to define the word security.  Many have.  For the purposes of this post, I define security as protection against your own accidental mistakes, protection against common malware techniques and protection against an attacker with access to your network or the internet path between you and the sites you visit.  Further, I consider security to include not leaking unnecessary information about yourself or your browsing habits to third parties that want that information, such as advertisers.

Run A Current Browser

Using an old browser begs for trouble.  Just don’t do it.  For now I have Firefox 25 installed and everything I write here applies to this version and hopefully future versions.  Go to the Tools menu, select Options, then click on Advanced and select the Update tab.  Enable the radio button next to “Automatically install updates”.

Simple Steps

The steps described here shouldn’t significantly degrade your web browser experience but will improve your security quite a bit.  Everything in this section lives in the Tools->Options dialog box.  Open it up now.

Options: Tabs

If checked, uncheck the box next to “Show tab previews in the Windows taskbar”.  Windows has a history of buffer overflows in graphics handlers, and a specially crafted tab preview could potentially exploit this.  I do not know of this ever happening but no need to take the risk simply for some eye candy.

Options: Content

Check the box next to “Block pop-up windows”.  Compromised or otherwise malicious sites love to put up confusing pop-up windows saying “your computer has a virus” and other such nonsense.  The next time you go to a site that attempts to raise a pop-up window, Firefox will ask if you wish to allow an exception for that site.  If this happens on a site you need, allow the exception.  If a bad site can’t pop up a window to attempt to fool you, you won’t click on their shady links.

Click the “Choose…” button next to “Choose your preferred language for displaying pages”.  Make sure the contents of the language dialog box reflect only those languages you wish to read.

Options: Applications

Click through every row of this screen and use the drop-down menu on the right-hand side to select “Always ask”, so that Firefox will prompt to ask how (and more importantly, if) you wish to access embedded content like videos, music, PDF documents, etc.  This may get inconvenient over time if you access a lot of media, so later on, when prompted to select an application to view media, you may choose to select the “Do this automatically for files like this from now on” checkbox in the prompt but know that this reduces your overall security slightly.

Options: Privacy

Enable the radio button next to “Tell sites that I do not want to be tracked”.  This will cause your browser to send the Do-Not-Track header. Few webservers will respect this setting, but some will, so you get some small value here.

In the History section, select “Use custom settings for history” from the “Firefox will:” dropdown menu.  For the sake of convenience, go ahead and leave the checkboxes enabled for “Remember my browsing and download history” and “Remember search and form history”.  I recommend disabling them, but the convenience of having recently visited sites available outweighs the risk of having to search for a site repeatedly and possibly clicking on a malicious search engine result.

Go ahead and leave the checkbox enabled for “Accept cookies from sites”, or very few websites will work.  Set the “Accept third-party cookies” dropdown menu to “From visited”, NOT to “Always”.  Many sites will not work if you set it to “Never”, nearly every site will still work fine with it set to “From visited”.  “Always”, in this case, begs to be tracked by marketers.

In the “Keep until:” dropdown menu, select “they expire”.  Some people would recommend deleting cookies every time the browser closes, but you will lose the convenience of having sites recognize you when you want them to.  If you can tolerate that loss of convenience go ahead and select “I close Firefox”.

Check out the “Exceptions…” button near the “Accept cookies from sites” checkbox.  Here you can add exceptions to specify sites always allowed to set cookies, or never allowed to set cookies.  I love this feature.  I coded this feature into the text-based Lynx web browser back in 1999 and it pleases me that the GUI browsers picked it up.

Options: Security

Check the checkboxes next to “Warn me when sites try to install add-ons”, “Block reported attack sites” and “Block reported web forgeries”.

Uncheck the “Remember passwords for sites” checkbox.  If you permit the browser to store your passwords, anyone with access to your browser can retrieve your passwords.  I suggest only enabling this if you have taken the further step of encrypting your hard drive.  If you do enable it, make sure you also enable the “Use a master password” option and select a strong password.

Options: Sync

Do not use Firefox Sync.  This will simply spread your information out over more devices, increasing your risk.

Options: Advanced

On the “General” tab, check the box next to “Warn me when websites try to redirect or reload the page”.

On the “Data Choices” tab, uncheck everything.  All of these options share information with Mozilla and you do not want that to happen.

On the “Network” tab, check the box next to “Tell me when a website asks to store data for offline use”.  Most likely you do not actually want any sites to do this.

On the “Certificates” tab, click the “Validation” button and enable the checkboxes to use the Online Certificate Status Protocol to confirm certificate validity and to treat certificates as invalid when an OCSP server connection fails.  While not foolproof, this can help protect against invalid or compromised server certificates.

Intermediate Steps

If you have followed everything so far, you have improved your browser security.  Not enough, in my opinion, but perhaps enough if you plan to hand this browser off to your tech-challenged grandparents to use to look up recipes and email pictures of their grandkids.  If you have a decent comfort level with basic internet and browser concepts, continue on.

Install Add-Ons

Numerous add-ons available for Firefox can further enhance your security.  Here I will list the ones I consider most critical, along with some comments on configuration/usage for each of them.

Disconnect

Install Disconnect. This add-on identifies and blocks various web trackers embedded throughout the sites you visit.  Mostly analytics and marketing, rather than anything truly security related, but you don’t want any part of those either.  The developers have released the source code and development supported by donations.  It takes note of sites that host trackers but also host page elements that may cause a page to function incorrectly if blocked.

I previously recommended Ghostery for this purpose.  However, after witnessing a recent Twitter conversation involving one of Ghostery’s developers, I felt he represented the product poorly and lost faith in it.  Further, the company behind Ghostery includes many former ad-agency employees, providing another strike against it on top of their opt-in data collection.

Adblock Plus Adblock Edge

Install Adblock Plus Adblock Edge. Ads on webpages may not represent an obvious security issue, but I still consider blocking them appropriate for a secured browser.  When your browser loads an ad from a page the advertiser will know that somebody from your IP address viewed a page containing that ad, and depending on how the ad gets served up they may also learn the page you intended to view at the same time.  Further, traffic analysis of specially placed ads may reveal information about the sites you visit as ads typically do not use https connections, and if somebody with access to your network sees that you repeatedly load some specific ad that only appears on a particular site, they would then have strong evidence that you visit that site repeatedly.

Within the Adblock Plus Edge options, subscribe to EasyList EasyPrivacy+EasyList, Fanboy’s Social Blocking List and Malware Domains., and uncheck the “Allow some non-intrusive advertising” checkbox.  If you live outside the USA, subscribe to some of the additional filter lists dedicated to your region.

I have changed my recommendation as of December 12, 2013. Adblock Edge performs better and does not receive money from Internet advertisers to permit “some non-intrusive advertising”.

BetterPrivacy

Install BetterPrivacy. This add-on removes persistent Flash cookies, for which browsers generally provide no control mechanism.  Within the options screen, select the radio button for “Delete Flash cookies on Firefox exit”.  Select the checkboxes for “Auto protect LSO sub-folders” and “Notify if new LSO is stored”.  Check the box for “Disable Ping Tracking”.

Certificate Patrol

Install Certificate Patrol. This add-on stores all SSL certificates you encounter when accessing https sites, and notifies you when a site you connect to has changed certificates since your last visit.  A changed certificate may indicate an attempted man-in-the-middle attack that would compromise your encrypted session.  I receive a lot of false positives with this add-on, which defeats its utility somewhat, but I review every single change.  If you want to skip one of these add-ons, make it this one.  I haven’t convinced myself that I take enough care to actually identify a man-in-the-middle attack, and I can’t exactly call someone at Google every time their cert changes to confirm they meant to do so.

Ghostery

Install Ghostery. This add-on identifies and blocks various web trackers embedded throughout the sites you visit.  Mostly analytics and marketing, rather than anything truly security related, but you don’t want any part of those either.  Unfortunately some sites will not function properly with Ghostery installed, but it provides options to whitelist those sites or temporarily pause blocking so that you can easily determine if Ghostery has caused the page to fail.  I end up having to whitelist bank sites, WordPress, a few others, but for just clicking through search results, I love it.  It also has the ability to block advertising cookies.

I have changed my recommendation to use Ghostery as of December 12, 2013.  Please see the “Disconnect” section above for details on why I no longer recommend Ghostery.

Long URL Please Mod

Install Long URL Please Mod.  Shortened URLs suck.  You don’t know where they will lead, and if you take security seriously you probably won’t click on them.  This add-on expands short URLs for you so that you know where they lead and can make an educated decision as to whether or not you want to follow that link.

NoScript

Install NoScript. Perhaps the most important add-on to use. This add-on provides the ability to permit or reject active scripting to run on a per-domain or per-host basis.  It will, initially, block all JavaScript on every site, which will break large portions of the web for you.  In this case, as you find sites that don’t work, you use the button it adds to the browser bar to enable scripting (temporarily or permanently) for that particular site, reload the page, and everything should then function as intended.  Sites get classified into trusted (whitelisted), untrusted, and those that you haven’t yet evaluated.

As a bonus, it also provides protection against cross-site-scripting and clickjacking (where a malicious site overlays an invisible object over a page element, intercepting a click on that element as a click directed at the malicious site, allowing it to load a page/code/etc).

NoScript has numerous configuration options.  I recommend the following:

Do NOT check the “Scripts Globally Allowed” box, as this essentially disables the add-on and leaves you back in the usual situation of freely running all JavaScript submitted to your browser.

On the “Embeddings” tab, you can specify restrictions for untrusted sites that do not apply to whitelisted sites.  This gives you a chance to use paranoid settings, as you can always whitelist a site later.  I don’t want to make them so restrictive that I end up whitelisting every other site, so I don’t block frames, but I do block: Java, Flash, Silverlight, other plugins, audio/video tags, and font-face, and I also block every object coming from sites marked as untrusted.  I also enable “Show placeholder icon”, “No placeholder for objects coming from sites marked as untrusted”, “Ask for confirmation before temporarily unblocking an object” and “Collapse blocked objects”.  I also check the box for ClearClick (clickjacking) protection on untrusted pages.  Some whitelisted pages don’t work if I enable ClearClick protection for trusted pages, so I leave that one off.

In the “Advanced” tab, on the “Untrusted” sub-tab, check “Forbid <a ping…>”, “Forbid META redirections inside <NOSCRIPT> elements”, “Forbid XSLT” and “Attempt to fix JavaScript links”.  On the “XSS” tab, I check “Sanitize cross-site suspicious requests” and “Turn cross-site POST requests into data-less GET requests”.

NoScript can do even more than this, and you should look into the other options.  The configuration set I have described works well for my browsing habits.

WOT

Install WOT. This add-on uses a crowdsourced set of website rankings to provide you with a simple red (bad) / yellow (maybe bad) / green (good) ranking for every site you visit and all sites that appear in search results from Google.  It further takes advantage of blacklists published by anti-virus vendors and other independent sources to identify malicious sites.  You do not have to do so, but if you choose to create an account with them you can submit your own ratings.  WOT uses a complex reputation mechanism to determine how much weight to give your ratings when compiling them with others’ to determine a site’s overall rating; this helps prevent malicious individuals from installing the add-on and voting up a bunch of malware infested sites.

Expert Steps

Doing everything, or even some of the things, that I’ve listed to this point will greatly improve your browser security.  But you can do more.  At this point I will get into the weeds a bit and make some significant changes to browser operation.  These changes may (and probably will) cause problems accessing poorly configured sites, but if you use sites configured so poorly, maybe you shouldn’t.  I recommend, if you follow these suggestions, that you implement them one at a time, and test all the sites you consider most important.  If you change a dozen things and suddenly some page stops working, you won’t know what to undo to restore it to functionality.  As an example, while writing up this post I noticed that addons.mozilla.org started to throw intermittent SSL errors when I tried to connect to it.  Hitting reload would usually load the page just fine.  It turned out that disabling RC4 cipher suites for SSL negotiation caused that problem: apparently not all of the servers behind their load balancer have the same configuration, and some of them just don’t work if the client browser does not accept RC4.

about:config

Everything else happens in the about:config screen.  If you haven’t used it before, type “about:config” into your address bar and hit enter.  Click through the warning that says it might break stuff, but recognize they put it there for a reason.

Disable RC4

The RC4 symmetric cipher contains significant failings.  You should not use it.  In fact, if you admin any webservers, leave this blog now and go figure out how to disable RC4 on them.  Then come back and finish securing your browser.  If you need convincing, read this: “Attack of the week: RC4 is kind of broken in TLS“.

In the about:config page, type “rc4″ into the search bar and press enter.  You will see several cipher suites listed (with names like “security.ssl3.rsa_rc4_128_sha”).  Double-click on each of them so that the value field on the right reads “false”.  Your browser will no longer advertise willingness to accept RC4 as a component in an SSL connection.

Require TLS

Type “tls” into the about:config search bar and press enter.  Find the “security.tls.version.min” key, which defaults to 0, and change it to 1.  Set the “security.tls.version.max” key, which defaults to 1, to 3. [EDIT 20131112: I previously recommended 2 here, for TLS 1.1, thinking it would cause fewer connection failures than 3 for TLS 1.2. This won't be a problem once Firefox has fallback code from TLS 1.2. But if you are following these steps you should know how to debug and fix any connection problems you have.] For more information on these settings and what they do, see this link.

Disable additional insecure cipher suites

(Added 20140107) Type “rsa_des_ede3″ into the about:config search bar and press enter.  Find the “security.ssl3.rsa_des_ede3_sha” key and double-click it to set the value to false.  This will remove SSL_RSA_FIPS_WITH_3DES_EDE_CBC_SHA from the cipher suites for which your client will advertise support.  Thanks to Jeff Hodges for creating howsmyssl.com through which I noticed this item.

Other Settings

(This section edited on 20140110, after the comment below from Ismail Dönmez.  Please see that comment for a link to the Firefox bug database entry concerning security.ssl.enable_false_start.)

Type “security” into the about:config search bar and press enter.  Find the “security.ssl.enable_false_start” key and double-click it to set the value to true.  Do the same for “security.ssl.false_start.require-forward-secrecy”, “security.ssl.require_safe_negotiation”, and “security.ssl.treat_unsafe_negotiation_as_broken”.  Read this link for more information about these settings.

Conclusion

If most of your web browsing still works after configuring all this stuff, congratulations.  You probably browse safely enough that you don’t have much to worry about.  If you run into sites that don’t work with these settings, consider whether or not you really need to visit them.  Good luck!

SQL to query table size and DBMS_REDEFINITION progress

Like so many other Oracle DBAs, I need a script to query the total disk space used by an individual table, including the data, indexes and LOBs, that works whether or not the table uses partitioning.  I also wanted a script to monitor the progress of DBMS_REDEFINITION actions.  Here I provide a single script that does both.

Sample output during a DBMS_REDEFINITION run, with my SAP system name redacted:

SQL> @s
Enter value for segment: reposrc

ACTION          TARGET                              REMAINS  PROGRESS
--------------- ----------------------------------- -------- ---------------
Table Scan      SAP***.REPOSRC                      00:08:45 4.89%

SEGTYPE         SEGMENT                               SIZEMB TABLESPACE
--------------- ----------------------------------- -------- ---------------
1-TABLE         SAP***.REPOSRC                          3230 PSAP***702
                SAP***.REPOSRC#$                         160 PSAP***702
***************                                     --------
sum                                                     3390

2-INDEX         SAP***.REPOSRC^0                         136 PSAP***702
                SAP***.REPOSRC^SPM                       136 PSAP***702
***************                                     --------
sum                                                      272

3-LOBDATA       DATA:SAP***.REPOSRC                     3365 PSAP***702
                DATA:SAP***.REPOSRC#$                    192 PSAP***702
***************                                     --------
sum                                                     3557

4-LOBINDEX      DATA:SAP***.REPOSRC                        0 PSAP***702
                DATA:SAP***.REPOSRC#$                      0 PSAP***702
***************                                     --------
sum                                                        0

                                                    --------
sum                                                     7219

The first result block shows the current action (a table scan, in this instance), the name of the table, time remaining in hours:minutes:seconds format and the completion percentage from V$SESSION_LONGOPS.  As a side benefit, if you run this against a table that has some other long operation running against it, you will see that here as well.  It works for more than just table redefinitions.

The second result block displays the space used by the original table (REPOSRC) and the intermediate table used during DBMS_REDEFINITION (REPOSRC#), along with all segment types in use by both tables (table data, indexes, LOB data and LOB indexes).  For the LOB data and indexes, the “SEGMENT” column shows the LOB column name followed by the table name.

Another example of output from the same script, this time for a partitioned table with no LOBs and no redefinition running, from my EM12c repository database:

SQL> @s
Enter value for segment: em_metric_values_daily

SEGTYPE         SEGMENT                               SIZEMB TABLESPACE
--------------- ----------------------------------- -------- ---------------
1-TABLE         SYSMAN.EM_METRIC_VALUES_DAILY            327 MGMT_TABLESPACE
***************                                     --------
sum                                                      327

2-INDEX         SYSMAN.EM_METRIC_VALUES_DAILY_PK          48 MGMT_TABLESPACE
***************                                     --------
sum                                                       48

                                                    --------
sum                                                      375

The script:

SET PAGES 30
SET VERIFY OFF
SET FEEDBACK OFF

COLUMN ACTION FORMAT A15
COLUMN TARGET FORMAT A35
COLUMN PROGRESS FORMAT A15
COLUMN REMAINS FORMAT A8

SELECT
  OPNAME ACTION,
  TARGET,
  TO_CHAR(TO_DATE(TIME_REMAINING, 'sssss'), 'hh24:mi:ss') REMAINS,
  TO_CHAR(TRUNC(ELAPSED_SECONDS/(ELAPSED_SECONDS+TIME_REMAINING)*100,2))
  || '%' PROGRESS
FROM
  V$SESSION_LONGOPS
WHERE
  TIME_REMAINING != 0
AND TARGET LIKE UPPER('%&&segment%');

COLUMN SEGTYPE FORMAT A15
COLUMN SEGMENT FORMAT A35
COLUMN SIZEMB FORMAT 9999999
COLUMN TABLESPACE FORMAT A15

BREAK ON SEGTYPE SKIP 1 ON REPORT

COMPUTE SUM OF SIZEMB ON SEGTYPE
COMPUTE SUM OF SIZEMB ON REPORT

SELECT
  SEGTYPE,
  SEG SEGMENT,
  SIZEMB,
  TABLESPACE_NAME TABLESPACE
FROM
  (
    SELECT
      '1-TABLE' SEGTYPE,
      S.OWNER
      || '.'
      || S.SEGMENT_NAME SEG,
      TRUNC(SUM(BYTES)/1024/1024) SIZEMB,
      S.TABLESPACE_NAME
    FROM
      DBA_SEGMENTS S
    WHERE
      (
        S.SEGMENT_NAME = UPPER('&&segment')
      OR S.SEGMENT_NAME LIKE UPPER('&&segment#%')
      )
    AND S.SEGMENT_TYPE LIKE 'TABLE%'
    GROUP BY
      S.OWNER
      || '.'
      || SEGMENT_NAME,
      TABLESPACE_NAME
    UNION
    SELECT
      '2-INDEX' SEGTYPE,
      S.OWNER
      || '.'
      || S.SEGMENT_NAME SEG,
      TRUNC(SUM(S.BYTES)/1024/1024) SIZEMB,
      S.TABLESPACE_NAME
    FROM
      DBA_SEGMENTS S,
      DBA_INDEXES I
    WHERE
      S.SEGMENT_NAME = I.INDEX_NAME
    AND S.SEGMENT_TYPE LIKE 'INDEX%'
    AND S.OWNER = I.OWNER
    AND
      (
        I.TABLE_NAME = UPPER('&&segment')
      OR I.TABLE_NAME LIKE UPPER('&&segment#%')
      )
    GROUP BY
      S.OWNER
      || '.'
      || S.SEGMENT_NAME,
      S.TABLESPACE_NAME
    UNION
    SELECT
      '3-LOBDATA' SEGTYPE,
      L.COLUMN_NAME
      || ':'
      || S.OWNER
      || '.'
      || L.TABLE_NAME SEG,
      TRUNC(SUM(S.BYTES)/1024/1024) SIZEMB,
      S.TABLESPACE_NAME
    FROM
      DBA_SEGMENTS S,
      DBA_LOBS L
    WHERE
      S.SEGMENT_NAME = L.SEGMENT_NAME
    AND
      (
        S.SEGMENT_TYPE = 'LOBSEGMENT'
      OR S.SEGMENT_TYPE LIKE 'LOB %'
      )
    AND S.OWNER = L.OWNER
    AND
      (
        L.TABLE_NAME = UPPER('&&segment')
      OR L.TABLE_NAME LIKE UPPER('&&segment#%')
      )
    GROUP BY
      L.COLUMN_NAME
      || ':'
      || S.OWNER
      || '.'
      || L.TABLE_NAME,
      S.TABLESPACE_NAME
    UNION
    SELECT
      '4-LOBINDEX' SEGTYPE,
      L.COLUMN_NAME
      || ':'
      || S.OWNER
      || '.'
      || L.TABLE_NAME SEG,
      TRUNC(SUM(S.BYTES)/1024/1024) SIZEMB,
      S.TABLESPACE_NAME
    FROM
      DBA_SEGMENTS S,
      DBA_LOBS L
    WHERE
      S.SEGMENT_NAME   = L.INDEX_NAME
    AND S.SEGMENT_TYPE = 'LOBINDEX'
    AND S.OWNER        = L.OWNER
    AND
      (
        L.TABLE_NAME = UPPER('&&segment')
      OR L.TABLE_NAME LIKE UPPER('&&segment#%')
      )
    GROUP BY
      L.COLUMN_NAME
      || ':'
      || S.OWNER
      || '.'
      || L.TABLE_NAME,
      S.TABLESPACE_NAME
  )
ORDER BY
  SEGTYPE,
  SEG ;
UNDEFINE segment;

I based this on a script I initially found at stackoverflow.

How to migrate EM12c R3 OMS and repository to a new host

(EDIT 20130917: If you simply need to change the IP address of your OEM server, please review MOS note 1562029.1.  The procedure in that note may allow you to change your OEM server’s IP address without following the lengthy process I describe below.)

In order to save power in our data center, I need to migrate my EM12c R3 environment from the host where it currently runs to a new host.  I have a simple configuration, with a single OMS, no load balancer, and the repository database runs on the same host as EM12c R3 itself.  I also have BI Publisher installed and integrated with EM12c, and a few third party plugins as I’ve detailed elsewhere on this blog.  If you use an OS other than Linux x86-64 I suggest you research thoroughly as this procedure may or may not apply to your environment.  Further, if you have a multi-OMS setup or use a load balancer, you must read the documentation and adapt the process accordingly to match your system’s needs.  Note that I wrote this as I did the migration, live, on my production system, so I have text in a few places showing where I would have done things differently if I knew what to expect in the first place.  It all ended up working, but it could have been simpler.

Oracle documents the procedure for this migration in the EM12c Administrator’s Guide, Part VII, section 29, “Backing Up and Recovering Enterprise Manager“.  As a first step, my system administrator installed SLES 11 SP3 on the new server and created an account for me along with the ‘oracle’ account for EM12c. I have a 70GB volume to use for the database and OEM binaries, a 1GB volume for the DB control files and a 2GB volume for redo logs supplemented with a 15GB FRA volume to support flashback.  Due to our tape backup strategy I use the FRA only for flashback, which we don’t wish to backup, and use a separate volume for RMAN backupsets.  To avoid a backup/restore cycle, the volumes holding the database datafiles will just be moved over to the new host on the storage side.

First I will relocate the management repository database to the new host, then complete the process by relocating the OMS.

Relocating the Management Repository Database

I run Oracle Database 11.2.0.3, Enterprise Edition, plus PSU Jul 2013.  Rather than installing the database software from scratch and patching it, I will clone the existing Oracle home to the new server.  Unfortunately I cannot use EM12c to do the cloning, as cloning via EM12c requires a management agent on the new host.  The software-only install of EM12c that I will run later installs a management agent as part of the process and I do not wish these two to conflict, so I do not want to install an agent on the new host at this time.

I will clone the database home according to the procedure in Appendix B of the 11gR2 database documentation.  You should review the documentation for full details.

Cloning the Database Home

Stop the OMS, database and management agent before cloning the existing Oracle home.

oracle$ $OMS_HOME/bin/emctl stop oms -all ; $AGENT_HOME/bin/emctl stop agent ; $ORACLE_HOME/bin/dbshut $ORACLE_HOME

Create a zip file of the existing database home.  Run this step as root (or using sudo) to make sure that you get all the files.

oracle$ sudo zip -r dbhome_1.zip /oracle/oem/product/11.2.0/dbhome_1

Now I will start the original database back up so that OEM continues running while I prepare the cloned Oracle home.  I will perform this migration over a few days, as I have time, so I need to keep OEM up and running as much as possible to support and manage my other databases.

oracle$ $ORACLE_HOME/bin/dbstart $ORACLE_HOME ; sleep 10 ; $OMS_HOME/bin/emctl start oms ; sleep 10 ; $AGENT_HOME/bin/emctl start agent

Copy this zip file to the new host.

oracle$  scp dbhome_1.zip oracle@newhost:/oracle/oem

On the new host, extract this zip file to the target directory.

oracle@newhost$ unzip -d / dbhome_1.zip

Remove all “*.ora” files from the extracted $ORACLE_HOME/network/admin directory.

oracle@newhost$  rm /oracle/oem/product/11.2.0/dbhome_1/network/admin/*.ora

Execute clone.pl from $ORACLE_HOME/clone/bin.

oracle@newhost$ export ORACLE_HOME=/oracle/oem/product/11.2.0/dbhome_1
oracle@newhost$ $ORACLE_HOME/perl/bin/perl clone.pl ORACLE_BASE="/oracle/oem" ORACLE_HOME="/oracle/oem/product/11.2.0/dbhome_1" OSDBA_GROUP=dba OSOPER_GROUP=oper -defaultHomeName

Unfortunately this creates an oraInventory directory in the oracle user’s home directory.  I prefer to keep oraInventory under ORACLE_BASE, so I moved it and edited the generated files to change the path from /home/oracle/oraInventory to /oracle/oem/oraInventory.  Most likely some environment variable, or a previously existing /etc/oraInst.loc would have prevented this optional step.

oracle@newhost$ cp -a ~/oraInventory /oracle/oem
oracle@newhost$ cd /oracle/oem/oraInventory
oracle@newhost$ perl -pi.bak -e 's#/home/oracle#/oracle/oem#' oraInst.loc orainstRoot.sh

Complete the cloning steps by running the orainstRoot.sh and root.sh scripts.

oracle@newhost$ sudo /oracle/oem/oraInventory/orainstRoot.sh
Changing permissions of /oracle/oem/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /oracle/oem/oraInventory to dba.
The execution of the script is complete.
oracle@newhost$ sudo /oracle/oem/product/11.2.0/dbhome_1/root.sh
Check /oracle/oem/product/11.2.0/dbhome_1/install/root_newhost_2013-08-27_13-04-51.log for the output of root script

I do not want to use netca to configure the listener, so I will just copy the $ORACLE_HOME/network/admin/*.ora files back over from the original server to the new server, and edit them accordingly.

oracle$ scp *.ora oracle@newhost:/oracle/oem/product/11.2.0/dbhome_1/network/admin/ 

oracle@newhost$ cd $ORACLE_HOME/network/admin
oracle@newhost$ perl -pi.bak -e 's#oldhost#newhost#' *.ora

This completes the database cloning.

Start Management Repository Database On New Host

At this point you will probably use RMAN to create a backup of your original repository database, then restore that backup onto the new host.  Instead, I will cheat a bit, shut down OEM and the database, and ask my sysadmin to move the repository database’s datafile LUN over to the new host and mount it at the same location.

Before moving the LUN, create directories that the database needs for a successful startup.  These include the admin/SID/adump directory, and in my case, the /oracle/mirror/SID/cntrl and /oracle/mirror/SID/log directories where I keep the multiplexed copies of my redo logs and controlfiles.

oracle@newhost$ mkdir -p /oracle/oem/admin/emrep/adump
oracle@newhost$ mkdir -p /oracle/mirror/emrep/cntrl ; mkdir -p /oracle/mirror/emrep/log

As a sanity check, you should try starting up the listener on the new server and starting the database in NOMOUNT mode before proceeding.  This will help catch any issues that may exist in your environment before you start the outage on your original server.  Investigate and resolve any issues found before proceeding.

Shutdown the OMS, agent and database on the original server.

oracle$ $OMS_HOME/bin/emctl stop oms -all ; $AGENT_HOME/bin/emctl stop agent ; $ORACLE_HOME/bin/dbshut $ORACLE_HOME

Copy the controlfiles and redo logs from the original server to the new server.

oracle$ scp /oracle/oem/cntrl/control01.ctl oracle@newhost:/oracle/oem/cntrl/control01.ctl
oracle$ scp /oracle/mirror/emrep/cntrl/control02.ctl oracle@newhost:/oracle/mirror/emrep/cntrl/control02.ctl
oracle$ scp /oracle/oem/log/redo* oracle@newhost:/oracle/oem/log
oracle$ scp /oracle/mirror/emrep/log/redo* oracle@newhost:/oracle/mirror/emrep/log

Back on the new server, start up the listener, then the database.  I probably should have disabled flashback first.

oracle@newhost$ lsnrctl start LISTENER
oracle@newhost$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Wed Aug 28 10:09:01 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup;
ORACLE instance started.

Total System Global Area 9620525056 bytes
Fixed Size                  2236488 bytes
Variable Size            6241128376 bytes
Database Buffers         3355443200 bytes
Redo Buffers               21716992 bytes

Database mounted.
ORA-38760: This database instance failed to turn on flashback database
SQL> select open_mode from v$database;

OPEN_MODE
--------------------
MOUNTED

SQL> alter database flashback off;

Database altered.

SQL> alter database open;

Database altered.

Reconfigure Existing OMS For New Repository Database

Start the OMS and agent on the original server.  OMS startup will fail, as you have not yet reconfigured the repository.

oracle$ $OMS_HOME/bin/emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server is not functioning because of the following reason:
Failed to connect to repository database. OMS will be automatically restarted once it identifies that database and listener are up.
Check EM Server log file for details: /oracle/oem/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
oracle$ $AGENT_HOME/bin/emctl start agent

Reconfigure the OMS repository database connection.  Provide SYSMAN’s password when prompted.

oracle$ $OMS_HOME/bin/emctl config oms -store_repos_details -repos_conndesc "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=newhost)(PORT=1521)))(CONNECT_DATA=(SID=emrep)))" -repos_user sysman
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Enter Repository User's Password : 
Successfully updated datasources and stored repository details in Credential Store.
If there are multiple OMSs in this environment, run this store_repos_details command on all of them.
And finally, restart all the OMSs using 'emctl stop oms -all' and 'emctl start oms'.
It is also necessary to restart the BI Publisher Managed Server.

Stop, then restart the OMS.

oracle$ $OMS_HOME/bin/emctl stop oms -all ; sleep 5 ; $OMS_HOME/bin/emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
AdminServer Successfully Stopped
Oracle Management Server is Down
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up

Login to OEM and confirm proper operation of the system.  I had a lot of alerts for failed backup jobs since my repository database hosts my RMAN catalog.  These can wait for now.  Also expect your repository target to show as down, since you have not yet updated the monitoring configuration.  Reconfigure it now, providing the SYSMAN password when prompted.

oracle$ $OMS_HOME/bin/emctl config emrep -conn_desc "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=newhost)(PORT=1521)))(CONNECT_DATA=(SID=emrep)))"
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Please enter repository password:                                    Enter password :                                               Login successful
Target "Management Services and Repository:oracle_emrep" modified successfully
Command completed successfully!

At this point you have successfully moved your repository database.  Don’t worry about any errors for now, though if you rely on an RMAN catalog and stored scripts for your backups, and these all live in your OEM repository database, you should go through now and update the monitoring configuration for the repository database and listener so that backups of your other databases do not fail.  I had to edit the recovery catalog and specify the host, port, and SID manually, since for some reason when I told it to use the repository database it kept trying to use the old hostname.  I will fix this after I complete the rest of the migration.

IMPORTANT NOTE: Since you have not yet migrated the repository database target to an agent local to that machine, backups of your repository database may not run.  Monitor your archived log directory on this system until you complete the rest of the migration, and manually run backups when necessary.

Installing OMS On A New Host

To install the OMS on a new host, perform a software-only installation from the same EM12c R3 installer that was used to install on the original host.  You will need to identify and retrieve all of the plugins that you have installed on the current OMS, as well as any patches that are currently installed on the OMS.  You must also make sure to use the same directory layout as on the original OMS.

Identifying Installed Patches

oracle$ $OMS_HOME/OPatch/opatch lsinv -oh $OMS_HOME
[...]
Interim patches (1) :

Patch  13983293     : applied on Thu Jul 11 09:56:16 EDT 2013
Unique Patch ID:  14779750
   Created on 25 Apr 2012, 02:18:06 hrs PST8PDT
   Bugs fixed:
     13587457, 13425845, 11822929

This patch gets installed by the EM12c R3 installer, so no need to bother with it any further.  If you have other patches installed, go fetch them, and install them after you have completed the plugin installation (see below).

Identifying Installed Plugins

Identify all plugins installed on your system using the query provided in the documentation, run as SYSMAN against your repository database.

SELECT epv.display_name, epv.plugin_id, epv.version, epv.rev_version,decode(su.aru_file, null, 'Media/External', 'https://updates.oracle.com/Orion/Services/download/'||aru_file||'?aru='||aru_id||chr(38)||'patch_file='||aru_file) URL
FROM em_plugin_version epv, em_current_deployed_plugin ecp, em_su_entities su
WHERE epv.plugin_type NOT IN ('BUILT_IN_TARGET_TYPE', 'INSTALL_HOME')
AND ecp.dest_type='2'
AND epv.plugin_version_id = ecp.plugin_version_id
AND su.entity_id = epv.su_entity_id;

Oracle-provided plugins will show a URL from which you must download the plugin.  Third-party plugins will not; you will need to make sure you have the appropriate downloaded plugin install .opar file from when you initially installed it.  Gather up all of these plugin files into a single directory on your NEW OMS host, changing the “.zip” filename extension to “.opar” for the Oracle-provided plugins.  You need EVERY plugin returned by this query or else your installation will NOT work.  I placed mine in /oracle/oem/migration/plugins.

You also need to copy over the three .zip files containing the OEM 12cR3 distribution: V38641-01.zip, V38642-01.zip and V38643-01.zip.  Save them into a convenient staging area on the new server (I use /oracle/oem/stage).

Perform Software-Only Installation Of EM12c R3

Go to the staging area on the new server and extract the three .zip files containing the EM12c R3 distribution, then start the installer.

oracle@newhost$ unzip V38641-01.zip ; unzip V38642-01.zip ; unzip V38643-01.zip 
[...]
oracle@newhost$ ./runInstaller

You can follow my previous post about upgrading EM12c R2 to R3 for more information about the installation process, just make sure you run it as a software only install and use the exact same path names as configured on the original OMS.  In my case this means a middleware home of /oracle/oem/Middleware12cR3 and an agent base directory of /oracle/oem/agent12c.

While the software installation proceeds, you should run an exportconfig on your current OMS to produce the configuration backup file you will need to use to reconfigure the new one.  Enter the SYSMAN password when prompted.

oracle$ $OMS_HOME/bin/emctl exportconfig oms
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password : 
ExportConfig started...
Machine is Admin Server host. Performing Admin Server backup...
Exporting emoms properties...
Exporting secure properties...

Export has determined that the OMS is not fronted 
by an SLB. The local hostname was NOT exported. 
The exported data can be imported on any host but 
resecure of all agents will be required. Please 
see the EM Advanced Configuration Guide for more 
details.

Exporting configuration for pluggable modules...
Preparing archive file...
Backup has been written to file: /oracle/oem/gc_inst/em/EMGC_OMS1/sysman/backup/opf_ADMIN_20130828_120424.bka

The export file contains sensitive data. 
 You must keep it secure.

ExportConfig completed successfully!

Copy that backup file to the new server.

oracle$  scp /oracle/oem/gc_inst/em/EMGC_OMS1/sysman/backup/opf_ADMIN_20130828_120424.bka oracle@newhost:/oracle/oem

Once the software-only install finishes, it will prompt you to run allroot.sh.  Do so.

oracle@newhost$ sudo /oracle/oem/Middleware12cR3/oms/allroot.sh 

Starting to execute allroot.sh ......... 

Starting to execute /oracle/oem/Middleware12cR3/oms/root.sh ......
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /oracle/oem/Middleware12cR3/oms

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: 
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: 
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: 

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
/etc exist

Creating /etc/oragchomelist file...
/oracle/oem/Middleware12cR3/oms
Finished execution of  /oracle/oem/Middleware12cR3/oms/root.sh ......

Starting to execute /oracle/oem/agent12c/core/12.1.0.3.0/root.sh ......
Finished product-specific root actions.
/etc exist
/oracle/oem/agent12c/core/12.1.0.3.0
Finished execution of  /oracle/oem/agent12c/core/12.1.0.3.0/root.sh ......

After running allroot.sh, you need to run the PluginInstall.sh script with the path where you saved the .opar files.  Make sure you select every plugin listed when you ran the query to retrieve the plugin list earlier, then hit install.

oracle@newhost$ /oracle/oem/Middleware12cR3/oms/sysman/install/PluginInstall.sh -pluginLocation /oracle/oem/migration/plugins
This must match the list you generated previously

This must match the list you generated previously

Prepare the Software Library

Go to the original server, and copy the contents of the software library to the new server.

oracle$ scp -r /oracle/oem/software_library/ oracle@newhost:/oracle/oem

Recreate the OMS with OMSCA

Shut everything down on your old server.

oracle$ $OMS_HOME/bin/emctl stop oms -all ; sleep 5 ; $AGENT_HOME/bin/emctl stop agent

Run OMSCA using the exportconfig backup file you generated earlier.  Enter the administration server, node manager, repository database user and agent registration passwords when prompted.

oracle@newhost$ $OMS_HOME/bin/omsca recover -as -ms -nostart -backup_file /oracle/oem/opf_ADMIN_20130828_120424.bka
Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.3.0
Copyright (c) 1996, 2013, Oracle. All rights reserved.

OS check passed.
OMS version check passed.
Performing Admin Server Recovery...
Retrieved Admin Server template.
Source Instance Host name where configuration is exported : [deleted]
Populated install params from backup...
Enter Administration Server user password:
Confirm Password:
Enter Node Manager Password:
Confirm Password:
Enter Repository database user password:
Enter Agent Registration password:
Confirm Password:
Doing pre requisite checks ......
Pre requisite checks completed successfully

Checking Plugin software bits
Proceed to recovery
Setting up domain from template...
Setup EM infrastructure succeeded!
Admin Server recovered from backup.
Now performing cleanup of OMS EMGC_OMS1...
Now launching DeleteOMS...
OMS Deleted successfully

Delete finished successfully
Now launching AddOMS...
Infrastructure setup of EM completed successfully.

Doing pre deployment operations ......
Pre deployment of EM completed successfully.

Deploying EM ......
Deployment of EM completed successfully.

Configuring webtier ......
Configuring webTier completed successfully.

Importing OMS configuration from recovery file...

If you have software library configured 
please make sure it is functional and accessible 
from this OMS by visiting:
 Setup->Provisioning and Patching->Software Library

Securing OMS ......
Adapter already exists: emgc_USER
Adapter already exists: emgc_GROUP
Post "Deploy and Repos Setup" operations completed successfully.

Performing Post deploy operations ....
Total 0 errors, 78 warnings. 0 entities imported.
pluginID:oracle.sysman.core
Done with csg import
pluginID:oracle.sysman.core
Done with csg import
No logging has been configured and default agent logging support is unavailable.
Post deploy operations completed successfully.

EM configuration completed successfully.
EM URL is:https://newhost:7803/em

Add OMS finished successfully
Recovery of server EMGC_OMS1 completed successfully
OMSCA Recover completed successfully

Start the OMS on the new server.

oracle@newhost$ $OMS_HOME/bin/emctl start oms

Configure the central agent on the new server, then run the root.sh script.

oracle@newhost$ /oracle/oem/agent12c/core/12.1.0.3.0/sysman/install/agentDeploy.sh AGENT_BASE_DIR=/oracle/oem/agent12c AGENT_INSTANCE_HOME=/oracle/oem/agent12c/agent_inst AGENT_PORT=3872 -configOnly OMS_HOST=newhost EM_UPLOAD_PORT=4902 AGENT_REGISTRATION_PASSWORD=password
[...]
oracle@newhost$ sudo /oracle/oem/agent12c/core/12.1.0.3.0/root.sh

Relocate the oracle_emrep target to the new OMS host.

oracle@newhost$ $OMS_HOME/bin/emcli login -username=sysman
Enter password : 

Login successful
oracle@newhost$ $OMS_HOME/bin/emcli sync
Synchronized successfully
oracle@newhost$ $OMS_HOME/bin/emctl config emrep -agent newhost:3872
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Please enter repository password: 
Enter password :                                                               
Login successful
Moved all targets from oldhost:3872 to newhost:3872
Command completed successfully!
Enter password :                                                               
Login successful
Moved all targets from oldhost:3872 to newhost:3872
Command completed successfully!

Step through each of your existing agents to re-secure them against the new OMS.  Provide the OMS HTTP port (not HTTPS) in this command, and enter the agent registration password when prompted.

$ $AGENT_INSTANCE_DIR/bin/emctl secure agent -emdWalletSrcUrl "http://newhost:4890/em"
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Agent successfully stopped...   Done.
Securing agent...   Started.
Enter Agent Registration Password : 
Agent successfully restarted...   Done.
EMD gensudoprops completed successfully
Securing agent...   Successful.

Start the agent on the old OMS server.  You should not need to do this, but I could not update the WebLogic Domain monitoring configuration without doing so first.  Also re-secure this agent to point to the new OMS.

oracle$ $AGENT_HOME/bin/emctl start agent
oracle$ $AGENT_INSTANCE_DIR/bin/emctl secure agent -emdWalletSrcUrl "http://newhost:4890/em"

Login to the OEM GUI running on the new server and navigate to the WebLogic Domain target for the Cloud Control domain.  In the Target Setup -> Monitoring Credentials section, update the Administration server host value to the new server name, then hit OK.  Then execute a Refresh WebLogic Domain, selecting Add/Update Targets, to move all WebLogic targets to the new central agent.

I use third-party plugins to monitor VMWare targets, NetApp storage and MySQL servers.  I had many of them set up to run from the OMS agent (except for the VMWare ones, since Blue Medora helpfully advised not to use the OMS agent for this — great advice).  I now need to relocate each of these targets to the new central agent using emcli.  You won’t need to do this step unless you also have things set up this way.  If I had to do this again, I would not use the OMS agent for these targets, since I would not need to change anything if I just had these on some other agent.

oracle@newhost$ ./emcli relocate_targets -src_agent=oldhost:3872 -dest_agent=newhost:3872 -copy_from_src -target_name=nameoftarget -target_type=typeoftarget

Final Cleanup Steps

By now you have completed the bulk of the work necessary to migrate your EM12c stack to a new server.  Only a few steps remain.  If you use any utility scripts on the old server, go ahead and copy those over now.  I have scripts to automate starting/stopping the OMS and agent, so I’ve copied those over.  Also make sure the oracle user on the new server has all the environment variables set up in their shell initialization files.

oracle$ scp ~/bin/CCstart ~/bin/CCstop oracle@newhost:bin/

The GCDomain Oracle WebLogic Domain target did not get moved to my new agent.  If this happened to you, go to the target home page and select the Modify Agents menu item.  Click Continue, then find GCDomain in the list, scroll to the right, and assign the new OMS server’s agent as the monitoring agent for this target, then click the Modify Agents button.

Reinstall BI Publisher

Since I had BI Publisher installed on the old server, I need to install it again on the new one.  Retrieve the 11.1.1.6.0 BI Publisher installation files used previously, and copy them to your staging area.  Run the “runInstaller” program from bishiphome/Disk1, and perform a software-only installation with the middleware home set to your EM12c installation middleware home, and leave the Oracle home as Oracle_BI1.

Instead of running the configureBIP script as you normally would to integrate BI Publisher with EM12c, just go to the WebLogic administration console after the software-only install completes, and navigate to the BIP server configuration page.  Lock the configuration for editing, and edit the configuration to change the listen address to reference the new server’s hostname and change the machine to the machine name where the admin server runs (in my case it showed up as EMGC_MACHINE2).  Save and activate the changes, then start the BIP server.

After the server has started, return to the WebLogic Domain page and re-run the Refresh WebLogic Domain step, again with Add/Update targets, to move BIP to your new OMS agent.

I actually had to do the Refresh WebLogic Domain step here twice.  I may have simply not waited long enough after starting BIP before I ran it, but I do not know for sure.

Update EM Console Service

I have only one target showing down at this point, the EM Console Service.  Go to the target, and click on the Monitoring Configuration tab.  Click on Service Tests and Beacons.  Select the EM Console Service Test, and click the Edit button.  Make sure you have the “Access Login page” step selected, and click Edit.  Change the URL to reflect your new OEM server, and save the changes.

Remove Previous OMS Server From OEM

Stop the agent on your original OMS server.

oracle$ $AGENT_HOME/bin/emctl stop agent

Remove the host target where your original OMS ran.  Then remove the agent target.

One Last Bounce

Finally, bounce the whole thing one last time, then start it back up.  All green.

Conclusion

I would prefer a simpler process to migrate the EM12c stack to a new server, but this works.  If you find yourself in a similar position to mine, I hope this helps you.  I’ve spent a lot of time working in EM12c so I feel capable to diagnose and resolve issues encountered during the process, but if you run into problems do not hesitate to contact Oracle Support and file a service requests.  If you want your system to stay supportable, stick with the experts and just use blogs as a guide to get started.  Good luck.

How to connect to the default EM12c R3 self-signed WebLogic SSL port with WLST

After upgrading to Oracle Enterprise Manager 12c R3, I decided it was time to get roles configured properly for BI Publisher so that I can use it under my regular account rather than only permitting SYSMAN to access it.  Adeesh Fulay (@AdeeshF) helpfully provided me with a link to the documentation about setting up BI Publisher for EM12c.  The first step to perform the configuration involves connecting to the secured WebLogic adminserver via wlst.sh, but I immediately encountered an error:

wls:/offline> connect('weblogic', 'password', 't3s://host.domain.com:7103')
Connecting to t3s://host.domain.com:7103 with userid weblogic ...
<Jul 19, 2013 9:41:15 AM EDT> <Warning> <Security> <BEA-090542> <Certificate chain received from host.domain.com - x.x.x.x was not trusted causing SSL handshake failure. Check the certificate chain to determine if it should be trusted or not. If it should be trusted, then update the client trusted CA configuration to trust the CA certificate that signed the peer certificate chain. If you are connecting to a WLS server that is using demo certificates (the default WLS server behavior), and you want this client to trust demo certificates, then specify -Dweblogic.security.TrustKeyStore=DemoTrust on the command line for this client.> 
Traceback (innermost last):
  File "<console>", line 1, in ?
  File "<iostream>", line 22, in connect
  File "<iostream>", line 648, in raiseWLSTException
WLSTException: Error occured while performing connect : Error getting the initial context. There is no server running at t3s://host.domain.com:7103 
Use dumpStack() to view the full stacktrace

I could not find any obvious reference in the documentation on how to add the “-Dweblogic.security.TrustKeyStore=DemoTrust” options on the command line.  I attempted to just run wlst.sh with that parameter but I also received an error.

After a little searching I found a fix and figured I would post it.

In the documentation for the WebLogic 10.3.6 Oracle WebLogic Scripting Tool, section “Invoking WLST”, an example is included where it shows how to provide a different command line option to the WLST tool, by setting the environment variable CONFIG_JVM_ARGS. (EDITED 20130719: Adeesh has let me know that the preferred environment variable to use for this string is WLST_PROPERTIES, not CONFIG_JVM_ARGS.  Both work at the moment, but the documentation will be updated to refer to WLST_PROPERTIES so I advise you to use that one.)

I tried that before making my wlst.sh call, and everything worked successfully:

oracle@host:~> export WLST_PROPERTIES=-Dweblogic.security.TrustKeyStore=DemoTrust
oracle@host:~> /oracle/oem/Middleware12cR3/oracle_common/common/bin/wlst.sh 
[...]
Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

wls:/offline> connect('weblogic', 'password', 't3s://host.domain.com:7103')
[...]
Successfully connected to Admin Server 'EMGC_ADMINSERVER' that belongs to domain 'GCDomain'.wls:/GCDomain/serverConfig>

So if you are having trouble connecting to your WebLogic admin server using the default self-signed certificate via wlst.sh, this environment variable is the answer.  I was now able to proceed with granting my account access to BI Publisher, and now I am able to access BI Publisher features as needed without using the SYSMAN account.

wls:/GCDomain/serverConfig> grantAppRole(appStripe="obi",appRoleName="EMBIPViewer",principalClass="weblogic.security.principal.WLSUserImpl",principalName="USERNAME")    
Location changed to domainRuntime tree. This is a read-only tree with DomainMBean as the root. 
For more help, use help(domainRuntime)

wls:/GCDomain/serverConfig> grantAppRole(appStripe="obi",appRoleName="EMBIPAdministrator",principalClass="weblogic.security.principal.WLSUserImply", principalName="USERNAME")                                                
Already in Domain Runtime Tree

wls:/GCDomain/serverConfig> grantAppRole(appStripe="obi",appRoleName="EMBIPScheduler",principalClass="weblogic.security.principal.WLSUserImply", principalName="USERNAME")
Already in Domain Runtime Tree

wls:/GCDomain/serverConfig> grantAppRole(appStripe="obi",appRoleName="EMBIPAuthor",principalClass="weblogic.security.principal.WLSUserImply", principalName="USERNAME")
Already in Domain Runtime Tree

wls:/GCDomain/serverConfig> exit()

Exiting WebLogic Scripting Tool.