Oracle Enterprise Manager 10g Grid Control Implementation Guide P1

PB
Oracle Enterprise Manager 10g Grid Control Implementation Guide
®
Oracle Enterprise
Manager 10g
Grid Control
Implementation Guide
Michael New
New York Chicago San Francisco
Lisbon London Madrid Mexico City Milan
New Delhi San Juan Seoul Singapore Sydney Toronto
Copyright © 2009 by The McGraw-Hill Companies, Inc. All rights reserved. Except as permitted under the United States Copyright
Act of 1976, no part of this publication may be reproduced or distributed in any form or by any means, or stored in a database or
retrieval system, without the prior written permission of the publisher.
ISBN: 978-0-07-159503-2
MHID: 0-07-159503-1
The material in this eBook also appears in the print version of this title: ISBN: 978-0-07-149275-1, MHID: 0-07-149275-5.
All trademarks are trademarks of their respective owners. Rather than put a trademark symbol after every occurrence of a trademarked
name, we use names in an editorial fashion only, and to the benefit of the trademark owner, with no intention of infringement of the
trademark. Where such designations appear in this book, they have been printed with initial caps.
McGraw-Hill eBooks are available at special quantity discounts to use as premiums and sales promotions, or for use in corporate
training programs. To contact a representative please visit the Contact Us page at www.mhprofessional.com.
Information has been obtained by Publisher from sources believed to be reliable. However, because of the possibility of human or
mechanical error by our sources, Publisher, or others, Publisher does not guarantee to the accuracy, adequacy, or completeness of
anyinformation included in this work and is not responsible for any errors or omissions or the results obtained from the use of such
information.
Oracle Corporation does not make any representations or warranties as to the accuracy, adequacy, or completeness of any informa-
tion contained in this Work, and is not responsible for any errors or omissions.
TERMS OF USE
This is a copyrighted work and The McGraw-Hill Companies, Inc. (“McGraw-Hill”) and its licensors reserve all rights in and to the

work. Use of this work is subject to these terms. Except as permitted under the Copyright Act of 1976 and the right to store and
retrieve one copy of the work, you may not decompile, disassemble, reverse engineer, reproduce, modify, create derivative works
based upon, transmit, distribute, disseminate, sell, publish or sublicense the work or any part of it without McGraw-Hill’s prior con-
sent. You may use the work for your own noncommercial and personal use; any other use of the work is strictly prohibited. Your right
to use the work may be terminated if you fail to comply with these terms.
THE WORK IS PROVIDED “AS IS.” McGRAW-HILL AND ITS LICENSORS MAKE NO GUARANTEES OR WARRANTIES AS
TO THE ACCURACY, ADEQUACY OR COMPLETENESS OF OR RESULTS TO BE OBTAINED FROM USING THE WORK,
INCLUDING ANY INFORMATION THAT CAN BE ACCESSED THROUGH THE WORK VIA HYPERLINK OR OTHERWISE,
AND EXPRESSLY DISCLAIM ANY WARRANTY, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. McGraw-Hill and its licensors do not
warrant or guarantee that the functions contained in the work will meet your requirements or that its operation will be uninterrupted
or error free. Neither McGraw-Hill nor its licensors shall be liable to you or anyone else for any inaccuracy, error or omission,
regardless of cause, in the work or for any damages resulting therefrom. McGraw-Hill has no responsibility for the content of any
information accessed through the work. Under no circumstances shall McGraw-Hill and/or its licensors be liable for any indirect, inci-
dental, special, punitive, consequential or similar damages that result from the use of or inability to use the work, even if any of them
has been advised of the possibility of such damages. This limitation of liability shall apply to any claim or cause whatsoever whether
such claim or cause arises in contract, tort or otherwise.
I dedicate this book to my daughter, Leah, and to my late brother, Robbie.
About the Author
Michael New is a senior information systems architect with 17 years of experience
in the software industry. After receiving his B.S. from M.I.T. in Aeronautics and
Astronautics, he worked for six years as a software engineer, principally on the
GPS satellite program at Rockwell. For the past 11 years, Michael has worked as
an Oracle core and Oracle Applications DBA. He began as a full-time DBA,
moved into consulting with an Oracle partner, then served as a Technical
Manager for Oracle Consulting before launching his own firm, New DB Solutions.
Michael has implemented the full array of Oracle products for clients in many
industries, including the Oracle E-Business Suite, RAC, Data Guard, RMAN,
Application Server, and Grid Control (in which he is a recognized authority).
Michael has authored several white papers on database administration, has

been a presenter at Oracle OpenWorld, and has been a technical editor for a
McGraw-Hill Professional book on Oracle 10g. Michael lives in New York City
with his five-year-old daughter, Leah. You can reach him at Michael@
NewDBsolutions.com or through his web site, www.newdbsolutions.com.
About the Technical Editors
Phil Choi worked at Oracle for several years documenting the Enterprise Manager
suite of products. Before Oracle, he worked in a variety of industries, including
consulting, software, and e-commerce. Phil has yet to put his Mechanical
Engineering degree to good use, but the degree in Creative Writing has come in
handy during his attempts to write short stories and essays for publication. Phil
(and his dog Ajax) lives and works in San Francisco.
Edward Whalen is the founder and Chief Technology Officer of Performance
Tuning Corporation. During his career, he completed many complex consulting
projects and continues to write books and technical papers on various Oracle and
Microsoft technologies. Ed has published more than eight books on SQL Server
and Oracle, most recently Microsoft SQL Server 2005 Administrator’s Companion
and Oracle Database 10g Linux Administration. Ed is recognized as a leader in
database design and performance optimization and provides critical services to
companies worldwide.
v

Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv


 Overview of the Grid Control Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 3
What Is Grid Computing? ............................................... 4
What Is Grid Control? .................................................. 7
Grid Control Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Grid Control Console ............................................ 13

Oracle Management Agent ........................................ 14
Oracle Management Service ....................................... 15
Oracle Management Repository .................................... 17
Data Flow Between Grid Control Components ......................... 19
Grid Control, Database Control, and AS Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Grid Control vs. Database Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Grid Control vs. AS Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Agents for Grid Control, Database Control, and AS Control . . . . . . . . . . . . . . . 26
Grid Control, Database Control, and AS Control: All Together Now ......... 26
 Grid Control Preinstallation ...................................... 29
Key Architectural Design Decisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Decide How Many Grid Control Environments to Build .................. 35
How Many Regional Sites Are Required? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Make Required Installation Choices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Network Configuration Steps ............................................ 47
Set Up Host Name Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Fully Qualify Host Name References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Host Name Constraints ........................................... 52
Use Static IP Addresses ........................................... 53
Connectivity Checks ............................................. 53
v
vi
Oracle Enterprise Manager 10g Grid Control Implementation Guide
Contents

vii
vi
Oracle Enterprise Manager 10g Grid Control Implementation Guide
Hardware Requirements ................................................ 55
Hardware Installation Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

Hardware Operating Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Software Requirements ................................................. 60
Verify Certification Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Create Required OS Groups and Users ............................... 61
Create Required Directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Stop Database Listeners Using Port 1521 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Synchronize OS Timestamps/Time Zones ............................. 70
Confirm Platform-Specific Software Requirements ...................... 72
3 Grid Control Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Gather Needed Installation Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Address Installation Bugs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Initialize the Oracle User Environment ..................................... 83
Set Up X Server (UNIX Only) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Set/Unset OS Environment Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
Running the Installer with Desired Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
Silent Installation Method ......................................... 87
Static Ports Feature .............................................. 88
Starting and Monitoring the Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Install Grid Control Using a New Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Specify Installation Type .......................................... 91
Specify Installation Location ....................................... 91
Language Selection .............................................. 93
Specify Inventory Directory and Credentials ........................... 93
Product-Specific Prerequisite Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
Specify Configuration ............................................ 95
Specify Optional Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Specify Security Options .......................................... 97
Oracle Configuration Manager Registration . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
Install ........................................................ 101

Configuration Assistants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
End of Installation ............................................... 103
Install Grid Control Using an Existing Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
Steps to Install the OMR Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
Run the Installer—GC Existing Database Option . . . . . . . . . . . . . . . . . . . . . . . 111
Install Standalone Agents on Dedicated OMR Nodes .................... 114
Log in to the Web Console .............................................. 114
Install the Grid Control Security Certificate in Your Browser ............... 115
Install an Additional OMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
vi
Oracle Enterprise Manager 10g Grid Control Implementation Guide
Contents

vii
vi
Oracle Enterprise Manager 10g Grid Control Implementation Guide
4 Grid Control Post Installation ..................................... 119
Patch Grid Control .................................................... 120
Apply the Latest Grid Control Patch Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
Apply Latest Database Patch Set Certified for OMR . . . . . . . . . . . . . . . . . . . . . 123
Apply the Latest EM Critical Patch Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
Apply Required Grid Control One-off Patches . . . . . . . . . . . . . . . . . . . . . . . . . 125
Oracle Management Service Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
Reduce Grid Control Logging ...................................... 130
Set Up Oracle User Environment on OMS Hosts . . . . . . . . . . . . . . . . . . . . . . . 135
Modify the Default Console Timeout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
Configure OMS for Failover and Load Balancing . . . . . . . . . . . . . . . . . . . . . . . 138
Tune OMS Thread Pool Size ....................................... 139
Add OMR Alias to OMS tnsnames.ora . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
Back Up Critical OMS Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

Oracle Management Repository Configuration ............................... 141
Set Up Oracle User Environment on OMR Nodes . . . . . . . . . . . . . . . . . . . . . . 141
Confirm That Listeners Load-Balance Across RAC Nodes ................. 141
Configure i SQL*Plus Access in Grid Control . . . . . . . . . . . . . . . . . . . . . . . . . . 142
Modify the Data Retention Policy ................................... 148
Modify Job Purge Policy .......................................... 150
Secure the emkey ............................................... 150
Set Up Grid Control Auditing ...................................... 151
Configure and Tune the OMR Database .............................. 152
Install Oracle Configuration Manager ................................ 152
5 Preinstall Standalone Management Agents ........................... 155
An Overview of Common Agent Preinstallation Steps .......................... 157
Agent Key Installation Decisions .......................................... 158
Select Agent Installation Methods ................................... 158
Use an Existing User or Create a Separate Agent User . . . . . . . . . . . . . . . . . . . 162
Install a Local or Cluster Agent ..................................... 163
Configure a Server Load Balancer First ............................... 165
Use an LDAP or Local Agent User . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
Choose to Secure or Not Secure the Agent ............................ 166
Agent Hardware Requirements ........................................... 166
Agent Disk Space Requirements .................................... 166
Agent CPU Requirements ......................................... 167
Agent RAM Requirements ......................................... 168
Summary of Agent Hardware Requirements ........................... 168
Agent Software Requirements ............................................ 168
Initialize the oracle User Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
Confirm No Existing Agent Is Installed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
Change Response File if Using an SLB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
Prepare for Cross-Platform Agent Installation . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Prepare Targets for Discovery ...................................... 173

Gather Needed Agent Installation Information . . . . . . . . . . . . . . . . . . . . . . . . . 173
viii
Oracle Enterprise Manager 10g Grid Control Implementation Guide
Contents

ix
viii
Oracle Enterprise Manager 10g Grid Control Implementation Guide
6 Install Management Agents via Agent Deploy . . . . . . . . . . . . . . . . . . . . . . . . 175
Agent Deploy Installation Overview ....................................... 176
Install Required Packages ............................................... 178
Configure SSH User Equivalence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
Set Up SSH Server (SSHD) on Windows .............................. 179
Back Up Current SSH Configuration ................................. 179
Set ORACLE HOME on OMS Host .................................. 179
Run sshConnectivity.sh Script ...................................... 180
Verify SSH User Equivalence Is Configured . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
Set Up Time Zone for SSH Server ......................................... 184
Validate All Command Locations ......................................... 185
Modify Agent Deploy Properties File for SLB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
Choose Inventory Location .............................................. 187
Verify Agent User Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
Prepare for Cross-Platform Agent Push . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
Include Additional Files (Optional) ........................................ 189
Run Agent Deploy Application ........................................... 190
Fresh Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
NFS Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
Upgrade Agent ................................................. 196
Agent Deploy Post-Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
Non-NFS Agents ................................................ 198

NFS Agents .................................................... 198
Troubleshoot Agent Deploy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
7 Install Management Agents Locally . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
nfsagentinstall Installation ............................................... 202
Configure Shared Storage ......................................... 205
Install Master Agent on Shared Storage ............................... 209
Install NFS Agents ............................................... 211
agentDownload Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
Prepare for agentDownload Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
Copy agentDownload Script from OMS to Target Host ................... 217
Execute agentDownload on Target Host .............................. 219
Agent Cloning Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
Silent Agent Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
Interactive Agent Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
Install Required Packages ......................................... 228
Run the Interactive Installer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
Agent Post-Installation Steps ............................................. 236
Set Up Agent User Environment .................................... 237
Confirm Agent Is Working . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
Refresh Host Configuration (If Needed) ............................... 240
Run agentca for Cluster Agent (Windows Only) . . . . . . . . . . . . . . . . . . . . . . . . 240
Confirm Agent Restart on Reboot Is Configured . . . . . . . . . . . . . . . . . . . . . . . . 240
Back Up the Agent .............................................. 241

Oracle Enterprise Manager 10g Grid Control Implementation Guide
Contents



Oracle Enterprise Manager 10g Grid Control Implementation Guide

 Install Grid Control Clients ....................................... 243
Install and Configure the EM Java Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
Determine Whether You Need the EM Java Console . . . . . . . . . . . . . . . . . . . . 245
Installation Steps for the EM Java Console ............................. 246
Start the EM Java Console ......................................... 249
Configure Change Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
Install Adobe SVG Viewer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
Install EM Command Line Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
Install Oracle Configuration Manager Client ................................. 264
 EM Login and Component Control ................................. 267
Log in to EM ......................................................... 269
Web Console Login to Grid Control ................................. 269
EM Java Console Login ........................................... 272
AS Console Login ............................................... 273
iSQL*Plus Login ................................................ 274
Metrics Browser Login . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
Control EM Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
Control iSQL*Plus Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
Control the Management Repository ................................. 279
Control the AS Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
Control the Management Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
Control the Management Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
Windows EM Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290
EM Startup/Shutdown Order ....................................... 292


 General Console Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
Tour of the Setup and Preferences Menus ............................. 299
Follow an Event from Trigger to Notification . . . . . . . . . . . . . . . . . . . . . . . . . . 303

Reasoning Behind this Chapter’s Menu Navigation Order . . . . . . . . . . . . . . . . 305
Set Up Administrators .................................................. 307
Create Roles ................................................... 307
Create Administrators ............................................ 313
Set Administrator Preferences ...................................... 317
Set Preferred Credentials .......................................... 320
Configure Notifications ................................................. 330
Define Notification Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
Set Notification Schedules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334
Create and Subscribe to Notification Rules ............................ 339
Enable Patching Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356
Complete the Patching Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356
Schedule the RefreshFromMetalink Job ............................... 362
x
Oracle Enterprise Manager 10g Grid Control Implementation Guide
Contents

xi
x
Oracle Enterprise Manager 10g Grid Control Implementation Guide
11 Configure Target Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365
Remove Targets from Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367
Ensure Agents Are Started ......................................... 367
Back Up targets.xml File .......................................... 367
Confirm No Targets Are in Blackout ................................. 367
Remove Targets from Grid Control .................................. 368
Discover and Configure Targets for Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 370
Three Paths to Discover/Configure a Target . . . . . . . . . . . . . . . . . . . . . . . . . . . 371
Agent Configuration ............................................. 376
ASM Discovery and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383

Database Discovery and Configuration ............................... 386
Listener Discovery and Configuration ................................ 407
Host Additional Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
OracleAS Discovery and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411
Enter Target Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413
Grant Management Pack Access .......................................... 414
Management Options: Packs, Plug-ins, and Connectors .................. 414
Schedule Blackouts for Planned Downtimes ................................. 421
Set Blackouts in the Console ....................................... 421
Set Blackouts at the Command Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427
12 Configure Group and Service Monitoring ............................ 429
Group Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 430
Create a Group ................................................. 431
Group Features ................................................. 435
Service Configuration .................................................. 438
SLA Objectives for Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 439
Key Elements of Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
Configure a Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445
13 Tune Metrics and Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 489
Change Default Metrics and Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492
Change Default Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492
Change Default Policies .......................................... 501
Enable Metric Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504
Metric Baseline Concepts and Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 506
Create Metric Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 509
Disabling Metric Baselines ........................................ 514
Add Corrective Actions ................................................. 515
Create Corrective Actions ......................................... 516
Add Corrective Actions to Metrics or Policies .......................... 518
Implement User-Defined Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521

Create an OS User-Defined Metric .................................. 522
Create a SQL User-Defined Metric .................................. 527
x
Oracle Enterprise Manager 10g Grid Control Implementation Guide
Contents

xi
x
Oracle Enterprise Manager 10g Grid Control Implementation Guide
Use Metric Snapshots .................................................. 529
Create Metric Snapshots .......................................... 530
Apply Metric Snapshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 532
Leverage Monitoring Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 534
Create Monitoring Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 536
Maintain Custom Metrics when Applying Monitoring Templates . . . . . . . . . . . 540
Apply Monitoring Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 541
Compare Settings Between Targets and the Template .................... 544
14 Backup and Recovery of Grid Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549
Introduction to Grid Control B&R ......................................... 551
Grid Control B&R Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 551
Resetting Agents ................................................ 554
Recommended Grid Control Backups ................................ 557
OMR Database Backup and Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 560
Back Up the OMR Database in Grid Control . . . . . . . . . . . . . . . . . . . . . . . . . . 561
Recover the OMR Database ....................................... 597
Leverage Flashback Technology for OMR Database Recovery . . . . . . . . . . . . . 606
Manage OMR Database Backups in Grid Control ....................... 608
Grid Control Software Backup and Restoration ............................... 612
OMR Database Software Backup and Restoration ....................... 613
OMS Software Backup and Restoration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615

Agent Software Backup and Restoration .............................. 619
15 Configure Grid Control for High Availability and Disaster Recovery . . . . . . . 623
Grid Control High Availability Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625
OMS HA Recommendations ....................................... 627
OMR Database HA Recommendations ............................... 633
Agent HA Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 636
Grid Control Disaster Recovery Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . 637
OMR Database DR Recommendations ............................... 639
OMS DR Recommendations ....................................... 644
Agent DR Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 648
16 Securing Grid Control Data Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 651
Enable EM Framework Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 656
Enable Framework Security Between Agents and OMS ................... 656
Secure Repository Data Transmissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 663
Secure Console Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 675
Enforce HTTPS Between the Browser and AS Console . . . . . . . . . . . . . . . . . . . 676
Enforce HTTPS Between Browser and OHS . . . . . . . . . . . . . . . . . . . . . . . . . . . 677
Security Considerations for Console Web Cache Access .................. 679
Limit GC Console Access to Certain Clients . . . . . . . . . . . . . . . . . . . . . . . . . . . 680
xii
Oracle Enterprise Manager 10g Grid Control Implementation Guide
17 Maintain and Tune Grid Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 685
Perform Routine Grid Control Maintenance Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 686
Weekly Online Maintenance Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 687
Monthly Offline Maintenance Tasks ................................. 693
Gather and Document Grid Control Metrics ................................. 699
Types of Grid Control Metrics to Gather .............................. 699
Specific Grid Control Metrics to Gather . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 703
Procedure to Evaluate Grid Control Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 708
Tune Grid Control to Reduce Bottlenecks ................................... 709

Reduce High CPU Utilization ...................................... 710
Resolve Loader Backlog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 715
Minimize Rollup Delays .......................................... 719
Evaluate Job, Notification, and Alert Metrics ........................... 722
Tune I/O Bottlenecks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 723
Improve Poor Console Performance ................................. 727
Platform-Specific Tuning Recommendations ........................... 734
Appendixes Online at www.OraclePressBooks.com
A Agent Subsystems
B Certification and Platform-Specific Installation Requirements
C Configure and Tune a Database for the Management Repository
D Set Up SSH User Equivalence
E Configure Grid Control Automatic Startup
F RMAN Scripts from Oracle-Suggested Backup Strategy in Grid Control
G Configuring OAS Using Oracle Net Manager
H Tools for Grid Control Performance Metrics
I How to Install the Grid Control Security Certificate in Your Browser
Index ......................................................... 737
xiii
xii
Oracle Enterprise Manager 10g Grid Control Implementation Guide
Acknowledgments
This book sprung from material for a Grid Control “Deep Dive” class that I developed and
taught to Oracle Consulting personnel at the behest of Dennis Horton, a Senior Director at
Oracle. Without his impetus, I would never have tackled this book.
Thanks to the Oracle Press team at McGraw-Hill Professional: Lisa McClain, Mandy
Canales, Jennifer Housh; the copyedit team, notably Vastavikta Sharma, Carolyn Welch,
and Bill McManus; and the Illustration and Production departments for their work in
producing this book. I am very grateful to Ed Whalen and Phil Choi for their roles as
technical reviewers. Ed introduced me to the McGraw-Hill team by bringing me in as a

technical editor for his book, Oracle Database 10g Linux Administration (Oracle Press,
McGraw-Hill Professional, 2005). Later, he told me about McGraw-Hill’s desire to publish
a Grid Control book. After I signed up to write this book, Ed provided his company’s
computer resources as a platform for most of the Grid Control installation and configuration
steps shown in the book. I am also largely indebted to Phil Choi, formerly the lead editor at
Oracle for Enterprise Manager products, for extensively editing the book’s first draft. Phil
taught me a great deal about technical writing.
I greatly appreciate Oracle’s help in supplying and validating technical content as
needed. Thanks to Oracle Support in answering some tough questions along the way,
particularly Mathieu Hornsperger for explaining the inner workings of Grid Control security.
Narain Jagathesan and Anirban Chatterjee in the Enterprise Manager development group at
Oracle provided much of the detailed information about the Management Agent given in
Appendix A (available online at www.OraclePressBooks.com).
xiii
xiv
Oracle Enterprise Manager 10g Grid Control Implementation Guide
I’d also like to acknowledge my Oracle mentors: Howard Ostrow, a great all-around
technical resource and RAC expert; Matthew Burke, a DBA master who tutored me for four years
in DBA paths less traveled and encouraged me to present my first white paper; and Ashok
Nagabothu, who took me, when a fledgling DBA, under his wing.
I am grateful to my family and friends for seeing me through a trying time personally while
writing this book. My brother Jon, his wife Gina, their son Daniel, my mother, and good friends
Adam Bierman and Brad Stonberg were very supportive. I am indebted to Amy Seawright,
erstwhile companion, who brought me back to life before my mistress, Oracle, reclaimed it.
Finally, I thank my daughter, Leah, and my muse for resurrecting me yet again, once and for all.
xv
xiv
Oracle Enterprise Manager 10g Grid Control Implementation Guide
Introduction
This is the first book dedicated solely to the subject of implementing Oracle Enterprise

Manager 10g Grid Control. It was only a matter of time before such a book was published.
With the first release (10.1) of Grid Control roughly seven years ago, Oracle ushered in a true
system management product. The latest release (10.2) of the product is enjoying large-scale
adoption by companies looking to centrally manage their complete IT infrastructure—not just
their Oracle systems, but non-Oracle ones as well. Firms around the world are significantly
cutting IT expenditures by administering their data center components with Grid Control.
As you would expect from such a capable software product, the Enterprise Manager 10g
Grid Control documentation library from Oracle Corporation is bulky, yet still does not
contain solutions to certain common installation and configuration hurdles. Answers are
found in supplemental material available on OracleMetaLink, the Oracle Technology
Network (OTN), and elsewhere on the Web. (How much third-party material alone is out
there? Google “Oracle Enterprise Manager 10g Grid Control” and you’ll get 122,000 hits.)
All in all, it’s a lot of material to digest, especially if you’re new to Grid Control.
Contrast a Grid Control novice’s first installation using product and supplemental
documentation with that of an expert who has performed many installations, having already
waded through and assimilated the reference material and learned the bugs, workarounds,
optimal installation order, shortcuts, tips, and tricks. The expert will likely implement Grid
Control successfully, on time and under budget, whereas the novice may not be so fortunate.
The aim of this book is to lend an expert hand to a DBA who is not intimately acquainted
with Grid Control so that he or she can implement it according to best practices, as quickly
and painlessly as possible.
This book distills what I’ve learned from completing many Grid Control engagements
over the past seven years. Each implementation was unique: many were large installations at
Fortune 500 companies, while others were smaller, proof-of-concept assignments; the
platforms were different, as were the high availability requirements. Yet, these engagements
also shared many commonalities: I ran across same issues over and over again—some
xv