Student Guide - IES-421 Sun Fire™ High-End Server Maintenance and Administration

Sun Fire™ High-End Server
Maintenance and Administration
IES-421
Student Guide With Instructor Notes

Sun Microsystems, Inc.
UBRM05-104
500 Eldorado Blvd.
Broomfield, CO 80021
U.S.A.
Sun Proprietary: Internal Use Only

Revision E


February 3, 2004 12:47 pm


Copyright 2004 Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, California 95054, U.S.A. All rights reserved.
This product or document is protected by copyright and distributed under licenses restricting its use, copying, distribution, and
decompilation. No part of this product or document may be reproduced in any form by any means without prior written authorization of
Sun and its licensors, if any.
Third-party software, including font technology, is copyrighted and licensed from Sun suppliers.
Sun, Sun Microsystems, the Sun logo, OpenBoot, Solaris, Solstice DiskSuite, SunATM, Sun Blade, Sun Enterprise, Sun Fire, Sun Java,
SunOS, SunSolve, SunSolve Online, SunSpectrum, Sun StorEdge, SunVTS, and Ultra are trademarks or registered trademarks of Sun
Microsystems, Inc. in the U.S. and other countries.
All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the U.S. and
other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc.
ORACLE is a registered trademark of Oracle Corporation.
Federal Acquisitions: Commercial Software – Government Users Subject to Standard License Terms and Conditions
Export Laws. Products, Services, and technical data delivered by Sun may be subject to U.S. export controls or the trade laws of other

countries. You will comply with all such laws and obtain all licenses to export, re-export, or import as may be required after delivery to
You. You will not export or re-export to entities on the most current U.S. export exclusions lists or to any country subject to U.S. embargo
or terrorist controls as specified in the U.S. export laws. You will not use or provide Products, Services, or technical data for nuclear, missile,
or chemical biological weaponry end uses.
DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS, AND
WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE
OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE
LEGALLY INVALID.
THIS MANUAL IS DESIGNED TO SUPPORT AN INSTRUCTOR-LED TRAINING (ILT) COURSE AND IS INTENDED TO BE
USED FOR REFERENCE PURPOSES IN CONJUNCTION WITH THE ILT COURSE. THE MANUAL IS NOT A STANDALONE
TRAINING TOOL. USE OF THE MANUAL FOR SELF-STUDY WITHOUT CLASS ATTENDANCE IS NOT RECOMMENDED.
Export Commodity Classification Number (ECCN) assigned: 12 December 2002

Sun Proprietary: Internal Use Only
Please
Recycle


Copyright 2004 Sun Microsystems Inc., 4150 Network Circle, Santa Clara, California 95054, Etats-Unis. Tous droits réservés.
Ce produit ou document est protégé par un copyright et distribué avec des licences qui en restreignent l’utilisation, la copie, la distribution,
et la décompilation. Aucune partie de ce produit ou document ne peut être reproduite sous aucune forme, par quelque moyen que ce soit,
sans l’autorisation préalable et écrite de Sun et de ses bailleurs de licence, s’il y en a.
Le logiciel détenu par des tiers, et qui comprend la technologie relative aux polices de caractères, est protégé par un copyright et licencié
par des fournisseurs de Sun.
Sun, Sun Microsystems, le logo Sun, OpenBoot, Solaris, Solstice DiskSuite, SunATM, Sun Blade, Sun Enterprise, Sun Fire, Sun Java, SunOS,
SunSolve, SunSolve Online, SunSpectrum, Sun StorEdge, SunVTS, et Ultra sont des marques de fabrique ou des marques déposées de Sun
Microsystems, Inc. aux Etats-Unis et dans d’autres pays.
Toutes les marques SPARC sont utilisées sous licence sont des marques de fabrique ou des marques déposées de SPARC International, Inc.
aux Etats-Unis et dans d’autres pays. Les produits portant les marques SPARC sont basés sur une architecture développée par Sun
Microsystems, Inc.

ORACLE est une marque déposée registre de Oracle Corporation.
Législation en matière dexportations. Les Produits, Services et données techniques livrés par Sun peuvent être soumis aux contrôles
américains sur les exportations, ou à la législation commerciale dautres pays. Nous nous conformerons à lensemble de ces textes et nous
obtiendrons toutes licences dexportation, de ré-exportation ou dimportation susceptibles dêtre requises après livraison à Vous. Vous
nexporterez, ni ne ré-exporterez en aucun cas à des entités figurant sur les listes américaines dinterdiction dexportation les plus courantes,
ni vers un quelconque pays soumis à embargo par les Etats-Unis, ou à des contrôles anti-terroristes, comme prévu par la législation
américaine en matière dexportations. Vous nutiliserez, ni ne fournirez les Produits, Services ou données techniques pour aucune utilisation
finale liée aux armes nucléaires, chimiques ou biologiques ou aux missiles.
LA DOCUMENTATION EST FOURNIE “EN L’ETAT” ET TOUTES AUTRES CONDITIONS, DECLARATIONS ET GARANTIES
EXPRESSES OU TACITES SONT FORMELLEMENT EXCLUES, DANS LA MESURE AUTORISEE PAR LA LOI APPLICABLE, Y
COMPRIS NOTAMMENT TOUTE GARANTIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE, A L’APTITUDE A UNE
UTILISATION PARTICULIERE OU A L’ABSENCE DE CONTREFAÇON.
CE MANUEL DE RÉFÉRENCE DOIT ÊTRE UTILISÉ DANS LE CADRE D’UN COURS DE FORMATION DIRIGÉ PAR UN
INSTRUCTEUR (ILT). IL NE S’AGIT PAS D’UN OUTIL DE FORMATION INDÉPENDANT. NOUS VOUS DÉCONSEILLONS DE
L’UTILISER DANS LE CADRE D’UNE AUTO-FORMATION.

Sun Proprietary: Internal Use Only
Please
Recycle


Table of Contents
About This Course .............................................................Preface-xxi
Course Goals........................................................................ Preface-xxi
Course Map..........................................................................Preface-xxii
Topics Not Covered...........................................................Preface-xxiii
How to Use the Course Materials....................................Preface-xxiv
Conventions .........................................................................Preface-xxv
Icons .............................................................................Preface-xxv
Typographical Conventions ....................................Preface-xxvi

Sun Fire™ High-End Server Product and Architecture
Review and Assessment..................................................................1-1
Objectives ........................................................................................... 1-1
Relevance............................................................................................. 1-2
Additional Resources ........................................................................ 1-3
Reviewing Sun Fire HES ................................................................... 1-4
Self-Assessment Review.................................................................... 1-5
Product Review ................................................................................ 1-10
Sun Fire HES Products ........................................................... 1-10
Board Sets................................................................................. 1-10
Domain Configurable Units (DCUs)................................... 1-11
Reviewing the System Controllers ...................................... 1-12
Reviewing the Fireplane Operations............................................. 1-13
Understanding Memory Coherency ............................................. 1-15
Coherency Protocol................................................................. 1-15
Snoopy Coherency ................................................................. 1-17
Scalable Shared Memory........................................................ 1-18
Understanding Bus Interconnect Levels....................................... 1-19
Interconnect Levels ................................................................ 1-20
Address Interconnect............................................................. 1-21
Data Interconnect ................................................................... 1-22
Self-Assessment Review Answers................................................. 1-25
Installing Sun Fire High-End Servers .............................................2-1
Objectives ........................................................................................... 2-1
Relevance............................................................................................. 2-2
Sun Proprietary: Internal Use Only
vii
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Additional Resources ........................................................................ 2-3
Architecture Overview...................................................................... 2-4
Understanding Enterprise Installation Services
Methodology.................................................................................... 2-5
Understanding the EISdoc Tool....................................................... 2-6
Goals of the EISdoc Tool ......................................................... 2-7
EISdoc Resources ..................................................................... 2-8
Using the EIS Installation and Patch CD-ROM ............................. 2-9
Sun Fire HES EIS Installation Checklist................................. 2-9
Other EIS-CD Tools ............................................................... 2-10
Exploring System Components ..................................................... 2-11
Exploring Board Sets ....................................................................... 2-12
System Board Set.................................................................... 2-13
Exploring System Boards................................................................ 2-14
CPU and Memory Configuration ........................................ 2-16
Memory Configuration Rules ............................................... 2-18
System Board LEDs ................................................................ 2-20
Exploring Sun Fire HES I/O Support ........................................... 2-22
hsPCI I/O Assembly Slot Assignments............................... 2-23
hsPCI I/O Assembly Status LEDs ....................................... 2-24
Exploring the MaxCPU Board........................................................ 2-26
MaxCPU Board Components and Status LEDs.................. 2-26
Exploring the Sun Fire Link Board................................................ 2-28
Sun Fire Link Board Components and Status LEDs .......... 2-28
Exploring the Expander Board....................................................... 2-30
Exploring the System Controller ................................................... 2-32
System Controller Components........................................... 2-33
System Controller Faceplate................................................. 2-35
System Controller LEDs and Controls................................ 2-37
System Controller Peripheral Board LEDs.......................... 2-40

System Controller Physical Locations.................................. 2-43
Exploring Centerplanes................................................................... 2-44
Power Centerplane ................................................................. 2-46
Fan Backplane.......................................................................... 2-47
Exploring Carrier Plates.................................................................. 2-48
Carrier Plate Label ................................................................. 2-50
Understanding Electrical Specifications ....................................... 2-51
Processor Cabinet Power System.......................................... 2-51
Specifications (Sun Fire 15K and 12K Servers Only) ........ 2-52
Understanding Input Power Requirements................................. 2-53
Sun Fire 12K Server Power Requirements ......................... 2-54
Power Factor ............................................................................ 2-54
Power Distribution ................................................................. 2-55
Exploring AC Power Supplies ....................................................... 2-56
Power Supply LEDs................................................................ 2-57
Installing Power Cables................................................................... 2-59

viii

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E


Powering the System On and Off .................................................. 2-61
Powering On the Sun Fire HES............................................. 2-61
Powering Off the Sun Fire HES ............................................ 2-62
Exploring the Processor Cabinet Cooling System....................... 2-63
Exploring Fan Trays ........................................................................ 2-65
Exploring FrameManager............................................................... 2-67

FrameManager Front Panel .................................................. 2-69
Physical Locations.................................................................. 2-71
Exercise 1: Identifying the Contents of the EIS-CD..................... 2-73
Preparation............................................................................... 2-73
Tasks ......................................................................................... 2-73
Exercise 2: Removing and Installing Sun Fire HES FRUs.......... 2-74
Preparation............................................................................... 2-74
Tasks ........................................................................................ 2-76
Exercise 3: Powering the Platform On and Off............................ 2-79
Preparation............................................................................... 2-79
Tasks ......................................................................................... 2-79
Exercise Summary............................................................................ 2-80
Exercise 1 Solutions ......................................................................... 2-81
Tasks ......................................................................................... 2-81
Exercise 2 Solutions ......................................................................... 2-82
Tasks ......................................................................................... 2-82
Exercise 3 Solutions ......................................................................... 2-85
Tasks ......................................................................................... 2-85
Managing the Sun Fire HES System Controller Networks
and Software .....................................................................................3-1
Objectives ........................................................................................... 3-1
Relevance............................................................................................. 3-2
Additional Resources ........................................................................ 3-3
Reviewing System Architecture....................................................... 3-4
Exploring System Controller Networking ..................................... 3-5
Management Networks (MANs) ........................................... 3-7
Console Bus............................................................................. 3-14
Exploring System Controller Software ......................................... 3-15
SMS Capabilities .................................................................... 3-16
Reviewing SMS Startup .................................................................. 3-18

Message Logging Daemon ................................................... 3-21
SMS Startup Daemon ............................................................ 3-23
Hardware Access Daemon ................................................... 3-26
Management Network Daemon .......................................... 3-28
FRU Access Daemon ............................................................. 3-32
Failover Management Daemon............................................ 3-33
Platform Configuration Database Daemon........................ 3-34
Task Management Daemon.................................................. 3-35
Domain Status Monitoring Daemon ................................... 3-36

Sun Proprietary: Internal Use Only
ix
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E


Environmental Status Monitoring Daemon....................... 3-37
OpenBoot™ PROM Support Daemon ................................ 3-38
Key Management Daemon ................................................... 3-39
Domain X Server .................................................................... 3-43
Domain Configuration Administration Server.................. 3-44
Exploring SC Environment Variables ........................................... 3-45
Exploring SMS File System Structure ........................................... 3-46
Examining SMS Installation ........................................................... 3-50
Installing the System Management Services
Software................................................................................ 3-51
Installing the SMS Software Packages Using the
smsinstall Command...................................................... 3-52
Installing the SMS Software Packages Using the
smsupgrade Command...................................................... 3-55
Upgrading SMS Versions...................................................... 3-58

Configuring the System Controller ............................................... 3-60
Changing the Configuration During the Initial
Startup ................................................................................... 3-60
Configuring the Management Network ............................. 3-61
The smsconfig Script............................................................ 3-63
Using the smsconfig Script ................................................. 3-66
Configuring the Name Services ............................................ 3-70
Securing the Sun Fire HES System Controller ............................. 3-71
Using the smsconfig Security Option................................. 3-71
Exercise: Configuring the System Management Services .......... 3-74
Preparation............................................................................... 3-74
Task 1 – Installing and Configuring SMS Software ........... 3-74
Task 2 – Logging In to the Actual SC ................................... 3-74
Exercise Summary............................................................................ 3-76
Exercise Solutions ............................................................................ 3-77
Task 1 – Installing and Configuring SMS Software ........... 3-77
Task 2 – Logging In to the Actual SC ................................... 3-77
Configuring the Sun Fire HES Platform......................................... 4-1
Objectives ........................................................................................... 4-1
Relevance............................................................................................. 4-2
Additional Resources ........................................................................ 4-3
Architecture Overview...................................................................... 4-4
Sun Fire HES Administrative Privileges ........................................ 4-5
Managing Administration Groups.................................................. 4-6
Adding Administrators......................................................... 4-10
Group Privileges .............................................................................. 4-13
Platform Administrator Group ............................................. 4-13
Platform Operator Group ..................................................... 4-15
Platform Service Group......................................................... 4-16
Domain Administrator Group ............................................. 4-17

x

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E


Domain Configuration Group.............................................. 4-18
Superuser Privileges .............................................................. 4-19
Command-by-Command Privileges .................................... 4-19
Platform Administration Tasks...................................................... 4-20
Exploring the Available Component List..................................... 4-21
The setupplatform Command .................................................... 4-22
The setcsn Command.................................................................... 4-25
Displaying Platform Configuration .............................................. 4-26
The smsversion Command ................................................. 4-27
The showplatform Command............................................. 4-28
The showboards Command ................................................. 4-35
Monitoring Platform Environmentals and Status ....................... 4-38
The showenvironment Command ....................................... 4-38
The showlogs Command ..................................................... 4-43
Setting Up the Platform................................................................... 4-45
The setdate Command......................................................... 4-45
The setdefaults Command............................................... 4-47
Powering Platform Components On and Off .............................. 4-49
The poweron Command......................................................... 4-49
The poweroff Command ..................................................... 4-51
Updating the Firmware................................................................... 4-53
The flashupdate Command............................................... 4-54

Examples of the flashupdate Command .......................... 4-55
Backing Up and Restoring the SMS Environment ...................... 4-56
The smsbackup Command .................................................... 4-56
The smsrestore Command .................................................. 4-57
Re-Initializing the System Controller............................................ 4-58
The resetsc Command......................................................... 4-58
Exploring System Controller Failover .......................................... 4-59
How System Controller Failover Works ...................................... 4-60
Startup ..................................................................................... 4-61
Main SC Role During Startup ............................................... 4-61
Spare SC Role During Startup............................................... 4-61
File Propagation ...................................................................... 4-62
Failover Triggering Faults .................................................... 4-64
Failover Disabling Faults ....................................................... 4-64
Exploring Detailed Failover Scenarios.......................................... 4-65
Fault on Main SC Detected by the Main SC........................ 4-65
Fault on Main SC Detected by the Spare SC ....................... 4-66
I2 Network Fault .................................................................... 4-67
Fault on Main SC and I2 Network Also Down................... 4-67
Using System Controller Failover Commands ............................ 4-68
The showfailover Command.............................................. 4-68
The setfailover Command............................................... 4-70
Forcing System Controller Failover............................................... 4-71

Sun Proprietary: Internal Use Only
xi
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E


Exercise: Monitoring and Configuring Platforms ....................... 4-74

Preparation............................................................................... 4-74
Task 1 – Configuring Administration Groups.................... 4-74
Task 2 – Monitoring the Platform......................................... 4-75
Task 3 – Configuring Domain ACLs .................................... 4-76
Task 4 – Updating Flash PROM Images .............................. 4-76
Task 5 – Backing Up and Restoring the SMS
Environment ........................................................................ 4-77
Exercise Summary............................................................................ 4-78
Exercise Solutions ............................................................................ 4-79
Task 1 – Configuring Administration Groups.................... 4-79
Task 2 – Monitoring the Platform......................................... 4-80
Task 3 – Configuring Domain ACLs .................................... 4-81
Task 4 – Updating Flash PROM Images .............................. 4-81
Task 5 – Backing Up and Restoring the SMS
Environment ........................................................................ 4-82
Configuring Sun Fire HES Domains............................................... 5-1
Objectives ........................................................................................... 5-1
Relevance............................................................................................. 5-2
Additional Resources ........................................................................ 5-3
Architecture Overview...................................................................... 5-4
Exploring Sun Fire HES Domains ................................................... 5-5
Static and Dynamic Domain Configuration.......................... 5-5
Domain Configuration ...................................................................... 5-6
Domain Configuration Unit .................................................... 5-6
Domain Configuration Requirements ................................... 5-7
Configuring Static Domains ............................................................. 5-8
The addtag Command.......................................................... 5-10
The deletetag Command ................................................... 5-12
The addboard Command ..................................................... 5-13
The deleteboard Command............................................... 5-16

The moveboard Command ................................................... 5-19
The setobpparams Command............................................. 5-20
Domain Configuration Example.................................................... 5-22
Virtual Keyswitch ............................................................................ 5-25
The setkeyswitch Command.............................................. 5-25
Summary of Keyswitch Transitions .................................... 5-28
Displaying the Virtual Keyswitch Setting in a
Domain .................................................................................. 5-29
The hpost Utility ............................................................................. 5-30
File Locations of the .postrc File ........................................ 5-30
Controlling Level and Verbosity in the .postrc File ..... 5-31
Other Directives in the .postrc File ................................... 5-31
Accessing the Domain Console...................................................... 5-32
The console Command......................................................... 5-32

xii

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E


Displaying the Domain Devices .................................................... 5-35
The showdevices Command................................................ 5-35
Resetting the Domain ...................................................................... 5-38
The reset Command ............................................................. 5-38
Exploring the OpenBoot PROM Device Tree .............................. 5-40
OpenBoot PROM Device Tree.............................................. 5-41
Sun Fire Server Physical Device Mapping ................................... 5-44
Device Mapping Algorithm .................................................. 5-44

Decoding CPU and Memory Locations ........................................ 5-46
Decoding I/O Card Locations........................................................ 5-50
Agent ID ................................................................................... 5-50
IOC PCI Buses ........................................................................ 5-53
Slot Number............................................................................. 5-53
Sun Fire HES Server on the Solaris™ OS ..................................... 5-55
Configuring the Solaris OS in a New Domain............................. 5-56
Setting Up the OpenBoot PROM for the Default Boot
Device .................................................................................... 5-56
Configuring the Domain’s MAN Interface ........................ 5-57
Setting the Domain’s Time-of-Day Clock........................... 5-61
Configuring NTP in the Domain ......................................... 5-62
Finishing the Installation ....................................................... 5-62
System Controller Domain Log Files ............................................ 5-63
Exercise: Creating a Sun Fire HES Domain.................................. 5-64
Preparation............................................................................... 5-64
Task 1 – Configuring a Domain ........................................... 5-67
Task 2 – Analyzing the OpenBoot PROM Device
Tree........................................................................................ 5-69
Task 3 – Booting the Solaris OS for the First Time ............. 5-71
Exercise Summary............................................................................ 5-72
Exercise Solutions ............................................................................ 5-73
Task 1 – Configuring a Domain ............................................ 5-73
Task 2 – Analyzing the OpenBoot PROM Device Tree ..... 5-75
Task 3 – Booting the Solaris OS for the First Time ............. 5-77
Exploring Dynamic Reconfiguration ..............................................6-1
Objectives ........................................................................................... 6-1
Relevance............................................................................................. 6-2
Additional Resources ........................................................................ 6-3
Introducing Dynamic Reconfiguration........................................... 6-4

Benefits of DR ............................................................................ 6-4
DR Operational Locations ................................................................ 6-5
Sun Fire HES System Controller............................................. 6-5
Sun Management Center 3.x Software.................................. 6-6
Domain ....................................................................................... 6-6
DR Concepts ....................................................................................... 6-7
Preparing the Domain for DR Operations ............................ 6-8

Sun Proprietary: Internal Use Only
xiii
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E


DR attach Operation...................................................................... 6-10
DR attach States .................................................................... 6-10
DR detach Operation...................................................................... 6-11
DR detach States .................................................................... 6-11
I/O Board Considerations .............................................................. 6-12
Multipathed I/O Configuration ........................................... 6-12
Non-Multipath Configuration .............................................. 6-13
Detach-Safe and Suspend-Safe Devices............................... 6-14
Hot-Pluggable Hardware ..................................................... 6-16
System Memory Considerations.................................................... 6-17
Detaching Permanent Memory............................................. 6-17
Detaching Swap Space ........................................................... 6-18
CPU Considerations ........................................................................ 6-20
Bound Threads ........................................................................ 6-20
Processor Sets .......................................................................... 6-20
Performing DR From the System Controller ............................... 6-21
SMS Commands ...................................................................... 6-21

The addboard Command ..................................................... 6-22
The deleteboard Command............................................... 6-23
The moveboard Command ................................................... 6-25
Using the rcfgadm Utility............................................................... 6-27
Viewing DR Status................................................................. 6-29
Standard View ......................................................................... 6-29
Detailed View .......................................................................... 6-33
DR Procedures Using the rcfgadm Command ........................... 6-35
Replacing a System Board or MaxCPU Board.................... 6-35
Replacing an I/O Board or a Card ...................................... 6-39
DR Process With VERITAS DMP ......................................... 6-40
Moving Physical Resources Between Domains................. 6-43
Using Reconfiguration Coordination Manager (RCM) .............. 6-45
How RCM Works................................................................... 6-46
Sun Cluster Example .............................................................. 6-46
Exercise: Performing DR ................................................................. 6-47
Preparation............................................................................... 6-47
Task 1 – Replacing a System Board ...................................... 6-47
Task 2 – Replacing an I/O Board......................................... 6-50
Task 3 – Replacing an I/O Card .......................................... 6-52
Task 4 – Moving Physical Resources Between
Domains................................................................................ 6-54
Task 4a – Moving Physical Resources Between
Domains................................................................................ 6-55
Exercise Summary............................................................................ 6-56
Exercise Solutions ............................................................................ 6-57
Task 1 – Replacing a System Board ...................................... 6-57
Task 2 – Replacing an I/O Board.......................................... 6-59
Task 3 – Replacing an I/O Card ........................................... 6-61

xiv

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E


Exploring Sun Fire HES Capacity-on-Demand Version 2.0 ..........7-1
Objectives ........................................................................................... 7-1
Relevance............................................................................................. 7-2
Additional Resources ........................................................................ 7-3
Exploring COD Version 2.0 .............................................................. 7-4
Identifying COD Systems ........................................................ 7-4
Understanding COD Part Number and License
Requirements.......................................................................... 7-4
Managing COD RTU Licenses ................................................ 7-5
Understanding COD RTU Licenses and
UltraSPARC IV ....................................................................... 7-6
Understanding Instant Access CPUs and Headroom ......... 7-6
Identifying COD SMS Commands ........................................ 7-7
Managing the codd Daemon on the Sun Fire HES SC ....... 7-8
Installing and Removing a COD RTU License Key
to and From the COD License Database.......................... 7-10
Enabling COD Resources................................................................ 7-13
The setupplatform Command .......................................... 7-14
Verifying the COD Resource Configuration....................... 7-15
Monitoring COD Resources ................................................. 7-16
Obtaining COD Resource Usage information ................... 7-18
Identifying COD-Disabled CPUs......................................... 7-22
Identifying Other Useful Commands ................................. 7-23

Servicing COD V2.0 FRUs .................................................... 7-24
Converting a Non-COD CPU/Memory Board .................. 7-25
Exercise: Managing COD Resources ............................................. 7-27
Preparation............................................................................... 7-27
Tasks ......................................................................................... 7-27
Exercise Summary............................................................................ 7-29
Exercise Solutions ............................................................................ 7-30
Tasks ......................................................................................... 7-30
Troubleshooting Sun Fire HES .......................................................8-1
Objectives ........................................................................................... 8-1
Relevance............................................................................................. 8-2
Additional Resources ........................................................................ 8-3
Diagnostic Programs ......................................................................... 8-4
Power-On Self-Test ................................................................... 8-4
SunVTS™ Software................................................................... 8-4
SunSolve OnlineSM Service ...................................................... 8-5
Sun Explorer Software.............................................................. 8-5
The smshelp Command........................................................... 8-5
Examining the POST Process ........................................................... 8-6
Domain Configuration ............................................................. 8-6
Probing ....................................................................................... 8-7
Built-In Self-Test (BIST)........................................................... 8-8
CPU/Memory LPOST .............................................................. 8-9
Sun Proprietary: Internal Use Only
xv
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E


I/O LPOST ............................................................................... 8-10
Sun Fire HES LPOST .............................................................. 8-10

POST Logs................................................................................ 8-11
Using the hpost Test ....................................................................... 8-12
File Locations of the .postrc File ....................................... 8-14
Using the blacklist Files ............................................................. 8-15
The blacklist File................................................................. 8-15
Using Automatic System Reconfiguration (ASR) ....................... 8-18
Identifying an ASR Event ..................................................... 8-19
Isolating Centerplane Problems..................................................... 8-20
The setbus Command........................................................... 8-20
Centerplane Fault Isolation Example................................... 8-21
System Failures Requiring Specific Handling ............................. 8-24
Reboot Request................................................................................. 8-25
System Panic ..................................................................................... 8-26
Considerations......................................................................... 8-26
Watchdog, Redmode, and XIR Resets .......................................... 8-27
Forcing a Domain Reset .................................................................. 8-28
The reset Command ............................................................. 8-28
The setkeyswitch Command.............................................. 8-30
Recovering From a Hung Domain ................................................ 8-31
Cannot Log In to the Domain................................................ 8-31
Hard-Hung Domain ............................................................... 8-31
Heartbeat Failure (Hung Host) ...................................................... 8-32
Manual Intervention With a Hung Host ............................. 8-32
Hung System Controller Console.................................................. 8-33
Domainstops and Recordstops ...................................................... 8-34
Creating a Hardware State Dump File ......................................... 8-35
Viewing Error Messages ................................................................. 8-36
Starting the smshelp Utility .................................................. 8-36
Types of Errors ....................................................................... 8-39
Error Categories ...................................................................... 8-39

Using the redx Utility ..................................................................... 8-42
Modes of Operation ................................................................ 8-42
Configuring the redx Utility .......................................................... 8-43
The redx Utility Run Control Files (.redxrc) ................... 8-43
Configuring the Offline redx Standalone Script................ 8-44
Verifying the redx Utility Configuration in Offline
Mode From the System Controller .................................... 8-44
Running the redx Utility in Offline Modes ................................. 8-45
Running the redx Utility From a Workstation................... 8-45
Running the redx Utility From a System Controller........ 8-46
Loading Domain Stop and Record Stop Logs.............................. 8-48
Loading the Dump File .......................................................... 8-48
Viewing a Loaded Dump File ........................................................ 8-50
The wfail Operation.............................................................. 8-50

xvi

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E


Auto-Diagnosis Engines and CHS ................................................ 8-53
Component Health Status...................................................... 8-53
Auto-Diagnosis Engines......................................................... 8-54
Scenarios With and Without the Auto-Diagnosis
Engines ................................................................................. 8-55
Event Framework............................................................................. 8-56
Fault Management Architecture (FMA) .............................. 8-56
Summary of Diagnostic Processes................................................. 8-57

SMS Diagnostic Engine .......................................................... 8-57
POST of Domain After Component Failure....................... 8-58
POST Test Failure................................................................... 8-59
Solaris OS Diagnostic Engine ............................................... 8-60
Displaying and Setting the CHS .................................................... 8-61
The showchs Command......................................................... 8-61
The setchs Command.......................................................... 8-63
Automatic Email Event Notification ............................................. 8-64
The testemail Command ................................................... 8-69
Using Sun Explorer Data Collector Software............................... 8-70
Installing and Running the Sun Explorer Utility ............... 8-70
Viewing a Sun Explorer Capture.......................................... 8-71
Reviewing Sun Fire HES Unique Files................................. 8-72
Reviewing Technical Information for Escalation ........................ 8-73
General Information Needed ................................................ 8-73
Problem-Specific Information Needed ................................ 8-73
Exercise 1: Troubleshooting the Sun Fire HES ............................ 8-75
Preparation............................................................................... 8-75
Task 1 – Running POST Diagnostics .................................... 8-75
Task 2 – Managing the Blacklist............................................ 8-76
Task 3 – Assessing a Domain Configuration ..................... 8-77
Task 4 – Assessing a dstop and rstop Dump File
Using the redx Utility ......................................................... 8-78
Task 5 – Setting and Viewing the Component
Health Status........................................................................ 8-80
Task 6 – Viewing the Interaction of CHS and POST.......... 8-80
Task 7 – Resetting the Component Health Status ............. 8-81
Exercise 2: Troubleshooting Practice............................................. 8-82
Preparation............................................................................... 8-82
Tasks ......................................................................................... 8-82

Worksheet 1 ............................................................................ 8-87
Worksheet 2 ............................................................................ 8-88
Exercise Summary............................................................................ 8-89
Exercise 1 Solutions ......................................................................... 8-90
Task 1 – Running POST Diagnostics .................................... 8-90
Task 2 – Managing the Blacklist........................................... 8-91
Task 3 – Assessing a Domain Configuration ..................... 8-92

Sun Proprietary: Internal Use Only
xvii
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E


Task 4 – Assessing a dstop and rstop Dump File
Using the redx Utility ......................................................... 8-93
Task 5 – Setting and Viewing the Component
Health Status......................................................................... 8-94
Task 6 – Viewing the Interaction of CHS and POST.......... 8-95
Task 7 – Resetting the Component Health Status .............. 8-95
Sun Fire HES EIS Checklist ............................................................A-1
Objectives .......................................................................................... A-1
EIS Installation Checklist for Sun Fire 12K and 15K
Server Systems.............................................................................. A-1
Examining Sun Fire HES Interconnect Architecture ....................B-1
Objectives ........................................................................................... B-1
Introducing System Interconnect Architecture .............................B-2
Snoopy Coherency ....................................................................B-2
Sun Fireplane Interconnect Address Transactions ..............B-6
SSM Coherency .........................................................................B-9
SSM Operations in Sun Fire HES..........................................B-11

Understanding Interconnect Transaction Sequences .................B-12
Read-to-Share From the Same Bus .......................................B-12
Read-to-Share From a Different Bus ....................................B-14
Read-to-Share of an Owned Cache Line..............................B-16
Read-to-Own From a Different Bus .................................... B-20
Writeback to a Different Bus .................................................B-23
Reviewing Interconnect Timing.....................................................B-26
Read From Memory on the Same Bus..................................B-26
Read From Memory on a Different Bus.............................. B-28
Examining Sun Fire HES Group Privileges ...................................C-1
Objectives .......................................................................................... C-1
Additional Resources ....................................................................... C-2
Reviewing Sun Fire HES Administrator Privileges ..................... C-3
Multipathing I/O Management.........................................................D-1
Objectives .......................................................................................... D-1
Relevance............................................................................................ D-2
Additional Resources ....................................................................... D-3
How Sun StorEdge™ Traffic Manager Software Works............. D-4
Sun StorEdge Traffic Manager Software Device Tree ................. D-6
Sun StorEdge Traffic Manager Software Device Paths ...... D-6
Sun StorEdge Traffic Manager Software Path Properties ........... D-9
Path States ................................................................................. D-9
SCSI Device Attributes ......................................................... D-10
Configuring Sun StorEdge Traffic Manager Software .............. D-11
Domain Configuration Verification .................................... D-11
Sun StorEdge Traffic Manager Software Installation ...... D-13
Post-Installation Setup and Verification............................. D-14

xviii

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E


Sun StorEdge Traffic Manager Software Configuration
File .................................................................................................. D-15
Future Upgrades to the Operating System ................................. D-16
Automatic Path Discovery............................................................. D-17
Sun StorEdge Traffic Manager Software Management ............. D-18
The luxadm Command.......................................................... D-18
The format Command......................................................... D-23
Automatic Failover ......................................................................... D-24
IPMP.................................................................................................. D-26
IPMP Features ........................................................................ D-26
IPMP Components................................................................. D-27
Sun Trunking Software, IPMP, and HAnet Comparison.......... D-28
IPMP Hardware and Software Requirements ............................ D-30
Hardware Requirements....................................................... D-30
Software Requirements ......................................................... D-30
IPMP Group Requirements ........................................................... D-31
How IPMP Works ........................................................................... D-32
Network Path Failure Detection .......................................... D-32
Network Path Failover .......................................................... D-33
Network Path Failback.......................................................... D-33
Multipathing Configuration File .................................................. D-34
Settings ................................................................................... D-35
Starting the in.mpathd Daemon ......................................... D-35
New ifconfig Status Flags and Subcommands ....................... D-36
New ifconfig Status Flags ................................................. D-36

New ifconfig Subcommands ........................................... D-37
Configuring IPMP........................................................................... D-38
Creating an IP Multipathing Group.................................... D-38
Adding and Deleting Physical Network Interfaces ......... D-39
Configuring Test Interfaces .................................................. D-39
Configuring Standby Interfaces ........................................... D-40
Configuring Logical Interfaces............................................ D-41
IPMP Implementation .................................................................... D-42
Configuring Single Active Path With Standby.................. D-42
Multiple Active Paths Without Standby............................ D-46
Verifying IPMP Load Spreading .................................................. D-50
Automating IPMP Configuration in IPv4 ................................... D-52
Configuring IPMP in IPv6 ............................................................. D-53
Manually Configuring IPMP in IPv6 .................................. D-53
Automating IPMP Configuration in IPv6 .......................... D-53
Optional Exercise 1: Installing Sun StorEdge Traffic
Manager Software ........................................................................ D-54
Preparation.............................................................................. D-54
Task 1 – Configuring Sun StorEdge Traffic
Manager Software ............................................................... D-54

Sun Proprietary: Internal Use Only
xix
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E


Task 2 – Confirming Sun StorEdge Traffic
Manager Software ............................................................... D-55
Optional Exercise 2: Configuring and Managing IPMP............ D-56
Preparation.............................................................................. D-56

Task 1 – Exploring IPMP...................................................... D-57
Task 2 – Configuring IPMP .................................................. D-58
Exercise Summary........................................................................... D-59
Optional Exercise 1 Solutions........................................................ D-60
Task 1 – Configuring Sun StorEdge Traffic Manager
Software................................................................................ D-60
Task 2 – Confirming Sun StorEdge Traffic Manager
Software............................................................................... D-61
Optional Exercise 2 Solutions........................................................ D-62
Task 1 – Exploring IPMP....................................................... D-62
Task 2 – Configuring IPMP .................................................. D-63
Reconfiguration Coordination Manager (RCM).............................E-1
Objectives ........................................................................................... E-1
Additional Resources ........................................................................E-2
Resource Consumer Plug-ins ...........................................................E-3
RCM Scripts ...............................................................................E-3
RCM Script Commands ....................................................................E-5
RCM Script Example ............................................................... E-7
Installing a RCM Script ......................................................... E-10
Removing a RCM Script.........................................................E-10
Testing a RCM Script............................................................. E-11
Sun Enterprise™ 10000 Server-to-Sun Fire HES Dictionary ....... F-1
Daemons.............................................................................................. F-2
Commands .......................................................................................... F-3
Configuration Files ............................................................................ F-5
Miscellaneous Names........................................................................ F-6

xx

Sun Proprietary: Internal Use Only

Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E


Preface

About This Course
Course Goals
Upon completion of this course, you should be able to:
●

Remove and install most Sun Fire™ high-end server (HES)
field-replaceable units (FRUs)

●

Configure the Sun Fire HES platform and domains

●

Perform platform administration activities on the Sun Fire HES

●

Perform dynamic reconfiguration operations on the Sun Fire HES
domains

●

Use available Solaris™ Operating System (Solaris OS) and

platform-specific tools to troubleshoot the Sun Fire HES

●

Identify and administer Capacity-on-Demand (COD) systems

Sun Proprietary: Internal Use Only
Preface-xxi
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E


Course Map

Course Map
The following course map enables you to see what you have
accomplished and where you are going in reference to the course goals.

Sun Fire Server Review and Installation
Sun Fire HES
Product and
Architecture Review
and Assessment

Installing Sun Fire
High-End Servers

Sun Fire HES Configuration
Managing the Sun Fire
HES System Controller
Networks and

Software

Configuring the
Sun Fire HES
Platform

Configuring
Sun Fire HES
Domains

Sun Fire HES Administration and Troubleshooting

Exploring Dynamic
Reconfiguration

Preface-xxii

Exploring Sun Fire
HES
Capacity-on-Demand
Version 2.0

Troubleshooting
Sun Fire HES

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Topics Not Covered

Topics Not Covered
This course does not cover the following topics. Many of these topics are
covered in other courses offered by Sun™ Services:
●

Solaris OS administration – Covered in SA-118: Fundamentals of
Solaris™ 8 Operating Environment for System Administrators

●

Storage administration – Covered in ES-255: Sun™ Hardware RAID
and T3 Storage System Administration

●

Sun Fire 3800, Sun Fire 48x0, Sun Fire 4810, and Sun Fire 6800 server
administration – Covered in ES-420: Sun Fire™ Workgroup/Enterprise
Server Administration

This course does not cover advanced architecture theory.
Refer to the education.central/ITT Web site for specific information
and registration.

Sun Proprietary: Internal Use Only
About This Course
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Preface-xxiii

How to Use the Course Materials

How to Use the Course Materials
To enable you to succeed in this course, these course materials use a
learning module that is composed of the following components:

Preface-xxiv

●

Goals – You should be able to accomplish the goals after finishing
this course and meeting all of its objectives.

●

Objectives – You should be able to accomplish the objectives after
completing a portion of instructional content. Objectives support
goals and can support other higher-level objectives.

●

Lecture – The instructor will present information specific to the
objective of the module. This information will help you learn the
knowledge and skills necessary to succeed with the activities.

●

Activities – The activities take on various forms, such as an exercise,

self-check, discussion, and demonstration. Activities are used to
facilitate mastery of an objective.

●

Visual aids – The instructor might use several visual aids to convey
a concept, such as a process, in a visual form. Visual aids commonly
contain graphics, animation, and video.

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E


Conventions

Conventions
The following conventions are used in this course to represent various
training elements and alternative learning resources.

Icons
Additional resources – Indicates other references that provide additional
information on the topics described in the module.

URL Resources – Indicates additional information on the subjects being
described can be found at the indicated Universal Resource Locators
(URLs).

!

Discussion – Indicates a small-group or class discussion on the current
topic is recommended at this time.

?

Power user – Indicates additional supportive topics, ideas, or other
optional information.

Note – Indicates additional information that can help students but is not
crucial to their understanding of the concept being described. Students
should be able to understand the concept or complete the task without
this information. Examples of notational information include keyword
shortcuts and minor system adjustments.
Caution – Indicates that there is a risk of personal injury from a
nonelectrical hazard, or risk of irreversible damage to data, software, or
the operating system. A caution indicates that the possibility of a hazard
(as opposed to certainty) might happen, depending on the action of the
user.

Sun Proprietary: Internal Use Only
About This Course
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Preface-xxv


Conventions
Caution – Indicates that either personal injury or irreversible damage of
data, software, or the operating system will occur if the user performs this
action. A warning does not indicate potential events; if the action is

performed, catastrophic events will occur.

Typographical Conventions
Courier is used for the names of commands, files, directories,
programming code, and on-screen computer output; for example:
Use ls -al to list all files.
system% You have mail.
Courier is also used to indicate programming constructs, such as class
names, methods, and keywords; for example:
The getServletInfo method is used to get author information.
The java.awt.Dialog class contains Dialog constructor.
Courier bold is used for characters and numbers that you type; for
example:
To list the files in this directory, type:
# ls
Courier bold is also used for each line of programming code that is
referenced in a textual description; for example:
1 import java.io.*;
2 import javax.servlet.*;
3 import javax.servlet.http.*;
Notice the javax.servlet interface is imported to allow access to its life
cycle methods (Line 2).

Courier italic is used for variables and command-line placeholders
that are replaced with a real name or value; for example:
To delete a file, use the rm filename command.

Courier italic bold is used to represent variables whose values are to
be entered by the student as part of an activity; for example:
Type chmod a+rwx filename to grant read, write, and execute

rights for filename to world, group, and users.

Preface-xxvi

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E


Conventions
Palatino italic is used for book titles, new words or terms, or words that
you want to emphasize; for example:
Read Chapter 6 in the User’s Guide.
These are called class options.

Sun Proprietary: Internal Use Only
About This Course
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Preface-xxvii