Student Guide - IES-421 Sun Fire™ High-End Server Maintenance and Administration - Slide

SunU

Sun Fire™ High-End Server
Maintenance and Administration

IES-421

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration


Copyright 2004 Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, California 95054, U.S.A. All rights reserved.
This product or document is protected by copyright and distributed under licenses restricting its use, copying, distribution, and decompilation. No part of this product or document may
be reproduced in any form by any means without prior written authorization of Sun and its licensors, if any.
Third-party software, including font technology, is copyrighted and licensed from Sun suppliers.
Sun, Sun Microsystems, the Sun logo, OpenBoot, Solaris, Sun StorEdge, SunVTS, Sun Fire, SunSolve, and SunSolve Online are trademarks or registered trademarks of Sun Microsystems,
Inc. in the U.S. and other countries.
All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the U.S. and other countries. Products bearing SPARC
trademarks are based upon an architecture developed by Sun Microsystems, Inc.
UNIX is a registered trademark in the U.S. and other countries, exclusively licensed through X/Open Company, Ltd.
U.S. Government approval might be required when exporting the product.
RESTRICTED RIGHTS: Use, duplication, or disclosure by the U.S. Government is subject to restrictions of FAR 52.227-14(g)(2)(6/87) and FAR 52.227-19(6/87), or DFAR 252.227-7015
(b)(6/95) and DFAR 227.7202-3(a).
DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS, AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE
HELD TO BE LEGALLY INVALID.


Copyright 2004 Sun Microsystems Inc., 4150 Network Circle, Santa Clara, California 95054, Etats-Unis. Tous droits réservés.
Ce produit ou document est protégé par un copyright et distribué avec des licences qui en restreignent l’utilisation, la copie, la distribution, et la décompilation. Aucune partie de ce
produit ou document ne peut être reproduite sous aucune forme, par quelque moyen que ce soit, sans l’autorisation préalable et écrite de Sun et de ses bailleurs de licence, s’il y en a.

Le logiciel détenu par des tiers, et qui comprend la technologie relative aux polices de caractères, est protégé par un copyright et licencié par des fournisseurs de Sun.
Sun, Sun Microsystems, le logo Sun, OpenBoot, Solaris, Sun StorEdge, SunVTS, Sun Fire, SunSolve, et SunSolve Online sont des marques de fabrique ou des marques déposées de Sun
Microsystems, Inc. aux Etats-Unis et dans d’autres pays.
Toutes les marques SPARC sont utilisées sous licence sont des marques de fabrique ou des marques déposées de SPARC International, Inc. aux Etats-Unis et dans d’autres pays. Les
produits portant les marques SPARC sont basés sur une architecture développée par Sun Microsystems, Inc.
UNIX est une marques déposée aux Etats-Unis et dans d’autres pays et licenciée exclusivement par X/Open Company, Ltd.
L’accord du gouvernement américain est requis avant l’exportation du produit.
LA DOCUMENTATION EST FOURNIE “EN L’ETAT” ET TOUTES AUTRES CONDITIONS, DECLARATIONS ET GARANTIES EXPRESSES OU TACITES SONT FORMELLEMENT
EXCLUES, DANS LA MESURE AUTORISEE PAR LA LOI APPLICABLE, Y COMPRIS NOTAMMENT TOUTE GARANTIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE, A
L’APTITUDE A UNE UTILISATION PARTICULIERE OU A L’ABSENCE DE CONTREFAÇON.


SunU

Preface
About This Course

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration


SunU

Course Goals
Upon completion of this course, you should be able to:
• Remove and install most Sun Fire™ high-end server
(HES) field-replaceable units (FRUs)
• Configure the Sun Fire HES server platform and
domains
• Perform platform administration activities on the Sun

Fire HES
• Perform dynamic reconfiguration operations on the
Sun Fire HES domains

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Preface, slide v of xii


SunU

Course Goals
• Use available Solaris™ Operating System (Solaris OS)
and platform-specific tools to troubleshoot the Sun Fire
HES
• Identify and administer Capacity-on-Demand (COD)
systems

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Preface, slide vi of xii


SunU

Course Map

Sun Fire Server Review and Installation
Sun Fire HES
Product and
Architecture Review
and Assessment

Installing Sun Fire
High-End Servers

Sun Fire HES Configuration
Managing the Sun Fire
HES System Controller
Networks and
Software

Configuring the
Sun Fire HES
Platform

Configuring
Sun Fire HES
Domains

Sun Fire HES Administration and Troubleshooting

Exploring Dynamic
Reconfiguration

Exploring Sun Fire
HES

Capacity-on-Demand
Version 2.0

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Troubleshooting
Sun Fire HES

Preface, slide vii of xii


SunU

Topics Not Covered
• Solaris OS administration – Covered in SA-118:
Fundamentals of Solaris™ 8 Operating Environment for
System Administrators
• Storage administration – Covered in ES-255: Sun™
Hardware RAID and T3 Storage System Administration
• Sun Fire 3800, Sun Fire 4800, Sun Fire 4810, and Sun
Fire 6800 server administration – Covered in ES-420:
Sun Fire™ Workgroup/Enterprise Server Administration
This course does not cover advanced architecture theory.

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Preface, slide viii of xii


SunU

Icons
Additional resources

!

URL Resources
Discussion

?

Power user

Note

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Preface, slide ix of xii


SunU

Icons
Caution

Caution – Electrical

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Preface, slide x of xii


SunU

Typographical Conventions
• Courier is used for the names of commands, files,
directories, programming code, programming
constructs, and on-screen computer output.
• Courier bold is used for characters and numbers
that you type, and for each line of programming code
that is referenced in a textual description.
• Courier italic is used for variables and
command-line placeholders that are replaced with a
real name or value.
• Courier italic bold is used to represent variables
whose values are to be entered by the student as part of
an activity.
Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Preface, slide xi of xii

SunU

Typographical Conventions
• Palatino italic is used for book titles, new words or
terms, or words that are emphasized.

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Preface, slide xii of xii


SunU

Module 1
Sun Fire™ High-End Server Product and
Architecture Review and Assessment

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration


SunU

Objectives
Describe the Sun Fire HES products and architecture.

Sun Proprietary: Internal Use Only

Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Module 1, slide 2 of 13


SunU

Reviewing Sun Fire HES
This course is designed to provide you with the information
to:
• Remove and replace FRUs on the high-end servers
• Install and configure the System Management Services
(SMS) software
• Manage the platform with the SMS software
• Create and manage domains on the HES
• Perform dynamic reconfiguration
• Perform basic troubleshooting using both manual and
automatic diagnosis techniques

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Module 1, slide 3 of 13


SunU

Sun Fire HES

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Module 1, slide 4 of 13


SunU

Self-Assessment Review
• Review your knowledge now or later, as instructed by
your instructor
• Review the prerequisite tech-talk, if needed

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Module 1, slide 5 of 13


SunU

Product Review
• Cabinet with up to nine board sets on each side
• System controller (SC) board set on each side
• Products
• Sun Fire 15K/12K server
• Sun Fire E25K/E20K server

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Module 1, slide 6 of 13


SunU

Terminology
• Board sets
• Slot 0 boards
• Slot 1 boards
• Expanders
• Domain configuration units (DCUs)
• Slot 0 boards and Slot 1 boards
• Expander (can be split between domains)

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Module 1, slide 7 of 13


SunU

Reviewing the System Controllers
•

•
•
•
•
•
•
•
•
•
•

Provides an interface to the customer’s network
Provides a programmable system and processor clock
Sets up the server and coordinates the boot process
Monitors environmental sensors
Indicates the status and control of power supplies
Analyzes errors and takes corrective action
Provides the server console functionality
Provides a fully redundant system controller board
Sets up the server domains
Provides centralized time-of-day
Provides centralized reset logic

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Module 1, slide 8 of 13

SunU

Sun Fire 15K Server System Interconnect
Unique 3-level bus architecture
System Control boardset

9

43 GB/s
centerplane

PCI card
PCI card
PCI card

PCI
ctl
PCI
ctl

1.2

1.2

Address

Mem

Dual
3X3

CPU
PCI
ctl
PCI
ctl

CPU/Mem board

CPU
3x3 xbar

Address control

2.4

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Mem

Mem
PCI card
PCI card
PCI card

PCI board

PCI card

Address

4.8

CPU

Address

CPU

Mem

Dual
3X3

3X3

Mem

4.8

CPU

3x3 data crossbar

4.8

18x18
Response

CPU

18x18 Data crossbar

Mem

Address control

CPU

3x3 data crossbar

Mem

3x3 xbar

Dual
3X3

Dual
3X3
Slot-1 boards

Address

2.4

3X3

Slot-0 boards

CPU

18x18
Address

s

et

ds

ar

Bo

Mem
2.4

PCI card

Module 1, slide 9 of 13


SunU

Understanding Memory
Coherency
• Snooping within an expander board
• Scalable shared memory (SSM)

coherency across board sets
Snoopy coherence domain

IO

IO

P

P

P

P

M

M

M

M

SSM
Agent

Snoopy coherence domain

IO

IO

P

P

P

P

M

M

M

M

SSM
Agent

Snoopy coherence domain

IO

IO

P

P

P

P

M

M

M

M

P

Processor

M

Memory unit
(controlled by a processor

IO

I/O controller

Scalable Shared Memory (SSM) Interconnect

Snoopy coherence domain

SSM
Agent

SSM
Agent

P

P

M

M

M

M

IO

IO

P

P

P

P

M

M

M

M

IO

IO

Snoopy coherence domain

Broadcast transactions
on an address bus

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

P

Snoopy coherence domain

SSM
Agent

SSM
Agent

P

P

P

P

P

M

M

M

M

IO

IO

Point-to-point transactions
on the SSM interconnect

Module 1, slide 10 of 13


SunU

Interconnect Levels
Level 3
Level 2
Level 1
Level 0

C/M
C/M

Slot 1

e

an

rpl

nte

C/M

Ce

Expander

Slot 0

C/M

I/O
I/O

C/M
C/M

Slot 1

C/M

I/O
I/O

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Expander

Slot 0

C/M

Module 1, slide 11 of 13


SunU

Address Interconnect Levels
18 x 18 address crossbars and 18 x 18 response crossbar

Level 3:
Centerplane
To 17 other
board sets

System address controller

Level 2/3:
Expander

Point-to-point
coherency

L2 address repeater

Address repeater

Level 2:
Board

Proc

Proc

Proc

Address repeater

Proc

Memory Memory Memory Memory

Slot-0 board type:
4 CPUs/4 memory units

PCI
controller

Link
controller

PCI cards

Link cards

Slot-1 board type:
2 PCI controllers

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Snoopy
coherency

Module 1, slide 12 of 13


SunU

Data Interconnect Levels
Level 3:
Centerplane

Centerplane data switch (18x18) (43 GB/s bisection bandwidth)
4.8 GB/s
To 17 other
board sets

System data interface
Level 2:
Expander

4.8 GB/s
Level 1:
Board
4.8 GB/s

2.4 GB/s
Level 0:
Processor/
memory

2.4 GB/s

Data switch

Proc

Proc

2.4 GB/s

Dual proc
data switch
2.4 GB/s

Memory

Point-to-point
coherency

Level-2 data switch

Data switch
4.8 GB/s

Proc
2.4 GB/s

2.4 GB/s

Dual proc
data switch

2.4 GB/s
2.4 GB/s

Memory

Proc

Memory

Slot-0 board type:
4 CPUs/4 memory units

2.4 GB/s

Memory

1.2 GB/s
PCI
controller

2.4 GB/s
Link
controller

0.2 GB/s

PCI
card

Optics
card

0.4 GB/s

PCI
card

Optics
card

Slot-1 board types:
2 PCI controllers
1 PCI /1 link controller
2 CPUs

Sun Proprietary: Internal Use Only
Sun Fire™ High-End Server Maintenance and Administration
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved. SunU, Revision E

Board set

"GB/s" numbers are peak
bandwidths at each
part of the interconnect

0.8 GB/s each way

Module 1, slide 13 of 13