Cisco FabricPath Lab

Cisco FabricPath Lab!
!
Michał Skiba
Software Engineer




Feedback:

Release 1.0
Cisco FabricPath Lab
© 2011 Cisco Systems, Inc. All rights reserved 2 of 39



Cisco FabricPath Lab

Michał Skiba

Copyright  2011 Cisco Systems, Inc.

All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means,
electronic or mechanical, including photocopying, recording, or by any information storage and retrieval
system, without written permission from the publisher, except for the inclusion of brief quotations in a
review.

Printed in the United States of America

First Printing March 2011

Warning and Disclaimer

This book is designed to provide information about the Nexus Operating system and Nexus family of
products. Every effort has been made to make this book as complete and as accurate as possible, but
no warranty or fitness is implied. The information is provided on an “as is” basis. The authors and Cisco
Systems, Inc. shall have neither liability nor responsibility to any person or entity with respect to any loss
or damages arising from the information contained in this book or from the use of the discs or programs
that may accompany it. The opinions expressed in this book belong to the author and are not
necessarily those of Cisco Systems, Inc.

Trademark Acknowledgments

All terms mentioned in this book that are known to be trademarks or service marks have been
appropriately capitalized. Cisco Systems, Inc. cannot attest to the accuracy of this information. Use of a
term in this book should not be regarded as affecting the validity of any trademark or service mark.

Feedback Information

Our goal is to create in-depth technical books of the highest quality and value. Each book is crafted with
care and precision, undergoing rigorous development that involves the unique expertise of members
from the professional technical community. Readers’ feedback is a natural continuation of this process.
If you have any comments regarding how we could improve the quality of this book, or otherwise alter it
to better suit your needs, you can contact us through email at:

We greatly appreciate your assistance.

Cisco has more than 200 offices worldwide. Addresses, phone numbers, and fax numbers are listed on the Cisco Website at www.cisco.com/go/offices

CCDE, CCENT, Cisco Eos, Cisco HealthPresence, the Cisco logo, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, DCE, and Welcome to the Human
Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You,
Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital,
the Cisco Systems logo, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet
Quotient, IOS, iPhone, iQuick Study, IronPort, the IronPort logo, LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy,
Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient,
TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.
All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between
Cisco and any other company. (0812R)
Cisco FabricPath Lab
© 2011 Cisco Systems, Inc. All rights reserved 3 of 39


Introduction

The modern data center is rapidly changing and evolving to support the current and future
demands of technology. At the center of this change is the network—the single entity that
connects everything and touches all components of the data center. With that in mind, Cisco
has launched a new series of switches, Nexus, based on a revolutionary new operating
system, NX-OS, to meet these changes and provide a platform with the scalability, reliability,
and comprehensive feature set required in the next generation data center.
Nexus 7000

The Cisco Nexus 7000 Series of modular switches is available in 10-slot and 18-slot
configurations, and is capable of more than 15 terabits per second (Tbps) of switching
capacity while offering market-leading Gigabit Ethernet and 10 Gigabit Ethernet density. Built
on a zero-service-loss hardware and software architecture, the Cisco Nexus 7000 Series
offers the kind of high availability needed in a next-generation data center, in which

virtualization increases the scope of downtime and Unified Fabric demands Fibre Channel-
like availability to properly support storage services. With manageability in mind, the Cisco
Nexus 7000 Series incorporates a number of unique features, including integrated lights-out
management and integrated packet capture and decoding. The Cisco Nexus 7000 also offers
innovative switch virtualization capabilities, which, in combination with the switch's density,
allows customers to greatly simplify their switching infrastructure, reducing costs, power and
cooling load, and management complexity.
Cisco FabricPath Lab
© 2011 Cisco Systems, Inc. All rights reserved 4 of 39


NX-OS

The NX-OS data center-class operating system was built by Cisco with scalability, modularity,
resiliency, and serviceability at its foundation. NX-OS is based on the industry-proven Cisco
Storage Area Network Operating System (SAN-OS) Software and helps ensure continuous
availability to set the standard for mission-critical data center environments. The self-healing
and highly modular design of Cisco NX-OS enables excellent service levels and exceptional
operational flexibility.
Lab Objectives

This self-paced hands-on lab will introduce participants to Cisco FabricPath, a Layer-2 NX-
OS innovation. FabricPath’s ease of provisioning, highly scalable east-west bandwidth and
fast network convergence time allow administrators to provision very large layer-2 domains
with confidence while maximizing interconnect bandwidth. Cisco FabricPath enables this
without the need for complex configurations. By the end of this lab session participants should
be able to understand Cisco FabricPath and vPC+ functionality and configuration with the
Nexus 7000 & 5000 platforms.

Lab Procedure

The Lab consists of multiple pods, each of which represents a simplified Nexus 7000 Data
Center site. Nexus 7010 switches are used as aggregation layer or spine devices. The Nexus
5548 switch represents the access layer and attached to it are servers with VMware’s ESX
hypervisor.

A group of two students is assigned to each pod, and each student will be configuring two
Nexus 7000 devices. The length of the lab will depend on the student’s level of experience
with NX-OS, but should not last longer than two hours. While the student has access to
physical Nexus 7000 hardware, Virtual Device Contexts (VDCs) are used to consolidate
multiple nodes and optimize the number of chassis and amount of power required.


The lab procedure has four main steps:
1. NX-OS CLI familiarization by performing system verification, management VRF
testing, and establishing basic connectivity (optional)
2. Base configuration with the Spanning Tree Protocol
3. Configuring Cisco FabricPath to establish active/active 4-way connections between
Nexus 7000 devices
4. Connecting edge devices to the FabricPath core with vPC+

General Disclaimer
The content in this book is current as of the NX-OS 5.1(3) software release. It is expected that
the FabricPath feature set will grow over time. The most up-to-date information regarding NX-
OS on Nexus 7000 can be found online at:


Cisco FabricPath Lab
© 2011 Cisco Systems, Inc. All rights reserved 5 of 39

Feedback
Our goal is to create in-depth technical lab procedures of the highest quality and value. If you
have any comments regarding how we could improve the quality of this document, or
otherwise alter it to better suit your needs, you can contact us through email at
We greatly appreciate your input.

Acknowledgements
This lab was implemented with support from the Cisco GOLD Lab Operations team members
Liz Chen, Shane Hudson, Gernot Kindel and Divyaben Bhalani. Extremely valuable support
and feedback was provided by members of the Nexus 7000 marketing team; most notably
Nikhil Kelshikar, Ben Basler, Tim Stevenson and Francois Tallet.
Cisco FabricPath Lab
© 2011 Cisco Systems, Inc. All rights reserved 6 of 39


Lab Topology and Access



The diagram above shows the complete connectivity that can be achieved within a pod. By
the end of this lab procedure, all links between Nexus 7000 switches, as well as the vPC+
connection to the Nexus 5000 switch, should be configured and operational. All links between
the Nexus 7000 switches will be connected with the Cisco FabricPath protocol, while the
remainder will leverage the Spanning Tree Protocol. Two students will be assigned to each
pod, and each student will be configuring two different Nexus 7000 devices (N7K-TOP and
N7K-BOTTOM) from their Windows 2003 Virtual Machine. The Nexus 5548 (N5K) is shared
amongst students and only needs to be configured by one of the students. The Catalyst 3750
will not be accessible to students. Future releases of this lab procedure will outline how to
connect the Windows 2003 Virtual Machines, and the Catalyst 3750 switch.

N7K-BOTTOM N7K-BOTTOM
N7K-TOPN7K-TOP
STUDENT VM STUDENT VM
N5K
Catalyst 3750
vPC+
STUDENT A STUDENT B
Cisco FabricPath Lab
© 2011 Cisco Systems, Inc. All rights reserved 7 of 39





The pod topology can be thought of in terms of network layers, as shown in the diagram
above. The Nexus 5548 acts as an access layer switch in this topology. The aggregation and
core layers are built on Nexus 7010 (10-slot) chassis, each running NX-OS version 5.1(3).
Each chassis has an F1 I/O module (32-port 10GE fiber module, model N7K-F132XP-15)
installed. These modules will be used to create parallel 4-way Cisco FabricPath connections
to each Nexus 7010 switch within the aggregation and core layers. N7K-BOTTOM, the
aggregation layer chassis facing the access layer, have an additional F1 module installed in
order to provide connectivity to the Nexus 5548 switch. N7K-TOP, the chassis facing the
Layer 3 network core, have an M1 I/O module (48-port Gigabit Ethernet Copper module,
model N7K-M148GT-11) installed, which supports Layer 3 routing protocols. The Layer 3
network is abstracted by the Catalyst 3750.

Cisco FabricPath Lab
© 2011 Cisco Systems, Inc. All rights reserved 8 of 39

While the student has access to physical Nexus 7000 hardware, Virtual Device Contexts
(VDCs) are used to consolidate multiple nodes and optimize the number of chassis and
amount of power required. Only the interfaces that are necessary for achieving the previously
described connectivity have been assigned to the VDCs. Following table maps the N7K-TOP
and N7K-BOTTOM shorthand designations to the names of the actual VDCs that students will
be configuring:

POD # Student # N7K-TOP N7K-BOTTOM
1 A N7K-1-POD-1 N7K-3-POD-1
1 B N7K-2-POD-1 N7K-4-POD-1
2 A N7K-1-POD-2 N7K-3-POD-2
2 B N7K-2-POD-2 N7K-4-POD-2
3 A N7K-1-POD-3 N7K-3-POD-3
3 B N7K-2-POD-3 N7K-4-POD-3
4 A N7K-1-POD-4 N7K-3-POD-4
4 B N7K-2-POD-4 N7K-4-POD-4
5 A N7K-5-POD-5 N7K-7-POD-5
5 B N7K-6-POD-5 N7K-8-POD-5
6 A N7K-5-POD-6 N7K-7-POD-6
6 B N7K-6-POD-6 N7K-8-POD-6
7 A N7K-5-POD-7 N7K-7-POD-7
7 B N7K-6-POD-7 N7K-8-POD-7
8 A N7K-5-POD-8 N7K-7-POD-8
8 B N7K-6-POD-8 N7K-8-POD-8


Lab Access

This Lab is accessible through the Cisco Global On-line Learning Distribution (GOLD)
infrastructure, which provides students with the ability to learn about Cisco products and

solutions through a convenient, convincing and hands-on approach. In this lab, you will have
the opportunity to gain the invaluable experience of configuring physical hardware in the
topology that is reflective of a realistic deployment.

Each FabricPath GOLD lab is a scheduled session for which you will need to register.
Following registration, you will obtain a session timeslot and class name. At the start time of
your session, you will need to access the GOLD Labs student portal and provide your email
and class name. the portal can be found at:

Note: If you are using the GOLD lab for the first time, then you need to register prior logging
in. This is a one-time process and can be initiated by clicking on “REGISTER” under the
above URL. Once registered, you will be redirected back to this page and can proceed to log
in with your email address and class name.

Cisco FabricPath Lab
© 2011 Cisco Systems, Inc. All rights reserved 9 of 39



Following a successful login, you will find a table listing the FabricPath Lab pods. As
explained in the previous section, each pod is the topology that you will be configuring along
with a partner. If your partner has already registered with the pod, then their name will be
displayed next to the pod number. To enter the pod, click on the “pick” link.

Cisco FabricPath Lab
© 2011 Cisco Systems, Inc. All rights reserved 10 of 39



Important: If you find yourself without a partner, then you will need to configure both sides of

the pod by yourself.

Having entered the pod, you will see a frame within your browser window with four tabs:
Toplogy, Overview, Instructions and Help. This manual is found under the instructions tab,
while under the topology tab you will find a wiring diagram for this lab. No interfaces have
been labeled on the diagram since it is a generic diagram for all pods in this lab infrastructure.



The Nexus 7010 & 5548 switches are accessible through the Windows 2003 Virtual Machines
that are associated with each pod. These machines can be accessed with a client that
supports the Remote Desktop Protocol (RDP). Users with the Apple OS-X operating system
can download the Microsoft RDP client from:


The connection can be initiated by clicking on the laptop icon in the topology, which is located
at the bottom on the diagram. A shortcut to the RDP connection will begin to download, and
you should chose to open it with your RDP client. The login credentials are:





Login Password
student Nbv_12345
Cisco FabricPath Lab
© 2011 Cisco Systems, Inc. All rights reserved 11 of 39

Once you have logged into the Windows 2003 Virtual Machine, you can use PuTTY, which is
located on the desktop, to connect with the management interfaces of the Nexus 7010 &
5548 switches that you will be configuring during the lab procedure. After opening PuTTY,
you will have a list of saved sessions. Two will correspond to the Nexus 7010 switches and
one to the Nexus 5548 switch, which is shared in the pod amongst you and your partner. In
the table below, “N7K-TOP” denotes the Nexus 7010 switch facing the Layer 3 cloud, and
“N7K-BOTTOM” denotes the Nexus 7010 switch facing the Nexus 5548 switch (see topology
diagram). As with the RDP session, the login for these devices is student and
corresponding password is Nbv_12345

Important: if you are performing this lab without a partner, then you will have to perform the
configuration for both sides of your pod. To gain access to the left and the right Nexus 7000
switches you will need open both RDP sessions.




Note: When establishing an SSH connection with a Nexus 7000 management interface for
the first time, you may be asked to confirm the host verification. Please do so.


Cisco FabricPath Lab
© 2011 Cisco Systems, Inc. All rights reserved 12 of 39


Step 1: NX-OS Familiarization


This section serves to familiarize users with the NX-OS operating system and is not required
to complete the FabricPath topology configuration. You can skip this step if you are already

familiar with the Nexus 7000 hardware and software infrastructure and proceed directly to the
next step, Spanning Tree.

All configuration in this lab will be performed through the management console of each
switch, and over the SSH protocol. Nexus switches also support the Telnet protocol.
This section will begin by checking the system settings of each switch in the pod. If you have
not already done so, access the Windows 2003 virtual machine corresponding to your student
number and use Putty to access either chassis in your half of the pod. The appropriate
network addresses and login credentials can be found on the previous page. A shortcut to
Putty can be found on the desktop of the Windows 2003 virtual machine. The following output
corresponds to that of the N7K-TOP switch in Pod #1, but will be very similar for all other
switches and pods.
N7K-1-POD-1# show module

Mod Ports Module-Type Model Status
--- ----- -------------------------------- ------------------ ------------
1 32 1/10 Gbps Ethernet Module N7K-F132XP-15 ok
3 48 10/100/1000 Mbps Ethernet Module N7K-M148GT-11 ok
5 0 Supervisor module-1X N7K-SUP1 active *

Mod Sw Hw
--- -------------- ------
1 5.1(3) 1.1
3 5.1(3) 1.3
5 5.1(3) 1.4

Mod MAC-Address(es) Serial-Num
--- -------------------------------------- ----------
1 e0-5f-b9-42-8d-d8 to e0-5f-b9-42-8e-1c JAF1449DRRP
3 00-24-f7-1d-00-fc to 00-24-f7-1d-01-30 JAF1313AERM

5 00-26-51-c8-7d-a0 to 00-26-51-c8-7d-a8 JAF1326ACEL

Mod Online Diag Status
--- ------------------
1 Pass
3 Pass
5 Pass

Xbar Ports Module-Type Model Status
--- ----- -------------------------------- ------------------ ------------
1 0 Fabric Module 1 N7K-C7010-FAB-1 ok
2 0 Fabric Module 1 N7K-C7010-FAB-1 ok
3 0 Fabric Module 1 N7K-C7010-FAB-1 ok

Xbar Sw Hw
--- -------------- ------
1 NA 1.0
2 NA 1.0
3 NA 1.0


Cisco FabricPath Lab
© 2011 Cisco Systems, Inc. All rights reserved 13 of 39


Xbar MAC-Address(es) Serial-Num
--- -------------------------------------- ----------
1 NA JAB122800F9
2 NA JAB121602JC
3 NA JAB122000J1

* this terminal session

N7K-1-POD-1# show version
Cisco Nexus Operating System (NX-OS) Software
TAC support:
Copyright (c) 2002-2010, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
and


Software
BIOS: version 3.22.0
kickstart: version 5.1(3)
system: version 5.1(3)
BIOS compile time: 02/20/10
kickstart image file is: bootflash://n7000-s1-kickstart.5.1.3.bin
kickstart compile time: 12/25/2010 12:00:00 [02/23/2012 01:03:42]
system image file is: bootflash://n7000-s1-dk9.5.1.3.bin
system compile time: 1/21/2011 19:00:00 [02/23/2011 22:01:45]

Hardware
cisco Nexus7000 C7010 (10 Slot) Chassis ("Supervisor module-1X")
Intel(R) Xeon(R) CPU with 4109560 kB of memory.
Processor Board ID JAF1326ACEL

Device name: N7K-1
bootflash: 2030616 kB
slot0: 0 kB (expansion flash)

Kernel uptime is 7 day(s), 20 hour(s), 17 minute(s), 45 second(s)

Last reset at 143637 usecs after Thu Feb 24 10:45:14 2011

Reason: Reset Requested by CLI command reload
System version: 5.1(3)
Service:

plugin
Core Plugin, Ethernet Plugin



NX-OS is composed by two images: a kickstart image that contains a custom Linux kernel
and a system image that contains the NX-OS software components. Both are listed in the
configuration.


Active Plug-ins
Storage Devices
CPU
Image locations

NX-OS Version
Cisco FabricPath Lab

© 2011 Cisco Systems, Inc. All rights reserved 14 of 39



Let’s take a look at the running configuration.


N7K-1-POD-1# show running-config
version 5.1(3)

<omitted interface config>
interface Ethernet1/1

interface Ethernet1/2

interface Ethernet1/3

interface Ethernet1/4


<omitted interface config>



interface mgmt0
ip address 10.0.8.11/24
line vty


Let’s also see the interfaces that are available to us for configuration:

N7K-1-POD-1# show interface brief

----------------------------------------------------------------------------
Port VRF Status IP Address Speed MTU
----------------------------------------------------------------------------
mgmt0 -- up 10.0.8.11 1000 1500

----------------------------------------------------------------------------
Ethernet VLAN Type Mode Status Reason Speed Port
Interface Ch #
----------------------------------------------------------------------------
Eth1/1 1 eth access down Administratively down auto(D) --
Eth1/2 1 eth access down Administratively down auto(D) --
Eth1/3 1 eth access down Administratively down auto(D) --
Eth1/4 1 eth access down Administratively down auto(D) --
Eth1/5 1 eth access down Administratively down auto(D) --
Eth1/6 1 eth access down Administratively down auto(D) --
Eth1/7 1 eth access down Administratively down auto(D) --
Eth1/8 1 eth access down Administratively down auto(D) --
Eth3/1 -- eth routed down Administratively down auto(D) --


The outputs above show the configuration for pod number 1. As explained earlier, each pod
runs within a Virtual Device Context (VDC). By using the VDC feature, we can segment the
physical Nexus 7000 into multiple logical switches, each of which runs in a separate memory
space and only has visibility into the hardware resources that it owns, providing total isolation
between the VDCs. This is the reason why only the subset of interfaces that will be wired in
the lab procedure for the pod 1 has been is made available to the VDC.

These are the interfaces available to
your Pod (Virtual Device Context)
Management
Interface
Cisco FabricPath Lab
© 2011 Cisco Systems, Inc. All rights reserved 15 of 39


One of the features of the show running-config command is the ability to display both
the running configuration and default values, and this can be done with the keyword all, as
shown below.

N7K-1-POD-1# show running-config all | section mgmt0
interface mgmt0
no description
speed auto
duplex auto no shutdown
cdp enable
ip address 10.0.8.11/24



Management VRF and Basic Connectivity

The management interface, mgmt0, is always part of the management VRF and is the only
interface allowed to be part of this VRF. The Management VRF provides total isolation of
management traffic from the rest of the traffic flowing through the Nexus 7000 chassis.
In the following steps, you will learn how to verify that:
- Only the mgmt0 interface is part of the management VRF

- No other interface can be part of the management VRF
- The default gateway is reachable only using the management VRF

N7K-1-POD-1# show vrf

VRF-Name VRF-ID State Reason
default 1 Up --
management 2 Up --

N7K-1-POD-1# show vrf management interface
Interface VRF-Name VRF-ID
mgmt0 management 2

The management VRF is part of the default configuration, and the management interface
mgmt0 is the only interface that can be made a member of this VRF. This membership will be
verified in the following procedure.



N7K-1-POD-1# conf t
Enter configuration commands, one per line. End with CNTL/Z.
N7K-1-POD-1(config)# interface ethernet 3/1

FastEthernet? GigabitEthernet?...
no, just “ethernet” interfaces