Step-by-Step: Simple Stand Alone UCS Setup in a Lab from Start to Finish

These are the notes I took during the first time I got to set up a non redundant UCS with 1 fabric interconnect, 1 IOM and 1 chassis. Please note that this is just a quick and dirty setup of UCS and should by no means be used for production purposes. The complete build document for the UCS components I wrote for an actual project was 40 pages long complete with proper design of UUID, MAC, WWP pools that requires careful thought based on the client's datacenter requirements and rack layout.

I hope these notes help other professionals out there that are just starting out and I will try to find some time to post a thorough "how to" for a fully redundant setup in the future.

Please also excuse any typos or mistakes below as I had to copy and paste the content from an Outlook email I sent myself, into notepad then back here and some of the text didn't paste properly.

To reset Cisco UCS 6100 Series Fabric Interconnect, run the following commands:

1. connect local-mgmt

2. erase configuration

Setting up management IP on the 6100 via Console:

1. Plug in network cable for management interface (Mgmt0 on 6100).

2. Console into Cisco UCS 6100 Series Fabric Interconnect (Baud rate 9600, Data bits 8, Parity None, Stop bits 1).

3. Use “console” configuration method.

4. Choose “setup” for setup mode.

5. Set password (password).

6. No to “create a new cluster” to configure standalone.

7. Enter system name (UCSDemo).

8. Enter Physical Switch Mgmt0:

a. IPv4 address (172.200.5.131).
b. IPv4 netmask (255.255.255.0).
c. Default gateway (172.200.5.1).
d. DNS: 172.200.1.5
e. Domain name: domain.com.

9. Apply and save the configuration: yes.

Continue Configuration of 6100 via GUI:

1. Access GUI via management IP address.

2. Login to GUI.

3. Navigate to the Equipment tab and select the Equipment node.



4. On the right side of the GUI, select Policies.


5. Under the Global Policies tab, select the proper Chassis Discovery Policy based on the amount of twinax cables that will be used to connect the UCS-6120XP and the UCS Chassis.


6. Save the changes.

7. End-Host Mode is the default setting but if you would like to validate this, navigate to Equipment -> Fabric Interconnects -> Fabric Interconnect A.

8. Under the General tab and under the Actions area, you will see which mode is active.


9. Configure VLANs that will be used for ESX (i.e. service console, vmotion, iSCSI, data traffic).

10. To add VLANs to the Fabric Interconnect, navigate to Equipment -> Fabric Interconnects -> Fabric Interconnect A.

11. Under the General tab under the Actions area, click on LAN Uplinks Manager.



12. Navigate to the VLANs tab and create the additional required VLANs.


13. Once the VLANs have been added in, configure the uplink Ethernet port used for northbound traffic.

14. Navigate to Equipment à Fabric Interconnects -> Fabric Interconnect A -> Unconfigured Ports.

15. Right-click on the ports used to connect to the chassis and select Configure as Uplink port.


16. Verify that the port’s mode is trunk.


17. Connect the physical cables from the Fabric Interconnect to network switch.

18. Configure the northbound switch port as a trunk port.

Connecting UCS-6120XP Fabric Interconnect to the UCS-5108 Chassis:

1. Connect the twinax cable from the Fabric Extender behind the UCS-5108 Chassis to the UCS-6120XP Fabric Interconnect.

2. Plug in Fabric Extender port 1 to Fabric Interconnect port 1. Repeat for port 2.

3. The left Fabric Extender is primary so unless there is another Fabric Interconnect to be used, the right Fabric Extender does not have to be plugged in.

4. Once the cables are plugged, we will need to configure the ports on the Fabric Interconnect as server ports.

5. Navigate to Equipment -> Fabric Interconnects -> Fabric Interconnect A -> Unconfigured Ports.



6. Right-click on the ports used to connect to the chassis and select Configure as Server port.


7. To confirm that the ports have been assigned as Server Ports, navigate to Equipment -> Fabric Interconnects -> Fabric Interconnect A -> Server Ports and review the list of ports displayed.


8. Once the Server Ports have been configured, the Chassis should now show up in the Equipment tab.


Service Profile Creation

1. Navigate to the Servers tab.


2. Right-click on the Service Profiles node and select Create Service Profile (expert).


3. Name: ESX

4. UUID Assignment: 00000000-0000-0000-0000-000000000001


5. Local Storage: Create a Specific Storage Policy

6. Mode: RAID Mirrored

7. Scrub Policy:

8. How would you like to configure SAN storage?: No vHBAs.


9. How would you like to configure LAN connectivity?: Expert


10. Name: eth0

11. MAC Address Assignment: 00:25:B5:XX:XX:XX

12. MAC Address: 00:25:B5:00:00:CE


13. Boot Policy: Create a Specific Boot Policy


Associate Service Profile to Server:

1. Navigate to the Servers tab.

2. Navigate to Servers -> Service Profiles -> Root.

3. Right-click on the Service Profile Name and select Change Service Profile Association.


4. Server Assignment: Select existing Server.


Create a set of Management IP Pool for external management:

1. The Cisco UCS manager needs a pool of IPs for external access to a server through the following:

a. KVM console
b. Serial over LAN
c. IPMI

So in order to access the console of the blade, a pool of management IPs will need to be defined (172.200.5.230, 231, 232).

2. Navigate to the Admin tab.

3. Navigate to All -> Communication Services -> Management IP Pool (ext-mgmt).

4. Select Create Block of IP Address.


-----------------------------------------------------------------------

The following is a copy and paste of the console output:

% invalid command detected at '^' marker.
UCS01-B# init-system ?
^
% invalid command detected at '^' marker.
UCS01-B# ?
acknowledge Acknowledge
clear Reset functions
commit-buffer Commit transaction buffer
connect Connect to Another CLI
decommission Decommission managed objects
discard-buffer Discard transaction buffer
exit Exit from command interpreter
recommission Recommission Server Resources
remove Remove
scope Changes the current mode
set Set property values
show Show running system information
terminal Set terminal line parameters
top Go to the top mode
up Go up one mode
where Show information about the current mode

UCS01-B# erase configuration
% Invalid command
UCS01-B# erase ?
^
% invalid command detected at '^' marker.
UCS01-B# write erase
% Invalid command
UCS01-B# config
% Invalid command
UCS01-B# remove ?
server Server

UCS01-B# set ?
cli CLI Information

UCS01-B# set cli ?
suppress-field-spillover Suppress Field Spillover
suppress-headers Suppress Table Headers
table-field-delimiter Field Delimiter for Tables

UCS01-B# scope ?
adapter Mezzanine Adapter
chassis Chassis
eth-server Ethernet Server Domain
eth-uplink Ethernet Uplink
fabric-interconnect Fabric Interconnect
fc-uplink FC Uplink
firmware Firmware
host-eth-if Host Ethernet Interface
host-fc-if Host FC Interface
monitoring Monitor the system
org Organizations
security security mode
server Server
service-profile Service Profile
system Systems
vhba vHBA
vnic vNIC

UCS01-B# scope system ?


UCS01-B# scope system
Error: DME is not ready
UCS01-B# scope fabric-interconnect
% Incomplete command
UCS01-B# scope fabric-interconnect ?
a Fabric A
b Fabric B

UCS01-B# scope fabric-interconnect b ?


UCS01-B# scope fabric-interconnect b
Error: DME is not ready
UCS01-B# connect local-mgmt
Cisco UCS 6100 Series Fabric Interconnect

TAC support: http://www.cisco.com/tac

Copyright (c) 2009, Cisco Systems, Inc. All rights reserved.

The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software may be covered under the GNU Public
License or the GNU Lesser General Public License. A copy of
each such license is available at
http://www.gnu.org/licenses/gpl.html and
http://www.gnu.org/licenses/lgpl.html
UCS01-B(local-mgmt)# erase ?
configuration System configuration

UCS01-B(local-mgmt)# erase configuration ?


UCS01-B(local-mgmt)# erase configuration
All UCS configurations will be erased and system will reboot. Are you sure? (yes/no):

yes
Removing all the configuration. Please wait....
Configurations are cleaned up. Rebooting....
2010 Apr 21 13:49:30 UCS01-B %$ VDC-1 %$ %USER-2-SYSTEM_MSG: no feature-name to add -

clis
2010 Apr 21 13:49:30 UCS01-B %$ VDC-1 %$ %PLATFORM-2-PFM_SYSTEM_RESET: Manual system

restart from Command Line Interface
writing reset reason 9,

INIT: Sending processes the TERM signal
Apr 21 13:49:35 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "Security Daemon" (PID 2895)

is forced exit.

Apr 21 Apr 21 13:49:35 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "vshd" (PID 2817) is

forced exit.

Apr 21 13:49:35 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "vdc_mgr" (PID 2856) is forced

exit.

Apr 21 13:49:35 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "statsclient" (PID 2859) is

forced exit.

Apr 21 13:49:35 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "res_mgr" (PID 2861) is forced

exit.

Apr 21 13:49:35 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "netstack" (PID 2990) is

forced exit.

Apr 21 13:49:35 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "licmgr" (PID 2822) is forced

exit.

Apr 21 13:49:35 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "l3vm" (PID 2908) is forced

exit.

Apr 21 13:49:35 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "icmpv6" (PID 2989) is forced

exit.

Apr 21 13:49:35 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "ascii-cfg" (PID 2891) is

forced exit.

Apr 21 13:49:35 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "adjmgr" (PID 2958) is forced

exit.

Apr 21 13:49:36 %SYSMGR-3-SHUTDOWN_OVER The System Manager is shutting down now.
Apr 21 13:49:36 %SYSMGR-2-RESTART_SYSTEM_LOG We will be restarting system soon for vdc

4.
Apr 21 13:49:36 %SYSMGR-2-RESTART_SYSTEM_LOG We will be restarting system soon for vdc

3.
Apr 21 13:49:36 %SYSMGR-2-RESTART_SYSTEM_LOG We will be restarting system soon for vdc

2.
Apr 21 13:49:36 %SYSMGR-2-RESTART_SYSTEM_LOG We will be restarting system soon for vdc

1.
Unexporting directories for NFS kernel daemon...done.
Stopping NFS kernel daemon: rpc.mountd rpc.nfsddone.
Unexporting directories for NFS kernel daemon...
done.
Stopping portmap daemon: portmap.
Stopping kernel log daemon: klogd.
Sending all processes the TERM signal... done.
Sending all processes the KILL signal... done.
Unmounting remote filesystems... done.
Deactivating swap...done.
Unmounting local filesystems...done.
mount: you must specify the filesystem type
Starting reboot command: reboot
Rebooting...
Restarting system.


----------------------------


Booting kickstart image: bootflash:/installables/switch/ucs-6100-k9-kickstart.4
.0.1a.N2.1.2d.bin....
.......................................................................Image ve
rification OK

Starting kernel...
Usage: init 0123456SsQqAaBbCcUu
INIT: version 2.85 booting
Starting UCS6120XP POST...
Executing Mod 1 1 SEEPROM Test......done
Executing Mod 1 1 GigE Port Test.......done
Executing Mod 1 1 Inband GigE Test.....done
Executing Mod 1 1 NVRAM Test....done
Executing Mod 1 1 PCIE Test..............................done
Mod 1 1 Post Completed Successfully
Mod 2 Post Completed Successfully
POST is completed
Creating /callhome..
Mounting /callhome..
Creating /callhome done.
Callhome spool file system init done.
Checking all filesystems.....rrr done.
Checking NVRAM block device ... done
The startup-config won't be used until the next reboot.
.
Loading system software
Uncompressing system image:

bootflash:/installables/switch/ucs-6100-k9-system.4.0.1a.N2.1.2d.bin

Loading plugin 0: core_plugin...
plugin_link_to_exec_path: plugin_path = /isan/plugin/0, tar_log =

/isan/plugin_extract_log/0
Loading plugin 1: eth_plugin...
plugin_link_to_exec_path: plugin_path = /isan/plugin/1, tar_log =

/isan/plugin_extract_log/1
plugin_file_is_excluded_from_exec_path: /boot/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so is excluded from

linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so.0.0.0 is

excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so.0 is excluded

from linking
plugin_file_is_excluded_from_exec_path: /boot/etc/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/etc/plugin_exclude.conf is excluded from

linking
plugin_file_is_excluded_from_exec_path: /boot/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so is excluded from

linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so.0.0.0 is

excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so.0 is excluded

from linking
plugin_file_is_excluded_from_exec_path: /boot/etc/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/etc/plugin_exclude.conf is excluded from

linking
Loading plugin 2: fc_plugin...
plugin_link_to_exec_path: plugin_path = /isan/plugin/2, tar_log =

/isan/plugin_extract_log/2
plugin_file_is_excluded_from_exec_path: /boot/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so is excluded from

linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so.0.0.0 is

excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so.0 is excluded

from linking
plugin_file_is_excluded_from_exec_path: /boot/etc/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/etc/plugin_exclude.conf is excluded from

linking
plugin_file_is_excluded_from_exec_path: /boot/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so is excluded from

linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so.0.0.0 is

excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so.0 is excluded

from linking
plugin_file_is_excluded_from_exec_path: /boot/etc/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/etc/plugin_exclude.conf is excluded from

linking
121316+1 records in
121316+1 records out
ethernet end-host mode on CA
FC end-host mode on CA
n_port virtualizer mode.
---------------------------------------------------------------
ethernet end-host mode
INIT: Entering runlevel: 3
Exporting directories for NFS kernel daemon...done.
Starting NFS kernel daemon:rpc.nfsd.
rpc.mountddone.

Setting envvar: SYSMGR_SERVICE_NAME to muxif_service
Set envvar SYSMGR_SERVICE_NAME to muxif_service
/isan/bin/muxif_config: argc:2
muxif_init....vacl: ret: 0
Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
Added VLAN with VID == 4042 to IF -:muxif:-
---------------------
enabled fc feature
---------------------
pss_store_sys_list: handle 0x831c158 vdc 1 err 0x0
2010 Apr 21 13:53:22 %$ VDC-1 %$ %KERN-2-SYSTEM_MSG: Starting kernel... - kernel
2010 Apr 21 13:53:38 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: no feature-name to add - clis
Executing Port Power On Tests...........System is coming up ... Please wait ...
.System is coming up ... Please wait ...
Done
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
2010 Apr 21 13:54:53 %$ VDC-1 %$ %VDC_MGR-2-VDC_ONLINE: vdc 1 has come online
System is coming up ... Please wait ...
nohup: appending output to `nohup.out'

---- Basic System Configuration Dialog ----

This setup utility will guide you through the basic configuration of
the system. Only minimal configuration including IP connectivity to
the Fabric interconnect and its clustering mode is performed through these steps.

Type Ctrl-C at any time to abort configuration and reboot system.
To back track or make modifications to already entered values,
complete input till end of section and answer no when prompted
to apply configuration.


Enter the configuration method. (console/gui) ? console

Enter the setup mode; setup newly or restore from backup. (setup/restore) ? setup

You have chosen to setup a new Fabric interconnect. Continue? (y/n): y

Enter the password for "admin":
Confirm the password for "admin":
password should be atleast 8 characters

Enter the password for "admin":
Confirm the password for "admin":
it is based on a dictionary word

Enter the password for "admin":
Confirm the password for "admin":
Admin password confirmation failed. Please try again.

Enter the password for "admin":
Confirm the password for "admin":

Do you want to create a new cluster on this Fabric interconnect (select 'no' for

standalone setup or if you want this Fabric interconnect to be added to an existing

cluster)? (yes/no) [n]: no

Enter the system name: UCSDemo

Physical Switch Mgmt0 IPv4 address : 172.200.5.131

Physical Switch Mgmt0 IPv4 netmask : 255.255.255.0

IPv4 address of the default gateway : 172.200.5.1

Configure the DNS Server IPv4 address? (yes/no) [n]: y

DNS IPv4 address : 172.200.1.5

Configure the default domain name? (yes/no) [n]: unislumin.com

Configure the default domain name? (yes/no) [n]: y

Default domain name : unislumin.com

Following configurations will be applied:

Switch Fabric=A
System Name=UCSDemo
Physical Switch Mgmt0 IP Address=172.200.5.131
Physical Switch Mgmt0 IP Netmask=255.255.255.0
Default Gateway=172.200.5.1
DNS Server=172.200.1.5
Domain Name=unislumin.com

Apply and save the configuration (select 'no' if you want to re-enter)? (yes/no): y
Applying configuration. Please wait.

Configuration file - Ok


Cisco UCS 6100 Series Fabric Interconnect
UCSDemo-A login: admin
Password:
Login incorrect

UCSDemo-A login: admin
Password:
Cisco UCS 6100 Series Fabric Interconnect

TAC support: http://www.cisco.com/tac

Copyright (c) 2009, Cisco Systems, Inc. All rights reserved.

The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software may be covered under the GNU Public
License or the GNU Lesser General Public License. A copy of
each such license is available at
http://www.gnu.org/licenses/gpl.html and
http://www.gnu.org/licenses/lgpl.html
UCSDemo-A# ^C
UCSDemo-A#