Quick note about reconfiguring an ESX 4.x server’s service console to different vmnics

So I had quite the “kick off” on my first day back at work for the year 2011. To cut a long story short, I was asked to figure out why 3 of our UCS blades were knocked offline from vCenter and the rest of the network even though accessing them from UCSM’s KVM console showed that they were up. Aside from spending over an hour reviewing logs and finally realizing that someone managed to delete the the service profiles for the UCS blades (I’ll write a post about the troubleshooting process and resolution at a later time), I found myself at the point where I had to shuffle around the 2 vmnics (this UCS has a menlo card) from a Nexus 1000v switch back to a regular vSwitch all through the CLI. I’d have to say I’m not much of an expert with the commands and while I do know the basics, I can never remember the switches. What I ended up using the following Cisco document that includes all the commands I ended up using to:

1. Remove a vmnic from a Nexus 1000v.
2. Delete a vswif interface mapped to a port on the Nexus 1000v.
3. Create a regular vSwitch.
4. Recreate a vswif interface on a regular vSwitch.
5. Add the vswif to a Service Console port on the regular vSwitch.

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9902/white_paper_c11-558242.html

I’ve also pasted the section I used at the bottom in case I ever need to reference this again and the link is no longer available.

Update:  The information provided below doesn’t show you how to remove a vmkernel port from a Nexus 1000v or distributed switch so I have followed up with another blog post to show how to do this: http://terenceluk.blogspot.com/2011/01/quick-note-about-reconfiguring-esx-4x_05.html

----------------------------------------------------------------------------------------------------------------------------------------------------

VMware ESX and ESXi Commands to Troubleshoot Cisco Nexus 1000V Series Switches in the Cisco Unified Computing System

In VMware ESX with a service console OS, these commands are in the /usr/sbin directory. Press ALT-F1 to display the service console login prompt, and press ALT-F11 to leave the prompt. To get to a misconfigured Cisco UCS blade for recovery, click the KVM item under the server view. This procedure assumes that the KVM IP addresses have been set up and assigned.

In VMware ESXi, the esxcfg-vswitch functions are all present as in VMware ESX. Since VMware ESXi does have a service console OS, use ALT-F2 and ALT-F11 key presses to go to and from the VMware ESXi command line. All the commands listed here for esxcfg-vswif will now be replaced with esxcfg-vmknic, and you can update specific details as required.

To List Information About the Current Configuration

esxcfg-vswif -l To list information about the current vswif interface

esxcfg-vswitch -l To list information about the current vSwitches and DVSs

To Manage the Existence of a vSwitch

esxcfg-vswitch -a vSwitch0 To add a vSwitch named vSwitch0 to the server

esxcfg-vswitch -d vSwitch0 To delete a vSwitch named vSwitch0 from the server

To Manage the Port Groups on a vSwitch

esxcfg-vswitch -A "Service Console" vSwitch0 To add a port group named Service Console to vSwitch0

esxcfg-vswitch -D "Service Console" vSwitch0 To remove a port group named Service Console from vSwitch0

To Assign VLAN Membership to a Port Group on a vSwitch

esxcfg-vswitch -p "Service Console" -v 102 vSwitch0 To set a port group named Service Console to user VLAN 102 on vSwitch0

To View the Available NICs That VMware ESX Can See

esxcfg-nics -l To see a list of VMNICs

To Manage Uplinks from the DVS or Cisco Nexus 1000V Series Switch

First run the list commands (esxcfg-vswif -l and esxcfg-vswitch -l ) to get the port numbers for the existing uplinks (vmnic0, vmnic1, etc.); this will be port#.

esxcfg-vswitch -Q vmnic0 -V [port#] myN1kDVS To remove the vmnic0 uplink from the DVS named myN1kDVS

esxcfg-vswitch -P vmnic0 -V [port#] myN1kDVS To add the vmnic0 uplink to the DVS named myN1kDVS

To add the VMNICs back into a DVS, the recommended approach is to use VMware vCenter and not the command line.

To Manage Uplinks to a vSwitch

esxcfg-vswitch -L vmnic0 vSwitch0 To add vmnic0 as an uplink to vSwitch0

esxcfg-vswitch -U vmnic0 vSwitch0 To remove vmnic0 as an uplink to vSwitch0

To remove a VMNIC from a vSwitch, the recommended approach is to use VMware vCenter and not the command line while migrating from a vSwitch to a Cisco Nexus 1000V Series DVS.

To Delete a vswif Interface from a VMware ESX Server Port Group

esxcfg-vswif -d vswif0 -p "Service Console" vSwitch0 To delete a vswif interface from the port group on vSwitch0

Esxcfg-vswif -d vswif0 -p "Service Console" -P xxx -V myN1kDVS To delete a vswif interface from the port group on the Cisco Nexus 1000V Series Switch

This deletion is a common task if an error situation arises and you lose connectivity to the VMware ESX server when migrating to the Cisco Nexus 1000V Series. You will have to add the port group to vSwitch0 and then add the vswif interface on it.

To Add a vswif Interface to the VMware ESX Server

esxcfg-vswif -a vswif0 -p "Service Console" -i 10.1.1.10 -n 255.255.255.0 To add vswif0 with IP 10.1.1.10/24 to port group Service Console

esxcfg-vswif -a vswif0 -p "Service Console" -i 10.1.1.10 -n 255.255.255.0 -P [port#]-V myN1kDVS To add vswif0 with IP 10.1.1.10/24 to port group Service Console to port = port# on DVS myN1kDVS. The port# value is usually just the last unused entry in the list of available port numbers when the DVS configuration is listed with esxcfg-vswitch -l.

To Manage a Default Gateway for the VMware ESX Server

On the VMware vCenter GUI, you can change the gateway directly, but on the command-line interface (CLI), you must edit the /etc/sysconfig/network file and update the vswif interface's default gateway. The file will look similar to this:

NETWORKING=yes

HOSTNAME=ESXServerBlade4.cisco.com

GATEWAY=10.1.1.1

After saving the file, you must reboot the VMware ESX server to implement the change.

Troubleshooting Commands

To Ping Test a VMware VMkernel Interface

esxcfg-route -l To get the routes on the VMware VMkernel network

vmkping X.X.X.X To ping an IP address on the VMware VMkernel network

To Test for the VEM Installation on a VMware ESX Host

vmkload_mod -l grep vem To verify that the appropriate VEM is loaded (the command should return the modules)