Thursday, January 20, 2011

VMware ESXi 4.1 show’s Intel 82576 Gigabit NIC on Cisco UCS C210 M2 as 10Mbps Full

One of the reasons why I like what I do an my position at the company is that whenever there’s any problems with a Cisco UCS B or C series servers, I’m one of the few who always get the email, call or IM about it so here’s another one to add to all the other posts I have about the problems I’ve experienced:

Problem

I received an email from one of my colleagues about two of our Cisco UCS C210 M2 servers which are about to go live in a month are showing that the Intel Corporation 82576 Gigabit Network Connection NIC is connected at 10Mbps Full.  Here’s what it looks like via the VI Client:

image

Solution

So what solved it?  A simple firmware update on the UCS C210 M2 got ESXi 4.1 Build 260247 to show the NIC at the proper speed.  Now before I include more screenshots, I’d like to point out that I don’t think this problem was caused by an older firmware and the reason is because if that was the case, wouldn’t this have become a public Cisco KB?  My suspicion is more that somehow a mismatch of BIOS, CIMC and LOM firmware is what caused it.  I wasn’t the one who set up this server and I know an update was made to the CIMC but not the rest.  Here are the version I found prior to updating the firmware:

BIOS Version: C200.1.1.1.a.0 (Build Date: 03/29/2010)

image

CIMC Firmware:

Running Version: 1.2(1b)

Backup Version: 1.1(1d)

Boot-loader Version: 1.2(1b).6

image

Here’s another shot via the Host Based Upgrade boot CD:

BIOS: C200.1.1.1a.0.032920100525

CIMC firmware: 1.2(1b)

image

Once I saw the mismatch in firmware version, I went ahead and updated the firmware to:

CIMC Firmware: 1.2(2d)

BIOS: C200.1.2.2f.0

LOM: Not sure but it was updated

image

image

image

I’m not going into the details of how to update the firmware as I am supposed to write another post about it but just haven’t gone around to doing it.  Once I completed the firmware update, rebooted the server and had it boot back into ESXi 4.1, the NIC speed started to show up properly:

image

Before I end and publish this post, I’d like to reiterate that I highly doubt the problem was because of an old firmware and is most likely something else but if anyone ever runs into a problem similar to this, try updating the firmware (which is probably what Cisco Support would have told you if you had called them).

3 comments:

Jimi said...

I got this from Ciscos support:

EXT-INTEL-CNTLR
Symptom:
esxcfg-nics -l on VMware shows that LOM ports (82576) negotiated only 10 mbps on C-Series.
Workaround
Follow these steps:
----

Step 1:
If running ESX, collect the output from the following commands for Cisco TAC:
esxcfg-nics -l
ethtool -i vmnic0
ethtool -i vmnic1
Then verify that the ports are in auto/auto for speed and duplex by running the following commands:
esxcfg-nics -a vmnic0
esxcfg-nics -a vmnic1

Step 2:
Toggle the NIC mode between Shared LOM and Dedicated and back to the desired mode in CIMC, while saving the changes in between. If the interface does not come up in the 1 Gbps mode, try power cycling the server, leaving the entire device disconnected from power for 3 minutes and then power on. If the NIC still does not negotiate at 1 Gbps, try step 3.

Step 3:
Make sure the LOM firmware is at version 1.4-3, if the ethtool -i output shows the firmware-version is not 1.4-3, then try step 2 again. (CSCtj80604)

---

And I have tried it on two systems.
The detail about leaving it powered off for 3min is a important detail.

Terence Luk said...

Thanks for sharing the information you got from Cisco Jimi!

Anonymous said...

Jimi's information worked on me as well. Ihad the same issue on two Cisco USC 210 M1 servers.
Thank you all guys for sharing.

Stelios