Monday, February 28, 2011

Problem with initial Cisco UCS 6100 series fabric interconnect setup with error message: “Failed to configure given password on the switch, system will be rebooted , please wait… !”

The following was probably one of the most annoying errors I’ve ever come across while trying update the firmware on a Cisco UCS 6120XP fabric interconnect.  What basically happened was that the fabric interconnects we got were at firmware version 1.3 and because we were using the new B230 blades, we wanted to get it updated to 1.4 before we began building the cluster.  The plan was to temporarily set up each fabric interconnect as a standalone, update the firmware, then erase the configuration and set up the cluster.  I managed to get the firmware for the fabric interconnect updated to 1.4(1j) but had some issues with the UCS Manager.  To make a long story short, the firmware was at:

UCS Manager: 1.3(1c)

Fabric Interconnect: 1.4(1j)


… the configuration was blown away and setup was reran and the following error message will be displayed:

Failed to configure given password on the switch, system will be rebooted , please wait… !


This problem was extremely annoying because the 6120XP will essentially reboot automatically and you will be asked to run the initial setup again.  Each reboot takes upwards to 10 minutes so every reboot wastes a lot of time.

I went through the exercise of trying to run initial setup 3 times and then thought I could try doing a full state restore but then ran into the following error:

Error: End point timed out. Please retry


The TFTP server was off my laptop and I was able to successfully perform a full state backup the day before but since this workaround wasn’t working and my colleagues were waiting for me to leave the datacenter (it was a Sunday and we’ve been working for the past 2 weekends), I rebooted the fabric interconnect and thought I’ll write up a list of “things to try” at night and just tackle it the next day.

As I thought about the problem at night, I realized that the only other configuration parameter I could change during the initial setup that I haven’t tired was the Enforce strong password setting during the initial setup.  The setting I choose for this configuration was NO so I could just use the password: “password” and I repeated this over the 3 failed setups I ran.


Fast forward to the next day and as stupid as this sounds, when I choose YES for the Enforce strong password option, the setup completed and I was able to successfully configure the fabric interconnect as a standalone setup.

I’m not sure what might have caused this but I do suspect that it may be a mismatch between the UCS Manager (1.3) and the Fabric Interconnect (1.4).  I didn’t have time to confirm this so I hope that the information I’ve provided here may help anyone out there that comes across the exact same problem.


ucscrazy said...

Hi Terence,
I did this back when you did it going from 1.31c to 1.41i for the same reason. Why in heaven's name did you make the FINs standalone rather than leaving them in a cluster (not criticizing so much as curious)? Also interested in what you ran into when doing the upgrade. I've since gone to 1.42b and am now at 1.43l

Terence Luk said...

The reason why I chose standalone was because I had problems setting up the cluster and during the troubleshooting process, I decided to just configure the first node as a standalone then add the secondary FI to build a cluster. You could do it either way but I just wanted to eliminate any factors that may be causing my problems.