Updating Cisco UCS B Series infrastructure firmware from 1.4(1i) to 1.4(1j)

It’s been awhile since I’ve written a post about updating Cisco UCS B Series Infrastructure firmware and while preparing to upgrade the firmware for a client I’m working with, I noticed that the last post I wrote was:

Updating Cisco UCS firmware from 1.3(1c) to 1.3(1n)

http://terenceluk.blogspot.com/2010/10/updating-cisco-ucs-firmware-from-131c.html 

Seeing how it’s been awhile and I don’t have a post for 1.4, I figure I should take the opportunity to do some screenshots to write this.

Downloading UCS B Series Firmware

Keep in mind that the way in which Cisco bundles the UCS B Series firmware is has changed and for more information about this, see one of my previous post: http://terenceluk.blogspot.com/2011/01/missing-14-firmware-option-when.html.  Navigate to the Cisco software download page and navigated to:

Products –> Unified Computing –> Cisco UCS Infrastructure Software

Once you see the folders with the various versions, choose the one you would like to download (in this example, we’re using 1.4(1j):

Since the components required to update the UCS B Series Infrastructure is split into 2 packages, continue by selecting Select Another Product:

Navigate on the Cisco software download page to:

Products –> Unified Computing –> Cisco UCS Manager Server Software

Select UCS B-Series Blade Server Software:

Once you see the folders with the various versions, choose the one you would like to download (in this example, we’re using 1.4(1j):

Proceed with downloading the 2 packages queued:

1. UCS B-Series Blade Server Software
2. Cisco UCS Infrastructure Software

Uploading UCS B Series Firmware

Once you have the packages downloaded, check to ensure that your fabric interconnects have sufficient space:

Once you’ve confirmed you have enough space on the bootflash, proceed with uploading the firmware packages to the 6100 series Fabric Interconnects:

 

 

UCS B Series Firmware Update Guide

Unfortunately, there isn’t a upgrade guide available for 1.4 to 1.4 upgrades so I will simply use 1.3 to 1.4 as a reference:  http://www.cisco.com/en/US/products/ps10281/prod_installation_guides_list.html 

 

 

As shown in the update guide, the following will be the update order for the devices in the UCS B Series Infrastructure:

1. Adapter (interface card)
2. CIMC
3. I/O module
4. Cisco UCS Manager
5. Fabric Interconnect
6. Host firmware package

 

In case you’re interested in comparing the ordering provided by the 1.2 to 1.3 guide, it’s actually the same other than the reference to CIMC as BMC:

1. Adapter (interface card)
2. BMC
3. I/O module
4. Cisco UCS Manager
5. Fabric Interconnect
6. Host firmware package

 

Also note that if we were updating the firmware from 1.3 to 1.4, this snippet would be important:

However, because this example is from 1.4(1i) to 1.4(1j), we can safely ignore this.

Updating UCS B Series Firmware

Once you’ve completed the upload, you will need to update the UCS B Series Infrastructure components from firmware version that is currently being used to the new version that was uploaded.  Proceed by navigating to the Equipment tab –> Equipment node –> Firmware Management tab –> Installed Firmware then click on the Update Firmware button:

 

In the Update Firmware window, set the Filter to All and set the Set Version to the version you would like to update to (in this example, it will be 1.4(1j):

 

 

Pay close attention to the Backup Version of each component and you will see the new version listed:

Click on the Apply button and you will see the Update Status column as updating:

 

If you’re interested in seeing the update status of a component, simply navigate to that component under the Equipment tab, click on the General tab and scroll to the section Update Status:

As seen in the following screenshot, the FSM tab isn’t going to tell you much:

Navigate back to the Firmware Management tab –> Installed Firmware and wait for all of the components to be listed as ready for the Update Status column:

Once all of the components are listed as ready for the Update Status column, proceed with clicking on the Activate Firmware button:

Activate Interface Card Firmware

Within the Activate Firmware window, set:

Filter: Interface Cards

Set Version: 1.4(1j)

Ignore Compatibility Check: Checked

Set Startup Version Only: Checked

I find that I get asked a lot about why we’re supposed to check the Ignore Compatibility Check checkbox so I’ve included the explanation from the upgrade guide:

Check the Ignore Compatibility Check check box.
The firmware for this release is not compatible with previous releases. Therefore, you must check the
Ignore Compatibility Check check box to ensure that the activation succeeds.

Once you proceed and click on the OK button, the interface cards’ firmware activation will begin:

Once the activation completes, the Activate Status will be listed as: pending-next-reboot:

Activate CIMC Firmware

Once the firmware for the interface cards have been activated, we can proceed with the activation of the CIMC.  Within the Activate Firmware window, set:

Filter: CIMC

Set Version: 1.4(1j)

Ignore Compatibility Check: Checked

Set Startup Version Only: Checked

Once you proceed and click on the OK button, the CIMC’s firmware activation will begin and once it has completed, it will be rebooted.  Once the reboot completes, the Activate Status will be listed as: ready:

 

Activate IO Module Firmware

Once the firmware for the CIMC have been activated, we can proceed with the activation of the IO Module but before we continue, please be aware of the following snippet from the upgrade guide:

When you configure Set Startup Version Only for an I/O module, the I/O module is rebooted
when the fabric interconnect in its data path is rebooted. If you do not configure Set Startup
Version Only for an I/O module, the I/O module reboots and disrupts traffic. In addition, if
Cisco UCS Manager detects a protocol and firmware version mismatch between it and the
I/O module, Cisco UCS Manager automatically updates the I/O module with the firmware
version that matches its own and then activates the firmware and reboots the I/O module
again.

When ready to activate the I/O module firmware, click on the Active Firmware button:

Within the Activate Firmware window, set:

Filter: IO Module

Set Version: 1.4(1j)

Ignore Compatibility Check: Checked

Set Startup Version Only: Checked

Once you proceed and click on the OK button, the IO Module firmware activation will begin and once the activation completes, the Activate Status will be listed as: pending-next-reboot:

As noted in the snippet I included above, you can safely leave the IO Modules with the Activate Status as: pending-next-reboot.

Activate Board Controller Firmware

Once the firmware for the IO Modules have been activated, we can proceed with the activation of the Board Controller but before we continue, please be aware of the following snippet from the upgrade guide:

Activating the Board Controller Firmware on a Server to Release 1.4(1)
Only certain servers, such as the Cisco UCS B440 High Performance blade server and the Cisco UCS B230
blade server, have board controller firmware. The board controller firmware controls many of the server
functions, including eUSBs, LEDs, and I/O connectors.
This procedure continues from the previous one and assumes that you are on the Installed Firmware tab.
This activation procedure causes the server to reboot. Depending upon whether or not the service profile
associated with the server includes a maintenance policy, the reboot can occur immediately. To reduce
the number of times a server needs to be rebooted during the upgrade process, we recommend that you
upgrade the board controller firmware through the host firmware package in the service profile.

The snippet above pertains to the UCS Infrastructure used in this example because all of the servers in the 6 chassis are actually Cisco UCS B230 blades.

When ready to activate the Board Controller firmware, click on the Active Firmware button:

Within the Activate Firmware window, set:

Filter: Board Controller

Set Version: This option is grayed out

Ignore Compatibility Check: Checked

Now from the Startup Version column, select the version B230100A:

Once you proceed and click on the OK button, the Board Controller firmware activation will change from ready to activating:

… then ready:

Activate UCS Manager Firmware

Once the firmware for the Board Controllers have been activated, we can proceed with the activation of the UCS Manager so proceed with clicking on the Activate Firmware button:

Within the Activate Firmware window, set:

Filter: UCS Manager

Set Version: 1.4(1j)

Ignore Compatibility Check: Checked

Once you proceed with clicking the OK button, you’ll briefly get a warning message about ignoring the compatibility check.  Continue by clicking the Yes button to proceed:

 

You will be kicked off of UCS Manager soon after the activation begins:

Wait till you are able to hit the UCS Manager webpage:

Continue to log in:

Once you’ve successfully logged back into UCS Manager, navigate to the Installed Firmware tab to make sure UCS Manager has been updated:

Activate Clustered Fabric Interconnect Firmware

Activate Subordinate Fabric Interconnect:

As noted in the upgrade guide:

… it is important to perform the firmware activation on the subordinate fabric interconnect first before you activate the active to avoid unplanned disruption.

Also, prior to activating the firmware on the passive fabric interconnect, it is important that you verify that the cluster’s High Availability Details state is:

Ready: Yes

State: Up

Cluster Link State: Full

Once you’ve validated that the cluster is in good health, proceed with the activation of the subordinate fabric interconnect by clicking on the Activate Firmware button:

Within the Activate Firmware window, set:

Filter: Fabric Interconnects

Set Version: <do not set>

Ignore Compatibility Check: Checked

Now from the Startup Version column of the subordinate fabric interconnect, select the version you’re updating to for the Kernel and System:

Once you proceed with clicking the OK button, you’ll get a warning message about activating the fabric interconnects will cause them to reboot.  Continue by clicking the Yes button to proceed:

After you’ve answered Yes to the warning message, you will see the Active Status status of the subordinate fabric interconnect listed as activating:

You’ll notice that during the initial start of the activation, the cluster state of your primary and subordinate fabric interconnects will still be listed as being in good health:

If you would like to review the status of the update, you can navigate to the subordinate fabric interconnect’s General tab and scroll down to the Update Status section:

As the activation proceeds, you’ll see that the status of the subordinate fabric interconnect turn to red:

The High Availability Details for the subordinate will also change to:

Ready: No

State: Down

Failure Reason: Node down

Leadership: Inapplicable

Cluster Link Stat: Full

The High Availability Details for the primary will also change to:

Ready: No

State: Up

Failure Reason: Peer node down

Leadership: Primary

Cluster Link Stat: Full

Note that the activation of the new firmware on fabric interconnects take quite a bit of time so be patient:

You’ll see the fabric interconnect eventually complete a reboot and go through various states (i.e. election) with states such as:

Failure Reason: Chassis configuration incomplete

Leadership: Election in progress

… but remember that no action is required:

You’ll also notice that even when the Activate Status in the Installed Firmware tab lists the subordinate as being as ready, you may still a yellow bracket around the subordinate fabric:

Waiting for a bit longer will show it change to blue:

Then finally change to normal (no bracket) but the High Availability Details may still list:

Ready: Downgraded

State: Up

Failure Reason: Chassis configuration incomplete on peer node

Leadership: Primary

Cluster Link State: Full

A few more seconds and the yellow bracket may come back:

The bottom line is that the errors and faults will eventually clear:

Activate Primary Fabric Interconnect:

As with updating the subordinate, prior to activating the firmware on the active fabric interconnect, it is important that you verify that the cluster’s High Availability Details state is:

Ready: Yes

State: Up

Cluster Link State: Full

Once you’ve validated that the cluster is in good health, proceed with the activation of the active fabric interconnect by clicking on the Activate Firmware button:

Within the Activate Firmware window, set:

Filter: Fabric Interconnects

Set Version: <do not set>

Ignore Compatibility Check: Checked

Now from the Startup Version column of the primary fabric interconnect, select the version you’re updating to for the Kernel and System:

Once you proceed with clicking the OK button, you’ll get a warning message about activating the fabric interconnects will cause them to reboot.  Continue by clicking the Yes button to proceed:

After you’ve answered OK to the informational message, you will see the Active Status status of the primary fabric interconnect listed as activating:

You’ll notice that during the initial start of the activation, the cluster state of your primary and subordinate fabric interconnects will still be listed as being in good health:

 

If you would like to review the status of the update, you can navigate to the subordinate fabric interconnect’s General tab and scroll down to the Update Status section:

Once the activation of the firmware completes for the primary fabric interconnect, it will reboot and kick you out of UCS Manager:

As with the subordinate fabric interconnect, the state of the primary fabric interconnect will start with a red bracket indicating there are major faults then eventually cycle through the less critical faults and warnings:

You’ll also notice that the fabric interconnect is still going through the update process after logging back into UCS Manager:

The firmware activation will eventually complete and you’ll notice that your previously primary fabric interconnect is now the subordinate:

Completing Interface Card’s “Pending-next-boot” status

The last task is to reboot your servers so that the firmware activation for the interface cards will complete:

Note that even if you don’t have a service profile associated to the server and it’s powered off, you will still need to power cycle the server to complete the update process:

The server will go through the regular boot cycle:

… and when it completes, will have the interface card updated:

If you’re dealing with a new deployment and nothing is currently ran on all of your blades, you can actually navigate to the Chassis tab –> Blade Servers tab, then highlight all the servers, right click and choose Reset:

This will reset all of the servers to complete the update.

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

This concludes the update process for Cisco UCS B Series Infrastructure.