hba / vmhba not showing up with ESXi 4.0 / 4.1

I ran into various problems while deploying a new server in an ESX cluster and I hope this post will help anyone who might run into the same problem.

Specifications:

Server: HP ProLiant DL360 G6

BIOS: ProLiant System BIOS – P64 (03/30/2010)

CPU: 2 x Intel CPU X5550 @ 2.67GHz

QPI: Speed: 6.4 GT/s

Onboard NIC: PCI Embedded HP NC382i PCIe DP Multifunction 1GB Adapter Port 1, 2 (IRQ 7, 11)

HBA (Host bus adapter): HP StorageWorks 42B PCI Fibre Channel Adapter IRQ 7, 11

Additional NIC: NC375T PCIe Quad Port Gigabit Server Adapter (IRQ: 7)

------------------------------------------------------------------------------------------------------------------------------------------

Other than having incorrect memory ordered and some other issues, I was unable to have ESXi 4.0.0 see the HP StorageWorks 42B HBA.

As shown in the screenshot below, I was unable to have ESXi 4.0.0 build 261974 see the vmhba after a fresh install.

After doing a few searches on Google and finding various posts about people having problems with installing 2 of the quad port cards but managed to get it going after upgrading the drivers or ESXi, I decided to try upgrading ESXi from 4.0.0 to 4.1.0 (http://terenceluk.blogspot.com/2010/09/updating-vsphere-esxi-from-40-to-41.html) assuming that the newer ESXi would may have the right drivers for the HBA.

The upgrade went without a hitch but I was still unable to see the HBA after getting ESXi to version 4.1.0 so I went back to the internet to do some searches and found that there were ways to install additional drivers for ESXi and coincidentally, you use the VMware vSphere CLI to do it (see my previous post about why this was coincidental). The next step I did was to try and find the HBA drivers. Through reviewing the description the BOM, I find the description:

StorageWorks 42B - Host bus adapter - PCI Express low profile - 4Gb Fibre Channel (SW) - fiber optic - 2 ports

… which lead me to the following page on the HP site @:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareIndex.jsp?lang=en&cc=us&prodNameId=3954646&prodTypeId=12169&prodSeriesId=1809835&swLang=13&taskId=135&swEnvOID=4040

The driver I thought would work was the:

VMware and Host Connectivity Manager (HCM) Installation Kit

…so I went ahead and downloaded this package:

brocade_driver_esx40_v2-1-1-0.tar.gz

…from: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=12169&prodSeriesId=1809835&prodNameId=3954646&swEnvOID=4040&swLang=13&mode=2&taskId=135&swItem=co-84517-1

I went ahead to try and execute the package with the command:

C:\Program Files\VMware\VMware vSphere CLI\bin>vihostupdate.pl -server 172.20.70.13 -username root -install -bundle c:\brocade_driver_esx40_v2-1-1-0.tar.gz

…but got the following message:

Please wait patch installation is in progress …

Failed to download metadata.Error extracting metadata.zip from /tmp/updatecache/brocade_driver_esx40_v2-1-1-0.tar.gz: File is not a zip

Seeing how it doesn’t look like the update is expecting a tar.gz file, I tried to extra the .tar.gz file into a folder, then zipping it back up to try again:

Unfortunately, that didn’t work either:

Please wait patch installation is in progress …

Failed to download metadata.Error extracting metadata.zip from /tmp/updatecache/brocade_driver_esx40_v2-1-1-0.zip: “There is no item named ‘metadata.zip’ in the archive”

This was when I went back to the package I downloaded for the ESXi 4.0.0 update to 4.1.0 and found that the package indeed had a metadata.zip package:

Browsing around the package I downloaded from the HP site and not finding a metadata.zip file made me suspect that I probably downloaded some other package that was not meant for ESXi updates.

Realizing that perhaps I should first try to ensure that this HBA was on the HCL list, I went ahead to try and find it on VMware’s HCL:

http://www.vmware.com/resources/compatibility/search_results_ajax.php?action=search&deviceCategory=io&&&&&&&&&&&&partnerId%5B%5D=41&&&&sort=manufacturer~desc&&&ioTypeId=5&&startDisplayRow=50

http://www.vmware.com/resources/compatibility/detail.php?device_cat=io&device_id=16075&release_id=24

http://www.vmware.com/resources/compatibility/detail.php?device_cat=io&device_id=16075&release_id=24

Based on what I found on the HCL, it does look like it’s supported so I went back to the VMware downloads section to try and find drivers offered directly from that site. What I found was the following:

http://downloads.vmware.com/d/info/datacenter_downloads/vmware_vsphere_4/4#drivers_tools

I was a bit confused as to which one to download because I was not able to find a 4Gb Fibre Channel HBA option and the only option close to it was the:

VMware ESX/ESXi 4.X Driver CD for Brocade 8Gb Fibre Channel HBA

Since I wouldn’t know if I don’t try, I went ahead to download that package:

http://downloads.vmware.com/d/details/esx_4x_brocade_bfa2111_dt/ZHcqYmRAZCpiZHdlZQ==

Download
VMware ESX/ESXi 4.X Driver CD for Brocade 8Gb Fibre Channel HBA

Description
This driver CD release includes support for version 2.1.1.1 of the Brocade BFA driver on ESX/ESXi 4.0. This BFA driver supports products based on the Brocade 825, 815, 425, and 415 Fibre Channel host bus adapters (HBA).

Version
2.1.1.1

Build Number
285864

Release Date
2010/08/23

Type
Drivers & Tools

Language Support
English

Components
This download contains the following components. Hide Details

ESX/ESXi 4.X Brocade bfa 2.1.1.1 Driver
File type: iso
English

The download gave me the following ISO package:

vmware-esx-drivers-scsi-bfa_400.2.1.1.1-1OEM.285864.iso

I opened the package with WinRAR and found the following:

..

.rpm

doc

offline-bundle

drivers.xml

TRANS.TBL

Looking into the offline-bundle folder, I found the following package:

BRCD-bfa-2.1.1.1-00000-offline_bundle-285864.zip

Drilling into that .zip package showed that it contains the following:

metadata.zip

vmware-esx-drivers-scsi-bfa-400.2.1.1.1-1OEM.x86_64.vib

This was when I was sure that this package was the one I wanted so i went ahead and extracted the BRCD-bfa-2.1.1.1-00000-offline_bundle-285864.zip package then went back to vSphere CLI to try and update the drivers with it:

C:\Program Files\VMware\VMware vSphere CLI\bin>vihostupdate.pl -server 172.20.70

.13 -username root -install -bundle c:\vmware-esx-drivers-scsi-bfa_400.2.1.1.1-1

OEM.285864\offline-bundle\BRCD-bfa-2.1.1.1-00000-offline_bundle-285864.zip

Enter password:

Please wait patch installation is in progress ...

The update completed successfully, but the system needs to be rebooted for the c

hanges to be effective.

C:\Program Files\VMware\VMware vSphere CLI\bin>

Success!

Once the update completed, I went and fired up VI Client, connected to the host, navigated to the storage adapters section and now I can see the vmhba listed!

Now that I got the HBA to show up on the host, I knew I needed to do one last step and that was to ensure that I did use the right driver because once this goes into production, we won’t be able to do any more testing with it.

Within the storage adapters section, ESXi lists the adapters as Brocade-425/825:

… so I went ahead to look up the reference guide on the Brocade site to determine if there was a cross reference guide available and indeed there was:

http://www.brocade.com/downloads/documents/reference_guides/Brocade_HBA_Cross_Reference_01.pdf

As indicated on the guide in the above screenshot, the Brocade 425 actually corresponds to the StorageWorks 42B HP model.

--------------------------------------------------------------------------------------------------------------------------------------------------

I’m surprised that the ESXi 4.1.0 package did not seem to load this driver during the install so I hope this helps anyone out there that may run into this or a similar problem in the future.

Updating vSphere ESXi from 4.0 to 4.1 with vSphere CLI

We’re currently in the process of refreshing a client’s VI3 environment to vSphere 4.1 and procured a new server to add to the existing cluster. While performing the ESXi install on the new server, I did not have a ESXi 4.1 CD available so I went ahead to install 4.0 and figure I’d update it with the vSphere Host Update Utility. For those who have read the vSphere 4.1 release probably already know that you cannot update the host from 4.0 to 4.1 with that utility so this post serves as to show how you can update the host with the vSphere CLI.

I started off with using the VMware vSphere Host Update Utility I had installed on my laptop to try and update the ESXi 4.0.0 build-261974.

As shown in the following screenshot, scanning a fully patched ESXi 4.0.0 won’t give you an option to upgrade the host to version 4.1.0.

As per the following release notes:

http://www.vmware.com/support/vsphere4/doc/vsp_esxi41_vc41_rel_notes.html

ESXi Upgrades

vSphere 4.1 offers the following tools for upgrading ESXi hosts:

VMware vCenter Update Manager. vSphere module that supports direct upgrades from ESXi 3.5 and ESXi 4.0 to ESXi 4.1. See the vCenter Update Manager Installation and Administration Guide.

vihostupdate. Command-line utility that supports direct upgrades from ESXi 4.0 to ESXi 4.1. This utility requires the vSphere CLI. See the vSphere Upgrade Guide.

Reviewing the upgrade guide at: http://www.vmware.com/pdf/vsphere4/r41/vsp_41_upgrade_guide.pdf shows that we need to download and install the vSphere CLI client.

So I went ahead to download the client from http://www.vmware.com/downloads:

(The build I downloaded for this upgrade was: VMware-vSphere-CLI-4.1.0-254719.exe)

… and began installing it:

The following screen too extremely long to finish and I remember not having this issue on my last deployment when I installed it on a server so my guess is that I had some other application on my laptop that caused the delay.

Once I completed the installation, I went ahead to download the upgrade package. Make sure you download the proper upgrade package in a ZIP package and not the regular installable ISO as the latter will not allow you to use vSphere CLI to upgrade the host.

While downloading the package, we can spend the time we need to wait to put the host into maintenance mode:

Once you’ve downloaded the zip package, DO NOT uncompress it. Simply place it into a directory of your choice and then open up the VMware vSphere CLI.

C:\Program Files\VMware\VMware vSphere CLI>vihostupdate

'vihostupdate' is not recognized as an internal or external command,

operable program or batch file.

C:\Program Files\VMware\VMware vSphere CLI>dir

Volume in drive C has no label.

Volume Serial Number is 4802-7E84

Directory of C:\Program Files\VMware\VMware vSphere CLI

09/23/2010 06:50 AM <DIR> .

09/23/2010 06:50 AM <DIR> ..

09/23/2010 06:50 AM <DIR> bin

09/23/2010 07:03 AM <DIR> Perl

09/23/2010 06:49 AM <DIR> PPM

0 File(s) 0 bytes

5 Dir(s) 7,619,219,456 bytes free

C:\Program Files\VMware\VMware vSphere CLI>cd bin

C:\Program Files\VMware\VMware vSphere CLI\bin>

As shown in the above screenshot, the vihostupdate.pl script is actually in the C:\program files\VMware\VMware vSphere CLI\bin directory.

In the screenshot above, I actually made 2 mistakes, the first one being running vihostupdate without the .pl extension.

The 2nd mistake is shown in the screenshot below:

I originally unzipped the package because I thought executing the vihostupdate.pl was supposed to be done on a directory when in fact it actually expects a zip package. The following is the output and I’ve also highlighted the error if you were to specify a directory:

C:\Program Files\VMware\VMware vSphere CLI>vihostupdate

'vihostupdate' is not recognized as an internal or external command,

operable program or batch file.

C:\Program Files\VMware\VMware vSphere CLI>dir

Volume in drive C has no label.

Volume Serial Number is 4802-7E84

Directory of C:\Program Files\VMware\VMware vSphere CLI

09/23/2010 06:50 AM <DIR> .

09/23/2010 06:50 AM <DIR> ..

09/23/2010 06:50 AM <DIR> bin

09/23/2010 07:03 AM <DIR> Perl

09/23/2010 06:49 AM <DIR> PPM

0 File(s) 0 bytes

5 Dir(s) 7,619,219,456 bytes free

C:\Program Files\VMware\VMware vSphere CLI>cd bin

C:\Program Files\VMware\VMware vSphere CLI\bin>vihostupdate -server 172.20.70.13

-i -b c:\upgrade-from-ESXi4.0-to-4.1.0-0.0.260247-release\

'vihostupdate' is not recognized as an internal or external command,

operable program or batch file.

C:\Program Files\VMware\VMware vSphere CLI\bin>vihostupdate.pl -server 172.20.70

.13 -i -b c:\upgrade-from-ESXi4.0-to-4.1.0-0.0.260247-release\

Enter username: root

Enter password:

Please wait patch installation is in progress ...

Invalid bundle ZIP archive, or missing metadata.zip inside.Bundle.zip [/tmp/updatecache/upgrade-from-ESXi4.0-to-4.1.0-0.0.260247-release]: File /tmp/updatecache/upgrade-from-ESXi4.0-to-4.1.0-0.0.260247-release is too small to be a .zip file

C:\Program Files\VMware\VMware vSphere CLI\bin>vihostupdate.pl -server 172.20.70

.13 -i -b c:\upgrade-from-ESXi4.0-to-4.1.0-0.0.260247-release.zip

Once I specified the zip package instead, the update proceeds and completes properly:

C:\Program Files\VMware\VMware vSphere CLI\bin>vihostupdate.pl -server 172.20.70

.13 -i -b c:\upgrade-from-ESXi4.0-to-4.1.0-0.0.260247-release.zip

Enter username: root

Enter password:

Please wait patch installation is in progress ...

The update completed successfully, but the system needs to be rebooted for the changes to be effective.

C:\Program Files\VMware\VMware vSphere CLI\bin>

Reviewing what is displayed in the vSphere Client, we can see that the version is still 4.0.0 so all we need to do now is to reboot the server.

Note: Notice the Recent tasks below? Those are from the host update utility’s scans.

Once the host is successfully updated, you’ll see the correct version:

----------------------------------------------------------------------------------------------------------------------------------

Thoughts: Coming from a Windows background, I personally don’t like to do upgrades and I was told by my colleague that our practice lead recommends simply reinstalling ESXi on the host. The problem I have with that is that you lose all your settings so if you have a lot of hosts, this option might be a better route to take.

I hope this has been beneficial to the other professionals out there and possibly even save them some time.

VMware Virtual Infrastructure 3 (VI3) - Adding a Raw Device Mapping hard disk grayed out

We still have a client who are still on VI3 and I had a maintenance window today to reconfigure their virtualized SQL cluster and while executing my plan, I ran into an issue with mapping a newly provisioned LUN as an RDM. While this isn’t much of a difficult problem to solve, I’d like to blog it in case someone happens to Google this problem for a quick answer.

So here I am going through the regular tasks of attaching a new disk to an existing virtual machine.

…and I get to the screen with:

A virtual disk is composed of one or more files on the host file system. Together these files appear as a single hard disk to the guest operating system. Select the type of disk to use from the choices below.

Disk

Create a new virtual disk

Choose this option to create a new virtual disk.

Use an existing virtual disk

Choose this option to reuse a previously configured virtual disk.

Raw Device Mappings

Give your virtual machine direct access to the SAN. This option allows you to use existing SAN commands to manage the storage and continue to access it using a datastore.

The problem I had was that the option is grayed out. Fortunately, I sort of knew what might be the problem so I navigated to the Configuration tab, Storage Adapters and found that the newly provisioned LUN hasn’t shown up yet:

What I ended up doing was do a quick Rescan (top right hand corner in the screenshot above) of on the HBAs, got the LUN to show up and then went back to adding the disk. From there on, the option was available.

I hope this post will end up saving someone’s troubleshooting time.

Backup Exec 2010 fails with “e000848c - Unable to attach to a resource” and other errors

While configuring backups for a customer with Backup Exec 2010 to backup a vSphere 4 environment, we ran into multiple errors such as the following:

Final error: 0xe000848c - Unable to attach to a resource. Make sure that all selected resources exist and are online, and then try again. If the server or resource no longer exists, remove it from the selection list. Edit the selection list properties, click the View Selection Details tab, and then remove the resource.
Final error category: Resource Errors

For additional information regarding this error refer to link V-79-57344-33932

V-79-57344-38277 – Unable to open a disk of the virtual machine

VixDiskLib_Open() reported the error:
V-79-57344-38277 - Unable to open a disk of the virtual machine.

VixDiskLib_Open() reported the error:

Problem Resolution

After troubleshooting the problem for an hour and being pressed for time, we ended up calling Symantec since we had a support contract. What ended up being the problem was because we were not using the R2 version. Here’s the version we were initially using:

Symantec

Backup Exec 2010

Media Server: Version 13.0 Rev. 2896 (64-bit)

Administration Console: Version 13.0 Rev. 2896 (64-bit)

Desktop and Laptop Option: Version 3.1 Rev. 3.42.44a

Once we upgraded to the R2 version, the backups began to work:

Symantec

Backup Exec 2010 R2

Media Server: Version 13.0 Rev. 4164 (64-bit)

Administration Console: Version 13.0 Rev. 4164 (64-bit)

Desktop and Laptop Option: Version 3.1 Rev. 3.43.17a

Here are some additional copy and paste of the error logs:

Job name : test

Job type : Backup

Job status : Failed

Job log : C:\Program Files\Symantec\Backup Exec\Data\BEX_WCITORBTBEXEC_00155.xml

Server name : BEXEC

Selection list name : test-1
Device name : Test

Target name : Test

Media set name : Daily Full Backups
Error category : Resource Errors

Error : e000848c - Unable to attach to a resource. Make sure that all selected resources exist and are online, and then try again. If the server or resource no longer exists, remove it from the selection list. Edit the selection list properties, click the View Selection D

For additional information regarding this error refer to link V-79-57344-33932

-------------------------------------------------------------------------------------------------------------------------------------------------------------------

Set type : Backup

Set status : Completed

Set description : test
Resource name : \\VCVM\VMVCB::\\VCVM\VCGuestVm\vm\WFPS

Logon account : System Logon Account

Encryption used : None
Error : e0009585 - Unable to open a disk of the virtual machine.
Agent used : Yes

Advanced Open File Option used : No

-------------------------------------------------------------------------------------------------------------------------------------------------------------------

Job ended: September-15-10 at 3:47:55 PM
Completed status: Failed
Final error: 0xe000848c - Unable to attach to a resource. Make sure that all selected resources exist and are online, and then try again. If the server or resource no longer exists, remove it from the selection list. Edit the selection list properties, click the View Selection Details tab, and then remove the resource.
Final error category: Resource Errors

For additional information regarding this error refer to link V-79-57344-33932

-------------------------------------------------------------------------------------------------------------------------------------------------------------------

Click an error below to locate it in the job log

Backup- VMVCB::\\VCVM\VCGuestVm\vm\FPS

V-79-57344-38277 - Unable to open a disk of the virtual machine.

VixDiskLib_Open() reported the error:
V-79-57344-38277 - Unable to open a disk of the virtual machine.

VixDiskLib_Open() reported the error:

Problem powering on a virtual machine - “Cannot open the disk ‘/vmfs/volumes/unique identifier/virtualmachine/file.vmdk’

While this error can be easily solved by reviewing the events in VI Client, I figure I should blog it anyways in case anyone was searching for this string and had ran out of ideas.

I was doing some maintenance on a hosted environment a few days ago where we had to add new NetApp shelf to the existing FAS that hosted storage for a 3 node ESX cluster with 20 virtual machines currently used to host a medical tracking application. The setup was nothing fancy and small enough to be easily built with the NetApp providing SAN storage with SAS drives and an iSCSI SAN with SATA drives for backups.

The work done in the maintenance window was mainly ESX and storage so we only had my storage colleague and myself. The problem I ran into early was that since the environment was so locked down, I was unable to get to the ESX hosts directly with VI Client when I shutdown Virtual Center (It’s VI3). I ended up hooking my laptop to the iSCSI storage’s port to get to the hosts directly.

Once we finished provisioning the new storage and started booting back up the VMs, I received an error on the 2 virtual machines configured with MSCS and SQL clustering. The VMs had RDMs and disks located on the iSCSI vmfs store for backups. The first error I saw was:

Failed to power on virtualMachine: A general system error occurred:

Definitely not helpful at all. The next message was much better:

Message on virtualMachine: Cannot open the disk ‘/vmfs/volumes/unique identifier/virtualmachine/file.vmdk

At the bottom of the Events entries in the Events Details window, we can see the following message:

Message on virtualMachine on esxHost in datacenterName: Cannot open the disk ‘/vmfs/volumes/unique identifier/virtualmachine/file.vmdk or one of….

Reason: Device or resource busy.

Once I saw the 2nd error I knew immediately why the VM wasn’t powering on (it has lost access to some of the disks) so I replaced the network cable from the iSCSI storage controller back to the port that it was on. From there on, the server booted back up.

Windows Server 2008 R2 64-bit cluster verification error with VMware virtual machines

Ran into a problem a few months ago while building a MSCS (Microsoft Clustering Services) while verifying the cluster nodes when I would continue to get the following message:

An error occurred while executing the test. There was an error verifying the firewall configuration. An item with the same key has already been added.

After searching for awhile for the answer, I found that this was because the NICs on the virtual machines had the same GUIDs and MSCS doesn’t like that. This generally wouldn’t be a problem if the servers were physical because the GUIDs would be different but this customer opted to deploy the cluster in their vSphere environment and since the servers were deployed from templates, the GUIDs ended up being the same. One of the blog posts I found indicated that I can remove the NIC from within Windows but when I tried doing so through device manager, the GUID ended up being the same. What I ended up doing to resolve the problem was:

1. Uninstall the NIC from Device Manager within Windows.
2. Shut off the virtual machine.
3. Removed the NIC by editing the virtual machine’s settings from vCenter, click OK.
4. Edit the virtual machine’s settings and add a new NIC, click OK.

Once Windows completed the startup, I went into the registry and can now see the new NIC haven’t a different GUID.