Problem powering on a virtual machine - “Cannot open the disk ‘/vmfs/volumes/unique identifier/virtualmachine/file.vmdk’

While this error can be easily solved by reviewing the events in VI Client, I figure I should blog it anyways in case anyone was searching for this string and had ran out of ideas.

I was doing some maintenance on a hosted environment a few days ago where we had to add new NetApp shelf to the existing FAS that hosted storage for a 3 node ESX cluster with 20 virtual machines currently used to host a medical tracking application. The setup was nothing fancy and small enough to be easily built with the NetApp providing SAN storage with SAS drives and an iSCSI SAN with SATA drives for backups.

The work done in the maintenance window was mainly ESX and storage so we only had my storage colleague and myself. The problem I ran into early was that since the environment was so locked down, I was unable to get to the ESX hosts directly with VI Client when I shutdown Virtual Center (It’s VI3). I ended up hooking my laptop to the iSCSI storage’s port to get to the hosts directly.

Once we finished provisioning the new storage and started booting back up the VMs, I received an error on the 2 virtual machines configured with MSCS and SQL clustering. The VMs had RDMs and disks located on the iSCSI vmfs store for backups. The first error I saw was:

Failed to power on virtualMachine: A general system error occurred:

Definitely not helpful at all. The next message was much better:

Message on virtualMachine: Cannot open the disk ‘/vmfs/volumes/unique identifier/virtualmachine/file.vmdk

At the bottom of the Events entries in the Events Details window, we can see the following message:

Message on virtualMachine on esxHost in datacenterName: Cannot open the disk ‘/vmfs/volumes/unique identifier/virtualmachine/file.vmdk or one of….

Reason: Device or resource busy.

Once I saw the 2nd error I knew immediately why the VM wasn’t powering on (it has lost access to some of the disks) so I replaced the network cable from the iSCSI storage controller back to the port that it was on. From there on, the server booted back up.