Attempting to upgrade ESXi on a Nutanix node manually fails with: "OSError: [Errno 39 Directory not empty: ‘/vmfs/volumes/e2be1277-f8844313-564e-dbe0fc3821c4/Nutanix’"

Problem

You’re attempting to manually upgrade a Nutanix node from VMware ESXi, 6.5.0, 17167537 to VMware ESXi, 6.7.0, 15160138 with the ISO VMware-VMvisor-Installer-201912001-15160138.x86_64.iso then patch it to VMware ESXi, 6.7.0, 17700523 with ESXi670-202103001.zip:

Upgrade ESXi, preserve VMFS datastore

… but the upgrade from 6.5 to 6.7 would fail with the following error after you select the upgrade option:

------ An unexpected error occurred ------

See logs for details

OSError: [Errno 39 Directory not empty: ‘/vmfs/volumes/e2be1277-f8844313-564e-dbe0fc3821c4/Nutanix’

The upgrade does not proceed further until you restart the host, which will boot into ESXi 6.5 normally.

Subsequent attempts to re-run the upgrade will now present the option of installing a fresh ESXi installation and not upgrade.

Install ESXi, preserve VMFS datastore

Solution

I could not find any Nutanix KB but I did find quite a few forum and blog posts indicating that the files in /bootbank need to be copied to /altbootbank (copy not move) so I performed the following with WinSCP:

1. Backed up single file in /altbootbank named boot.cfg.backup and then deleted it
2. Backed up the Nutanix folder in bootbank folder and then deleted it
3. Copied the files in /bootbank to /altbootbank

Note that removing the Nutanix folder is important as failure to do so will not allow you to upgrade.

bootbank directory:

altbookbank directory:

I then rebooted the ESXi host with the 6.7 ISO mounted and was able to proceed with the upgrade and proceed to patch 6.7.

To be safe, I performed the following after the upgrade:

1. Copy the file named boot.cfg.backup back into in /altbootbank
2. Copy the the Nutanix folder back into the bootbank folder

Proceeded to upgrade

esxcli software vib install -d /vmfs/volumes/ntnx-lun03/ISO/ESXi670-202103001.zip

Hope this helps anyone who may encounter this issue.

VMware vSphere 6.5 CPU per socket calculation

I’ve recently had a developer ask me why the socket calculation in VMware vSphere 6.5 is seemingly so different than the legacy 5.5 environment he was used to and I found it difficult to verbally walk him through the changes so I went ahead and quickly wrote up an explanation, which I thought would be useful to blog in case I need to explain it again in the future.

First off, the following VMware blog does a great job with explaining this but I try not to respond to people with a link as they may feel I haven’t put much effort into an answer.

Virtual Machine vCPU and vNUMA Rightsizing – Rules of Thumb
https://blogs.vmware.com/performance/2017/03/virtual-machine-vcpu-and-vnuma-rightsizing-rules-of-thumb.html

Here is the write up I sent:

The following are the two parameters we are presented with when attempting to allocate processors and cores to the VM:

CPU

Cores per Socket

The CPU value is used to determine the total amount of cores that we want to present to the VM.

The Cores per Socket is a setting we use to determine the amount of physical sockets that VM would have.  By this, I mean that if we wanted the VM to have, say, 10 cores of processing power, we can use either of the three settings:

Option #1

CPU: 10

Cores per Socket: 1

The above would yield a socket configuration of 10:

Option #2

CPU: 10

Cores per Socket: 5

The above would yield a socket configuration of 2:

Option #3

CPU: 10

Cores per Socket: 2

The above would yield a socket configuration of 5:

The Cores per Socket actually has no bearing on the amount of usable cores available to the VMs as it is used to determine how many sockets are presented to yield the amount of cores available.

With the above in mind, it is not possible to have a Cores per Socket value that is higher than the CPU value because that would equate to a total amount of cores that is higher than the CPU value or the last socket would need to have zero cores for math to work.  The following are options help demonstrate this:

Note how the maximum value allowed for Cores per Socket is always equal to or less than value of the CPU.

If I took the last example of having 16 for the value of CPU and configuring 8 cores per socket then that configuration would consist of 2 physical sockets with 8 cores each totaling 16 CPUs:

You’ve probably already noticed but the math is basically CPU divide by Cores Per Socket.

This has been a significant change to what was previously used to in version 5.5 (the above is 6.5) where the calculation makes more sense to many who have used that environment:

Notice how the way the older version calculates the cores is simply Number of virtual sockets multiple by the Number of cores per socket.

Patching a VMware ESXi 5.5 host offline via CLI

One of the more common questions I’m asked every year is how to patch an ESXi host without using VMware Update Manager because it is quite common for ESXi hosts as well as vCenter to not have internet access. What I typically do when asked is point them to one of my previous posts where I demonstrated the process for a Nutanix infrastructure:

Upgrading Nutanix blades from ESXi 5.1 to 5.5
http://terenceluk.blogspot.com/2014/12/upgrading-nutanix-blades-from-esxi-51.html

… because the post includes the command for non-Nutanix deployments at the end but since I’ve received follow up questions about the process, I thought it would be a good idea to write a post that specifically demonstrates this for any ESXi deployments.

Step #1 – Identify and download desired ESXi build level

Begin by identifying the which build you’d like to patch the ESXi to:

https://kb.vmware.com/s/article/2143832

With the build number identified, proceed to the following URL to locate and download the patch:

https://my.vmware.com/group/vmware/patch#search

Download the patch:

The package you download should be a ZIP file containing files similar to the following screenshot:

vib20

index.xml

metadata.zip

vendor-index.xml

Step #2 – Upload patch to datastore

Upload the package to a datastore that the host you’re patching has access to by using a utility such as WinSCP:

Step #3 – Patch ESXi host via CLI

Proceed to either access the console directly on the server or SSH to it.

The command we’ll be using to install the patch will look as such:

esxcli software vib install -d <full path to patch file>

Note that it is important to specify the full path to the patch file because if you don’t then the follow error will be thrown:

/vmfs/volumes/58da7103-7810a474-5615-000eb6b4b6ea # esxcli software vib install

-d ESXi550-201612001.zip

[MetadataDownloadError]

Could not download from depot at zip:/var/log/vmware/ESXi550-201612001.zip?index.xml, skipping (('zip:/var/log/vmware/ESXi550-201612001.zip?index.xml', '', "Error extracting index.xml from /var/log/vmware/ESXi550-201612001.zip: [Errno 2] No such file or directory: '/var/log/vmware/ESXi550-201612001.zip'"))

url = zip:/var/log/vmware/ESXi550-201612001.zip?index.xml

Please refer to the log file for more details.

/vmfs/volumes/58da7103-7810a474-5615-000eb6b4b6ea #

Forgetting to include the -d will throw the following error:

/vmfs/volumes/58da7103-7810a474-5615-000eb6b4b6ea # esxcli software vib install

/vmfs/volumes/rvbd_vsp_datastore/ESXi550-201612001.zip

Error: Unknown command or namespace software vib install /vmfs/volumes/rvbd_vsp_datastore/ESXi550-201612001.zip

The patch will successfully install if the full path is specified and the output would look as such:

/vmfs/volumes/58da7103-7810a474-5615-000eb6b4b6ea # esxcli software vib install

-d /vmfs/volumes/rvbd_vsp_datastore/ESXi550-201612001.zip

Installation Result

Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

Reboot Required: true

VIBs Installed: VMware_bootbank_ehci-ehci-hcd_1.0-3vmw.550.3.95.4345813, VMware_bootbank_esx-base_5.5.0-3.100.4722766, VMware_bootbank_esx-tboot_5.5.0-3.100.4722766, VMware_bootbank_esx-ui_1.12.0-4722375, VMware_bootbank_lsi-msgpt3_00.255.03.03-2vmw.550.3.78.3248547, VMware_bootbank_misc-drivers_5.5.0-3.95.4345813, VMware_bootbank_net-e1000e_3.2.2.1-2vmw.550.3.78.3248547, VMware_bootbank_net-igb_5.0.5.1.1-1vmw.550.2.54.2403361, VMware_bootbank_net-ixgbe_3.7.13.7.14iov-12vmw.550.2.62.2718055, VMware_bootbank_sata-ahci_3.0-22vmw.550.3.89.4179633, VMware_bootbank_xhci-xhci_1.0-3vmw.550.3.95.4345813, VMware_locker_tools-light_5.5.0-3.92.4345810

VIBs Removed: Intel_bootbank_net-igb_5.2.7-1OEM.550.0.0.1331820, Intel_bootbank_net-ixgbe_3.21.4-1OEM.550.0.0.1331820, VMware_bootbank_ehci-ehci-hcd_1.0-3vmw.550.0.0.1331820, VMware_bootbank_esx-base_5.5.0-3.71.3116895, VMware_bootbank_esx-tboot_5.5.0-2.33.2068190, VMware_bootbank_lsi-msgpt3_00.255.03.03-1vmw.550.1.15.1623387, VMware_bootbank_misc-drivers_5.5.0-3.68.3029944, VMware_bootbank_net-e1000e_3.2.2.1-2vmw.550.3.68.3029944, VMware_bootbank_sata-ahci_3.0-22vmw.550.3.68.3029944, VMware_bootbank_xhci-xhci_1.0-2vmw.550.3.68.3029944, VMware_locker_tools-light_5.5.0-3.68.3029944

VIBs Skipped: VMware_bootbank_ata-pata-amd_0.3.10-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-atiixp_0.4.6-4vmw.550.0.0.1331820, VMware_bootbank_ata-pata-cmd64x_0.2.5-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-pdc2027x_1.0-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-serverworks_0.4.3-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-sil680_0.4.8-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-via_0.3.3-2vmw.550.0.0.1331820, VMware_bootbank_block-cciss_3.6.14-10vmw.550.0.0.1331820, VMware_bootbank_elxnet_10.2.309.6v-1vmw.550.3.68.3029944, VMware_bootbank_esx-dvfilter-generic-fastpath_5.5.0-0.0.1331820, VMware_bootbank_esx-xlibs_5.5.0-0.0.1331820, VMware_bootbank_esx-xserver_5.5.0-0.0.1331820, VMware_bootbank_ima-qla4xxx_2.01.31-1vmw.550.0.0.1331820, VMware_bootbank_ipmi-ipmi-devintf_39.1-4vmw.550.0.0.1331820, VMware_bootbank_ipmi-ipmi-msghandler_39.1-4vmw.550.0.0.1331820, VMware_bootbank_ipmi-ipmi-si-drv_39.1-4vmw.550.0.0.1331820, VMware_bootbank_lpfc_10.0.100.1-1vmw.550.0.0.1331820, VMware_bootbank_lsi-mr3_0.255.03.01-2vmw.550.3.68.3029944, VMware_bootbank_misc-cnic-register_1.72.1.v50.1i-1vmw.550.0.0.1331820, VMware_bootbank_mtip32xx-native_3.3.4-1vmw.550.1.15.1623387, VMware_bootbank_net-be2net_4.6.100.0v-1vmw.550.0.0.1331820, VMware_bootbank_net-bnx2_2.2.3d.v55.2-1vmw.550.0.0.1331820, VMware_bootbank_net-bnx2x_1.72.56.v55.2-1vmw.550.0.0.1331820, VMware_bootbank_net-cnic_1.72.52.v55.1-1vmw.550.0.0.1331820, VMware_bootbank_net-e1000_8.0.3.1-3vmw.550.0.0.1331820, VMware_bootbank_net-enic_1.4.2.15a-1vmw.550.0.0.1331820, VMware_bootbank_net-forcedeth_0.61-2vmw.550.0.0.1331820, VMware_bootbank_net-mlx4-core_1.9.7.0-1vmw.550.0.0.1331820, VMware_bootbank_net-mlx4-en_1.9.7.0-1vmw.550.0.0.1331820, VMware_bootbank_net-nx-nic_5.0.621-1vmw.550.0.0.1331820, VMware_bootbank_net-tg3_3.123c.v55.5-1vmw.550.2.33.2068190, VMware_bootbank_net-vmxnet3_1.1.3.0-3vmw.550.2.39.2143827, VMware_bootbank_ohci-usb-ohci_1.0-3vmw.550.0.0.1331820, VMware_bootbank_qlnativefc_1.0.12.0-1vmw.550.0.0.1331820, VMware_bootbank_rste_2.0.2.0088-4vmw.550.1.15.1623387, VMware_bootbank_sata-ata-piix_2.12-10vmw.550.2.33.2068190, VMware_bootbank_sata-sata-nv_3.5-4vmw.550.0.0.1331820, VMware_bootbank_sata-sata-promise_2.12-3vmw.550.0.0.1331820, VMware_bootbank_sata-sata-sil24_1.1-1vmw.550.0.0.1331820, VMware_bootbank_sata-sata-sil_2.3-4vmw.550.0.0.1331820, VMware_bootbank_sata-sata-svw_2.3-3vmw.550.0.0.1331820, VMware_bootbank_scsi-aacraid_1.1.5.1-9vmw.550.0.0.1331820, VMware_bootbank_scsi-adp94xx_1.0.8.12-6vmw.550.0.0.1331820, VMware_bootbank_scsi-aic79xx_3.1-5vmw.550.0.0.1331820, VMware_bootbank_scsi-bnx2fc_1.72.53.v55.1-1vmw.550.0.0.1331820, VMware_bootbank_scsi-bnx2i_2.72.11.v55.4-1vmw.550.0.0.1331820, VMware_bootbank_scsi-fnic_1.5.0.4-1vmw.550.0.0.1331820, VMware_bootbank_scsi-hpsa_5.5.0-44vmw.550.0.0.1331820, VMware_bootbank_scsi-ips_7.12.05-4vmw.550.0.0.1331820, VMware_bootbank_scsi-lpfc820_8.2.3.1-129vmw.550.0.0.1331820, VMware_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.550.0.0.1331820, VMware_bootbank_scsi-megaraid-sas_5.34-9vmw.550.3.68.3029944, VMware_bootbank_scsi-megaraid2_2.00.4-9vmw.550.0.0.1331820, VMware_bootbank_scsi-mpt2sas_14.00.00.00-3vmw.550.3.68.3029944, VMware_bootbank_scsi-mptsas_4.23.01.00-9vmw.550.3.68.3029944, VMware_bootbank_scsi-mptspi_4.23.01.00-9vmw.550.3.68.3029944, VMware_bootbank_scsi-qla2xxx_902.k1.1-12vmw.550.3.68.3029944, VMware_bootbank_scsi-qla4xxx_5.01.03.2-6vmw.550.0.0.1331820, VMware_bootbank_uhci-usb-uhci_1.0-3vmw.550.0.0.1331820

/vmfs/volumes/58da7103-7810a474-5615-000eb6b4b6ea #

Proceed to vMotion the VMs on this host to another host or shut them down if they don’t need to be up and restart the host:

The new build number should be displayed once the host has successfully restarted:

Unable to reconfigure HA on VMware ESXi 5.5.0 hosts after failed vCenter upgrade

Problem

You’ve recently attempted to upgrade vCenter 5.5 from 3b to 3e but had to rollback the upgrade to a previous snapshot due to an issue.  Since the upgraded vCenter server was briefly started and had upgraded the HA agents on the ESXi hosts, you now see alarms thrown for the cluster as well as the hosts. The following is one of those errors:

Configuration Issues

There was an error unconfiguring the vSphere HA agent on this host. To solve this problem, connect the host to a vCenter Server of version 5.0 or later.

Attempting to use the Reconfigure for vSphere HA feature on the hosts fails and disabling then re-enabling HA on the cluster also fails.

Solution

To resolve this issue, the first step to try is disable HA on the cluster then follow instructions for reinstalling the HA agent as outlined in the following KB:

Verifying and reinstalling the correct version of the VMware vCenter Server agents (1003714)
https://kb.vmware.com/kb/1003714

Since the environment shown above is ESXi 5.5, the following commands were executed:

~ # cd opt/vmware/uninstallers/

/opt/vmware/uninstallers # ls

VMware-fdm-uninstall.sh

/opt/vmware/uninstallers # cp /opt/vmware/uninstallers/VMware-fdm-uninstall.sh /tmp

/opt/vmware/uninstallers # chmod +x /tmp/VMware-fdm-uninstall.sh

/opt/vmware/uninstallers # /tmp/VMware-fdm-uninstall.sh

/opt/vmware/uninstallers #

If completing the above continues to throw the same error, proceed and try to use the following commands to manually remove the FDM VIB:

~ # esxcli software vib list | grep -i fdm

vmware-fdm 5.5.0-3252642 VMware VMwareCertified 2016-06-23

~ # esxcli software vib remove -n vmware-fdm

Removal Result

Message: Operation finished successfully.

Reboot Required: false

VIBs Installed:

VIBs Removed: VMware_bootbank_vmware-fdm_5.5.0-3252642

VIBs Skipped:

~ #

~ # esxcli software vib list | grep -i fdm

~ #

Once completed, restart the host and attempt to re-enable HA on the cluster to configure the hosts.

Unable to power on virtual machines on ESXi host with the error: "Invalid or unsupported virtual machine configuration."

Problem

You’ve noticed that the Recent Tasks in your vCenter’s task list is displaying the following error when attempting to perform a Power On virtual machine task:

Invalid or unsupported virtual machine configuration.

Reviewing the detail of the task displays the following Task Details:

Name: Power On virtual machine

Status: Invalid or unsupported virtual machine configuration.

Error Stack:

An error was received from the ESX host while powering on VM <vmName>.

Transport (VMDB) error -45: Failed to connect to peer process.

Failed to power on’/vmfs/volumes/<GUID>/<vmName>/<vmName>.vmx’.

The problem appears to be host specific because you are able to power on the virtual machine if you move it to a different host.

Establishing an SSH session to the host and reviewing the logs show the following message constantly logged:

Cannot create file /tmp/.SwapInfoSysSwap.lock.LOCK for process hostd-worker because the inode table of its ramdisk (tmp) is full.

Solution

The environment where I encountered this issue was running HP ProLiant BL660c Gen8 blades which the VMware support engineer told me apparently had a bug that constantly wrote logs to the /tmp/vmware-root folder that eventually filled up the partition. To verify this, navigate to the /tmp/vmware-root folder and use the ls command to list the contents:

Note the amount of vmware-vmx-xxxxxxx.log files in the directory in the screenshot above. To correct the issue, either move the files out of the directory to an external storage device or simply delete them. In this example, I will use the rm vmware-vmx-xxx* command to delete the files. The reason why I am required to narrow down the file to a 3 digit + wildcard is because there are simply too many files in the directory to use vmware-vmx-* (if you try using that, you will get a Argument list too long message).

The problem should go away once the files are removed.