VMware vSphere 5.5 sysprep customization never completes and fails for virtual machine

I recently ran into an extremely strange issue at a client’s environment that a VMware support engineer never managed to resolve and it was only by chance that I was able to figure out a workaround.  The likelihood of running into this issue again is probably quite low but I thought I’d write this blog post in case anyone else comes across it as I did.

Problem

You’ve noticed that deploying virtual machines from template with customization specifications no longer complete and result with the following error in VMware View when attempting to deploy full clone VDIs:

Customization operation timed out

Pariing state: In pairing…

Reviewing the full clone virtual machine logs show the start of customization:

Started customization of virtualMachine.

Customization log located at C:\Windows\TEMP\vmware-imc\guestcust.log in the guest OS.

Reviewing the guestcust.log does not show any errors:

Solution

While the problem described above can be caused by many reasons, the reason I encountered a few weeks ago was one that I would have never guessed.  What caused the customization process to fail in this environment I had to troubleshoot in was that the ESXi version was from a release date that was later than the vCenter version.

The ESXi version in this environment was 5.5.0, 3568722

The vCenter version in this environment was 5.5.0, 3252642

Using the information provided by the following KBs:

Build numbers and versions of VMware ESXi/ESX (2143832)
https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2143832

Build numbers and versions of VMware vCenter Server (2143838)
https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2143838

Will show the following information for the ESXi host and vCenter:

Version: ESXi 5.5 Express Patch 10
Release Date: 2016-02-22
Build Number: 3568722

Version: vCenter Server 5.5 Update 3b
Release Date: 2015-12-08
Build Number: 3252642

To correct the issue, update vCenter to a version with a later release date than the ESXi host and note that having the vCenter version newer than the ESXi host is best practice.

If upgrading the host is not an option, a strange workaround I noticed was that the customization would complete successfully if I disable HA on the cluster:

I’m not sure if this is specific to this environment but if there are no other options, try temporarily disabling HA and see if a new cloned VM would complete the customization.