VDP (vSphere Data Protection) 6.1.2 fails to backup after upgrading vCenter 5.5 from Update 3c to 3e

Problem

You’ve recently upgraded vCenter 5.5 from Update 3c to 3e and immediately noticed that your existing VDP 6.1.2.19 appliance backup jobs fail and the closest KB you can find related to this issue is the following:

825Communication fails in vSphere Data Protection with vCenter Server 5.5 Update 3 (2146825)https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2146825

However, attempting to download VDPHotfix.sh, uploading it to the VDP appliance and applying the hotfix does not appear to work:

login as: root
Using keyboard-interactive authentication.
Password:
Last login: Wed Jan 25 12:20:34 2017
root@vdp1prd:~/#: cd /home
root@vdp1prd:/home/#: cd admin
root@vdp1prd:/home/admin/#: ls
0.0           autorestart  cert.pem  getnodelogs  gsan.out  key.pem    truncate
VDPHotfix.sh  bin          cgsan     gsan         hfsclean  timerange
root@vdp1prd:/home/admin/#: .VDPHotfix.sh
-bash: .VDPHotfix.sh: command not found
root@vdp1prd:/home/admin/#: ./home/admin/VDPHotfix.sh
-bash: ./home/admin/VDPHotfix.sh: No such file or directory
root@vdp1prd:/home/admin/#: ls
0.0           autorestart  cert.pem  getnodelogs  gsan.out  key.pem    truncate
VDPHotfix.sh  bin          cgsan     gsan         hfsclean  timerange
root@vdp1prd:/home/admin/#: ./home/admin/VDPHotfix.sh
-bash: ./home/admin/VDPHotfix.sh: No such file or directory

Attempting to upload the full 2146825_vdp-hotfix.zip file onto the appliance and using the tar -zxvf command to extract the file does not work either:

admin@vdp1prd:/tmp/#: unzip 2146825_vdp-hotfix.zip
Archive:  2146825_vdp-hotfix.zip
  inflating: 2146825_vdp-hotfix.tar
admin@vdp1prd:/tmp/#: tar -zxvf 2146825_vdp-hotfix.
2146825_vdp-hotfix.tar  2146825_vdp-hotfix.zip
admin@vdp1prd:/tmp/#: tar -zxvf 2146825_vdp-hotfix.tar

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
admin@vdp1prd:/tmp/#: sh VDPHotfix.sh
sh: VDPHotfix.sh: Permission denied
admin@vdp1prd:/tmp/#: sh VDPHotfix.sh
sh: VDPHotfix.sh: Permission denied
admin@vdp1prd:/tmp/#: ls

Solution

Prior to calling VMware support, the only solution that worked for this environment where I encountered the problem was the workaround mentioned in the following forum post:

VDP 6.1.2 appliance can't connect to vCenter after initial configuration
https://communities.vmware.com/thread/542344?tstart=0

The workaround involved modifying the following file in the directory /usr/local/avamar/lib:

mcsutils.pm

… and adding the following line highlighted in red in between the two lines in black:

. "-Dfile.encoding=UTF-8 "
. "-Dsecurity.provider.rsa.JsafeJCE.position=last "
. "-Dlog4j.configuration=file://$mcsvar::lib_dir/log4j.properties "; # vmware/axis

Once this workaround was applied, the backup jobs began to run and complete.

Knowing that this workaround probably wasn’t the best solution, we opened a support call with VMware then was asked to upgrade the VDP appliance from 6.1.2 to 6.1.3 and noticed that the upgrade does indeed fix the issue.  I hope this post helps anyone who may come across this issue as I did not see an official KB article indicating upgrading was the solution.