Pages

Monday, October 22, 2018

Unable to start the Content Library service on a Windows vCenter Server 6.5

Problem

You’ve noticed that the Content Library service on your Windows vCenter Server 6.5 has stopped:

image

C:\Program Files\VMware\vCenter Server\bin>service-control --status

Running:

VMWareAfdService VMWareCertificateService VMWareDirectoryService VMwareComponentManager VMwareDNSService VMwareIdentityMgmtService VMwareSTS VServiceManager rhttpproxy vPostgres vapiEndpoint vimPBSM vmon vmonapi vmsyslogcollector vmware-cis-config vmware-license vmware-perfcharts vmware-psc-client vmwareServiceControlAgent vpxd vpxd-svcs vsan-health vsphere-ui vspherewebclientsvc

Stopped:

EsxAgentManager VMWareCAMService content-library mbcs vmware-autodeploy-waiter vmware-imagebuilder vmware-network-coredump

C:\Program Files\VMware\vCenter Server\bin>

image

**Note that the content-library service on a Windows vCenter is named content-library while the VCSA has it named vmware-content-library so if you attempt to start the service with the supplied command in the KB then you’ll receive the error below:

C:\Program Files\VMware\vCenter Server\bin>service-control --status vmware-content-library

Failed to get service vmware-content-library status. Err Given service name vmware-content-library is invalid

Service-control failed. Error Given service name vmware-content-library is invalid

C:\Program Files\VMware\vCenter Server\bin>

image

Proceeding to start the service on the Windows vCenter 6.5 server failed with the following error:

C:\Program Files\VMware\vCenter Server\bin>service-control --start content-library

Perform start operation. vmon_profile=None, svc_names=['content-library'], include_coreossvcs=False, include_leafossvcs=False

2018-10-19T16:19:40.231Z Service content-library state STOPPED

Error executing start on service content-library. Details {

"resolution": null,

"detail": [

{

"args": [

"content-library"

],

"id": "install.ciscommon.service.failstart",

"localized": "An error occurred while starting service 'content-library'",

"translatable": "An error occurred while starting service '%(0)s'"

}

],

"componentKey": null,

"problemId": null

}

Service-control failed. Error {

"resolution": null,

"detail": [

{

"args": [

"content-library"

],

"id": "install.ciscommon.service.failstart",

"localized": "An error occurred while starting service 'content-library'",

"translatable": "An error occurred while starting service '%(0)s'"

}

],

"componentKey": null,

"problemId": null

}

C:\Program Files\VMware\vCenter Server\bin>

image

Attempting to start the Content Library Service from within the vSphere Web Client (Home > Administration > System Configuration > Services > Objects > Services > Content Library Service) will also fail:

image

The "Start service" operation failed for the entity with the following error message.

Error (com.vmware.vapi.std.errors.error) => {

messages = [],

data = <null>

}

image

Reviewing the content library logs show that it has not been updated during the time of the troubleshooting (this is because it is unable to start so no logs would be written):

C:\ProgramData\VMware\vCenterServer\logs\content-library

image

Solution

One of the reasons why the content library service on a Windows Server vCenter 6.5 server won’t start is if the appropriate local account created during the vCenter 6.5 server install no longer has the Log on as a batch job permission on the Windows server. In the case of this example, checking the properties of the permissions showed that the local server content library account was missing:

image

Manually adding the account back into the security permission corrected the issue:

image

It is also important to note that the accounts listed in the screenshots above are incomplete for all the vCenter Server services to function properly as there are many more accounts that need to be added as shown in the list below:

  • cm
  • content-library
  • eam
  • imagebuilder
  • mbcs
  • netdumper
  • perfcharts
  • rbd
  • vapiEndpoint
  • vmware-vpostgres
  • vsan-health
  • vsm
  • vsphere-client
  • vsphere-ui

Note that the list above can be found in this VMware KB:

Error "Logon failure: the user has not been granted the requested logon type at this computer" (2148054)
https://kb.vmware.com/s/article/2148054

The properties of the Log on as a batch job should look something like the screenshots below:

imageimage

With the appropriate account added, the content library service should start as expected:

C:\Program Files\VMware\vCenter Server\bin>service-control --start content-library

Perform start operation. vmon_profile=None, svc_names=['content-library'], include_coreossvcs=False, include_leafossvcs=False

2018-10-19T17:00:34.224Z Service content-library state STOPPED

Successfully started service content-library

image

image

Friday, October 19, 2018

Troubleshooting VMware Horizon View 7.5.1 Virtual Desktop in “Agent unreachable” status

I’ve had several colleagues reach out to me in the past to ask about how I normally troubleshoot an Agent unreachable issue when all of the usual checks were verified (e.g. desktop being powered on, an IP address was obtained, etc) so I thought I’d take the opportunity to write this blog post using a recent issue I encountered at one of the environments I work with.

Problem

You’ve noticed that desktops in a Desktop pool has the Status reported as being Agent unreachable:

image

Clicking on the ellipsis displays the following details:

Status: Agent unreachable

Pairing state:Paired and secured

Configured by: <viewConnectionServerFQDN>

Attempted theft by:

image

Troubleshooting

Assuming the following obvious items have already been verified:

  1. IP address was obtained
  2. There is connectivity between the View Connection server and VDI
  3. Desktop is powered on
  4. All Horizon View Agent services have been started
  5. Desktop’s domain secure connection is not broken

… the first step for troubleshooting issues that have limited information provided in the error details is to review the Horizon View Agent logs on the problematic virtual desktop via the following folder:

C:\ProgramData\VMware\VDM\logs

image

What I typically do when in the log folder is to sort the files by Date modified so the most recently modified files are displayed on top.  For this example, the latest debug log that had been modified was:

debug-2018-10-09-154215.txt

… so I opened the file in Notepad and began reviewing the entries:

image

Parsing through the logs allowed me to identify the following 3 suspicious entries:

2018-10-09T15:42:25.726-03:00 DEBUG (0F6C-09CC) <Thread-4> [BrokerUpdateUtility] Published CHANGEKEY request

2018-10-09T15:42:34.277-03:00 DEBUG (0724-0728) <Main Thread> [wsnm_desktop] WTS_CONSOLE_CONNECT, sessionId=1

2018-10-09T15:42:35.745-03:00 DEBUG (0F6C-09CC) <Thread-4> [BrokerUpdateUtility] Timeout waiting for success response

image

2018-10-09T15:42:16.553-03:00 WARN (0724-079C) <PluginInitThread> [ws_configmgr] Update rejected, attempt to reconfigure as paired with missing key information

image

2018-10-09T15:44:23.232-03:00 INFO (0724-0F38) <JavaBridge> [wsnm_jmsbridge] wsnm_jms died, restarting in a minute

image

After reviewing the agent logs of the virtual desktop, I proceeded to log onto the VMware View Horizon Connection Server to review the server logs located in the folder:

C:\ProgramData\VMware\VDM\logs

The most recent modified log file was:

log-2018-10-09.txt

… so I opened the file in Notepad and began reviewing the entries until I found the following:

2018-10-09T00:00:08.386-03:00 WARN (0EC8-0A34) <DesktopControlJMS> [JMSMessageSecurity] Message could not be validated: Signature invalid for identity agent/4f631619-2cf1-4676-a63e-a37ecc4b9e80

2018-10-09T00:00:08.386-03:00 WARN (0EC8-0A34) <DesktopControlJMS> [JMSMessageSecurity] Identity validation failed: UNKNOWN

2018-10-09T00:00:08.386-03:00 WARN (0EC8-0A34) <DesktopControlJMS> [DesktopTracker] CHANGEKEY message from agent/4f631619-2cf1-4676-a63e-a37ecc4b9e80 is discarded as it cannot be validated

image

The entry references a GUID of a desktop and not its name so to determine which desktop this GUID belongs to, I launched adsiedit.msc on the View Connection server:

imageimage

Then used these instructions to open the local VMware Horizon View ADAM database:

Connecting to the View ADAM Database (2012377)
https://kb.vmware.com/s/article/2012377

Name: View ADAM Database

Distinguished Name: dc=vdi,dc=vmware,dc=int

Server: localhost:389

image

Once connected to the ADAM database, navigate to the Servers OU then locate the unique GUID in the right hand pane:

image

Open the properties of the object and navigate down to the attribute named pae-DisplayName to locate the name:

image

Solution

Quickly searching for the identified log entries above brought me to the following KB:

After reinstalling or upgrading the View agent, the View Administrator console reports the message: Agent Unreachable (2038679)
https://kb.vmware.com/s/article/2038679

Using the provided command::

vdmadmin -A -d desktop-pool-name -m name-of-machine-in-pool -resetkey

… located in the directory:

C:\Program Files\VMware\VMware View\Server\tools\bin

Resolved the issue.

The following are the screenshots of what the output for the command would look like:

image

Note that a restart is not required for the status of the VDIs to return to normal:

image

Tuesday, October 16, 2018

Deploying vSphere Data Protection 6.1.9 on VMware vSphere 6.5

It has been a while since I’ve deployed a VDP appliance and since I had the opportunity to do so this week, I took the time to screenshot the process so I can write what would likely be the last post for this application since VMware will no longer be releasing new versions in the future.  For those interested, here is the latest compatibility matrix for vCenter Server and VDP:

https://www.vmware.com/resources/compatibility/sim/interop_matrix.php#interop&2=&68=

image

The deployment hasn’t changed much throughout the versions so this post serves more to demonstrate what the deployment looks like rather than how.

Deploying the OVA

One of the features I’ve enjoyed using since vSphere 6.0 is Content Libraries but I’d like to note that you can only deploy OVF packages and not OVA files from Content Libraries as outlined in this VMware Docs:

Templates in Content Libraries
https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.vm_admin.doc/GUID-F7BF0E6B-7C4F-4E46-8BBF-76229AEA7220.html

Templates are master copies of virtual machines that you can use to deploy virtual machines that are customized and ready for use. Templates promote consistency throughout your vSphere environment. Content libraries support only OVF templates. As a result, VM and Vapp templates are converted to OVF files when you upload the to a content library. You can use the templates to deploy virtual machines and vApps in the vSphere inventory.

image

With the above in mind, the best way to start the appliance deployment is simply use the Deploy OVF Template… option from within the vSphere Client or vSphere Web Client to select the download OVA file:

image

Select the Local file radio button and click on Choose Files to select the downloaded OVA VDP appliance:

image

vSphereDataProtection-6.1.9.ova

image

Proceed by selecting the folder location and configuring a meaningful name for the appliance:

image

Select the appropriate cluster for the appliance:

image

Review the details:

image

Accept the EULA as we always do:

image

Select the datastore that will store the appliance:

image

Configure the port group (VLAN) for the appliance:

image

Configure the network properties for the appliance:

image

Review the configuration and commence the deployment:

image

image

Configuring the appliance

Power on the VDP appliance once the OVA deployment has completed and wait until the operating system has completed resulting with the following screen displayed:

image

Navigate to the appliance via https://<IP Address>/vdp-configure and login with:

root / changeme

image

The following configuration will be presented:

image

Configure the IP settings as well as create an A (forward) and PTR (reverse) record for the appliance in your DNS:

image

Forgetting to create the DNS records will throw the following error:

IP/DNS could not be resolved. Check the IP, DNS values for forward and reverse lookup validation.

image

Configure the appropriate time zone:

image

Configure a password for the appliance that will change the default changeme password:

image

Configure the connect to the vCenter that this VDP appliance will connect to:

image

Click on the Test Connection button before attempting to hit the Next button as it is a requirement:

image

This part of the wizard is where you define the type of storage the VDP appliance will use.  The first option is to create new storage by adding VMDK disks to the VDP appliance to store backups, the second option is to attached existing disks from another VDP appliance and the third option is to migrate VDP storage data from another VDP release.  For the purpose of this example, we’ll be creating new storage that is 0.5TB in size because the intention is to attached this VDP deployment to a Data Domain.

image

Leave the Device Allocation settings as the default unless you want to separate the disks from the actual appliance:

image

Configure the CPU and Memory according to the recommended settings for the size of your backups as noted here:

https://docs.vmware.com/en/VMware-vSphere/6.5/vmware-data-protection-administration-guide-61.pdf

image

imageimage

Decide whether you want to join the feedback program (not sure whether this is relevant since this would be the last release:

image

Complete the deployment and decide whether to run the performance analysis on the storage configuration.  I will not be selecting this because we’ll be attaching this to a data domain:

imageimage

imageimage

The deployment will display the login screen during the configuration but don’t bother trying to log in as you won’t be able to:

image

Instead, proceed to the console of the appliance and wait until it restarts and boots back into the OS:

image

Log back into the appliance and you should see the appliance successfully deployed:

image

image

Note the size of the disks that are configured for a 0.5TB deployment:

boot – 200GB

data01 – 256GB

data02 – 256GB

data03 – 256GB

image

For comparison purposes, these are the size of the disks for a 1TB deployment:

boot – 200GB

data01 – 512GB

data02 – 512GB

data03 – 512GB

image

One of the things I eventually realized at the end of this deployment was that the Data Domain in this environment had a DDOS (Data Domain Operating System) that was at a version that no longer supported the VDP appliance since VMware was no longer going to release this product so when I attempted to connect to the DD, I will receive the following error:

image

Failed to add the Data Domain.

Reasons: Failed to connect to Data Domain system. The hostname, user name and/or password may be invalid.

image

Here is a are two snippets from the EMC site:

On April 5th, 2017, VMware announced the End of Availability (EOA) of the VMware vSphere Data Protection (VDP) product. At this time the latest release of VDP uses DDboost library 3.1.0.1 which is compatible up to DDOS 5.7.X. Due to the End of Availability announcement there are no plans to introduce additional DDboost libraries to allow support of later DDOS releases.  
For additional information on VDP/Data Domain compatibility please see the DDBoost for VMware compatibility Matrix found here:

DDBoost Compatibility Matrix

For additional information on the end of service life for VMware Data Protection please see the following documentation from VMware:

End of Availability (EOA) of VMware vSphere Data Protection (2149614)

----------------------------------------------------------------------------------------------------------------------------------------------------

Cause:

The latest version of the vDP contains the gsan 7.2.80.133 release. This version contains the 3.1.0.1-481386 ddboost libraries. This is not supported with the DDOS 6.1.x or greater releases.

Resolution:

1. Deploy an AVE and consider migrating the data from the vDP to AVE
2. If the Data Domain doesn't have any data, starting over and installing a lower version of the DDOS is another option.
Reference:
KB 503953
Compatibility App