Pages

Wednesday, November 14, 2018

Troubleshooting Citrix NetScaler VPX licensing issues

I receive a lot of calls from colleagues and customers for Citrix NetScaler licensing issues over the past few years so I thought I’d write a quick blog post to demonstrate what steps to take to troubleshoot the issue.

Problem

You’ve just allocated a license on the Citrix portal for the a NetScaler VPX appliance using the Host Id (which is the MAC address of the appliance’s NIC) with the following command:

shell

lmutil lmhostid

image

Or via the GUI:

image

image

You proceed to install it onto the appliance but noticed that none of the features are turned on and the top left corner still indicates that it is the Citrix AD VPX (Freemium) version with the following properties:

License Type: Standard

Model ID: 20

Licensing Mode: Express

image

Solution

The best way to determine why the applied license is not working is to review the license.log located on the appliance in the following directory:

/var/log/

Execute cat license.log to display the log entries:

root@ns# cat license.log
18:23:07 (lmgrd) -----------------------------------------------
18:23:07 (lmgrd)   Please Note:
18:23:07 (lmgrd)
18:23:07 (lmgrd)   This log is intended for debug purposes only.
18:23:07 (lmgrd)   In order to capture accurate license
18:23:07 (lmgrd)   usage data into an organized repository,
18:23:07 (lmgrd)   please enable report logging. Use Flexera Software LLC's
18:23:07 (lmgrd)   software license administration  solution,
18:23:07 (lmgrd)   FlexNet Manager, to  readily gain visibility
18:23:07 (lmgrd)   into license usage data and to create
18:23:07 (lmgrd)   insightful reports on critical information like
18:23:07 (lmgrd)   license availability and usage. FlexNet Manager
18:23:07 (lmgrd)   can be fully automated to run these reports on
18:23:07 (lmgrd)   schedule and can be used to track license
18:23:07 (lmgrd)   servers and usage across a heterogeneous
18:23:07 (lmgrd)   network of servers including Windows NT, Linux
18:23:07 (lmgrd)   and UNIX.
18:23:07 (lmgrd)
18:23:07 (lmgrd) -----------------------------------------------
18:23:07 (lmgrd)
18:23:07 (lmgrd)
18:23:07 (lmgrd) Server's System Date and Time: Tue Nov 06 2018 18:23:07 UTC
18:23:07 (lmgrd) SLOG: Summary LOG statistics is enabled.
18:23:07 (lmgrd) The license server manager (lmgrd) running as root:
18:23:07 (lmgrd)        This is a potential security problem
18:23:07 (lmgrd)        and is not recommended.
18:23:07 (lmgrd) FlexNet Licensing (v11.14.0.2 build 191018 i86_f8) started on ns () (11/6/2018)
18:23:07 (lmgrd) Copyright (c) 1988-2016 Flexera Software LLC. All Rights Reserved.
18:23:07 (lmgrd) World Wide Web: 
http://www.flexerasoftware.com
18:23:07 (lmgrd) License file(s): /nsconfig/license/FID_ea88c82a_a1d1_47c5_960c_b518d36f6413.lic
18:23:07 (lmgrd) lmgrd tcp-port 27000
18:23:07 (lmgrd) (@lmgrd-SLOG@) ===============================================
18:23:07 (lmgrd) (@lmgrd-SLOG@) === LMGRD ===
18:23:07 (lmgrd) (@lmgrd-SLOG@) Start-Date: Tue Nov 06 2018 18:23:07 UTC
18:23:07 (lmgrd) (@lmgrd-SLOG@) PID: 10015
18:23:07 (lmgrd) (@lmgrd-SLOG@) LMGRD Version: v11.14.0.2 build 191018 i86_f8 ( build 191018 (ipv4))
18:23:07 (lmgrd) (@lmgrd-SLOG@)
18:23:07 (lmgrd) (@lmgrd-SLOG@) === Network Info ===
18:23:07 (lmgrd) (@lmgrd-SLOG@) Listening port: 27000
18:23:07 (lmgrd) (@lmgrd-SLOG@)
18:23:07 (lmgrd) (@lmgrd-SLOG@) === Startup Info ===
18:23:07 (lmgrd) (@lmgrd-SLOG@) Server Configuration: Single Server
18:23:07 (lmgrd) (@lmgrd-SLOG@) Command-line options used at LS startup: -l /var/log/license.log -c /nsconfig/license
18:23:07 (lmgrd) (@lmgrd-SLOG@) License file(s) used:  /nsconfig/license/FID_ea88c82a_a1d1_47c5_960c_b518d36f6413.lic
18:23:07 (lmgrd) (@lmgrd-SLOG@) ===============================================
18:23:07 (lmgrd) Starting vendor daemons ...
18:23:07 (lmgrd) Started CITRIX (internet tcp_port 14389 pid 10016)
18:23:07 (CITRIX) FlexNet Licensing version v11.14.0.2 build 191018 i86_f8
18:23:07 (CITRIX) SLOG: Summary LOG statistics is enabled.
18:23:07 (CITRIX) Server started on ns for:     CNS_V200S_SSERVER
18:23:07 (CITRIX) CNS_V200_SERVER CNS_SSE_SERVER
18:23:07 (CITRIX)
18:23:07 (CITRIX) Licenses are case sensitive for CITRIX
18:23:07 (CITRIX)
18:23:07 (CITRIX) Wrong hostid on SERVER line for license file:
18:23:07 (CITRIX)       /nsconfig/license/FID_ea88c82a_a1d1_47c5_960c_b518d36f6413.lic
18:23:07 (CITRIX) SERVER line says 0050569224f9, hostid is 005056927696
18:23:07 (CITRIX) Invalid hostid on SERVER line

18:23:07 (CITRIX) Disabling 1 license from feature CNS_SSE_SERVER(0B9B 56BD C8F9 3B01 )
18:23:07 (CITRIX) Disabling 1 license from feature CNS_V200S_SSERVER(05D9 67D7 CE21 1146 )
18:23:07 (CITRIX) Disabling 1 license from feature CNS_V200_SERVER(1CB2 E478 6D73 1DE8 )
18:23:07 (CITRIX) EXTERNAL FILTERS are OFF
18:23:07 (lmgrd) CITRIX using TCP-port 14389
18:23:07 (CITRIX) SLOG: Statistics Log Frequency is 240 minute(s).
18:23:07 (CITRIX) (@CITRIX-SLOG@) ===============================================
18:23:07 (CITRIX) (@CITRIX-SLOG@) === Vendor Daemon ===
18:23:07 (CITRIX) (@CITRIX-SLOG@) Vendor daemon: CITRIX
18:23:07 (CITRIX) (@CITRIX-SLOG@) Start-Date: Tue Nov 06 2018 18:23:07 UTC
18:23:07 (CITRIX) (@CITRIX-SLOG@) PID: 10016
18:23:07 (CITRIX) (@CITRIX-SLOG@) VD Version: v11.14.0.2 build 191018 i86_f8 ( build 191018 (ipv4))
18:23:07 (CITRIX) (@CITRIX-SLOG@)
18:23:07 (CITRIX) (@CITRIX-SLOG@) === Startup/Restart Info ===
18:23:07 (CITRIX) (@CITRIX-SLOG@) Options file used: None
18:23:07 (CITRIX) (@CITRIX-SLOG@) Is vendor daemon a CVD: No
18:23:07 (CITRIX) (@CITRIX-SLOG@) Number of VD restarts since LS startup: 0
18:23:07 (CITRIX) (@CITRIX-SLOG@)
18:23:07 (CITRIX) (@CITRIX-SLOG@) === Network Info ===
18:23:07 (CITRIX) (@CITRIX-SLOG@) Listening port: 14389
18:23:07 (CITRIX) (@CITRIX-SLOG@) Daemon select timeout (in seconds): 1
18:23:07 (CITRIX) (@CITRIX-SLOG@)
18:23:07 (CITRIX) (@CITRIX-SLOG@) === Host Info ===
18:23:07 (CITRIX) (@CITRIX-SLOG@) Host used in license file: ns
18:23:07 (CITRIX) (@CITRIX-SLOG@) Running on Hypervisor: Not determined - treat as Physical
18:23:07 (CITRIX) (@CITRIX-SLOG@) ===============================================
18:23:07 (CITRIX) No valid hostids, exiting
18:23:07 (CITRIX) EXITING DUE TO SIGNAL 34 Exit reason 2
18:23:07 (lmgrd) CITRIX exited with status 34 (Invalid host)
18:23:07 (lmgrd) Please correct problem and restart daemons
lmstat - Copyright (c) 1989-2016 Flexera Software LLC. All Rights Reserved.
Flexible License Manager status on Tue 11/6/2018 18:23

License server status: 27000@ns
     License file(s) on ns: /nsconfig/license/FID_ea88c82a_a1d1_47c5_960c_b518d36f6413.lic:

        ns: license server UP (MASTER) v11.14.0

Vendor daemon status (on ns):

    CITRIX: The desired vendor daemon is down. (-97,121)


18:23:10 (lmgrd) lmgrd will now shut down all the vendor daemons

18:23:10 (lmgrd) EXITING DUE TO SIGNAL 15
root@ns#

Reviewing the output above will usually provide the reason why the appliance isn’t licensed as expected and in the case of this example, the cause is an incorrect Host Id used to generate the license:

18:23:07 (CITRIX) Wrong hostid on SERVER line for license file:
18:23:07 (CITRIX)       /nsconfig/license/FID_ea88c82a_a1d1_47c5_960c_b518d36f6413.lic
18:23:07 (CITRIX) SERVER line says 0050569224f9, hostid is 005056927696
18:23:07 (CITRIX) Invalid hostid on SERVER line

Proceeding to reallocate the license with the appropriate Host Id will license the appliance as expected:

image

Monday, November 12, 2018

Attempting to install Server Certificate on NetScaler VPX fails with the error: "Object doesn't support property or method 'endsWith"

Problem

You’re attempting to install a Server Certificate on a Citrix NetScaler VPX NS12.1 49.23.nc with the required .cer and .key files but receive the following error:

"Object doesn't support property or method 'endsWith"

image

Solution

These error had me stumped for a while as I kept thinking this was caused by corrupted files but I realized a bit later that it was browser related.  This error would be thrown when I use IE 11.447.14393.0:

image

… but not when I use Chrome 70.0.3538.77.

Wednesday, November 7, 2018

VMware vCenter Site Recovery Manager 5.5.1.8569 service starts and stops

Problem

You’ve noticed that VMware vCenter Site Recovery Manager Server service briefly starts and then stops:

imageimage

The System event logs has the following error entry:

Log Name: System

Source: Service Control Manager

Event ID: 7034

Level: Error

The VMware vCenter Site Recovery Manager Server service terminated unexpectedly. It has done this 3 time(s).

image

Reviewing the SRM latest log in the folder:

C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\Logs\

image

… reveals the following entry:

Section for VMware vCenter Site Recovery Manager, pid=5092, version=5.5.1, build=1647061, option=Release
2018-10-24T14:49:07.083+01:00 [03480 info 'Default'] Logging uses fast path: false
2018-10-24T14:49:07.083+01:00 [03480 info 'Default'] Handling bora/lib logs with VmaCore facilities
2018-10-24T14:49:07.083+01:00 [03480 info 'Default'] Initialized channel manager
2018-10-24T14:49:07.083+01:00 [03480 info 'Default'] Current working directory: C:\Program Files\VMware\VMware vCenter Site Recovery Manager\bin
2018-10-24T14:49:07.083+01:00 [03480 verbose 'Default'] Setting COM threading model to MTA
2018-10-24T14:49:07.083+01:00 [03480 info 'Default'] ThreadPool windowsStackImmediateCommit = true
2018-10-24T14:49:07.083+01:00 [03480 info 'ThreadPool'] Thread pool on asio: Min Io, Max Io, Min Task, Max Task, Max Concurency: 2, 401, 2, 200, 2147483647
2018-10-24T14:49:07.083+01:00 [03480 info 'ThreadPool'] Thread enlisted
2018-10-24T14:49:07.083+01:00 [03480 info 'Default'] Set dump dir to 'C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\DumpFiles'
2018-10-24T14:49:07.083+01:00 [04204 info 'ThreadPool'] Thread enlisted
2018-10-24T14:49:07.083+01:00 [04684 info 'ThreadPool'] Thread enlisted
2018-10-24T14:49:07.083+01:00 [03652 info 'ThreadPool'] Thread enlisted
2018-10-24T14:49:07.083+01:00 [00496 info 'ThreadPool'] Thread enlisted
2018-10-24T14:49:07.177+01:00 [03480 info 'Default'] Vmacore::InitSSL: handshakeTimeoutUs = 20000000
2018-10-24T14:49:07.239+01:00 [03480 error 'Default'] Certificate has expired.
2018-10-24T14:49:07.270+01:00 [03480 verbose 'HttpConnectionPool-000000'] HttpConnectionPoolImpl created. maxPoolConnections = 200; idleTimeout = 900000000; maxOpenConnections = 50; maxConnectionAge = 0
2018-10-24T14:49:07.317+01:00 [03652 verbose 'Default'] Local and remote versions are the same.  Talking with version vim.version.version9
2018-10-24T14:49:07.426+01:00 [03480 info 'Default'] VC Connection: Logging in extension by subject name
2018-10-24T14:49:07.426+01:00 [03480 info 'vmomi.soapStub[0]'] Resetting stub adapter for server <cs p:00000000041821b0, TCP:vcenter03.contoso.com:80> : Closed
2018-10-24T14:49:07.442+01:00 [03480 error 'Default'] VC server does not trust our client certificate.
2018-10-24T14:49:07.520+01:00 [00496 info 'ThreadPool'] Thread delisted
2018-10-24T14:49:07.520+01:00 [03652 info 'ThreadPool'] Thread delisted
2018-10-24T14:49:07.520+01:00 [04684 info 'ThreadPool'] Thread delisted
2018-10-24T14:49:07.520+01:00 [04204 info 'ThreadPool'] Thread delisted

image

Solution

As indicated in the log file above, the certificate that SRM uses for communication with vCenter has expired.  This can be confirmed by launching the certificate console and reviewing the properties of the certificate used by SRM.

image

To correct this issue, simply renew the certificate and update SRM to use the certificate by using the Change option in Programs and Features:

image

image

Select the Modify option:

image

You will need the service account you use to connect to the vCenter server:

image

The Automatically generate a certificate. option will generate a self-signed certificate.  For this example, I have generated a certificate with an internal Enterprise CA so I’ll be selecting Use a PKCS#12 certificate file.:

image

**Note that the bottom indicates the Installed certificate status: Certificate has expired.

Proceed and enter the SRM database information in the wizard:

image

Select the Use existing database. option:

image

Continue by clicking Install to apply the changes:

imageimage

image

--------------------------------------------------------------------------------------------------------------------------------------------------------

A few items worth mentioning for the certificate are:

  • You can export a certificate as a PFX format the rename it to have the .p12 extension for importing it in the wizard.
  • The requirements for the certificate may not be what you typically anticipate (e.g. you need the IP address in it for some reason) so refer to the following KB and carefully read the requirements (https://kb.vmware.com/s/article/2085644).  The following are a few prompts that you may receive if the certificate being used does not meet the requirements:

Failed to validate certificate.

Details:

The certificate does not contain the SRM hots name. SRM server certificates must contain the SRM host name in the Subject Alternative Name field.

image

Failed to validate certificate.

Details:

The host name (somehostName.domain.com) in the Subject Alternative Name of the provided certificate does not identically match the SRM host name (10.31.30.12).

image

Monday, November 5, 2018

Attempting to upload a file onto a datastore with vSphere Client 6.5 fails with: "The operation failed."

Problem

You attempt to upload a file onto a datastore in a vSphere 6.5 environment with the vSphere Client but notice that it fails with the error message:

image

Clicking onto the Details… link beside The operation failed. reveals the following message:

The operation failed

The operation failed for an undetermined reason. Typically this problem occurs due to certificates that the browser does not trust. If you are using self-signed or custom certificates, open the URL below in a new browser tab and accept the certificate, then retry the operation. https://esxi07.domain.com If this does not resolve the problem, other possible solutions are shown in this KB article: http://kb.vmware.com/kb/2147256

image

You proceed to view the certificate with the browser and install it into the Local Computer Trusted Root Certification Authorities but the upload continues to fail:

imageimage

Solution

Other than issuing a certificate from a trusted authority such as an Enterprise CA or public CA, you can quickly get around this by browsing to the webpage of the vCenter and download the self-signed certificate via the Download trusted root CA certificates link at the bottom right corner:

image

The download.zip file will contain a certs folder with a subfolders representing different operating systems such as Linux, Mac and Windows.  Open the appropriate operating system folder:

image

Then proceed to install the root certificate onto the desktop launching the vSphere Client:

imageimageimage

image

image

imageimage

imageimageimageimage

image

Browsing back to the vCenter’s root website should no longer present a certificate warning:

image

… and datastore uploads should now work.

Friday, November 2, 2018

Attempting to authenticate with SecurEnvoy passcode for VMware Horizon View fails with: “Access Denied” and “Incorrect Soft Token Code Received From Client”

Problem

You’ve completed configuring VMware Horizon View with SecurEnvoy but when authentication fails with Access Denied:

image

Reviewing the SecurEnvoy logs reveal the following error:

Incorrect Soft Token Code Received From ClientIP=10.34.30.58 RemoteID=

image

Solution

One of the possible reasons why authentication would not work and this message is logged in the Log Viewer is if the Shared Secret configured on the VMware Horizon View Connection Server does not match the one configured in the corresponding Radius server in SecurEnvoy:

imageimage

The following message should be logged once the authentication succeeds:

Access Accepted with Soft Token From ClientIP=10.34.30.58 RemoteID=

image

Monday, October 29, 2018

Unable to deploy OVF with VMware vCenter Server 6.5 on Microsoft Windows Server

Problem

You’re attempting to deploy and OVF within a VMware vCenter Server 6.5 installed on a Microsoft Windows Server but noticed that the deployment within the vSphere Web client fails with the following error:

This version of vCenter Server does not support Deploy OVF Template using this version of vSphere Web Client. To Deploy OVF Template, login with version 6.5.0.0 of vSphere Web Client.

image

You noticed that an attempt to use the vSphere Client (HTML5) will show that the Deploy OVF Template… is greyed out:

image

You’ve confirmed that all of the services within the Windows services console are started:

image

Searching the internet with the error message and symptoms returns plenty of results for the VCSA (vCenter Server Appliance) pointing to the following KB:

OVF deployment fails after upgrading to vCenter Server Appliance 6.5 U1 (2151085)
https://kb.vmware.com/s/article/2151085

However, the vCenter Server for this example is not an appliance but rather Windows and the version is 6.5 U2c (Build: 8815520):

Build numbers and versions of VMware vCenter Server (2143838)
https://kb.vmware.com/s/article/2143838

image

Troubleshooting

Having no luck finding any other articles or blog posts on the internet that applied to issue with the Windows version of vCenter, I went ahead and checked the Content Library service as noted in the VCSA KB article and noticed that it was indeed stopped:

C:\Program Files\VMware\vCenter Server\bin>service-control --status

Running:

VMWareAfdService VMWareCertificateService VMWareDirectoryService VMwareComponentManager VMwareDNSService VMwareIdentityMgmtService VMwareSTS VServiceManager rhttpproxy vPostgres vapiEndpoint vimPBSM vmon vmonapi vmsyslogcollector vmware-cis-config vmware-license vmware-perfcharts vmware-psc-client vmwareServiceControlAgent vpxd vpxd-svcs vsan-health vsphere-ui vspherewebclientsvc

Stopped:

EsxAgentManager VMWareCAMService content-library mbcs vmware-autodeploy-waiter vmware-imagebuilder vmware-network-coredump

C:\Program Files\VMware\vCenter Server\bin>

image

**Note that the content-library service on a Windows vCenter is named content-library while the VCSA has it named vmware-content-library so if you attempt to start the service with the supplied command in the KB then you’ll receive the error below:

C:\Program Files\VMware\vCenter Server\bin>service-control --status vmware-content-library

Failed to get service vmware-content-library status. Err Given service name vmware-content-library is invalid

Service-control failed. Error Given service name vmware-content-library is invalid

C:\Program Files\VMware\vCenter Server\bin>

image

Proceeding to start the service on the Windows vCenter 6.5 server failed with the following error:

C:\Program Files\VMware\vCenter Server\bin>service-control --start content-library

Perform start operation. vmon_profile=None, svc_names=['content-library'], include_coreossvcs=False, include_leafossvcs=False

2018-10-19T16:19:40.231Z Service content-library state STOPPED

Error executing start on service content-library. Details {

"resolution": null,

"detail": [

{

"args": [

"content-library"

],

"id": "install.ciscommon.service.failstart",

"localized": "An error occurred while starting service 'content-library'",

"translatable": "An error occurred while starting service '%(0)s'"

}

],

"componentKey": null,

"problemId": null

}

Service-control failed. Error {

"resolution": null,

"detail": [

{

"args": [

"content-library"

],

"id": "install.ciscommon.service.failstart",

"localized": "An error occurred while starting service 'content-library'",

"translatable": "An error occurred while starting service '%(0)s'"

}

],

"componentKey": null,

"problemId": null

}

C:\Program Files\VMware\vCenter Server\bin>

image

Attempting to start the Content Library Service from within the vSphere Web Client (Home > Administration > System Configuration > Services > Objects > Services > Content Library Service) will also fail:

image

The "Start service" operation failed for the entity with the following error message.

Error (com.vmware.vapi.std.errors.error) => {

messages = [],

data = <null>

}

image

Attempting to locate the ts-config.properties file as shown in the VCSA KB article will show that it exists but the corresponding ts-config.properties.rpmnew does not:

C:\ProgramData\VMware\vCenterServer\cfg\content-library\config

image

The content library logs also has not been updated during the time of the troubleshooting (this is because it is unable to start so no logs would be written):

C:\ProgramData\VMware\vCenterServer\logs\content-library

image

Solution

One of the reasons why the content library service on a Windows Server vCenter 6.5 server won’t start is if the appropriate local account created during the vCenter 6.5 server install no longer has the Log on as a batch job permission on the Windows server. In the case of this example, checking the properties of the permissions showed that the local server content library account was missing:

image

Manually adding the account back into the security permission corrected the issue:

image

It is also important to note that the accounts listed in the screenshots above are incomplete as there are many more accounts that need to be added as shown in the list below:

  • cm
  • content-library
  • eam
  • imagebuilder
  • mbcs
  • netdumper
  • perfcharts
  • rbd
  • vapiEndpoint
  • vmware-vpostgres
  • vsan-health
  • vsm
  • vsphere-client
  • vsphere-ui

Note that the list above can be found in this VMware KB:

Error "Logon failure: the user has not been granted the requested logon type at this computer" (2148054)
https://kb.vmware.com/s/article/2148054

The properties of the Log on as a batch job should look something like the screenshots below:

imageimage

With the appropriate account added, the content library service should start as expected:

C:\Program Files\VMware\vCenter Server\bin>service-control --start content-library

Perform start operation. vmon_profile=None, svc_names=['content-library'], include_coreossvcs=False, include_leafossvcs=False

2018-10-19T17:00:34.224Z Service content-library state STOPPED

Successfully started service content-library

image

image

You should now be able to deploy an OVF from either the vSphere Web Client or vSphere Client (HTML5):

imageimage