Troubleshooting Citrix NetScaler VPX licensing issues

I receive a lot of calls from colleagues and customers for Citrix NetScaler licensing issues over the past few years so I thought I’d write a quick blog post to demonstrate what steps to take to troubleshoot the issue.

Problem

You’ve just allocated a license on the Citrix portal for the a NetScaler VPX appliance using the Host Id (which is the MAC address of the appliance’s NIC) with the following command:

shell

lmutil lmhostid

Or via the GUI:

You proceed to install it onto the appliance but noticed that none of the features are turned on and the top left corner still indicates that it is the Citrix AD VPX (Freemium) version with the following properties:

License Type: Standard

Model ID: 20

Licensing Mode: Express

Solution

The best way to determine why the applied license is not working is to review the license.log located on the appliance in the following directory:

/var/log/

Execute cat license.log to display the log entries:

root@ns# cat license.log
18:23:07 (lmgrd) -----------------------------------------------
18:23:07 (lmgrd)   Please Note:
18:23:07 (lmgrd)
18:23:07 (lmgrd)   This log is intended for debug purposes only.
18:23:07 (lmgrd)   In order to capture accurate license
18:23:07 (lmgrd)   usage data into an organized repository,
18:23:07 (lmgrd)   please enable report logging. Use Flexera Software LLC's
18:23:07 (lmgrd)   software license administration  solution,
18:23:07 (lmgrd)   FlexNet Manager, to  readily gain visibility
18:23:07 (lmgrd)   into license usage data and to create
18:23:07 (lmgrd)   insightful reports on critical information like
18:23:07 (lmgrd)   license availability and usage. FlexNet Manager
18:23:07 (lmgrd)   can be fully automated to run these reports on
18:23:07 (lmgrd)   schedule and can be used to track license
18:23:07 (lmgrd)   servers and usage across a heterogeneous
18:23:07 (lmgrd)   network of servers including Windows NT, Linux
18:23:07 (lmgrd)   and UNIX.
18:23:07 (lmgrd)
18:23:07 (lmgrd) -----------------------------------------------
18:23:07 (lmgrd)
18:23:07 (lmgrd)
18:23:07 (lmgrd) Server's System Date and Time: Tue Nov 06 2018 18:23:07 UTC
18:23:07 (lmgrd) SLOG: Summary LOG statistics is enabled.
18:23:07 (lmgrd) The license server manager (lmgrd) running as root:
18:23:07 (lmgrd)        This is a potential security problem
18:23:07 (lmgrd)        and is not recommended.
18:23:07 (lmgrd) FlexNet Licensing (v11.14.0.2 build 191018 i86_f8) started on ns () (11/6/2018)
18:23:07 (lmgrd) Copyright (c) 1988-2016 Flexera Software LLC. All Rights Reserved.
18:23:07 (lmgrd) World Wide Web:  http://www.flexerasoftware.com
18:23:07 (lmgrd) License file(s): /nsconfig/license/FID_ea88c82a_a1d1_47c5_960c_b518d36f6413.lic
18:23:07 (lmgrd) lmgrd tcp-port 27000
18:23:07 (lmgrd) (@lmgrd-SLOG@) ===============================================
18:23:07 (lmgrd) (@lmgrd-SLOG@) === LMGRD ===
18:23:07 (lmgrd) (@lmgrd-SLOG@) Start-Date: Tue Nov 06 2018 18:23:07 UTC
18:23:07 (lmgrd) (@lmgrd-SLOG@) PID: 10015
18:23:07 (lmgrd) (@lmgrd-SLOG@) LMGRD Version: v11.14.0.2 build 191018 i86_f8 ( build 191018 (ipv4))
18:23:07 (lmgrd) (@lmgrd-SLOG@)
18:23:07 (lmgrd) (@lmgrd-SLOG@) === Network Info ===
18:23:07 (lmgrd) (@lmgrd-SLOG@) Listening port: 27000
18:23:07 (lmgrd) (@lmgrd-SLOG@)
18:23:07 (lmgrd) (@lmgrd-SLOG@) === Startup Info ===
18:23:07 (lmgrd) (@lmgrd-SLOG@) Server Configuration: Single Server
18:23:07 (lmgrd) (@lmgrd-SLOG@) Command-line options used at LS startup: -l /var/log/license.log -c /nsconfig/license
18:23:07 (lmgrd) (@lmgrd-SLOG@) License file(s) used:  /nsconfig/license/FID_ea88c82a_a1d1_47c5_960c_b518d36f6413.lic
18:23:07 (lmgrd) (@lmgrd-SLOG@) ===============================================
18:23:07 (lmgrd) Starting vendor daemons ...
18:23:07 (lmgrd) Started CITRIX (internet tcp_port 14389 pid 10016)
18:23:07 (CITRIX) FlexNet Licensing version v11.14.0.2 build 191018 i86_f8
18:23:07 (CITRIX) SLOG: Summary LOG statistics is enabled.
18:23:07 (CITRIX) Server started on ns for:     CNS_V200S_SSERVER
18:23:07 (CITRIX) CNS_V200_SERVER CNS_SSE_SERVER
18:23:07 (CITRIX)
18:23:07 (CITRIX) Licenses are case sensitive for CITRIX
18:23:07 (CITRIX)
18:23:07 (CITRIX) Wrong hostid on SERVER line for license file:
18:23:07 (CITRIX)       /nsconfig/license/FID_ea88c82a_a1d1_47c5_960c_b518d36f6413.lic
18:23:07 (CITRIX) SERVER line says 0050569224f9, hostid is 005056927696
18:23:07 (CITRIX) Invalid hostid on SERVER line
18:23:07 (CITRIX) Disabling 1 license from feature CNS_SSE_SERVER(0B9B 56BD C8F9 3B01 )
18:23:07 (CITRIX) Disabling 1 license from feature CNS_V200S_SSERVER(05D9 67D7 CE21 1146 )
18:23:07 (CITRIX) Disabling 1 license from feature CNS_V200_SERVER(1CB2 E478 6D73 1DE8 )
18:23:07 (CITRIX) EXTERNAL FILTERS are OFF
18:23:07 (lmgrd) CITRIX using TCP-port 14389
18:23:07 (CITRIX) SLOG: Statistics Log Frequency is 240 minute(s).
18:23:07 (CITRIX) (@CITRIX-SLOG@) ===============================================
18:23:07 (CITRIX) (@CITRIX-SLOG@) === Vendor Daemon ===
18:23:07 (CITRIX) (@CITRIX-SLOG@) Vendor daemon: CITRIX
18:23:07 (CITRIX) (@CITRIX-SLOG@) Start-Date: Tue Nov 06 2018 18:23:07 UTC
18:23:07 (CITRIX) (@CITRIX-SLOG@) PID: 10016
18:23:07 (CITRIX) (@CITRIX-SLOG@) VD Version: v11.14.0.2 build 191018 i86_f8 ( build 191018 (ipv4))
18:23:07 (CITRIX) (@CITRIX-SLOG@)
18:23:07 (CITRIX) (@CITRIX-SLOG@) === Startup/Restart Info ===
18:23:07 (CITRIX) (@CITRIX-SLOG@) Options file used: None
18:23:07 (CITRIX) (@CITRIX-SLOG@) Is vendor daemon a CVD: No
18:23:07 (CITRIX) (@CITRIX-SLOG@) Number of VD restarts since LS startup: 0
18:23:07 (CITRIX) (@CITRIX-SLOG@)
18:23:07 (CITRIX) (@CITRIX-SLOG@) === Network Info ===
18:23:07 (CITRIX) (@CITRIX-SLOG@) Listening port: 14389
18:23:07 (CITRIX) (@CITRIX-SLOG@) Daemon select timeout (in seconds): 1
18:23:07 (CITRIX) (@CITRIX-SLOG@)
18:23:07 (CITRIX) (@CITRIX-SLOG@) === Host Info ===
18:23:07 (CITRIX) (@CITRIX-SLOG@) Host used in license file: ns
18:23:07 (CITRIX) (@CITRIX-SLOG@) Running on Hypervisor: Not determined - treat as Physical
18:23:07 (CITRIX) (@CITRIX-SLOG@) ===============================================
18:23:07 (CITRIX) No valid hostids, exiting
18:23:07 (CITRIX) EXITING DUE TO SIGNAL 34 Exit reason 2
18:23:07 (lmgrd) CITRIX exited with status 34 (Invalid host)
18:23:07 (lmgrd) Please correct problem and restart daemons
lmstat - Copyright (c) 1989-2016 Flexera Software LLC. All Rights Reserved.
Flexible License Manager status on Tue 11/6/2018 18:23

License server status: 27000@ns
     License file(s) on ns: /nsconfig/license/FID_ea88c82a_a1d1_47c5_960c_b518d36f6413.lic:

        ns: license server UP (MASTER) v11.14.0

Vendor daemon status (on ns):

    CITRIX: The desired vendor daemon is down. (-97,121)

18:23:10 (lmgrd) lmgrd will now shut down all the vendor daemons

18:23:10 (lmgrd) EXITING DUE TO SIGNAL 15
root@ns#

Reviewing the output above will usually provide the reason why the appliance isn’t licensed as expected and in the case of this example, the cause is an incorrect Host Id used to generate the license:

18:23:07 (CITRIX) Wrong hostid on SERVER line for license file:
18:23:07 (CITRIX)       /nsconfig/license/FID_ea88c82a_a1d1_47c5_960c_b518d36f6413.lic
18:23:07 (CITRIX) SERVER line says 0050569224f9, hostid is 005056927696
18:23:07 (CITRIX) Invalid hostid on SERVER line

Proceeding to reallocate the license with the appropriate Host Id will license the appliance as expected:

Attempting to install Server Certificate on NetScaler VPX fails with the error: "Object doesn't support property or method 'endsWith"

Problem

You’re attempting to install a Server Certificate on a Citrix NetScaler VPX NS12.1 49.23.nc with the required .cer and .key files but receive the following error:

"Object doesn't support property or method 'endsWith"

Solution

These error had me stumped for a while as I kept thinking this was caused by corrupted files but I realized a bit later that it was browser related.  This error would be thrown when I use IE 11.447.14393.0:

… but not when I use Chrome 70.0.3538.77.

VMware vCenter Site Recovery Manager 5.5.1.8569 service starts and stops

Problem

You’ve noticed that VMware vCenter Site Recovery Manager Server service briefly starts and then stops:

The System event logs has the following error entry:

Log Name: System

Source: Service Control Manager

Event ID: 7034

Level: Error

The VMware vCenter Site Recovery Manager Server service terminated unexpectedly. It has done this 3 time(s).

Reviewing the SRM latest log in the folder:

C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\Logs\

… reveals the following entry:

Section for VMware vCenter Site Recovery Manager, pid=5092, version=5.5.1, build=1647061, option=Release
2018-10-24T14:49:07.083+01:00 [03480 info 'Default'] Logging uses fast path: false
2018-10-24T14:49:07.083+01:00 [03480 info 'Default'] Handling bora/lib logs with VmaCore facilities
2018-10-24T14:49:07.083+01:00 [03480 info 'Default'] Initialized channel manager
2018-10-24T14:49:07.083+01:00 [03480 info 'Default'] Current working directory: C:\Program Files\VMware\VMware vCenter Site Recovery Manager\bin
2018-10-24T14:49:07.083+01:00 [03480 verbose 'Default'] Setting COM threading model to MTA
2018-10-24T14:49:07.083+01:00 [03480 info 'Default'] ThreadPool windowsStackImmediateCommit = true
2018-10-24T14:49:07.083+01:00 [03480 info 'ThreadPool'] Thread pool on asio: Min Io, Max Io, Min Task, Max Task, Max Concurency: 2, 401, 2, 200, 2147483647
2018-10-24T14:49:07.083+01:00 [03480 info 'ThreadPool'] Thread enlisted
2018-10-24T14:49:07.083+01:00 [03480 info 'Default'] Set dump dir to 'C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\DumpFiles'
2018-10-24T14:49:07.083+01:00 [04204 info 'ThreadPool'] Thread enlisted
2018-10-24T14:49:07.083+01:00 [04684 info 'ThreadPool'] Thread enlisted
2018-10-24T14:49:07.083+01:00 [03652 info 'ThreadPool'] Thread enlisted
2018-10-24T14:49:07.083+01:00 [00496 info 'ThreadPool'] Thread enlisted
2018-10-24T14:49:07.177+01:00 [03480 info 'Default'] Vmacore::InitSSL: handshakeTimeoutUs = 20000000
2018-10-24T14:49:07.239+01:00 [03480 error 'Default'] Certificate has expired.
2018-10-24T14:49:07.270+01:00 [03480 verbose 'HttpConnectionPool-000000'] HttpConnectionPoolImpl created. maxPoolConnections = 200; idleTimeout = 900000000; maxOpenConnections = 50; maxConnectionAge = 0
2018-10-24T14:49:07.317+01:00 [03652 verbose 'Default'] Local and remote versions are the same.  Talking with version vim.version.version9
2018-10-24T14:49:07.426+01:00 [03480 info 'Default'] VC Connection: Logging in extension by subject name
2018-10-24T14:49:07.426+01:00 [03480 info 'vmomi.soapStub[0]'] Resetting stub adapter for server <cs p:00000000041821b0, TCP:vcenter03.contoso.com:80> : Closed
2018-10-24T14:49:07.442+01:00 [03480 error 'Default'] VC server does not trust our client certificate.
2018-10-24T14:49:07.520+01:00 [00496 info 'ThreadPool'] Thread delisted
2018-10-24T14:49:07.520+01:00 [03652 info 'ThreadPool'] Thread delisted
2018-10-24T14:49:07.520+01:00 [04684 info 'ThreadPool'] Thread delisted
2018-10-24T14:49:07.520+01:00 [04204 info 'ThreadPool'] Thread delisted

Solution

As indicated in the log file above, the certificate that SRM uses for communication with vCenter has expired.  This can be confirmed by launching the certificate console and reviewing the properties of the certificate used by SRM.

To correct this issue, simply renew the certificate and update SRM to use the certificate by using the Change option in Programs and Features:

Select the Modify option:

You will need the service account you use to connect to the vCenter server:

The Automatically generate a certificate. option will generate a self-signed certificate.  For this example, I have generated a certificate with an internal Enterprise CA so I’ll be selecting Use a PKCS#12 certificate file.:

**Note that the bottom indicates the Installed certificate status: Certificate has expired.

Proceed and enter the SRM database information in the wizard:

Select the Use existing database. option:

Continue by clicking Install to apply the changes:

--------------------------------------------------------------------------------------------------------------------------------------------------------

A few items worth mentioning for the certificate are:

• You can export a certificate as a PFX format the rename it to have the .p12 extension for importing it in the wizard.
• The requirements for the certificate may not be what you typically anticipate (e.g. you need the IP address in it for some reason) so refer to the following KB and carefully read the requirements (https://kb.vmware.com/s/article/2085644).  The following are a few prompts that you may receive if the certificate being used does not meet the requirements:

Failed to validate certificate.

Details:

The certificate does not contain the SRM hots name. SRM server certificates must contain the SRM host name in the Subject Alternative Name field.

Failed to validate certificate.

Details:

The host name (somehostName.domain.com) in the Subject Alternative Name of the provided certificate does not identically match the SRM host name (10.31.30.12).

Attempting to upload a file onto a datastore with vSphere Client 6.5 fails with: "The operation failed."

Problem

You attempt to upload a file onto a datastore in a vSphere 6.5 environment with the vSphere Client but notice that it fails with the error message:

Clicking onto the Details… link beside The operation failed. reveals the following message:

The operation failed

The operation failed for an undetermined reason. Typically this problem occurs due to certificates that the browser does not trust. If you are using self-signed or custom certificates, open the URL below in a new browser tab and accept the certificate, then retry the operation. https://esxi07.domain.com If this does not resolve the problem, other possible solutions are shown in this KB article: http://kb.vmware.com/kb/2147256

You proceed to view the certificate with the browser and install it into the Local Computer Trusted Root Certification Authorities but the upload continues to fail:

Solution

Other than issuing a certificate from a trusted authority such as an Enterprise CA or public CA, you can quickly get around this by browsing to the webpage of the vCenter and download the self-signed certificate via the Download trusted root CA certificates link at the bottom right corner:

The download.zip file will contain a certs folder with a subfolders representing different operating systems such as Linux, Mac and Windows.  Open the appropriate operating system folder:

Then proceed to install the root certificate onto the desktop launching the vSphere Client:

Browsing back to the vCenter’s root website should no longer present a certificate warning:

… and datastore uploads should now work.

Attempting to authenticate with SecurEnvoy passcode for VMware Horizon View fails with: “Access Denied” and “Incorrect Soft Token Code Received From Client”

Problem

You’ve completed configuring VMware Horizon View with SecurEnvoy but when authentication fails with Access Denied:

Reviewing the SecurEnvoy logs reveal the following error:

Incorrect Soft Token Code Received From ClientIP=10.34.30.58 RemoteID=

Solution

One of the possible reasons why authentication would not work and this message is logged in the Log Viewer is if the Shared Secret configured on the VMware Horizon View Connection Server does not match the one configured in the corresponding Radius server in SecurEnvoy:

The following message should be logged once the authentication succeeds:

Access Accepted with Soft Token From ClientIP=10.34.30.58 RemoteID=