Pages

Thursday, January 17, 2019

Citrix NetScaler CLI command cheat sheet

I worked with a Citrix NetScaler engineer a year ago on a case where we had to had to review historic and live logs to troubleshoot an issue and was told that they had a cheat sheet of commonly used commands so I asked her to send it to me.  Below are commands I’ve come to find very useful during troubleshooting:

Uncompress an archived log file:

gunzip newnslog.21.gz

Discover the time period covered by the log:

nsconmsg -K newnslog.21 –d setime

View load-balancing statistics from the archived log:

nsconmsg -K newnslog.21 -s ConLb=2 -d oldconmsg

Extract logging information for a shorter duration:

nsconmsg -K newnslog.21 -s time=12Jan2006:00:00 -k short_log.nsl -T 1200 -d copy

Start the log process for newnslog:

nsconmsg -k /var/nslog/newnslog -T 172800 &

If you want to:

View the time span of the current newnslog file:

nsconmsg -K newnslog -d setime

View the time span of the archived newnslog file:

zcat filename | nsconmsg -K pipe -d setime

View events in the current newnslog file:

nsconmsg -K newnslog -d event

View console messages in the current newnslog file:

nsconmsg -K newnslog -d consmsg

View counter values in the current newnslog file:

nsconmsg -K newnslog -d stats

View counter values in the current newnslog file:

nsconmsg -K newnslog -d stats –d current

View non-zero counter values in the current newnslog file:

nsconmsg -K newnslog -d statswt0 –d current

To display event information, such as entity up/down, alerts and configuration saves in the shell:

nsconmsg -K newnslog -d event

To display CPU usage in the shell:

nsconmsg -K newnslog -s totalcount=200 -g cpu_use -d current

To display memory utilization in the shell:

nsconmsg -s ConMEM=1 -d oldconmsg

To display established HTTP connections in the shell:

nsconmsg -j server_NSSVC_HTTP_vserver -d current

To display load balancing statistics in the shell:

nsconmsg -K newnslog –s ConLb=x –d oldconmsg

This command gives basic information when x=1 and detailed information when x=2.

Use the following command to view traffic distribution from the shell:

nsconmsg -K /var/nslog/newnslog -s time -s ConLB=2 -2 distrconmsg

To display load-balancing information in the shell:

nsconmsg -s ConLb=1 -d oldconmsg

To display monitoring statistics in the shell:

nsconmsg -K newnslog –s ConMon=x –d oldconmsg

This command gives basic information when x=1 and gives detailed information when x=2.

If you want to:

View SSL stats for front-end connections:

nsconmsg -K newnslog -s ConSSL=1 -d oldconmsg

View SSL stats for back-end connections:

nsconmsg -K newnslog -s ConSSL=2 -d oldconmsg

View SSL stats for front and back-end connections:

nsconmsg -K newnslog -s ConSSL=3 -d oldconmsg

To display content switching statistics in the shell:

nsconmsg -K newnslog –s ConCSW=1 -d oldconmsg

To display compression statistics in the shell:

nsconmsg -K newnslog –s ConCMP=x -d oldconmsg

This command gives old compression method related statistics when x=1 and gives new compression method related statistics when x=2

To display integrated caching statistics in the shell:

nsconmsg -K newnslog -s ConIC=1 -d oldconmsg

Tuesday, January 15, 2019

Attempting to open Excel files within Outlook 2016 in protected mode fails with: “Microsoft Excel cannot open or save any more documents because there is not enough available memory or disk space.”

Problem

You’ve noticed that attempting to open Excel files from within Outlook 2016:

image

… fails with the following message:

Microsoft Excel cannot open or save any more documents because there is not enough available memory or disk space.

To make more memory available, close workbooks or programs you no longer need.

To free disk space, delete files you no longer need from the disk you are saving to.

image

image

Further troubleshooting of the issue shows that this only happens to Excel files received from external senders outside of the Exchange organization and files sent from internal users are fine because files received from external senders are blocked:

image

… while files from internal senders are not:

image

**Note that you can determine where the temporary Excel file is stored when you launch Outlook by navigating to the following registry key item:

HKEY_CURRENT_USER\SOFTWARE\Microsoft\Office\16.0\outlook\Security

OutlookSecureTempFolder

imageimage

Reviewing forums and KB articles suggest the following workaround of disabling Enable Protected View for Outlook attachments:

image

image

While this solution works, it reduces the security protection for users because they will now open Excel Outlook attachments in non-protected mode.

Attempting to add the Outlook OutlookSecureTempFolder path as a Microsoft Office Trusted Location would not be a good solution because it also reduces security and Excel would not allow it either:

image

Solution

I wasn’t able to figure this out as all the forum posts I came across suggested the workaround so I opened up a ticket with Microsoft and the engineer eventually found the following KB:

Unable to open to Microsoft Word Documents or Excel Spreadsheets from Outlook 2013
https://support.microsoft.com/en-gb/help/3020607/unable-to-open-to-microsoft-word-documents-or-excel-spreadsheets-from

Reviewing the RDS server that had this problem revealed that the firewall was indeed disabled:

image

Re-enabling it and restarting the server corrected the problem.

Friday, December 21, 2018

VMware Horizon 7.6.0 agent unable to contact Connection Server with error: “com.vmware.vdi.messagesecurity.MessageSecurityException: Paired key does not exist”

Problem

You’ve noticed that some desktops are in Agent Unreachable state in the VMware Horizon 7 Administrator console and clicking on the elipsis displays the following detail:

Status:Agent unreachable

Pairing state:Paired and secured

Configured by:vcs.domain.com

Attempted theft by:

image

Reviewing the debug logs on the virtual desktop in the directory C:\programdata\VMware\VDM\Logs:

image

… displays the following entries:

2018-12-17T16:48:34.881-04:00 INFO  (0EAC-0D78) <Thread-4> [AgentJmsConfig] Attempting to securely pair agent for JMS communication
2018-12-17T16:48:34.881-04:00 DEBUG (0EAC-0D78) <Thread-4> [AgentJmsConfig] Using paired signing key
2018-12-17T16:48:34.881-04:00 DEBUG (0EAC-0D78) <Thread-4> [JmsManager] Unable to connect to JMS server conto-vcs.contoso.com com.vmware.vdi.logger.Logger.debug(Logger.java:44)
com.vmware.vdi.messagesecurity.MessageSecurityException: Paired key does not exist
     at com.vmware.vdi.agent.messageserver.AgentJmsConfig.pairOverJms(SourceFile:318)
     at com.vmware.vdi.agent.messageserver.JmsManager.connect(SourceFile:288)
     at com.vmware.vdi.agent.messageserver.Main.Start(SourceFile:1265)
2018-12-17T16:48:44.975-04:00 ERROR (0EAC-09C4) <Main Thread> [wsnm_jms] JavaBridge reader thread init: timed out
2018-12-17T16:48:44.975-04:00 DEBUG (0EAC-0400) <1024> [wsnm_jms] AddressCache::ThreadEntry: Stopping
2018-12-17T16:48:45.177-04:00 DEBUG (0644-0958) <SharedMemReaderThread> [MessageFrameWork] SharedMem reader opt, reader 0x02EA21E8 for channel 0x00311280 detaching from thread 0x02E62168 handleCount 7
2018-12-17T16:48:45.177-04:00 DEBUG (0644-0958) <SharedMemReaderThread> [MessageFrameWork] SharedMem reader opt, peer is dead - reader 0x02EA21E8 in thread 0x02E62168
2018-12-17T16:48:45.177-04:00 DEBUG (0644-0958) <SharedMemReaderThread> [MessageFrameWork] MessageFrameWork Worker Shutdown OnChannelDelete, Name=EventLoggerService
2018-12-17T16:48:45.177-04:00 DEBUG (0644-0958) <SharedMemReaderThread> [MessageFrameWork] MessageFrameWork Worker Shutdown OnChannelDelete, Name=JavaView-3756
2018-12-17T16:48:45.177-04:00 DEBUG (0644-0958) <SharedMemReaderThread> [MessageFrameWork] MessageFrameWork Worker Shutdown OnChannelDelete, Name=JMSBridge
2018-12-17T16:48:45.177-04:00 DEBUG (0644-0958) <SharedMemReaderThread> [MessageFrameWork] MessageFrameWork Worker Shutdown OnChannelDelete, Name=TimingProfilerService
2018-12-17T16:48:45.177-04:00 DEBUG (0644-0958) <SharedMemReaderThread> [MessageFrameWork] CORE::AuthChannelInt::~AuthChannelInt(): Closed incoming SharedMemory channel from machine conto-PLANET002.contoso.com, user contoso\conto-PLANET002$,  channel 00311280
2018-12-17T16:48:45.177-04:00 INFO  (0644-08F4) <JavaBridge> [wsnm_jmsbridge] wsnm_jms died, restarting in a minute

image

Further review of the logs displays the following entries:

2018-12-18T11:08:00.715-04:00 DEBUG (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] coregate 'KeyVault' locking

2018-12-18T11:08:00.715-04:00 DEBUG (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] coregate 'KeyVault' locked

2018-12-18T11:08:00.715-04:00 INFO (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVaultCAPI upgrade rekey start for wsnm_jms

2018-12-18T11:08:00.716-04:00 ERROR (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVaultCAPI encipher key 'local storage cipher key#1' MISSING, error:

2018-12-18T11:08:00.716-04:00 WARN (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault failed to convert 'DataRef-9IQloB7D' from CAPI

2018-12-18T11:08:00.718-04:00 ERROR (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault cannot verify signature, error = 0x80090006 (Invalid Signature.)

2018-12-18T11:08:00.718-04:00 ERROR (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault registry data BAD SIGNATURE

2018-12-18T11:08:00.719-04:00 WARN (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault failed to convert 'DataRef-AmH0fBxY' from CAPI

2018-12-18T11:08:00.719-04:00 ERROR (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVaultCAPI encipher key 'local storage cipher key#1' MISSING, error:

2018-12-18T11:08:00.720-04:00 WARN (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault failed to convert 'DataRef-hfBerq31' from CAPI

2018-12-18T11:08:00.720-04:00 ERROR (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault cannot verify signature, error = 0x80090006 (Invalid Signature.)

2018-12-18T11:08:00.720-04:00 ERROR (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault registry data BAD SIGNATURE

2018-12-18T11:08:00.721-04:00 WARN (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault failed to convert 'DataRef-I4nSYR9Q' from CAPI

2018-12-18T11:08:00.721-04:00 ERROR (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault cannot verify signature, error = 0x80090006 (Invalid Signature.)

2018-12-18T11:08:00.722-04:00 ERROR (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault registry data BAD SIGNATURE

2018-12-18T11:08:00.722-04:00 WARN (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault failed to convert 'DataRef-IYonrwPb' from CAPI

2018-12-18T11:08:00.722-04:00 ERROR (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault cannot verify signature, error = 87 (The parameter is incorrect.)

2018-12-18T11:08:00.722-04:00 ERROR (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault registry data BAD SIGNATURE

2018-12-18T11:08:00.722-04:00 WARN (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault failed to convert 'local master encryption key' from CAPI

2018-12-18T11:08:00.723-04:00 ERROR (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault cannot verify signature, error = 0x80090006 (Invalid Signature.)

2018-12-18T11:08:00.723-04:00 ERROR (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault registry data BAD SIGNATURE

2018-12-18T11:08:00.723-04:00 WARN (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault failed to convert 'local master encryption key' from CAPI

2018-12-18T11:08:00.724-04:00 ERROR (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault cannot verify signature, error = 87 (The parameter is incorrect.)

2018-12-18T11:08:00.724-04:00 ERROR (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault registry data BAD SIGNATURE

2018-12-18T11:08:00.724-04:00 WARN (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault failed to convert 'local master encryption key' from CAPI

2018-12-18T11:08:00.725-04:00 ERROR (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault cannot verify signature, error = 0x80090006 (Invalid Signature.)

2018-12-18T11:08:00.725-04:00 ERROR (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault registry data BAD SIGNATURE

2018-12-18T11:08:00.725-04:00 WARN (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault failed to convert 'local storage context' from CAPI

2018-12-18T11:08:00.726-04:00 ERROR (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault cannot verify signature, error = 0x80090006 (Invalid Signature.)

2018-12-18T11:08:00.726-04:00 ERROR (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault registry data BAD SIGNATURE

2018-12-18T11:08:00.726-04:00 WARN (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault failed to convert 'local storage context' from CAPI

2018-12-18T11:08:00.766-04:00 ERROR (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault cannot verify signature, error = 0x80090006 (Invalid Signature.)

2018-12-18T11:08:00.766-04:00 ERROR (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault registry data BAD SIGNATURE

2018-12-18T11:08:00.767-04:00 WARN (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVault failed to convert 'local storage context' from CAPI

2018-12-18T11:08:00.767-04:00 INFO (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] KeyVaultCAPI upgrade rekey completed for wsnm_jms

2018-12-18T11:08:00.767-04:00 DEBUG (12A0-0E88) <delayedRekeyCAPI> [MessageFrameWork] coregate 'KeyVault' un-locked

image

Solution

This issue had me stumped for a while as I could not find any results from the internet with the error messages above.  What ended up correcting the problem was a solution to another issue I had at another client that displayed a different error.  The solution can be found in my previous blog post here:

Troubleshooting VMware Horizon View 7.5.1 Virtual Desktop in “Agent unreachable” status
http://terenceluk.blogspot.com/2018/10/troubleshooting-vmware-horizon-view-751.html

Executing the vdmadmin command to reset the key pair resolved the issue:

vdmadmin -A -d desktop-pool-name -m name-of-machine-in-pool -resetkey

C:\Program Files\VMware\VMware View\Server\tools\bin>vdmadmin -A -d desktop-pool

-name -m planet002 -resetkey

Agent Public Key

================

MIHwMIGoBgcqhkjOOAQBMIGcAkEA/KaCzo4Syrom78z3EQ5SbbB4sF7ey80etKII864WF64B81uRpH5t

9jQTxeEu0ImbzRMqzVDZkVG9xD7nN1kuFwIVAJYu3cw2nLqOuyYO5rahJtk0bjjFAkBnhHGyepz0Tuka

ScUUfbGpqvJE8FpDTWSGkx0tFCcbnjUDC3H9c9oXkGmzLik1Yw4cIGI1TQ2iCmxBblC+eUykA0MAAkBA

lgbnFPWs2bOZJHQqyOtFVDf5IndS2riwKqaJTTUxCDutBJg4AsJqRVVa/Ktfc2Nq1joL+FF6AbAOxMtG

tyHT

C:\Program Files\VMware\VMware View\Server\tools\bin>

image

Wednesday, November 14, 2018

Troubleshooting Citrix NetScaler VPX licensing issues

I receive a lot of calls from colleagues and customers for Citrix NetScaler licensing issues over the past few years so I thought I’d write a quick blog post to demonstrate what steps to take to troubleshoot the issue.

Problem

You’ve just allocated a license on the Citrix portal for the a NetScaler VPX appliance using the Host Id (which is the MAC address of the appliance’s NIC) with the following command:

shell

lmutil lmhostid

image

Or via the GUI:

image

image

You proceed to install it onto the appliance but noticed that none of the features are turned on and the top left corner still indicates that it is the Citrix AD VPX (Freemium) version with the following properties:

License Type: Standard

Model ID: 20

Licensing Mode: Express

image

Solution

The best way to determine why the applied license is not working is to review the license.log located on the appliance in the following directory:

/var/log/

Execute cat license.log to display the log entries:

root@ns# cat license.log
18:23:07 (lmgrd) -----------------------------------------------
18:23:07 (lmgrd)   Please Note:
18:23:07 (lmgrd)
18:23:07 (lmgrd)   This log is intended for debug purposes only.
18:23:07 (lmgrd)   In order to capture accurate license
18:23:07 (lmgrd)   usage data into an organized repository,
18:23:07 (lmgrd)   please enable report logging. Use Flexera Software LLC's
18:23:07 (lmgrd)   software license administration  solution,
18:23:07 (lmgrd)   FlexNet Manager, to  readily gain visibility
18:23:07 (lmgrd)   into license usage data and to create
18:23:07 (lmgrd)   insightful reports on critical information like
18:23:07 (lmgrd)   license availability and usage. FlexNet Manager
18:23:07 (lmgrd)   can be fully automated to run these reports on
18:23:07 (lmgrd)   schedule and can be used to track license
18:23:07 (lmgrd)   servers and usage across a heterogeneous
18:23:07 (lmgrd)   network of servers including Windows NT, Linux
18:23:07 (lmgrd)   and UNIX.
18:23:07 (lmgrd)
18:23:07 (lmgrd) -----------------------------------------------
18:23:07 (lmgrd)
18:23:07 (lmgrd)
18:23:07 (lmgrd) Server's System Date and Time: Tue Nov 06 2018 18:23:07 UTC
18:23:07 (lmgrd) SLOG: Summary LOG statistics is enabled.
18:23:07 (lmgrd) The license server manager (lmgrd) running as root:
18:23:07 (lmgrd)        This is a potential security problem
18:23:07 (lmgrd)        and is not recommended.
18:23:07 (lmgrd) FlexNet Licensing (v11.14.0.2 build 191018 i86_f8) started on ns () (11/6/2018)
18:23:07 (lmgrd) Copyright (c) 1988-2016 Flexera Software LLC. All Rights Reserved.
18:23:07 (lmgrd) World Wide Web: 
http://www.flexerasoftware.com
18:23:07 (lmgrd) License file(s): /nsconfig/license/FID_ea88c82a_a1d1_47c5_960c_b518d36f6413.lic
18:23:07 (lmgrd) lmgrd tcp-port 27000
18:23:07 (lmgrd) (@lmgrd-SLOG@) ===============================================
18:23:07 (lmgrd) (@lmgrd-SLOG@) === LMGRD ===
18:23:07 (lmgrd) (@lmgrd-SLOG@) Start-Date: Tue Nov 06 2018 18:23:07 UTC
18:23:07 (lmgrd) (@lmgrd-SLOG@) PID: 10015
18:23:07 (lmgrd) (@lmgrd-SLOG@) LMGRD Version: v11.14.0.2 build 191018 i86_f8 ( build 191018 (ipv4))
18:23:07 (lmgrd) (@lmgrd-SLOG@)
18:23:07 (lmgrd) (@lmgrd-SLOG@) === Network Info ===
18:23:07 (lmgrd) (@lmgrd-SLOG@) Listening port: 27000
18:23:07 (lmgrd) (@lmgrd-SLOG@)
18:23:07 (lmgrd) (@lmgrd-SLOG@) === Startup Info ===
18:23:07 (lmgrd) (@lmgrd-SLOG@) Server Configuration: Single Server
18:23:07 (lmgrd) (@lmgrd-SLOG@) Command-line options used at LS startup: -l /var/log/license.log -c /nsconfig/license
18:23:07 (lmgrd) (@lmgrd-SLOG@) License file(s) used:  /nsconfig/license/FID_ea88c82a_a1d1_47c5_960c_b518d36f6413.lic
18:23:07 (lmgrd) (@lmgrd-SLOG@) ===============================================
18:23:07 (lmgrd) Starting vendor daemons ...
18:23:07 (lmgrd) Started CITRIX (internet tcp_port 14389 pid 10016)
18:23:07 (CITRIX) FlexNet Licensing version v11.14.0.2 build 191018 i86_f8
18:23:07 (CITRIX) SLOG: Summary LOG statistics is enabled.
18:23:07 (CITRIX) Server started on ns for:     CNS_V200S_SSERVER
18:23:07 (CITRIX) CNS_V200_SERVER CNS_SSE_SERVER
18:23:07 (CITRIX)
18:23:07 (CITRIX) Licenses are case sensitive for CITRIX
18:23:07 (CITRIX)
18:23:07 (CITRIX) Wrong hostid on SERVER line for license file:
18:23:07 (CITRIX)       /nsconfig/license/FID_ea88c82a_a1d1_47c5_960c_b518d36f6413.lic
18:23:07 (CITRIX) SERVER line says 0050569224f9, hostid is 005056927696
18:23:07 (CITRIX) Invalid hostid on SERVER line

18:23:07 (CITRIX) Disabling 1 license from feature CNS_SSE_SERVER(0B9B 56BD C8F9 3B01 )
18:23:07 (CITRIX) Disabling 1 license from feature CNS_V200S_SSERVER(05D9 67D7 CE21 1146 )
18:23:07 (CITRIX) Disabling 1 license from feature CNS_V200_SERVER(1CB2 E478 6D73 1DE8 )
18:23:07 (CITRIX) EXTERNAL FILTERS are OFF
18:23:07 (lmgrd) CITRIX using TCP-port 14389
18:23:07 (CITRIX) SLOG: Statistics Log Frequency is 240 minute(s).
18:23:07 (CITRIX) (@CITRIX-SLOG@) ===============================================
18:23:07 (CITRIX) (@CITRIX-SLOG@) === Vendor Daemon ===
18:23:07 (CITRIX) (@CITRIX-SLOG@) Vendor daemon: CITRIX
18:23:07 (CITRIX) (@CITRIX-SLOG@) Start-Date: Tue Nov 06 2018 18:23:07 UTC
18:23:07 (CITRIX) (@CITRIX-SLOG@) PID: 10016
18:23:07 (CITRIX) (@CITRIX-SLOG@) VD Version: v11.14.0.2 build 191018 i86_f8 ( build 191018 (ipv4))
18:23:07 (CITRIX) (@CITRIX-SLOG@)
18:23:07 (CITRIX) (@CITRIX-SLOG@) === Startup/Restart Info ===
18:23:07 (CITRIX) (@CITRIX-SLOG@) Options file used: None
18:23:07 (CITRIX) (@CITRIX-SLOG@) Is vendor daemon a CVD: No
18:23:07 (CITRIX) (@CITRIX-SLOG@) Number of VD restarts since LS startup: 0
18:23:07 (CITRIX) (@CITRIX-SLOG@)
18:23:07 (CITRIX) (@CITRIX-SLOG@) === Network Info ===
18:23:07 (CITRIX) (@CITRIX-SLOG@) Listening port: 14389
18:23:07 (CITRIX) (@CITRIX-SLOG@) Daemon select timeout (in seconds): 1
18:23:07 (CITRIX) (@CITRIX-SLOG@)
18:23:07 (CITRIX) (@CITRIX-SLOG@) === Host Info ===
18:23:07 (CITRIX) (@CITRIX-SLOG@) Host used in license file: ns
18:23:07 (CITRIX) (@CITRIX-SLOG@) Running on Hypervisor: Not determined - treat as Physical
18:23:07 (CITRIX) (@CITRIX-SLOG@) ===============================================
18:23:07 (CITRIX) No valid hostids, exiting
18:23:07 (CITRIX) EXITING DUE TO SIGNAL 34 Exit reason 2
18:23:07 (lmgrd) CITRIX exited with status 34 (Invalid host)
18:23:07 (lmgrd) Please correct problem and restart daemons
lmstat - Copyright (c) 1989-2016 Flexera Software LLC. All Rights Reserved.
Flexible License Manager status on Tue 11/6/2018 18:23

License server status: 27000@ns
     License file(s) on ns: /nsconfig/license/FID_ea88c82a_a1d1_47c5_960c_b518d36f6413.lic:

        ns: license server UP (MASTER) v11.14.0

Vendor daemon status (on ns):

    CITRIX: The desired vendor daemon is down. (-97,121)


18:23:10 (lmgrd) lmgrd will now shut down all the vendor daemons

18:23:10 (lmgrd) EXITING DUE TO SIGNAL 15
root@ns#

Reviewing the output above will usually provide the reason why the appliance isn’t licensed as expected and in the case of this example, the cause is an incorrect Host Id used to generate the license:

18:23:07 (CITRIX) Wrong hostid on SERVER line for license file:
18:23:07 (CITRIX)       /nsconfig/license/FID_ea88c82a_a1d1_47c5_960c_b518d36f6413.lic
18:23:07 (CITRIX) SERVER line says 0050569224f9, hostid is 005056927696
18:23:07 (CITRIX) Invalid hostid on SERVER line

Proceeding to reallocate the license with the appropriate Host Id will license the appliance as expected:

image

Monday, November 12, 2018

Attempting to install Server Certificate on NetScaler VPX fails with the error: "Object doesn't support property or method 'endsWith"

Problem

You’re attempting to install a Server Certificate on a Citrix NetScaler VPX NS12.1 49.23.nc with the required .cer and .key files but receive the following error:

"Object doesn't support property or method 'endsWith"

image

Solution

These error had me stumped for a while as I kept thinking this was caused by corrupted files but I realized a bit later that it was browser related.  This error would be thrown when I use IE 11.447.14393.0:

image

… but not when I use Chrome 70.0.3538.77.

Wednesday, November 7, 2018

VMware vCenter Site Recovery Manager 5.5.1.8569 service starts and stops

Problem

You’ve noticed that VMware vCenter Site Recovery Manager Server service briefly starts and then stops:

imageimage

The System event logs has the following error entry:

Log Name: System

Source: Service Control Manager

Event ID: 7034

Level: Error

The VMware vCenter Site Recovery Manager Server service terminated unexpectedly. It has done this 3 time(s).

image

Reviewing the SRM latest log in the folder:

C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\Logs\

image

… reveals the following entry:

Section for VMware vCenter Site Recovery Manager, pid=5092, version=5.5.1, build=1647061, option=Release
2018-10-24T14:49:07.083+01:00 [03480 info 'Default'] Logging uses fast path: false
2018-10-24T14:49:07.083+01:00 [03480 info 'Default'] Handling bora/lib logs with VmaCore facilities
2018-10-24T14:49:07.083+01:00 [03480 info 'Default'] Initialized channel manager
2018-10-24T14:49:07.083+01:00 [03480 info 'Default'] Current working directory: C:\Program Files\VMware\VMware vCenter Site Recovery Manager\bin
2018-10-24T14:49:07.083+01:00 [03480 verbose 'Default'] Setting COM threading model to MTA
2018-10-24T14:49:07.083+01:00 [03480 info 'Default'] ThreadPool windowsStackImmediateCommit = true
2018-10-24T14:49:07.083+01:00 [03480 info 'ThreadPool'] Thread pool on asio: Min Io, Max Io, Min Task, Max Task, Max Concurency: 2, 401, 2, 200, 2147483647
2018-10-24T14:49:07.083+01:00 [03480 info 'ThreadPool'] Thread enlisted
2018-10-24T14:49:07.083+01:00 [03480 info 'Default'] Set dump dir to 'C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\DumpFiles'
2018-10-24T14:49:07.083+01:00 [04204 info 'ThreadPool'] Thread enlisted
2018-10-24T14:49:07.083+01:00 [04684 info 'ThreadPool'] Thread enlisted
2018-10-24T14:49:07.083+01:00 [03652 info 'ThreadPool'] Thread enlisted
2018-10-24T14:49:07.083+01:00 [00496 info 'ThreadPool'] Thread enlisted
2018-10-24T14:49:07.177+01:00 [03480 info 'Default'] Vmacore::InitSSL: handshakeTimeoutUs = 20000000
2018-10-24T14:49:07.239+01:00 [03480 error 'Default'] Certificate has expired.
2018-10-24T14:49:07.270+01:00 [03480 verbose 'HttpConnectionPool-000000'] HttpConnectionPoolImpl created. maxPoolConnections = 200; idleTimeout = 900000000; maxOpenConnections = 50; maxConnectionAge = 0
2018-10-24T14:49:07.317+01:00 [03652 verbose 'Default'] Local and remote versions are the same.  Talking with version vim.version.version9
2018-10-24T14:49:07.426+01:00 [03480 info 'Default'] VC Connection: Logging in extension by subject name
2018-10-24T14:49:07.426+01:00 [03480 info 'vmomi.soapStub[0]'] Resetting stub adapter for server <cs p:00000000041821b0, TCP:vcenter03.contoso.com:80> : Closed
2018-10-24T14:49:07.442+01:00 [03480 error 'Default'] VC server does not trust our client certificate.
2018-10-24T14:49:07.520+01:00 [00496 info 'ThreadPool'] Thread delisted
2018-10-24T14:49:07.520+01:00 [03652 info 'ThreadPool'] Thread delisted
2018-10-24T14:49:07.520+01:00 [04684 info 'ThreadPool'] Thread delisted
2018-10-24T14:49:07.520+01:00 [04204 info 'ThreadPool'] Thread delisted

image

Solution

As indicated in the log file above, the certificate that SRM uses for communication with vCenter has expired.  This can be confirmed by launching the certificate console and reviewing the properties of the certificate used by SRM.

image

To correct this issue, simply renew the certificate and update SRM to use the certificate by using the Change option in Programs and Features:

image

image

Select the Modify option:

image

You will need the service account you use to connect to the vCenter server:

image

The Automatically generate a certificate. option will generate a self-signed certificate.  For this example, I have generated a certificate with an internal Enterprise CA so I’ll be selecting Use a PKCS#12 certificate file.:

image

**Note that the bottom indicates the Installed certificate status: Certificate has expired.

Proceed and enter the SRM database information in the wizard:

image

Select the Use existing database. option:

image

Continue by clicking Install to apply the changes:

imageimage

image

--------------------------------------------------------------------------------------------------------------------------------------------------------

A few items worth mentioning for the certificate are:

  • You can export a certificate as a PFX format the rename it to have the .p12 extension for importing it in the wizard.
  • The requirements for the certificate may not be what you typically anticipate (e.g. you need the IP address in it for some reason) so refer to the following KB and carefully read the requirements (https://kb.vmware.com/s/article/2085644).  The following are a few prompts that you may receive if the certificate being used does not meet the requirements:

Failed to validate certificate.

Details:

The certificate does not contain the SRM hots name. SRM server certificates must contain the SRM host name in the Subject Alternative Name field.

image

Failed to validate certificate.

Details:

The host name (somehostName.domain.com) in the Subject Alternative Name of the provided certificate does not identically match the SRM host name (10.31.30.12).

image

Monday, November 5, 2018

Attempting to upload a file onto a datastore with vSphere Client 6.5 fails with: "The operation failed."

Problem

You attempt to upload a file onto a datastore in a vSphere 6.5 environment with the vSphere Client but notice that it fails with the error message:

image

Clicking onto the Details… link beside The operation failed. reveals the following message:

The operation failed

The operation failed for an undetermined reason. Typically this problem occurs due to certificates that the browser does not trust. If you are using self-signed or custom certificates, open the URL below in a new browser tab and accept the certificate, then retry the operation. https://esxi07.domain.com If this does not resolve the problem, other possible solutions are shown in this KB article: http://kb.vmware.com/kb/2147256

image

You proceed to view the certificate with the browser and install it into the Local Computer Trusted Root Certification Authorities but the upload continues to fail:

imageimage

Solution

Other than issuing a certificate from a trusted authority such as an Enterprise CA or public CA, you can quickly get around this by browsing to the webpage of the vCenter and download the self-signed certificate via the Download trusted root CA certificates link at the bottom right corner:

image

The download.zip file will contain a certs folder with a subfolders representing different operating systems such as Linux, Mac and Windows.  Open the appropriate operating system folder:

image

Then proceed to install the root certificate onto the desktop launching the vSphere Client:

imageimageimage

image

image

imageimage

imageimageimageimage

image

Browsing back to the vCenter’s root website should no longer present a certificate warning:

image

… and datastore uploads should now work.