Wednesday, February 12, 2020

Confirming a Citrix ADC is CVE-2019-19781 vulnerable and compromised then proceeding to remediate

I took an extended leave from IT for most of 2009 until returning later in December and therefore haven’t had to deal with the Citrix CVE-2019-19781 vulnerability but I just so happened to come across an unpatched Citrix ADC the other day so I thought I’d go through the process as outlined by my friend Thomas Poppelgaard ( in detail to capture and show some screenshots of what an exploited Citrix ADC would reveal during the tests as well as the remediation I performed at the end. I won’t be reiterating all the details about the tests so please refer to his blog post for a more in depth description of why and what the test are for.

Confirming Vulnerability and Compromise

CISA and Citrix both offer tools to check the Citrix ADC for the vulnerability:



… but a community tool created by zentura_cp is available on Azure to perform the check quickly. That tool can be found here:


Scanning the Citrix ADC in this environment confirmed that it is vulnerable to CVE-2019-19781:


The wildcard certificate check did not reveal a match but I have recommended the client to revoke and reissue the certificate.

Checking the Template Files

As indicated by Thomas Pooelgaard’s article, the exploits all write files to the following three directories:




There is no need to switch to the shell prompt from the CLI as you can execute an ls on these directories as such:

shell ls /netscaler/portal/templates/*.xml

shell ls /var/tmp/netscaler/portal/templates

shell ls /var/vpn/bookmark/*.xml

Initiating a ls command for those directories return the following:

shell ls /netscaler/portal/templates/*.xml


> shell ls /netscaler/portal/templates/*.xml





























































































The results from above indicate that there are plenty of xml files that have filenames suggesting the Citrix ADC has been compromised.

Proceeding to execute the following:

shell ls /var/tmp/netscaler/portal/templates


… reveals that the /var/tmp/netscaler/portal/templates directory appears to be fine.

Executing the last command:

shell ls /var/vpn/bookmark/*.xml


> shell ls /var/vpn/bookmark/*.xml















… reveals that the /var/vpn/bookmark/ directory also displays the same type of files and suggests the ADC has been compromised.

Reviewing Apache Log Files

Executing the first 3 commands to review the Apache logs does not reveal any results:

shell cat /var/log/httpaccess.log | grep vpns | grep xml

shell cat /var/log/httpaccess.log | grep "/\.\./"

shell gzcat /var/log/httpaccess.log.*.gz | grep vpns | grep xml

However, execute the following command does:

shell gzcat /var/log/httpaccess.log.*.gz | grep "/\.\./"


The suspicious xml files are notably present in the results above.

Executing the commands to determine whether there are any scheduled tasks to maintain access after patches does not reveal any jobs that a compromised Citrix ADC would have:

shell cat /etc/crontab

shell crontab -l -u nobody


Checking for Backdoor Scripts

Executing the following two commands to look for backdoor scripts:

shell ps -aux | grep python

shell ps -aux | grep perl

… reveal the following:

> shell ps -aux | grep python

nobody 82094 0.0 0.1 60980 1852 ?? Ss 28Jan20 0:37.76 /var/python/bin/python2 -c -c (python2.7)

root 36830 0.0 0.1 9096 912 0 RL+ 1:51PM 0:00.00 grep python

> shell ps -aux | grep perl

nobody 53632 100.0 0.0 5804 272 ?? R Sun05AM 3176:31.20 /tmp/.perl

nobody 77911 0.0 0.0 5804 412 ?? I 28Jan20 0:07.52 /tmp/.perl

root 36832 0.0 0.1 9096 1024 0 S+ 1:51PM 0:00.00 grep perl



The presence of nobody scripts is a red flag that there are backdoor scripts currently on the NetScaler and we can see that there are 3 of them listed.

Checking for Cyrpto Mining

Executing the following command to list the current running processes:

shell top -n 10

… reveal the following:

> shell top -n 10

last pid: 36937; load averages: 2.00, 2.00, 2.00 up 123+23:58:22 13:53:29

77 processes: 3 running, 71 sleeping, 3 zombie

Mem: 27M Active, 3504K Inact, 1487M Wired, 11M Cache, 169M Buf, 2528K Free

Swap: 4198M Total, 299M Used, 3899M Free, 7% Inuse


1174 root 1 44 r0 1210M 1210M CPU1 1 2976.0 100.00% NSPPE-00

53632 nobody 1 119 0 5804K 272K RUN 0 53.0H 100.00% .perl

1176 root 1 44 0 83392K 5668K kqread 0 30.7H 0.00% nsnetsvc

1297 root 1 44 0 37128K 3232K kqread 0 387:11 0.00% nsrised

1434 root 1 44 0 54752K 444K nanslp 0 316:34 0.00% nsprofmon

994 root 1 44 0 76720K 1356K select 0 112:35 0.00% vmtoolsd

1199 root 1 44 0 99M 11940K kqread 0 111:55 0.00% nsaggregatord

1331 root 2 76 0 70184K 13168K ucond 0 41:09 0.00% nscopo

25 root 1 44 0 29784K 29872K kqread 0 37:05 0.00% pitboss

1314 root 1 44 0 63928K 9820K nanslp 0 30:48 0.00% nscollect




The presence of the nobody account running a perl script that is consuming 100.00% of the WCPU suggests malicious activity currently running on the Citrix ADC.

Reviewing the Bash Logs

Executing the following 2 commands to review the bash logs:

shell cat /var/log/bash.log | grep nobody

shell gzcat /var/log/bash.*.gz | grep nobody

… does not reveal the nobody account present:


The following 2 commands display the full output of both logs allowing you to manually comb through the entries but the results from all the tests above are sufficient enough to deem the appliance as being compromised:

shell cat /var/log/httperror.log

shell gzcat /var/log/httperror.log.*.gz



This particular environment did not have a backup pre-December 2019 for me to restore so the only viable option that was available is to redeploy the VPX appliance and restore the configuration. The latest backup I could find dates back to 2 years ago but given that there has not been any change to the Citrix ADC, I decided to download the same build, redeploy, restore backup configuration, upgrade appliance and swap out the existing one.

Prior to completing the swap it is also important to perform the following:

  1. Review all of the servers that the Citrix ADC interacts with to determine whether they have been breached. These services could include domain controllers, Citrix StoreFronts, Citrix Delivery Controllers, Licensing servers and others.
  2. Update all of the passwords both local to the ADC and credentials used to interact with other services such as LDAP lookups against Domain Controllers.
  3. Force password changes for all users who have logged onto the ADC to access resources. As most Citrix ADCs allow Active Directory integration, it is important to force password changes for AD accounts.
  4. Revoke and renew all SSL certificates on the Citrix ADC. Use the list of files on the ADC’s /nsconfig/ssl/ directory as the hackers who have breached the appliance would have been able to retrieve the files in there.
  5. Ensure that the version of the Citrix ADC is not vulnerable to the CVE-2019-19781.
  6. Once the above have been completed, proceed to place the Citrix ADC back on the internet and run the vulnerability scan against it.

Restoring Configuration from Backup

As indicated above, the backup I had to work with was a few versions back from the NS12.1 that the appliance was currently running:


Citrix’s official Backup and Restore documentation ( indicates we can use the same or newer build but just to avoid any unintended issues, I downloaded the same build and redeployed the appliance as the running version would need to be upgraded anyways. The first step for the restore is to get the backup file uploaded and since a restore can only be performed via the GUI, I will use it to upload the configuration backup.

Begin by navigating to System > Backup and Restore then click on the Backup button:


Select the Add button:


Click on the down arrow beside Choose File and select Local as the source then select the backup TGZ file:


**Note that it is important to choose the Local option because the Appliance option would allow you to upload but it will not be displayed as a backup in the options to restore.

With the backup uploaded, proceed to restore it from the TGZ package:


Note that there is no need to have backup the appliance since it’s a new deployment so select the Skip Backup checkbox:


The restore should be fairly quick and will display a Restore Successful at the bottom right hand corner of the window:


The restored configuration will not be active until you restart the appliance and it is important to not select Save configuration if you restart it from the GUI or else the empty configuration will overwrite what you have just restored.

Also note that the IP address configuration will match the original appliance so if you have left the other one online for forensics but prevented it from accessing the internet then you should either change its IP address to avoid conflict or move it to another network.


If you’re staging the new appliance and have not given it network connectivity then the following are useful commands to verify the restored configuration:

  • show ip – list the NSIP, SNIP, SIP, MIP, VIP IP addresses
  • show lb vserver – displays the Load Balancing Virtual Servers details
  • show lb vserver -summary -fullvalues – displays the list of Load Balancing Virtual Servers without truncating names (use | grep to add specific values to list such as UP or DOWN)
  • show ssl certKey – to list all certificates on the appliance
  • show ssl certLink – to list the links between Root CA and Subordinate CA certificates

It’s important to note that the previously installed Client certificates will be present but you should deem these as being compromised so proceed to revoke, reissue and reinstall those certificates.

Thursday, January 23, 2020

Microsoft Outlook fails to authenticate with Office 365 configured with DUO MFA

I was recently contacted to troubleshoot an issue where a user’s Outlook was unable to connect to Office 365 after a password change over the evening and while I found various forum posts describing the issue, the suggested solution required a slight change to work in the environment I was dealing with so this post serves to describe what I encountered and the solution.


  • Office 365 is configured for MFA with the product named DUO, which is now owned by Cisco
  • Users are automatically redirected to a Citrix NetScaler configured with DUO MFA authentication webpage when they attempt to log into Office 365 either via, or


A user is no longer able to connect to Office 365 with their Outlook client after the following actions:

  1. Her password was going to expire so she changed it at the end of the day
  2. She logged off after the password change and went home for the evening
  3. She arrived at the office this morning, logged into her laptop and noticed that her Outlook no longer connected

You’ve confirmed that their password was updated within Azure Active Directory (AAD) yesterday evening:


You’ve confirmed that the cached credentials were cleared:


You proceed to connect to their desktop/laptop and notice that her Outlook had the status displayed as:

Trying to connect…


Clicking on the Trying to connect… button would briefly bring up the what appears to be authentication prompt for Office 365:


The window is displayed for about 3 seconds and disappears.

Thinking that this may be an authentication issue, you try having the user authenticate via the Office Account sign in page but it does not resolve the issue:


You perform a bit of Googling on the internet and find the following two forum posts:

The discussion indicates that the following two registry keys be added:


DWORD: DisableADALatopWAMOverride

Value: 1


Value: 0

You proceed to add these two DWORDs to the registry:


Adding these two keys managed to display this classic authentication prompt when Outlook is restarted:


However, logging in with her new credentials did not correct the problem as the status would continue to be stuck at Trying to connect…:


What does end up working for this issue was when the following registry key was deleted:



Value: 0

… but with the following one configured:


DWORD: DisableADALatopWAMOverride

Value: 1


With the above setup, the Citrix NetScaler page loaded correctly when Outlook is started:


Having the user enter their credentials got Outlook to finally connect to Office 365:


What should also be noted is that if the following DWORD:



Value: 0

… is re-added back in after Outlook has connected then it will cause it to fail to connect again so this DWORD should be left unadded.

Friday, April 26, 2019

Attempting to rebalance VMware Horizon View linked-clones desktop pool datastore fails with: "Refit operation rebalance failed"


You attempt to migrate a linked-clone pool’s storage from one datastore to another by using the rebalance feature as outlined in the following KB:

Migrating linked clone pools to a different or new datastore (1028754)

… but notice that the operation attempts the migration on a few desktops but fails with the error: Refit operation rebalance failed in the Events log of the linked-clones desktop pool:

Clicking on the ellipsis icon for the error for more information does not reveal additional information:

Navigating outside of the Events log of the linked-clones desktop pool then into the Inventory view and clicking on the ellipsis icon for the desktop in error displays the following message:

Apr 5, 2019 8:34:03 PM ADT: View Composer Invalid Parent VM Fault: View Composer Invalid Parent VM Fault: Selected snapshot on parent VM is not found on VC server

Pairing state:

Configured by:

Attempted theft by:


This error is usually caused by situations where the master image for the linked-clones pool no longer have the snapshots that the desktops being moved are associated with.  The rebalance operation requires access to the snapshots in order to re-clone the replicas (snapshots) to the new datastore so if they no longer exist then this error would be thrown.  In the case of the environment, the pool of 200 desktops had a mix of 5 different snapshots where only 1 of them was still available on the master image.  To view the linked-clones desktops and the associated snapshot that are being used, navigate to Inventory > Machines (View Composer Details) and review the Image tab:

If the snapshots have already been deleted then the way to remediate the error is to recompose all the snapshots with an existing snapshot on the master image or create a new one and recompose with it.  The rebalance operation should complete successfully once all the linked-clone pools are using a snapshot that exists.

Thursday, April 25, 2019

Deploying Skype for Business Server 2019 Archiving and Monitoring Services

Deploying Skype for Business Server 2019 archiving and monitoring services are fairly much the same as the previous version but I thought it would be a good idea to write this updated post to demonstrate the process.  Before I begin, note that the official planning documentation Monitoring can be found here:

Deploy archiving for Skype for Business Server

Deploy monitoring in Skype for Business Server

Step #1 – Define Archiving and Monitoring services on Front End Server / Pool Properties

Begin by editing the properties of the front-end pool, navigate to the General properties and scroll down to the Associations section:

Enable the Archiving and Monitoring (CDR and QoE metrics) options:

Enter the FQDN of the SQL server that will be hosting the two databases when presented with the Define New SQL Server Store menu:

Select the same or create a new SQL instance for the monitoring service then click OK:

The archiving SQL Server store and Monitoring SQL Server store should now have the new SQL server instance defined:

Publish the topology:

The Publish Topology wizard will present you with the option to either automatically let the wizard determine where to store the database or manually changing the path by going into the advanced options:

Selecting advanced will present the following menu:

What I don’t like about this is that the database doesn’t get stored in the paths defined but rather in sub folders such as D:\Database\ArchivingStore\(default)\DbPath.  You can, obviously, change this afterwards or at least have them on the correct drives.

Step #2 (Optional) – Move SQL database and log files to appropriate location

If you are not satisfied with the path in which the database and the log files are stored, you can use the following SQL queries to change the path after taking the database offline:


    MODIFY FILE ( NAME = LcsCDR_data,  

                  FILENAME = 'D:\Database\LcsCDR.mdf'); 



    MODIFY FILE ( NAME = LcsCDR_log,  

                  FILENAME = 'D:\Log\LcsCDR.ldf'); 



    MODIFY FILE ( NAME = LcsLog_data,  

                  FILENAME = 'D:\Database\LcsLog.mdf'); 



    MODIFY FILE ( NAME = LcsLog_log,  

                  FILENAME = 'D:\Log\LcsLog.ldf'); 


    MODIFY FILE ( NAME = QoEMetrics_data,  

                  FILENAME = 'D:\Database\QoEMetrics.mdf'); 



    MODIFY FILE ( NAME = QoEMetrics_log,  

                  FILENAME = 'D:\Log\QoEMetrics.ldf'); 


Step #3 – Deploy Monitoring Reports on SSRS

With the databases created, proceed by launching the Deployment Wizard and click on Deploy Monitoring Report:

Select the appropriate SQL instance and define the SQL Server Reporting Services (SSRS) instance:

**Note that as the wizard indicates, this will overwrite existing reports so be cautious of any SSRS reports that may already exist on the server.

Enter an account that will be used to access the Monitoring database:

I’ve been asked many times in the past about what credentials should be entered and the short answer is to use a service account because this account will be granted required logon and database permissions to the database in order for it to access the database.  The documentation about this can be found here:

Install Monitoring Reports in Skype for Business Server

On the Specify Credentials page, in the User name box, type the domain name and user name of the account to be used when accessing the Monitoring Reports (for example, litwareinc\kenmyer). If you do not use this format (domain\user name) an error will occur.

Type the user account password in the Password box, and then click Next. Note that no special rights are required for this account. The account will automatically be granted the required logon and database permissions when setup completes.

I usually prefer to create a service account for this:

Define a group that will have read-only access to SQL Server Reporting Services (SSRS):

**Note that the default is RTCUniversalReadOnlyAdmins

The wizard will complete the deployment with the information provided:

Once completed, you should see the service account granted ReportsReadOnlyRole to the databases:

Step #4 – Test Monitoring services reports

The reports can now be accessed via the following URL:


Step #6 – Configure Monitoring and Archiving Reports

It is generally not advisable to just leave the configuration for the monitoring or archiving reports as the default (you should at least review the default settings to ensure it meets the requirements of the organization) so proceed to launch the Skype for Business Server 2019 Control Panel and navigate to Monitoring and Archiving:

Go through the menus and edit the global properties or create a new one for the organization:

Wednesday, April 24, 2019

Running Remote Connectivity Analyzer against Exchange 2013/2016/2019 fails with: “MAPI over HTTP connectivity failed.”


You use the Remote Connectivity Analyzer to test external connectivity for an Exchange 2013/2016/2019 environment but notice that it fails with the following error:

Testing MAPI over HTTP connectivity to server
      MAPI over HTTP connectivity failed.
Additional Details
      Elapsed Time: 11 ms.

Test Steps
Attempting to resolve the host name in DNS.
      The host name couldn't be resolved.
        Tell me more about this issue and how to resolve it

Additional Details
      Host couldn't be resolved in DNS InfoDomainNonexistent.
Elapsed Time: 11 ms.

What’s strange about the error is that the server FQDN referenced in the message has the FQDN of the internal server name and domain name.


As much as the error suggests this caused by MAPI over HTTP, you may notice that executing the cmdlet:

Get-MapiVirtualDirectory | FL ServerName, *url*, *auth*

… would reveal that the configuration for the external MAPI Virtual Directory is already configured with the appropriate URL.

Executing the cmdlet:

Get-OrganizationConfig | fl identity,*mapi*

… will also indicate that MAPI over HTTP is enabled.

What typically causes this issue is if Outlook Anywhere has not been configured.  You can review the settings on each of the servers either by executing the following cmdlet:

Get-OutlookAnywhere | FL Identity,*Host*,*requireSSL*

… or navigating to the server properties’ Outlook Anywhere section, which would likely have the internal server’s FQDN as that is the default:

Either adjust the configuration in the EAC for each server appropriately or use the following cmdlet to configure the settings:

Set-OutlookAnywhere -Identity:"<serverName>\Rpc (Default Web Site)" -DefaultAuthenticationMethod NTLM -InternalHostname <internalFQDNURL> -InternalClientsRequireSsl $true -ExternalHostname <externalFQDNURL> -ExternalClientsRequireSsl $true -SSLOffloading $false

The Remote Connectivity Analyzer should return successful results once this has been corrected.

Tuesday, April 23, 2019

VMware Horizon View virtual desktop in “agent unreachable” status because OS is in repair boot mode

One of the common issues I’ve seen in many virtual desktop environments such as VMware Horizon View (or Citrix XenDesktop) is when a Windows 10 virtual desktop has crashed and gone into the repair mode upon reboot thus causing Horizon View to report the desktop as agent unreachable:

Reviewing the console of the virtual desktop in vCenter will confirm that it has booted into the recovery screen:

There isn’t really a benefit for having this feature on because it causes desktops to never boot into the operating system and therefore generating help desk calls.  The way to disable this feature is through the use of the command: bcdedit

Executing this command in the command prompt will display the configuration of the Windows Boot Loader and whether the recoveryenabled setting is set to Yes or No:

To disable the feature, simply execute the following command:

bcdedit /set recoveryenabled NO

Executing bcdedit after the configuration should show the recoveryenabled configuration set to No:

Those who still have Windows 7 virtual desktops in their environment can use the following command:

bcdedit /set {default} bootstatuspolicy ignoreshutdownfailures