Wednesday, February 12, 2020

Confirming a Citrix ADC is CVE-2019-19781 vulnerable and compromised then proceeding to remediate

I took an extended leave from IT for most of 2009 until returning later in December and therefore haven’t had to deal with the Citrix CVE-2019-19781 vulnerability but I just so happened to come across an unpatched Citrix ADC the other day so I thought I’d go through the process as outlined by my friend Thomas Poppelgaard ( in detail to capture and show some screenshots of what an exploited Citrix ADC would reveal during the tests as well as the remediation I performed at the end. I won’t be reiterating all the details about the tests so please refer to his blog post for a more in depth description of why and what the test are for.

Confirming Vulnerability and Compromise

CISA and Citrix both offer tools to check the Citrix ADC for the vulnerability:



… but a community tool created by zentura_cp is available on Azure to perform the check quickly. That tool can be found here:


Scanning the Citrix ADC in this environment confirmed that it is vulnerable to CVE-2019-19781:


The wildcard certificate check did not reveal a match but I have recommended the client to revoke and reissue the certificate.

Checking the Template Files

As indicated by Thomas Pooelgaard’s article, the exploits all write files to the following three directories:




There is no need to switch to the shell prompt from the CLI as you can execute an ls on these directories as such:

shell ls /netscaler/portal/templates/*.xml

shell ls /var/tmp/netscaler/portal/templates

shell ls /var/vpn/bookmark/*.xml

Initiating a ls command for those directories return the following:

shell ls /netscaler/portal/templates/*.xml


> shell ls /netscaler/portal/templates/*.xml





























































































The results from above indicate that there are plenty of xml files that have filenames suggesting the Citrix ADC has been compromised.

Proceeding to execute the following:

shell ls /var/tmp/netscaler/portal/templates


… reveals that the /var/tmp/netscaler/portal/templates directory appears to be fine.

Executing the last command:

shell ls /var/vpn/bookmark/*.xml


> shell ls /var/vpn/bookmark/*.xml















… reveals that the /var/vpn/bookmark/ directory also displays the same type of files and suggests the ADC has been compromised.

Reviewing Apache Log Files

Executing the first 3 commands to review the Apache logs does not reveal any results:

shell cat /var/log/httpaccess.log | grep vpns | grep xml

shell cat /var/log/httpaccess.log | grep "/\.\./"

shell gzcat /var/log/httpaccess.log.*.gz | grep vpns | grep xml

However, execute the following command does:

shell gzcat /var/log/httpaccess.log.*.gz | grep "/\.\./"


The suspicious xml files are notably present in the results above.

Executing the commands to determine whether there are any scheduled tasks to maintain access after patches does not reveal any jobs that a compromised Citrix ADC would have:

shell cat /etc/crontab

shell crontab -l -u nobody


Checking for Backdoor Scripts

Executing the following two commands to look for backdoor scripts:

shell ps -aux | grep python

shell ps -aux | grep perl

… reveal the following:

> shell ps -aux | grep python

nobody 82094 0.0 0.1 60980 1852 ?? Ss 28Jan20 0:37.76 /var/python/bin/python2 -c -c (python2.7)

root 36830 0.0 0.1 9096 912 0 RL+ 1:51PM 0:00.00 grep python

> shell ps -aux | grep perl

nobody 53632 100.0 0.0 5804 272 ?? R Sun05AM 3176:31.20 /tmp/.perl

nobody 77911 0.0 0.0 5804 412 ?? I 28Jan20 0:07.52 /tmp/.perl

root 36832 0.0 0.1 9096 1024 0 S+ 1:51PM 0:00.00 grep perl



The presence of nobody scripts is a red flag that there are backdoor scripts currently on the NetScaler and we can see that there are 3 of them listed.

Checking for Cyrpto Mining

Executing the following command to list the current running processes:

shell top -n 10

… reveal the following:

> shell top -n 10

last pid: 36937; load averages: 2.00, 2.00, 2.00 up 123+23:58:22 13:53:29

77 processes: 3 running, 71 sleeping, 3 zombie

Mem: 27M Active, 3504K Inact, 1487M Wired, 11M Cache, 169M Buf, 2528K Free

Swap: 4198M Total, 299M Used, 3899M Free, 7% Inuse


1174 root 1 44 r0 1210M 1210M CPU1 1 2976.0 100.00% NSPPE-00

53632 nobody 1 119 0 5804K 272K RUN 0 53.0H 100.00% .perl

1176 root 1 44 0 83392K 5668K kqread 0 30.7H 0.00% nsnetsvc

1297 root 1 44 0 37128K 3232K kqread 0 387:11 0.00% nsrised

1434 root 1 44 0 54752K 444K nanslp 0 316:34 0.00% nsprofmon

994 root 1 44 0 76720K 1356K select 0 112:35 0.00% vmtoolsd

1199 root 1 44 0 99M 11940K kqread 0 111:55 0.00% nsaggregatord

1331 root 2 76 0 70184K 13168K ucond 0 41:09 0.00% nscopo

25 root 1 44 0 29784K 29872K kqread 0 37:05 0.00% pitboss

1314 root 1 44 0 63928K 9820K nanslp 0 30:48 0.00% nscollect




The presence of the nobody account running a perl script that is consuming 100.00% of the WCPU suggests malicious activity currently running on the Citrix ADC.

Reviewing the Bash Logs

Executing the following 2 commands to review the bash logs:

shell cat /var/log/bash.log | grep nobody

shell gzcat /var/log/bash.*.gz | grep nobody

… does not reveal the nobody account present:


The following 2 commands display the full output of both logs allowing you to manually comb through the entries but the results from all the tests above are sufficient enough to deem the appliance as being compromised:

shell cat /var/log/httperror.log

shell gzcat /var/log/httperror.log.*.gz



This particular environment did not have a backup pre-December 2019 for me to restore so the only viable option that was available is to redeploy the VPX appliance and restore the configuration. The latest backup I could find dates back to 2 years ago but given that there has not been any change to the Citrix ADC, I decided to download the same build, redeploy, restore backup configuration, upgrade appliance and swap out the existing one.

Prior to completing the swap it is also important to perform the following:

  1. Review all of the servers that the Citrix ADC interacts with to determine whether they have been breached. These services could include domain controllers, Citrix StoreFronts, Citrix Delivery Controllers, Licensing servers and others.
  2. Update all of the passwords both local to the ADC and credentials used to interact with other services such as LDAP lookups against Domain Controllers.
  3. Force password changes for all users who have logged onto the ADC to access resources. As most Citrix ADCs allow Active Directory integration, it is important to force password changes for AD accounts.
  4. Revoke and renew all SSL certificates on the Citrix ADC. Use the list of files on the ADC’s /nsconfig/ssl/ directory as the hackers who have breached the appliance would have been able to retrieve the files in there.
  5. Ensure that the version of the Citrix ADC is not vulnerable to the CVE-2019-19781.
  6. Once the above have been completed, proceed to place the Citrix ADC back on the internet and run the vulnerability scan against it.

Restoring Configuration from Backup

As indicated above, the backup I had to work with was a few versions back from the NS12.1 that the appliance was currently running:


Citrix’s official Backup and Restore documentation ( indicates we can use the same or newer build but just to avoid any unintended issues, I downloaded the same build and redeployed the appliance as the running version would need to be upgraded anyways. The first step for the restore is to get the backup file uploaded and since a restore can only be performed via the GUI, I will use it to upload the configuration backup.

Begin by navigating to System > Backup and Restore then click on the Backup button:


Select the Add button:


Click on the down arrow beside Choose File and select Local as the source then select the backup TGZ file:


**Note that it is important to choose the Local option because the Appliance option would allow you to upload but it will not be displayed as a backup in the options to restore.

With the backup uploaded, proceed to restore it from the TGZ package:


Note that there is no need to have backup the appliance since it’s a new deployment so select the Skip Backup checkbox:


The restore should be fairly quick and will display a Restore Successful at the bottom right hand corner of the window:


The restored configuration will not be active until you restart the appliance and it is important to not select Save configuration if you restart it from the GUI or else the empty configuration will overwrite what you have just restored.

Also note that the IP address configuration will match the original appliance so if you have left the other one online for forensics but prevented it from accessing the internet then you should either change its IP address to avoid conflict or move it to another network.


If you’re staging the new appliance and have not given it network connectivity then the following are useful commands to verify the restored configuration:

  • show ip – list the NSIP, SNIP, SIP, MIP, VIP IP addresses
  • show lb vserver – displays the Load Balancing Virtual Servers details
  • show lb vserver -summary -fullvalues – displays the list of Load Balancing Virtual Servers without truncating names (use | grep to add specific values to list such as UP or DOWN)
  • show ssl certKey – to list all certificates on the appliance
  • show ssl certLink – to list the links between Root CA and Subordinate CA certificates

It’s important to note that the previously installed Client certificates will be present but you should deem these as being compromised so proceed to revoke, reissue and reinstall those certificates.

Thursday, January 23, 2020

Microsoft Outlook fails to authenticate with Office 365 configured with DUO MFA

I was recently contacted to troubleshoot an issue where a user’s Outlook was unable to connect to Office 365 after a password change over the evening and while I found various forum posts describing the issue, the suggested solution required a slight change to work in the environment I was dealing with so this post serves to describe what I encountered and the solution.


  • Office 365 is configured for MFA with the product named DUO, which is now owned by Cisco
  • Users are automatically redirected to a Citrix NetScaler configured with DUO MFA authentication webpage when they attempt to log into Office 365 either via, or


A user is no longer able to connect to Office 365 with their Outlook client after the following actions:

  1. Her password was going to expire so she changed it at the end of the day
  2. She logged off after the password change and went home for the evening
  3. She arrived at the office this morning, logged into her laptop and noticed that her Outlook no longer connected

You’ve confirmed that their password was updated within Azure Active Directory (AAD) yesterday evening:


You’ve confirmed that the cached credentials were cleared:


You proceed to connect to their desktop/laptop and notice that her Outlook had the status displayed as:

Trying to connect…


Clicking on the Trying to connect… button would briefly bring up the what appears to be authentication prompt for Office 365:


The window is displayed for about 3 seconds and disappears.

Thinking that this may be an authentication issue, you try having the user authenticate via the Office Account sign in page but it does not resolve the issue:


You perform a bit of Googling on the internet and find the following two forum posts:

The discussion indicates that the following two registry keys be added:


DWORD: DisableADALatopWAMOverride

Value: 1


Value: 0

You proceed to add these two DWORDs to the registry:


Adding these two keys managed to display this classic authentication prompt when Outlook is restarted:


However, logging in with her new credentials did not correct the problem as the status would continue to be stuck at Trying to connect…:


What does end up working for this issue was when the following registry key was deleted:



Value: 0

… but with the following one configured:


DWORD: DisableADALatopWAMOverride

Value: 1


With the above setup, the Citrix NetScaler page loaded correctly when Outlook is started:


Having the user enter their credentials got Outlook to finally connect to Office 365:


What should also be noted is that if the following DWORD:



Value: 0

… is re-added back in after Outlook has connected then it will cause it to fail to connect again so this DWORD should be left unadded.