Pages

Wednesday, February 12, 2020

Confirming a Citrix ADC is CVE-2019-19781 vulnerable and compromised then proceeding to remediate

I took an extended leave from IT for most of 2009 until returning later in December and therefore haven’t had to deal with the Citrix CVE-2019-19781 vulnerability but I just so happened to come across an unpatched Citrix ADC the other day so I thought I’d go through the process as outlined by my friend Thomas Poppelgaard (https://www.poppelgaard.com/cve-2019-19781-what-you-should-know-and-how-to-fix-your-citrix-adc-access-gateway) in detail to capture and show some screenshots of what an exploited Citrix ADC would reveal during the tests as well as the remediation I performed at the end. I won’t be reiterating all the details about the tests so please refer to his blog post for a more in depth description of why and what the test are for.

Confirming Vulnerability and Compromise

CISA and Citrix both offer tools to check the Citrix ADC for the vulnerability:

CISA UTILITY TO CHECK IF CUSTOMER IS VULNERABLE

https://github.com/cisagov/check-cve-2019-19781

CVE-2019-19781 – VERIFICATION TOOL FROM CITRIX

https://support.citrix.com/article/CTX269180

… but a community tool created by zentura_cp is available on Azure to perform the check quickly. That tool can be found here: https://cve-2019-19781.azurewebsites.net

image

Scanning the Citrix ADC in this environment confirmed that it is vulnerable to CVE-2019-19781:

image

The wildcard certificate check did not reveal a match but I have recommended the client to revoke and reissue the certificate.

Checking the Template Files

As indicated by Thomas Pooelgaard’s article, the exploits all write files to the following three directories:

/netscaler/portal/templates/

/var/tmp/netscaler/portal/templates

/var/vpn/bookmark/

There is no need to switch to the shell prompt from the CLI as you can execute an ls on these directories as such:

shell ls /netscaler/portal/templates/*.xml

shell ls /var/tmp/netscaler/portal/templates

shell ls /var/vpn/bookmark/*.xml

Initiating a ls command for those directories return the following:

shell ls /netscaler/portal/templates/*.xml

image

> shell ls /netscaler/portal/templates/*.xml

/netscaler/portal/templates/02a81b35.xml

/netscaler/portal/templates/04c86902.xml

/netscaler/portal/templates/06fb0cee.xml

/netscaler/portal/templates/0bc583f8.xml

/netscaler/portal/templates/17EOk.xml

/netscaler/portal/templates/1Z5g92Aky.xml

/netscaler/portal/templates/1d9363f1.xml

/netscaler/portal/templates/2YMBK.xml

/netscaler/portal/templates/3QJW6.xml

/netscaler/portal/templates/3YDd6gheCtB8mrJL2zfyO4WKl7sG0MQa.xml

/netscaler/portal/templates/473ce5e1.xml

/netscaler/portal/templates/48ZwobXq0yjVtU3GeC2a95kBfAHNu1Qz.xml

/netscaler/portal/templates/4GgZh.xml

/netscaler/portal/templates/53963742.xml

/netscaler/portal/templates/5BdZA.xml

/netscaler/portal/templates/5qIJsMV1x9aUGzKirLuNtRYX032QjnvS.xml

/netscaler/portal/templates/5t4ls.xml

/netscaler/portal/templates/64c2192d.xml

/netscaler/portal/templates/6fef6135.xml

/netscaler/portal/templates/77bd07f9.xml

/netscaler/portal/templates/7BA1C.xml

/netscaler/portal/templates/7ESdi.xml

/netscaler/portal/templates/8HtqLVavu.xml

/netscaler/portal/templates/8W7Cgvyck.xml

/netscaler/portal/templates/8cc4d5fb.xml

/netscaler/portal/templates/9HS1qhTa5cyMXYjeIdbV06LQum3nlxB2.xml

/netscaler/portal/templates/BbMZ6.xml

/netscaler/portal/templates/CHQsB.xml

/netscaler/portal/templates/E9TG5.xml

/netscaler/portal/templates/FLQBEv6FQfjjxFfjn2dSRjGHf8WjgpN6.xml

/netscaler/portal/templates/FUcrY.xml

/netscaler/portal/templates/FnMd4HqxlN0Tar9eX3sYD1L8OtPbA5gG.xml

/netscaler/portal/templates/GWwH1.xml

/netscaler/portal/templates/GbRqm.xml

/netscaler/portal/templates/GgJlo.xml

/netscaler/portal/templates/GiDNJIaYtW7EubqBCLF3rHKodsQ16wRc.xml

/netscaler/portal/templates/H8yeh.xml

/netscaler/portal/templates/H9Qwv12zTE3bUcVZMK7Wga5hBfJGxiu0.xml

/netscaler/portal/templates/HitNA.xml

/netscaler/portal/templates/JTQ5sTyzWXWdQRLHrhWukGTgvAHX8Be9.xml

/netscaler/portal/templates/MAkw4mge4.xml

/netscaler/portal/templates/NpWPYQq2DCndZfFVix74PB6owv0QrOUp.xml

/netscaler/portal/templates/O0aAJQSycUFHDsk32gflC98iTK6bRpm1.xml

/netscaler/portal/templates/O9Bon2fAzapuXry7Q0JM41LTKS8l6DmR.xml

/netscaler/portal/templates/OhjZf.xml

/netscaler/portal/templates/R8p9l.xml

/netscaler/portal/templates/RWM4aJ9GhcvSTZLxAreg0qOo5fsCuz8y.xml

/netscaler/portal/templates/TlkhDg4uyEWRHX1fxV2t9AJ3OM0iKmYz.xml

/netscaler/portal/templates/UHKl9QzvkTpJi34wqWORhGN0bDYsCPX1.xml

/netscaler/portal/templates/X3h40kvVpMxjYIKbqlwsa5eLtuNTf281.xml

/netscaler/portal/templates/YOrPCVxZizJjLauTmRyoWUf5Xg1F4G6E.xml

/netscaler/portal/templates/YP68c.xml

/netscaler/portal/templates/YjpRX.xml

/netscaler/portal/templates/a436n.xml

/netscaler/portal/templates/ad140234.xml

/netscaler/portal/templates/aenp7.xml

/netscaler/portal/templates/bf327bcc.xml

/netscaler/portal/templates/c1ktNfwQ76mDi9y2bVEY4Ze8qJajSCvs.xml

/netscaler/portal/templates/cUAwMikvxEePnrOV0bs6NRzT28lt71jq.xml

/netscaler/portal/templates/cVSMUbaDF4TeAL8C9tpmGf15PNR6sd2W.xml

/netscaler/portal/templates/cs0lxv5ppnrxebo4.xml

/netscaler/portal/templates/czqhO.xml

/netscaler/portal/templates/d12affe3.xml

/netscaler/portal/templates/dddd[%template.new({'BLOCK'='print`id`'})%].xml

/netscaler/portal/templates/dqe82VQf8.xml

/netscaler/portal/templates/eMlUc.xml

/netscaler/portal/templates/f81961c6.xml

/netscaler/portal/templates/fhLCc.xml

/netscaler/portal/templates/hK3Sn.xml

/netscaler/portal/templates/hjXNi.xml

/netscaler/portal/templates/jLBscdX0Q.xml

/netscaler/portal/templates/k42bayNsu1hxf7pZ98UwWPviVYDOAmIM.xml

/netscaler/portal/templates/niphc4KmTRS9E0XkbAo5eBQWMtsYF3zG.xml

/netscaler/portal/templates/oQZFe.xml

/netscaler/portal/templates/orfuxqzupb.xml

/netscaler/portal/templates/pswIV.xml

/netscaler/portal/templates/qTvCLFw9HnfAzFycKGJSdQ9XxOnCityF.xml

/netscaler/portal/templates/qVIgn5Mljz7as2Q6PrkDHBbmoRuhx13v.xml

/netscaler/portal/templates/qrfgO.xml

/netscaler/portal/templates/qui4o.xml

/netscaler/portal/templates/ryMG7.xml

/netscaler/portal/templates/sDZIbl7ZNsL1HVBCzowAq3oVLY2aiexj.xml

/netscaler/portal/templates/somuniquestr.xml

/netscaler/portal/templates/tcHzZ6um0MrlEVXT8dyAwBOx3kYs2go7.xml

/netscaler/portal/templates/v905U.xml

/netscaler/portal/templates/wTJIb.xml

/netscaler/portal/templates/xfWjDEZ0K4lSaRgHvecQ2987GihIYTUX.xml

/netscaler/portal/templates/yR39d.xml

/netscaler/portal/templates/yVStWwCFy9BDXBxjIGvCk3h67Gx4Zm8E.xml

/netscaler/portal/templates/yndBMhT2L.xml

Done

image

The results from above indicate that there are plenty of xml files that have filenames suggesting the Citrix ADC has been compromised.

Proceeding to execute the following:

shell ls /var/tmp/netscaler/portal/templates

image

… reveals that the /var/tmp/netscaler/portal/templates directory appears to be fine.

Executing the last command:

shell ls /var/vpn/bookmark/*.xml

image

> shell ls /var/vpn/bookmark/*.xml

/var/vpn/bookmark/1Fx3kynwZ.xml

/var/vpn/bookmark/OqU8uS1bW.xml

/var/vpn/bookmark/bNRrtwSjF.xml

/var/vpn/bookmark/cx4yylzvg.xml

/var/vpn/bookmark/fKchklbXP.xml

/var/vpn/bookmark/fSdeA2hEn.xml

/var/vpn/bookmark/lztbVXgWEKqmoQLWPjbM.xml

/var/vpn/bookmark/nsroot.xml

/var/vpn/bookmark/pwnpzi1337.xml

/var/vpn/bookmark/testtest.xml

/var/vpn/bookmark/z8P0sSAGK.xml

Done

>

image

… reveals that the /var/vpn/bookmark/ directory also displays the same type of files and suggests the ADC has been compromised.

Reviewing Apache Log Files

Executing the first 3 commands to review the Apache logs does not reveal any results:

shell cat /var/log/httpaccess.log | grep vpns | grep xml

shell cat /var/log/httpaccess.log | grep "/\.\./"

shell gzcat /var/log/httpaccess.log.*.gz | grep vpns | grep xml

However, execute the following command does:

shell gzcat /var/log/httpaccess.log.*.gz | grep "/\.\./"

image

The suspicious xml files are notably present in the results above.

Executing the commands to determine whether there are any scheduled tasks to maintain access after patches does not reveal any jobs that a compromised Citrix ADC would have:

shell cat /etc/crontab

shell crontab -l -u nobody

image

Checking for Backdoor Scripts

Executing the following two commands to look for backdoor scripts:

shell ps -aux | grep python

shell ps -aux | grep perl

… reveal the following:

> shell ps -aux | grep python

nobody 82094 0.0 0.1 60980 1852 ?? Ss 28Jan20 0:37.76 /var/python/bin/python2 -c -c (python2.7)

root 36830 0.0 0.1 9096 912 0 RL+ 1:51PM 0:00.00 grep python

> shell ps -aux | grep perl

nobody 53632 100.0 0.0 5804 272 ?? R Sun05AM 3176:31.20 /tmp/.perl

nobody 77911 0.0 0.0 5804 412 ?? I 28Jan20 0:07.52 /tmp/.perl

root 36832 0.0 0.1 9096 1024 0 S+ 1:51PM 0:00.00 grep perl

>

image

The presence of nobody scripts is a red flag that there are backdoor scripts currently on the NetScaler and we can see that there are 3 of them listed.

Checking for Cyrpto Mining

Executing the following command to list the current running processes:

shell top -n 10

… reveal the following:

> shell top -n 10

last pid: 36937; load averages: 2.00, 2.00, 2.00 up 123+23:58:22 13:53:29

77 processes: 3 running, 71 sleeping, 3 zombie

Mem: 27M Active, 3504K Inact, 1487M Wired, 11M Cache, 169M Buf, 2528K Free

Swap: 4198M Total, 299M Used, 3899M Free, 7% Inuse

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND

1174 root 1 44 r0 1210M 1210M CPU1 1 2976.0 100.00% NSPPE-00

53632 nobody 1 119 0 5804K 272K RUN 0 53.0H 100.00% .perl

1176 root 1 44 0 83392K 5668K kqread 0 30.7H 0.00% nsnetsvc

1297 root 1 44 0 37128K 3232K kqread 0 387:11 0.00% nsrised

1434 root 1 44 0 54752K 444K nanslp 0 316:34 0.00% nsprofmon

994 root 1 44 0 76720K 1356K select 0 112:35 0.00% vmtoolsd

1199 root 1 44 0 99M 11940K kqread 0 111:55 0.00% nsaggregatord

1331 root 2 76 0 70184K 13168K ucond 0 41:09 0.00% nscopo

25 root 1 44 0 29784K 29872K kqread 0 37:05 0.00% pitboss

1314 root 1 44 0 63928K 9820K nanslp 0 30:48 0.00% nscollect

Done

>

image

The presence of the nobody account running a perl script that is consuming 100.00% of the WCPU suggests malicious activity currently running on the Citrix ADC.

Reviewing the Bash Logs

Executing the following 2 commands to review the bash logs:

shell cat /var/log/bash.log | grep nobody

shell gzcat /var/log/bash.*.gz | grep nobody

… does not reveal the nobody account present:

image

The following 2 commands display the full output of both logs allowing you to manually comb through the entries but the results from all the tests above are sufficient enough to deem the appliance as being compromised:

shell cat /var/log/httperror.log

shell gzcat /var/log/httperror.log.*.gz

image

Remediation

This particular environment did not have a backup pre-December 2019 for me to restore so the only viable option that was available is to redeploy the VPX appliance and restore the configuration. The latest backup I could find dates back to 2 years ago but given that there has not been any change to the Citrix ADC, I decided to download the same build, redeploy, restore backup configuration, upgrade appliance and swap out the existing one.

Prior to completing the swap it is also important to perform the following:

  1. Review all of the servers that the Citrix ADC interacts with to determine whether they have been breached. These services could include domain controllers, Citrix StoreFronts, Citrix Delivery Controllers, Licensing servers and others.
  2. Update all of the passwords both local to the ADC and credentials used to interact with other services such as LDAP lookups against Domain Controllers.
  3. Force password changes for all users who have logged onto the ADC to access resources. As most Citrix ADCs allow Active Directory integration, it is important to force password changes for AD accounts.
  4. Revoke and renew all SSL certificates on the Citrix ADC. Use the list of files on the ADC’s /nsconfig/ssl/ directory as the hackers who have breached the appliance would have been able to retrieve the files in there.
  5. Ensure that the version of the Citrix ADC is not vulnerable to the CVE-2019-19781.
  6. Once the above have been completed, proceed to place the Citrix ADC back on the internet and run the vulnerability scan against it.

Restoring Configuration from Backup

As indicated above, the backup I had to work with was a few versions back from the NS12.1 51.19.nc that the appliance was currently running:

image

Citrix’s official Backup and Restore documentation (https://docs.citrix.com/en-us/netscaler/12/system/basic-operations/backup-restore-netscaler-appliance.html) indicates we can use the same or newer build but just to avoid any unintended issues, I downloaded the same build and redeployed the appliance as the running version would need to be upgraded anyways. The first step for the restore is to get the backup file uploaded and since a restore can only be performed via the GUI, I will use it to upload the configuration backup.

Begin by navigating to System > Backup and Restore then click on the Backup button:

image

Select the Add button:

image

Click on the down arrow beside Choose File and select Local as the source then select the backup TGZ file:

image

**Note that it is important to choose the Local option because the Appliance option would allow you to upload but it will not be displayed as a backup in the options to restore.

With the backup uploaded, proceed to restore it from the TGZ package:

image

Note that there is no need to have backup the appliance since it’s a new deployment so select the Skip Backup checkbox:

image

The restore should be fairly quick and will display a Restore Successful at the bottom right hand corner of the window:

image

The restored configuration will not be active until you restart the appliance and it is important to not select Save configuration if you restart it from the GUI or else the empty configuration will overwrite what you have just restored.

Also note that the IP address configuration will match the original appliance so if you have left the other one online for forensics but prevented it from accessing the internet then you should either change its IP address to avoid conflict or move it to another network.

image

If you’re staging the new appliance and have not given it network connectivity then the following are useful commands to verify the restored configuration:

  • show ip – list the NSIP, SNIP, SIP, MIP, VIP IP addresses
  • show lb vserver – displays the Load Balancing Virtual Servers details
  • show lb vserver -summary -fullvalues – displays the list of Load Balancing Virtual Servers without truncating names (use | grep to add specific values to list such as UP or DOWN)
  • show ssl certKey – to list all certificates on the appliance
  • show ssl certLink – to list the links between Root CA and Subordinate CA certificates

It’s important to note that the previously installed Client certificates will be present but you should deem these as being compromised so proceed to revoke, reissue and reinstall those certificates.

Thursday, January 23, 2020

Microsoft Outlook fails to authenticate with Office 365 configured with DUO MFA

I was recently contacted to troubleshoot an issue where a user’s Outlook was unable to connect to Office 365 after a password change over the evening and while I found various forum posts describing the issue, the suggested solution required a slight change to work in the environment I was dealing with so this post serves to describe what I encountered and the solution.

Environment

  • Office 365 is configured for MFA with the product named DUO, which is now owned by Cisco
  • Users are automatically redirected to a Citrix NetScaler configured with DUO MFA authentication webpage https://aaa.domain.com when they attempt to log into Office 365 either via outlook.com/domain, outlook.office365.com or login.microsoftonline.com

Problem

A user is no longer able to connect to Office 365 with their Outlook client after the following actions:

  1. Her password was going to expire so she changed it at the end of the day
  2. She logged off after the password change and went home for the evening
  3. She arrived at the office this morning, logged into her laptop and noticed that her Outlook no longer connected

You’ve confirmed that their password was updated within Azure Active Directory (AAD) yesterday evening:

image

You’ve confirmed that the cached credentials were cleared:

clip_image001

You proceed to connect to their desktop/laptop and notice that her Outlook had the status displayed as:

Trying to connect…

clip_image002

Clicking on the Trying to connect… button would briefly bring up the what appears to be authentication prompt for Office 365:

image

The window is displayed for about 3 seconds and disappears.

Thinking that this may be an authentication issue, you try having the user authenticate via the Office Account sign in page but it does not resolve the issue:

image

You perform a bit of Googling on the internet and find the following two forum posts:

https://techcommunity.microsoft.com/t5/outlook/outlook-password-prompt-disappears-quickly/m-p/793317

https://superuser.com/questions/1349327/outlook-needs-password-but-dialog-box-disappears

The discussion indicates that the following two registry keys be added:

HKEY_CURRENT_USER\Software\Microsoft\Office\16.0\Common\Identity

DWORD: DisableADALatopWAMOverride

Value: 1

DWORD: EnableADAL

Value: 0

You proceed to add these two DWORDs to the registry:

clip_image002[4]

Adding these two keys managed to display this classic authentication prompt when Outlook is restarted:

image

However, logging in with her new credentials did not correct the problem as the status would continue to be stuck at Trying to connect…:

Solution

What does end up working for this issue was when the following registry key was deleted:

HKEY_CURRENT_USER\Software\Microsoft\Office\16.0\Common\Identity

DWORD: EnableADAL

Value: 0

… but with the following one configured:

HKEY_CURRENT_USER\Software\Microsoft\Office\16.0\Common\Identity

DWORD: DisableADALatopWAMOverride

Value: 1

clip_image002[6]

With the above setup, the aaa.domain.com Citrix NetScaler page loaded correctly when Outlook is started:

image

Having the user enter their credentials got Outlook to finally connect to Office 365:

clip_image002[8]

What should also be noted is that if the following DWORD:

HKEY_CURRENT_USER\Software\Microsoft\Office\16.0\Common\Identity

DWORD: EnableADAL

Value: 0

… is re-added back in after Outlook has connected then it will cause it to fail to connect again so this DWORD should be left unadded.