How to configure Citrix ADC / NetScaler to forward client Source IP to Exchange Server 2019 / 2016 or any IIS application

Those who have worked with load balancers for applications will know that it can be a pain to troubleshoot issue where the source IP address is required because from the application’s perspective, all incoming connections have the originating IP address as the load balancer. With Citrix ADC / NetScalers, there are several methods in achieving this such as using the X-Forwarded-For header to include the source client IP address (this only works with HTTP and SSL services) or configuring direct server return (DSR) mode to allow the server to respond to clients directly by using a return path that does not flow through the Citrix ADC appliance. There are advantages and disadvantages for each method but for the purpose of this post, I will demonstrate how to configure Exchange Server 2019 (or any IIS application) to receive the source client IP with the X-Forwarded-For header.

The Scenario

Let’s assume that you have a user who is continuously locked out of their account and you have identified the event to take place on an on-premise Exchange server as you can see event ID 4625 Audit Failure events in the Security log as shown in the screenshot below:

The Exchange server is placed behind Citrix ADC / NetScalers and therefore have the IP address 172.16.5.90 of the load balancer for the Source Network Address field in the event.

Proceeding to navigate into the IIS logs on the Exchange server in the W3SVC1 folder located in the C:\inetpub\logs\LogFiles\ directory:

… and opening the logs show only the source IP of the Citrix ADC / NetScaler:

Configuration IIS on Exchange Server to log the X-Forwarded-For request header

The first step to log the source IP address is to configure IIS on the Exchange server to log the X-Forwarded-For request header that is passed from the Citrix ADC / NetScaler load balancer. The following TechNet Blog does a fantastic job of demonstrating the process:

How to use X-Forwarded-For header to log actual client IP address?
https://techcommunity.microsoft.com/t5/iis-support-blog/how-to-use-x-forwarded-for-header-to-log-actual-client-ip/ba-p/873115

Below is a demonstration with an Exchange 2019 server on Windows Server 2019 and IIS version 10.0.17763.1:

Begin by launching Internet Information (IIS) Manager, navigate to either the Server node or one of the websites and then open on Logging:

We will add the X-Forwarded-For field by clicking on Select Fields beside the W3C Format dropdown menu:

Proceed to click on Add Field and add the X-Forwarded-For text as the Field Name and Source, with the Source Type as Request Header:

Note how the X-Forwarded-For is added as a Custom Field:

Apply the changes:

**Note that configuring the above on one site automatically applies it to the other sites.

Now navigate back to the IIS log files and open the latest log file and confirm that the X-Forwarded-For field is added as a header:

The following is a side by side comparison where the log on the top has the X-Forwarded-For custom field added and the bottom does not:

Configure the Citrix ADC / NetScaler to forward client source IP as X-Forwarded-For

With the IIS server configured to receive the custom X-Forwarded-For field, proceed to log into the Citrix ADC / NetScaler, navigate to Traffic Management > Load Balancing > Service Groups or Services:

For the purpose of this example, we will be configuring all of the Exchange service groups to forward the client source IP address as X-Forwarded-For (owa, activesync, rpc, ews, Autodiscover, oab, mapi, ecp).

Open the properties of the load balancing service group or service, navigate to the Settings area and click on the edit icon:

Enable the Insert Client IP Header and type in the X-Forwarded-For string for the Header text box:

Click OK to save the settings and proceed to save the settings by clicking Done.

Repeat for the rest of the Load Balancing Service Group by using the GUI or the CLI command:

set service <name> -CIP <Value> <cipHeader>

Here are the commands for each Exchange service:

set serviceGroup SVG_EX2019_owa -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_activesync -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_rpc -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_ews -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_autodiscover -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_oab -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_mapi -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_ecp -cip enabled X-Forwarded-For

Testing the configuration by verifying source IP address in IIS Logs

Switching back to the Exchange Server and navigating to the IIS logs should now have the latest log reveal a value for the X-Forwarded-For field. Below is a screenshot of the log before the configuration of the Citrix ADC / NetScaler:

Below is a screenshot after the change with an IP address added to the end of each connection with the source IP:

Hope this helps anyone looking for a way to log the originating source IP address of client requests on IIS that is load balanced by a Citrix ADC / NetScaler.

Attempting to log into a Citrix ADC / NetScaler published Citrix Gateway displays the prompt: "Cannot Complete Your Request"

Problem

You’ve completed configuring two Citrix StoreFront configured as a Load Balancing Virtual Server that is published by a Citrix ADC / NetScaler Gateway but noticed that the successfully logging on would continuously display the prompt:

Cannot complete your request.

Attempting to directly access the Load Balancing Virtual Server representing the StoreFront servers exhibits the same issue.

The following article’s recommendations does not appear to correct the issue:

Common Resolutions to “Cannot Complete Your Request” Error when connecting directly to StoreFront Server
https://support.citrix.com/article/CTX207162

Solution

The Cannot Complete Your Request prompt can be caused by many reasons but one of the more common reason I’ve come across is if the Load Balancing Virtual Server representing the StoreFront servers does not have Persistence configured:

To correct the issue, add the Persistence setting to the Load Balancing Virtual Server representing the StoreFront servers and configure

Select Persistence Type: SOURCEIP

Time-out (mins): the desired duration

Leave the rest as defaults.

Applying the settings should correct the issue:

Load Balancing Exchange Server 2019 with Citrix ADC / NetScaler

One of the most common questions I am asked by colleagues and clients is in regards to how they can publish Exchange Servers behind a Citrix ADC / NetScaler to provide load balancing for all services such as OWA, ActiveSync, RPC, EWS, AutoDiscover, MAPI, and ECP BUT NOT SMTP. I would like to further emphasize that SMTP (mail traffic) is excluded for this because I never liked suggesting load balancers to be used for this service and the reason is that the true originating IP of the traffic will be masked by the Citrix ADC / Netscaler. This effectively means that if you won’t be able to restrict IPs allowed for relaying via the Exchange receive connectors or troubleshoot an issue for a relaying client. An workaround for this solution could be to use a DSR (Direct Server Reply) configuration but limitations such as having the Exchange server and Citrix ADC on the same subnet is likely not something that will be possible for more environments. Lastly, a misconfiguration of the Citrix ADC / NetScaler (lack of filtering based on source IP address) could potentially create a open relay on the Exchange server because from the Exchange server perspective, it would relay any SMTP traffic sent by the Citrix ADC / NetScaler.

With the above said, this blog post will serve to provide the CLI commands to configure the Citrix ADC / NetScaler with the appropriate server objects, monitors, service groups, load balancing virtual server, and content switching server. Note that it is possible to not use a content switching server and simply use a load balancing virtual server to publish Exchange servers but you would not be able to independently redirect requests to specific services to another server if those a service is not available. An example to this is if ExchangeServer1 just has /owa unavailable, the content switching server will redirect requests to ExchangeServer2 but will continue to send /ews to ExchangeServer1 if it is available. Using just a load balancing server will deem ExchangeServer1 as down and redirect all traffic to another server.

The version of Exchange Server 2019 and Citrix ADC / NetScaler used for this post are as follows:

Exchange Server 2019: Version 15.2 ‎(Build 721.2)‎

Citrix ADC / NetScaler: NS13.0 67.39.nc

Note that certain version of Citrix ADC / NetScaler appear to exhibit issues with monitoring Exchange services as outlined one of my previous posts:

Citrix ADC / NetScaler monitors for Exchange 2019 fails with: "Failure - Time out during SSL handshake stage"
http://terenceluk.blogspot.com/2020/12/citrix-adc-netscaler-monitors-for.html

Prerequisites

• One private IP addresses allocated for the VIP of the Content Switching server
• Imported SSL certificate to secure traffic
• Firewall rule that allows the Citrix NSIP to the Exchange servers on port 443
• Firewall rule that allows internet traffic via the firewall to the VIP of the Content Switching server on port 443 and 80

Step #1 – Enable Features

Enable the features below by executing the following command: enable ns feature CS,RESPONDER,LB,SSL

• Content Switching
• Responder
• Load Balancing
• SSL

Step #2 – Import SSL Certificate

Either import the SSL certificate currently used on the Exchange servers to secure 443 traffic:

Importing PFX certificate from Microsoft Windows Server into Citrix NetScaler VPX
http://terenceluk.blogspot.com/2016/05/importing-pfx-certificate-from.html

Or request a new one if this is a greenfield deployment. We will need to assign the SSL certificate to the load balancing virtual server and the content switching server.

Make sure the issuing Root and Intermediate certificate chain is imported and linked as well.

Step #3 – Create the Server Objects

Create the server objects with the following command and replace the name and IP as it reflects your environment:

add server ExchServer1.contoso.com 172.16.1.81

add server ExchServer2.contoso.com 172.16.1.82

Step #4 – Create Exchange Service Monitors

The monitors we’ll be creating for are as follows:

1. OWA
2. ActiveSync
3. RPC
4. EWS
5. AutoDiscover
6. OAB
7. MAPI
8. ECP

Execute the following CLI commands to create the monitors:

add lb monitor mon_exch_owa HTTP-ECV -send "GET /owa/healthcheck.htm" -recv 200 -LRTM DISABLED -destPort 443 -secure YES

add lb monitor mon_exch_activesync HTTP-ECV -send "GET /Microsoft-Server-ActiveSync/healthcheck.htm" -recv 200 -LRTM DISABLED -secure YES

add lb monitor mon_exch_rpc HTTP-ECV -send "GET /rpc/healthcheck.htm" -recv 200 -LRTM DISABLED -secure YES

add lb monitor mon_exch_ews HTTP-ECV -send "GET /ews/healthcheck.htm" -recv 200 -LRTM DISABLED -secure YES

add lb monitor mon_exch_autodiscover HTTP-ECV -send "GET /Autodiscover/healthcheck.htm" -recv 200 -LRTM DISABLED -secure YES

add lb monitor mon_exch_oab HTTP-ECV -send "GET /oab/healthcheck.htm" -recv 200 -LRTM DISABLED -secure YES

add lb monitor mon_exch_mapi HTTP-ECV -send "GET /mapi/healthcheck.htm" -recv 200 -LRTM DISABLED -secure YES

add lb monitor mon_exch_ecp HTTP-ECV -send "GET /ecp/healthcheck.htm" -recv 200 -LRTM DISABLED -secure YES

Step #5 – Create Service Groups and Bind Servers

We’ll need to create a service group for each of the services we created a monitor for with the following CLI commands:

add serviceGroup SVG_EX2019_owa SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add serviceGroup SVG_EX2019_activesync SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add serviceGroup SVG_EX2019_rpc SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add serviceGroup SVG_EX2019_ews SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add serviceGroup SVG_EX2019_autodisover SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add serviceGroup SVG_EX2019_oab SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add serviceGroup SVG_EX2019_mapi SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add serviceGroup SVG_EX2019_ecp SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

With the service groups created, proceed to bind the Exchange servers and the monitors into them with the following CLI command:

bind serviceGroup SVG_EX2019_owa ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_owa ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_owa -monitorName mon_exch_owa

bind serviceGroup SVG_EX2019_activesync ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_activesync ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_activesync -monitorName mon_exch_activesync

bind serviceGroup SVG_EX2019_rpc ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_rpc ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_rpc -monitorName mon_exch_rpc

bind serviceGroup SVG_EX2019_ews ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_ews ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_ews -monitorName mon_exch_ews

bind serviceGroup SVG_EX2019_autodisover ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_autodisover ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_autodisover -monitorName mon_exch_autodiscover

bind serviceGroup SVG_EX2019_oab ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_oab ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_oab -monitorName mon_exch_oab

bind serviceGroup SVG_EX2019_mapi ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_mapi ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_mapi -monitorName mon_exch_mapi

bind serviceGroup SVG_EX2019_ecp ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_ecp ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_ecp -monitorName mon_exch_ecp

Step #6 – Create Load Balancing Virtual Server, Bind Service Groups and Bind SSL Certificate

A Load Balancing Virtual Server object with an IP Address Type configured as Non Addressable is required to be created for each service:

Proceed to create these load balancing virtual servers with the following CLI commands:

add lb vserver mail.contoso.com_owa-LBVS SSL 0.0.0.0 0 -persistenceType NONE -cltTimeout 180

add lb vserver mail.contoso.com_activesync-LBVS SSL 0.0.0.0 0 -persistenceType SRCIPDESTIP -cltTimeout 180

add lb vserver mail.contoso.com_rpc-LBVS SSL 0.0.0.0 0 -persistenceType SOURCEIP -timeout 30 -cltTimeout 180

add lb vserver mail.contoso.com_ews-LBVS SSL 0.0.0.0 0 -persistenceType NONE -cltTimeout 180

add lb vserver mail.contoso.com_autodiscover-LBVS SSL 0.0.0.0 0 -persistenceType SOURCEIP -timeout 30 -cltTimeout 180

add lb vserver mail.contoso.com_oab-LBVS SSL 0.0.0.0 0 -persistenceType NONE -cltTimeout 180

add lb vserver mail.contoso.com_mapi-LBVS SSL 0.0.0.0 0 -persistenceType SOURCEIP -timeout 30 -cltTimeout 180

add lb vserver mail.contoso.com_ecp-LBVS SSL 0.0.0.0 0 -persistenceType NONE -cltTimeout 180

With the load balancing virtual servers created, proceed to bind the service groups containing the Exchange Servers to them:

bind lb vserver mail.contoso.com_owa-LBVS SVG_EX2019_owa

bind lb vserver mail.contoso.com_activesync-LBVS SVG_EX2019_activesync

bind lb vserver mail.contoso.com_rpc-LBVS SVG_EX2019_rpc

bind lb vserver mail.contoso.com_ews-LBVS SVG_EX2019_ews

bind lb vserver mail.contoso.com_autodiscover-LBVS SVG_EX2019_autodisover

bind lb vserver mail.contoso.com_oab-LBVS SVG_EX2019_oab

bind lb vserver mail.contoso.com_mapi-LBVS SVG_EX2019_mapi

bind lb vserver mail.contoso.com_ecp-LBVS SVG_EX2019_ecp

Lastly, bind the imported SSL certificate (Step #2) to the load balancing virtual server with the following commands:

bind ssl vserver mail.contoso.com_owa-LBVS -certkeyName wildcard-contoso-exp-mar-2021

bind ssl vserver mail.contoso.com_activesync-LBVS -certkeyName wildcard-contoso-exp-mar-2021

bind ssl vserver mail.contoso.com_rpc-LBVS -certkeyName wildcard-contoso-exp-mar-2021

bind ssl vserver mail.contoso.com_ews-LBVS -certkeyName wildcard-contoso-exp-mar-2021

bind ssl vserver mail.contoso.com_autodiscover-LBVS -certkeyName wildcard-contoso-exp-mar-2021

bind ssl vserver mail.contoso.com_oab-LBVS -certkeyName wildcard-contoso-exp-mar-2021

bind ssl vserver mail.contoso.com_mapi-LBVS -certkeyName wildcard-contoso-exp-mar-2021

bind ssl vserver mail.contoso.com_ecp-LBVS -certkeyName wildcard-contoso-exp-mar-2021

Step #7 – Create a Content Switching Server, Bind SSL Certificate, Create the Content Switching Policies, Bind the Policies to the Content Switching Server

With all the dependent components created and configured, proceed to create the content switching server, which will have the pre-allocated VIP assigned and configured to direct requests to the Exchange services to a server that has a healthy status:

add cs vserver mail.contoso.com_http-CS HTTP 172.16.5.62 80 -cltTimeout 180 -persistenceType NONE

add cs vserver mail.contoso.com_ssl-CS SSL 172.16.5.62 443 -cltTimeout 180 -persistenceType NONE

Bind the same SSL certificate used for the load balancing virtual server to the SSL content switching server:

bind ssl vserver mail.contoso.com_ssl-CS -certkeyName wildcard-contoso-exp-mar-2021

Create the content switching policies that represent each of the services being published by the load balancing virtual servers:

add cs action cs_act_exch2019_owa -targetLBVserver mail.contoso.com_owa-LBVS

add cs action cs_act_exch2019_ews -targetLBVserver mail.contoso.com_ews-LBVS

add cs action cs_act_exch2019_autodiscover -targetLBVserver mail.contoso.com_autodiscover-LBVS

add cs action cs_act_exch2019_activesync -targetLBVserver mail.contoso.com_activesync-LBVS

add cs action cs_act_exch2019_oab -targetLBVserver mail.contoso.com_oab-LBVS

add cs action cs_act_exch2019_mapi -targetLBVserver mail.contoso.com_mapi-LBVS

add cs action cs_act_exch2019_rpc -targetLBVserver mail.contoso.com_rpc-LBVS

add cs action cs_act_exch2019_ecp -targetLBVserver mail.contoso.com_ecp-LBVS

add cs policy cs_pol_exch2019_owa -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/owa\")" -action cs_act_exch2019_owa

add cs policy cs_pol_exch2019_ews -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/ews\")" -action cs_act_exch2019_ews

add cs policy cs_pol_exch2019_autodiscover -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/autodiscover\")" -action cs_act_exch2019_autodiscover

add cs policy cs_pol_exch2019_activesync -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"Microsoft\")" -action cs_act_exch2019_activesync

add cs policy cs_pol_exch2019_oab -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/oab\")" -action cs_act_exch2019_oab

add cs policy cs_pol_exch2019_mapi -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/mapi\")" -action cs_act_exch2019_mapi

add cs policy cs_pol_exch2019_rpc -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/rpc\")" -action cs_act_exch2019_rpc

add cs policy cs_pol_exch2019_ecp -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/ecp\")" -action cs_act_exch2019_ecp

An additional OWA policy will also need to be added to address a but identified in the following KB:

OWA access via NetScaler is getting stuck at /cgi/selfauth?params= after authentication
https://support.citrix.com/article/CTX209060

add cs policy cs_pol_exch2019_cgi -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/cgi\")" -action cs_act_exch2019_owa

The following policy will redirect requests going to the mail.contoso.com address to mail.contoso.com/owa:

add cs policy cs_pol_exch2019_owa_redirect -rule "HTTP.REQ.HOSTNAME.EQ(\"mail.contoso.com\")" -action cs_act_exch2019_owa

Lastly, proceed to bind the policies created above to the content switching server:

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_owa -priority 110

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_ews -priority 120

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_autodiscover -priority 130

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_activesync -priority 140

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_oab -priority 150

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_mapi -priority 160

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_rpc -priority 170

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_ecp -priority 180

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_cgi -priority 190

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_owa_redirect -priority 200

Step #8 – Create a Responder to redirect http to https

The final step for the configuration is to create a responder to redirect http to https so users typing in http://mail.contoso.com will automatically be redirected to https://mail.contoso.com:

add responder action redirect_http_https_act_owa redirect "\"https://\"+HTTP.REQ.HOSTNAME+\"/owa/\""

add responder policy redirect_http_https_pol_owa "HTTP.REQ.HOSTNAME.CONTAINS(\"mail.contoso.com\")" redirect_http_https_act_owa

bind cs vserver mail.contoso.com_http-CS -policyName redirect_http_https_pol_owa -priority 100 -gotoPriorityExpression END -type REQUEST

The above steps should configuring the required components for load balancing Exchange servers with the Citrix ADC NetScaler. I will include some screenshots of the configuration in case anyone wants to see the configuration in the GUI.

Screenshots

Load Balancing Virtual Servers:

Service Groups:

Content Switching Server:

Citrix ADC / NetScaler monitors for Exchange 2019 fails with: "Failure - Time out during SSL handshake stage"

Problem

You’re attempting to publish / load balance your on-premise Exchange 2019 servers behind a Citrix ADC / NetScaler but notice that the health monitors created to check the health of the services (e.g. https://172.16.1.81/owa/healthcheck.htm) fail with the following error:

Failure - Time out during SSL handshake stage

The rest of the monitors are all reporting the same error:

Further troubleshooting reveals that this is due to the fact that the following server hardening registry keys are added to the Exchange 2019 servers:

Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL

AllowInsecureRenegoClients
REG_DWORD
0

AllowInsecureRenegoServers
REG_DWORD
0

Removing these entries one of the Exchange servers will correct the error allowing the probe to report that the server is up (note that it is partial because the other server still has the registry key):

Various forum discussions on Citrix points adjusting Deny SSL Renegotiation but none of the configuration settings corrected the issue for the environment I worked with:

https://discussions.citrix.com/topic/388325-netscaler-12-rfc-5746-on-backend-bug-limitation/#comment-1975755

https://discussions.citrix.com/topic/401441-basic-load-balancing-for-owa-exchange-2019/page/3/

Solution

After performing extensive troubleshooting but not able to come to a resolution, I decided to upgrade the Citrix ADC / NetScaler from NS13.0 52.24.nc to the latest build available at the time, which was NS13.0 67.39.nc:

NS13.0 52.24.nc

NS13.0 67.39.nc

This ended up resolving the issue with the services being correctly determined as UP:

Hope this helps anyone who might be facing this same issue as there isn’t much material available and the available solutions did not work for me.

Configuring X-Content-Type-Options and Permissions-Policy for Citrix ADC / NetScaler to score A on Security Headers for Exchange OWA

I’ve recently been involved with a few projects involving security vulnerabilities from penetration scans and one of the clients were asked to secure their Exchange OWA portal load balanced behind a Citrix ADC so that 2 of the 6 headers that are identified to be missing from the Security Headers (https://securityheaders.com/) scan are addressed:

• X-Content-Type-Options
• Permissions-Policy

Please refer to my previous blog post for instructions on how to implement the other headers identified to be present in the scan above:

Securing a Citrix ADC (formally known as NetScaler VPX) to score an A rating on Security Headers - March 2020
http://terenceluk.blogspot.com/2020/02/securing-citrix-adc-formally-known-as.html

X-Content-Type-Options

The X-Content-Type-Options header is fairly easy to implement as it is described in Citrix’s knowledge base article:

How to create rewrite policy for content security headers , XSS protection, HSTS, X-Content-Type-Options &amp; Content-Security-Policy.
https://support.citrix.com/article/CTX233095

The following are the CLI commands:

add rewrite action rw_act_insert_Xcontent_header_Policy insert_http_header X-Content-Type-Options "\"nosniff\""

add rewrite policy rw_pol_insert_XContent_Policy TRUE rw_act_insert_Xcontent_header_Policy

bind lb vserver mail.college.bm_external -policyName rw_pol_insert_XContent_Policy -type RESPONSE -priority 120 -gotoPriorityExpression NEXT

The following is the Rewrite Action in the Citrix ADC / NetScaler GUI:

The following is the Rewrite Policy in the Citrix ADC / NetScaler GUI:

With the Rewrite Action and Policy in place, and having the policy bound to the virtual server, the scan should now return the header as being present:

Permissions-Policy

The Permissions-Policy, which replaces the Feature Policy, isn’t as straight forward as Citrix does not have a KB for the implementation of this header. However, Scott Helme provides a very detailed explanation for this header:

Goodbye Feature Policy and hello Permissions Policy!
https://scotthelme.co.uk/goodbye-feature-policy-and-hello-permissions-policy/

The following are the CLI commands to create the Rewrite Action, Policy and bind it to the virtual server:

add rewrite action rw_act_insert_Permissions_Policy insert_http_header Permissions-Policy "\"vibrate=(self), sync-xhr=(self \'https://<owa.domain.com>\')\""

add rewrite policy rw_pol_insert_Permissions_Policy "HTTP.RES.HEADER(\"Permissions-Policy\").EXISTS.NOT" rw_act_insert_Permissions_Policy

bind lb vserver mail.contoso.com_external -policyName rw_pol_insert_Permissions_Policy -type RESPONSE 110 -gotoPriorityExpression NEXT

The following is the Rewrite Action in the Citrix ADC / NetScaler GUI:

The following is the Rewrite Policy in the Citrix ADC / NetScaler GUI:

With the Rewrite Action and Policy in place, and having the policy bound to the virtual server, the scan should now return the header as being present:

**Note that the reason why the score in the screenshot above is A rather than A+ is because the Content-Security-Policy header configured contains the ‘unsafe-inline’ in the script-src directive. I have yet to successfully omit ‘unsafe-inline’ for sites such as a Citrix Gateway or Exchange OWA without causing the login page to fail to load.

Successfully authenticating with Citrix ADC / Netscaler Gateway displays the error: "Http/1.1 Internal Server Error 43531"

I recently ran into an issue with a Citrix ADC / NetScaler NS13.0 36.27.nc after a reboot where the following error is displayed upon successfully authenticating:

Http/1.1 Internal Server Error 43531

The URL displayed ends with /cgi/dlge:

https://workspace.contoso.com/cgi/dlge

No configuration changes have been made for months. I combed through the configuration but could not determine why this error was being thrown so a ticket was opened with Citrix. The engineer went through the configuration and decided to change the Web Interface Address FQDN in the Citrix Gateway Session Profile to use the IP address instead of the DNS of the StoreFront server as shown in the screenshot below, which immediately corrected the issue:

We originally thought that there was something wrong with DNS but a DIG for the storefront.contoso.com URL returned the correct IP address for the Load Balancing Virtual Server that load balanced the two StoreFront servers:

root@CTXNETSCALER# dig storefront.contoso.com

; <<>> DiG 9.10.6 <<>> storefront.contoso.com

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31561

;; flags: qr aa rd ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:

; EDNS: version: 0, flags:; udp: 1280

;; QUESTION SECTION:

;storefront. contoso.com. IN A

;; ANSWER SECTION:

storefront. contoso.com. 3600 IN A 10.0.1.17

;; Query time: 0 msec

;; SERVER: 127.0.0.2#53(127.0.0.2)

;; WHEN: Tue Sep 29 19:25:36 UTC 2020

;; MSG SIZE rcvd: 69

root@CTXNETSCALER#

I haven’t gotten to the root cause of this issue but noticed that there were no recent posts for this error and thought I’d write a post in case someone else encounters this issue. We were told that an upgrade from the current version 13 Build 36.28 to version 13 Build 64.35 would resolve the issue so I will update this post when I determine whether it resolves the issue.