Attempting to log into a Citrix ADC / NetScaler Gateway portal with Duo configured as MFA fails with: "Additional Information Required Please type your response below"

I recently had to perform routine patching of various Citrix ADC / NetScalers to ensure that they are up-to-date with the latest NS13.1 17.42.nc build (released Feb 23, 2022) and one HA pair in particular displayed the following message when the expected Duo MFA authentication prompt was supposed to be displayed:

Additional Information Required

Please type your response below

Attempting to enter your passcode or other information into the prompt leads to an authentication failure.

Searching on the internet displays the following Duo KB:

Why do I see the Citrix Gateway or Netscaler error "Additional Information Required" during Duo authentication?
https://help.duo.com/s/article/3506?language=en_US

… which did not apply for my situation and other forum posts suggest that the theme configured for the gateway was the culprit and that did not apply for this appliance either. What I ended up having to do was downgrade from the updated firmware:

NS13.1 17.42.nc

… back down to:

NS13.1 12.50.nc

Then I opened up a ticket with Duo and the response I received was the following:

Hi Terence,

Thanks for getting back to me.
As per the docs here:
https://duo.com/docs/citrix-netscaler
If you have Citrix Gateway build 12.1-51.16 or later with an "Advanced" or "Premium" edition license, or Citrix Gateway build 13.0-67.x or later with "Standard", "Advanced", or "Premium" edition license, try adding Duo 2FA with nFactor.
After updating past 13, you must use the nfactor version of the integration.
This involves one single change in the proxy.
the line that is:
[radius_server_iframe]
type=citrix_netscaler
will become:
[radius_server_iframe]
type=citrix_netscaler_rfwebui

The way the Citrix ADC / NetScaler is configured to integrate with Duo is as shown in the following Duo document:

Duo for Citrix Gateway Basic Primary Authentication
https://duo.com/docs/citrix-netscaler#:~:text=Log%20in%20to%20the%20Duo,information%20to%20complete%20your%20setup.

I’ve seen other methods such as configuring Duo as:

Duo for Citrix Gateway - Basic Secondary Authentication Instructions
https://duo.com/docs/citrix-netscaler-alt

… and the nFactor method as the support engineer provided but noticed that the nFactor article indicates that it suggests to “try” so I did not realize it was mandatory. I asked the support engineer to clarify this and the response I receive was:

Hi Terence,

Thanks for getting back to me.
You are not wrong, the wording is definitely ambiguous.
However in all the previous cases with this exact issue I have seen, I’m confident in the solution, given I know it works.
I suspect it was worded this way as there is some overlap where both will actually work for various versions/license editions, however given your experience you do not seem to be in the overlap given it is not working.
Please follow through and make the change and I expect the issue will be resolved.
Please let me know if this helps.
Have a great day!
Kind regards,

Raphael

I haven’t actually tried switching over the nFactor yet but will do so and update this post in the future. Hope this helps anyone who may encounter the same issue as I have.

How to configure Citrix ADC / NetScaler to forward client Source IP to Exchange Server 2019 / 2016 or any IIS application

Those who have worked with load balancers for applications will know that it can be a pain to troubleshoot issue where the source IP address is required because from the application’s perspective, all incoming connections have the originating IP address as the load balancer. With Citrix ADC / NetScalers, there are several methods in achieving this such as using the X-Forwarded-For header to include the source client IP address (this only works with HTTP and SSL services) or configuring direct server return (DSR) mode to allow the server to respond to clients directly by using a return path that does not flow through the Citrix ADC appliance. There are advantages and disadvantages for each method but for the purpose of this post, I will demonstrate how to configure Exchange Server 2019 (or any IIS application) to receive the source client IP with the X-Forwarded-For header.

The Scenario

Let’s assume that you have a user who is continuously locked out of their account and you have identified the event to take place on an on-premise Exchange server as you can see event ID 4625 Audit Failure events in the Security log as shown in the screenshot below:

The Exchange server is placed behind Citrix ADC / NetScalers and therefore have the IP address 172.16.5.90 of the load balancer for the Source Network Address field in the event.

Proceeding to navigate into the IIS logs on the Exchange server in the W3SVC1 folder located in the C:\inetpub\logs\LogFiles\ directory:

… and opening the logs show only the source IP of the Citrix ADC / NetScaler:

Configuration IIS on Exchange Server to log the X-Forwarded-For request header

The first step to log the source IP address is to configure IIS on the Exchange server to log the X-Forwarded-For request header that is passed from the Citrix ADC / NetScaler load balancer. The following TechNet Blog does a fantastic job of demonstrating the process:

How to use X-Forwarded-For header to log actual client IP address?
https://techcommunity.microsoft.com/t5/iis-support-blog/how-to-use-x-forwarded-for-header-to-log-actual-client-ip/ba-p/873115

Below is a demonstration with an Exchange 2019 server on Windows Server 2019 and IIS version 10.0.17763.1:

Begin by launching Internet Information (IIS) Manager, navigate to either the Server node or one of the websites and then open on Logging:

We will add the X-Forwarded-For field by clicking on Select Fields beside the W3C Format dropdown menu:

Proceed to click on Add Field and add the X-Forwarded-For text as the Field Name and Source, with the Source Type as Request Header:

Note how the X-Forwarded-For is added as a Custom Field:

Apply the changes:

**Note that configuring the above on one site automatically applies it to the other sites.

Now navigate back to the IIS log files and open the latest log file and confirm that the X-Forwarded-For field is added as a header:

The following is a side by side comparison where the log on the top has the X-Forwarded-For custom field added and the bottom does not:

Configure the Citrix ADC / NetScaler to forward client source IP as X-Forwarded-For

With the IIS server configured to receive the custom X-Forwarded-For field, proceed to log into the Citrix ADC / NetScaler, navigate to Traffic Management > Load Balancing > Service Groups or Services:

For the purpose of this example, we will be configuring all of the Exchange service groups to forward the client source IP address as X-Forwarded-For (owa, activesync, rpc, ews, Autodiscover, oab, mapi, ecp).

Open the properties of the load balancing service group or service, navigate to the Settings area and click on the edit icon:

Enable the Insert Client IP Header and type in the X-Forwarded-For string for the Header text box:

Click OK to save the settings and proceed to save the settings by clicking Done.

Repeat for the rest of the Load Balancing Service Group by using the GUI or the CLI command:

set service <name> -CIP <Value> <cipHeader>

Here are the commands for each Exchange service:

set serviceGroup SVG_EX2019_owa -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_activesync -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_rpc -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_ews -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_autodiscover -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_oab -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_mapi -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_ecp -cip enabled X-Forwarded-For

Testing the configuration by verifying source IP address in IIS Logs

Switching back to the Exchange Server and navigating to the IIS logs should now have the latest log reveal a value for the X-Forwarded-For field. Below is a screenshot of the log before the configuration of the Citrix ADC / NetScaler:

Below is a screenshot after the change with an IP address added to the end of each connection with the source IP:

Hope this helps anyone looking for a way to log the originating source IP address of client requests on IIS that is load balanced by a Citrix ADC / NetScaler.

Logging onto a Citrix ADC / NetScaler hosted on Azure with nsroot fails with the error: "nsnet_connect: No such file or directory"

Problem

You’ve noticed that Citrix ADC / NetScaler hosted on Azure is no longer reachable and attempting to use the Serial console feature in Azure administration portal fails with the following error after entering the password for the nsroot account:

Nsnet_connect: No such file or directory

Solution

One of the possible reasons why the above error is displayed is if the /var directory is full but not being able to log into the appliance means that there is no way of cleaning up the drive. The workaround for this scenario is to try and use the nsrecover account to log in. The default password for this account is usually nsroot:

Once successfully logged in, proceed to use df -h to confirm the issue as shown in the screenshot above then use the following KB to clean up the /var directory:

How to free space on /var directory for logging issues with a Citrix ADC appliance
https://docs.citrix.com/en-us/citrix-adc/current-release/system/troubleshooting-citrix-adc/how-to-free-space-on-var-directory.html

Duo MFA login prompt does not show up after successfully authenticating username and password with Citrix ADC / NetScaler Citrix Gateway

Problem

You’ve completed configuring a Citrix Gateway with Duo MFA as described in the following configuration documentation:

Duo for Citrix Gateway Basic Primary Authentication
https://duo.com/docs/citrix-netscaler

However, you notice that authenticating through the portal does not display the Duo 2FA prompt as expected:

Note how only a field labeled as Password is displayed:

It should be noticed that entering the passcode from the Duo mobile app would successfully authenticate the identity but the following expected Duo login prompt is the desired outcome:

Solution

One of the common reasons why this behavior would be exhibited is if the portal them RFWebUI is selected (this the default theme in later versions of Citrix ADM / NetScalers) and the quick way to see if this is causing an issue is to switch the them to X1 as shown below:

However, the better solution is to refer to the following KB:

Is the Citrix Gateway RFWebUI theme compatible with Duo?
https://help.duo.com/s/article/3755?language=en_US

Which will refer to the complete instructions for information on how to configure the RFWebUI theme with Duo: https://duo.com/docs/citrix-netscaler#configure-the-proxy-for-your-citrix-gateway

Prior to implementing the configuration, verify that the requirements of having the Citrix ADC / NetScaler and Authentication Proxy at a certain version is met:

citrix_netscaler - Use citrix_netscaler when with the Default, Green Bubbles, or X1 themes.

citrix_netscaler_rfwebui - Use citrix_netscaler_rfwebui with the RFWebUI theme. Requires Authentication Proxy v3.1.0 and NS build 12.1-51.16 or later.

Attempting to log into a Citrix ADC / NetScaler published Citrix Gateway displays the prompt: "Cannot Complete Your Request"

Problem

You’ve completed configuring two Citrix StoreFront configured as a Load Balancing Virtual Server that is published by a Citrix ADC / NetScaler Gateway but noticed that the successfully logging on would continuously display the prompt:

Cannot complete your request.

Attempting to directly access the Load Balancing Virtual Server representing the StoreFront servers exhibits the same issue.

The following article’s recommendations does not appear to correct the issue:

Common Resolutions to “Cannot Complete Your Request” Error when connecting directly to StoreFront Server
https://support.citrix.com/article/CTX207162

Solution

The Cannot Complete Your Request prompt can be caused by many reasons but one of the more common reason I’ve come across is if the Load Balancing Virtual Server representing the StoreFront servers does not have Persistence configured:

To correct the issue, add the Persistence setting to the Load Balancing Virtual Server representing the StoreFront servers and configure

Select Persistence Type: SOURCEIP

Time-out (mins): the desired duration

Leave the rest as defaults.

Applying the settings should correct the issue:

Load Balancing Exchange Server 2019 with Citrix ADC / NetScaler

One of the most common questions I am asked by colleagues and clients is in regards to how they can publish Exchange Servers behind a Citrix ADC / NetScaler to provide load balancing for all services such as OWA, ActiveSync, RPC, EWS, AutoDiscover, MAPI, and ECP BUT NOT SMTP. I would like to further emphasize that SMTP (mail traffic) is excluded for this because I never liked suggesting load balancers to be used for this service and the reason is that the true originating IP of the traffic will be masked by the Citrix ADC / Netscaler. This effectively means that if you won’t be able to restrict IPs allowed for relaying via the Exchange receive connectors or troubleshoot an issue for a relaying client. An workaround for this solution could be to use a DSR (Direct Server Reply) configuration but limitations such as having the Exchange server and Citrix ADC on the same subnet is likely not something that will be possible for more environments. Lastly, a misconfiguration of the Citrix ADC / NetScaler (lack of filtering based on source IP address) could potentially create a open relay on the Exchange server because from the Exchange server perspective, it would relay any SMTP traffic sent by the Citrix ADC / NetScaler.

With the above said, this blog post will serve to provide the CLI commands to configure the Citrix ADC / NetScaler with the appropriate server objects, monitors, service groups, load balancing virtual server, and content switching server. Note that it is possible to not use a content switching server and simply use a load balancing virtual server to publish Exchange servers but you would not be able to independently redirect requests to specific services to another server if those a service is not available. An example to this is if ExchangeServer1 just has /owa unavailable, the content switching server will redirect requests to ExchangeServer2 but will continue to send /ews to ExchangeServer1 if it is available. Using just a load balancing server will deem ExchangeServer1 as down and redirect all traffic to another server.

The version of Exchange Server 2019 and Citrix ADC / NetScaler used for this post are as follows:

Exchange Server 2019: Version 15.2 ‎(Build 721.2)‎

Citrix ADC / NetScaler: NS13.0 67.39.nc

Note that certain version of Citrix ADC / NetScaler appear to exhibit issues with monitoring Exchange services as outlined one of my previous posts:

Citrix ADC / NetScaler monitors for Exchange 2019 fails with: "Failure - Time out during SSL handshake stage"
http://terenceluk.blogspot.com/2020/12/citrix-adc-netscaler-monitors-for.html

Prerequisites

• One private IP addresses allocated for the VIP of the Content Switching server
• Imported SSL certificate to secure traffic
• Firewall rule that allows the Citrix NSIP to the Exchange servers on port 443
• Firewall rule that allows internet traffic via the firewall to the VIP of the Content Switching server on port 443 and 80

Step #1 – Enable Features

Enable the features below by executing the following command: enable ns feature CS,RESPONDER,LB,SSL

• Content Switching
• Responder
• Load Balancing
• SSL

Step #2 – Import SSL Certificate

Either import the SSL certificate currently used on the Exchange servers to secure 443 traffic:

Importing PFX certificate from Microsoft Windows Server into Citrix NetScaler VPX
http://terenceluk.blogspot.com/2016/05/importing-pfx-certificate-from.html

Or request a new one if this is a greenfield deployment. We will need to assign the SSL certificate to the load balancing virtual server and the content switching server.

Make sure the issuing Root and Intermediate certificate chain is imported and linked as well.

Step #3 – Create the Server Objects

Create the server objects with the following command and replace the name and IP as it reflects your environment:

add server ExchServer1.contoso.com 172.16.1.81

add server ExchServer2.contoso.com 172.16.1.82

Step #4 – Create Exchange Service Monitors

The monitors we’ll be creating for are as follows:

1. OWA
2. ActiveSync
3. RPC
4. EWS
5. AutoDiscover
6. OAB
7. MAPI
8. ECP

Execute the following CLI commands to create the monitors:

add lb monitor mon_exch_owa HTTP-ECV -send "GET /owa/healthcheck.htm" -recv 200 -LRTM DISABLED -destPort 443 -secure YES

add lb monitor mon_exch_activesync HTTP-ECV -send "GET /Microsoft-Server-ActiveSync/healthcheck.htm" -recv 200 -LRTM DISABLED -secure YES

add lb monitor mon_exch_rpc HTTP-ECV -send "GET /rpc/healthcheck.htm" -recv 200 -LRTM DISABLED -secure YES

add lb monitor mon_exch_ews HTTP-ECV -send "GET /ews/healthcheck.htm" -recv 200 -LRTM DISABLED -secure YES

add lb monitor mon_exch_autodiscover HTTP-ECV -send "GET /Autodiscover/healthcheck.htm" -recv 200 -LRTM DISABLED -secure YES

add lb monitor mon_exch_oab HTTP-ECV -send "GET /oab/healthcheck.htm" -recv 200 -LRTM DISABLED -secure YES

add lb monitor mon_exch_mapi HTTP-ECV -send "GET /mapi/healthcheck.htm" -recv 200 -LRTM DISABLED -secure YES

add lb monitor mon_exch_ecp HTTP-ECV -send "GET /ecp/healthcheck.htm" -recv 200 -LRTM DISABLED -secure YES

Step #5 – Create Service Groups and Bind Servers

We’ll need to create a service group for each of the services we created a monitor for with the following CLI commands:

add serviceGroup SVG_EX2019_owa SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add serviceGroup SVG_EX2019_activesync SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add serviceGroup SVG_EX2019_rpc SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add serviceGroup SVG_EX2019_ews SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add serviceGroup SVG_EX2019_autodisover SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add serviceGroup SVG_EX2019_oab SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add serviceGroup SVG_EX2019_mapi SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add serviceGroup SVG_EX2019_ecp SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

With the service groups created, proceed to bind the Exchange servers and the monitors into them with the following CLI command:

bind serviceGroup SVG_EX2019_owa ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_owa ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_owa -monitorName mon_exch_owa

bind serviceGroup SVG_EX2019_activesync ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_activesync ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_activesync -monitorName mon_exch_activesync

bind serviceGroup SVG_EX2019_rpc ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_rpc ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_rpc -monitorName mon_exch_rpc

bind serviceGroup SVG_EX2019_ews ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_ews ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_ews -monitorName mon_exch_ews

bind serviceGroup SVG_EX2019_autodisover ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_autodisover ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_autodisover -monitorName mon_exch_autodiscover

bind serviceGroup SVG_EX2019_oab ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_oab ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_oab -monitorName mon_exch_oab

bind serviceGroup SVG_EX2019_mapi ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_mapi ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_mapi -monitorName mon_exch_mapi

bind serviceGroup SVG_EX2019_ecp ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_ecp ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_ecp -monitorName mon_exch_ecp

Step #6 – Create Load Balancing Virtual Server, Bind Service Groups and Bind SSL Certificate

A Load Balancing Virtual Server object with an IP Address Type configured as Non Addressable is required to be created for each service:

Proceed to create these load balancing virtual servers with the following CLI commands:

add lb vserver mail.contoso.com_owa-LBVS SSL 0.0.0.0 0 -persistenceType NONE -cltTimeout 180

add lb vserver mail.contoso.com_activesync-LBVS SSL 0.0.0.0 0 -persistenceType SRCIPDESTIP -cltTimeout 180

add lb vserver mail.contoso.com_rpc-LBVS SSL 0.0.0.0 0 -persistenceType SOURCEIP -timeout 30 -cltTimeout 180

add lb vserver mail.contoso.com_ews-LBVS SSL 0.0.0.0 0 -persistenceType NONE -cltTimeout 180

add lb vserver mail.contoso.com_autodiscover-LBVS SSL 0.0.0.0 0 -persistenceType SOURCEIP -timeout 30 -cltTimeout 180

add lb vserver mail.contoso.com_oab-LBVS SSL 0.0.0.0 0 -persistenceType NONE -cltTimeout 180

add lb vserver mail.contoso.com_mapi-LBVS SSL 0.0.0.0 0 -persistenceType SOURCEIP -timeout 30 -cltTimeout 180

add lb vserver mail.contoso.com_ecp-LBVS SSL 0.0.0.0 0 -persistenceType NONE -cltTimeout 180

With the load balancing virtual servers created, proceed to bind the service groups containing the Exchange Servers to them:

bind lb vserver mail.contoso.com_owa-LBVS SVG_EX2019_owa

bind lb vserver mail.contoso.com_activesync-LBVS SVG_EX2019_activesync

bind lb vserver mail.contoso.com_rpc-LBVS SVG_EX2019_rpc

bind lb vserver mail.contoso.com_ews-LBVS SVG_EX2019_ews

bind lb vserver mail.contoso.com_autodiscover-LBVS SVG_EX2019_autodisover

bind lb vserver mail.contoso.com_oab-LBVS SVG_EX2019_oab

bind lb vserver mail.contoso.com_mapi-LBVS SVG_EX2019_mapi

bind lb vserver mail.contoso.com_ecp-LBVS SVG_EX2019_ecp

Lastly, bind the imported SSL certificate (Step #2) to the load balancing virtual server with the following commands:

bind ssl vserver mail.contoso.com_owa-LBVS -certkeyName wildcard-contoso-exp-mar-2021

bind ssl vserver mail.contoso.com_activesync-LBVS -certkeyName wildcard-contoso-exp-mar-2021

bind ssl vserver mail.contoso.com_rpc-LBVS -certkeyName wildcard-contoso-exp-mar-2021

bind ssl vserver mail.contoso.com_ews-LBVS -certkeyName wildcard-contoso-exp-mar-2021

bind ssl vserver mail.contoso.com_autodiscover-LBVS -certkeyName wildcard-contoso-exp-mar-2021

bind ssl vserver mail.contoso.com_oab-LBVS -certkeyName wildcard-contoso-exp-mar-2021

bind ssl vserver mail.contoso.com_mapi-LBVS -certkeyName wildcard-contoso-exp-mar-2021

bind ssl vserver mail.contoso.com_ecp-LBVS -certkeyName wildcard-contoso-exp-mar-2021

Step #7 – Create a Content Switching Server, Bind SSL Certificate, Create the Content Switching Policies, Bind the Policies to the Content Switching Server

With all the dependent components created and configured, proceed to create the content switching server, which will have the pre-allocated VIP assigned and configured to direct requests to the Exchange services to a server that has a healthy status:

add cs vserver mail.contoso.com_http-CS HTTP 172.16.5.62 80 -cltTimeout 180 -persistenceType NONE

add cs vserver mail.contoso.com_ssl-CS SSL 172.16.5.62 443 -cltTimeout 180 -persistenceType NONE

Bind the same SSL certificate used for the load balancing virtual server to the SSL content switching server:

bind ssl vserver mail.contoso.com_ssl-CS -certkeyName wildcard-contoso-exp-mar-2021

Create the content switching policies that represent each of the services being published by the load balancing virtual servers:

add cs action cs_act_exch2019_owa -targetLBVserver mail.contoso.com_owa-LBVS

add cs action cs_act_exch2019_ews -targetLBVserver mail.contoso.com_ews-LBVS

add cs action cs_act_exch2019_autodiscover -targetLBVserver mail.contoso.com_autodiscover-LBVS

add cs action cs_act_exch2019_activesync -targetLBVserver mail.contoso.com_activesync-LBVS

add cs action cs_act_exch2019_oab -targetLBVserver mail.contoso.com_oab-LBVS

add cs action cs_act_exch2019_mapi -targetLBVserver mail.contoso.com_mapi-LBVS

add cs action cs_act_exch2019_rpc -targetLBVserver mail.contoso.com_rpc-LBVS

add cs action cs_act_exch2019_ecp -targetLBVserver mail.contoso.com_ecp-LBVS

add cs policy cs_pol_exch2019_owa -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/owa\")" -action cs_act_exch2019_owa

add cs policy cs_pol_exch2019_ews -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/ews\")" -action cs_act_exch2019_ews

add cs policy cs_pol_exch2019_autodiscover -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/autodiscover\")" -action cs_act_exch2019_autodiscover

add cs policy cs_pol_exch2019_activesync -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"Microsoft\")" -action cs_act_exch2019_activesync

add cs policy cs_pol_exch2019_oab -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/oab\")" -action cs_act_exch2019_oab

add cs policy cs_pol_exch2019_mapi -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/mapi\")" -action cs_act_exch2019_mapi

add cs policy cs_pol_exch2019_rpc -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/rpc\")" -action cs_act_exch2019_rpc

add cs policy cs_pol_exch2019_ecp -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/ecp\")" -action cs_act_exch2019_ecp

An additional OWA policy will also need to be added to address a but identified in the following KB:

OWA access via NetScaler is getting stuck at /cgi/selfauth?params= after authentication
https://support.citrix.com/article/CTX209060

add cs policy cs_pol_exch2019_cgi -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/cgi\")" -action cs_act_exch2019_owa

The following policy will redirect requests going to the mail.contoso.com address to mail.contoso.com/owa:

add cs policy cs_pol_exch2019_owa_redirect -rule "HTTP.REQ.HOSTNAME.EQ(\"mail.contoso.com\")" -action cs_act_exch2019_owa

Lastly, proceed to bind the policies created above to the content switching server:

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_owa -priority 110

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_ews -priority 120

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_autodiscover -priority 130

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_activesync -priority 140

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_oab -priority 150

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_mapi -priority 160

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_rpc -priority 170

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_ecp -priority 180

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_cgi -priority 190

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_owa_redirect -priority 200

Step #8 – Create a Responder to redirect http to https

The final step for the configuration is to create a responder to redirect http to https so users typing in http://mail.contoso.com will automatically be redirected to https://mail.contoso.com:

add responder action redirect_http_https_act_owa redirect "\"https://\"+HTTP.REQ.HOSTNAME+\"/owa/\""

add responder policy redirect_http_https_pol_owa "HTTP.REQ.HOSTNAME.CONTAINS(\"mail.contoso.com\")" redirect_http_https_act_owa

bind cs vserver mail.contoso.com_http-CS -policyName redirect_http_https_pol_owa -priority 100 -gotoPriorityExpression END -type REQUEST

The above steps should configuring the required components for load balancing Exchange servers with the Citrix ADC NetScaler. I will include some screenshots of the configuration in case anyone wants to see the configuration in the GUI.

Screenshots

Load Balancing Virtual Servers:

Service Groups:

Content Switching Server:

Citrix ADC / NetScaler monitors for Exchange 2019 fails with: "Failure - Time out during SSL handshake stage"

Problem

You’re attempting to publish / load balance your on-premise Exchange 2019 servers behind a Citrix ADC / NetScaler but notice that the health monitors created to check the health of the services (e.g. https://172.16.1.81/owa/healthcheck.htm) fail with the following error:

Failure - Time out during SSL handshake stage

The rest of the monitors are all reporting the same error:

Further troubleshooting reveals that this is due to the fact that the following server hardening registry keys are added to the Exchange 2019 servers:

Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL

AllowInsecureRenegoClients
REG_DWORD
0

AllowInsecureRenegoServers
REG_DWORD
0

Removing these entries one of the Exchange servers will correct the error allowing the probe to report that the server is up (note that it is partial because the other server still has the registry key):

Various forum discussions on Citrix points adjusting Deny SSL Renegotiation but none of the configuration settings corrected the issue for the environment I worked with:

https://discussions.citrix.com/topic/388325-netscaler-12-rfc-5746-on-backend-bug-limitation/#comment-1975755

https://discussions.citrix.com/topic/401441-basic-load-balancing-for-owa-exchange-2019/page/3/

Solution

After performing extensive troubleshooting but not able to come to a resolution, I decided to upgrade the Citrix ADC / NetScaler from NS13.0 52.24.nc to the latest build available at the time, which was NS13.0 67.39.nc:

NS13.0 52.24.nc

NS13.0 67.39.nc

This ended up resolving the issue with the services being correctly determined as UP:

Hope this helps anyone who might be facing this same issue as there isn’t much material available and the available solutions did not work for me.