Configuring an Azure Traffic manager to load balance a Windows Server AD FS WAP and Citrix NetScaler AD FS WAP

To continue from my previous post:

Configuring a Citrix ADC / NetScaler to provide AD FS (Active Directory Federation Services) WAP (Web Application Proxy) service
http://terenceluk.blogspot.com/2020/06/configuring-citrix-adc-netscaler-to.html

I had completed the configuration of a virtual server that directed external inbound AD FS traffic via SSL_Bridge to the Windows Server 2019 AD FS WAP server and another content switching server that responded to external inbound AD FS traffic via SSL (perform the functionalities of a Windows Serve 2019 AD FS WAP server). The next step was to load balance the two but there did not seem to be a way to configure a Content Switching Virtual Server as a Backup server to a Load Balancing Virtual Server because the intention was to have the Windows Server 2019 WAP server provide AD FS sign in services and only failover to the Citrix ADC WAP in the event of a failure. Another challenge I had was that the AD FS health probe required port 80 to be opened for the Azure Traffic Director to monitor the health of the AD FS service. Attempting to try and allow the Traffic Manager to monitor the health of the AD FS servers from the WAP to the internal farm has been a top of discussion in other blog posts but for this configuration, the Citrix ADC already monitors the AD FS servers so all I needed was really to allow the Azure Traffic Manager to monitor the WAP services presented by the Citrix ADC.

With the above state, the following is what the topology looks like:

Note the following design features:

1. There are two public IP addresses, 1 represents the Windows Server 2019 AD FS WAP and the other one represents the Content Switching Server WAP on the Citrix ADC.
2. The Azure Traffic Manager has the routing method configured as priority, which allows all traffic to be directed to the Windows Server 2019 WAP and only two the Content Switching Server WAP on the Citrix ADC when the Windows Server 2019 WAP is unavailable.
3. Health probes are configured on the Citrix ADC to monitor the Windows AD FS servers’ health via port 80 /adfs/probe.
4. The idpinitiatedsignon.htm page is enabled in the AD FS farm so it is publicly accessible via https://fs.contoso.com/adfs/ls/idpinitiatedsignon.htm.
5. The Azure Traffic Manager will deem the AD FS portal as being unavailable if the idpinitiatedsignon.htm page is not reachable.

Configuration of the Azure Traffic Director

The following screenshots are the configuration parameters used for the Azure Traffic Manager:

Overview:

Configuration: Routing method: Priority
DNS time to live (TTL): 60
Protocol: HTTPS
Port: 443
Path: /adfs/ls/idpinitiatedsignon.htm
Custom header settings: host:fs.berninare.com
Expected Status Code Ranges (default: 200): <blank> defaults to 200
Probing interval: 10
Tolerated number of failures: 3
Probe timeout: 5

Most of the configuration parameters are self-explanatory but the one I want to note is the DNS time to live (TTL) which has the following description:

A more in depth explanation can be found here: https://docs.microsoft.com/en-us/azure/traffic-manager/traffic-manager-faqs#what-is-dns-ttl-and-how-does-it-impact-my-users

What is DNS TTL and how does it impact my users?

When a DNS query lands on Traffic Manager, it sets a value in the response called time-to-live (TTL). This value, whose unit is in seconds, indicates to DNS resolvers downstream on how long to cache this response. While DNS resolvers are not guaranteed to cache this result, caching it enables them to respond to any subsequent queries off the cache instead of going to Traffic Manager DNS servers. This impacts the responses as follows:

• a higher TTL reduces the number of queries that land on the Traffic Manager DNS servers, which can reduce the cost for a customer since number of queries served is a billable usage.
• a higher TTL can potentially reduce the time it takes to do a DNS lookup.
• a higher TTL also means that your data does not reflect the latest health information that Traffic Manager has obtained through its probing agents.

As described above, a higher the value of the TTL reduces cost, reduces time for DNS lookups but at the cost of the results not reflecting the health of the AD FS WAP server. I’ve found that a value of 60 seconds was acceptable for the period it takes to failover to the secondary Citrix ADC WAP but please be cognizant of the increase in cost if this is a heavily accessed service.

Endpoints:

 

Hope this helps anyone who may be looking to configure their AD FS external access similar to the what I’ve put in place. I am also always open to suggestions and corrections if there are any issues with this configuration.

Configuring a Citrix ADC / NetScaler to provide AD FS (Active Directory Federation Services) WAP (Web Application Proxy) service

One of the clients I recently worked with was trying to move away from using their Citrix ADC / NetScaler appliance for authenticating Office 365 services because the federation between the appliance and their Azure AD prevented them from configuring hybrid Azure AD join as both Microsoft and Citrix could not confirm whether it would work (https://docs.microsoft.com/en-us/azure/active-directory/devices/hybrid-azuread-join-federated-domains). A few other issues such as the NetScaler themes being incompatible with the Teams authentication window and password change lead to the decision to move to AD FS (Active Directory Federation Services). As most administrators may know, configuring a redundant AD FS infrastructure requires at least 4 servers (2 x internal AD FS server farm and 2 x WAP servers) and while virtual machines aren’t very expensive to host in Azure, the client wanted to reduce the amount of servers required. With this requirement, the recommendation was made to provision 2 x internal AD FS server farm, 1 x AD FS WAP server, and configure the Citrix ADC / NetScaler to provide the AD FS WAP service as a virtual server / content switching server. This reduces the server count by 1 and leverages the Citrix ADC’s capabilities while still having a full Windows AD FS infrastructure. The following is what the topology looks like:

Before I begin, note that I am not configuring the following:

Guide to Deploying NetScaler as an Active Directory Federation Services Proxy

https://www.citrix.com/content/dam/citrix/en_us/documents/products-solutions/guide-to-deploying-netscaler-as-an-active-directory-federation-services-proxy.pdf

… because this configuration will perform authentication at the proxy and may present compatibility issues. The purpose of the WAP configured on the Citrix ADC / NetScaler will act as a AD FS WAP with passthrough configured.

Prerequisites

This post will assume that load balancing has already been set up for the internal AD FS farm servers. If it has not been completed then please have a look at my previous blog post:

Configure Citrix ADC to load balance Microsoft Active Directory Federation Services (AD FS) on Windows Server 2019

http://terenceluk.blogspot.com/2020/05/configure-citrix-adc-to-load-balance.html

Create a Service Group

Begin by creating a Service Group to represent the ADFS service provided by the internal AD FS servers. Note that you cannot reuse the one that was created for load balancing the internal AD FS servers as shown in my previous blog post because the one we’ll be creating will be have the Protocol configured as SSL instead of SSL_Bridge:

With the new service group created, click on the No Service Group Member line to add the internal AD FS servers:

Select the server objects representing the internal AD FS servers and specify the Port as 443:

With the service group members added, click on OK to proceed:

Scroll to the Settings section and click on the pencil icon to edit the properties:

Configure the settings as such:

SureConnect – Disabled
Surge Protection – Enabled
Use Proxy Port – Enabled
Down State Flush – Enabled
Use Client IP – Disabled
Client Keep-alive – Disabled
TCP Buffering – Disabled
HTTP Compression – Enabled
Header: X-MS-Forwarded-Client-IP

Click on the No Service Group to Monitor Binding to add the previously created monitor for the ADFS servers:

Select the previously created monitor (as outlined in my previous post) and click on the Bind button to bind the monitor to the service group:

The Monitors section should now display 1 Service Group to Monitor Binding:

Click Done to complete the configuration for the service group:

Create a Virtual Server

Proceed to create a new virtual server:

Provide a name for the Virtual Server, configure the protocol as SSL, and specify the IP Address Type as Non Addressable as we’ll be creating a Content Switching Server to referencing this Load Balancing Virtual Server:

With the newly created Load Balancing Virtual Server created, click on No Load Balancing Virtual Server ServiceGroup Binding to add the previously created Service Group:

Click on the Bind button to complete the configuration:

Proceed by selecting No Server Certificate:

Select the certificate that will be used for the AD FS WAP service and click Bind:

Complete the creation of the virtual server by clicking on Done:

Create Content Switching Policies

Navigate to Traffic Management > Content Switching > Policies and click Add:

Configure the policy as such:

Name: Provide a name that conforms with your naming convention (e.g. CSPolicy_ADFS)

Action: <blank>

Log Action: <blank>

Domain: Expression

Expression:

HTTP.REQ.HOSTNAME.SET_TEXT_MODE(IGNORECASE).EQ("fs.contoso.com") && HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS("/adfs")

**Replace fs.contoso.com with the AD FS URL and verify that the quotes are not changed.

Proceed and create a second policy for the AD FS metadata as such:

Name: Provide a name that conforms with your naming convention (e.g. CSPolicy_ADFS_Metadata)

Action: <blank>

Log Action: <blank>

Domain: Expression

Expression:

HTTP.REQ.HOSTNAME.SET_TEXT_MODE(IGNORECASE).EQ("fs.contoso.com") && HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS("/FederationMetadata")

**Replace fs.contoso.com with the AD FS URL.

The following two Content Switching Policies should be created:

Create a Content Switching Server

With the policies in place, proceed to create a Content Switching server:

Configure the Content Switch Virtual Server as such:

Name: Provide a name that conforms with your naming convention (e.g. CSVS_fs.contoso.com_NSWAP)

Protocol: SSL

Target: NONE

Persistent Type: <blank>

Persistent Mask: 255.255.255.255

IPv6 Persist Mask Length: 128

Timeout: 2

IP Address: An IP address for the Content Switch Virtual Server

Port: 443

Click on the No Content Switching Policy Bound line item:

Select the first policy that was created (non-metadata one) and configure the settings as such:

Priority: 100

Goto Expression: END

Invoke LabelType: None

Target Load Balancing Virtual Server: Select the one that was created earlier

Add the second policy that was created (metadata one) and configure the settings as such:

Priority: 110

Goto Expression: END

Invoke LabelType: None

Target Load Balancing Virtual Server: Select the one that was created earlier

The following 2 policies should be binded to the Content Switching Server:

Select the Certificate option under the Advanced Settings:

Select the No Server Certificate line item:

Select the certificate that will be used for the AD FS WAP service and click Bind:

Complete the creation of the Content Switching server and verify that the STATE is labeled as UP:

Create Rewrite Actions

Navigate to AppExpert > Rewrite > Actions and create a new action:

Create a new action with the following configuration:

Name: Provide a name that conforms with your naming convention (e.g. rw_act_adfs_proxyheader)

Type: INSERT_HTTP_HEADER

Header Name: X-MS-Proxy

Expression:

"NETSCALER"

Create a second rewrite action with the following configuration:

Name: Provide a name that conforms with your naming convention (e.g. rw_act_adfs_mex)

Type: REPLACE

Expression:

"/adfs/services/trust/proxymex" + HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).PATH_AND_QUERY.STRIP_START_CHARS("/adfs/services/trust/mex").HTTP_URL_SAFE

The following two Rewrite Actions should be created:

Create Rewrite Policies

Navigate to AppExpert > Rewrite > Policies and create a new action:

Create a new policy with the following configuration:

Name: Provide a name that conforms with your naming convention (e.g. rw_pol_adfs_ProxyHeader)

Action: rw_act_adfs_proxyheader

Log Action: <blank>

Undefined-Result Action*: -Global-undefined-result-action-

Expression:

HTTP.REQ.URL.TO_LOWER.STARTSWITH("/adfs")

Create a second rewrite action with the following configuration:

Name: Provide a name that conforms with your naming convention (e.g. rw_pol_adfs_mex)

Action: rw_act_adfs_mex

Log Action: <blank>

Undefined-Result Action*: -Global-undefined-result-action-

Expression:

HTTP.REQ.URL.TO_LOWER.STARTSWITH("/adfs/services/trust/mex")

The following two polices should be created:

Assign the Rewrite Policies to the Load Balancing Virtual Server

With the Rewrite Policies created, open the configuration of the Load Balancing Virtual Server that was created earlier:

Select Policies under Advanced Settings:

Click on the To add, please click on the + icon line item:

Assign a policy with the following configuration:

Choose Policy: Rewrite

Choose Type: Request

Select the ProxyHeader policy and configure the following:

Priority: 100

Goto Expression: NEXT

Invoke LabelType: None

Bind the mex policy with the following configuration:

Priority: 110

Goto Expression: END

Invoke LabelType: None

The following policies should be binded:

Proceed to test the Citrix ADC / NetScaler Content Switching server AD FS WAP by either the assigned IP address or the Public IP that is NAT-ed to the IP.

HTTP/1.1 Service Unavailable

If tests to the Citrix ADC AD FS WAP displays the error HTTP/1.1 Service Unavailable:

This is because SNI binding needs to be configured on the AD FS servers. Proceed to use the following command prompt to list the certificate used for the AD FS service:

netsh http show sslcert

Note the following certificate properties:

Hostname:port : fs.contoso.com:443

Certificate Hash : cc429f179e41c0d8a3bc74f92977d3bcb2f549e8

Application ID : {5d89a20c-beab-4389-9447-324788eb944a}

Certificate Store Name : MY

The command to configure the SNI binding is as follows:

netsh http add sslcert ipport=0.0.0.0:443 certhash=<the certificate hash> appid=<the certificate appID> certstorename=<the certificate datastore>

For this environment, the command would look as such:

netsh http add sslcert ipport=0.0.0.0:443 certhash= cc429f179e41c0d8a3bc74f92977d3bcb2f549e8 appid={5d89a20c-beab-4389-9447-324788eb944a} certstorename=MY

Repeat the same procedure on all of the AD FS servers.

Load Balancing Windows AD FS WAP and Citrix ADC WAP

Note that my original intention was to configure this Content Switching server as the backup of the Load Balancing Virtual Server that provides a SSL_Bridge connection to the Windows AD FS WAP server but realized that it is not possible to do so. What I ended up doing was configure an Azure Traffic Manager to direct traffic between the two services. I will write another blog post to demonstrate the configuration next week.

Update: February 22, 2021

I’ve received several requests to post the CLI commands for the configuration so I have extracted them and will paste it here:

Create ADFS Monitor:
add lb monitor mon_adfs_http HTTP -respCode 200 -httpRequest "GET /adfs/probe" -LRTM ENABLED -destPort 80

Create A Service Group:
add serviceGroup SVG_ADFS_NS SSL -maxClient 0 -maxReq 0 -cip ENABLED X-MS-Forwarded-Client-IP -usip NO -useproxyport YES -sp ON -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP YES

Bind Servers to Service Group:
bind serviceGroup SVG_ADFS_NS BREAZADFS1 443
bind serviceGroup SVG_ADFS_NS BREAZADFS2 443

Bind ADFS Monitor to Service Group:
bind serviceGroup SVG_ADFS_NS -monitorName mon_adfs_http

Create a Load Balancing Virtual Server:
add lb vserver LBVS_fs.contoso.com_NSWAP SSL 0.0.0.0 0 -persistenceType NONE -cltTimeout 180

Bind Service Group to LBVS:
bind lb vserver LBVS_fs.contoso.com_NSWAP SVG_ADFS_NS

Bind SSL Certificate to LBVS:
bind ssl vserver LBVS_fs.contoso.com_NSWAP -certkeyName <certificateName>

Disable SSL 3.0 and TLS 1.0:
set ssl vserver LBVS_fs.contoso.com_NSWAP -ssl3 DISABLED -dtls1 DISABLED

Create Content Switching Policies:
add cs policy CSPolicy_ADFS -rule "HTTP.REQ.HOSTNAME.SET_TEXT_MODE(IGNORECASE).EQ(\"fs.contoso.com\") && HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/adfs\")"
add cs policy CSPolicy_ADFS_Metadata -rule "HTTP.REQ.HOSTNAME.SET_TEXT_MODE(IGNORECASE).EQ(\"fs.contoso.com\") && HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/FederationMetadata\")"

Create a Content Switching Server:
add cs vserver CSVS_fs.contoso.com_NSWAP SSL 172.16.25.17 443 -cltTimeout 180 -persistenceType NONE

Bind Policy to Content Switching Server:
bind cs vserver CSVS_fs.contoso.com_NSWAP -policyName CSPolicy_ADFS -targetLBVserver LBVS_fs.contoso.com_NSWAP -priority 100
bind cs vserver CSVS_fs.contoso.com_NSWAP -policyName CSPolicy_ADFS_Metadata -targetLBVserver LBVS_fs.contoso.com_NSWAP -priority 110

Bind Service Group to Content Switching Server:
bind ssl vserver CSVS_fs.contoso.com_NSWAP -certkeyName <certificateName>

Create Rewrite Actions:
add rewrite action rw_act_adfs_proxyheader insert_http_header X-MS-Proxy "\"NETSCALER\""
add rewrite action rw_act_adfs_mex replace HTTP.REQ.URL.PATH_AND_QUERY "\"/adfs/services/trust/proxymex\" + HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).PATH_AND_QUERY.STRIP_START_CHARS(\"/adfs/services/trust/mex\").HTTP_URL_SAFE"

Create Rewrite Policies:
add rewrite policy rw_pol_adfs_ProxyHeader "HTTP.REQ.URL.TO_LOWER.STARTSWITH(\"/adfs\")" rw_act_adfs_proxyheader
add rewrite policy rw_pol_adfs_mex "HTTP.REQ.URL.TO_LOWER.STARTSWITH(\"/adfs/services/trust/mex\")" rw_act_adfs_mex

Assign the Rewrite Policies to the Load Balancing Virtual Server:
bind lb vserver LBVS_fs.contoso.com_NSWAP -policyName rw_pol_adfs_ProxyHeader -priority 100 -gotoPriorityExpression NEXT -type REQUEST
bind lb vserver LBVS_fs.contoso.com_NSWAP -policyName rw_pol_adfs_mex -priority 110 -gotoPriorityExpression END -type REQUEST