How to configure Citrix ADC / NetScaler to forward client Source IP to Exchange Server 2019 / 2016 or any IIS application

Those who have worked with load balancers for applications will know that it can be a pain to troubleshoot issue where the source IP address is required because from the application’s perspective, all incoming connections have the originating IP address as the load balancer. With Citrix ADC / NetScalers, there are several methods in achieving this such as using the X-Forwarded-For header to include the source client IP address (this only works with HTTP and SSL services) or configuring direct server return (DSR) mode to allow the server to respond to clients directly by using a return path that does not flow through the Citrix ADC appliance. There are advantages and disadvantages for each method but for the purpose of this post, I will demonstrate how to configure Exchange Server 2019 (or any IIS application) to receive the source client IP with the X-Forwarded-For header.

The Scenario

Let’s assume that you have a user who is continuously locked out of their account and you have identified the event to take place on an on-premise Exchange server as you can see event ID 4625 Audit Failure events in the Security log as shown in the screenshot below:

The Exchange server is placed behind Citrix ADC / NetScalers and therefore have the IP address 172.16.5.90 of the load balancer for the Source Network Address field in the event.

Proceeding to navigate into the IIS logs on the Exchange server in the W3SVC1 folder located in the C:\inetpub\logs\LogFiles\ directory:

… and opening the logs show only the source IP of the Citrix ADC / NetScaler:

Configuration IIS on Exchange Server to log the X-Forwarded-For request header

The first step to log the source IP address is to configure IIS on the Exchange server to log the X-Forwarded-For request header that is passed from the Citrix ADC / NetScaler load balancer. The following TechNet Blog does a fantastic job of demonstrating the process:

How to use X-Forwarded-For header to log actual client IP address?
https://techcommunity.microsoft.com/t5/iis-support-blog/how-to-use-x-forwarded-for-header-to-log-actual-client-ip/ba-p/873115

Below is a demonstration with an Exchange 2019 server on Windows Server 2019 and IIS version 10.0.17763.1:

Begin by launching Internet Information (IIS) Manager, navigate to either the Server node or one of the websites and then open on Logging:

We will add the X-Forwarded-For field by clicking on Select Fields beside the W3C Format dropdown menu:

Proceed to click on Add Field and add the X-Forwarded-For text as the Field Name and Source, with the Source Type as Request Header:

Note how the X-Forwarded-For is added as a Custom Field:

Apply the changes:

**Note that configuring the above on one site automatically applies it to the other sites.

Now navigate back to the IIS log files and open the latest log file and confirm that the X-Forwarded-For field is added as a header:

The following is a side by side comparison where the log on the top has the X-Forwarded-For custom field added and the bottom does not:

Configure the Citrix ADC / NetScaler to forward client source IP as X-Forwarded-For

With the IIS server configured to receive the custom X-Forwarded-For field, proceed to log into the Citrix ADC / NetScaler, navigate to Traffic Management > Load Balancing > Service Groups or Services:

For the purpose of this example, we will be configuring all of the Exchange service groups to forward the client source IP address as X-Forwarded-For (owa, activesync, rpc, ews, Autodiscover, oab, mapi, ecp).

Open the properties of the load balancing service group or service, navigate to the Settings area and click on the edit icon:

Enable the Insert Client IP Header and type in the X-Forwarded-For string for the Header text box:

Click OK to save the settings and proceed to save the settings by clicking Done.

Repeat for the rest of the Load Balancing Service Group by using the GUI or the CLI command:

set service <name> -CIP <Value> <cipHeader>

Here are the commands for each Exchange service:

set serviceGroup SVG_EX2019_owa -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_activesync -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_rpc -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_ews -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_autodiscover -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_oab -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_mapi -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_ecp -cip enabled X-Forwarded-For

Testing the configuration by verifying source IP address in IIS Logs

Switching back to the Exchange Server and navigating to the IIS logs should now have the latest log reveal a value for the X-Forwarded-For field. Below is a screenshot of the log before the configuration of the Citrix ADC / NetScaler:

Below is a screenshot after the change with an IP address added to the end of each connection with the source IP:

Hope this helps anyone looking for a way to log the originating source IP address of client requests on IIS that is load balanced by a Citrix ADC / NetScaler.

Accessing Exchange Server 2019 /OWA and /ECP throws the errors: "Status code: 500" and "Could not load file or assembly 'Microsoft.Exchange.Common, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies."

Problem

You’ve noticed that after patching Exchange Server 2019 servers (for this example, the patch for HAFNIUM was applied), access to /OWA and /ECP now displays the following errors:

OWA

:-(

Something went wrong

Your request couldn't be completed. HTTP Status code: 500.

X-ClientId: F8E662D41996402E8660EEEB0976EA50

request-id a5077939-13bd-4032-9471-d6c8dc221d5a

X-OWA-Error System.Web.HttpUnhandledException

X-OWA-Version 15.2.721.13

X-FEServer Exch01

X-BEServer Exch02

Date:3/8/2021 12:31:53 PM

InnerException: System.IO.DirectoryNotFoundException

Fewer details...

Refresh the page

ECP

Server Error in '/ecp' Application.

_______________________________________

Could not load file or assembly 'Microsoft.Exchange.Common, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Exchange.Common, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Assembly Load Trace: The following information can be helpful to determine why the assembly 'Microsoft.Exchange.Common, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' could not be loaded.

WRN: Assembly binding logging is turned OFF.

To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.

Note: There is some performance penalty associated with assembly bind failure logging.

To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

Stack Trace:

[FileNotFoundException: Could not load file or assembly 'Microsoft.Exchange.Common, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.]

System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMarkHandle stackMark, IntPtr pPrivHostBinder, Boolean loadTypeFromPartialName, ObjectHandleOnStack type) +0

System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean loadTypeFromPartialName) +96

System.Type.GetType(String typeName, Boolean throwOnError, Boolean ignoreCase) +65

System.Web.Compilation.BuildManager.GetType(String typeName, Boolean throwOnError, Boolean ignoreCase) +62

System.Web.Configuration.ConfigUtil.GetType(String typeName, String propertyName, ConfigurationElement configElement, XmlNode node, Boolean checkAptcaBit, Boolean ignoreCase) +50

[ConfigurationErrorsException: Could not load file or assembly 'Microsoft.Exchange.Common, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.]

System.Web.Configuration.ConfigUtil.GetType(String typeName, String propertyName, ConfigurationElement configElement, XmlNode node, Boolean checkAptcaBit, Boolean ignoreCase) +572

System.Web.Configuration.ConfigUtil.GetType(String typeName, String propertyName, ConfigurationElement configElement, Boolean checkAptcaBit) +31

System.Web.Configuration.Common.ModulesEntry.SecureGetType(String typeName, String propertyName, ConfigurationElement configElement) +59

System.Web.Configuration.Common.ModulesEntry..ctor(String name, String typeName, String propertyName, ConfigurationElement configElement) +59

System.Web.HttpApplication.BuildIntegratedModuleCollection(List`1 moduleList) +221

System.Web.HttpApplication.GetModuleCollection(IntPtr appContext) +1103

System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers) +122

System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context) +173

System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context) +255

System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext) +347

[HttpException (0x80004005): Could not load file or assembly 'Microsoft.Exchange.Common, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.]

System.Web.HttpRuntime.FirstRequestInit(HttpContext context) +552

System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context) +122

System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +737

You’ve noticed that reviewing the BinSearchFolders application settings for ecp folder in the Exchange Back End website shows that the Value is configured with %ExchangeInstallDir%:

Changing this to the path without using the variable appears to fix the ECP page but not OWA:

Solution

One of the possible solutions to correct the issue is to use the UpdateCas.ps1 script located in the \Microsoft\Exchange Server\V15\Bin folder to rebuild the /OWA and /ECP directory:

Proceed to test the /owa and /ecp directories once the PowerShell completes.

Load Balancing Exchange Server 2019 with Citrix ADC / NetScaler

One of the most common questions I am asked by colleagues and clients is in regards to how they can publish Exchange Servers behind a Citrix ADC / NetScaler to provide load balancing for all services such as OWA, ActiveSync, RPC, EWS, AutoDiscover, MAPI, and ECP BUT NOT SMTP. I would like to further emphasize that SMTP (mail traffic) is excluded for this because I never liked suggesting load balancers to be used for this service and the reason is that the true originating IP of the traffic will be masked by the Citrix ADC / Netscaler. This effectively means that if you won’t be able to restrict IPs allowed for relaying via the Exchange receive connectors or troubleshoot an issue for a relaying client. An workaround for this solution could be to use a DSR (Direct Server Reply) configuration but limitations such as having the Exchange server and Citrix ADC on the same subnet is likely not something that will be possible for more environments. Lastly, a misconfiguration of the Citrix ADC / NetScaler (lack of filtering based on source IP address) could potentially create a open relay on the Exchange server because from the Exchange server perspective, it would relay any SMTP traffic sent by the Citrix ADC / NetScaler.

With the above said, this blog post will serve to provide the CLI commands to configure the Citrix ADC / NetScaler with the appropriate server objects, monitors, service groups, load balancing virtual server, and content switching server. Note that it is possible to not use a content switching server and simply use a load balancing virtual server to publish Exchange servers but you would not be able to independently redirect requests to specific services to another server if those a service is not available. An example to this is if ExchangeServer1 just has /owa unavailable, the content switching server will redirect requests to ExchangeServer2 but will continue to send /ews to ExchangeServer1 if it is available. Using just a load balancing server will deem ExchangeServer1 as down and redirect all traffic to another server.

The version of Exchange Server 2019 and Citrix ADC / NetScaler used for this post are as follows:

Exchange Server 2019: Version 15.2 ‎(Build 721.2)‎

Citrix ADC / NetScaler: NS13.0 67.39.nc

Note that certain version of Citrix ADC / NetScaler appear to exhibit issues with monitoring Exchange services as outlined one of my previous posts:

Citrix ADC / NetScaler monitors for Exchange 2019 fails with: "Failure - Time out during SSL handshake stage"
http://terenceluk.blogspot.com/2020/12/citrix-adc-netscaler-monitors-for.html

Prerequisites

• One private IP addresses allocated for the VIP of the Content Switching server
• Imported SSL certificate to secure traffic
• Firewall rule that allows the Citrix NSIP to the Exchange servers on port 443
• Firewall rule that allows internet traffic via the firewall to the VIP of the Content Switching server on port 443 and 80

Step #1 – Enable Features

Enable the features below by executing the following command: enable ns feature CS,RESPONDER,LB,SSL

• Content Switching
• Responder
• Load Balancing
• SSL

Step #2 – Import SSL Certificate

Either import the SSL certificate currently used on the Exchange servers to secure 443 traffic:

Importing PFX certificate from Microsoft Windows Server into Citrix NetScaler VPX
http://terenceluk.blogspot.com/2016/05/importing-pfx-certificate-from.html

Or request a new one if this is a greenfield deployment. We will need to assign the SSL certificate to the load balancing virtual server and the content switching server.

Make sure the issuing Root and Intermediate certificate chain is imported and linked as well.

Step #3 – Create the Server Objects

Create the server objects with the following command and replace the name and IP as it reflects your environment:

add server ExchServer1.contoso.com 172.16.1.81

add server ExchServer2.contoso.com 172.16.1.82

Step #4 – Create Exchange Service Monitors

The monitors we’ll be creating for are as follows:

1. OWA
2. ActiveSync
3. RPC
4. EWS
5. AutoDiscover
6. OAB
7. MAPI
8. ECP

Execute the following CLI commands to create the monitors:

add lb monitor mon_exch_owa HTTP-ECV -send "GET /owa/healthcheck.htm" -recv 200 -LRTM DISABLED -destPort 443 -secure YES

add lb monitor mon_exch_activesync HTTP-ECV -send "GET /Microsoft-Server-ActiveSync/healthcheck.htm" -recv 200 -LRTM DISABLED -secure YES

add lb monitor mon_exch_rpc HTTP-ECV -send "GET /rpc/healthcheck.htm" -recv 200 -LRTM DISABLED -secure YES

add lb monitor mon_exch_ews HTTP-ECV -send "GET /ews/healthcheck.htm" -recv 200 -LRTM DISABLED -secure YES

add lb monitor mon_exch_autodiscover HTTP-ECV -send "GET /Autodiscover/healthcheck.htm" -recv 200 -LRTM DISABLED -secure YES

add lb monitor mon_exch_oab HTTP-ECV -send "GET /oab/healthcheck.htm" -recv 200 -LRTM DISABLED -secure YES

add lb monitor mon_exch_mapi HTTP-ECV -send "GET /mapi/healthcheck.htm" -recv 200 -LRTM DISABLED -secure YES

add lb monitor mon_exch_ecp HTTP-ECV -send "GET /ecp/healthcheck.htm" -recv 200 -LRTM DISABLED -secure YES

Step #5 – Create Service Groups and Bind Servers

We’ll need to create a service group for each of the services we created a monitor for with the following CLI commands:

add serviceGroup SVG_EX2019_owa SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add serviceGroup SVG_EX2019_activesync SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add serviceGroup SVG_EX2019_rpc SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add serviceGroup SVG_EX2019_ews SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add serviceGroup SVG_EX2019_autodisover SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add serviceGroup SVG_EX2019_oab SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add serviceGroup SVG_EX2019_mapi SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add serviceGroup SVG_EX2019_ecp SSL -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

With the service groups created, proceed to bind the Exchange servers and the monitors into them with the following CLI command:

bind serviceGroup SVG_EX2019_owa ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_owa ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_owa -monitorName mon_exch_owa

bind serviceGroup SVG_EX2019_activesync ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_activesync ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_activesync -monitorName mon_exch_activesync

bind serviceGroup SVG_EX2019_rpc ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_rpc ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_rpc -monitorName mon_exch_rpc

bind serviceGroup SVG_EX2019_ews ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_ews ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_ews -monitorName mon_exch_ews

bind serviceGroup SVG_EX2019_autodisover ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_autodisover ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_autodisover -monitorName mon_exch_autodiscover

bind serviceGroup SVG_EX2019_oab ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_oab ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_oab -monitorName mon_exch_oab

bind serviceGroup SVG_EX2019_mapi ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_mapi ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_mapi -monitorName mon_exch_mapi

bind serviceGroup SVG_EX2019_ecp ExchServer2.contoso.com 443

bind serviceGroup SVG_EX2019_ecp ExchServer1.contoso.com 443

bind serviceGroup SVG_EX2019_ecp -monitorName mon_exch_ecp

Step #6 – Create Load Balancing Virtual Server, Bind Service Groups and Bind SSL Certificate

A Load Balancing Virtual Server object with an IP Address Type configured as Non Addressable is required to be created for each service:

Proceed to create these load balancing virtual servers with the following CLI commands:

add lb vserver mail.contoso.com_owa-LBVS SSL 0.0.0.0 0 -persistenceType NONE -cltTimeout 180

add lb vserver mail.contoso.com_activesync-LBVS SSL 0.0.0.0 0 -persistenceType SRCIPDESTIP -cltTimeout 180

add lb vserver mail.contoso.com_rpc-LBVS SSL 0.0.0.0 0 -persistenceType SOURCEIP -timeout 30 -cltTimeout 180

add lb vserver mail.contoso.com_ews-LBVS SSL 0.0.0.0 0 -persistenceType NONE -cltTimeout 180

add lb vserver mail.contoso.com_autodiscover-LBVS SSL 0.0.0.0 0 -persistenceType SOURCEIP -timeout 30 -cltTimeout 180

add lb vserver mail.contoso.com_oab-LBVS SSL 0.0.0.0 0 -persistenceType NONE -cltTimeout 180

add lb vserver mail.contoso.com_mapi-LBVS SSL 0.0.0.0 0 -persistenceType SOURCEIP -timeout 30 -cltTimeout 180

add lb vserver mail.contoso.com_ecp-LBVS SSL 0.0.0.0 0 -persistenceType NONE -cltTimeout 180

With the load balancing virtual servers created, proceed to bind the service groups containing the Exchange Servers to them:

bind lb vserver mail.contoso.com_owa-LBVS SVG_EX2019_owa

bind lb vserver mail.contoso.com_activesync-LBVS SVG_EX2019_activesync

bind lb vserver mail.contoso.com_rpc-LBVS SVG_EX2019_rpc

bind lb vserver mail.contoso.com_ews-LBVS SVG_EX2019_ews

bind lb vserver mail.contoso.com_autodiscover-LBVS SVG_EX2019_autodisover

bind lb vserver mail.contoso.com_oab-LBVS SVG_EX2019_oab

bind lb vserver mail.contoso.com_mapi-LBVS SVG_EX2019_mapi

bind lb vserver mail.contoso.com_ecp-LBVS SVG_EX2019_ecp

Lastly, bind the imported SSL certificate (Step #2) to the load balancing virtual server with the following commands:

bind ssl vserver mail.contoso.com_owa-LBVS -certkeyName wildcard-contoso-exp-mar-2021

bind ssl vserver mail.contoso.com_activesync-LBVS -certkeyName wildcard-contoso-exp-mar-2021

bind ssl vserver mail.contoso.com_rpc-LBVS -certkeyName wildcard-contoso-exp-mar-2021

bind ssl vserver mail.contoso.com_ews-LBVS -certkeyName wildcard-contoso-exp-mar-2021

bind ssl vserver mail.contoso.com_autodiscover-LBVS -certkeyName wildcard-contoso-exp-mar-2021

bind ssl vserver mail.contoso.com_oab-LBVS -certkeyName wildcard-contoso-exp-mar-2021

bind ssl vserver mail.contoso.com_mapi-LBVS -certkeyName wildcard-contoso-exp-mar-2021

bind ssl vserver mail.contoso.com_ecp-LBVS -certkeyName wildcard-contoso-exp-mar-2021

Step #7 – Create a Content Switching Server, Bind SSL Certificate, Create the Content Switching Policies, Bind the Policies to the Content Switching Server

With all the dependent components created and configured, proceed to create the content switching server, which will have the pre-allocated VIP assigned and configured to direct requests to the Exchange services to a server that has a healthy status:

add cs vserver mail.contoso.com_http-CS HTTP 172.16.5.62 80 -cltTimeout 180 -persistenceType NONE

add cs vserver mail.contoso.com_ssl-CS SSL 172.16.5.62 443 -cltTimeout 180 -persistenceType NONE

Bind the same SSL certificate used for the load balancing virtual server to the SSL content switching server:

bind ssl vserver mail.contoso.com_ssl-CS -certkeyName wildcard-contoso-exp-mar-2021

Create the content switching policies that represent each of the services being published by the load balancing virtual servers:

add cs action cs_act_exch2019_owa -targetLBVserver mail.contoso.com_owa-LBVS

add cs action cs_act_exch2019_ews -targetLBVserver mail.contoso.com_ews-LBVS

add cs action cs_act_exch2019_autodiscover -targetLBVserver mail.contoso.com_autodiscover-LBVS

add cs action cs_act_exch2019_activesync -targetLBVserver mail.contoso.com_activesync-LBVS

add cs action cs_act_exch2019_oab -targetLBVserver mail.contoso.com_oab-LBVS

add cs action cs_act_exch2019_mapi -targetLBVserver mail.contoso.com_mapi-LBVS

add cs action cs_act_exch2019_rpc -targetLBVserver mail.contoso.com_rpc-LBVS

add cs action cs_act_exch2019_ecp -targetLBVserver mail.contoso.com_ecp-LBVS

add cs policy cs_pol_exch2019_owa -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/owa\")" -action cs_act_exch2019_owa

add cs policy cs_pol_exch2019_ews -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/ews\")" -action cs_act_exch2019_ews

add cs policy cs_pol_exch2019_autodiscover -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/autodiscover\")" -action cs_act_exch2019_autodiscover

add cs policy cs_pol_exch2019_activesync -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"Microsoft\")" -action cs_act_exch2019_activesync

add cs policy cs_pol_exch2019_oab -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/oab\")" -action cs_act_exch2019_oab

add cs policy cs_pol_exch2019_mapi -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/mapi\")" -action cs_act_exch2019_mapi

add cs policy cs_pol_exch2019_rpc -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/rpc\")" -action cs_act_exch2019_rpc

add cs policy cs_pol_exch2019_ecp -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/ecp\")" -action cs_act_exch2019_ecp

An additional OWA policy will also need to be added to address a but identified in the following KB:

OWA access via NetScaler is getting stuck at /cgi/selfauth?params= after authentication
https://support.citrix.com/article/CTX209060

add cs policy cs_pol_exch2019_cgi -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"/cgi\")" -action cs_act_exch2019_owa

The following policy will redirect requests going to the mail.contoso.com address to mail.contoso.com/owa:

add cs policy cs_pol_exch2019_owa_redirect -rule "HTTP.REQ.HOSTNAME.EQ(\"mail.contoso.com\")" -action cs_act_exch2019_owa

Lastly, proceed to bind the policies created above to the content switching server:

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_owa -priority 110

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_ews -priority 120

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_autodiscover -priority 130

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_activesync -priority 140

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_oab -priority 150

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_mapi -priority 160

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_rpc -priority 170

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_ecp -priority 180

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_cgi -priority 190

bind cs vserver mail.contoso.com_ssl-CS -policyName cs_pol_exch2019_owa_redirect -priority 200

Step #8 – Create a Responder to redirect http to https

The final step for the configuration is to create a responder to redirect http to https so users typing in http://mail.contoso.com will automatically be redirected to https://mail.contoso.com:

add responder action redirect_http_https_act_owa redirect "\"https://\"+HTTP.REQ.HOSTNAME+\"/owa/\""

add responder policy redirect_http_https_pol_owa "HTTP.REQ.HOSTNAME.CONTAINS(\"mail.contoso.com\")" redirect_http_https_act_owa

bind cs vserver mail.contoso.com_http-CS -policyName redirect_http_https_pol_owa -priority 100 -gotoPriorityExpression END -type REQUEST

The above steps should configuring the required components for load balancing Exchange servers with the Citrix ADC NetScaler. I will include some screenshots of the configuration in case anyone wants to see the configuration in the GUI.

Screenshots

Load Balancing Virtual Servers:

Service Groups:

Content Switching Server:

Citrix ADC / NetScaler monitors for Exchange 2019 fails with: "Failure - Time out during SSL handshake stage"

Problem

You’re attempting to publish / load balance your on-premise Exchange 2019 servers behind a Citrix ADC / NetScaler but notice that the health monitors created to check the health of the services (e.g. https://172.16.1.81/owa/healthcheck.htm) fail with the following error:

Failure - Time out during SSL handshake stage

The rest of the monitors are all reporting the same error:

Further troubleshooting reveals that this is due to the fact that the following server hardening registry keys are added to the Exchange 2019 servers:

Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL

AllowInsecureRenegoClients
REG_DWORD
0

AllowInsecureRenegoServers
REG_DWORD
0

Removing these entries one of the Exchange servers will correct the error allowing the probe to report that the server is up (note that it is partial because the other server still has the registry key):

Various forum discussions on Citrix points adjusting Deny SSL Renegotiation but none of the configuration settings corrected the issue for the environment I worked with:

https://discussions.citrix.com/topic/388325-netscaler-12-rfc-5746-on-backend-bug-limitation/#comment-1975755

https://discussions.citrix.com/topic/401441-basic-load-balancing-for-owa-exchange-2019/page/3/

Solution

After performing extensive troubleshooting but not able to come to a resolution, I decided to upgrade the Citrix ADC / NetScaler from NS13.0 52.24.nc to the latest build available at the time, which was NS13.0 67.39.nc:

NS13.0 52.24.nc

NS13.0 67.39.nc

This ended up resolving the issue with the services being correctly determined as UP:

Hope this helps anyone who might be facing this same issue as there isn’t much material available and the available solutions did not work for me.

Attempting to install Exchange Server 2019 to Cumulative Update 7 fails with: “Cannot start service MSExchangeIS on computer”

I do not have many clients with an on-premise Exchange Server but there was one who needed their Exchange Server 2019 CU3 patched to the latest CU7.

Error:

The following error was generated when "$error.Clear();

start-SetupService -ServiceName MSExchangeIS

" was run: "Microsoft.Exchange.Configuration.Tasks.ServiceDisabledException: Service 'MSExchangeIS' is disabled on this server. ---> System.InvalidOperationException: Cannot start service MSExchangeIS on computer '.'. ---> System.ComponentModel.Win32Exception: The service cannot be started, either because it is disabled or because it has no enabled devices associated with it

--- End of inner exception stack trace ---

at System.ServiceProcess.ServiceController.Start(String[] args)

at Microsoft.Exchange.Management.Tasks.ManageSetupService.StartServiceWorker(ServiceController serviceController, String[] serviceParameters)

--- End of inner exception stack trace ---

at Microsoft.Exchange.Configuration.Tasks.Task.ThrowError(Exception exception, ErrorCategory errorCategory, Object target, String helpUrl)

at Microsoft.Exchange.Management.Tasks.ManageSetupService.StartService(ServiceController serviceController, Boolean ignoreServiceStartTimeout, Boolean failIfServiceNotInstalled, Unlimited`1 maximumWaitTime, String[] serviceParameters)

at Microsoft.Exchange.Management.Tasks.ManageSetupService.StartService(String serviceName, Boolean ignoreServiceStartTimeout, Boolean failIfServiceNotInstalled, Unlimited`1 maximumWaitTime, String[] serviceParameters)

at Microsoft.Exchange.Management.Tasks.StartSetupService.InternalProcessRecord()

at Microsoft.Exchange.Configuration.Tasks.Task.<ProcessRecord>b__91_1()

at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)".

Attempting to restart the CU7 patch detects the unfinished patching and restarts but then fails again:

There aren’t any Microsoft KBs or blog posts with the error but the message implies the Microsoft Exchange Information Store (MSExchangeIS) could not be started and reviewing the services console does show that it is disabled so I manually enabled it, started it, restarted the installer but it would fail again with the MSExchangeIS service disabled.

A bit of searching on the internet lead me to the following blog post:

https://oddytee.wordpress.com/2017/11/09/cannot-start-service-msexchangeservicehost-on-computer-during-exchange-cu-update/

… which suggested that we chronically change any Microsoft Exchange services set to disabled to automatic with the following PowerShell command:

While (1 -le 2) { Sleep 1 ; Get-Service | Where { $_.DisplayName -Like ‘Microsoft Exchange*’ } | Set-Service –StartupType ‘Automatic’ }

I gave this a shot and can confirm that it allowed CU7 to complete patching the Exchange server.

Installing Microsoft .NET Framework 4.8 stalls at: "File security verification: All files were verified successfully. Installation progress:"

Problem

You’re attempting to upgrade an Exchange Server 2019 to Cumulative Update 7, which requires .NET Framework 4.8:

https://support.microsoft.com/en-ca/help/4571787/cumulative-update-7-for-exchange-server-2019

The prerequisites identifies that .NET Framework 4.8 is not present and provides a link to the following installer:

https://support.microsoft.com/en-us/help/4503548/microsoft-net-framework-4-8-offline-installer-for-windows

The following is the offline installer:

Executing it on the Exchange server loads the installer but stays stuck at the following prompt:

File security verification:

All files were verified successfully.

Installation progress:

Solution

Before begin troubleshooting, review the current version of .NET Framework installed with the following PowerShell cmdlet:

PS C:\Windows\system32> Get-ItemProperty -Path "HKLM:SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full" | Format-List

CBS : 1

Install : 1

InstallPath : C:\Windows\Microsoft.NET\Framework64\v4.0.30319\

Release : 461814

Servicing : 0

TargetVersion : 4.0.0

Version : 4.7.03190

PSPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\NET Framework

Setup\NDP\v4\Full

PSParentPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4

PSChildName : Full

PSDrive : HKLM

PSProvider : Microsoft.PowerShell.Core\Registry

PS C:\Windows\system32>

This issue is one that I’ve come across multiple times in the past as demonstrated in one of my previous posts:

Installing .NET Framework 4.7.1 on Windows Server 2016 as a prerequisite for patching Exchange 2016 CU8 to CU12 remains stuck at: “File security verification: All files were verified successfully.”
http://terenceluk.blogspot.com/2019/04/installing-net-framework-471-on-windows_4.html

There are times when waiting 20 minutes or more will proceed with the install:

However, what I’ve found that the MSU from the Microsoft Update Catalog installs much faster:

https://www.catalog.update.microsoft.com/Search.aspx?q=4486153

Once installed, execute the following cmdlet again to verify that .NET Framework 4.8 is installed:

Get-ItemProperty -Path "HKLM:SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full" | Format-List

Hope this helps anyone who may run into this issue as updating Exchange Server to a CU is usually performed after hours with limited time for troubleshooting.