How to configure Citrix ADC / NetScaler to forward client Source IP to Exchange Server 2019 / 2016 or any IIS application

Those who have worked with load balancers for applications will know that it can be a pain to troubleshoot issue where the source IP address is required because from the application’s perspective, all incoming connections have the originating IP address as the load balancer. With Citrix ADC / NetScalers, there are several methods in achieving this such as using the X-Forwarded-For header to include the source client IP address (this only works with HTTP and SSL services) or configuring direct server return (DSR) mode to allow the server to respond to clients directly by using a return path that does not flow through the Citrix ADC appliance. There are advantages and disadvantages for each method but for the purpose of this post, I will demonstrate how to configure Exchange Server 2019 (or any IIS application) to receive the source client IP with the X-Forwarded-For header.

The Scenario

Let’s assume that you have a user who is continuously locked out of their account and you have identified the event to take place on an on-premise Exchange server as you can see event ID 4625 Audit Failure events in the Security log as shown in the screenshot below:

The Exchange server is placed behind Citrix ADC / NetScalers and therefore have the IP address 172.16.5.90 of the load balancer for the Source Network Address field in the event.

Proceeding to navigate into the IIS logs on the Exchange server in the W3SVC1 folder located in the C:\inetpub\logs\LogFiles\ directory:

… and opening the logs show only the source IP of the Citrix ADC / NetScaler:

Configuration IIS on Exchange Server to log the X-Forwarded-For request header

The first step to log the source IP address is to configure IIS on the Exchange server to log the X-Forwarded-For request header that is passed from the Citrix ADC / NetScaler load balancer. The following TechNet Blog does a fantastic job of demonstrating the process:

How to use X-Forwarded-For header to log actual client IP address?
https://techcommunity.microsoft.com/t5/iis-support-blog/how-to-use-x-forwarded-for-header-to-log-actual-client-ip/ba-p/873115

Below is a demonstration with an Exchange 2019 server on Windows Server 2019 and IIS version 10.0.17763.1:

Begin by launching Internet Information (IIS) Manager, navigate to either the Server node or one of the websites and then open on Logging:

We will add the X-Forwarded-For field by clicking on Select Fields beside the W3C Format dropdown menu:

Proceed to click on Add Field and add the X-Forwarded-For text as the Field Name and Source, with the Source Type as Request Header:

Note how the X-Forwarded-For is added as a Custom Field:

Apply the changes:

**Note that configuring the above on one site automatically applies it to the other sites.

Now navigate back to the IIS log files and open the latest log file and confirm that the X-Forwarded-For field is added as a header:

The following is a side by side comparison where the log on the top has the X-Forwarded-For custom field added and the bottom does not:

Configure the Citrix ADC / NetScaler to forward client source IP as X-Forwarded-For

With the IIS server configured to receive the custom X-Forwarded-For field, proceed to log into the Citrix ADC / NetScaler, navigate to Traffic Management > Load Balancing > Service Groups or Services:

For the purpose of this example, we will be configuring all of the Exchange service groups to forward the client source IP address as X-Forwarded-For (owa, activesync, rpc, ews, Autodiscover, oab, mapi, ecp).

Open the properties of the load balancing service group or service, navigate to the Settings area and click on the edit icon:

Enable the Insert Client IP Header and type in the X-Forwarded-For string for the Header text box:

Click OK to save the settings and proceed to save the settings by clicking Done.

Repeat for the rest of the Load Balancing Service Group by using the GUI or the CLI command:

set service <name> -CIP <Value> <cipHeader>

Here are the commands for each Exchange service:

set serviceGroup SVG_EX2019_owa -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_activesync -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_rpc -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_ews -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_autodiscover -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_oab -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_mapi -cip enabled X-Forwarded-For
set serviceGroup SVG_EX2019_ecp -cip enabled X-Forwarded-For

Testing the configuration by verifying source IP address in IIS Logs

Switching back to the Exchange Server and navigating to the IIS logs should now have the latest log reveal a value for the X-Forwarded-For field. Below is a screenshot of the log before the configuration of the Citrix ADC / NetScaler:

Below is a screenshot after the change with an IP address added to the end of each connection with the source IP:

Hope this helps anyone looking for a way to log the originating source IP address of client requests on IIS that is load balanced by a Citrix ADC / NetScaler.

Using AirWatch to remotely run batch files for uninstalling applications

I don’t get to work with MDM or UEM applications much anymore due to my focus on Azure so I got pretty excited when an ex-colleague asked me if there was a way to use VMware AirWatch (also known as Workspace One) to remotely execute an uninstall command for an application because I had gone through the process before. After digging up some old notes to guide him through the setup, I thought I’d write this blog post in case anyone else happens to ask me in the future.

The official documentation, although a bit outdated, for this feature can be found here:

Using Product Provisioning to Deliver Files to Windows 10: Workspace ONE Operational Tutorial
https://techzone.vmware.com/using-product-provisioning-deliver-files-windows-10-workspace-one-operational-tutorial#_991596

For the purpose of this example, I will be using a batch file that manually (and forcefully) removes Cylance Protect from devices. Not long ago I ran into an issue where attempting to uninstall Cylance Protect from devices would display the following error:

Cylance PROTECT
You are attempting to run the 32-bit installer on a 64-bit version of Windows. Please run the 64-bit installer.

I couldn’t determine how to get around this as none of the methods such as editing registry keys and executing the following msiexec.exe would work:

msiexec /x {2E64FC5C-9286-4A31-916B-0D8AE4B22954} /quiet

Reaching out to Blackberry support had an support engineer point me to:

Fix problems that block programs from being installed or removed

https://support.microsoft.com/en-us/mats/program_install_and_uninstall

The tool worked but it had to be ran interactively and did not allow me to use it at scale.

A bit more research led me to the following script written by James Gallagher, which worked but please note that as this was originally provided by Cylance but later modified by Cyberforce, use this at your own risk. It worked for me but may not for others.

Manual Removal Of CylancePROTECT
https://cyberforcesecurityhelp.freshdesk.com/support/solutions/articles/44002036687-manual-removal-of-cylanceprotect

In case the post ever gets deleted, I will paste the contents for the customized-CylanceCleanupTool.bat at the end of this post.

With the above scenario described, let’s begin creating the configuration in AirWatch Version: 20.11.0.5 (2011):

Create the Files/Actions

The first step is to create a Files/Actions that will allow you to upload the batch file, define where to store it on the device, and how to execute the batch file.

Begin by navigating to Devices > Provisioning > Components > Files/Actions:

Click on the ADD FILES/ACTIONS:

Select Windows under Add Files/Actions:

Select Windows Desktop under Select Device Type:

Type in a name for the action and select the appropriate organization for Managed By:

Navigate to the Files tab, click on the ADD FILES tab, then Choose Files to select the batch file that will be uploaded and pushed to the clients:

Specify a download path where the batch file will be downloaded to on the client:

C:\Temp\AirWatch\

Save the configuration and the following line will be displayed in the Files tab:

Navigate to the Manifest tab and click on Install Manifest:

Select Run for Action(s) To Perform:

Select System for the Execution Context so the batch file is running with elevated permissions and specify the path and batch file (the location specified earlier for downloading the batch file and the batch file name that was just uploaded:

C:\Temp\AirWatch\customized-CylanceCleanupTool.bat

The following configuration will be displayed under Install Manifest. We can specify a command to uninstall but there will not be one configured for this example:

Proceed to save the new Files/Actions:

Create the Product to assign to devices

With the creation of the Files/Actions completed, the next step is to assign it to devices.

Navigate to Devices > Provisioning > Product List View and click on ADD PRODUCT:

Select Windows under Add Product:

Select Windows Desktop under Select Device Type:

Provide a name and description for the product, select the appropriate organization, and select the Smart Group this product should be applied to. For the purpose of this example, we will be assigning it to all devices.

Click on the Manifest tab and then the ADD button:

Select File/Action - Install for Action(s) To Perform and the previously created Files/Actions for the Files/Actions field:

The saved Manifest will be displayed as such:

You can further specify Conditions, Deployment and Dependencies options:

With the configuration completed, decide to click Save button to simply save the Manifest or Activate to save and immediately activate the configuration:

For the purpose of this example, I will click on Activate, which will display the list of devices it will be applied to:

The new product should now be displayed:

Waiting a few seconds and refreshing the update the In Progress, Compliant and Failed values:

Hope this helps anyone who might be looking for instructions on how to remotely run batch files with AirWatch.

customized-CylanceCleanupTool.bat

@ECHO OFF

title UNIFIED DRIVER CYLANCE CLEANUP TOOL

Echo UNIFIED DRIVER CYLANCE CLEANUP TOOL

:SwitchToWorkingDirectory

cd /d "%~dp0" 1>nul 2>&1

:AdminCheck

openfiles>nul 2>&1

IF %ERRORLEVEL% == 0 (

:AdminCheckPass

GOTO ManualUninstall

) ELSE (

:AdminCheckFail

Echo * Please re-run the Unified Driver Cylance Cleanup Tool as Administrator.

Echo * Exiting...

GOTO CyExit

)

:ManualUninstall

reg delete HKEY_LOCAL_MACHINE\SOFTWARE\Cylance /f

reg delete HKEY_CLASSES_ROOT\Installer\Products\C5CF46E2682913A419B6D0A84E2B9245 /f

reg delete HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\CylanceSvc /f

taskkill /im CylanceUI.exe

takeown /f "C:\Program Files\Cylance" /r /d y

icacls "C:\Program Files\Cylance" /reset /T

rd /s /q "C:\Program Files\Cylance"

takeown /f "C:\programdata\Cylance" /r /d y

rd /s /q C:\programdata\Cylance

:InstallCleanup

Echo * Installing the Unified Driver Cylance Cleanup Tool service...

CyCleanupSvc.exe "-install"

IF %ERRORLEVEL% == 0 (

GOTO WaitCleanup

) ELSE (

Echo * Failed to install the Unified Driver Cylance Cleanup Tool service.

Echo * Please check the Logs directory.

Echo * Exiting...

GOTO CyExit

)

:WaitCleanup

Echo * Waiting for the Unified Driver Cylance Cleanup Tool service to cleanup...

ping -n 30 127.0.0.1 1>nul 2>&1

Echo * Unified Driver Cylance Cleanup Tool is finished.

Echo * Removing the Unified Driver Cylance Cleanup Tool service...

CyCleanupSvc.exe "-uninstall"

IF %ERRORLEVEL% == 0 (

GOTO FinishCleanup

) ELSE (

Echo * Failed to remove the Unified Driver Cylance Cleanup Tool service.

Echo * Please check the Logs directory.

Echo * Exiting...

GOTO CyExit

)

:FinishCleanup

Echo * Unified Driver Cylance Cleanup Tool service has been removed.

Echo * Exiting...

:CyExit

exit

Skype for Business Online (SkypeOnlineConnector) PowerShell connections are blocked and Set-CsUser no longer works

Problem

You’ve noticed that the following cmdlets fail with the error message indicating Skype for Business Online (SkypeOnlineConnector) PowerShell connections are blocked:

Import-Module SkypeOnlineConnector

$sfbSession = New-CsOnlineSession

New-PSSession : [admin0b.online.lync.com] Processing data from remote server admin0b.online.lync.com failed with the

following error message: Skype for Business Online PowerShell connections are blocked. Please replace the Skype for

Business Online PowerShell connector module with the Teams PowerShell Module. Please visit https://aka.ms/sfbocon2tpm

for supported options. For more information, see the about_Remote_Troubleshooting Help topic.

At C:\Program Files\Common Files\Skype for Business

Online\Modules\SkypeOnlineConnector\SkypeOnlineConnectorStartup.psm1:254 char:16

+ ... $session = New-PSSession -Name $psSessionName -ConnectionUri $Connec ...

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ CategoryInfo : OpenError: (System.Manageme....RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotin

gTransportException

+ FullyQualifiedErrorId : IncorrectProtocolVersion,PSSessionOpenFailed

PS C:\WINDOWS\system32> Import-PSSession $sfbSession

Import-PSSession : Cannot validate argument on parameter 'Session'. The argument is null. Provide a valid value for

the argument, and then try running the command again.

At line:1 char:18

+ Import-PSSession $sfbSession

+ ~~~~~~~~~~~

+ CategoryInfo : InvalidData: (:) [Import-PSSession], ParameterBindingValidationException

+ FullyQualifiedErrorId : ParameterArgumentValidationError,Microsoft.PowerShell.Commands.ImportPSSessionCommand

As per the following Microsoft documentation:

Migrating from Skype for Business Online Connector to the Teams PowerShell module
https://aka.ms/sfbocon2tpm

Skype for Business Online (SkypeOnlineConnector) PowerShell connections will be rejected / blocked starting May 17, 2021 and to use Teams PowerShell Module for administration. However, attempting to use Connect-MicrosoftTeams and then executing the Set-CsUser cmdlet to enable a user for Enterprise Voice fails indicating it is not recognized:

Solution

One of the reasons why the Set-CsUser or any Cs- cmdlets will not work even if the MicrosofTeams module is used is if an old version is used. In order for the legacy Cs- cmdlets to be available, Teams PowerShell Module 2.0 or later needs to be installed and imported. The example above has an older module imported:

Get-Module

To update the module to the latest version, execute the following:

Uninstall-Module -Name MicrosoftTeams

Install-Module -Name MicrosoftTeams -RequiredVersion 2.0.0 -AllowClobber

Import-Module -Name MicrosoftTeams

The legacy Cs- cmdlets should now work:

Import-Module MicrosoftTeams

Connect-MicrosoftTeams

$usernameUPN = "tluk@contoso.com"

$extension = "tel:+7899"

Set-CsUser -Identity $usernameUPN -EnterpriseVoiceEnabled $true -HostedVoiceMail $true -OnPremLineURI $extension

Get-CsOnlineUser -Identity $usernameUPN | FL *uri

Grant-CsOnlineVoiceRoutingPolicy -Identity $usernameUPN -PolicyName "Toronto"

Grant-CsTenantDialPlan -PolicyName Toronto -Identity (Get-CsOnlineUser $usernameUPN).SipAddress