Pages

Monday, December 14, 2020

Citrix ADC / NetScaler monitors for Exchange 2019 fails with: "Failure - Time out during SSL handshake stage"

Problem

You’re attempting to publish / load balance your on-premise Exchange 2019 servers behind a Citrix ADC / NetScaler but notice that the health monitors created to check the health of the services (e.g. https://172.16.1.81/owa/healthcheck.htm) fail with the following error:

Failure - Time out during SSL handshake stage

image

The rest of the monitors are all reporting the same error:

image

Further troubleshooting reveals that this is due to the fact that the following server hardening registry keys are added to the Exchange 2019 servers:

Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL

AllowInsecureRenegoClients
REG_DWORD
0

AllowInsecureRenegoServers
REG_DWORD
0

image

Removing these entries one of the Exchange servers will correct the error allowing the probe to report that the server is up (note that it is partial because the other server still has the registry key):

image

Various forum discussions on Citrix points adjusting Deny SSL Renegotiation but none of the configuration settings corrected the issue for the environment I worked with:

https://discussions.citrix.com/topic/388325-netscaler-12-rfc-5746-on-backend-bug-limitation/#comment-1975755

https://discussions.citrix.com/topic/401441-basic-load-balancing-for-owa-exchange-2019/page/3/

image

Solution

After performing extensive troubleshooting but not able to come to a resolution, I decided to upgrade the Citrix ADC / NetScaler from NS13.0 52.24.nc to the latest build available at the time, which was NS13.0 67.39.nc:

NS13.0 52.24.nc

image

NS13.0 67.39.nc

image

This ended up resolving the issue with the services being correctly determined as UP:

image

Hope this helps anyone who might be facing this same issue as there isn’t much material available and the available solutions did not work for me.

Tuesday, December 8, 2020

Configuring an Azure Point-to-Site VPN using Microsoft Enterprise CA certificates for authentication for remote users to establish a secure tunnel into Azure

I’ve recently been asked by several clients about how they would go about setting up a P2S (Point-to-Site) VPN for their remote workers to VPN into Azure so I thought I’d write a short blog post demonstrating the process.

Before I begin, the following are the Microsoft documentation for configuring a Point-to-Site VPN that should be reviewed as it provides an explanation of how it works and the respective configuration parameters available during the deployment.

Configure a Point-to-Site VPN connection to a VNet using native Azure certificate authentication: Azure portal
https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-howto-point-to-site-resource-manager-portal

About Point-to-Site VPN
https://docs.microsoft.com/en-us/azure/vpn-gateway/point-to-site-about

Configure a Point-to-Site VPN connection to a VNet using native Azure certificate authentication: Azure portal
https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-howto-point-to-site-resource-manager-portal

Create and install VPN client configuration files for native Azure certificate authentication P2S configurations
https://docs.microsoft.com/en-us/azure/vpn-gateway/point-to-site-vpn-client-configuration-azure-cert

Configure a Point-to-Site VPN connection to a VNet using native Azure certificate authentication: Azure portal
https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-howto-point-to-site-resource-manager-portal

Step #1 - Configure Azure Point-to-Site VPN

Assuming that a Site-to-Site VPN between Azure and an on-premise datacenter or office is already configured, begin by logging into Azure and selecting the Virtual network gateways option to list the gateways configured:

image

Select the existing Site-to-Site VPN gateway that is already configured and then click on Point-to-site configuration:

image

The following options for the P2S VPN is displayed:

image

The Address pool is where you define the IP subnet that the VPN client will be in. Specify a subnet that does not overlap any existing address space specified in a Virtual Network.

The Tunnel type is where you select the type of VPN you want to provide to the users. I won’t go into the details of the differences between all of them and will simply reference the following Microsoft TechNet article: https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-howto-point-to-site-resource-manager-portal#tunneltype

image

For the purpose of demonstrating how to use Microsoft Enterprise CA certificates for authentication and using a Windows 10 operating system to connect natively with the OS built client, we’ll be configuring the following:

Tunnel type: IKEv2 and SSTP (SSL)
Authentication type: Azure certificate

Root Certificates:

We’ll need to obtain the Microsoft Enterprise CA root certificate’s Public certificate data by performing the following:

Either log onto the certificate authority, launch the Local Computer Certificate Store (certlm.msc), navigate to the Personal > Certificates folder to locate the Root CA Certificate Authority’s certificate (this is used to sign certificates this CA issues):

image

Or from any domain joined computer, launch the Local Computer Certificate Store (certlm.msc), navigate to the Trusted Root Certification Authorities > Certificates folder to locate the Root CA Certificate Authority’s certificate:

image

Open the properties of the certificate, click on the Details tab, click on Copy to File to copy the certificate without the private key to a file in Base-64 encoded X.509 format:

image

image

image

Open the file in Notepad, copy the content between:

-----BEGIN CERTIFICATE-----

… and:

-----END CERTIFICATE-----

image

… then paste it into the Public certificate data field:

image

Save the configuration and proceed to download the VPN client by clicking on Download VPN client:

image

A zip file containing the following folders will be downloaded:

image

We’ll come back to installing the client once we have issued the client certificate.

Update December 15, 2020 – Configuring OpenVPN SSL

I received a few questions after publishing this post that using the IKEv2 and SSTP (SSL) and native Windows 10 VPN client isn’t usually a viable option due to the local administrator right issue, which while has workarounds available, isn’t a great solution overall. For those who want to implement a solution that has a free VPN client can opt for using tunnel type OpenVPN SSL paired with the Azure VPN Client. OpenVPN SSL can be configured to either authenticate the user via the Azure Active Directory or with certificates:

Create an Azure Active Directory tenant for P2S OpenVPN protocol connections
https://docs.microsoft.com/en-us/azure/vpn-gateway/openvpn-azure-ad-tenant

image

The Azure VPN Client can be downloaded via the following URL:

https://www.microsoft.com/en-ca/p/azure-vpn-client/9np355qt2sqb?activetab=pivot:overviewtab#

imageimageimage

The following is a sample OpenVPN (SSL) configuration using certificate for authentication:

image

Once complete, click on the Download VPN client to download a package with the azurevpnconfig.xml configuration file then import it into the Azure VPN client:

image

image

Select the appropriate Certificate Information, change the Connection Name if required and leave all the other settings as the default:

image

Note how the message Client Certificate Hash cannot be empty is displayed if a certificate is not selected:

imageimage

Save the changes and proceed to connect:

image

Note that the client has successfully connected along with the appropriate routes injected without the user requiring local administrator rights:

image 

Step #2 – Generate Client Certifcates for Authentication

The Microsoft document doesn’t provide much information for using an Enterprise CA other than the following:

Generate client certificates
https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-howto-point-to-site-resource-manager-portal#generateclientcert

You can generate client certificates by using the following methods:

· Enterprise certificate:

o If you're using an enterprise certificate solution, generate a client certificate with the common name value format name@yourdomain.com. Use this format instead of the domain name\username format.

o Make sure the client certificate is based on a user certificate template that has Client Authentication listed as the first item in the user list. Check the certificate by double-clicking it and viewing Enhanced Key Usage in the Details tab.

The information isn’t very clear on what certificate template to use other than that the common name value format must be the UPN rather than domain name\username. The good news is that the default User certificate template that a Microsoft Enterprise CA has can be used. While logged in as the user, launch the Certificates Console under the Current User context with:

Certmgr.msc

Proceed to request a new certificate under the Personal > Certificates store:

image

image

Select Active Directory Enrollment Policy:

image

Select the User template:

image

As the User template does not require information to be supplied because it requests the template with the logged in user’s identity, a certificate will be issued:

image

Step #3 – Install and Configure Windows VPN Client

With the appropriate certificate requested, we can now install and configure the VPN client by navigating into the appropriate folder based on the operating system. For this example, the OS is Windows 10 so the VPNClientSetupAmd64.exe executable in the folder WindowsAmd64 will be ran:

image

image

***Note that the installation doesn’t really present anything on the screen as it just flashes for a bit (I hope this will change in the future).

With the VPN installation completed, proceed to launch the VPN settings:

image

Note that there will be a new VPN connection configured (in this case it is vnet-prod-eastus):

image

The following are the advanced settings of the VPN connection. Note that the Type of sign-in info is specified as Certificate:

image

The following are the settings that can be edited:

image

Proceeding to connect to Azure will display the following message:

image

Note that one of the most complained about topic for the Azure Point-to-Site VPN is that it requires the user connecting to Azure be an administrator because routes need to be injected into the routing table by using the CMROUTE.DLL file to copy them into a routes.txt file and this operation requires elevated permissions:

image

The problem with this approach is that it is executed under the context of the current user, and a standard user does not have the permissions to make changes to routes. There are plenty of workarounds available on the internet for this so I will not include them here.

More information can be found here:

What are the client configuration requirements?
https://docs.microsoft.com/en-us/azure/vpn-gateway/point-to-site-about#what-are-the-client-configuration-requirements

image

The connection will proceed once the routes have been copied:

image

Notice that the VPN connect is now labeled as Connected:

image

An IP address will also be assigned under the PPP adapter:

image

Reviewing the Allocated IP Addresses will display the IP address assigned to the VPN client:

image

Note that if the following message is displayed in the Connection Status when attempting to connect then it is because a valid certificate for authentication is not available:

A certificate could not be found that can be used with this Extensible Authentication Protocol. (Error 798)

image

Also, there are cases when administrators would like to generate the certificate for the user under their own login and if this is desired, simply duplicate the User certificate template and modify the properties of it from:

imageimage