Connectivity issues between Citrix XenDesktop DDC and virtual desktops causing intermittent connectivity issues and inability to connect

Problem

You notice that you are having connectivity issues between various virtual desktops where it would be in Ready status for a few minutes then switches to Unregistered.  Logging onto your Citrix XenDesktop virtual desktop shows the following event IDs logged:

• Warning Event ID 1014
• Warning Event ID 1002
• Warning Event ID 1017
• Warning Event ID 1022
• Warning Event ID 1012
• Warning Event ID 1048
• Information Event ID 0
• Warning Event ID 1001

Warning Event ID 1014

The Citrix Desktop Service lost contact with the Citrix Desktop Delivery Controller Service on server 'svrctxddc02.contoso.internal'.

The service will now attempt to register again.

Error details:

Exception 'There was no endpoint listening at http://10.2.1.21/Citrix/CdsController/IRegistrar that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.' of type 'System.ServiceModel.EndpointNotFoundException'

Warning Event ID 1002

The Citrix Desktop Service cannot connect to the delivery controller 'http://svrctxddc02.contoso.internal:80/Citrix/CdsController/IRegistrar' (IP Address '10.2.1.21')
Check that the system clock is in sync between this machine and the delivery controller. If this does not resolve the problem, please refer to Citrix Knowledge Base article CTX117248 for further information.
Error Details:
Exception 'Error occurred when attempting to connect to endpoint at address http://svrctxddc02.contoso.internal:80/Citrix/CdsController/IRegistrar, binding WsHttpBindingIRegistrarEndpoint and contract Citrix.Cds.Protocol.Controller.IRegistrar: System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at http://10.2.1.21/Citrix/CdsController/IRegistrar that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
   at System.Net.HttpWebRequest.GetResponse()
   at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
   --- End of inner exception stack trace ---
Server stack trace:
   at System.ServiceModel.Security.IssuanceTokenProviderBase`1.DoNegotiation(TimeSpan timeout)
   at System.ServiceModel.Security.SspiNegotiationTokenProvider.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Security.SymmetricSecurityProtocol.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.SecurityChannelFactory`1.ClientSecurityChannel`1.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.DoOperation(SecuritySessionOperation operation, EndpointAddress target, Uri via, SecurityToken currentToken, TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.GetTokenCore(TimeSpan timeout)
   at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionClientSettings`1.ClientSecuritySessionChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at System.ServiceModel.ICommunicationObject.Open()
   at Citrix.Cds.BrokerAgent.ControllerConnectionFactory.AttemptConnection[T](EndpointReference endpoint, Boolean throwOnError, Boolean allowNtlmAuthentication, String connectUsingIpThisIpAddress, Boolean cacheFactory)' of type 'Citrix.Cds.BrokerAgent.ConnectionFailedException'..

Warning Event ID 1017

The Citrix Desktop Service failed to register with any delivery controller.
The service will retry registering with controllers in approximately 17 seconds.
Please ensure that at least one delivery controller is available for Virtual Desktop Agents to register with. Refer to Citrix Knowledge Base article CTX117248 for further information

Warning Event ID 1022

The Citrix Desktop Service failed to register with any controllers in the last 2 minutes.

The service will now try to register with controllers at a reduced rate of every 2 minutes.

Warning Event ID 1012

The Citrix Desktop Service successfully registered with delivery

controller svrctxddc01.contoso.internal (IP Address 10.2.1.20).

The endpoint address of the controller is http://svrctxddc01.contoso.internal:80/Citrix/CdsController/IRegistrar.

Warning Event ID 1048

The Citrix Desktop Service is re-registering with the DDC: 'NotificationManager:NotificationServiceThread: WCF failure or rejection by broker (DDC: svrctxddc01.contoso.internal)'

From here, you get another informational Event ID 0:

The description for Event ID 0 from source Self-service Plug-in cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

Self-service Plug-in started (user=Contoso\hleaback).

the message resource is present but the message is not found in the string/message table

Then the event ID 1001 warning:

The Citrix Desktop Service failed to obtain a list of delivery controllers with which to register.
Please ensure that the Active Directory configuration for the farm is correct, that this machine is in the appropriate Active Directory domain and that one or more delivery controllers have been fully initialized.
Refer to Citrix Knowledge Base article CTX117248 for further information.
Error details:
Exception 'The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.' of type 'System.ServiceModel.CommunicationObjectFaultedException'

The logs then repeat again.

On the actual Citrix XenDesktop DDC, you see the following event IDs logged:

• Warning Event ID 1060
• Warning Event ID 1039
• Information Event ID 1066

Warning Event ID 1060

The Citrix Broker Service failed to apply settings on the virtual machine 'VDI-HOLLY.contoso.internal'.

Check that the virtual machine can be contacted from the controller and that any firewall on the virtual machine allows connections from the controller. See Citrix Knowledge Base article CTX126992.

Error details:

Exception 'The request channel timed out while waiting for a reply after 00:00:59.9919992. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.' of type 'System.TimeoutException'.

Warning Event ID 1039

The Citrix Broker Service failed to contact virtual machine 'VDI-MARINA.contoso.internal' (IP address 10.2.1.51).
Check that the virtual machine can be contacted from the controller and that any firewall on the virtual machine allows connections from the controller. See Citrix Knowledge Base article CTX126992.
Error details:
Exception 'Client is unable to finish the security negotiation within the configured timeout (00:00:04.9989999).  The current negotiation leg is 1 (00:00:04.9989999).  ' of type 'System.TimeoutException'.

Information Event ID 1066

The Citrix Broker Service successfully determined the base settings needed for the Virtual Desktop Agent of machine 'VDI-HOLLY.contoso.internal'.

Solution

While there can be multiple reasons as to why these errors would be thrown, I’ve found that these generally point to communications issues between the Citrix XenDesktop DDC and the actual VDA agent (the virtual desktop).  I’ve seen time drift between the DDC and the VDA agent causing such an issue and lately, a duplicate IP address for the Citrix XenDesktop DDC on the network where the DDC is not completely offline but the VDA agent is able to sometimes successfully send traffic to the DDC and sometimes unable to.  If you to encounter these warnings and notice your VDI is listed as unregistered in the Desktop Studio, check that there are no issues between the subnets of your VDI and DDC, no port issues and certainly no IP conflicts.