On-prem Skype for Business Server 2015 federation with Skype for Business Online logs the event: "Unable to resolve DNS SRV record";domain="fabrikam.com";dns-srv-result="NegativeResult";dns-source="WireQuery";source="sip.contoso.com"

Problem

You’ve configured your Skype for Business Server 2015 SIP FEDERATED PROVIDERS to federate with Skype for Business Online domains:

… but notice that one of the domains you are trying to reach is listing the presence information of the contact as Presence unknown:

Attempting to manually configure their domain in the SIP FEDERATED DOMAINS with the Edge Server configured as sipdir.online.lync.com:

… allows you to view their presence and trade messages but they are unable to view yours:

Using the Skype for Business Server 2015 Logging Tool to capture the IMAndPresence scenario:

Shows the following SIP/2.0 404 Not Found log entries:

TL_INFO(TF_PROTOCOL) [bmlyncedge01\bmlyncedge01]0608.10A4::03/02/2018-20:08:58.210.00002EA8 (SIPStack,SIPAdminLog::ProtocolRecord::Flush:ProtocolRecord.cpp(261)) [1193230895] Trace-Correlation-Id: 1193230895

Instance-Id: 9BD4F

Direction: outgoing;source="local";destination="internal edge"

Peer: bmlyncstd01.contoso.com:65371

Message-Type: response

Start-Line: SIP/2.0 404 Not Found

From: "Terence Luk"<sip:tluk@contoso.com>;tag=b57c0d9a53;epid=de538b165f

To: <sip:kseidl@fabrikam.com>;tag=61AAA34DF2202DB35CF394AA3FF1E420

Call-ID: ed355df33e6e40e8afe98664b3abe01b

CSeq: 1 SUBSCRIBE

Via: SIP/2.0/TLS 10.21.1.106:65371;branch=z9hG4bKBD5B17D0.7277A46B0013221F;branched=FALSE;ms-received-port=65371;ms-received-cid=9BD600

Via: SIP/2.0/TLS 10.23.0.28:58599;ms-received-port=58599;ms-received-cid=817000

Content-Length: 0

ms-diagnostics: 1008;reason="Unable to resolve DNS SRV record";domain="fabrikam.com";dns-srv-result="NegativeResult";dns-source="WireQuery";source="sip.contoso.com"

ms-edge-proxy-message-trust: ms-source-type=EdgeProxyGenerated;ms-ep-fqdn=bmlyncedge01.contoso.com;ms-source-verified-user=verified

$$end_record

Solution

One of the common causes of symptoms described above is if the domain hosted in Skype for Business Online does not have the appropriate SRV setup to allow the on-prem SfB to discover the company’s SfB. To ensure that the appropriate SRV is configured, simply attempt to look up the record with the following commands:

nslookup

set q=srv

_sipfederationtls.tcp.<domainFQDN>

A domain properly configured with the required DNS record should look like the output circled in green while a domain with the missing record should look like the output circled in red:

Creating the missing record will correct the issue.

Extra Troubleshooting Step

I went through numerous troubleshooting steps prior to looking at the obvious issue as outlined above because the problematic company had insisted that they did not have any issues with other partners so my assumption was that the domain I was working with wasn’t configured properly. The following may not have been the solution for this problem but I feel is worth checking as well.

One of the other possible causes to the issue above is if your domain is also configured for Office 365 services and you had inadvertently configured the domain to provide Skype for Business Online services. If this was the case then you would either have to configure interoperability between your on-premises deployment and Skype for Business Online as outlined in the following TechNet article:

Configure federation with Skype for Business Online

https://technet.microsoft.com/en-us/library/jj205126.aspx

If there is no intention on having Skype for Business Online accounts in Office 365 then ensure that the service is turned off. To check, simply launch the Azure Active Directory Module for Windows PowerShell and connect to the domain’s Office 365 account:

Use commands such as:

Get-MsolDomain -DomainName <domain>.com | FL

… or:

Get-MSolDomain | Select Name,Capabilities

… to determine whether Skype for Business Online is enabled for the domain:

Voicemails left with Skype for Business Server 2015 on Exchange Server 2016 does not get delivered to user’s inbox

Problem

You’ve just completed integrating Skype for Business Server 2015 with Exchange Server 2016 Unified Messaging to provide voicemail services but noticed that voicemails left for users are not being delivered and the following errors are being logged on the Exchange 2016 server:

Log Name: Application

Source: MSExchange Unified Messaging

Event ID: 1423

Level: Error

The Microsoft Exchange Unified Messaging service on the Mailbox server encountered an error while trying to process the message with header file "E:\Program Files\Microsoft\Exchange Server\V15\UnifiedMessaging\voicemail\c7d3e3ee-0212-4d27-99bd-f1960eca0698.txt". Error details: "Microsoft.Exchange.UM.UMCore.SmtpSubmissionException: Submission to the Hub Transport server failed. The operation will be retried. ---> Microsoft.Exchange.Net.ExSmtpClient.UnexpectedSmtpServerResponseException: Unexpected SMTP server response. Expected: 220, actual: 421, whole response: 421 4.3.2 Service not available

at Microsoft.Exchange.Net.ExSmtpClient.SmtpTalk.CheckResponse(ServerResponseInfo response, Int32 expectedCode)

at Microsoft.Exchange.Net.ExSmtpClient.SmtpTalk.Connect(String server, Int32 port, Boolean isSslPort)

at Microsoft.Exchange.Net.ExSmtpClient.SmtpClient.Submit()

at Microsoft.Exchange.UM.UMCore.SmtpSubmissionHelper.SubmitMessage(MessageItem message, String senderAddress, Guid senderOrgGuid, String recipientAddress, OutboundConversionOptions submissionConversionOptions, InternalExchangeServer smtpServer)

at Microsoft.Exchange.UM.UMCore.SmtpSubmissionHelper.SubmitMessage(MessageItem message, String senderAddress, Guid senderOrgGuid, String recipientAddress, OutboundConversionOptions submissionConversionOptions, String requestId)

--- End of inner exception stack trace ---

Server stack trace:

at Microsoft.Exchange.UM.UMCore.SmtpSubmissionHelper.HandleTransientSmtpFailure(Exception e, InternalExchangeServer smtpServer, String recipientAddress)

at Microsoft.Exchange.UM.UMCore.SmtpSubmissionHelper.SubmitMessage(MessageItem message, String senderAddress, Guid senderOrgGuid, String recipientAddress, OutboundConversionOptions submissionConversionOptions, String requestId)

at Microsoft.Exchange.UM.UMCore.SmtpSubmitStage.InternalDoSynchronousWork()

at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)

at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)

Exception rethrown at [0]:

at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase)

at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData& msgData)

at Microsoft.Exchange.UM.UMCore.SynchronousPipelineStageBase.SynchronousWorkDelegate.EndInvoke(IAsyncResult result)

at Microsoft.Exchange.UM.UMCore.SynchronousPipelineStageBase.EndSynchronousWork(IAsyncResult r)"

Log Name: Application

Source: MSExchange Unified Messaging

Event ID: 1446

Level: Error

The Microsoft Exchange Unified Messaging service on the Mailbox server failed to process the message with header file "E:\Program Files\Microsoft\Exchange Server\V15\UnifiedMessaging\voicemail\1fbf0f79-adeb-4b6c-a7e2-792ec27a5351.txt" within "11" minutes. The server will continue to process and deliver the message, but the "MSExchangeUMAvailability: % of Messages Successfully Processed Over the Last Hour" performance counter will be decreased.

Solution

Similar to one of my previous posts from 7 years ago:

Voicemails left on Exchange UM does not get delivered to inbox – Event ID 1082 and 1035 logged
http://terenceluk.blogspot.com/2010/07/voicemails-left-on-exchange-um-does-not.html

… the issue was caused by the receiving connector on the Exchange server which hosts the Unified Messaging role (note that Exchange 2016 no longer splits roles into other servers).  The following snippet from the event ID 1423 error logged displays what response the Exchange server is receiving when an attempt to deliver the voice is made:

The Microsoft Exchange Unified Messaging service on the Mailbox server encountered an error while trying to process the message with header file "E:\Program Files\Microsoft\Exchange Server\V15\UnifiedMessaging\voicemail\c7d3e3ee-0212-4d27-99bd-f1960eca0698.txt". Error details: "Microsoft.Exchange.UM.UMCore.SmtpSubmissionException: Submission to the Hub Transport server failed. The operation will be retried. ---> Microsoft.Exchange.Net.ExSmtpClient.UnexpectedSmtpServerResponseException: Unexpected SMTP server response. Expected: 220, actual: 421, whole response: 421 4.3.2 Service not available

You can verify this by logging onto one of the Exchange servers and initiate a telnet to port 25 to another as shown here:

421 4.3.2 Service not available

Connection to host lost.

C:\>

If the environment you’re working in has many receive connectors configured:

… my suggestion is to not to attempt figuring out which routing connector may be receiving the request and rejecting the connection but rather create new ones specifically for UM with the the scope configured with the Exchange server IP address and the following security configuration:

You should see the queued files stored in the folder:

… begin to clear out when the Exchange server begins accepting the voicemail delivery connections.

Migrating Skype for Business Server 2015 CMS from one server to another

I recently had to migrate and decommission a Skype for Business Server after upgrading it from Lync Server 2013 and noticed that the Windows 2012 Server would continuously blue screen every few hours.  One of the tasks involved with the migration was to move the CMS (Central Management Server) from the to-be-decommissioned server to the newly one and this post serves to demonstrate the process.

Note that the CMS is currently on the lyncstd01.domain.bm server:

Note the newly deployed lyncstd02.domain.bm that we will be moving the CMS to:

Begin by backing up the Skype for Business configuration with the following cmdlets:

Export-CsConfiguration -FileName C:\SfB-config.zip

Export-CsLisConfiguration -FileName C:\SfB-lis.zip

With the backup completed, run the Prepare first Standard Edition server wizard in the Skype for Business Server 2015 – Deployment Wizard on the CMS destination server (lyncstd02):

Once the wizard has completed, launch the Skype for Business Server Management Shell on the CMS destination server and execute the following:

Install-CsDatabase -CentralManagementDatabase -SQLServerFQDN <FQDN of your Standard Edition Server> -SQLInstanceName <name of instance - RTC by default>

The cmdlet executed in the screenshot below is:

Install-CsDatabase -CentralManagementDatabase -SQLServerFQDN lyncstd02.domain.bm -SQLInstanceName RTC

Once the cmdlet completes enable the topology with:

Enable-CsTopology

With the topology published, execute Move-CsManagementServer to move the Central Management Server to the destination server:

Note the following warning message once the cmdlet completes:

Warning: The move completed successfully but the following additional steps are required:

Run local setup on the following computers to remove Central Management services that are no longer defined in the topology:

- lyncstd01.domain.bm

Reviewing the html log will show the same warning message in the management shell:

Continue to run the Install or Update Skype for Business Server System on the new server hosting CMS:

Run the Setup or Remove Skype for Business Server Components wizard again:

Once the wizard completes, continue by running the Install or Update Skype for Business Server System on the old server that use host the CMS:

Run the Setup or Remove Skype for Business Server Components wizard:

There may be times when the wizard will complete but also throw the error:

Command execution failed: ###50021:XdsPublishItems:This central management store has moved to another location. No more changes can be made to this store.

Confirm that replication is successful by executing the cmdlet Get-CsManagementStoreReplicationStatus:

There may be times when replication does not appear to be working and the UpToDate variable never changes to True.  If this happens, check the Skype for Business Server File Transfer Agent service on both servers because sometimes it is stopped:

Restarting the services and rerunning the Get-CsManagementStoreReplicationStatus should show that all of the UpToDate variables is shown as True:

The CMS should now be moved and the Skype for Business Topology Builder should reflect the changes:

If the old server hosting the CMS is going to remain in the environment and you would like to remove the Central Management store files then execute the following:

Uninstall-CsDatabase -CentralManagementDatabase -SqlServerFqdn <FQDN of SQL Server> -SqlInstanceName <Name of source server>

**Note that it is important not to proceed with the removal of the previous database files until replication is complete and is stable. If you remove the files prior to completing replication, you will disrupt the replication process and leave the newly moved Central Management Server in an unknown state.

The following is an example of executing the cmdlet:

Uninstall-CsDatabase -CentralManagementDatabase -SqlServerFqdn lyncstd01.domain.bm -SqlInstanceName RTC

Once the cmdlet completes run the Setup or Remove Skype for Business Server Components wizard:

With the above completed, the Skype for Business Server Master Replicator Agent will no longer be present: