Audio calls via a Lync Server 2013 Edge server gets stuck at connecting with no audio for 10 seconds then disconnects

I was recently asked by a client to troubleshoot an issue with their Lync Server 2013 environment where users connecting remotely via the Edge server are unable to establish voice calls. A user would be able to successfully authenticate as well as see the call come in but when they pick up with their handset, they would not hear any audio and the call would disconnect within 10 seconds. Using the Skype for Business client would show the call in the Connecting call… status then disconnect:

While this type of issue can be caused by many reasons, this environment I had to troubleshoot the issue in had 3 contributing misconfigurations:

#1 – A/V Edge service NAT was not configured

The Edge server’s external interface IPs were all NAT-ed but the topology for the environment did not have the external IP address configured:

#2 – The DNS record of the Edge server’s internal interface was not correct

The Edge server for this environment was recently moved with the internal interface’s IP address was changed and since the server is not joined to the domain, the internal DNS A record which servers such as the front-end server uses to reach the Edge server was not updated.

#3 – Port 5062 was not opened between the Edge server and FE / SBA servers

The internal interface of the Edge server was in a secured network and TCP port 5062 used for authentication of A/V users was not opened. The following TechNet article provides more detail about the purpose of this port:

Edge Server environmental requirements in Skype for Business Server
https://docs.microsoft.com/en-us/skypeforbusiness/plan-your-deployment/edge-server-deployments/edge-environmental-requirements

----------------------------------------------------------------------------------------------------------------------------

Note that other than discovering audio does not work, the Remote Connectivity Analyzer tool that Microsoft provides can help with identifying such an issue: https://testconnectivity.microsoft.com

The following is the output from a test ran against the environment:

Testing remote connectivity for user tluk@domain.com to the Microsoft Lync server.
      Specified remote connectivity test(s) to Microsoft Lync server failed. See details below for specific failure reasons.
        Tell me more about this issue and how to resolve it

     
Additional Details
      Couldn't sign in. Error: User failed to get response from MRAS server.
SIP service request to MRAS server failed.
Error Message: A 504 (Server time-out) response was received from the network and the operation failed. See the exception details for more information..
Error Type: PublishSubscribeException.
Fault Code: .
Response Code: 504.
Response Text: Server time-out.
Diagnostic Header: ErrorCode=1038,Source=UKSBA03.contoso.COM,Reason=Failed to connect to a peer server,fqdn=ukles03.contoso.com:5062,peer-type=InternalServer,winsock-code=10060,ip-address=192.168.34.10,winsock-info=The peer did not respond to the connection attempt
Microsoft.Rtc.Signaling.DiagnosticHeader


Elapsed Time: 12486 ms.

Attempting to configure VMware SRM (Site Recovery Manager) Protection Group fails with: "... due to unresolved devices"

Problem

You attempt to configure a new VMware SRM protection group but receive the following error:

ERROR

Operation Failed Task completed with error CompositeException Unable to protect VM '<virtualMachineName>' due to unresolved devices Unable to protect VM '<virtualMachineName>’ due to unresolved devices Unable to protect VM '<virtualMachineName>' due to unresolved devices

Operation ID: 33a8561f-c6b5-4868-b2f7-01453d762ad6

Solution

This error is usually thrown due to components of SRM not being completely configured. The following are a few common items:

1. Network Mappings
2. Folder Mappings
3. Resource Mappings

There will be times when the administrator has configured the mappings the configuration in the mapping may not be correct and the best way to identify exactly what is preventing the protection group from being configured is to navigate to the Virtual Machines tab and review the Protection Status output, which would explicitly display the problem. In the case of this example, the network mappings were configured but various port groups and cluster mapping for these VMs were not configured:

The configuration for the Protection Groups should successfully complete once the missing mappings are configured:

Windows 10 fails to sysprep with the error message: “A fatal error occurred while trying to sysprep the machine.”

Problem

You attempt to run sysprep on a Windows 10 operating system by manually navigating to C:\windows\system32\sysprep to execute the sysprep.exe:

… but receive the following error:

System Preparation Tool 3.14

A fatal error occurred while trying to sysprep the machine.

You navigate to the directory:

C:\windows\system32\sysprep\Panther

.. and find the following the content in the setuperr.log file:

2019-03-14 07:29:46, Error [0x0f0073] SYSPRP RunExternalDlls:Not running DLLs; either the machine is in an invalid state or we couldn't update the recorded state, dwRet = 0x1f

2019-03-14 07:29:46, Error [0x0f00ae] SYSPRP WinMain:Hit failure while processing sysprep cleanup external providers; hr = 0x8007001f

Solution

One of the first items to check is that the rearm limit has not been reached by executing slmgr.vbs /dlv to review the Remaining Windows rearm count (the desktop I was working on has not reached 0):

If the Remaining Windows rearm count hasn’t been exceeded then proceed to check the following registry keys:

HKEY_LOCAL_MACHINE\SYSTEM\Setup\Status\SysprepStatus\

Verify that the CleanupState registry key is set to 2:

Verify that the GeneralizationState is set to 7 (the desktop I was working on had the value of 3):

Uninstall and reinstall the MSDTC with the following commands:

msdtc -uninstall

msdtc –install

Navigate to the following registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\SoftwareProtectionPlatform\

Verify that the SkipRearm key has the value of 1:

Rerunning the sysprep.exe executable to start the sysprep process should work as expected now:

Using wmic (Windows Management Interface Command) to remotely uninstall applications

I’ve recently been tasked to create a script that would remotely uninstall Adobe Flash on all desktops on the network and after not having any luck with PowerShell, I reverted to the wmic (Windows Management Interface Command) command I have used in the past.  While this isn’t the best way to guarantee the removal of the application in any environment it can be used in situations where you need a method that requires very little time.

The first step in the process is to obtain a list of computer names that you would like to remotely uninstall the application from and put it into a txt file with each name on a separate line.  If you intend on running it against all the computers in Active Directory then you can use the following PowerShell cmdlet to export the list in CSV format:

Get-ADComputer -Filter * -Property * | Select-Object Name,OperatingSystem,OperatingSystemServicePack,OperatingSystemVersion | Export-CSV AllWindows.csv -NoTypeInformation -Encoding UTF8

The above command would generate a CSV file as such:

You won’t need all the columns so simply copy the column with the computer names and paste it into a txt file then save it as computers.txt.

The command you’ll run with the reference to the computers.txt file will be the following:

wmic /failfast:on /node:@"computers.txt" product where "name like 'Adobe Flash%'" call uninstall /nointeractive

Note that the % sign is a wildcard and the following output will be displayed if the application is found and uninstalled on the remote computer:

Skype for Business Server 2019 Front-End service stuck at Starting status

Problem

You’ve just completed deploying a new Skype for Business Server 2019 server into an environment but noticed that the Skype for Business Server Front-End service remains stuck at the Starting status and never completes to Running or stops:

Executing the cmdlet Get-CsWindowsService displays the following:

Reviewing the Lync Server logs show the following entries:

Log Name: Lync Server

Source: LS User Services

Event ID: 32174

Level: Warning

Server startup is being delayed because fabric pool manager has not finished initial placement of users.

Currently waiting for routing group: {63BB8586-A9D8-5AF2-83FF-B5CE680594C0}.

Number of groups potentially not yet placed: 1.

Total number of groups: 1.

Cause: This is normal during cold-start of a Pool and during server startup.

If you continue to see this message many times, it indicates that insufficient number of Front-Ends are available in the Pool.

Resolution:

During a cold-start of a large Pool it can take up to an hour for the placement process to finish as it needs to populate all the Front-End databases with data from the Backup Store. If the Pool is running and the Front-End is just started, this is normal for some time. If this repeats for a long time, ensure that all the Front-Ends configured for this Pool are up and running. If multiple Front-Ends have been recently decommissioned, run Reset-CsPoolRegistrarState -ResetType QuorumLossRecovery to enable the Pool to recover from Quorum Loss and make progress.

Scrolling upwards from the warning displays the following error:

Log Name: Lync Server

Source: LS MCU Infrastructure

Event ID: 61029

Level: Error

In the past 30.0093507983333 minutes the process RtcHost(6756) received 1 invalid certificates. The last one was from server: contsfbstd01.contoso.com, IP Address: 10.198.40.152:60873, with subject: CN=contsfbstd01.contoso.com, OU=IT, O=contoso Re, L=Hamilton, S=Hamilton, C=BM, issued by: CN=contoso-CA, DC=contoso, DC=com. Validation error code was: 800B0109.

Resolution:

Please check the remote server and ensure that the certificate is valid. Also ensure that the full certificate chain of the Issuer is present in the local machine. If the remote certificate and chain appear to be valid and error code is 0x800B0109 (CERT_E_UNTRUSTEDROOT), check that the ROOT certificate store on the local machine does not contain any intermediate certificates (certificates with different values in 'Issued To' and 'Issued By' fields do not belong to the ROOT store and cause client certificate validation errors in HTTP.SYS)

The following warning is also logged:

Log Name: Microsoft-Service Fabric/Admin

Source: Microsoft-Service Fabric

Event ID: 4097

Level: Error

ignore error 0x80092013:certificate revocation list offline

You attempt to navigate to the directory:

C:\Program Files\Skype for Business Server 2019\Server\Core

… and edit the file:

ClusterManifests.Xml.Template

Changing the flag:

<Parameter Name="CrlCheckingFlag" Value="%CRLCHECKINGFLAG%" />

… to:

<Parameter Name="CrlCheckingFlag" Value="0" />

… which should disable CRL Checking for the certificates but this does not correct the issue.

Solution

The solution to this problem can actually be found in the previous error log:

Note the following text highlighted in red:

In the past 30.0093507983333 minutes the process RtcHost(6756) received 1 invalid certificates. The last one was from server: contsfbstd01.contoso.com, IP Address: 10.198.40.152:60873, with subject: CN=contsfbstd01.contoso.com, OU=IT, O=contoso Re, L=Hamilton, S=Hamilton, C=BM, issued by: CN=contoso-CA, DC=contoso, DC=com. Validation error code was: 800B0109.

Resolution:

Please check the remote server and ensure that the certificate is valid. Also ensure that the full certificate chain of the Issuer is present in the local machine. If the remote certificate and chain appear to be valid and error code is 0x800B0109 (CERT_E_UNTRUSTEDROOT), check that the ROOT certificate store on the local machine does not contain any intermediate certificates (certificates with different values in 'Issued To' and 'Issued By' fields do not belong to the ROOT store and cause client certificate validation errors in HTTP.SYS)

The reason why the front-end service is unable to start is because there is a certificate stored in the Trusted Root Certification Authority that isn’t actually a Root certificate.  To check this, load the local computer’s certificate store (certlm.msc) and review the certificates in the Trusted Root Certification Authority ensuring that the Issued To matches the Issued By word for work.

The following is a screenshot of the offending certificate I found in the Trusted Root Certification Authority where the Issued To is arersa01.domain.com while the Issued By is RSA root CA for arersa01.domain.com:

Opening the properties of this certificate will show that it is actually an Intermediate Certification Authority certificate:

Either removing the certificate or placing it in the appropriate datastore will correct the issue.

Note that having improperly placed certificates in certificate stores are known to cause service start and replication issues.  The following are a few of my older posts of Skype for Business / Lync Server environments:

Lync Server Access Edge service fails to start with: “… service-specific error code -2146762487”
http://terenceluk.blogspot.com/2013/05/lync-server-access-edge-service-fails.html

Lync Server 2013 Edge server replication issues on Windows Server 2012
http://terenceluk.blogspot.com/2013/04/lync-server-2013-edge-server.html

Deploying Skype for Business Server 2019 on Windows Server 2019 in a Skype for Business Server 2015 environment

I’ve recently had the opportunity to deploy Skype for Business Server 2019 on Windows Server 2019 in a Skype for Business Server 2015 environment and decided to capture the process so I can write this blog post demonstrating what the deployment process looks like.

Before I proceed, the deployment guide I will be using can be found here:

Skype for Business Server 2019
https://docs.microsoft.com/en-us/skypeforbusiness/skype-for-business-server-2019

Prerequisites

Forest and Domain Functional Level

Verify that the forest and domain functional level is at one of the following levels:

• Windows Server 2019
• Windows Server 2016
• Windows Server 2012 R2
• Windows Server 2012

Windows Roles and Features

Install the required roles and features with the following PowerShell cmdlet:

Add-WindowsFeature RSAT-ADDS, Web-Server, Web-Static-Content, Web-Default-Doc, Web-Http-Errors, Web-Asp-Net, Web-Net-Ext, Web-ISAPI-Ext, Web-ISAPI-Filter, Web-Http-Logging, Web-Log-Libraries, Web-Request-Monitor, Web-Http-Tracing, Web-Basic-Auth, Web-Windows-Auth, Web-Client-Auth, Web-Filtering, Web-Stat-Compression, Web-Dyn-Compression, NET-WCF-HTTP-Activation45, Web-Asp-Net45, Web-Mgmt-Tools, Web-Scripting-Tools, Web-Mgmt-Compat, Server-Media-Foundation, Telnet-Client

Skype for Business Server 2015 Coexistence

I could not find any documentation identifying which CU a legacy SfB 2015 environment needs to be in order for coexistence to work properly as the following article is the only section I found in the document:

https://docs.microsoft.com/en-us/skypeforbusiness/migration/apply-updates

I ended up going with the latest January 2019 (CU8) for the environment I was installing SfB 2019 in and have not had any problems.

Preparing Install for Skype for Business Server 2019

Launch the Setup.exe executable as administrator:

Those who have installed SfB 2015 would find that the interface hasn’t changed much:

As there is already a Skype for Business Server 2015 deployment in the environment, I would not need to run the Prepare first Standard Edition server until I am ready to remove the legacy pool (https://docs.microsoft.com/en-us/skypeforbusiness/migration/move-the-central-management-server). I’ve also noticed that the Prepare Active Directory step already had a Complete check mark beside it, which most likely because there hasn’t been any changes from SfB 2015.

Installing Administrative Tools

Proceed to run Install Administrative Tools:

Defining Skype for Business Server 2019 Topology

Launch the Topology Builder from the start menu and download the existing topology:

Begin by defining a file store for the new Skype for Business Server 2019 environment by navigate to Shared Components > File stores then right click and select New File Store…:

Fill in the FQDN (make sure you use the FQDN and not the short server name) of the file server that will host the SfB files and the share name:

With the file store defined, proceed to create the new standard front-end server by navigating to Skype for Business Server 2019 > Standard Edition Front End Servers then right click and select New Front End Pool..:

Enter the FQDN of the server name that the front-end services will be installed onto:

Select the appropriate features required for the front-end server:

As this is a standard edition deployment, a local SQL Server Express will be installed:

Select the previously defined file store:

**Note that the path in the screenshot below should be the FQDN of the server.

Fill in the external URL for the Web Services URL:

Associate the existing or new Office Web Apps Server if one exists in the environment:

The new front-end server should now be created:

Proceed to publish the topology:

Review the warnings to ensure that they are not deployment impacting.

For those who are interested, the required security permissions for the file store that was defined earlier are automatically configured after publishing the topology:

Installing Skype For Business Server 2019 Front-End Server

Proceed and run Install or Update Skype for Business Server System:

Run the Install Local Configuration Store:

Proceed to run the Setup or Remove Skype for Business Server Components:

With the components successfully installed, proceed to request, install and assign the certificates:

Note that the OAuthTokenIssuer already had a certificate issued because there is an existing SfB 2015 deployment in the environment.

Depending on the way you’ll be publishing the Web services external service, you may want to assign a certificate issued by a public Certificate Authority but for the purpose of this demonstration, we’ll create a certificate for all 3 services from an internal Microsoft Enterprise CA:

Proceed to leave the Assign this certificate to Skype for Business Server certificate usages checked and click Finish:

Assign the certificate to the services:

All of the services should now have a check mark beside them:

Start Skype for Business Server 2019 Services

Scroll down to the Start Services section:

Click on the Run button for Service Status (Optional) to bring up the Services console:

Run the Start-CsWindowsService cmdlet in the Skype for Business Server Management Shell to start all of the services:

Confirm that all the services start:

Launch the Skype for Business Server 2019 Control Panel to confirm accessibility:

You can now continue with any other migration tasks such as moving pilot users over to verify functionality and/or migrate over services over.