Voicemails left with Skype for Business Server 2015 on Exchange Server 2016 does not get delivered to user’s inbox

Problem

You’ve just completed integrating Skype for Business Server 2015 with Exchange Server 2016 Unified Messaging to provide voicemail services but noticed that voicemails left for users are not being delivered and the following errors are being logged on the Exchange 2016 server:

Log Name: Application

Source: MSExchange Unified Messaging

Event ID: 1423

Level: Error

The Microsoft Exchange Unified Messaging service on the Mailbox server encountered an error while trying to process the message with header file "E:\Program Files\Microsoft\Exchange Server\V15\UnifiedMessaging\voicemail\c7d3e3ee-0212-4d27-99bd-f1960eca0698.txt". Error details: "Microsoft.Exchange.UM.UMCore.SmtpSubmissionException: Submission to the Hub Transport server failed. The operation will be retried. ---> Microsoft.Exchange.Net.ExSmtpClient.UnexpectedSmtpServerResponseException: Unexpected SMTP server response. Expected: 220, actual: 421, whole response: 421 4.3.2 Service not available

at Microsoft.Exchange.Net.ExSmtpClient.SmtpTalk.CheckResponse(ServerResponseInfo response, Int32 expectedCode)

at Microsoft.Exchange.Net.ExSmtpClient.SmtpTalk.Connect(String server, Int32 port, Boolean isSslPort)

at Microsoft.Exchange.Net.ExSmtpClient.SmtpClient.Submit()

at Microsoft.Exchange.UM.UMCore.SmtpSubmissionHelper.SubmitMessage(MessageItem message, String senderAddress, Guid senderOrgGuid, String recipientAddress, OutboundConversionOptions submissionConversionOptions, InternalExchangeServer smtpServer)

at Microsoft.Exchange.UM.UMCore.SmtpSubmissionHelper.SubmitMessage(MessageItem message, String senderAddress, Guid senderOrgGuid, String recipientAddress, OutboundConversionOptions submissionConversionOptions, String requestId)

--- End of inner exception stack trace ---

Server stack trace:

at Microsoft.Exchange.UM.UMCore.SmtpSubmissionHelper.HandleTransientSmtpFailure(Exception e, InternalExchangeServer smtpServer, String recipientAddress)

at Microsoft.Exchange.UM.UMCore.SmtpSubmissionHelper.SubmitMessage(MessageItem message, String senderAddress, Guid senderOrgGuid, String recipientAddress, OutboundConversionOptions submissionConversionOptions, String requestId)

at Microsoft.Exchange.UM.UMCore.SmtpSubmitStage.InternalDoSynchronousWork()

at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)

at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)

Exception rethrown at [0]:

at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase)

at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData& msgData)

at Microsoft.Exchange.UM.UMCore.SynchronousPipelineStageBase.SynchronousWorkDelegate.EndInvoke(IAsyncResult result)

at Microsoft.Exchange.UM.UMCore.SynchronousPipelineStageBase.EndSynchronousWork(IAsyncResult r)"

Log Name: Application

Source: MSExchange Unified Messaging

Event ID: 1446

Level: Error

The Microsoft Exchange Unified Messaging service on the Mailbox server failed to process the message with header file "E:\Program Files\Microsoft\Exchange Server\V15\UnifiedMessaging\voicemail\1fbf0f79-adeb-4b6c-a7e2-792ec27a5351.txt" within "11" minutes. The server will continue to process and deliver the message, but the "MSExchangeUMAvailability: % of Messages Successfully Processed Over the Last Hour" performance counter will be decreased.

Solution

Similar to one of my previous posts from 7 years ago:

Voicemails left on Exchange UM does not get delivered to inbox – Event ID 1082 and 1035 logged
http://terenceluk.blogspot.com/2010/07/voicemails-left-on-exchange-um-does-not.html

… the issue was caused by the receiving connector on the Exchange server which hosts the Unified Messaging role (note that Exchange 2016 no longer splits roles into other servers).  The following snippet from the event ID 1423 error logged displays what response the Exchange server is receiving when an attempt to deliver the voice is made:

The Microsoft Exchange Unified Messaging service on the Mailbox server encountered an error while trying to process the message with header file "E:\Program Files\Microsoft\Exchange Server\V15\UnifiedMessaging\voicemail\c7d3e3ee-0212-4d27-99bd-f1960eca0698.txt". Error details: "Microsoft.Exchange.UM.UMCore.SmtpSubmissionException: Submission to the Hub Transport server failed. The operation will be retried. ---> Microsoft.Exchange.Net.ExSmtpClient.UnexpectedSmtpServerResponseException: Unexpected SMTP server response. Expected: 220, actual: 421, whole response: 421 4.3.2 Service not available

You can verify this by logging onto one of the Exchange servers and initiate a telnet to port 25 to another as shown here:

421 4.3.2 Service not available

Connection to host lost.

C:\>

If the environment you’re working in has many receive connectors configured:

… my suggestion is to not to attempt figuring out which routing connector may be receiving the request and rejecting the connection but rather create new ones specifically for UM with the the scope configured with the Exchange server IP address and the following security configuration:

You should see the queued files stored in the folder:

… begin to clear out when the Exchange server begins accepting the voicemail delivery connections.

VMware Horizon View 7.4.0 virtual desktops does not automatically power on after upgrade

I recently upgraded a client’s VMware Horizon View 7.0.2 infrastructure to the latest 7.4.0 version Build 7400497:

… and received reports the next day that virtual desktops were not being started up when the user had shut them down even though the Remote Machine Power Policy was configured as Ensure machines are always powered on:

After reviewing the event logs and performing various troubleshooting steps without any luck, I gave the recommendation VMware support always suggested:

Shut down all View servers then power them on one after another to allow the database to synchronize properly

… a try and was able to correct the issue.  Hope this post will help anyone who may encounter this problem.  Please refer to the following KB for a more detailed outline of the steps for restarting the servers:

Restart order of the View environment to clear ADLDS (ADAM) synchronization in Horizon View (2068381)
https://kb.vmware.com/s/article/2068381

Upgrading VMware Horizon View from 7.0.2 to 7.4.0 causes VMware Horizon Composers to display a Red status in the View Administrator with "The service is not working properly" and SSL as "Unknown"

Problem

You have just successfully upgraded your VMware Horizon View 7.0.2 environment to 7.4.0 but noticed that the System Health of the View Composer Servers now displays a Red status with The service is not working properly and SSL as Unknown with no option of accepting the self signed certificate that the Composer service is using as you would normally expect to see.

The View 7.4.0 environment is currently using an older vSphere 5.5 environment and you have confirmed that TLS 1.0 has been enabled as per the following document:

Enable TLSv1.0 on vCenter and ESXi Connections from View Compose

https://docs.vmware.com/en/VMware-Horizon-7/7.4/horizon-upgrades/GUID-1845E4AD-E84A-4304-A7DD-83170B8D21C5.html

Reviewing the event logs on the View Connection server show that the following error is logged:

Log Name: Application
Source: VMware

Event ID: 105

Level: Error

User: System

BROKER_SVI_CERT_INVALID

Certificate is invalid for Composer at address https://contosodrvc01.contoso.com:18443

Attributes:

            Node=contosoUKVV01.contoso.com

            ComposerId=https://contosodrvc01.contoso.com:18443

            Severity=ERROR

            Time=Fri Feb 02 22:01:58 GMT 2018

            Module=Broker

            Source=com.vmware.vdi.desktopcontroller.PublishVcCertToSviFederatedTask

            Acknowledged=true

The Events in the View Administrator console display the following message:

Certificate is invalid for Composer at address https://<vcenter>.FQDN.com:18443

Attempting to view the properties of a linked-clone pool displays the following error:

The certificate configured on View Comopser Server is invalid, blocking communication with this server. To resume communication, replace the certificate with a valid certificate signed by a CA. Alternatively, accept the certificate thumbprint by clicking Verify in the View Administrator dashboard.

You’ve confirmed that the correct self-signed certificate is assigned to the VMware Horizon View Composer service by executing the command:

sviconfig -operation=ReplaceCertificate -delete=false

You’ve tried changing the pae-SVIURL to the short NetBIOS name instead of the FQDN:

Solution

If you do not include on issuing a certificate from a trusted Certificate Authority and would like to use the self-signed certificate, the way to accept it is to navigate to the View Configuration > Servers menu, select the vCenter instance then click Edit to bring up the vCenter Server settings, then click Edit under the View Composer Server Settings section:

Getting into the configuration settings of the View Composer will now display the following invalid certificate detected prompt:

Click on the View Certificate… button will display the option to accept the self-signed certificate:

Accepting the certificate should now display the View Composer Servers health as green: