Thursday, February 22, 2018

Voicemails left with Skype for Business Server 2015 on Exchange Server 2016 does not get delivered to user’s inbox


You’ve just completed integrating Skype for Business Server 2015 with Exchange Server 2016 Unified Messaging to provide voicemail services but noticed that voicemails left for users are not being delivered and the following errors are being logged on the Exchange 2016 server:

Log Name: Application

Source: MSExchange Unified Messaging

Event ID: 1423

Level: Error


The Microsoft Exchange Unified Messaging service on the Mailbox server encountered an error while trying to process the message with header file "E:\Program Files\Microsoft\Exchange Server\V15\UnifiedMessaging\voicemail\c7d3e3ee-0212-4d27-99bd-f1960eca0698.txt". Error details: "Microsoft.Exchange.UM.UMCore.SmtpSubmissionException: Submission to the Hub Transport server failed. The operation will be retried. ---> Microsoft.Exchange.Net.ExSmtpClient.UnexpectedSmtpServerResponseException: Unexpected SMTP server response. Expected: 220, actual: 421, whole response: 421 4.3.2 Service not available

at Microsoft.Exchange.Net.ExSmtpClient.SmtpTalk.CheckResponse(ServerResponseInfo response, Int32 expectedCode)

at Microsoft.Exchange.Net.ExSmtpClient.SmtpTalk.Connect(String server, Int32 port, Boolean isSslPort)

at Microsoft.Exchange.Net.ExSmtpClient.SmtpClient.Submit()

at Microsoft.Exchange.UM.UMCore.SmtpSubmissionHelper.SubmitMessage(MessageItem message, String senderAddress, Guid senderOrgGuid, String recipientAddress, OutboundConversionOptions submissionConversionOptions, InternalExchangeServer smtpServer)

at Microsoft.Exchange.UM.UMCore.SmtpSubmissionHelper.SubmitMessage(MessageItem message, String senderAddress, Guid senderOrgGuid, String recipientAddress, OutboundConversionOptions submissionConversionOptions, String requestId)

--- End of inner exception stack trace ---

Server stack trace:

at Microsoft.Exchange.UM.UMCore.SmtpSubmissionHelper.HandleTransientSmtpFailure(Exception e, InternalExchangeServer smtpServer, String recipientAddress)

at Microsoft.Exchange.UM.UMCore.SmtpSubmissionHelper.SubmitMessage(MessageItem message, String senderAddress, Guid senderOrgGuid, String recipientAddress, OutboundConversionOptions submissionConversionOptions, String requestId)

at Microsoft.Exchange.UM.UMCore.SmtpSubmitStage.InternalDoSynchronousWork()

at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)

at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)

Exception rethrown at [0]:

at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase)

at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData& msgData)

at Microsoft.Exchange.UM.UMCore.SynchronousPipelineStageBase.SynchronousWorkDelegate.EndInvoke(IAsyncResult result)

at Microsoft.Exchange.UM.UMCore.SynchronousPipelineStageBase.EndSynchronousWork(IAsyncResult r)"


Log Name: Application

Source: MSExchange Unified Messaging

Event ID: 1446

Level: Error


The Microsoft Exchange Unified Messaging service on the Mailbox server failed to process the message with header file "E:\Program Files\Microsoft\Exchange Server\V15\UnifiedMessaging\voicemail\1fbf0f79-adeb-4b6c-a7e2-792ec27a5351.txt" within "11" minutes. The server will continue to process and deliver the message, but the "MSExchangeUMAvailability: % of Messages Successfully Processed Over the Last Hour" performance counter will be decreased.



Similar to one of my previous posts from 7 years ago:

Voicemails left on Exchange UM does not get delivered to inbox – Event ID 1082 and 1035 logged

… the issue was caused by the receiving connector on the Exchange server which hosts the Unified Messaging role (note that Exchange 2016 no longer splits roles into other servers).  The following snippet from the event ID 1423 error logged displays what response the Exchange server is receiving when an attempt to deliver the voice is made:

The Microsoft Exchange Unified Messaging service on the Mailbox server encountered an error while trying to process the message with header file "E:\Program Files\Microsoft\Exchange Server\V15\UnifiedMessaging\voicemail\c7d3e3ee-0212-4d27-99bd-f1960eca0698.txt". Error details: "Microsoft.Exchange.UM.UMCore.SmtpSubmissionException: Submission to the Hub Transport server failed. The operation will be retried. ---> Microsoft.Exchange.Net.ExSmtpClient.UnexpectedSmtpServerResponseException: Unexpected SMTP server response. Expected: 220, actual: 421, whole response: 421 4.3.2 Service not available

You can verify this by logging onto one of the Exchange servers and initiate a telnet to port 25 to another as shown here:

421 4.3.2 Service not available

Connection to host lost.



If the environment you’re working in has many receive connectors configured:


… my suggestion is to not to attempt figuring out which routing connector may be receiving the request and rejecting the connection but rather create new ones specifically for UM with the the scope configured with the Exchange server IP address and the following security configuration:


You should see the queued files stored in the folder:


… begin to clear out when the Exchange server begins accepting the voicemail delivery connections.

Thursday, February 15, 2018

VMware Horizon View 7.4.0 virtual desktops does not automatically power on after upgrade

I recently upgraded a client’s VMware Horizon View 7.0.2 infrastructure to the latest 7.4.0 version Build 7400497:



… and received reports the next day that virtual desktops were not being started up when the user had shut them down even though the Remote Machine Power Policy was configured as Ensure machines are always powered on:


After reviewing the event logs and performing various troubleshooting steps without any luck, I gave the recommendation VMware support always suggested:

Shut down all View servers then power them on one after another to allow the database to synchronize properly

… a try and was able to correct the issue.  Hope this post will help anyone who may encounter this problem.  Please refer to the following KB for a more detailed outline of the steps for restarting the servers:

Restart order of the View environment to clear ADLDS (ADAM) synchronization in Horizon View (2068381)

Saturday, February 3, 2018

Upgrading VMware Horizon View from 7.0.2 to 7.4.0 causes VMware Horizon Composers to display a Red status in the View Administrator with "The service is not working properly" and SSL as "Unknown"


You have just successfully upgraded your VMware Horizon View 7.0.2 environment to 7.4.0 but noticed that the System Health of the View Composer Servers now displays a Red status with The service is not working properly and SSL as Unknown with no option of accepting the self signed certificate that the Composer service is using as you would normally expect to see.

The View 7.4.0 environment is currently using an older vSphere 5.5 environment and you have confirmed that TLS 1.0 has been enabled as per the following document:

Enable TLSv1.0 on vCenter and ESXi Connections from View Compose

Reviewing the event logs on the View Connection server show that the following error is logged:

Log Name: Application
Source: VMware

Event ID: 105

Level: Error

User: System


Certificate is invalid for Composer at address





            Time=Fri Feb 02 22:01:58 GMT 2018





The Events in the View Administrator console display the following message:

Certificate is invalid for Composer at address https://<vcenter>


Attempting to view the properties of a linked-clone pool displays the following error:

The certificate configured on View Comopser Server is invalid, blocking communication with this server. To resume communication, replace the certificate with a valid certificate signed by a CA. Alternatively, accept the certificate thumbprint by clicking Verify in the View Administrator dashboard.


You’ve confirmed that the correct self-signed certificate is assigned to the VMware Horizon View Composer service by executing the command:

sviconfig -operation=ReplaceCertificate -delete=false


You’ve tried changing the pae-SVIURL to the short NetBIOS name instead of the FQDN:



If you do not include on issuing a certificate from a trusted Certificate Authority and would like to use the self-signed certificate, the way to accept it is to navigate to the View Configuration > Servers menu, select the vCenter instance then click Edit to bring up the vCenter Server settings, then click Edit under the View Composer Server Settings section:


Getting into the configuration settings of the View Composer will now display the following invalid certificate detected prompt:


Click on the View Certificate… button will display the option to accept the self-signed certificate:


Accepting the certificate should now display the View Composer Servers health as green:


Saturday, January 6, 2018

Patching a VMware ESXi 5.5 host offline via CLI

One of the more common questions I’m asked every year is how to patch an ESXi host without using VMware Update Manager because it is quite common for ESXi hosts as well as vCenter to not have internet access. What I typically do when asked is point them to one of my previous posts where I demonstrated the process for a Nutanix infrastructure:

Upgrading Nutanix blades from ESXi 5.1 to 5.5

… because the post includes the command for non-Nutanix deployments at the end but since I’ve received follow up questions about the process, I thought it would be a good idea to write a post that specifically demonstrates this for any ESXi deployments.

Step #1 – Identify and download desired ESXi build level

Begin by identifying the which build you’d like to patch the ESXi to:


With the build number identified, proceed to the following URL to locate and download the patch:



Download the patch:


The package you download should be a ZIP file containing files similar to the following screenshot:





Step #2 – Upload patch to datastore

Upload the package to a datastore that the host you’re patching has access to by using a utility such as WinSCP:


Step #3 – Patch ESXi host via CLI

Proceed to either access the console directly on the server or SSH to it.


The command we’ll be using to install the patch will look as such:

esxcli software vib install -d <full path to patch file>

Note that it is important to specify the full path to the patch file because if you don’t then the follow error will be thrown:

/vmfs/volumes/58da7103-7810a474-5615-000eb6b4b6ea # esxcli software vib install



Could not download from depot at zip:/var/log/vmware/, skipping (('zip:/var/log/vmware/', '', "Error extracting index.xml from /var/log/vmware/ [Errno 2] No such file or directory: '/var/log/vmware/'"))

url = zip:/var/log/vmware/

Please refer to the log file for more details.

/vmfs/volumes/58da7103-7810a474-5615-000eb6b4b6ea #


Forgetting to include the -d will throw the following error:

/vmfs/volumes/58da7103-7810a474-5615-000eb6b4b6ea # esxcli software vib install


Error: Unknown command or namespace software vib install /vmfs/volumes/rvbd_vsp_datastore/


The patch will successfully install if the full path is specified and the output would look as such:

/vmfs/volumes/58da7103-7810a474-5615-000eb6b4b6ea # esxcli software vib install

-d /vmfs/volumes/rvbd_vsp_datastore/

Installation Result

Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

Reboot Required: true

VIBs Installed: VMware_bootbank_ehci-ehci-hcd_1.0-3vmw.550.3.95.4345813, VMware_bootbank_esx-base_5.5.0-3.100.4722766, VMware_bootbank_esx-tboot_5.5.0-3.100.4722766, VMware_bootbank_esx-ui_1.12.0-4722375, VMware_bootbank_lsi-msgpt3_00.255.03.03-2vmw.550.3.78.3248547, VMware_bootbank_misc-drivers_5.5.0-3.95.4345813, VMware_bootbank_net-e1000e_3.2.2.1-2vmw.550.3.78.3248547, VMware_bootbank_net-igb_5., VMware_bootbank_net-ixgbe_3., VMware_bootbank_sata-ahci_3.0-22vmw.550.3.89.4179633, VMware_bootbank_xhci-xhci_1.0-3vmw.550.3.95.4345813, VMware_locker_tools-light_5.5.0-3.92.4345810

VIBs Removed: Intel_bootbank_net-igb_5.2.7-1OEM.550.0.0.1331820, Intel_bootbank_net-ixgbe_3.21.4-1OEM.550.0.0.1331820, VMware_bootbank_ehci-ehci-hcd_1.0-3vmw.550.0.0.1331820, VMware_bootbank_esx-base_5.5.0-3.71.3116895, VMware_bootbank_esx-tboot_5.5.0-2.33.2068190, VMware_bootbank_lsi-msgpt3_00.255.03.03-1vmw.550.1.15.1623387, VMware_bootbank_misc-drivers_5.5.0-3.68.3029944, VMware_bootbank_net-e1000e_3.2.2.1-2vmw.550.3.68.3029944, VMware_bootbank_sata-ahci_3.0-22vmw.550.3.68.3029944, VMware_bootbank_xhci-xhci_1.0-2vmw.550.3.68.3029944, VMware_locker_tools-light_5.5.0-3.68.3029944

VIBs Skipped: VMware_bootbank_ata-pata-amd_0.3.10-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-atiixp_0.4.6-4vmw.550.0.0.1331820, VMware_bootbank_ata-pata-cmd64x_0.2.5-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-pdc2027x_1.0-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-serverworks_0.4.3-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-sil680_0.4.8-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-via_0.3.3-2vmw.550.0.0.1331820, VMware_bootbank_block-cciss_3.6.14-10vmw.550.0.0.1331820, VMware_bootbank_elxnet_10.2.309.6v-1vmw.550.3.68.3029944, VMware_bootbank_esx-dvfilter-generic-fastpath_5.5.0-0.0.1331820, VMware_bootbank_esx-xlibs_5.5.0-0.0.1331820, VMware_bootbank_esx-xserver_5.5.0-0.0.1331820, VMware_bootbank_ima-qla4xxx_2.01.31-1vmw.550.0.0.1331820, VMware_bootbank_ipmi-ipmi-devintf_39.1-4vmw.550.0.0.1331820, VMware_bootbank_ipmi-ipmi-msghandler_39.1-4vmw.550.0.0.1331820, VMware_bootbank_ipmi-ipmi-si-drv_39.1-4vmw.550.0.0.1331820, VMware_bootbank_lpfc_10.0.100.1-1vmw.550.0.0.1331820, VMware_bootbank_lsi-mr3_0.255.03.01-2vmw.550.3.68.3029944, VMware_bootbank_misc-cnic-register_1.72.1.v50.1i-1vmw.550.0.0.1331820, VMware_bootbank_mtip32xx-native_3.3.4-1vmw.550.1.15.1623387, VMware_bootbank_net-be2net_4.6.100.0v-1vmw.550.0.0.1331820, VMware_bootbank_net-bnx2_2.2.3d.v55.2-1vmw.550.0.0.1331820, VMware_bootbank_net-bnx2x_1.72.56.v55.2-1vmw.550.0.0.1331820, VMware_bootbank_net-cnic_1.72.52.v55.1-1vmw.550.0.0.1331820, VMware_bootbank_net-e1000_8.0.3.1-3vmw.550.0.0.1331820, VMware_bootbank_net-enic_1.4.2.15a-1vmw.550.0.0.1331820, VMware_bootbank_net-forcedeth_0.61-2vmw.550.0.0.1331820, VMware_bootbank_net-mlx4-core_1.9.7.0-1vmw.550.0.0.1331820, VMware_bootbank_net-mlx4-en_1.9.7.0-1vmw.550.0.0.1331820, VMware_bootbank_net-nx-nic_5.0.621-1vmw.550.0.0.1331820, VMware_bootbank_net-tg3_3.123c.v55.5-1vmw.550.2.33.2068190, VMware_bootbank_net-vmxnet3_1.1.3.0-3vmw.550.2.39.2143827, VMware_bootbank_ohci-usb-ohci_1.0-3vmw.550.0.0.1331820, VMware_bootbank_qlnativefc_1.0.12.0-1vmw.550.0.0.1331820, VMware_bootbank_rste_2.0.2.0088-4vmw.550.1.15.1623387, VMware_bootbank_sata-ata-piix_2.12-10vmw.550.2.33.2068190, VMware_bootbank_sata-sata-nv_3.5-4vmw.550.0.0.1331820, VMware_bootbank_sata-sata-promise_2.12-3vmw.550.0.0.1331820, VMware_bootbank_sata-sata-sil24_1.1-1vmw.550.0.0.1331820, VMware_bootbank_sata-sata-sil_2.3-4vmw.550.0.0.1331820, VMware_bootbank_sata-sata-svw_2.3-3vmw.550.0.0.1331820, VMware_bootbank_scsi-aacraid_1.1.5.1-9vmw.550.0.0.1331820, VMware_bootbank_scsi-adp94xx_1.0.8.12-6vmw.550.0.0.1331820, VMware_bootbank_scsi-aic79xx_3.1-5vmw.550.0.0.1331820, VMware_bootbank_scsi-bnx2fc_1.72.53.v55.1-1vmw.550.0.0.1331820, VMware_bootbank_scsi-bnx2i_2.72.11.v55.4-1vmw.550.0.0.1331820, VMware_bootbank_scsi-fnic_1.5.0.4-1vmw.550.0.0.1331820, VMware_bootbank_scsi-hpsa_5.5.0-44vmw.550.0.0.1331820, VMware_bootbank_scsi-ips_7.12.05-4vmw.550.0.0.1331820, VMware_bootbank_scsi-lpfc820_8.2.3.1-129vmw.550.0.0.1331820, VMware_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.550.0.0.1331820, VMware_bootbank_scsi-megaraid-sas_5.34-9vmw.550.3.68.3029944, VMware_bootbank_scsi-megaraid2_2.00.4-9vmw.550.0.0.1331820, VMware_bootbank_scsi-mpt2sas_14.00.00.00-3vmw.550.3.68.3029944, VMware_bootbank_scsi-mptsas_4.23.01.00-9vmw.550.3.68.3029944, VMware_bootbank_scsi-mptspi_4.23.01.00-9vmw.550.3.68.3029944, VMware_bootbank_scsi-qla2xxx_902.k1.1-12vmw.550.3.68.3029944, VMware_bootbank_scsi-qla4xxx_5.01.03.2-6vmw.550.0.0.1331820, VMware_bootbank_uhci-usb-uhci_1.0-3vmw.550.0.0.1331820

/vmfs/volumes/58da7103-7810a474-5615-000eb6b4b6ea #


Proceed to vMotion the VMs on this host to another host or shut them down if they don’t need to be up and restart the host:


The new build number should be displayed once the host has successfully restarted:


Wednesday, December 20, 2017

Unable to expand virtual machine hard disk that is backed up by VMware VDP


You have a virtual machine in your vSphere environment that you would like to expand one of the hard disks but as you browse to the hard disk that needs to get expanded, you see that it is locked:


You’ve checked to confirm that there are no snapshots on the virtual machine:


Reviewing the files for the virtual machine show that there are CTK files for the disks that are locked and unable to be expanded in capacity:


Reviewing the disk for the VM in the vSphere Web Client displays the same behavior:


Shutting down and reviewing the VM’s summary tab indicate that the disks need to be consolidated:

Configuration Issues

Virtual machine disks consolidation is needed.


Attempting to consolidate the disks throws the following error:

Consolidate virtual machine disk files

Unable to access file since it is locked

See the error stack for details on the cause of this problem.

Error Stack

An error occurred while consolidating disks: msg.snapshot.error-DISKLOCKED.




One of the reasons why this error would be thrown is if you have this virtual machine configured to be backed up by a VMware VDP appliance and the appliance hasn’t released the disk at the end of a backup job or perhaps it did not complete properly.  To determine whether this is the case, open the settings of the VDP appliance and review whether this virtual machine’s disk is attached to it as such:


You can verify the disks that belong to the VDR appliance by reviewing the administration console as such:


To correct the problem, simply remove the disk from the VDP appliance and select the Remove from virtual machine when prompted with the Removal Options (this does not delete the disk):


Note that the disk will still be locked at this point:


Proceed to consolidate the disks:


The disk should be unlocked once the consolidation task completes.

Wednesday, December 13, 2017

Attempting to launch Citrix XenApp / XenDesktop 7.x desktop or application published by NetScaler fails with the error: “The connection to “XenApp Desktop” failed with status (Unknown client error 1110).”


You attempt to launch a XenApp or XenDesktop published application or desktop published by NetScaler but receive the following error message:

The connection to “XenApp Desktop” failed with status (Unknown client error 1110).



One of the possible causes of this error message is if you have incorrect or have not configured any STA (Secure Ticket Authority) on your NetScalers.  Navigate to NetScaler Gateway > Virtual Servers, open the properties of the virtual server, scroll down to Published Applications and ensure STA Servers are configured:


Monday, December 4, 2017

Attempting to run Windows update on a Windows 7 desktop fails with the error code 80248015


You attempt to run Windows updates on a Windows 7 desktop but notice that it fails with the following message:

Windows could not search for new updates

An error occurred while checking for new updates for your computer.

Error(s) found:

Code 80248015

Windows Update encountered an unknown error.



One of the ways to correct this issue is to navigate to the Windows directory and rename the SoftwareDistribution folder so that it would get recreated. 



Restart the desktop once the folder has been renamed and Windows update should now work as expected: