Thursday, July 15, 2010

Voicemails left on Exchange UM does not get delivered to inbox – Event ID 1082 and 1035 logged

I ran into an interesting problem when a company out West purchased emergency service from us regarding their voicemail not being delivered to their inbox. Their environment consists of Exchange 2007 with UM and a Mitel phone system integrated with OCS enterprise voice through and Audiocodes Mediant 1000 gateway. The client had just re-IP-ed the majority of their servers but left the OCS, gateway and Mitel servers untouched. People were able to call and reach their Exchange UM voicemail, leave a message, but the messages are never delivered to their Exchange mailbox. The problem lasted for 3 days before they decided to ask us for help.

Without looking at the environment, I originally thought the problem was DNS not updating but once I logged into the environment after hours, I discovered it was more complicated than that. Issue was resolved within 30 minutes but the report to the customer took a lot longer to type and here it is:

The following outlines what was discovered and the resolution for the issue between Exchange Hub/CAS/Mailbox and Exchange UM.

Problem Description

When calling a into any user’s extension, the caller can leave a voicemail but the voicemail is never delivered to the called user’s mailbox. The issue began after the network readdressing from 172.26.x.x to 10.3.x.x. All DNS records were updated appropriately.

Analysis/Troubleshooting

One of the errors being logged on EXCH-UM (Unified Messaging Server) provides the following information.

Event Type: Error

Event Source: MSExchange Unified Messaging

Event Category: UMService

Event ID: 1082

Date: 6/3/2010

Time: 12:26:25 PM

User: N/A

Computer: EXCH-UM

Description:

The Unified Messaging server was unable to submit messages to a Hub Transport server because there is no Hub Transport server available to process the request with UM header file "C:\Program Files\Microsoft\Exchange Server\UnifiedMessaging\voicemail\04b3668f-6af8-4da7-aa4f-3272a74051cd.txt". Make sure that there is a Hub Transport server located in the same Active Directory site as the UM server. In addition, make sure that the Microsoft Exchange Transport service is started on the Hub Transport server.

Another warning being logged on EXCH-02 (Exchange Server) provides the following message:

Event Type: Warning

Event Source: MSExchangeTransport

Event Category: SmtpReceive

Event ID: 1035

Date: 5/31/2010

Time: 12:09:15 AM

User: N/A

Computer: EXCH-02

Description:

Inbound authentication failed with error IllegalMessage for Receive connector Default EXCH-02. The authentication mechanism is ExchangeAuth. The source IP address of the client who tried to authenticate to Microsoft Exchange is [172.26.186.14].

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

The 2 logs provided by The Company appears to suggest that there are communication problems between the hub transport server and the UM server (both separated on different physical servers).

The first error message suggests that there may be an issue with the physical topology of Active Directory. However, a quick review of the physical topology shows that the appropriate subnets (new and old) have been added to the appropriate sites:

image

I double checked the subnet of 172.26.176.1/20 and it includes a host address range of 172.26.176.1 to 172.26.191.254 which includes the 2 Exchange server’s IP:

Server

Name

IP

Exchange Hub/CAS/MB

EXCH-02

172.26.186.49

Exchange UM

EXCH-UM

172.26.186.14

The second warning entry actually provided a more important clue of what the cause might be since it shows that Exchange UM does indeed start a session with Exchange Hub but the communication between the 2 servers runs into a problem when authenticating.

If we look at the logs on the Exchange UM server, we’ll notice that there’s actually another warning log that has been flooding the event logs since the IP change:

image

This warning has been logged since 5/30/2010 @ 8:50:44PM.

The details of the error are as follows:

The Unified Messaging server was unable to submit a message to Hub Transport server "EXCH-02" because the following error occurred: Unexpected SMTP server response. Expected: 220, actual: 500, whole response: 500 5.3.3 Unrecognized command.

I did a simple test from Exchange UM to Exchange Hub to see what an EHLO command outputs and here are the results:

From EXCH-UM.domain.local to EXCH-01.domain.local:

220 exch-02.domain.local Microsoft ESMTP MAIL Service ready at Thu, 3 Jun 20

10 16:25:34 -0600

EHLO

250-exch-02.domain.local Hello [172.26.186.14]

250-SIZE

250-PIPELINING

250-DSN

250-ENHANCEDSTATUSCODES

250-AUTH

250-8BITMIME

250-BINARYMIME

250-CHUNKING

250 XEXCH50

An important 250 response required for UM to Hub communication is actually: 250-X-ANONYMOUSTLS.

The next step is to review the receive connectors on the hub transport server (EXCH-02).

image

There are 3 receive connectors. The only configured connector that will actually allow Exchange UM to connect properly is the “Default EXCH-02” connector. All the other ones do not have the proper settings to allow Exchange UM to establish a connection.

If we review each connector, we will notice that the “Internal EXCH-02” connector actually has specific IP addresses defined while the “Default EXCH-02” connector does not. Since the former has specific IP addresses defined, it will take precedence over the other.

image

If we look at the authentication and permissions, we’ll notice that while the permissions are set to allow UM connectivity, the Authentication isn’t:

image

Resolution

Rather than modify The company’s existing connectors, I went ahead and created a new one as shown here that has only one IP defined (EXCH-UM) so the receive connector catches all Exchange UM connections:






image

After creating new “EXCH-UM to EXCH-02”, a telnet session from Exchange UM to Exchange UM @ port 25 displays the following after an EHLO command:

220 exch-02.domain.local Microsoft ESMTP MAIL Service ready at Thu, 3 Jun 20

10 16:34:20 -0600

EHLO

250-exch-02.domain.local Hello [172.26.186.14]

250-SIZE 10485760

250-PIPELINING

250-DSN

250-ENHANCEDSTATUSCODES

250-STARTTLS

250-X-ANONYMOUSTLS

250-AUTH NTLM

250-X-EXPS GSSAPI NTLM

250-8BITMIME

250-BINARYMIME

250-CHUNKING

250-XEXCH50

250 XRDST

----------------------------------------------------------------------------------------------------------------

I hope this post helps anyone out there that may experience the same problems as I did.

7 comments:

Anonymous said...

Thank you very much. This post saved my day. KR, C.A.

Anonymous said...

Thanks this hlped me immensly.

Anonymous said...

Thank you! I was having the same issue. I never knew UM used this type of connection.

Anonymous said...

Solved the problem. Thanks!

Anonymous said...

Saved my bacon today. you're the man.

Anonymous said...

Thanks for the tip! In my environment I have a single Exchange 2013 server running all roles. I do have some custom receive connectors so I had to add the local IPs of the server to the "allow remote mail servers". Seems silly but it was necessary for my UM server to send out.

Anonymous said...

I also meant to indicate in the last post that I did this on the "Default Frontend " receive connector instead of creating a new connector.