Voicemails left on Exchange UM does not get delivered to inbox – Event ID 1082 and 1035 logged

I ran into an interesting problem when a company out West purchased emergency service from us regarding their voicemail not being delivered to their inbox. Their environment consists of Exchange 2007 with UM and a Mitel phone system integrated with OCS enterprise voice through and Audiocodes Mediant 1000 gateway. The client had just re-IP-ed the majority of their servers but left the OCS, gateway and Mitel servers untouched. People were able to call and reach their Exchange UM voicemail, leave a message, but the messages are never delivered to their Exchange mailbox. The problem lasted for 3 days before they decided to ask us for help.

Without looking at the environment, I originally thought the problem was DNS not updating but once I logged into the environment after hours, I discovered it was more complicated than that. Issue was resolved within 30 minutes but the report to the customer took a lot longer to type and here it is:

The following outlines what was discovered and the resolution for the issue between Exchange Hub/CAS/Mailbox and Exchange UM.

Problem Description

When calling a into any user’s extension, the caller can leave a voicemail but the voicemail is never delivered to the called user’s mailbox. The issue began after the network readdressing from 172.26.x.x to 10.3.x.x. All DNS records were updated appropriately.

Analysis/Troubleshooting

One of the errors being logged on EXCH-UM (Unified Messaging Server) provides the following information.

Event Type: Error

Event Source: MSExchange Unified Messaging

Event Category: UMService

Event ID: 1082

Date: 6/3/2010

Time: 12:26:25 PM

User: N/A

Computer: EXCH-UM

Description:

The Unified Messaging server was unable to submit messages to a Hub Transport server because there is no Hub Transport server available to process the request with UM header file "C:\Program Files\Microsoft\Exchange Server\UnifiedMessaging\voicemail\04b3668f-6af8-4da7-aa4f-3272a74051cd.txt". Make sure that there is a Hub Transport server located in the same Active Directory site as the UM server. In addition, make sure that the Microsoft Exchange Transport service is started on the Hub Transport server.

Another warning being logged on EXCH-02 (Exchange Server) provides the following message:

Event Type: Warning

Event Source: MSExchangeTransport

Event Category: SmtpReceive

Event ID: 1035

Date: 5/31/2010

Time: 12:09:15 AM

User: N/A

Computer: EXCH-02

Description:

Inbound authentication failed with error IllegalMessage for Receive connector Default EXCH-02. The authentication mechanism is ExchangeAuth. The source IP address of the client who tried to authenticate to Microsoft Exchange is [172.26.186.14].

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

The 2 logs provided by The Company appears to suggest that there are communication problems between the hub transport server and the UM server (both separated on different physical servers).

The first error message suggests that there may be an issue with the physical topology of Active Directory. However, a quick review of the physical topology shows that the appropriate subnets (new and old) have been added to the appropriate sites:

I double checked the subnet of 172.26.176.1/20 and it includes a host address range of 172.26.176.1 to 172.26.191.254 which includes the 2 Exchange server’s IP:

Server	Name	IP
Exchange Hub/CAS/MB	EXCH-02	172.26.186.49
Exchange UM	EXCH-UM	172.26.186.14

The second warning entry actually provided a more important clue of what the cause might be since it shows that Exchange UM does indeed start a session with Exchange Hub but the communication between the 2 servers runs into a problem when authenticating.

If we look at the logs on the Exchange UM server, we’ll notice that there’s actually another warning log that has been flooding the event logs since the IP change:

This warning has been logged since 5/30/2010 @ 8:50:44PM.

The details of the error are as follows:

The Unified Messaging server was unable to submit a message to Hub Transport server "EXCH-02" because the following error occurred: Unexpected SMTP server response. Expected: 220, actual: 500, whole response: 500 5.3.3 Unrecognized command.

I did a simple test from Exchange UM to Exchange Hub to see what an EHLO command outputs and here are the results:

From EXCH-UM.domain.local to EXCH-01.domain.local:

220 exch-02.domain.local Microsoft ESMTP MAIL Service ready at Thu, 3 Jun 20

10 16:25:34 -0600

EHLO

250-exch-02.domain.local Hello [172.26.186.14]

250-SIZE

250-PIPELINING

250-DSN

250-ENHANCEDSTATUSCODES

250-AUTH

250-8BITMIME

250-BINARYMIME

250-CHUNKING

250 XEXCH50

An important 250 response required for UM to Hub communication is actually: 250-X-ANONYMOUSTLS.

The next step is to review the receive connectors on the hub transport server (EXCH-02).

There are 3 receive connectors. The only configured connector that will actually allow Exchange UM to connect properly is the “Default EXCH-02” connector. All the other ones do not have the proper settings to allow Exchange UM to establish a connection.

If we review each connector, we will notice that the “Internal EXCH-02” connector actually has specific IP addresses defined while the “Default EXCH-02” connector does not. Since the former has specific IP addresses defined, it will take precedence over the other.

If we look at the authentication and permissions, we’ll notice that while the permissions are set to allow UM connectivity, the Authentication isn’t:

Resolution

Rather than modify The company’s existing connectors, I went ahead and created a new one as shown here that has only one IP defined (EXCH-UM) so the receive connector catches all Exchange UM connections:

After creating new “EXCH-UM to EXCH-02”, a telnet session from Exchange UM to Exchange UM @ port 25 displays the following after an EHLO command:

220 exch-02.domain.local Microsoft ESMTP MAIL Service ready at Thu, 3 Jun 20

10 16:34:20 -0600

EHLO

250-exch-02.domain.local Hello [172.26.186.14]

250-SIZE 10485760

250-PIPELINING

250-DSN

250-ENHANCEDSTATUSCODES

250-STARTTLS

250-X-ANONYMOUSTLS

250-AUTH NTLM

250-X-EXPS GSSAPI NTLM

250-8BITMIME

250-BINARYMIME

250-CHUNKING

250-XEXCH50

250 XRDST

----------------------------------------------------------------------------------------------------------------

I hope this post helps anyone out there that may experience the same problems as I did.

OCS 2007 R2 Mediation Server with Firewall On Issue

I ran into an interesting problem when I was deploying OCS 2007 R2 enterprise voice for a law firm. I usually turn off the firewall for all of the server roles aside from the Edge server but missed the mediation server for this deployment for whatever reason. What’s interesting is that whenever you deploy OCS 2007 R2 on a Windows 2008 server, you always get warned at the end if you already have the firewall turned off that your services would not get put in as exceptions. In my case during this deployment, I left it on so the exception for the services were put in.

The following are the notes I took documenting the issue:

When installing the OCS 2007 R2 Mediation server role on a Windows 2008 R1 server with the firewall turned on, the installation automatically puts the mediation server service in the firewall’s exception. Calls continue to work but the following behavior are experienced by users:

1. OCS user initiates outbound PSTN call.

2. PSTN user picks up and says: “Hello.”

3. OCS user does not hear the user because he/she has not said anything and cannot hear the PSTN user.

4. After 5 seconds, the OCS user finally says: “Hello.”

5. Now that the OCS user has caused mediation server to send traffic out, the traffic generated by the PSTN user now makes it to the OCS user.

6. Call proceeds without any issues.

It almost seems like since the device behind the firewall has not sent any packets out, all traffic coming in is blocked. Once the device behind the firewall sends out a packet, the responding traffic is then allowed in.

Hope this helps anyone out there that may experience this issue.

HA Cluster Error: “… Could not reach isolation address: none specified”

I received a call today from a colleague today as he was deploying a new ESXi cluster for a client where 1 of the hosts has the following warning:

"Issue detected on “esx1.domain.local” in “Cluster Name”: Could not reach isolation address: none specified".

The server was up so it definitely wasn’t in isolation mode but I did a few sanity checks to make sure network connectivity wasn’t the problem. After spending 5 minutes or so without finding any configuration error, I went ahead and googled it. Turns out many people have experienced this before and what was missing was the default gateway it the service console settings. Once I put in the default gateway of the address the message went away.

UCS & vCenter’s dvSwitches: One issue leads to another

I was out to a client’s site to close off a project with a list of items that I wrote down to completed. As 3:30p.m. came around the corner and everything was happily chugging along, I started writing a status report for my PM:

Task	Details	Status	Completed On
Redeploy vCenter on 64-bit OS and with full SQL Server	We are currently still using the preproduction vCenter which resides on a 32-bit Windows 2008 R1 OS (it was incompatible with R2 when we originally deployed it).	Completed	July 9th, 2010
Configure VUM (Update Manager)	Update manager has not been configured and will be installed and configured after the new vCenter has been deployed.	Completed	July 9th, 2010
Cisco UCS with vSphere Distribute Power Management	The Cisco UCS IPMI profile has not been configured with vSphere Distributed Power Management.	Completed	July 9th, 2010

Task	Details	Status	Scheduled
Update ESX servers to update 2	The UCS blades currently have ESX 4.0 Update 1 installed and will need to get updated.	Incomplete	TBD

Other than the last item I had on my list which was incomplete because there were 2 SQL servers configured with MSCS that could not be taken offline, everything was going according to schedule.

An important note I’d like to make before I continue is that the client requested vCenter to use another SQL instance on a separate virtual machine and I was fine with it so I actually went ahead and deployed a new virtual machine from a pristine template of Windows Server 2008 R2 I deployed according to best practices (locked down domain accounts for service accounts). More on this a bit later.

So come 4:00p.m., I figure I should start to pack up and head out to a dinner I planned but as the time ticked off to 4:08p.m., everything started falling apart. The northbound connection from fabric interconnect B went down. We’ve been experiencing this problem for the past few weeks and had just swapped out both switches last week and thought it was fixed.

I went ahead and tried to reseat the x2 module and it came back up but then a few minutes later, fabric interconnect A’s link went down. What’s different than what we’ve experienced prior to swapping out the x2 modules was that now both links would go down one after another. What was also different was that now we would have a fabric interconnect labeled as “subordinate” would not have an active link while our “primary” fabric interconnect would have it’s northbound link down.

Seeing how one of the links would keep going down one after another, I figure we’ll just have to contact Cisco at some point so I went ahead and putty-ed into the primary fabric interconnect and reboot it so it becomes the subordinate. I know I didn’t have to because the redundant link would continue working and I wished I didn’t because once I rebooted it, our virtual machines suddenly lost connectivity to each other. I got cut off from vCenter and could not connect.

I managed to get onto the host directly via it’s service console IP and logged into vCenter’s console. It wasn’t going to start because it simply couldn’t contact the SQL server which was on another server. I couldn’t ping the other server but I can see that the vmnic0 was up (vmnic1) was down. I didn’t start panicking yet and figure I’ll just remove that virtual machine from inventory and add it onto the same host as the SQL server.

I successfully complete the operation as described above and when I logged into the vCenter and tried to start the service but it wouldn’t start. Then I noticed that within Windows, the NIC had a red X beside it. Being a Windows guy, I knew the NIC was disconnected so I went into VI Client to check the check box.

Long behold the checkbox was unchecked, no worries, let’s check it and hit OK. Umm, what’s this?

Why now? Ok, let me do a goggle search. Interesting, a known problem. No worries, I’m a smart guy, I can outsmart this bug:

Umm, I guess I wasn’t that smart:

Note to self and remember this for the future:

<If your vCenter is down, you cannot modify the switch>

Terence: Yes I knew this.

<Yes but did you know you can’t map existing port groups either?>

Terence: No, I did not.

So this was when I started to panic. Dinner plans in 1 hour, what do I do? Why don’t I try to create a new vSwitch?

Yikes, almost forgot this was a UCS with a Menlo card and thus I have 2 vmnics to work with. No problem, I can remove 1 of them right? No, because you lost 1 link!

Now that everything that can possibly go wrong has gone wrong, I’m going to remove my humor and explain what I ended up doing:

Seeing how 1 was down and I cannot edit these dvSwitches from console, here’s what I did:

1. Create a new vSwitch with no vmnics mapped to it.

2. Remove one of the physical adapter that’s down.

3. Assign adapter to the vSwitch.

4. Recreate the port groups you need:

a) virtual machine network

b) service console

Ok, so now we have a broken adapter assigned to a regular vSwitch. Just as I was about to ask and reseat the x2 module, I was told that they didn’t want to do this because the client wasn’t comfortable with anything being done to the UCS until Cisco determines what the problem is (I got the client to call Cisco and troubleshoot).

I proceeded to this the following (no screenshots from here on as I was in a rush):

5. You cannot edit the dvSwitches from the console so you have to unmap the left over active vmnic from the dvSwitch. This will cut you off but at least you will get the vmnic back so go ahead and do it.

6. Log into UCSM or the KVM manager and go directly to the console.

7. Log in, use esxcfg-vswif, esxcfg-nics, esxcfg-vswitch to remap the vmnic to the regular vSwitch.

8. What I noticed was that even after doing so, I was unable to VI Client to the service console IP so I rebooted the server. Once the server was up, I managed to VI Client into the server, modify the NIC settings for the vCenter and SQL virtual machine to be on the regular vSwitch, get vCenter back up (vCenter and SQL are on the same host now).

What an ordeal. This still doesn’t solve the virtual machines not being able to ping each other across ESX hosts YET if I console into any of the ESX hosts, i can ping each other (hence no isolation mode and why the virtual machines haven’t shut off). Strange. I won’t elaborate on how I solved this problem because I haven’t yet. I went back in the next day on Saturday and tried removing the dvSwitch and all of the port groups, recreated the dvSwitch but still cannot establish connection. Since I had little time on Saturday, I ended up re-creating regular vSwitches to get the infrastructure back up. I will try to schedule some time in the following weeks to open a VMware support case to get this resolved and will post back.

Harsh lessons learned:

1. I really hope VMware changes the design of dvSwitches and:

a) allow modifying them through the console and GUI even if vCenter is down.

b) allow you to map dvSwitch port groups even if vCenter is down.

2. Go hybrid model and leave the service console on a regular vSwitch in the future (you cannot do this with UCS and a menlo card).

3. If licensing is not an issue, deploy SQL on the vCenter server.

4. NEVER EVER EVER touch the NIC properties if vCenter is down. See bug mentioned above.

5. Think twice before removing virtual machines from inventory if vCenter is down. See bug mentioned above.

I hope this helps someone in the future if they ever encounter the same problem as I did. I will blog the resolution to the northbound connection to the 3750e switches when we get that sorted out (http://terenceluk.blogspot.com/2010/07/problem-with-ucs-northbound-connection.html) and the dvSwitch issue when I get that resolved either by myself or through VMware support.

As a final note just in case I'm sending off the vibe that I may not be happy with my experiences with UCS, I'd like to make it clear that I've had a blast deploying this infrastructure and would highly recommend this to clients. The issues I encountered here are simply bugs and if we had used another type of x2 extender, we would have never encountered this.

As for vCenter's new distributed switches, I still like the idea and I'm sure VMware would address the design in the next release.

Nightmare with dvSwitches and UCS

I had an extremely bad end to a work week last Friday when one issue lead to another and after accumulating multiple small issues, a UCS and VMware infrastructure that was happily chugging along fell apart. I finally got a taste of all the blogs that spoke of the limitations of dvSwitches and what you need to look out for when working with a UCS with the Menlo card.

I’m happy to say that although I had to sacrifice a sunny Saturday to fix the issue, I did manage to get the whole infrastructure back up. I have a lot of notes and screenshots but lack the time to write the post so stay tuned.

Oh, and Malcom Gladwell was right in his "The Tipping Point". :)

Clean MOSS Farm Install Error - 403 - Forbidden: Access is denied.

After finishing the MOSS 2007 64bit SP2 on Windows Server 2008 base OS, I went ahead to create a web application and then a site collection as per document's instructions: http://technet.microsoft.com/en-us/library/cc262714.aspx

The problem I experienced was that when I hit the new site, everything loads except for the graphics. I went ahead to figure out the directory for the image and I can go to the directory to see the picture. Since the pictures weren't showing up, I also tried to directly access the image through the URL: http://99.99.5.91/_layouts/images/navshape.jpg and this is where I get a:

· 403 - Forbidden: Access is denied.

You do not have permission to view this directory or page using the credentials that you supplied.

I have no issues with all the images in the Sharepoint Administration page and when I browse the image via http://server1:32873/_layouts/images/recycbin.gif I can see the recycling bin.

I went ahead to try to review the permissions to see if they were set appropriately and they were (Users and WSS_admin_wpg were listed). As a troubleshooting step, I also added “everyone” and the “IUSER” account to read access but even by doing so, I was still getting the 403 error.

After spending some more time to troubleshoot the issue, I noticed that both:

1. Allow unspecified CGI modules

2. Allow unspecified ISAPI modules

…were both unchecked. This was when I finally found a workaround here: http://support.microsoft.com/default.aspx/kb/942065

This fixed the problem but it was only for this site. As soon as I deployed another site, it would have the same issues.

With a bit more troubleshooting, I realized that there was a missing component required for IIS and that was the “ASP.NET” in the Web server role services.

This was not included in this document: http://technet.microsoft.com/en-us/library/cc731760.aspx

I asked a Microsoft engineer as to why this was not included in the document and he pointed me to this link: http://technet.microsoft.com/en-us/library/cc263408.aspx I wasn’t able to find a list of components required either so I ended up documenting the information in the following table so I have a reference list of what is required to be installed in case I need it in the future.

Description

Front-end Server

Roles

Web Server (IIS)

· Common HTTP Features

· Static Content

· Default Document

· Directory Browsing

· HTTP Errors

· Application Development

· ASP.NET

· .NET Extensibility

· ISAPI Extensions

· ISAPI Filters

· Health and Diagnostics

· HTTP Logging

· Logging Tools

· Request Monitor

· Security

· Basic Authentication

· Windows Authentication

· Request Filtering

· Performance

· Static Content Compressions

· Management Tools

· IIS Management Console

· IIS 6 Management Compatibility

· IIS 6 Metabase Compatibility

Features

Microsoft .Net Framework 3.0 Features

· .NET Framework 3.0

· XPS Viewer

Remote Server Administration Tools

· Role Administration Tools

· Web Server (IIS) Tools

Windows Process Activation Service

· Process Model

· .NET Environment

· Configuration

Reinstalling MOSS and Required Components (IIS 7.0)

This was one of the problems I encountered a year or two ago while during a project at a college when installing MOSS 2007 on a Windows 2008 server:

One of the problems I encountered while reinstalling a MOSS farm front-end servers was when I had removed IIS 7.0 from Windows Server 2008 and tried to reinstall it. What happened was that it would keep failing when I tried to reinstall IIS 7.0 and the various components required for the installation. After troubleshooting for a bit, I realized that in order for IIS 7.0 to reinstall properly, you need to uninstall Windows Process Activation Service in the Features section. This component is required component and is automatically installed when you install IIS 7.0 but is not automatically uninstalled if you remove IIS 7.0 from the roles. The following is a screenshot of where it can be found: