Handling BIS (Blackberry Internet Service) users who access their corporate email via OWA when transitioning from Exchange 2003 to 2010

I ran into an interesting challenge a few months ago during a project where I was migrating users from Exchange 2003 to 2010.  These migrations are usually very straight forward but what was different for this project was that there were several users using BIS (Blackberry Internet Service) to retrieve their corporate email because of the lack of a BES server available. 

Problem #1 – Re-directly OWA requests

The challenge here was that during the coexistence phase, all legacy or regular OWA requests always hits the Exchange 2010 CAS server first, then if the user is in the legacy Exchange 2003 environment, Exchange 2010’s CAS server will redirect the traffic over. 

This causes a problem for users who retrieve corporate email via their Blackberry’s BIS service because it will not work if you point it to an Exchange 2010’s OWA and have it redirect requests to the Exchange 2003 front-end or mailbox server. 

Problem #2 – Authentication configuration requirements for BIS

The next challenge is that we were not able to simply modify all of the BIS servers to use the legacy Exchange’s new URL such as: https://legacy.contoso.com/Exchange because when we did so, the Blackberries were not able to establish a connection.  This was strange because as per the following KB: http://btsc.webapps.blackberry.com/btsc/search.do?cmd=displayKC&docType=kc&externalId=KB23361 the configuration we had for the Exchange 2003 OWA should be supported.  What’s even more interesting was that when I turned off Form Based authentication for the Exchange 2003 OWA:

… https://legacy.contoso.com/exchange worked for the BIS service.  This obviously contradicts what the KB says and even though disabling the feature would get BIS to work, we would end up breaking the Exchange 2010 redirect for legacy users.  Since we were pressed for time and needed a solution, what I ended up doing was the following:

1. Purchased another certificate named blackberry.contoso.com.
2. Added a new public DNS record for blackberry.contoso.com.
3. Modified our firewall to accept blackberry.contoso.com traffic and redirect it to the backend mailbox server.
4. Configured the backend mailbox server to have Form Based authentication disabled.
5. Configured the users’ BIS account to use blackberry.contoso.com as the OWA URL.

This interim solution worked during the transition and while it’s not perfect, it’s a viable solution if you happen to come across the same issues I had.

Moving an Exchange 2010 mailbox with New-MoveRequest fails with: "The discovery mailbox, a hidden default mailbox that is required to search mailboxes, can't be found."

Problem

You’re transitioning from Exchange 2003 or 2010 and with the infrastructure in place, you notice that your mailbox move fails with the following error which is also logged in the application log as event ID 5000:

Failed to save admin audit log for this cmdlet invocation.
Organization: 
Log content:
Subject: Contosonet.com/ContosoNoPolicy/Terence Luk : New-MoveRequest
Body:
Cmdlet Name: New-MoveRequest
Object Modified:
Parameter: TargetDatabase = Mailbox Database 0162106233
Parameter: Identity = Contosonet.com/Users/testuser
Property Modified: UserId = Contosonet.com/Users/testuser
Property Original: UserId =
Caller: Contosonet.com/ContosoNoPolicy/Terence Luk
Succeeded: True
Error: None
Run Date: 2011-07-12T14:13:56
OriginatingServer: ContosoMBX01 (14.01.0218.011)

Error:
Exception thrown during AdminLogProvisioningHandler.Validate: Microsoft.Exchange.Data.Storage.ObjectNotFoundException: The discovery mailbox, a hidden default mailbox that is required to search mailboxes, can't be found. It may have been inadvertently deleted. This mailbox must be re-created before you can search mailboxes.
   at Microsoft.Exchange.Data.Storage.Infoworker.MailboxSearch.MailboxDataProvider.GetDiscoveryMailbox(ADRecipientSession session)
   at Microsoft.Exchange.Management.SystemConfigurationTasks.AdminAuditLogHelper.CheckArbitrationMailboxStatus(OrganizationId organizationId, ADUser& user, ExchangePrincipal& principal, String& errorMessage)

Solution

The reason why Exchange Server 2010 is throwing this error is because there is something wrong with the DiscoverySearchMailbox which is a hidden mailbox that Exchange Server 2010 uses to search through mailboxes and is required when it attempts to move a mailbox.  Unless you’ve moved the object with the associated mailbox to another OU, you should be able to locate the object under the Users container in the domain.  If the object no longer exists, follow the TechNet article here: http://technet.microsoft.com/en-us/library/gg588318.aspx to recreate it with the PowerShell cmdlet.

If the object still exist as shown here:

… the object may have incorrect values for the homeMDB and/or homeMTA attributes.  The following shows the the object’s properties opened in ADSIedit:

Notice how the attribute values are set to <not set>. 

If you’re sure that someone or a process has inadvertently removed the values, you can manually populate the attributes to correct the problem.

The homeMDB attribute is the mailbox database’s distinguished name so if you navigate to:

CN=Databases,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Contoso,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=Contosonet,DC=com

… you should see a list of mailbox databases that you can choose from.  In the case of this example, there’s only 1 to choose from and it’s the CN=Mailbox Database 0162106233 database:

From here, open up the properties of the mailbox database object, navigate down to the distinguishedName attribute, open the value in the String Attribute Editor and copy it:

From the DiscoverySearchMailbox object’s properties window, paste the value into the homeMDB attribute:

The homeMTA attribute is the mailbox server’s Microsoft MTA’s distinguished name so navigate to:

CN=Microsoft MTA,CN=CONTOSOMBX01,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Contoso,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=Contosonet,DC=com

… and perform the same action as you did for the homeMDB attribute by opening up the properties of the Microsoft MTA object, navigate down to the distinguishedName attribute, open the value in the String Attribute Editor and copy it:

From the DiscoverySearchMailbox object’s properties window, paste the value into the homeMTA attribute:

Once these 2 attributes have their values populated with the appropriate values, you should now be able to move mailboxes.

Installing Exchange 2010’s Hub Transport role throws the error: “$error.Clear(); if ( ($server -eq $null) -and ($RoleIsDatacenter -ne $true) ) { Update-RmsSharedIdentity -ServerName $RoleNetBIOSName } " was run: "Database is mandatory on UserMailbox. Property Name: Database". Database is mandatory on UserMailbox. Property Name: Database”

Problem

You attempt to install the hub transport role for Exchange 2010 but the process errors out with the following error message:

Hub Transport Role

Failed

Error:

The following error was generated when "$error.Clear();

if ( ($server -eq $null) -and ($RoleIsDatacenter -ne $true) )

{

Update-RmsSharedIdentity -ServerName $RoleNetBIOSName

}

" was run: "Database is mandatory on UserMailbox. Property Name: Database".

Database is mandatory on UserMailbox. Property Name: Database

Click here for help... http://technet.microsoft.com/en-US/library/ms.exch.err.default(EXCHG.141).aspx?v=14.1.218.11&e=ms.exch.err.Ex88D115&l=0&cl=cp

The application logs of the server also logs the error with event ID 1002:

Exchange Server component Hub Transport Role failed.

Error: Error:

The following error was generated when "$error.Clear();

if ( ($server -eq $null) -and ($RoleIsDatacenter -ne $true) )

{

Update-RmsSharedIdentity -ServerName $RoleNetBIOSName

}

" was run: "Database is mandatory on UserMailbox. Property Name: Database".

Database is mandatory on UserMailbox. Property Name: Database

… and warning with event ID 2937:

Process ExSetupUI.exe (PID=1704). Object [CN=FederatedEmail.4c1f4d8b-8179-4148-93bf-00a95fa1e042,CN=Users,DC=contosonet,DC=com]. Property [HomeMTA] is set to value [contosonet.com/Configuration/Deleted Objects/Microsoft MTA
DEL:8c20a5cb-cbb5-4e6d-b6f8-05427d8c4751], it is pointing to the Deleted Objects container in Active Directory. This property should be fixed as soon as possible.

The warning is logged first then followed by the error:

Solution

The reason why Exchange 2010’s setup is throwing this error is because of the federated built-in e-mail account either no longer exists or has has missing attributes.  The following is an example of a federated built-in e-mail account that has the homeMTA attribute pointed to the deleted items container and the homeMDB attribute not set:

Option #1 – Recreate the Federated Account

The first way of rectifying this issue, which I suggest to be the preferred option, is to perform the steps outlined in the following KB: http://support.microsoft.com/kb/978776 to recreate the Federated object.  The following is what the steps look like:

Execute the following command in the Exchange Management Shell:

New-Mailbox -Arbitration -Name FederatedEmail.4c1f4d8b-8179-4148-93bf-00a95fa1e042 -UserPrincipalName FederatedEmail.4c1f4d8b-8179-4148-93bf-00a95fa1e042@<Default_Accepted_Domain>

Hit the refresh button in ADSIedit a few times and you should see the FederatedEmail account reappear:

Option #2 – Manually correct the attributes

If you’re sure that the mailbox’s attributes were accidentally modified and that nothing else was changed, you can actually manually edit the homeMDB and homeMTA attribute to fix it.  One of the ways to retrieve those values is to open up another Exchange 2010 system mailbox object:

-------------------------------------------------------------------------------------------------------------------------------------------------------------------

Once the federated mailbox is corrected, the hub transport role should now install correctly:

Installing ESXi 4.1 on Cisco UCS B series server throws the error: “vmkctl.HostCtlException: Unable to load module /usr/lib/vmware/vmkmod/vmfs3: Failure”

I ran into an interesting error a few months ago while deploying a new Cisco UCS B series infrastructure with B200 M2 and B440 M1 blades.  As some of you may know, there are Cisco packages of VMware ESXi 4.1 available for download but as I browsed through the downloads section and was only able to find VMware ESXi 4.1 but not ESXi 4.1 Update 1, the thought of trying to use a regular package to see what happens crossed my mind and this post serves to demonstrates what happens.

Problem

You’ve downloaded the VMware ESXi 4.1 Update 1 package (VMware-VMvisor-Installer-4.1.0.update1-348481.x86_64.iso) from VMware’s site and proceeded to install onto the UCS blades:

Accepted the EULA:

The ESXi 4.1 installer begins to scan for devices:

… then quickly errors out with the following error:

vmfs3Module.Load()
File “/lib/python2.5/vmkctl.py”, line 11331, in Load
def Load(*args): return _vmkctl.ModuleImpl_Load(*args)
vmkctl.HostCtlException: Unable to load module /usr/lib/vmware/vmkmod/vmfs3: Failure

You are not able to proceed with the install and is forced to reboot.

Solution

The reason why you’re encountering this error is because the ESXi 4.1 Update 1 installer encounters a device it doesn’t have a driver for and fails without recovering. I’ve spoken to Cisco about this and they say you can either slipstream an ESXi 4.1 Update 1 CD with the missing drivers or just use the available ESXi 4.1 ISO (VMware-VMvisor-Installer-4.1.0-260247_Cisco.iso) provided by Cisco to install ESXi 4.1 onto the blades, then update them afterwards.

I didn’t want to go through the hassle of slipstreaming so what I ended up doing was install ESXi 4.1 onto the blades then used vCenter’s Update Manager to update the build from 260247 to 433742).

Installing NIC drivers for VMware ESX/ESXi 4.x

Those who have come across one of my previous post that documented the troubleshooting steps I went through when I did not see the fibre channel hba adapters appear in the storage adapters section after installing ESXi 4.1:

http://terenceluk.blogspot.com/2010/09/hba-vmhba-not-showing-up-with-esxi-40.html

hba / vmhba not showing up with ESXi 4.0 / 4.1

… may remember that while the post does demonstrate what needs to be done, it also includes more information than needed because I opted to show the mistakes I made on the way.  Fast forward to a few months later, one of our clients needed someone to assist them in their datacenter deployment and one of the first items I was asked to look at was whether additional drivers had to get installed into their new ESXi 4.1 U1 servers to get the Intel Ethernet Server Adapter X520-T2 to show up in the network adapters list. 

Since I have gone through this exercise already and remember blogging it, I went back to reference the post and after plowing through the content skipping the additional information I did not need, I decided to document the process again and write up a new post to provide a more direct step-by-step instructions.

Note: for the purpose of this example, I will use the Intel Ethernet Server Adapter X520-T2 as the device we need to download and install the drivers for.

Device

No matter what device you’re about to install, I would highly recommend that you look up the vendor’s website to determine what chipset the adapter uses.  In the case of this example, although the Intel adapter we’re going to install the drivers for is named Intel Ethernet Server Adapter X520-T2, it’s also useful to know what chipset (82599) is used:

http://www.intel.com/content/www/us/en/network-adapters/gigabit-network-adapters/ethernet-x520-t2.html

Hardware Compatibility List (HCL)

As with any device you are going to use in your production VMware ESXi hosts, always remember to check the hardware compatibility list from VMware to ensure that it is listed:

http://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=18611&deviceCategory=io&keyword=x520&deviceTypes=6&page=1&display_interval=10&sortColumn=Partner&sortOrder=Asc

Downloading the Drivers

As shown in the screenshot below, you now know why it is highly recommended to lookup the detailed information of the device on the vendor’s site because if you were to try and search for Intel Ethernet Server Adapter X520-T2 adapter, you wouldn’t find it.  The drivers for this adapter is actually listed as the chipset name:

http://downloads.vmware.com/d/details/esx4x_intel_8259882599_dt/ZHcqYnRlQCVidGR3

Scroll down and you’ll see the following list:

Download vSphere Command-Line Interface to install the driver

The process for installing the drivers is to have a management server or laptop/desktop that has network access to the ESX/ESXi host’s management network to push the drivers out and the utility for this is the vSphere Command-Line Interface application.  If you don’t already have it installed on, say, your laptop, you can download it at the following URL:

http://www.vmware.com/support/developer/vcli/

Downloaded Driver Bundle

Once the the driver bundle has been downloaded, double check to make sure it’s a zip package with the proper extension.  I’m not sure what happened when I downloaded the package because upon completion, I noticed that the package was named as the following:

INT-intel-lad-ddk-ixgbe-400.3.1.17.1-1.2.17.249663-offline_bundl

Since I knew the package was supposed to be in zip format, I went ahead and renamed the file to following:

INT-intel-lad-ddk-ixgbe-400.3.1.17.1-1.2.17.249663-offline_bundle.zip

Just to be safe, I opened up the zip package to ensure that my WinRAR recognized it to be a valid compressed package and that metadata.zip was in the compressed file.

If you’re as paranoid as I am, you can open up the metadata.zip package to see if the following folders are listed as well:

Don’t forget to put the host into maintenance mode

Now that we have the package downloaded and verified, it’s important to note that you will need to put the ESX/ESXi host into maintenance mode before executing the install command or you’ll see the following error:

C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>vihostupdate.pl -server 172

.16.15.101 -username root -install -bundle c:\INT-intel-lad-ddk-ixgbe-400.3.1.1

7.1-1.2.17.249663-offline_bundle.zip

Enter password:

Please wait patch installation is in progress ...

Maintenance mode is not enabled or could not be determined.The following VIBs re

quire this host to be in maintenance mode: cross_vmware-esx-drivers-net-ixgbe_40

0.3.1.17.1-1vmw.2.17.249663. Please put the host in maintenance mode to proceed.

C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>

 

Installing Drivers

Installing the drivers is a 4 step process and is possible to perform completely in vSphere CLI.  The following outlines the steps:

Step #1 – Put server into maintenance mode

C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>vicfg-hostops.pl -server 17
2.16.15.103 -username root -o enter
Enter password:
Host CONESXi-03.contoso.com entered into maintenance mode successfully.

172.16.15.103 <—The IP of the management console of your ESX/ESXi host.

CONESXi-03.contoso.com <—The FQDN of your ESX/ESXi host.

Step #2 – Install the drivers

C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>vihostupdate.pl -server 172
.16.15.103 -username root -install -bundle c:\INT-intel-lad-ddk-ixgbe-400.3.1.1
7.1-1.2.17.249663-offline_bundle.zip
Enter password:
Please wait patch installation is in progress ...
The update completed successfully, but the system needs to be rebooted for the c
hanges to be effective.

172.16.15.103 <—The IP of the management console of your ESX/ESXi host.

C:\INT-intel-lad--ddk-ixgbe-400.3.1.1.7.1-1.2.17.249663-offline_bundle.zip <—The full path to the drivers package.

Step #3 – Reboot the server

C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>vicfg-hostops.pl -server 17
2.16.15.102 -username root -o reboot
Enter password:
Host CONESXi-03.contoso.com rebooted successfully.

172.16.15.103 <—The IP of the management console of your ESX/ESXi host.

Step #4– Take server out of maintenance mode

C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>vicfg-hostops.pl -server 17
2.16.15.103 -username root -o exit
Enter password:
Host CONESXi-03.contoso.com exited from maintenance mode successfully.

C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>

172.16.15.103 <—The IP of the management console of your ESX/ESXi host.

Now that the drivers have been installed, use the vSphere Client and log into the server to verify that the adapters are recognized:

Hope this helps anyone who may be looking for instructions on how to install additional drivers for their ESX/ESXi 4.x servers.

Modifying an Exchange 2010 E-Mail Address Policy throws the error: “You can't specify the recipient container because legacy servers are detected.”

Problem

You’re in the process of transitioning from Exchange 2003 to 2010 and you have just upgraded your Exchange 2003 E-Mail Address Policy for Exchange Server 2010.  Now that they’re upgraded, you would like to make some changes to it so you go through the wizard for modifying the policy but once you attempt to apply the policy, you receive the following error:

New student.contoso.com
Failed

Error:
You can't specify the recipient container because legacy servers are detected.

Exchange Management Shell command attempted:
new-EmailAddressPolicy -Name 'student.contoso.com' -RecipientContainer 'contoso.local.com/School' -IncludedRecipients 'MailboxUsers' -ConditionalCompany 'student.contoso.com' -Priority 'Lowest' -EnabledEmailAddressTemplates 'smtp:%m@contoso.local.com','SMTP:%m@student.contoso.com'

Elapsed Time: 00:00:00

Apply student.contoso.com
Cancelled

Solution

The reason why Exchange Server 2010 is throwing this error is because you used the following feature:

Select the recipient container where you want to apply the filter:

Exchange Server 2003 actually doesn’t support this feature and since you still have your Exchange 2003 server in the organization and therefore operating in a coexistence topology, Exchange Server 2010 will not allow you to create a policy that uses this feature that does not exist in Exchange 2003.

To get around this, all you need to do is leave the Select the recipient container where you want to apply the filter: field blank if you’re using the Exchange Management Console wizard or use the PowerShell cmdlet to create a new one with the following syntax:

new-EmailAddressPolicy –Name ‘domain.com (i.e. contoso.com)’ -IncludedRecipients ‘AllRecipients’ –ConditionalCompany ‘Attribute (i.e. Accounting)’ -Priority ‘Lowest’ -EnabledEmailAddressTemplates ‘SMTP:%g.%s@domain.com’,'smtp:%g.%s@secondarydomain.com’

Here’s an example of the cmdlet:

new-EmailAddressPolicy -Name 'student.contoso.com' -IncludedRecipients 'MailboxUsers' -ConditionalCompany 'student.contoso.com' -Priority 'Lowest' -EnabledEmailAddressTemplates 'smtp:%m@contoso.local','SMTP:%m@student.contoso.com'

By leaving the Select the recipient container where you want to apply the filter: blank, the policy should not successfully update.

Creating a new E-Mail Address Policy with the cmdlet would look something like this:

[PS] C:\Windows\system32>new-EmailAddressPolicy -Name 'student.contoso.com' -IncludedRecipients 'MailboxUsers' -ConditionalCompany 'student.contoso.com' -Priority 'Lowest' -EnabledEmailAddressTemplates 'smtp:%m@contoso.local','SMTP:%m@student.contoso.com'
Creating a new session for implicit remoting of "New-EmailAddressPolicy" command...

Name                                    Priority                                RecipientFilter
----                                    --------                                ---------------
student.contoso.com                       2                                       ((Company -eq 'student.contoso.com') -...

[PS] C:\Windows\system32>

VMware Data Recovery error: Failed to create snapshot for <virtualMachine>, error -3960 ( cannot quiesce virtual machine)

Note that this error can be caused by various reasons and the example in this post is just one of the many possible causes.

Problem

You attempt to use VMware Data Recovery to perform a full backup of a virtual machine but the process errors out with:

Failed to create snapshot for <virtualMachine>, error -3960 ( cannot quiesce virtual machine)

Solution

The problem I had was because the virtual machine did not have network access which meant VMware Data Recovery wouldn’t be able to use the Windows VSS writers to quiesce the snapshot).

A quick check on the virtual machine’s summary tab shows that this virtual machine didn’t have an IP address:

Reviewing the properties of the virtual machine showed that the network adapter did not have port group selected.

Proceed with selecting the proper port group:

Ensure that an IP address shows up:

… and now the VDR backup should proceed without error out: