Upgrading Exchange Server 2010 to Service Pack 3 fails with: “The following error was generated when "$error.Clear(); …”

Problem

You have installed 2 new Exchange Server 2010 with SP1 mailbox servers (future DAG) and 2 hub transport / CAS servers an existing Exchange 2007 organization and proceed to install SP3 onto the servers but noticed that you are able to install it onto the HT/CAS servers and only 1 of the 2 mailbox servers.  The first mailbox server installs without an issue but the second one fails at the Mailbox Role stage with the error:

Summary: 6 item(s). 3 succeeded, 1 failed.
Elapsed time: 00:08:01

Language Files
Completed
Elapsed Time: 00:02:55

Restoring services
Completed
Elapsed Time: 00:00:01

Languages
Completed
Elapsed Time: 00:01:17

Mailbox Role
Failed
Error:
The following error was generated when "$error.Clear();
          $name = [Microsoft.Exchange.Management.RecipientTasks.EnableMailbox]::DiscoveryMailboxUniqueName;
          $dispname = [Microsoft.Exchange.Management.RecipientTasks.EnableMailbox]::DiscoveryMailboxDisplayName;
          $dismbx = get-mailbox -Filter {name -eq $name} -IgnoreDefaultScope -resultSize 1;
          if( $dismbx -ne $null)
          {
            $srvname = $dismbx.ServerName;
            if( $dismbx.Database -ne $null -and $RoleFqdnOrName -like "$srvname.*" )
            {
              Write-ExchangeSetupLog -info "Setup DiscoverySearchMailbox Permission.";
              $mountedMdb = get-mailboxdatabase $dismbx.Database -status | where { $_.Mounted -eq $true };
              if( $mountedMdb -eq $null )
              {
                Write-ExchangeSetupLog -info "Mounting database before stamp DiscoverySearchMailbox Permission...";
                mount-database $dismbx.Database;
              }
              $mountedMdb = get-mailboxdatabase $dismbx.Database -status | where { $_.Mounted -eq $true };
              if( $mountedMdb -ne $null )
              {
                $dmRoleGroupGuid = [Microsoft.Exchange.Data.Directory.Management.RoleGroup]::DiscoveryManagementWkGuid;
                $dmRoleGroup = Get-RoleGroup -Identity $dmRoleGroupGuid -DomainController $RoleDomainController -ErrorAction:SilentlyContinue;
                if( $dmRoleGroup -ne $null )
                {
                  Add-MailboxPermission $dismbx -User $dmRoleGroup.Identity -AccessRights FullAccess -DomainController $RoleDomainController -WarningAction SilentlyContinue;
                }
              }
            }
          }
        " was run: "Couldn't resolve the user or group "contoso.com/Microsoft Exchange Security Groups/Discovery Management." If the user or group is a foreign forest principal, you must have either a two-way trust or an outgoing trust.".
Couldn't resolve the user or group "contoso.com/Microsoft Exchange Security Groups/Discovery Management." If the user or group is a foreign forest principal, you must have either a two-way trust or an outgoing trust.
The trust relationship between the primary domain and the trusted domain failed.
Click here for help... http://technet.microsoft.com/en-US/library/ms.exch.err.default(EXCHG.141).aspx?v=14.3.123.3&e=ms.exch.err.Ex88D115&l=0&cl=cp
Elapsed Time: 00:03:46

Management Tools
Cancelled

Finalizing Setup
Cancelled

The error message includes a link but clicking on it brings you to a page indicating there’s no article written for this error:

Searching on the internet returns a lot of posts that suggest either disabling the Discovery Mailbox in the Exchange Management Console deleting the account completely, install SP3 then recreate it.  The challenge I had was that these suggestions did not resolve the issue because it would fail at the same stage and it also appears that the Discovery Mailbox also gets recreated during the process.

Another solution I read off of a forum post was to check the permissions was to check the Full Access permissions of the Discovery Mailbox through the EMC:

… and while it was missing a lot of permissions because it only had NT AUTHORITY\SELF:

 

… while the list should look more like this:

• DOMAIN\Discovery Management
• DOMAIN\Exchange Domain Servers
• DOMAIN\Exchange Servers
• DOMAIN\Exchange Services
• DOMAIN\Exchange Trusted Subsystem

… adding the above permissions and running the SP3 install continued to fail.

Solution

Having exhausted all of the available resources I could find and knowing I’d probably have to figure this one out myself, I went ahead and reviewed the error message line by line again to see if anything pop out at me but nothing did.  What I ended up doing was run through the following list of what I knew:

1. The discovery mailbox actually gets recreated by this second mailbox server during the SP3 install
2. The discovery mailbox is not available when the server that’s being upgraded to SP3 is down

Though probably not accurate at all, the error message appears to suggest some sort of access issue related to this mailbox and if the server was being upgraded, wouldn’t the store be down at some point?  This gave me the idea that since I had 2 mailbox servers with 2 mailbox databases, why not move this discovery mailbox off of the SP3 upgrade failing mailbox server’s database and onto the one that already has SP3 successfully installed:

 

Executed a move request:

 

Notice the mailbox is being moved:

The move completed:

With the discovery mailbox in a different store, I ran the SP3 upgrade again and this time it completed successfully:

A bit odd but I hope this helps anyone who might come across this issue as I did.