Wednesday, August 21, 2013

Exchange 2010 Server DAG Seeding Errors

I recently had to rebuild a failed Exchange 2010 server that was a part of a DAG and while the rebuild went quite smoothly, I had quite a few issues reseeding the database back onto the new server so I thought it would be good to blog 2 of the main issues I had:

Problem #1

Attempting to add a mailbox database copy with the cmdlet Add-MailboxDatabaseCopy throws the error:

[PS] C:\Windows\system32>Add-MailboxDatabaseCopy -Identity "EXDB-02" -MailboxServer MBX-01
A source-side operation failed. Error An error occurred while performing the seed operation. Error: Failed to open a lo
g truncation context to source server 'MBX-02.someDomain.NET'. Hresult: 0xfffffae7. Error: The database was eithe
r not found or was not replicated.. [Database: EXDB-02, Server: MBX-01.someDomain.NET]
    + CategoryInfo          : InvalidOperation: (:) [Add-MailboxDatabaseCopy], SeedInProgressException
    + FullyQualifiedErrorId : 89031D8B,Microsoft.Exchange.Management.SystemConfigurationTasks.AddMailboxDatabaseCopy

[PS] C:\Windows\system32>

image

You notice that the mailbox server has been added as one of the servers for the mailbox database but the Copy Status is listed as Suspended:

image

You attempt to use the Resume Database Copy option in the EMC to resume the seeding:

image

image

… but notice that the Copy Status quickly changes into Failed and Suspended:

image

Alternatively, if you use the EMC to add the mailbox database copy, you see the following error:

Summary: 1 item(s). 0 succeeded, 1 failed.
Elapsed time: 00:00:34


EXDB-02
Failed

Error:
A source-side operation failed. Error An error occurred while performing the seed operation. Error: An error occurred while processing a request on server 'MBX-02'. Error: Couldn't open backup file handle for database 'EXDB-02' to server 'MBX-02'. Hresult: 0x50d. Error: A database backup is already in progress. Please verify that no other seeding or incremental reseeding operations are started for this database, and then try the operation again by rerunning the Update-MailboxDatabaseCopy cmdlet.. [Database: EXDB-02, Server: MBX-01.someDomain.NET]

An error occurred while processing a request on server 'MBX-02'. Error: Couldn't open backup file handle for database 'EXDB-02' to server 'MBX-02'. Hresult: 0x50d. Error: A database backup is already in progress. Please verify that no other seeding or incremental reseeding operations are started for this database, and then try the operation again by rerunning the Update-MailboxDatabaseCopy cmdlet.

Couldn't open backup file handle for database 'EXDB-02' to server 'MBX-02'. Hresult: 0x50d. Error: A database backup is already in progress. Please verify that no other seeding or incremental reseeding operations are started for this database, and then try the operation again by rerunning the Update-MailboxDatabaseCopy cmdlet.
Click here for help...
http://technet.microsoft.com/en-US/library/ms.exch.err.default(EXCHG.141).aspx?v=14.2.247.1&t=exchgf1&e=ms.exch.err.Ex4543D9

Exchange Management Shell command attempted:
Add-MailboxDatabaseCopy -Identity 'EXDB-02' -MailboxServer 'MBX-01' -ActivationPreference '2'

Elapsed Time: 00:00:34

image

As ridiculous as may sound, the way I got the seeding to work was to dismount all of the databases on the active server (the one that wasn’t rebuilt), restart the server, mount the databases, and started the seeding. 

Problem #2

You notice that while the seeding of a database starts and seemingly completes, the Copy Status ends up being as Failed and Suspended:

image

… if you attempt to resume the seeding, the Copy Status will end up being Failed.

Another symptom observed is that the logs on the active database are not truncated even if you turn on circular logging.

Also, you notice the following event logged on the active server:

Exchange Search Indexer has temporarily disabled indexing of the mailbox database EXDB-01 (GUID = 52c3b6c8-c3ef-4912-8b04-e6f7e3eeb438) due to an error (Microsoft.Mapi.MapiExceptionMdbOffline: MapiExceptionMdbOffline: Unable to read events. (hr=0x80004005, ec=1142)
Diagnostic context:
    Lid: 33865 
    Lid: 1494    ---- Remote Context Beg ----
    Lid: 44215 
    Lid: 60049   StoreEc: 0x8004010F
    Lid: 49469 
    Lid: 65341   StoreEc: 0x8004010F
    Lid: 56125 
    Lid: 47933   StoreEc: 0x8004010F
    Lid: 32829 
    Lid: 49213   StoreEc: 0x8004010F
    Lid: 48573 
    Lid: 64957   StoreEc: 0x8004010F
    Lid: 20057   StoreEc: 0x476    
    Lid: 1750    ---- Remote Context End ----
    Lid: 28777   StoreEc: 0x476    
    Lid: 20098 
    Lid: 20585   StoreEc: 0x476    
   at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)
   at Microsoft.Mapi.MapiEventManager.ReadEvents(Int64 startCounter, Int32 eventCountWanted, Int32 eventCountToCheck, Restriction filter, ReadEventsFlags flags, Boolean includeSid, Int64& endCounter)
   at Microsoft.Exchange.Search.RetriableOperations.ReadEvents(ThreadLocalCrawlData unused1, MapiEventManager eventManager, Int64 watermark, Int32 eventCount, Int64& endCounter)
   at Microsoft.Exchange.Search.RetriableOperations.DoRetriableMapiOperation[SourceType,ReturnType,Parameter1Type,Parameter2Type,Parameter3Type](ThreadLocalCrawlData crawlData, SourceType source, Parameter1Type parameter1, Parameter2Type parameter2, Parameter3Type& parameter3, MapiOperationDelegate`5 operationDelegate)
   at Microsoft.Exchange.Search.NotificationWatcher.GetMapiEvents(Int32 maxEvents, NotificationQueue notificationQueue, Int64& endCount)
   at Microsoft.Exchange.Search.NotificationWatcher.NotificationWatcherThread()), and Operations Manager would never alert it.

image

What ended up fixing this issue for me was to rebuild the full-text index catalog for all of my mailbox databases as per the following KB:

How to Rebuild the Full-Text Index Catalog
http://technet.microsoft.com/en-us/library/aa995966(EXCHG.80).aspx

**Note that the article states that it applies to Exchange 2007 but it works for 2010 as well.

image

Then proceeded to dismount and remount the database that wasn’t truncating the logs to get the logs to truncate.

Once the problematic database was back in good health, I was able to reseed the database onto the new mailbox server without running into the same issue.

No comments: