Pages

Friday, July 14, 2017

“vSphere Replication does not support changing the length of a replicated disk.” error is thrown when attempting to expand a hard disk of replicate virtual machine

Problem

You attempt to expand a hard disk of a vSphere Replication replicated virtual machine but immediately receive the following error:

Reconfigure virtual machine

Invalid or unsupported virtual machine configuration.

See the error stack for details on the cause of this problem.

vSphere Replication does not support changing the length of a replicated disk.

image

Solution

The reason why the error is thrown is because vSphere Replication prevents sizing changes to the protected copy of the virtual machine (source) when it is replicated to a recovery copy of the virtual machine (target).  This makes sense because the replication engine likely tracks the changes in a certain way where changing the disk size would cause issues.  VMware has released the following two KBs to explaining the steps required to expand a replicated virtual machine’s hard disk:

Resizing virtual machine disk files that are protected by vSphere Replication (VR) using VMware vCenter Site Recovery Manager (2042790)
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2042790

Cannot resize the vmdk files during replication which are protected by vSphere replication (2052883)
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2052883

… but I find that the instructions aren’t completely clear so I thought I’d demonstrate the process with screenshots so there are no confusions.

Step #1 - Document Replicated Virtual Machine's vSphere Replication Configuration

Begin by documenting replicated virtual machine's vSphere Replication configuration because you will need this information when specifying the Target Location in one of the later steps (screenshots usually suffice assuming the path fits in the text field):

image

Ensure that you can read the full path of the Target Location:

image

The same applies to every Hard disk Target Location field:

image

image

Document the Quiescing method configuration:

image

Document the Recovery Settings and I usually select Cancel to avoid unintentionally making any changes:

image

Step #2 - Rename the Replicated Virtual Machine's Datastore Folder(s)

Proceed to browsing the Target Location datastore of where the replicated virtual machine is stored.  Note that this is the replicated copy and NOT the live copy:

image

One of the reasons why it is important to document the Target Location of the replicated copy is because the VMDK files are not always stored in the same directory as the VMX files as shown in this example:

image

Proceed to rename the replicated copy's folder as such.  Remember that this is the replicated copy and NOT the live copy:

image

Rename any additional folders that store the replicated virtual machine's files:

image

Step #3 - Stop the Virtual Machine's Replication 

I find the step #2 outlined in the KB:

Cannot resize the vmdk files during replication which are protected by vSphere replication(2052883)
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2052883

... unclear but it states:

2. Disable replication of the virtual machine you want to resize.

The fact that there is no disable option causes confusion.  Step #3 in the KB article:

Resizing virtual machine disk files that are protected by vSphere Replication (VR) using VMware vCenter Site Recovery Manager (2042790)
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2042790

... is much clearer as it states:

3. Stop replication for the virtual machine at the protected site using the vSphere Replication User Interface (UI).

So with this in mind, proceed to right-clicking on the replicate virtual machine in the vSphere Replication console and select Stop:

image

image

Note that it is important to you have renamed the Target Location virtual machine folders.  From what I've seen, if the replicated VM was seeded then the target files would not be deleted but if the replicated VM was not seeded then the files would get deleted.

image

You should now see tasks executed under the Recent Tasks pane indicating replication is being disabled for the virtual machine:

image

The virtual machine should no longer be displayed once the operation completes.

Step #4 - Expand Source/Live Virtual Machine's VMDK

With replication stopped, you should now be able to expand the source/live virtual machine's VMDKs so proceed to expanding them to the size required.

Step #5 - Expand Target/Replicated Virtual Machine's VMDK

Since the target/replicated virtual machine is not inventoried on a host, expanding the drives will need to be done with the vmkfstools command.  Proceed by either accessing the console or SSH to a host that has access to the datastore and navigate to the directory of the renamed folders containing the replicated virtual machine files. 

**The ls -lah command can be used to display the files in the command line.

Once in the directory containing the files, proceed to increase the hard drive VMDK file with the command:

vmkfstools -x XXXGB <filenameOfVMDK>

A similar output below will be displayed upon successfully increasing the VMDK:

image

Refreshing the datastore browser will show the new size of the VMDK:

image

Continue by renaming the folder back to the original name:

image

image

Proceed to reconfigure replication for the virtual machine:

image

image

imageimage

image

The default Target location that is used will most likely be different than the folder with the replicated VMDKs so use the previously documented configuration to select the same Target location as the original location:

image

Once the previous Target location has been configured, the amount of hard disks of the replicated virtual machine will now be displayed (there are 4 in this case):

image

Configure all of the hard disks to use the same folder as the previous location:

image

image

Selecting the folder with the existing replicated VMDK will display the following message:

Replication Seed Confirmation

Duplicate file found. Do you want to use this file as a seed?

Select Yes when receiving this prompt:

image

image

Continue and repeat the same procedure for the rest of the disks and the same Replication Seed Confirmation prompt should be displayed:

imageimage

image

----------------------------------------------------------------------------------------------------------------------------------------------

Note that I’ve noticed there are times when the wizard would prompt all of the disks at the same time rather than prompting for each individual disks as shown above:

image

----------------------------------------------------------------------------------------------------------------------------------------------

Configure the Quiescing method as previously documented:

image

Configure the Recovery settings as previously documented and complete the configuration by clicking Finish::

imageimage

The replicated virtual machine should now be displayed again with an Initial Full Sync Status:

image

Clicking on the i icon in the GUI would provide information:

image

To obtain more information on the status of the synchronization, log onto the esxi host with the protected VM is inventoried and execute:

vim-cmd vmsvc/getallvms

... to list all the VMs along with their Vmid:

image

Note the Vmid and execute the command:

vim-cmd hbrsvc/vmreplica.getState <Vmid>

This will display an output similar to the following:

image

I usually periodically execute the vim-cmd hbrsvc/vmreplica.getState command to check the progress as it provides more information:

image

The time it takes for the process to complete will vary depending on the size of the virtual machine but note that only changes are replicated over and not the full virtual machine.  The following are some screenshots taken during the synchronization:

imageimage

image

After the required time, the virtual machine should get back to an OK Status as such:

image

Hope this would help anyone looking for what the full process of expanding a replicated virtual machine’s hard disk would look like.

Monday, July 3, 2017

Attempting to migrate mailboxes from Exchange 2010 to 2016 stalls with StatusDetail: StalledDueToTarget_MailboxCapacityExceeded

Problem

You’re in the process of migrating mailboxes from Exchange 2010 to Exchange 2016 with both a live as well as an archive mailbox and while some mailboxes successfully move to the new databases, you noticed that others remains in Syncing status indefinitely:

image

image

Expanding the More Details… option show the synchronization has stalled for quite some time:

image

Executing the cmdlet Get-MoveRequest | Get-MoveRequestStatstics -Format-Table -AutoSize displays the StatusDetail StalledDueToTarget_MailboxCapacityExceeded:

image

Executing the cmdlets:

Get-MoveRequest administrator | FL

… or:

Get-MoveRequest | Get-MoveRequestStatistics | FL

… does not provide additional information.

Reviewing the properties of the migration job and clicking on Report: Download the report for this user:

image

… displays a report with the following log output:

7/2/2017 10:14:11 AM [CONTBMEXMB01] '' created move request.
7/2/2017 10:14:11 AM [CONTBMEXMB01] '' allowed a large amount of data loss when moving the mailbox (50 bad items, 0 large items).
7/2/2017 11:07:56 AM [CONTBMEXMB01] Relinquishing job because of large delays due to unfavorable server health or budget limitations with a request throttling state 'StalledDueToTarget_Processor'.
7/2/2017 11:51:16 AM [CONTBMEXMB01] Relinquishing job because of large delays due to unfavorable server health or budget limitations with a request throttling state 'StalledDueToTarget_Processor'.
7/2/2017 11:15:00 PM [CONTBMEXMB01] '' suspended move request.
7/2/2017 11:15:02 PM [CONTBMEXMB01] Suspending job.
7/2/2017 11:15:02 PM [CONTBMEXMB01] Relinquishing job.
7/3/2017 10:04:41 AM [CONTBMEXMB01] '' resumed move request.
7/3/2017 10:04:45 AM [CONTBMEXMB01] Job resumed with status 'Queued'.
7/3/2017 10:04:45 AM [CONTBMEXMB01] Relinquishing job.

Attempting to log onto the Exchange 2016 server and adjusting the parameters MaxActiveJobsPerSourceMailbox and MaxActiveJobsPerTargetMailbox in the configuration file MSExchangeMailboxReplication.exe.config located in the directory E:\Program Files\Microsoft\Exchange Server\V15\Bin does not correct the issue:

image

imageimage

image

Solution

Attempting to search for the error messages:

StalledDueToTarget_MailboxCapacityExceeded:

… and:

StalledDueToTarget_Processor

… did not return any helpful posts and what ended up being the problem was the archive mailbox server we were using to move the archive mailboxes to.  The server’s CPU utilization wasn’t particularly high (2%), memory usage was average (90%) but the server uptime was 82 days and there were pending Windows patches asking for a reboot. Previous mailboxes that were stalled would successfully completed after the server restart:

image

Hope this helps anyone who may encounter this issue and unable to find any useful information on the internet.