Newly P2V / VMware Converter converted HP DL series server hangs on Windows startup

It has probably been 4 or 5 years since I’ve had to P2V old HP DL380 G3 servers with Windows Server 2003 but a recent datacenter virtualization project I’ve been involved in had 2 full racks of them. Going through the motions of virtualizing the servers with VMware Converter brought back good memories as well as reminded me of all the issues I’ve had in the past with these servers where one of them was when a newly converted server would hang at various startup phases for hours and seemingly never really load the operating system properly.

Observe the first screenshot of the:

Please wait…

Preparing network connections…

… process. I could walk around from the virtual machine for an hour, come back and still see this screen being the same. The next screenshot:

Please wait…

Loading your personal settings…

… comes after you’re able to log in and also seemingly takes hours until it loads into the blue desktop wallpaper. In any event, you could wait for hours or days and the server wouldn’t boot or load any faster. I still remember the first time I came across this 5 years ago and how frustrating it was so in hopes that I can save someone else’s frustration, the way to fix this is to restart Windows in safe mode then open the services console as such:

From here, disable all of the HP services so that they don’t start. Notice that you may be prompted with a Found New Hardware Wizard but you should just ignore it because it will go away when you install VMware tools.

Once all the HP services have been disabled, proceed by restarting the server and you should notice that the boot and login times will revert back to normal. Once in the OS, perform the post virtualization cleanup process (remove unneeded applications, hidden devices that no longer exist, etc.

Robocopy script for copying and synchronizing files and folders when migrating Windows file servers

I’m not sure why I haven’t blogged this Robocopy script that I’ve been using over the last 10 years to migrate servers until today when I had trouble finding it because it was in my old emails.  In any case, the following script will mirror the directories between the source and destination servers.  Note that this script uses the /MIR switch which will ensure that the destination folder is completely the same as the source.  This means that if there is a file on the destination folder that isn’t found on the source, it will get deleted so if you accidentally try to mirror an empty source folder that doesn’t contain files with a destination folder that does, all the contents will get deleted!

With dangerous of this script stated, I typically use this script to continuously synchronize files and folders along with the NTFS permissions from an older production file server and a to become production new file server until the night of the cutover when I perform one final synchronization then migrate over.  For information on the switches that are being used, see the following URL:

http://technet.microsoft.com/en-us/library/cc733145.aspx

@ECHO OFF
SETLOCAL

FOR /f "tokens=1-4 delims=/ " %%a in ('date /t') do set vl=%%b%%c%%d
FOR /f "tokens=1-4 delims=/: " %%a in ('time /t') do set tl=%%a_%%b%%c%%dm

SET _source="\\sourceServerName\f$"

SET _dest="\\destinationServerName\f$"

SET _what=/COPYALL /B /SEC /E /MIR /Z
:: /COPYALL :: COPY ALL file info
:: /B :: copy files in Backup mode.
:: /SEC :: copy files with SECurity
:: /E :: copy subdirectories, including Empty ones
:: /MIR :: MIRror a directory tree
:: /Z :: Copy files in restartable mode (survive network glitch).

SET _options=/XO /XF *.LCK /R:5 /W:5 /LOG:Log-SourceServertoDestinationServer_%vl%_%tl%.txt /NFL /NDL /IPG:5
:: /XO :: eXclude Older - if destination file exists and is the same date or newer than the source - don't bother to overwrite it.
:: /XF :: eXclude Files matching given names/paths/wildcards.
:: /R:n :: number of Retries
:: /W:n :: Wait time between retries
:: /LOG :: Output log file
:: /NFL :: No file logging
:: /NDL :: No dir logging
:: /IPG:n :: Inter-Packet Gap (ms), to free bandwidth on slow lines
:: /MON:n :: minimum number of changes that must occur before Robocopy runs again
:: /MOT:m :: minimum time, in minutes, that must elapse before Robocopy runs again

ROBOCOPY %_source% %_dest% %_what% %_options%

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

One of the other switches I commonly use that is not included in the script is the:

/xd <Directory>[ ...]

… that is used to exclude full directories.  This switch comes in handy when you’re trying to copy contents between DFS shares where there a DFS system subfolders handling conflicts.

Attempting to P2V a Windows 2003 SBS Server with VMware Converter fails with: “FAILED: An error occurred during the conversion: ‘BlockLevelVolumeCloneMgr::CloneVolume: Detected a write error during the cloning of volume \WindowsBitmapDriverVolumeID=[04-D4-91-3A-00-20-D8-07-00-00-00-00]. Error: 67109009 (type: 1, code: 4194313)’”

A colleague of mine recently called me during a P2V operation using the VMware Standalone Converter on an older Dell Tower with Windows 2003 SBS server.  He noticed that the cloning process would briefly start and then fail within 10 minutes with the following error:

Error: Unable to clone volume ‘C:’.

FAILED: An error occurred during the conversion: ‘BlockLevelVolumeCloneMgr::CloneVolume: Detected a write error during the cloning of volume \WindowsBitmapDriverVolumeID=[04-D4-91-3A-00-20-D8-07-00-00-00-00]. Error: 67109009 (type: 1, code: 4194313)’

Seeing how the error message was seemingly referencing a bad block on the C drive, I asked him to run a chkdsk /f on the C (note that running a chkdsk on a system drive requires a reboot) as well as any other data drives that were on the server.  My colleague went ahead and ran chkdsk on all of the drives and while some errors were detected and were fixed, the cloning would continue to fail.

After giving it a bit of thought, I asked if he had stopped all of the services (i.e. databases, Exchange, backup agents) on the SBS server prior to launching the cloning operation and the answer was no so I told him to proceed to do so and he was then able to successfully convert the SBS server to a virtual machine:

While I can’t really say what service caused the issue, I hope this post will help anyone running into the same issue as I found very little information when trying to search for the error on the internet.

Exchange 2010 Server DAG Seeding Errors

I recently had to rebuild a failed Exchange 2010 server that was a part of a DAG and while the rebuild went quite smoothly, I had quite a few issues reseeding the database back onto the new server so I thought it would be good to blog 2 of the main issues I had:

Problem #1

Attempting to add a mailbox database copy with the cmdlet Add-MailboxDatabaseCopy throws the error:

[PS] C:\Windows\system32>Add-MailboxDatabaseCopy -Identity "EXDB-02" -MailboxServer MBX-01
A source-side operation failed. Error An error occurred while performing the seed operation. Error: Failed to open a lo
g truncation context to source server 'MBX-02.someDomain.NET'. Hresult: 0xfffffae7. Error: The database was eithe
r not found or was not replicated.. [Database: EXDB-02, Server: MBX-01.someDomain.NET]
    + CategoryInfo          : InvalidOperation: (:) [Add-MailboxDatabaseCopy], SeedInProgressException
    + FullyQualifiedErrorId : 89031D8B,Microsoft.Exchange.Management.SystemConfigurationTasks.AddMailboxDatabaseCopy

[PS] C:\Windows\system32>

You notice that the mailbox server has been added as one of the servers for the mailbox database but the Copy Status is listed as Suspended:

You attempt to use the Resume Database Copy option in the EMC to resume the seeding:

… but notice that the Copy Status quickly changes into Failed and Suspended:

Alternatively, if you use the EMC to add the mailbox database copy, you see the following error:

Summary: 1 item(s). 0 succeeded, 1 failed.
Elapsed time: 00:00:34

EXDB-02
Failed

Error:
A source-side operation failed. Error An error occurred while performing the seed operation. Error: An error occurred while processing a request on server 'MBX-02'. Error: Couldn't open backup file handle for database 'EXDB-02' to server 'MBX-02'. Hresult: 0x50d. Error: A database backup is already in progress. Please verify that no other seeding or incremental reseeding operations are started for this database, and then try the operation again by rerunning the Update-MailboxDatabaseCopy cmdlet.. [Database: EXDB-02, Server: MBX-01.someDomain.NET]

An error occurred while processing a request on server 'MBX-02'. Error: Couldn't open backup file handle for database 'EXDB-02' to server 'MBX-02'. Hresult: 0x50d. Error: A database backup is already in progress. Please verify that no other seeding or incremental reseeding operations are started for this database, and then try the operation again by rerunning the Update-MailboxDatabaseCopy cmdlet.

Couldn't open backup file handle for database 'EXDB-02' to server 'MBX-02'. Hresult: 0x50d. Error: A database backup is already in progress. Please verify that no other seeding or incremental reseeding operations are started for this database, and then try the operation again by rerunning the Update-MailboxDatabaseCopy cmdlet.
Click here for help... http://technet.microsoft.com/en-US/library/ms.exch.err.default(EXCHG.141).aspx?v=14.2.247.1&t=exchgf1&e=ms.exch.err.Ex4543D9

Exchange Management Shell command attempted:
Add-MailboxDatabaseCopy -Identity 'EXDB-02' -MailboxServer 'MBX-01' -ActivationPreference '2'

Elapsed Time: 00:00:34

As ridiculous as may sound, the way I got the seeding to work was to dismount all of the databases on the active server (the one that wasn’t rebuilt), restart the server, mount the databases, and started the seeding. 

Problem #2

You notice that while the seeding of a database starts and seemingly completes, the Copy Status ends up being as Failed and Suspended:

… if you attempt to resume the seeding, the Copy Status will end up being Failed.

Another symptom observed is that the logs on the active database are not truncated even if you turn on circular logging.

Also, you notice the following event logged on the active server:

Exchange Search Indexer has temporarily disabled indexing of the mailbox database EXDB-01 (GUID = 52c3b6c8-c3ef-4912-8b04-e6f7e3eeb438) due to an error (Microsoft.Mapi.MapiExceptionMdbOffline: MapiExceptionMdbOffline: Unable to read events. (hr=0x80004005, ec=1142)
Diagnostic context:
    Lid: 33865 
    Lid: 1494    ---- Remote Context Beg ----
    Lid: 44215 
    Lid: 60049   StoreEc: 0x8004010F
    Lid: 49469 
    Lid: 65341   StoreEc: 0x8004010F
    Lid: 56125 
    Lid: 47933   StoreEc: 0x8004010F
    Lid: 32829 
    Lid: 49213   StoreEc: 0x8004010F
    Lid: 48573 
    Lid: 64957   StoreEc: 0x8004010F
    Lid: 20057   StoreEc: 0x476    
    Lid: 1750    ---- Remote Context End ----
    Lid: 28777   StoreEc: 0x476    
    Lid: 20098 
    Lid: 20585   StoreEc: 0x476    
   at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)
   at Microsoft.Mapi.MapiEventManager.ReadEvents(Int64 startCounter, Int32 eventCountWanted, Int32 eventCountToCheck, Restriction filter, ReadEventsFlags flags, Boolean includeSid, Int64& endCounter)
   at Microsoft.Exchange.Search.RetriableOperations.ReadEvents(ThreadLocalCrawlData unused1, MapiEventManager eventManager, Int64 watermark, Int32 eventCount, Int64& endCounter)
   at Microsoft.Exchange.Search.RetriableOperations.DoRetriableMapiOperation[SourceType,ReturnType,Parameter1Type,Parameter2Type,Parameter3Type](ThreadLocalCrawlData crawlData, SourceType source, Parameter1Type parameter1, Parameter2Type parameter2, Parameter3Type& parameter3, MapiOperationDelegate`5 operationDelegate)
   at Microsoft.Exchange.Search.NotificationWatcher.GetMapiEvents(Int32 maxEvents, NotificationQueue notificationQueue, Int64& endCount)
   at Microsoft.Exchange.Search.NotificationWatcher.NotificationWatcherThread()), and Operations Manager would never alert it.

What ended up fixing this issue for me was to rebuild the full-text index catalog for all of my mailbox databases as per the following KB:

How to Rebuild the Full-Text Index Catalog
http://technet.microsoft.com/en-us/library/aa995966(EXCHG.80).aspx

**Note that the article states that it applies to Exchange 2007 but it works for 2010 as well.

Then proceeded to dismount and remount the database that wasn’t truncating the logs to get the logs to truncate.

Once the problematic database was back in good health, I was able to reseed the database onto the new mailbox server without running into the same issue.

Attempting to add an Exchange 2010 server to a DAG throws the error: “WARNING: Network name 'DAG-01' is not online. Please check that the IP address configuration for the database availability group is correct.”

Problem

You attempt to add a newly built Exchange 2010 mailbox server to an existing DAG but receive the following error:

[PS] C:\Windows\system32>Add-DatabaseAvailabilityGroupServer -Identity DAG-01 -MailboxServer MBX-01
WARNING: Network name 'DAG-01' is not online. Please check that the IP address configuration for the database
availability group is correct.
[PS] C:\Windows\system32>

Solution

While there are probably many different reasons why this error would be thrown, the issue I had was because I had an incorrect subnet mask.  The environment I was working in had a /16 subnet for their server subnet (don’t ask me why) and I had incorrectly used a /24:

Once I corrected the subnet mask to a /16:

The Add-DatabaseAvailabilityGroupServer cmdlet ran successfully and the new mailbox database was added to the DAG:

Exchange Server 2010 Setup.exe with /m:recoverserver switch fails with the error: “Exchange Server 2010 build 14.2.247.5 or later must be used to perform a recovery of this server.”

A colleague recently called me when one of his clients had an Exchange 2010 server fail and because I’ve done this in the past and blogged about it, I pointed him to one of my previous posts here:

Recovering an Exchange 2010 Mailbox Server from a corrupted operating system
http://terenceluk.blogspot.com/2011/08/recovering-exchange-2010-mailbox-server.html

However, while recovering a failed Exchange 2010 server with the setup.exe /m:recoverserver switch he received the following error:

D:\>setup /m:recoverserver

Welcome to Microsoft Exchange Server 2010 Unattended Setup

Setup will continue momentarily, unless you press any key and cancel the

installation. By continuing the installation process, you agree to the license

terms of Microsoft Exchange Server 2010.

If you don't accept these license terms, please cancel the installation. To

review the license terms, please go to

http://go.microsoft.com/fwlink/?LinkId=150127&clcid=0x409/

Press any key to cancel setup................

No key presses were detected. Setup will continue.

Preparing Exchange Setup

Copying Setup Files COMPLETED

The following server roles will be recovered

Languages

Mailbox Role

Management Tools

Performing Microsoft Exchange Server Prerequisite Check

Configuring Prerequisites COMPLETED

Language Pack Checks COMPLETED

Mailbox Role Checks FAILED

Exchange Server 2010 build 14.2.247.5 or later must be used to perform a re

covery of this server.

The Exchange Server setup operation didn't complete. More details can be found

in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.

Exchange Server setup encountered an error.

D:\>

One of the items I forgot to mention in my blog post which was also brought up by one of the visitors in the comments is that you need to use installation binaries that are at the same service pack level as the server you are recovering.  In my previous blog post, it was SP1 but for the server my colleague was recovering, it was SP2.  As most would know, there is no Exchange Server 2010 with SP2 ISOs for download on the Microsoft site and for some odd reason, I haven’t come across any documentation from Microsoft or the TechNet article:

Recover a Database Availability Group Member Server
http://technet.microsoft.com/en-us/library/dd638206(v=exchg.141).aspx

… stating that you can actually download Exchange Server 2010 SP2, extract it and use it as installation binaries.  So if you are to receive this error, simply download the 535MB Exchange2010-SP2-x64.exe file:

… extract it into a folder:

… then run setup.exe /m:recoverserver to perform the install:

Installing Windows Server 2012 on a VMware ESXi 5.1 hypervisor throws the error message: “A media driver your computer needs is missing. This could be a DVD, USB or Hard disk driver…”

A colleague of mine recently reached out to me about an issue while installing Windows Server 2012 on a VMware ESXi 5.1 hypervisor where he would receive the following error message:

A media driver your computer needs is missing. This could be a DVD, USB or Hard disk driver. If you have a CD, DVD, or USB flash drive with the driver on it, please inesrt it now.

Note: If the installation media for Windows is in the DVD drive or on a USB drive, you can safely remove it for this step.

Hide drivers that aren’t compatible with this computer’s hardware.

I recall running into a similar issue with Windows Server 2008 R2 before where an error similar to the above was thrown when an incomplete or corrupted ISO was used for the install so I asked him to compare the file sizes between his download and one that I had on my laptop and we immediately noticed that the sizes were different:

Bad ISO:

Name: SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO
File Size: 2,635,776 KB

Good ISO:

Name: SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO
File Size: 3,607,534 KB

Basically what happened with his download was that an inspection module on firewalls he initiated the download from caused his download to be partly downloaded while the download manager suggested it was completed.  My guess is that the ISO wasn’t semi readable and therefore the install kicked off but when the drivers required were missing, the install would not continue.  What I ended up doing was giving him the ISO I had to try the install again and the install completed as expected.