Moving O365 (Office 365) archive mailbox to on-prem Exchange server

I’ve recently been involved with an Office 365 archiving pilot project to demonstrate the user experience and performance for a client to see if it met their requirements and noticed that while it was quite easy to move the archive mailbox from the on-prem Exchange database to O365, there did not seem to be a way to move the mailbox back to the on-prem Exchange. After failing to find instructions through search, I ended up calling Microsoft O365 support for assistance and what ended up being the solution was to use PowerShell. As I’m sure there will be others who find themselves in the same situation as I did, I thought it would be good to write this blog post to demonstrate the steps.

To move the mailbox from O365 to your on-prem Exchange, execute the following New-MoveRequest cmdlet replacing the unique parameters:

$cred = get-credential // Enter your On Premise Admin credential

New-MoverRquest -identity user@domain.com -remotehostname mail.domain.com -archiveonly -archivedomain “domain.com” -Outbound -remotearchivetargetdatabase “db1” -remotecredential $cred

Note the following parameters that need to be changed for your environment:

user@domain.com – replace with the user’s email address
mail.domain.com – your migration endpoint configured in Office 365
domain.com – your SMTP domain
db1 – the name of the on-prem archive database that this user’s archive will be moved to

Note that the above immediately begins the move and as with all move requests done on an on-prem Exchange, this move would fail if it encounters any corrupted items. Skipping corrupted items is the same as a regular on-prem Exchange move and to specify a threshold for corrupted items, use the parameter -BadItemLimit # where # is the amount of corrupted item threshold you would like to set.

Get-MoveRequest on its own or with the | FL parameter can be used to review the status of the mailbox move:

The Exchange Management Console and navigating to Office 365 > Recipient Configuration > Move Request can also be used for reviewing the status of the archive mailbox migration as such:

Continue to wait until the move is completed and you should be able to see that the on-prem Exchange archive database is specified as the user’s Archive database:

The on-prem Exchange Management Shell can also be used to confirm that the user’s archive mailbox database is now on-prem by executing:

Get-Mailbox -identity user@domain.com –archive | FL

Note that the ArchiveDatabase is listed as blank when the user’s archive mailbox is still on O365:

After the move, you should see that the field is now populated with the on-prem Exchange archive database:

Hope this will help anyone looking for off boarding an Office 365 Online Archive mailbox back onto their on-prem Exchange.

------------------------------------------------------------------------------------------------------------------------------------------------------------------

An additional note I’d like to make is if you would like to list all the archive mailboxes active on Office 365, execute the following cmdlet:

Get-MailUser | Where-Object {$_.ArchiveStatus -match "Active"} | fl DisplayName,*archive*

AD FS and DirSync services fail to start after server restart

Problem

You’ve successfully installed AD FS and DirSync on their respective Windows Server 2012 R2 servers and have confirmed that both are working as expected. However, you also realize that the services on the AD FS and DirSync servers no fail to start as soon as you restart the server:

DirSync

Service Name: FIMSynchronizationService
Display Name: Forefront Identity Manager Synchronization Service
Service Account: .\AAD_d5b89680b957

Service Name: MSOnlineSyncScheduler
Display Name: Windows Azure Active Directory Sync Service
Service Account: .\AAD_d5b89680b957

AD FS

Service Name: Adfssrv
Display Name: Active Directory Federation Services
Service Account: <nonGeneric>

Windows could not start the Active Directory Federation Services service on the Local Computer.

Error 1069: The service did not start due to a logon failure.

Solution

While there could be various reasons why this issue may occur, one of them is if you have a GPO configured in your domain that specifies what accounts are allowed to have Log on as service rights.  In the environment I worked in, there was such a policy so when I launched the Local Computer Policy editor with gpedit.msc:

… I can see that the options to edit the Log on as a service configuration greyed out:

The reason why the AD FS and DirSync worked initially is because the install manually granted these service accounts the rights but a restart of the server removed them.

Troubleshooting this issue didn’t actually take me too much time but I can see that it could have if I missed this so I hope this will safe some time for anyone who may encounter the same issue.

Removing duplicate disposable disks assigned to SanDisk Fusion-io ioVDI enabled virtual desktop

Problem

While reviewing the Hard disk configuration of VMware Horizon View ioVDI enabled virtual desktops, you notice that there are several disposable disks assigned to the VDI:

Note the Disk File path: 18/disp0/18disp0.vmdk

Note the Disk File path: 18/disp1/18disp0.vmdk

Note the Disk File path: 18/disp2/18disp0.vmdk

The following are the paths of each Hard disk with the disposable disks highlighted in red:

[datastore01b:view_lun11] 18/18-checkpoint.vmdk

[datastore01b:view_lun11] 18/disp0/18disp0.vmdk

[datastore01b:view_lun11] 18/disp1/18disp0.vmdk

[datastore01b:view_lun11] 18/disp2/18disp0.vmdk

[datastore01a:view_lun11] 4-vdm-user-disk-D-e7fd3c19-ca5e-4c7e-9531-ac6a22830c49.vmdk

[datastore01b:view_lun11] 18/181-internal.vmdk

Solution

As shown in the desktop properties above, we have 3 disposable disks assigned to the VDI but only one is really being used.  To identify which disk is in use, SSH into the ioVDI management appliance and connect to the vCenter via the following commands:

fio3prd:~ # iovdi vcenter -ln -vu sladmin -va vc01.contoso.com

Enter the vCenter password:

Re-Enter the vCenter password:

Logged in to VMP : vc01.contoso.com

Once successfully logged into vCenter, proceed to execute the following command to list the VDI’s Cache Mode properties for each disk:

iovdi guest -dr -np 18 -gu a-tluk -v

An output similar to the follow will be displayed:

fio3prd:~ # iovdi guest -dr -np 18 -gu a-tluk -v
Enter Guest Password :
Re-Enter Guest Password :
Checking vSCSI filter
VM name: 18
scsi0:1.Cache Mode = write_vector
scsi0:2.Cache Mode = hyper_cache  (Write Vector Candidate)
scsi0:3.Cache Mode = hyper_cache  (Write Vector Candidate)
scsi1:0.Cache Mode = hyper_cache
scsi1:1.Cache Mode = hyper_cache
scsi0:0.Cache Mode = hyper_cache
Duplicate write-vector disks found

vSCSI filter status: Not OK

Checking Write Vector status
VM name: 18
Pagefile and temp folder are redirected

Write Vectoring status: OK

1 tests failed
Add failed guests to an ioVDI config to fix the issues. If already present, reapply the config.

fio3prd:~ #

From the output above, the disposable disks that are actually being used are the ones that are labeled as with:

(Write Vector Candidate)

This means that SCSI (0:2):

… and SCSI (0:3):

… are the ones we can delete so proceed to remove them from within the Virtual Machine Properties then re-execute the command again. Output similar to the following will be displayed:

fio3prd:~ # iovdi guest -dr -np 18 -gu a-tluk -v
Enter Guest Password :
Re-Enter Guest Password :
Checking vSCSI filter
VM name: 18
scsi0:1.Cache Mode = write_vector
scsi1:0.Cache Mode = hyper_cache
scsi1:1.Cache Mode = hyper_cache
scsi0:0.Cache Mode = hyper_cache

vSCSI filter status: OK

Checking Write Vector status
VM name: 18
Pagefile and temp folder are redirected

Write Vectoring status: OK

All OK
fio3prd:~ #

If you would like to traverse through all of the desktops rather than doing them independently, the command also accepts wildcards as demonstrated in the following command:

fio3prd:~ # iovdi guest -dr -np VDInamingConvention-* -gu a-tluk -v

Attempting to update a VMware Horizon View linked-clone pool’s snapshot throws the error: “Active Directory Host Unreachable”

Problem

You’ve updated your linked-clone pool’s master image and attempt to reassign the Default Image Snapshot but while you are able to assign the new snapshot, you receive the following error when you try to apply the setting:

Server Error

Active Directory Host Unreachable

Solution

While there can be several reasons why this error would be thrown, this is usually caused by connectivity issues between the server with the View Composer role installed and your Active Directory controllers.  In the example above, I noticed that the vCenter server which had the View Composer role installed was assigned a primary and secondary DNS server that would not be able to look up the internal Active Directory domain zone which would prevent the View Composer role to locate and communicate with the Active Directory Domain Controllers.  The error above went away as soon as I removed those DNS servers and configured the primary and secondary servers to point to the Active Directory Domain Controllers which had DNS installed on them.

VMware Horizon View virtual desktops experience temporary drive space issues with SanDisk Fusion-io ioVDI integration

Before I begin, it’s important to note that I am not an expert with the SanDisk Fusion-io ioVDI product and the only exposure I’ve had was with a client who had another consulting company implement it in their VMware Horizon View 6 environment. With that out of the way, I’ve been troubleshooting issues with the VDIs ever since the the ioVDI product was upgraded from 1.0 to 2.0.  The virtual desktops would exhibit sporadic issues with various applications such as BGInfo not being able to load the customized desktop background or Silverlight web pages not loading at all.  The Silverlight web page issue isn’t as obvious so I’ll use the BGInfo as an example.  The screenshot below displays an error message when we try to manually apply the customized background:

Error during WriteFile():

There is not enough space on the disk.

This behavior lead us to believe that it had to do with the ioVDI cache / disposable disk so we opened up a ticket to work with a support engineer and we noticed that the Non-Persistent / DisposableDisk was indeed low on space with 1.52MB free:

This would sort of explain why the BGInfo couldn’t load the background because the wallpaper would have consumed around 2MB of space which the disposable disk did not have.

Next, we browsed to the folder that ioVDI redirect files to:

\\VirtualDesktopName\c$\Windows\Temp\iotdx-disposable

Lassoing the folders and reviewing the properties shows that only 6.30MB of space is actually consumed:

Having a feeling that perhaps there were hidden files, we went ahead and configured the folder to list all files and folders:

Which immediately revealed a redirected 5GB pagefile.sys file that was just as big as the 5GB disposable disk:

The page file size was expected because the virtual machine was configured with 6GB of memory.

I’m currently still waiting for the ioVDI support engineer to call me back with a recommendation whether to increase the drive space or perhaps do not redirect the page file and will update this post when I get a firm answer.

Update Aug 9, 2015

I received confirmation from the support engineer that the pagefile.sys should be redirected to the disposable disk as it is by design.  The case is currently being escalated to the engineering group because there have been several customers with the same issue.  One of the another engineer I worked with is was able to locate a command to disable the redirection of this file:

iottool redirectpagefile { enable | disable } : Enable or Disable redirection of pagefile.sys. The command

takes effect after reboot.

I haven’t tried this yet because I wanted to wait for engineering to get back to us on a better resolution.

Update Aug 10, 2015

We didn’t get an update from the other support engineer who’s trying to escalate the case so I went ahead and made the change to disable the redirect of the pagefile via the command above, rebooted the VDI and immediately noticed that all the sporadic out of memory, out of disk space error messages and other problems we had went away.  The VDI also feels a lot faster.  Here is a screenshot what the command prompt output looks like when executed directly on the VDI:

Using remote PowerShell to log into Office 365 and review archive mailboxes

I’m not much of an Office 365 expert but have recently had the opportunity to work with a client to migrate their on prem Exchange 2010 archive over to O365.  The process of deploying ADFS and DirSync was fairly painless but there seemed to be some confusion when I called into Microsoft Office 365 support where the support engineer did not understand why the migrated archived mailbox did not show up in the EAC (Exchange Admin Center).  To make a long story short, it was not until I worked with the third engineer when I was finally told it’s not to supposed to show up if the user’s mailbox is hosted on prem while the archive is hosted on Office 365.  The purpose of this post is for me to list the steps for reviewing the migrated archive mailboxes in case I need it again in the future.

The first step is to connect to Office 365 by launching PowerShell and execute the following:

$LiveCred = Get-Credential

$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://ps.outlook.com/powershell/ -Credential $LiveCred -Authentication Basic -AllowRedirection

Set-ExecutionPolicy Unrestricted -Force

Import-PSSession $Session

Log in with your administrative O365 credentials:

You should see the following output once successfully authenticated:

Proceed an execute the following cmdlet to list a specific user’s archive mailbox:

Get-MailUser -Identity <userName@domain.com> |fl *archive*

Do not attempt to use the cmdlet:

Get-Mailbox -archive

… to list a user a user’s archive because it will not work when the user has an on prem mailbox with an O365 hosted archive:

To verify that the archive mailbox located on O365 is belongs to the on prem mailbox, compare the ArchiveGuid listed for the archive on O365 and the ArchiveGuid for the mailbox on the on prem mailbox by executing the following cmdlet:

Get-Mailbox -Identity <userName@domain.com> |fl *archive*

If you ever wanted to check whether a user’s archive is located on prem or Office 365, you can launch the Test E-mail AutoConfiguration option for Outlook, run the test, navigate to the XML tab and look for the <Type>Archive</T> tag that specifies the SmtpAddress as Office 365 rather than the internal on prem Exchange server:

To list all the archive mailboxes that are hosted on Office 365, execute the following cmdlet:

Get-MailUser | Where-Object {$_.ArchiveStatus -match "Active"} | fl DisplayName,*archive*

Searching through OWA (Outlook Web App) on Exchange 2013 returns only 1 month of results

Problem

You’ve received complaints that users searching their inbox with OWA or Outlook in Online mode only returns 1 month of results.  Using the following Get-MailboxDatabaseCopyStatus cmdlet:

Get-MailboxDatabaseCopyStatus –server <mailboxServerName> | FL *index*,*ma*ser*,databasename

… shows that the ContentIndexState is listed as Healthy for the mailbox databases.

You proceed to stop the following services:

• Microsoft Exchange Search Host Controller
• Microsoft Exchange Search

Then rename or delete the content index folder named with the GUID of the database and restart the services again forcing Exchange to rebuild the content indexes.  However, you notice that searching still continues to return the same incomplete results.

Solution

This issue took me a bit of time to troubleshoot because attempting to search for anything related to searching points to the solution above but the environment I was working on did not have the ContentIndexState listed as FailedAndSuspended.  I tried searching for the ContentIndexRetryQueueSize variable because the value was high and all of the results pointed me to install CU7 when I already had CU8 installed.

What I found that eventually led me to the underlying issue was the following warning that got repeatedly written to the application log:

Log Name: Application

Source: MSExchangeFastSearch

Event ID: 1009

Level: Warning

The indexing of mailbox database Admin encountered an unexpected exception. Error details: Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. ---> Microsoft.Exchange.Search.Core.Abstraction.ComponentFailedPermanentException: Failed to read notifications, MDB: 8f76b2d9-77dd-44e6-a8ef-73d2a2539ae1. ---> Microsoft.Mapi.MapiExceptionMdbOffline: MapiExceptionMdbOffline: Unable to read events. (hr=0x80004005, ec=1142)

Diagnostic context:

Lid: 49384

Lid: 51176 StoreEc: 0x476

Lid: 40680 StoreEc: 0x476

Lid: 43980

Lid: 16354 StoreEc: 0x476

Lid: 38985 StoreEc: 0x476

Lid: 20098

Lid: 20585 StoreEc: 0x476

at Microsoft.Mapi.MapiExceptionHelper.InternalThrowIfErrorOrWarning(String message, Int32 hresult, Boolean allowWarnings, Int32 ec, DiagnosticContext diagCtx, Exception innerException)

at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, IExInterface iUnknown, Exception innerException)

at Microsoft.Mapi.MapiEventManager.ReadEvents(Int64 startCounter, Int32 eventCountWanted, Int32 eventCountToCheck, Restriction filter, ReadEventsFlags flags, Boolean includeSid, Int64& endCounter)

at Microsoft.Exchange.Search.Mdb.NotificationsEventSource.<>c__DisplayClass3.<ReadEvents>b__1()

at Microsoft.Exchange.Search.Mdb.MapiUtil.<>c__DisplayClass1`1.<TranslateMapiExceptionsWithReturnValue>b__0()

at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptions(IDiagnosticsSession tracer, LocalizedString errorString, Action mapiCall)

--- End of inner exception stack trace ---

at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptions(IDiagnosticsSession tracer, LocalizedString errorString, Action mapiCall)

at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptionsWithReturnValue[TReturnValue](IDiagnosticsSession tracer, LocalizedString errorString, Func`1 mapiCall)

at Microsoft.Exchange.Search.Mdb.NotificationsEventSource.ReadEvents(Int64 startCounter, Int32 eventCountWanted, ReadEventsFlags flags, Int64& endCounter)

at Microsoft.Exchange.Search.Mdb.NotificationsEventSource.ReadFirstEventCounter()

at Microsoft.Exchange.Search.Engine.NotificationsEventSourceInfo..ctor(IWatermarkStorage watermarkStorage, INotificationsEventSource eventSource, IDiagnosticsSession diagnosticsSession, MdbInfo mdbInfo)

at Microsoft.Exchange.Search.Engine.SearchFeedingController.DetermineFeederStateAndStartFeeders()

at Microsoft.Exchange.Search.Engine.SearchFeedingController.InternalExecutionStart()

at Microsoft.Exchange.Search.Core.Common.Executable.InternalExecutionStart(Object state)

--- End of inner exception stack trace ---

at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)

at Microsoft.Exchange.Search.Engine.SearchRootController.ExecuteComplete(IAsyncResult asyncResult)

From here, I went ahead to compare the content index catalog sizes between the 3 mailbox databases in the environment and quickly noticed that the problematic mailbox database had a catalog that was 1.5GB while the others were 9.5GBs or more.

My hunch was that the Veeam backups which take place at 11:00p.m. every evening was conflicting with the index building engine so I proceeded to stop the backups for the evening, forced Exchange to rebuild the index catalog over the evening.  After leaving the environment for a day, I went back to test searching on both and Outlook client in Online mode and through OWA and was able to retrieve results older than 1 month.