Tuesday, August 11, 2015

Removing duplicate disposable disks assigned to SanDisk Fusion-io ioVDI enabled virtual desktop

Problem

While reviewing the Hard disk configuration of VMware Horizon View ioVDI enabled virtual desktops, you notice that there are several disposable disks assigned to the VDI:

image

Note the Disk File path: 18/disp0/18disp0.vmdk

image

Note the Disk File path: 18/disp1/18disp0.vmdk

image

Note the Disk File path: 18/disp2/18disp0.vmdk

image

image

image

The following are the paths of each Hard disk with the disposable disks highlighted in red:

[datastore01b:view_lun11] 18/18-checkpoint.vmdk

[datastore01b:view_lun11] 18/disp0/18disp0.vmdk

[datastore01b:view_lun11] 18/disp1/18disp0.vmdk

[datastore01b:view_lun11] 18/disp2/18disp0.vmdk

[datastore01a:view_lun11] 4-vdm-user-disk-D-e7fd3c19-ca5e-4c7e-9531-ac6a22830c49.vmdk

[datastore01b:view_lun11] 18/181-internal.vmdk

Solution

As shown in the desktop properties above, we have 3 disposable disks assigned to the VDI but only one is really being used.  To identify which disk is in use, SSH into the ioVDI management appliance and connect to the vCenter via the following commands:

fio3prd:~ # iovdi vcenter -ln -vu sladmin -va vc01.contoso.com

Enter the vCenter password:

Re-Enter the vCenter password:

Logged in to VMP : vc01.contoso.com

Once successfully logged into vCenter, proceed to execute the following command to list the VDI’s Cache Mode properties for each disk:

iovdi guest -dr -np 18 -gu a-tluk -v

An output similar to the follow will be displayed:

fio3prd:~ # iovdi guest -dr -np 18 -gu a-tluk -v
Enter Guest Password :
Re-Enter Guest Password :
Checking vSCSI filter
VM name: 18
scsi0:1.Cache Mode = write_vector
scsi0:2.Cache Mode = hyper_cache  (Write Vector Candidate)
scsi0:3.Cache Mode = hyper_cache  (Write Vector Candidate)
scsi1:0.Cache Mode = hyper_cache
scsi1:1.Cache Mode = hyper_cache
scsi0:0.Cache Mode = hyper_cache
Duplicate write-vector disks found

vSCSI filter status: Not OK

Checking Write Vector status
VM name: 18
Pagefile and temp folder are redirected

Write Vectoring status: OK

1 tests failed
Add failed guests to an ioVDI config to fix the issues. If already present, reapply the config.

fio3prd:~ #

image

From the output above, the disposable disks that are actually being used are the ones that are labeled as with:

(Write Vector Candidate)

This means that SCSI (0:2):

image

… and SCSI (0:3):

image

… are the ones we can delete so proceed to remove them from within the Virtual Machine Properties then re-execute the command again. Output similar to the following will be displayed:

fio3prd:~ # iovdi guest -dr -np 18 -gu a-tluk -v
Enter Guest Password :
Re-Enter Guest Password :
Checking vSCSI filter
VM name: 18
scsi0:1.Cache Mode = write_vector
scsi1:0.Cache Mode = hyper_cache
scsi1:1.Cache Mode = hyper_cache
scsi0:0.Cache Mode = hyper_cache

vSCSI filter status: OK

Checking Write Vector status
VM name: 18
Pagefile and temp folder are redirected

Write Vectoring status: OK

All OK
fio3prd:~ #

image

If you would like to traverse through all of the desktops rather than doing them independently, the command also accepts wildcards as demonstrated in the following command:

fio3prd:~ # iovdi guest -dr -np VDInamingConvention-* -gu a-tluk -v

Sunday, August 9, 2015

Attempting to update a VMware Horizon View linked-clone pool’s snapshot throws the error: “Active Directory Host Unreachable”

Problem

You’ve updated your linked-clone pool’s master image and attempt to reassign the Default Image Snapshot but while you are able to assign the new snapshot, you receive the following error when you try to apply the setting:

Server Error

Active Directory Host Unreachable

image

Solution

While there can be several reasons why this error would be thrown, this is usually caused by connectivity issues between the server with the View Composer role installed and your Active Directory controllers.  In the example above, I noticed that the vCenter server which had the View Composer role installed was assigned a primary and secondary DNS server that would not be able to look up the internal Active Directory domain zone which would prevent the View Composer role to locate and communicate with the Active Directory Domain Controllers.  The error above went away as soon as I removed those DNS servers and configured the primary and secondary servers to point to the Active Directory Domain Controllers which had DNS installed on them.

Wednesday, August 5, 2015

VMware Horizon View virtual desktops experience temporary drive space issues with SanDisk Fusion-io ioVDI integration

Before I begin, it’s important to note that I am not an expert with the SanDisk Fusion-io ioVDI product and the only exposure I’ve had was with a client who had another consulting company implement it in their VMware Horizon View 6 environment. With that out of the way, I’ve been troubleshooting issues with the VDIs ever since the the ioVDI product was upgraded from 1.0 to 2.0.  The virtual desktops would exhibit sporadic issues with various applications such as BGInfo not being able to load the customized desktop background or Silverlight web pages not loading at all.  The Silverlight web page issue isn’t as obvious so I’ll use the BGInfo as an example.  The screenshot below displays an error message when we try to manually apply the customized background:

Error during WriteFile():

There is not enough space on the disk.

image

This behavior lead us to believe that it had to do with the ioVDI cache / disposable disk so we opened up a ticket to work with a support engineer and we noticed that the Non-Persistent / DisposableDisk was indeed low on space with 1.52MB free:

imageimage

This would sort of explain why the BGInfo couldn’t load the background because the wallpaper would have consumed around 2MB of space which the disposable disk did not have.

Next, we browsed to the folder that ioVDI redirect files to:

\\VirtualDesktopName\c$\Windows\Temp\iotdx-disposable

image

Lassoing the folders and reviewing the properties shows that only 6.30MB of space is actually consumed:

image

Having a feeling that perhaps there were hidden files, we went ahead and configured the folder to list all files and folders:

image

Which immediately revealed a redirected 5GB pagefile.sys file that was just as big as the 5GB disposable disk:

image

image

The page file size was expected because the virtual machine was configured with 6GB of memory.

image

I’m currently still waiting for the ioVDI support engineer to call me back with a recommendation whether to increase the drive space or perhaps do not redirect the page file and will update this post when I get a firm answer.

Update Aug 9, 2015

I received confirmation from the support engineer that the pagefile.sys should be redirected to the disposable disk as it is by design.  The case is currently being escalated to the engineering group because there have been several customers with the same issue.  One of the another engineer I worked with is was able to locate a command to disable the redirection of this file:

iottool redirectpagefile { enable | disable } : Enable or Disable redirection of pagefile.sys. The command

takes effect after reboot.

I haven’t tried this yet because I wanted to wait for engineering to get back to us on a better resolution.

Update Aug 10, 2015

We didn’t get an update from the other support engineer who’s trying to escalate the case so I went ahead and made the change to disable the redirect of the pagefile via the command above, rebooted the VDI and immediately noticed that all the sporadic out of memory, out of disk space error messages and other problems we had went away.  The VDI also feels a lot faster.  Here is a screenshot what the command prompt output looks like when executed directly on the VDI:

image

Friday, July 10, 2015

Using remote PowerShell to log into Office 365 and review archive mailboxes

I’m not much of an Office 365 expert but have recently had the opportunity to work with a client to migrate their on prem Exchange 2010 archive over to O365.  The process of deploying ADFS and DirSync was fairly painless but there seemed to be some confusion when I called into Microsoft Office 365 support where the support engineer did not understand why the migrated archived mailbox did not show up in the EAC (Exchange Admin Center).  To make a long story short, it was not until I worked with the third engineer when I was finally told it’s not to supposed to show up if the user’s mailbox is hosted on prem while the archive is hosted on Office 365.  The purpose of this post is for me to list the steps for reviewing the migrated archive mailboxes in case I need it again in the future.

The first step is to connect to Office 365 by launching PowerShell and execute the following:

$LiveCred = Get-Credential

$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://ps.outlook.com/powershell/ -Credential $LiveCred -Authentication Basic -AllowRedirection

Set-ExecutionPolicy Unrestricted -Force

Import-PSSession $Session

clip_image002

Log in with your administrative O365 credentials:

image

You should see the following output once successfully authenticated:

clip_image002[4]

Proceed an execute the following cmdlet to list a specific user’s archive mailbox:

Get-MailUser -Identity <userName@domain.com> |fl *archive*

image

Do not attempt to use the cmdlet:

Get-Mailbox -archive

… to list a user a user’s archive because it will not work when the user has an on prem mailbox with an O365 hosted archive:

image

To verify that the archive mailbox located on O365 is belongs to the on prem mailbox, compare the ArchiveGuid listed for the archive on O365 and the ArchiveGuid for the mailbox on the on prem mailbox by executing the following cmdlet:

Get-Mailbox -Identity <userName@domain.com> |fl *archive*

image

If you ever wanted to check whether a user’s archive is located on prem or Office 365, you can launch the Test E-mail AutoConfiguration option for Outlook, run the test, navigate to the XML tab and look for the <Type>Archive</T> tag that specifies the SmtpAddress as Office 365 rather than the internal on prem Exchange server:

imageimage

Wednesday, July 8, 2015

Searching through OWA (Outlook Web App) on Exchange 2013 returns only 1 month of results

Problem

You’ve received complaints that users searching their inbox with OWA or Outlook in Online mode only returns 1 month of results.  Using the following Get-MailboxDatabaseCopyStatus cmdlet:

Get-MailboxDatabaseCopyStatus –server <mailboxServerName> | FL *index*,*ma*ser*,databasename

image

… shows that the ContentIndexState is listed as Healthy for the mailbox databases.

You proceed to stop the following services:

  • Microsoft Exchange Search Host Controller
  • Microsoft Exchange Search

Then rename or delete the content index folder named with the GUID of the database and restart the services again forcing Exchange to rebuild the content indexes.  However, you notice that searching still continues to return the same incomplete results.

Solution

This issue took me a bit of time to troubleshoot because attempting to search for anything related to searching points to the solution above but the environment I was working on did not have the ContentIndexState listed as FailedAndSuspended.  I tried searching for the ContentIndexRetryQueueSize variable because the value was high and all of the results pointed me to install CU7 when I already had CU8 installed.

What I found that eventually led me to the underlying issue was the following warning that got repeatedly written to the application log:

Log Name: Application

Source: MSExchangeFastSearch

Event ID: 1009

Level: Warning

image

The indexing of mailbox database Admin encountered an unexpected exception. Error details: Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. ---> Microsoft.Exchange.Search.Core.Abstraction.ComponentFailedPermanentException: Failed to read notifications, MDB: 8f76b2d9-77dd-44e6-a8ef-73d2a2539ae1. ---> Microsoft.Mapi.MapiExceptionMdbOffline: MapiExceptionMdbOffline: Unable to read events. (hr=0x80004005, ec=1142)

Diagnostic context:

Lid: 49384

Lid: 51176 StoreEc: 0x476

Lid: 40680 StoreEc: 0x476

Lid: 43980

Lid: 16354 StoreEc: 0x476

Lid: 38985 StoreEc: 0x476

Lid: 20098

Lid: 20585 StoreEc: 0x476

at Microsoft.Mapi.MapiExceptionHelper.InternalThrowIfErrorOrWarning(String message, Int32 hresult, Boolean allowWarnings, Int32 ec, DiagnosticContext diagCtx, Exception innerException)

at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, IExInterface iUnknown, Exception innerException)

at Microsoft.Mapi.MapiEventManager.ReadEvents(Int64 startCounter, Int32 eventCountWanted, Int32 eventCountToCheck, Restriction filter, ReadEventsFlags flags, Boolean includeSid, Int64& endCounter)

at Microsoft.Exchange.Search.Mdb.NotificationsEventSource.<>c__DisplayClass3.<ReadEvents>b__1()

at Microsoft.Exchange.Search.Mdb.MapiUtil.<>c__DisplayClass1`1.<TranslateMapiExceptionsWithReturnValue>b__0()

at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptions(IDiagnosticsSession tracer, LocalizedString errorString, Action mapiCall)

--- End of inner exception stack trace ---

at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptions(IDiagnosticsSession tracer, LocalizedString errorString, Action mapiCall)

at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptionsWithReturnValue[TReturnValue](IDiagnosticsSession tracer, LocalizedString errorString, Func`1 mapiCall)

at Microsoft.Exchange.Search.Mdb.NotificationsEventSource.ReadEvents(Int64 startCounter, Int32 eventCountWanted, ReadEventsFlags flags, Int64& endCounter)

at Microsoft.Exchange.Search.Mdb.NotificationsEventSource.ReadFirstEventCounter()

at Microsoft.Exchange.Search.Engine.NotificationsEventSourceInfo..ctor(IWatermarkStorage watermarkStorage, INotificationsEventSource eventSource, IDiagnosticsSession diagnosticsSession, MdbInfo mdbInfo)

at Microsoft.Exchange.Search.Engine.SearchFeedingController.DetermineFeederStateAndStartFeeders()

at Microsoft.Exchange.Search.Engine.SearchFeedingController.InternalExecutionStart()

at Microsoft.Exchange.Search.Core.Common.Executable.InternalExecutionStart(Object state)

--- End of inner exception stack trace ---

at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)

at Microsoft.Exchange.Search.Engine.SearchRootController.ExecuteComplete(IAsyncResult asyncResult)

From here, I went ahead to compare the content index catalog sizes between the 3 mailbox databases in the environment and quickly noticed that the problematic mailbox database had a catalog that was 1.5GB while the others were 9.5GBs or more.

My hunch was that the Veeam backups which take place at 11:00p.m. every evening was conflicting with the index building engine so I proceeded to stop the backups for the evening, forced Exchange to rebuild the index catalog over the evening.  After leaving the environment for a day, I went back to test searching on both and Outlook client in Online mode and through OWA and was able to retrieve results older than 1 month.

Thursday, June 25, 2015

Inbound mail submission disabled with event ID: 15004 warning logged on Exchange Server 2013

I recently had to troubleshoot an issue for a client who only had about 30 people in the organization with a mix of Macs and PCs notebooks/desktops accessing a single Exchange Server 2013 server with all roles installed via a mix of MAPI, IMAP4, and EWS protocols. The volume of email exchange within the organization isn’t particularly large but users do tend to send attachments as large as 40MB and Exchange is configured to journal to a 3rd party provider and therefore doubling every attachment email that is sent.

What one of the users noticed was that he would receive the following message on his Mac mail intermittently at various times of the week:

Cannot send message using the server

The sender address some@emailaddress.com was rejected by the server webmail.url.com

The server response was: Insufficient system resources

Select a different outgoing mail server from the list below or click Try Later to leave the message in your Outbox until it can be sent.

image

Reviewing the event logs show that when the user receives the error message above, Exchange also logs the following:

Log Name: Application

Source: MSExchangeTransport

Event ID: 15004 warning:

The resource pressure increased from Medium to High.

The following resources are under pressure:

Version buckets = 278 [High] [Normal=80 Medium=120 High=200]

The following components are disabled due to back pressure:

Inbound mail submission from Hub Transport servers

Inbound mail submission from the Internet

Mail submission from Pickup directory

Mail submission from Replay directory

Mail submission from Mailbox server

Mail delivery to remote domains

Content aggregation

Mail resubmission from the Message Resubmission component.

Mail resubmission from the Shadow Redundancy Component

The following resources are in normal state:

Queue database and disk space ("C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue\mail.que") = 77% [Normal] [Normal=95% Medium=97% High=99%]

Queue database logging disk space ("C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue\") = 80% [Normal] [Normal=94% Medium=96% High=98%]

Private bytes = 6% [Normal] [Normal=71% Medium=73% High=75%]

Physical memory load = 67% [limit is 94% to start dehydrating messages.]

Submission Queue = 0 [Normal] [Normal=2000 Medium=4000 High=10000]

Temporary Storage disk space ("C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Temp") = 80% [Normal] [Normal=95% Medium=97% High=99%]

image

Aside from seeing the pressure go from Medium to High, I’ve also seen pressure go from Normal to High:

The resource pressure increased from Normal to High.

The following resources are under pressure:

Version buckets = 155 [High] [Normal=80 Medium=120 High=200]

The following components are disabled due to back pressure:

Inbound mail submission from Hub Transport servers

Inbound mail submission from the Internet

Mail submission from Pickup directory

Mail submission from Replay directory

Mail submission from Mailbox server

Mail delivery to remote domains

Content aggregation

Mail resubmission from the Message Resubmission component.

Mail resubmission from the Shadow Redundancy Component

The following resources are in normal state:

Queue database and disk space ("C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue\mail.que") = 77% [Normal] [Normal=95% Medium=97% High=99%]

Queue database logging disk space ("C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue\") = 80% [Normal] [Normal=94% Medium=96% High=98%]

Private bytes = 5% [Normal] [Normal=71% Medium=73% High=75%]

Physical memory load = 67% [limit is 94% to start dehydrating messages.]

Submission Queue = 0 [Normal] [Normal=2000 Medium=4000 High=10000]

Temporary Storage disk space ("C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Temp") = 80% [Normal] [Normal=95% Medium=97% High=99%]

image

A bit of researching on the internet pointed me to various reasons why this would happen but one that appeared to the cause of this error in the environment was that users were sending attachments that are too large and therefore filling up the version bucket too fast for Exchange to process. One of the cmdlets a forum users suggested to check was the following:

get-messagetrackinglog -resultsize unlimited -start "04/03/2014 00:00:00" | select sender, subject, recipients,totalbytes | where {$_.totalbytes -gt "20240000"}

Executing the cmdlet above displayed the following:

image

Further digging into the logs revealed that there were quite a few emails with large attachments sent around the time when the warning was logged and after consulting with our Partner Forum support engineer, I decided to go ahead and increase the thresholds that deem the pressure as being Normal, Medium or High.  The keys of interest are found on the Exchange server in the following folder:

C:\Program Files\Microsoft\Exchange Server\V15\Bin

… in the following file:

EdgeTransport.exe.config

image

From within this file, look for the following keys:

<add key=”VersionBucketsHighThreshold” value=”200″ />

<add key=”VersionBucketsMediumThreshold” value=”120″ />

<add key=”VersionBucketsNormalThreshold” value=”80″ />

image

Proceed and change the values to a higher number (I simply doubled the number):

<add key=”VersionBucketsHighThreshold” value=”400″ />

<add key=”VersionBucketsMediumThreshold” value=”240″ />

<add key=”VersionBucketsNormalThreshold” value=”160″ />

image

Another suggested key to change was the:

<add key=”DatabaseMaxCacheSize” value=”134217728” />

… but the value on Exchange 2013 already defaults to the 512MB value so there was no need to modify it.

With these new values in pace, I went ahead and restarted the Microsoft Exchange Transport service to make the new variables take effect.

A month goes by without any complaints and logs reveal that while the version buckets have risen to the Medium threshold, it does not reach the new 400 high threshold. 

image

Another month passes and I get another call from the same user indicating the problem has reappeared.  A quick look at the logs shows that the threshold did reach the new high of 400. With no ideas left, I went ahead and opened a support call with Microsoft and the engineer basically told me the same cause and that was there is most likely a lot of large attachments being sent and thus Exchange is unable to flush the messages held in memory which results in this.  I explained to the engineer that there’s only 30 people and the attachments aren’t that big so the engineer went ahead to export the tracking logs so she could review them.  After about 30 minutes she said that I was right in the sense that while there are attachments but none of them exceed 40MB.  At this point, she said the configuration changes we can try are the following:

  1. Modify the Normal, Medium, High version buckets keys
  2. Modify the DatabaseMaxCacheSize
  3. Modify the QueueDatabaseLoggingFileSize
  4. Modify the DatabaseCheckPointDepthMax
  5. Set limits on send message sizes
  6. Increase the memory on the server

After reviewing the version bucket thresholds I’ve set when I doubled the default values, she said we don’t need to increase them further.  The questions I immediately asked was whether I could if I wanted to and she said yes but we could run the risk of causing the server to crash if it’s set to high that the server runs out of memory.

The DatabaseMaxCacheSize key was unchanged at 512MB so she asked me to change it to:

<add key=”DatabaseMaxCacheSize” value=”1073741824” />

image

The <add key="QueueDatabaseLoggingFileSize" value="5MB" />:

image

… was changed to <add key="QueueDatabaseLoggingFileSize" value="31457280" />:

image

The <add key="DatabaseCheckPointDepthMax" = value="384MB" />:

image

… was changed to <add key="DatabaseCheckPointDepthMax" = value="512MB" />:

image

Next, she used the following cmdlets to review the message size limits currently set:

Get-ExchangeServer | fl name, admindisplayversion, serverrole, site, edition

Get-Transportconfig | fl *size*

Get-sendconnector | fl Name, *size*

Get-receiveconnector | fl Name, *size*

Get-Mailbox -ResultSize Unlimited | fl Name, MaxSendSize, MaxReceiveSize >C:\mbx.txt

Get-MailboxDatabase | FL

Get-Mailboxdatabase -Server <serverName> | FL Identity,IssueWarningQuota,ProhibitSendQuota,ProhibitSendReceiveQuota

Get-Mailbox -server <ServerName> -ResultSize unlimited | Where {$_.UseDatabaseQuotaDefaults -eq $false} |ft DisplayName,IssueWarningQuota,ProhibitSendQuota,@{label="TotalItemSize(MB)";expression={(get-mailboxstatistics $_).TotalItemSize.Value.ToMB()}}

She noticed that I’ve already set limits on the mailbox database but wanted me to sent all of the other connectors and some other configurations to have 100MB limits by executing the following cmdlet:

Get-Transportconfig | Set-Transportconfig -maxreceivesize 100MB Get-Transportconfig | Set-Transportconfig -maxsendsize 100MB Get-sendconnector | Set-sendconnector -MaxMessageSize 100MB Get-receiveconnector | set-receiveconnector -MaxMessageSize 100MB Get-TransportConfig | Set-TransportConfig -ExternalDSNmaxmessageattachsize 100MB -InternalDSNmaxmessageattachsize 100MB Get-Mailbox -ResultSize Unlimited | Set-Mailbox -MaxSendSize 100MB -MaxReceiveSize 100MB

The last item that we could modify was the memory of the server but as there was already 16GB assigned, she said we could leave it as is for now and monitor the event logs over the next while.  It has been 3 weeks and while the version bucket has reached medium, it has been consistently 282 or less than the 400 High threshold set.

Troubleshooting this issue was quite frustrating for me as there wasn’t really a KB article with clear instructions for all these checks so I hope this post will help anyone out there who may experience this issue in their environment.

Friday, June 19, 2015

Adding an account from an external domain with a forest trust configured throws the error: “The security identifer could not be resolved…”

Problem

You’ve successfully deployed a new Windows Server 2012 R2 Remote Desktop Services farm in your environment and have begun assigning permissions to users located in another forest that you are forest trust with:

image

While you are able to browse the domain in the separate forest and select a user or group, you quickly notice you receive the following error message when you attempt to apply the settings:

The security identifier could not be resolved. Ensure that a two-way trust exists for the domain of the selected users.

Exception: The network path was not found.

image

Solution

I’ve come across the same problem with a Windows Server 2008 R2 Remote Desktop Services deployment and it looks like this problem still persists in the newer Windows Server 2012 R2 version. To get around this issue, we would need to create a Domain local group in the domain where the RDS server is installed:

image

… then proceed and add the user or group from the federated forest domain into the Domain local group:

image

… and because we can’t add a Domain local group into any other type of group such as Global or Universal in the domain, we would have to assign it directly to the RDS Collection and RemoteApp:

image

Not exactly the most elegant solution but good enough for a workaround.