Friday, April 21, 2017

Attempting to run an Export job with Microsoft Forefront Identity Manager 2010 R2 throws the error: “stopped-extension-dll-exception”

Problem

You’ve noticed that your previously operational Microsoft Forefront Identity Manager 2010 R2 throws the error the following error when you execute an Export job:

stopped-extension-dll-exception

image

Exchange 2010 contacts in the are either no longer updated or created in the source domain. 

You proceed into the connector’s properties under Management Agents:

image

Review and confirm that the service account is correct:

imageimage

Reviewing the event logs show the following errors displayed in the Application logs:

Log Name: Application

Source: FIMSynchronizationService

Event ID: 6803

Level: Error

Task Category: Management Agent Run Profile

image

The management agent "FIM Connector" failed on run profile "Export" because the server encountered errors.

image

Log Name: Application

Source: FIMSynchronizationService

Event ID: 0

Level: Error

Task Category: None

image

The description for Event ID 0 from source FIMSynchronizationService cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

There is an error in Exch2010Extension BeginExportToCd() function.Type: System.Management.Automation.Remoting.PSRemotingTransportException

Message: Connecting to remote server failed with the following error message : The WS-Management service cannot process the request. The system load quota of 1000 requests per 2 seconds has been exceeded. Send future requests at a slower rate or raise the system quota. The next request from this user will not be approved for at least 1316187520 milliseconds. For more information, see the about_Remote_Troubleshooting Help topic.

Stack Trace: at System.Management.Automation.Runspaces.Internal.RunspacePoolInternal.EndOpen(IAsyncResult asyncResult)

at System.Management.Automation.Runspaces.RunspacePool.Open()

at System.Management.Automation.RemoteRunspace.Open()

at Exch2010Extension.Exch2010ExtensionClass.OpenConnection(String uri, PSCredential credential)

at Exch2010Extension.Exch2010ExtensionClass.BeginExportToCd(String connectTo, String domain, String server, String user, String password)

the message resource is present but the message is not found in the string/message table

You attempt to use the following TechNet article to further troubleshoot by disabling Exchange provisioning confirming that the export now completes and manually executing the included PowerShell cmdlet:

FIM Troubleshooting: stopped-dll-exception: WinRM cannot process the request: Access Denied
https://social.technet.microsoft.com/wiki/contents/articles/15091.fim-troubleshooting-stopped-dll-exception-winrm-cannot-process-the-request-access-denied.aspx

imageimage

.. but you run into the error:

'contoso.com/contoso/Employees/TMRUK/GalContacts/Matthew  Evans' have been modified.
WARNING: The command completed successfully but no settings of
'contoso.com/contoso/Employees/TMRUK/GalContacts/Buu Truong' have been modified.
WARNING: The command completed successfully but no settings of
'contoso.com/contoso/Employees/TMRUK/GalContacts/Gemma Gregson' have been modified.
"DG_TMRUK_Pricing" is a MailForestContact and can't be modified.
    + CategoryInfo          : NotSpecified: (contoso...G_TMRUK_Pricing:ADObjectId) [Set-MailContact], TaskInva
   lidOperationException
    + FullyQualifiedErrorId : 69D6CABF,Microsoft.Exchange.Management.RecipientTasks.SetMailContact

"Terence Luk" is a MailForestContact and can't be modified.
    + CategoryInfo          : NotSpecified: (contoso...cts/Terence Luk:ADObjectId) [Set-MailContact], TaskInva
   lidOperationException
    + FullyQualifiedErrorId : 1DAD038F,Microsoft.Exchange.Management.RecipientTasks.SetMailContact

WARNING: The command completed successfully but no settings of
'contoso.com/contoso/Employees/TMRUK/GalContacts/Anna Ivanova' have been modified.
WARNING: The command completed successfully but no settings of
'contoso.com/contoso/Employees/TMRUK/GalContacts/DG_Operations' have been modified.
"Taro Murakami" is a MailForestContact and can't be modified.
    + CategoryInfo          : NotSpecified: (contoso...s/Taro Murakami:ADObjectId) [Set-MailContact], TaskInva
   lidOperationException
    + FullyQualifiedErrorId : 5FD06EB8,Microsoft.Exchange.Management.RecipientTasks.SetMailContact

"Sara Perdichizzi" is a MailForestContact and can't be modified.
    + CategoryInfo          : NotSpecified: (contoso...ara Perdichizzi:ADObjectId) [Set-MailContact], TaskInva
   lidOperationException
    + FullyQualifiedErrorId : E298C7BF,Microsoft.Exchange.Management.RecipientTasks.SetMailContact

"Giuseppe Ieraci" is a MailForestContact and can't be modified.
    + CategoryInfo          : NotSpecified: (contoso...Giuseppe Ieraci:ADObjectId) [Set-MailContact], TaskInva
   lidOperationException
    + FullyQualifiedErrorId : 28CFBAA8,Microsoft.Exchange.Management.RecipientTasks.SetMailContact

"Ken Tarbet" is a MailForestContact and can't be modified.
    + CategoryInfo          : NotSpecified: (contoso...acts/Ken Tarbet:ADObjectId) [Set-MailContact], TaskInva
   lidOperationException
+ FullyQualifiedErrorId : 2EE5477,Microsoft.Exchange.Management.RecipientTasks.SetMailContact

image

The AD and Exchange contacts also does not get created.

Other TechNet articles such as the following does not correct the issue:

FIM Troubleshooting: stopped-dll-exception troubleshooter document
https://social.technet.microsoft.com/wiki/contents/articles/8759.fim-troubleshooting-stopped-dll-exception-troubleshooter-document.aspx

Solution

After going through numerous TechNet articles and posts without making any progress, I went ahead and tried changing the Exchange 2010 RPS URI to another Exchange 2010 HT/CAS server:

image

… and the export job immediately worked.  This lead me to change my search query, which was when I found the following blog post that resolved the issue:

http://www.vspbreda.nl/nl/exchange/exchange-2010/exchange-2010-load-quota-1000-requests-exceeded/

What I needed to do was simply perform an iisreset on the problematic server to prevent the export job from erroring out:

image

Wednesday, April 19, 2017

Accelerating Exchange 2016 DAG (Database Availability Group) replication with Riverbed SteelHead

I’ve recently had to work with a Riverbed engineer to determine why the traffic between an Exchange server in Bermuda and London was not being accelerated by the Riverbeds between the two sites.  The following Riverbed  configuration guide was what I used in the past for Exchange 2010 environments:

Optimizing Database Availability Group (DAG) Replication for Microsoft Exchange 2010 (Testing Guide)
https://splash.riverbed.com/docs/DOC-1280

Since the the configuration guide above was last modified on December 1, 2012 and I was working with Exchange 2016, I went ahead to try and find a more updated guide but was unable to find one so I went ahead and used the instructions in the older guide.  The following are the results from my tests.

Begin by reviewing the DAG configuration with the following cmdlet:

Get-DatabaseAvailabilityGroup <dagName> | FL *network*

The following is an example of the output from the cmdlet:

Get-DatabaseAvailabilityGroup 16dag | FL *network*

NetworkCompression : InterSubnetOnly

NetworkEncryption : InterSubnetOnly

ManualDagNetworkConfiguration : False

NetworkNames : {}

image

The two configuration settings we’re interested in are:

  • NetworkCompression
  • NetworkEncrytion

More information about these two settings can be found in the following TechNet article:

Managing database availability groups
https://technet.microsoft.com/en-us/library/dd298065(v=exchg.150).aspx

In order to allow the Riverbed to accelerate the traffic, these two configuration settings need to be disabled.  As a test, I disabled NetworkEncryption first with the following cmdlet:

Set-DatabaseAvailabilityGroup 16dag -NetworkEncryption disabled

The following is the output and configuration settings of the DAG after the change:

Set-DatabaseAvailabilityGroup 16dag -NetworkEncryption disabled

Get-DatabaseAvailabilityGroup 16dag | FL *network*

NetworkCompression : InterSubnetOnly

NetworkEncryption : Disabled

ManualDagNetworkConfiguration : False

NetworkNames : {}

image

With NetworkEncryption disabled, the Riverbed was able to provide approximately 16% reduction of data:

imageimage

After performing the above test, I proceeded to disable NetworkCompression:

Set-DatabaseAvailabilityGroup 16dag -NetworkCompression disabled

Get-DatabaseAvailabilityGroup 16dag | FL *network*

NetworkCompression : Disabled

NetworkEncryption : Disabled

ManualDagNetworkConfiguration : False

NetworkNames : {}

image

With both NetworkCompression and NetworkEncryption disabled, the Riverbed was able to provide approximately 61% reduction of data:

imageimage

Thursday, April 13, 2017

Attempting to enable a user for Exchange UM displays the error message: “Extension xxx is already assigned to another user on dial plan UMDialPlan or on an equivalent dial plan.”

Problem

You attempt to reassign an Exchange UM extension that was previously assigned to a user who you have disabled for UM but receive the following error:

error

Extension xxxx is already assigned to another user on dial plan UMDialPlan or on an equivalent dial plan.

image

You try using the Get-UMMailbox cmdlet to list all of the users and review which one currently has the extension assigned:

Get-UMMailbox | Format-Table -Wrap -AutoSize

… but do not see the extension listed in Extensions column:

image

You review the attributes for the user who was previously assigned the extension but do not see any reference of it in the Exchange attributes:

image

The Lync / Skype for Business msRTCSIP-Line attribute is confirmed not to exist for the user either:

image

Solution

One of the possible causes of this issue is if the user who previously had this extension assigned still has the EUM email address with the extension as the value:

image

image

To correct this issue, remove the email address from the previous user.

Monday, April 10, 2017

Attempting to move an archive mailbox from one mailbox database to another in Exchange 2016 takes a long time

Problem

You’re attempting to move an archive mailbox of a user from one mailbox database to another in Exchange 2016 but notice that Status in the EAC appears to be stuck in Syncing for a long time and does not complete:

image

The statistics of the migration request indicates data is either being moved very slowly or not moving at all because the Last synced time can be hours before the current time:

Statistics

Created by: a-tluk@domain.com

Create time: 4/10/2017 8:51:52 AM

Start time: 4/10/2017 8:51:52 AM

Initial sync time:

Initial sync duration:

Last synced time: 4/10/2017 10:24:29 AM

image

Opening the details of the migration request via the View details link shows the migration rate listed as 0 bytes

mjoell@domain.com

Status: Syncing

mjoell@domain.com

Skipped item details

Data migrated: 1.376 GB ‎(1,477,282,535 bytes)‎

Migration rate: 0 B ‎(0 bytes)‎

Error:

Report: mjoell@domain.com Download the report for this user

Last successful sync date: 4/10/2017 10:41:38 AM

Status:

Queued duration: 00:00:09.5392418

In-progress duration: 00:12:19.0602398

Synced duration: 00:00:00

Stalled duration: 01:32:35.5891119

image

Opening the properties of the move request and navigating to the migration reports section does not show any reports available:

image

Solution

There could be various reasons why a migration request could appear to be forever syncing and not completing but for the example above, the reason why the move is taking a long time can be found by using the following cmdlets to obtain more information.

Begin by verifying that the cmdlet is displaying the same results as the EAC GUI:

Get-MigrationBatch -identity "Move Marchelle's archive to EMAR02" -includereport

image

Continue by listing the detailed statistics of the move requests in the queue:

Get-MoveRequest | Get-MoveRequestStatistics

image

Notice that the StatusDetail column display StalledDueToTarget_Mdb… in the screenshot above.

To view the full description of the StatusDetail column, execute the following:

Get-MoveRequestStatistics -Identity "Marchelle Joell" | Format-Wide -Property StatusDetail

image

Notice that the StatusDetail is shown as StalledDueToTarget_MdbReplication.

Reviewing the following TechNet article:

---------------------------------------------------------------------------------------------------------------------------------------------------------

Exchange 2016 Migration Status
https://social.technet.microsoft.com/wiki/contents/articles/36516.exchange-2016-migration-status.aspx

Explains the Stalledduetotarget_mdbreplication status as:

Stalledduetotarget_mdbreplication:
This value is also returned from Data Guarantee API on checking the replication health of the target database copies if they are a member of DAG and have database copies.
We might get this message if the MRS service is waiting to get this information from the target server about the replication status of the database copies.

So, in this case, the passive copy must be:
1) Healthy.
2) Must have a replay queue with 10 mins of replay lag time.
3) Have a copy queue length less than 10 logs.
4) Have an average copy queue length less than 10 logs.

---------------------------------------------------------------------------------------------------------------------------------------------------------

In the example above, the reason why the migration is taking so long is because there is a delay with the replication between the servers in the DAG hosting the target mailbox database.  This can be determined by reviewing the passive copy’s Copy queue length:

image

image

I’ve noticed that the environment I was working in would have this value jump between 0 to 50, which basically indicates there’s a slight replication delay between the two servers in the DAG.  The archive mailbox move is indeed being moved but at a very slow pace.  This can be verified by executing:

Get-MoveRequest | Get-MoveRequestStatistics

… or:

Get-MoveRequestStatistics -Identity "Marchelle Joell"

… and confirming that the value for PercentComplete is indeed increasing:

image

The only options you’ll have if the above is your scenario is to either correct the replication delay or simply wait for a longer duration to move the archive mailbox due to the problem.

Wednesday, April 5, 2017

Unable to expand Exchange 2010 public folders from an Exchange 2016 hosted mailbox with Outlook 2016

Problem

You’ve used the following TechNet article to allow Exchange 2016 mailboxes to access your Exchange 2010 public folders during a migration:

Configure legacy public folders where user mailboxes are on Exchange 2013 servers
https://technet.microsoft.com/en-us/library/dn690134(v=exchg.150).aspx

You’ve confirmed that Outlook 2010 and 2013 clients are able to expand the Exchange 2010 hosted public folders but the following error messages are displayed when attempting to expand with Outlook 2016:

image

Cannot expand the folder. The set of folders cannot be opened. Network problems are preventing connection to Microsoft Exchange.

image

The set of folders cannot be opened. Network problems are preventing connection to Microsoft Exchange.

image

***Note that it is not accurate to test via Exchange 2016 OWA because it would not be able to proxy requests over to Exchange 2010 so you are guaranteed to receive the following error if you try opening the public folders:

Can’t complete your request

No public folders are available. Check that a public folders deployment exists in this Exchange organization. If so, please try again later

image

Solution

This issue had me stumped for half a day because attempting to search this error message with the combintation of Exchange 2010, 2016 and Outlook 2016 returns many other KBs and forum posts that are unrelated to the issue.  After a bit more searching with different strings, I managed to find the solution in the following KB:

Outlook can't access public folders hosted on legacy Exchange servers
https://support.microsoft.com/en-us/help/3177600/outlook-can-t-access-public-folders-hosted-on-legacy-exchange-servers

Executing the following cmdlet on an Exchange 2016 hosted mailbox displays the DefaultPublicFolderMailbox field as being blank:

Get-Mailbox tluk | FL *public*

image

Use the following cmdlet as described in the KB to define the proxy mailbox that was created for Exchange 2016 users to access the Exchange 2010 public folders:

Set-Mailbox <User> -DefaultPublicFolderMailbox <ProxyMailbox>

image

Execute the following cmdlet to assign the public folder database for the user's mailbox database:

Set-MailboxDatabase (Get-Mailbox <User>).Database -PublicFolderDatabase (Get-MailboxDatabase (Get- Mailbox <ProxyMailbox>).Database).PublicFolderDatabase

image

Complete the above steps by executing the following cmdlet to restart the Microsoft Exchange Rpc Client Access service on the server that's running Exchange Server that hosts the public folders:

Restart-Service MSExchangeRPC

With the above completed, the Outlook 2016 client should now be able to expand the public folder structure:

image

Monday, April 3, 2017

Unable to expand Exchange 2010 public folders from an Exchange 2016 hosted mailbox with Outlook 2013

Problem

You’ve used the following TechNet article to allow Exchange 2016 mailboxes to access your Exchange 2010 public folders during a migration:

Configure legacy public folders where user mailboxes are on Exchange 2013 servers
https://technet.microsoft.com/en-us/library/dn690134(v=exchg.150).aspx

You’ve confirmed that the configuration in the article has been completed but receive the following message when you attempt to expand an Exchange 2010 hosted public folder with Outlook 2013:

Cannot expand the folder. Microsoft Exchange is not available. Either there are network problems or the Exchange server is down for maintenance.

image

You’ve also used the following KB article to configure Outlook Anywhere to use NTLM as the authentication method:

Users of Exchange Server 2013 or later or Exchange Online can't open public folders or shared mailboxes on a legacy Exchange server
https://support.microsoft.com/en-us/help/2834139/users-of-exchange-server-2013-or-later-or-exchange-online-can-t-open-public-folders-or-shared-mailboxes-on-a-legacy-exchange-server

Solution

One of the reasons why the error message above would be displayed is if one or more of the RPC directory on the Exchange CAS servers have Negotiate listed above NTLM as an Enabled Provider.  To check, log onto each CAS server and launch the Internet Information Services (IIS) Manager, expand the Default Web Site, select the RPC directory and click on Authentication:

image

Select Windows Authentication and click on the Providers… link under Actions:

image

Notice that Negotiate could be listed at the top of the list in the Enabled Providers section even if you’ve configured NTLM as the ClientAuthenticationMethod or IISAuthenticationMethods:

image

Change this by selecting NTLM in the list and clicking on the Move Up button:

image

Perform an IISReset and this would correct the issue allowing you to expand the public folder hosted on an Exchange 2010 server in an Outlook 2013 client.

Friday, March 31, 2017

Attempting to add the CAS role to an Exchange 2010 mailbox server with SP3 Rollup 13 throws the error: “The installed product does not match the installation source(s)…”

Problem

You need to install the CAS (Client Access Server) role onto an existing Exchange 2010 server with SP3 Rollup 13 that has the mailbox role already installed.  You’ve downloaded Exchange 2010 SP3, unpacked it, run setup.exe:

image

… select the CAS role to be installed but receive the following message during the install:

Update Rollup 13 for Exchange Server 2010 Service Pack 3

The installed product does not match the installation source(s). Until a matching source is provided or the installed product and source are synchronized, this action can not be performed.

image

Solution

The solution to this is actually quite simple and that is to click on the Browse button and manually select the exchangeserver.msi file in the unpacked Exchange 2010 SP3 folder:

image

Manually selecting this file will allow the install to proceed:

image

It is important to reapply the rollup update to the server once the install is complete.  In the example above, the version listed via the following cmdlet is SP3 RU13:

Get-Command ExSetup | ForEach {$_.FileVersionInfo}

Exchange Server Updates: build numbers and release dates
https://technet.microsoft.com/en-us/library/hh135098(v=exchg.150).aspx

image