Azure AD synchronization fails with: "user_realm_discovery_failed: User realm discovery failed" and "The remote server returned an error: (407) Proxy Authentication Required."

Problem

You’ve noticed that your Azure AD synchronization has stopped synchronizing for a period of time:

Launching the Synchronization Service Manager indicates the export job is failing with stopped-extension-dll-exception:

Reviewing the event logs show the following three events consistently logged:

• Event ID: 6900
• Event ID: 659
• Event ID: 906

Log Name: Application
Source: ADSync

Event ID: 6900

Level: Error

The server encountered an unexpected error while processing a password change notification:

"user_realm_discovery_failed: User realm discovery failed

at InitializeAndGetTargetExtension(Object lockObject, TargetTaskScheduler taskScheduler, Dictionary`2 targetExtensions, ECMAInformation* ecmaInformation)

at TargetExtensionManager.ExportPasswords(TargetExtensionManager* , ECMAInformation* ecmaInformation, DynamicArray<ActiveDirectoryPasswordChange \*>* targetPasswordChanges, Char* forestInfo)

InnerException=>

The remote server returned an error: (407) Proxy Authentication Required.

at System.Net.HttpWebRequest.GetResponse()

at Microsoft.IdentityModel.Clients.ActiveDirectory.HttpWebRequestWrapper.<GetResponseSyncOrAsync>d__2.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Microsoft.IdentityModel.Clients.ActiveDirectory.UserRealmDiscoveryResponse.<CreateByDiscoveryAsync>d__0.MoveNext()

InnerException=>

none

"

Log Name: Application
Source: Directory Synchronization

Event ID: 659

Level: Error

Error while retrieving password policy sync configuration. Microsoft.IdentityModel.Clients.ActiveDirectory.AdalServiceException: user_realm_discovery_failed: User realm discovery failed ---> System.Net.WebException: The remote server returned an error: (407) Proxy Authentication Required.

at System.Net.HttpWebRequest.GetResponse()

at Microsoft.IdentityModel.Clients.ActiveDirectory.HttpWebRequestWrapper.<GetResponseSyncOrAsync>d__2.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Microsoft.IdentityModel.Clients.ActiveDirectory.UserRealmDiscoveryResponse.<CreateByDiscoveryAsync>d__0.MoveNext()

--- End of inner exception stack trace ---

at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.RunAsyncTask[T](Task`1 task)

at Microsoft.Online.Coexistence.ProvisionHelper.GetADALToken(String userName, String userPassword, MSOInstance adalServiceResource)

at Microsoft.Online.Coexistence.ProvisionHelper.GetSecurityToken(String userName, String userPassword, MSOInstance adalServiceResource)

at Microsoft.Azure.ActiveDirectory.Connector.ProvisioningServiceAdapter.InitializeProvisionHelper()

at Microsoft.Azure.ActiveDirectory.Connector.ProvisioningServiceAdapter.Initialize()

at Microsoft.Azure.ActiveDirectory.Connector.ProvisioningServiceAdapter.GetCompanyConfiguration()

at Microsoft.Azure.ActiveDirectory.Connector.PasswordPolicy.RefreshPasswordPolicySyncSettings()

ErrorCode: user_realm_discovery_failed

StatusCode: 0

Log Name: Application
Source: Directory Synchronization

Event ID: 906

Level: Error

GetADALToken [user_realm_discovery_failed]: unexpected authentication failure. STS endpoint (https://login.windows.net), userName (Sync_DIRSYNC01_d5b89680b957@contoso.onmicrosoft.com), tenantName (contoso.onmicrosoft.com), adalAuthority(https://login.windows.net/contoso.onmicrosoft.com) user_realm_discovery_failed: User realm discovery failed | The remote server returned an error: (407) Proxy Authentication Required..

Solution

One of the possible causes to this error is if the DirSync service is attempting reach Azure via a proxy server and is unable to authenticate. In the case of this example, the DirSync server was able to synchronize directly via the internet but had inadvertently inherited proxy settings due to a network misconfiguration. Changing the proxy settings did not correct the problem as DirSync continued to connect to the proxy. To correct this problem, you’ll need to edit a configuration file for the synchronization service manager to stop it from using a proxy.  Navigate to the following directly:

C:\Windows\Microsoft.NET\Framework64\v4.0.30319\config

.. and locate the machine.config file:

Duplicate the file to create a backup:

Open the file and navigate to the end of the configuration:

Add the following under the </system.web> tag:

<system.net>

<defaultProxy enabled="false"></defaultProxy>

</system.net>

Save the file, restart the Azure synchronization services and rerun the synchronization.

Dirsync export job in Synchronization Service Manager displays “InvalidSoftMatch” in the Export Errors

Problem

You’ve noticed that a newly created user account in your on premise Active Directory is not showing up in your Office 365 Admin center so you review the Operations menu in the Synchronization Service Manager and notice that the export job displays the error InvalidSoftMatch in the Export Errors window pane:

Opening the InvalidSoftMatch entry brings up the following Connector Space Object Properties Pending Export tab with information confirming that this is the missing user account:

Continuing to click on the Export Error tab displays the following information with a Detail button:

Clicking on the Detail button will display the following Error Information:

Unable to update this object because the following attributes associated with this object have values that may already be associated with another object in your local directory services: [ProxyAddresses SMTP:crussell@Contoso.com,smtp:crussell@ContosoReAG.mail.onmicrosoft.com,Mail crussell@Contoso.com;].  Correct or remove the duplicate values in your local directory.  Please refer to http://support.microsoft.com/kb/2647098 for more information on identifying objects with duplicate attribute values.

Tracking Id: 466344fe-a7c5-403e-8b0a-8621752ac178

You attempt to use the following PowerShell cmdlets via the WAAD (Windows Azure Active Directory) console to determine whether there is another account with the same smtp address:

Connect-MsolService

Get-MsolUser -all | Select DisplayName,ProxyAddresses | where-object {$_.ProxyAddresses -like "*crussell*"} | Format-Table -Wrap -Autosize

… but no results are returned.

Going through the steps outlined the in the KB:

Duplicate or invalid attributes prevent directory synchronization in Office 365
https://support.microsoft.com/en-us/kb/2647098

… and using the IdFix DirSync Error Remediation Tool (https://www.microsoft.com/en-ca/download/details.aspx?id=36832) does not list any references to the problematic account.

Solution

After trying all of the above without having any luck, I reread the contents in the following KB:

Duplicate or invalid attributes prevent directory synchronization in Office 365
https://support.microsoft.com/en-us/kb/2647098

… and noticed this:

All alias values in Office 365 must be unique for a given organization. Even if you have multiple unique suffixes after the at sign (@) in the Simple Mail Transfer Protocol (SMTP) address, all alias values must be unique.

Knowing that the user of the user account in question also had a pre-existing contact with an external SMTP email address, I began reviewing the properties of the existing contact in the Admin center:

Proceeded to click on the Edit Exchange settings link:

Which brought me to the Office 365 Exchange console of the contact object and it immediately became obvious that the problem was caused by the Alias of the exist contact (also configured as crussell):

Attempting to change the Alias would fail with:

error

The action ‘Set-MailContact’, ‘Alias,EmailAddresses’, can’t be performed on the object ‘Craig Russell’ because the object is being synchronized from your on-premises organization. This action should be performed on the object in your on-premises organization.

Attempting to delete the mail contact would throw the following error:

error

The action ‘Remove-MailContact’, ‘Identity’, can’t be performed on the object ‘Craig Russell’ because the object is being synchronized from your on-premises organization. This action should be performed on the object in your on-premises organization.

Having no luck with the GUI, I proceeded to review the Remove-MsolContact cmdlet:

https://docs.microsoft.com/en-us/powershell/msonline/v1/remove-msolcontact

Used the following Get-MsolContact cmdlet to export contact objects and their respective properties.  Used the find feature to locate the Craig Russell contact’s ObjectID:

Then proceeded to use the Remove-MsolContact cmdlet to delete the contact from the directory:

With the contact deleted, re-running the export job in the Synchronization Service Manager no longer displayed the InvalidSoftMatch in the Export Errors window pane:

Logging back onto the Office 365 Admin center console now displayed the user object.

AD FS and DirSync services fail to start after server restart

Problem

You’ve successfully installed AD FS and DirSync on their respective Windows Server 2012 R2 servers and have confirmed that both are working as expected. However, you also realize that the services on the AD FS and DirSync servers no fail to start as soon as you restart the server:

DirSync

Service Name: FIMSynchronizationService
Display Name: Forefront Identity Manager Synchronization Service
Service Account: .\AAD_d5b89680b957

Service Name: MSOnlineSyncScheduler
Display Name: Windows Azure Active Directory Sync Service
Service Account: .\AAD_d5b89680b957

AD FS

Service Name: Adfssrv
Display Name: Active Directory Federation Services
Service Account: <nonGeneric>

Windows could not start the Active Directory Federation Services service on the Local Computer.

Error 1069: The service did not start due to a logon failure.

Solution

While there could be various reasons why this issue may occur, one of them is if you have a GPO configured in your domain that specifies what accounts are allowed to have Log on as service rights.  In the environment I worked in, there was such a policy so when I launched the Local Computer Policy editor with gpedit.msc:

… I can see that the options to edit the Log on as a service configuration greyed out:

The reason why the AD FS and DirSync worked initially is because the install manually granted these service accounts the rights but a restart of the server removed them.

Troubleshooting this issue didn’t actually take me too much time but I can see that it could have if I missed this so I hope this will safe some time for anyone who may encounter the same issue.