Upgrading Active Directory forest level throws the warning: “The functional level could not be raised. The error is: The directory service is busy.”

I ran into an interesting error that I’ve never seen before through the many times I’ve upgrade an Active Directory’s forest functional level where I would receive a:

The functional level could not be raised. The error is: The directory service is busy.

What’s interesting was that I had no problems upgrading the domain functional level prior to upgrading the forest functional level but of all the times I’ve had problems modifying schemas or objects in Active Directory, it almost always was due to communication problems between domain controllers so the first thing I did was use repadmin to see if any replication errors were logged:

C:\Program Files (x86)\Support Tools>repadmin /showrepl

repadmin running command /showrepl against server localhost

Default-First-Site-Name\DC1
DC Options: IS_GC
Site Options: (none)
DC object GUID: 1ca423a8-872b-4c84-be97-790149f515db
DC invocationID: 1ca423a8-872b-4c84-be97-790149f515db

==== INBOUND NEIGHBORS ======================================

DC=someDomain,DC=local
    Default-First-Site-Name\DC2 via RPC
        DC object GUID: 5fb9ecdd-a78e-4067-8299-0ce9fef9c076
        Last attempt @ 2011-04-19 22:27:58 was successful.

CN=Configuration,DC=someDomain,DC=local
    Default-First-Site-Name\DC2 via RPC
        DC object GUID: 5fb9ecdd-a78e-4067-8299-0ce9fef9c076
        Last attempt @ 2011-04-19 22:14:38 failed, result 1908 (0x774):
            Could not find the domain controller for this domain.
        1 consecutive failure(s).
        Last success @ 2011-04-19 21:53:49.

CN=Schema,CN=Configuration,DC=someDomain,DC=local
    Default-First-Site-Name\DC2 via RPC
        DC object GUID: 5fb9ecdd-a78e-4067-8299-0ce9fef9c076
        Last attempt @ 2011-04-19 22:16:02 failed, result 1908 (0x774):
            Could not find the domain controller for this domain.
        1 consecutive failure(s).
        Last success @ 2011-04-19 21:53:49.

DC=DomainDnsZones,DC=someDomain,DC=local
    Default-First-Site-Name\DC2 via RPC
        DC object GUID: 5fb9ecdd-a78e-4067-8299-0ce9fef9c076
        Last attempt @ 2011-04-19 22:18:26 failed, result 1908 (0x774):
            Could not find the domain controller for this domain.
        1 consecutive failure(s).
        Last success @ 2011-04-19 21:53:49.

DC=ForestDnsZones,DC=someDomain,DC=local
    Default-First-Site-Name\DC2 via RPC
        DC object GUID: 5fb9ecdd-a78e-4067-8299-0ce9fef9c076
        Last attempt @ 2011-04-19 22:30:10 was successful.

Source: Default-First-Site-Name\DC2
******* 1 CONSECUTIVE FAILURES since 2011-04-19 21:53:49
Last error: 1908 (0x774):
            Could not find the domain controller for this domain.

C:\Program Files (x86)\Support Tools>

After seeing that there were errors, I opened Replication Monitor and tried to force a replication to see if an error was logged and if there was, what the informational message was:

As shown in the above screenshot in Replication Monitor, the error message:

Replication Failure: The reason is: Could not find the domain controller for this domain.

… appears to suggest that DC1 is unable to contact DC2.  After plowing through the logs on the domain controllers, I ended up fixing a few things in DNS and the domain controller’s DNS settings to get replication working again.

Once the replication issues were gone, I was then able to upgrade the forest functional level.

Since replication problems can have many causes, if you experience the same error message as I did, I’d recommend checking Active Directory replication between your domain controllers with tools such as:

1. Replication Monitor
2. DCDiag
3. NetDiag

Hope this helps.

Microsoft Lync Server 2010 Enterprise Pool Local SQL Express (RTCLOCAL) Database Permissions

One of the new features of Microsoft Lync Server 2010 is that enterprise pool deployments adds a local instance of SQL Server Express to each front-end server and edge servers named RTCLOCAL so they become less reliant on being able to access the Central Management Store data on the backend SQL server.  Through this replica model, the front end servers would still be able to provide Lync services to clients even if the backend SQL server is unavailable.  As some of you may already know, I took some time to document the permissions of the backend databases a few months back and wrote the following post:

Microsoft Lync Server 2010 Enterprise Pool SQL Database Permissions

http://terenceluk.blogspot.com/2011/01/microsoft-lync-server-2010-enterprise_31.html

… and a few weeks ago, a reader posted in the comments section asking me if I had the permissions for the RTCLOCAL instance documented and while I did document it, I never got the chance to write a blog post about it due to the lack of time.  Now that I’ve got some time, I thought I’d write this post in case anyone out there is looking for this information.

To access the local instance of SQL Express on the front-end servers, simply open up SQL Server Management Studio from a server that has it installed and connect to: yourFrontEndServer\RTCLOCAL. 

The databases that the local RTCLOCAL instance has are as follows:

1. rtc
2. rtcdyn
3. xds

RTC Database

RTC Component Local Group

RTC Local Administrators

RTC Local Read-only Administrators

RTC Server Local Group

RTCDYN Database

RTC Component Local Group

RTC Local Administrators

RTC Server Local Group

XDS Database

RTC Component Local Group

RTC Local Administrators

RTC Local Config Replicator

RTC Local Read-only Administrators

RTC Server Local Group

RTCLOCAL Security Principals

MSSQL$RTCLOCAL

RTC Component Local Group

RTC Local Administrators

RTC Local Config Replicator

RTC Local Read-only Administrators

RTC Server Local Group

Assuming that nothing disastrous has happened to your Lync Server 2010 front-end servers, you won’t ever need to modify these settings but if the situation does happen and you need to reconcile permission settings for the your servers, I hope this post will help.