While performing some tasks at a client’s office for their directory summer maintenance, I ran into a problem I haven’t encountered for quite some time and figured I blog about it this time.
Scenario:
- Windows Server 2003 is being used
- 4 domain controllers in the environment.
- 2 domain controllers will be demoted and retired.
- 2 virtual machines have been staged and will replace the 2 domain controllers to be decommissioned with the same name and IP.
Actions Performed:
1. Demote DC1.
2. Force replication.
3. Verify replication.
4. Run NTDSUtil to ensure DC was cleaned out.
5. Re-IP and rename new virtual machine with proper name.
6. Promote new virtual machine to DC.
7. Force replication, verify replication.
8. Repeat for 2nd DC.
Problem:
We went ahead and started to review the event logs after replacing the 2 old domain controllers and noticed that 2 of the old domain controllers (not the virtual machines) were logging a lot of event ID: 1411. One of the DCs were logging more errors while the other less but both were complaining about 2 GUID that appeared to belong to the 2 removed domain controllers:
------------------------------------------------------
Active Directory failed to construct a mutual authentication service principal name (SPN) for the following domain controller.
Domain controller:
ceb25b3a-7741-4dce-9447-d02f9b0bd526._msdcs.domain.net
The call was denied. Communication with this domain controller might be affected.
Additional Data
Error value:
8589 The DS cannot derive a service principal name (SPN) with which to mutually authenticate the target server because the corresponding server object in the local DS database has no serverReference attribute.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
------------------------------------------------------
As seen in the above screenshots, this doesn’t look good. The following were the troubleshooting steps I did:
1. Open Active Directory Sites & Services to review the NTDS replication objects. Don’t see a reference to these 2 GUIDs.
2. Went into ADSIEdit to look for the repsTo attribute. Don’t see any references to the 2 GUIDs.
3. Forced replication via Replication Monitor. Don’t see any errors or references.
4. Ran DCDiag and NetDiag. No errors.
Everything looked good and based on the following KB: http://support.microsoft.com/kb/938704, it says that KCC will eventually remove these connections so that’s when we decided to wait.
Resolution:
There wasn’t really a resolution as the KB article says, KCC will run again in 24 hours to remove those links and that was what happened. Just so I add a bit of value here for those that may read this, the Event ID you want to wait for that will clear up this error is event ID: 1104.
What I noticed was that this required to be logged 2 or more times before the error referencing that GUID was removed:
In the above screenshot, you see this event @ 11:47:03, then the error gets logged at 11:50:52, then another event ID 1104 gets logged again. Then after a few more hours, I noticed another 1104 being logged.
If I scroll through these events, they are all referencing the same GUID but the output is a bit different:
The Knowledge Consistency Checker (KCC) successfully terminated the following change notifications.
Directory partition:
CN=Configuration,DC=domain,DC=net
Destination network address:
ceb25b3a-7741-4dce-9447-d02f9b0bd526._msdcs.domain.net
Destination domain controller (if available):
CN=NTDS Settings\0ADEL:ceb25b3a-7741-4dce-9447-d02f9b0bd526,CN=DC1016\0ADEL:851e0305-2d6c-4016-89dc-fd0a18882b7b,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=net
This event can occur if either this domain controller or the destination domain controller has been moved to another site.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
The Knowledge Consistency Checker (KCC) successfully terminated the following change notifications.
Directory partition:
DC=domain,DC=net
Destination network address:
ceb25b3a-7741-4dce-9447-d02f9b0bd526._msdcs.domain.net
Destination domain controller (if available):
CN=NTDS Settings\0ADEL:ceb25b3a-7741-4dce-9447-d02f9b0bd526,CN=DC1016\0ADEL:851e0305-2d6c-4016-89dc-fd0a18882b7b,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=net
This event can occur if either this domain controller or the destination domain controller has been moved to another site.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
The Knowledge Consistency Checker (KCC) successfully terminated the following change notifications.
Directory partition:
DC=ForestDnsZones,DC=domain,DC=net
Destination network address:
ceb25b3a-7741-4dce-9447-d02f9b0bd526._msdcs.domain.net
Destination domain controller (if available):
CN=NTDS Settings\0ADEL:ceb25b3a-7741-4dce-9447-d02f9b0bd526,CN=DC1016\0ADEL:851e0305-2d6c-4016-89dc-fd0a18882b7b,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=net
This event can occur if either this domain controller or the destination domain controller has been moved to another site.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
----------------------------------------------------------------------
The errors will eventually go away and it looks like it takes more than 24 hours to do so.