A recent replication attempt failed: The replication generated an error (8453): Replication access was denied.
A true story!
The Background
To start off, all names have been changed for security reasons. We had two new read-only domain controllers (RDC3 & RDC4). These domain controllers (Windows Server 2016) seemed to have successfully replicated the database. It became apparent that there was an issue with them as some properties and group memberships did not replicate. There have been a few complaints about users not being able to log on to applications or open network shares.
We worked with the networks team to make sure all ports are allowed for AD Services. Turned out that some ports were blocked during the initial promotion which caused replication errors. We got the networks team to allow the below ports to the appropriate writable domain controllers that were previously blocked.
These ports I have listed below.
We thought our issues were resolved but seem like it was only getting started. The domain controllers were still not reflecting group memberships of users that was created recently.
Our next step was to demote the domain controllers, clean up the metadata and then re-promote them. A few days after the promotion we started noticing errors again.
The Errors
Starting test: Replications
[Replications Check, RDC3] A recent replication attempt
failed:
From DC2 to RDC3
Naming Context: DC=Domain,DC=local
The replication generated an error (8453):
Replication access was denied.
The failure occurred at 2021-06-11 10:27:35.
The last success occurred at (never).
863 failures have occurred since the last success.
The machine account for the destination RDC3.
is not configured properly.
Check the userAccountControl field.
Kerberos Error.
The machine account is not present, or does not match on the.
destination, source or KDC servers.
Verify domain partition of KDC is in sync with rest of enterprise.
The tool repadmin/syncall can be used for this purpose
C:\Windows\system32>repadmin /replicate RDC4 DC1 "DC=Root,DC=Domain,DC=local"
DsReplicaSync() failed with status 8440 (0x20f8):
The naming context specified for this replication operation is invalid.
C:\Windows\system32>repadmin /replsummary
Replication Summary Start Time: 2021-07-01 08:56:27
Beginning data collection for replication summary, this may take a while:........................
Destination DSA largest delta fails/total% error
BDC4 11m:27s 0 / 15 0
DC4 04m:58s 0 / 10 0
RDC1 07m:26s 0 / 5 0
RDC2 03m:09s 0 / 10 0
RDC3 04m:00s 10 / 50 20 (8453) Replication access was denied.
RDC4 09m:44s 10 / 50 20 (8453) Replication access was denied.
C:\Windows\system32>DCDIAG /TEST:KccEvent
Doing primary tests
Testing server: Site\RDC4
Starting test: KccEvent
An error event occurred. EventID: 0xC000043C
Time Generated: 07/01/2021 08:59:03
Event String:
Internal event: Active Directory Domain Services could not update the following object with changes received from the following source directory service. This is because an error occurred during the application of the changes to Active Directory Domain Services on the directory service.
The Investigation
Enough is enough let’s get Microsoft to see if they can assist us.
After some log checking, we found Event ID 1084 with some valuable information.
Internal event: Active Directory Domain Services could not update the following object with changes received from the following source directory service. This is because an error occurred during the application of the changes to Active Directory Domain Services on the directory service.
From the below screenshot of the error, it’s worth noting that the krbtgt accounts are in the deleted objects container.
We used ldp.exe to check the deleted object folder and confirm that both krbtgt accounts are in the deleted objects container in AD. Doh!
LDP.exe - Performs operations such as connect, bind, search, modify, add, delete against any Lightweight Directory Access Protocol (LDAP)-compatible directory, such as Active Directory Domain Services (AD DS). Ldp is an LDAP client that you use to view objects that are stored in AD DS along with their metadata, such as security descriptors and replication metadata.
The KRBTGT is a local default account that acts as a service account for the Key Distribution Center (KDC) service. It's created automatically when a new domain is created. It cannot be deleted. its name cannot be changed. it cannot be enabled.
The Root cause
As luck would have it, when we demoted the RDCs, the krbtgt accounts were moved to the deleted objects container in AD. When we promoted the RDCs again we used the same hostnames.
Recommended by LinkedIn
Had we changed the names; this might not have happened. But because the names were the same, the domain controllers, were still trying to refer to its krbtgt accounts which were inside the deleted objects container.
The below command will confirm the accounts used by each of the Read-only Domain controllers in your domain and the status thereof.
get-adobject -server PDCname -filter { (objectclass -eq "user") -and ( name -like "krbtgt*") } -properties * | FT Name, msDS-KrbTgtLinkBl
It showed the working RODCs in our domain. However, the two affected ones did not show.
The Recommendation
As recycle Bin was not enabled in the environment, and the krbtgt account has a security gotcha that we could not revert, the only way is to demote, change the names and repromote the DCs.
The Fix
1. RODC Removal and Reinstallation step by step
2. Don’t use “retain metadata” when demoting
3. After the demotion is finished, make sure the metadata is cleaned up:
· Active directory users and computers
· Active sites and services
· DNS (Name server, A records & SRV records)
4. Then reimage the machine, promote it as the new RODC with a different name.
The Test
After the promotion, you could use the following command to verify the krbtgt for the RODCs:
get-adobject -server PDCname -filter { (objectclass -eq "user") -and ( name -like "krbtgt*") } -properties * | FT Name, msDS-KrbTgtLinkBl
References;
https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/cc771022(v=ws.11)
Thank you. This helped us as well. Kudos on the write-up!
Thank you for sharing the details write up, It's very informative and life saver 👍