This post has been republished via RSS; it originally appeared at: Ask the Directory Services Team articles.
First published on TechNet on Sep 15, 2014Introducing the Lingering Object Liquidator
Hi all, Justin Turner here ---it's been a while since my last update . The goal of this post is to discuss what causes lingering objects and show you how to download, and then use the new GUI-based Lingering Object Liquidator (LOL) tool to remove them. This is a beta version of the tool, and it is currently not yet optimized for use in large Active Directory environments.
This is a long article with lots of background and screen shots, so plug-in or connect to a fast connection when viewing the full entry. The bottom of this post contains a link to my AD replication troubleshooting TechNet lab for those that want to get their hands dirty with the joy that comes with finding and fixing AD replication errors. I’ve also updated the post with a link to my Lingering Objects hands-on lab from TechEd Europe.
Overview of Lingering Objects
Lingering objects are objects in AD than have been created, replicated, deleted, and then garbage collected on at least the DC that originated the deletion but still exist as live objects on one or more DCs in the same forest. Lingering object removal has traditionally required lengthy cleanup sessions using tools like LDP or repadmin /removelingeringobjects. The removal story improved significantly with the release of repldiag.exe . We now have another tool for our tool belt: Lingering Object Liquidator. There are related topics such as “lingering links” which will not be covered in this post.
Lingering Objects Drilldown
The dominant causes of lingering objects are
1. Long-term replication failures
While knowledge of creates and modifies are persisted in Active Directory forever, replication partners must inbound replicate knowledge of deleted objects within a rolling Tombstone Lifetime (TSL) # of days (default 60 or 180 days depending on what OS version created your AD forest). For this reason, it is important to keep your DCs online and replicating all partitions between all partners within a rolling TSL # of days. Tools like REPADMIN /SHOWREPL * /CSV, REPADMIN /REPLSUM and AD Replication Status should be used to continually identify and resolve replication errors in your AD forest.
2. Time jumps
System time jump more than TSL # of days in the past or future can cause deleted objects to be prematurely garbage collected before all DCs have inbound replicated knowledge of all deletes. The protection against this is to ensure that :
- your forest root PDC is continually configured with a reference time source (including following FSMO transfers
- All other DCs in the forest are configured to use NT5DS hierarchy
- Time rollback and roll-forward protection has been enabled via the maxnegphasecorrection and maxposphasecorrection registry settings or their policy-based equivalents.
The importance of configuring safeguards can't be stressed enough. Look at this post to see what happens when time gets out of whack.
3. USN Rollbacks
USN rollbacks are caused when the contents of an Active Directory database move back in time via an unsupported restore. Root causes for USN Rollbacks include:
- Manually copying previous version of the database into place when the DC is offline
- P2V conversions in multi-domain forests
- Snapshot restores of physical and especially virtual DCs. For virtual environments, both the virtual host environment AND the underlying guest DCs should be Virtual Machine Generation ID capable . Windows Server 2012 or later. Both Microsoft and VMWARE make VM-Generation ID aware Hyper-V host.
Events, errors and symptoms that indicate you have lingering objects
Active Directory logs an array of events and replication status codes when lingering objects are detected. It is important to note that while errors appear on the destination DC, it is the source DC being replicated from that contains the lingering object that is blocking replication. A summary of events and replication status codes is listed in the table below:
| Event or Error status | Event or error text | Implication |
| AD Replication status 8606 | "Insufficient attributes were given to create an object. This object may not exist because it may have been deleted." | Lingering objects are present on the source DC (destination DC is operating in Strict Replication Consistency mode) |
| AD Replication status 8614 | The directory service cannot replicate with this server because the time since the last replication with this server has exceeded the tombstone lifetime. | Lingering objects likely exist in the environment |
| AD Replication status 8240 | There is no such object on the server | Lingering object may exist on the source DC |
| Directory Service event ID 1988 | Active Directory Domain Services Replication encountered the existence of objects in the following partition that have been deleted from the local domain controllers (DCs) Active Directory Domain Services database. |
Lingering objects exist on the source DC specified in the event
(Destination DC is running with Strict Replication Consistency) |
| Directory Service event ID 1388 | This destination system received an update for an object that should have been present locally but was not. |
Lingering objects were reanimated on the DC logging the event
Destination DC is running with Loose Replication Consistency |
| Directory Service event ID 2042 | It has been too long since this server last replicated with the named source server. | Lingering object may exist on the source DC |
A comparison of Tools to remove Lingering Objects
The table below compares the Lingering Object Liquidator with currently available tools that can remove lingering objects
| Removal method | Object / Partition & and Removal Capabilities | Details |
| Lingering Object Liquidator |
Per-object and per-partition removal
Leverages:
|
|
| Repldiag /removelingeringobjects |
Per-partition removal
Leverages:
|
|
| LDAP RemoveLingeringObjects rootDSE primative (most commonly executed using LDP.EXE or an LDIFDE import script) | Per-object removal |
|
| Repadmin /removelingeringobjects |
Per-partition removal
Leverages:
|
|
The Repldiag and Lingering Object Liquidator tools are preferred for lingering object removal because of their ease of use and holistic approach to lingering object removal.
Why you should care about lingering object removal
Widely known as the gift that keeps on giving, it is important to remove lingering objects for the following reasons
- Lingering objects can result in a long term divergence for objects and attributes residing on different DCs in your Active Directory forest
- The presence of lingering objects prevents the replication of newer creates, deletes and modifications to destination DCs configured to use strict replication consistency. These un-replicated changes may apply to objects or attributes on users, computers, groups, group membership or ACLS.
- Objects intentionally deleted by admins or application continue to exist as live objects on DCs that have yet to inbound replicate knowledge of the deletes.
Once present, lingering objects rarely go away until you implement a comprehensive removal solution. Lingering objects are the unwanted houseguests in AD that you just can't get rid of.
Mother in law jokes… a timeless classic.
We commonly find these little buggers to be the root cause of an array of symptom ranging from logon failures to Exchange, Lync and AD DS service outages. Some outages are resolved after some lengthy troubleshooting only to find the issue return weeks later.
The remainder of this post, we will give you everything needed to eradicate lingering objects from your environment using the Lingering Object Liquidator.
Repldiag.exe is another tool that will automate lingering object removal. It is good for most environments, but it does not provide an interface to see the objects, clean up RODCs (yet) or remove abandoned objects.
Introducing Lingering Object Liquidator
More:
|
Lingering Object Liquidator automates the discovery and removal of lingering objects by using the
DRSReplicaVerifyObjects
method used by repadmin /removelingeringobjects and repldiag combined with the
removeLingeringObject
rootDSE primitive used by LDP.EXE. Tool features include:
|
How to obtain Lingering Object Liquidator
Updated October 9th, 2017 with link to new released version.
Download LoL from this link: http://aka.ms/MSFTLoL Read about this newer released version here: http://aka.ms/LingeringObjectLiquidator
If the download does not work, try opening the link from an InPrivate browser tab. If it still does not work, follow these steps:
1. Log on to the Microsoft Connect site (using the Sign in ) link with a Microsoft account:
Note: You may have to create a profile on the site if you have never participated in Connect.
2. Open the Non-feedback Product Directory:
https://connect.microsoft.com/directory/non-feedback
3. Join the following program:
AD Health
Product Azure Active Directory Connection Join link
4. Click the
Downloads
link to see a list of downloads or this
link
to go directly to the Lingering Objects Liquidator download. (Note: the direct link may become invalid as the tool gets updated.)
Updated 8/01/2016 with link to latest version of the tool
5. Download all associated files
6. Double click on the downloaded executable to open the tool.
Tool Requirements
1. Install Lingering Object Liquidator on a DC or member computer in the forest you want to remove lingering objects from.
2. .NET 4.5 must be installed on the computer that is executing the tool.
3. Permissions: The user account running the tool must have Domain Admin credentials for each domain in the forest that the executing computer resides in. Members of the Enterprise Admins group have domain admin credentials in all domains within a forest by default. Domain Admin credentials are sufficient in a single domain or single domain forest.
4. The admin workstation must have connectivity over the same port and protocol required of a domain-joined member computer or domain controller against any DC in the forest. Protocols of interest include DNS, Kerberos, RPC, LDAP and ephemeral port range used by the targeted DC See TechNet for more detail. Of specific concern: Pre-W2K8 DCs communicate over the “low” ephemeral port between 1024 and 5000 while post W2K3 DCs use the “high” ephemeral port range between 49152 to 65535. Environments containing both OS version families will need to enable connectivity over both port ranges.
5. You must enable the Remote Event Log Management (RPC) firewall rule on any DC that needs scanning. Otherwise, the tool displays a window stating, "Exception: The RPC server is unavailable"
6. The liquidation of lingering objects in AD Lightweight Directory Services (AD LDS / ADAM) environments is not supported.
7. You cannot use the tool to cleanup lingering objects on DCs running Windows Server 2003. The tool leverages the event subscriptions feature which wasn’t added until Windows Server 2008.
Walkthrough
Lingering Object Detection:
Run the tool as Domain Administrator (Enterprise Administrator if you want to scan the entire forest) Error 8453 is observed if the tool is not run elevated.
1. Launch LoL.exe .
2. From the
Topology Detection
section, select
Fast
.
Fast detection populates the Naming Context, Reference DC and Target DC lists by querying the local DC. Thorough detection does a more exhaustive search of all DCs and leverages DC Locator and DSBind calls. Note that Thorough detection will likely fail if one or more DCs are unreachable.
3. Take a quick walk through the UI:
Naming Context:
Reference DC: the DC you will compare to the target DC. The reference DC hosts a writeable copy of the partition.
Note: ChildDC2 should not be listed here since it is an RODC, and RODCs are not valid reference DCs for lingering object removal.
More:
|
The version of the tool is still in development and does not represent the finished product. In other words, expect crashes, quirks and everything else normally encountered with beta software. |
Target DC: the DC that lingering objects are to be removed from
4. Click Detect to use these DCs for the comparison. If you want to scan all partitions and all DCs: Leave all fields blank to have the entire environment scanned, and then click Detect.
The tool does a comparison amongst all DCs for all partitions in a pairwise fashion when all fields are left blank. In a large environment, this comparison will take a great deal of time (possibly even days) as the operation targets (n * (n-1)) number of DCs in the forest for all locally held partitions. For shorter, targeted operations, select a naming context, reference DC and target DC. The reference DC must hold a writable copy of the selected naming context. Note that clicking Stop does not actually stop the server-side API, it merely stops the work in the client-side tool.
During the scan, several buttons are disabled. The current count of lingering objects is displayed in the status bar at the bottom of the screen along with the current tool status. During this execution phase, the tool runs in an advisory mode and reads the event log data reported on each target DC.
Note: The Directory Service event log may completely fill up if the environment contains large numbers of lingering objects and the Directory Services event log is using its default maximum log size. The tool leverages the same lingering object discovery method as repadmin and repldiag, logging one event per lingering object found.
When the scan is complete, the status bar updates, buttons are re-enabled and total count of lingering objects is displayed. The Result pane at the bottom of the window updates with any errors encountered during the scan.
If you see error 1396 or Error 8440 in the status pane, you are using an early beta-preview version of the tool and should use the latest version.
Error 1396 is logged if the tool incorrectly used an RODC as a reference DC.
Error 8440 is logged when the targeted reference DC doesn't host a writable copy of the partition.
Note:
|
Lingering Object Liquidator discovery method
|
The tool leverages the Advisory Mode method exposed by DRSReplicaVerifyObjects that both repadmin /removelingeringobjects /Advisory_Mode and repldiag /removelingeringobjects /advisorymode use. In addition to the normal Advisory Mode related events logged on each DC, it displays each of the lingering objects within the main content pane.
Results of the scan are logged in the Results pane. Many more details of all operations are logged in the linger<Date-TimeStamp>.log.txt file in the same directory as the tool's executable.
The Export button allows you to export a list of all lingering objects listed in the main pane into a CSV file. View the file in Excel, modify if necessary and use the Import button later to view the objects without having to do a new scan. The Import feature is also useful if you discover abandoned objects (not discoverable with DRSReplicaVerifyObjects) that you need to remove. We briefly discuss abandoned objects later in this post.
A note about transient lingering objects:
Garbage collection is an independent process which runs on each DC every 12 hours by default. One of its jobs is to remove objects that have been deleted and have existed as a tombstone for greater than the tombstone lifetime number of days. There is a rolling 12-hour period where an object eligible for garbage collection exists on some DCs but has already been removed by the garbage collection process on other DCs. These objects will also be reported as lingering object by the tool, however no action is required as they will automatically get removed the next time the garbage collector process runs on the DC.
Removal of individual objects
The tool allows you to remove objects a handful at a time, if desired, using the Remove button:
5. To remove individual objects, you can select a single object or multi-select multiple objects using the Ctrl or SHIFT keys. (hold down the Ctrl key to select multiple objects, or the SHIFT key to select a range of objects) and then select Remove .
The status bar updates with the new count of lingering objects and the status of the removal operation:
Logging for removed objects
The tool dumps a list of attributes for each object before removal and logs this along with the results of the object removal in the removedLingeringObjects.log.txt log file. This log file is in the same location as the tool's executable.
C:\tools\LingeringObjects\removedLingeringObjects<DATE-TIMEStamp.log.txt
the obj DN: <GUID=0bb376aa1c82a348997e5187ff012f4a>;<SID=010500000000000515000000609701d7b0ce8f6a3e529d669f040000>;CN=Dick Schenk,OU=R&D,DC=root,DC=contoso,DC=com
objectClass:top, person, organizationalPerson, user;
sn:Schenk ;
whenCreated:20121126224220.0Z;
name:Dick Schenk;
objectSid:S-1-5-21-3607205728-1787809456-1721586238-1183;primaryGroupID:513;
sAMAccountType:805306368;
uSNChanged:32958;
objectCategory:<GUID=11ba1167b1b0af429187547c7d089c61>;CN=Person,CN=Schema,CN=Configuration,DC=root,DC=contoso,DC=com;
whenChanged:20121126224322.0Z;
cn:Dick Schenk;
uSNCreated:32958;
l:Boulder;
distinguishedName:<GUID=0bb376aa1c82a348997e5187ff012f4a>;<SID=010500000000000515000000609701d7b0ce8f6a3e529d669f040000>;CN=Dick Schenk,OU=R&D,DC=root,DC=contoso,DC=com;
displayName:Dick Schenk ;
st:Colorado;
dSCorePropagationData:16010101000000.0Z;
userPrincipalName:Dick@root.contoso.com;
givenName:Dick;
instanceType:0;
sAMAccountName:Dick;
userAccountControl:650;
objectGUID:aa76b30b-821c-48a3-997e-5187ff012f4a;
value is :<GUID=70ff33ce-2f41-4bf4-b7ca-7fa71d4ca13e>:<GUID=aa76b30b-821c-48a3-997e-5187ff012f4a>
Lingering Obj CN=Dick Schenk,OU=R&D,DC=root,DC=contoso,DC=com is removed from the directory, mod response result code = Success
----------------------------------------------
RemoveLingeringObject returned Success
Removal of all objects
The Remove All button, removes all lingering objects from all DCs in the environment.
To remove all lingering objects from the environment:
1. Click the Remove All button. The status bar updates with the count of lingering objects removed. (the count may differ to the discovered amount due to a bug in the tool-this is a display issue only and the objects are actually removed)
2. Close the tool and reopen it so that the main content pane clears.
3. Click the Detect button and verify no lingering objects are found.
Abandoned object removal using the new tool
None of the currently available lingering object removal tools will identify a special sub-class of lingering objects referred to internally as, "Abandoned objects".
An abandoned object is an object created on one DC that never got replicated to other DCs hosting a writable copy of the NC but does get replicated to DCs/GCs hosting a read-only copy of the NC. The originating DC goes offline prior to replicating the originating write to other DCs that contain a writable copy of the partition.
The lingering object liquidator tool does not currently discover abandoned objects automatically so a manual method is required.
1. Identify abandoned objects based on Oabvalidate and replication metadata output.
Abandoned objects can be removed with the LDAP RemoveLingeringObject rootDSE modify procedure, and so Lingering Objects Liquidator is able to remove these objects.
2. Build a CSV file for import into the tool. Once, they are visible in the tool, simply click the Remove button to get rid of them.
a. To create a Lingering Objects Liquidator tool importable CSV file:
Collect the data in a comma separated value (CSV) with the following data:
| FQDN of RWDC | CNAME of RWDC | FQDN of DC to remove object from | DN of the object | Object GUID of the object | DN of the object's partition |
3. Once you have the file, open the Lingering Objects tool and select the Import button, browse to the file and choose Open.
4. Select all objects and then choose Remove .
Review replication metadata to verify the objects were removed.
Resources
For those that want even more detail on lingering object troubleshooting, check out the following:
- TechNet - Fixing Replication Lingering Object Problems (Event IDs 1388, 1988, 2042)
- KB 910205 - Information about lingering objects in a Windows Server Active Directory forest
- KB 2028495 - Troubleshooting AD Replication error 8606: "Insufficient attributes were given to create an object"
- Glenn LeCheminant’s weblog: Clean that Active Directory forest of lingering objects
To prevent lingering objects:
- Actively monitor for AD replication failures using a tool like the AD Replication Status tool.
- Resolve AD replication errors within tombstone lifetime number of days.
- Ensure your DCs are operating in Strict Replication Consistency mode
- Protect against large jumps in system time
-
Use only supported methods or procedures to restore DCs. Do not:
- Restore backups older than TSL
- Perform snapshot restores on pre Windows Server 2012 virtualized DCs on any virtualization platform
- Perform snapshot restores on a Windows Server 2012 or later virtualized DC on a virtualization host that doesn't support VMGenerationID
If you want hands-on practice troubleshooting AD replication errors, check out my lab on TechNet Virtual labs . Alternatively, come to an instructor-led lab at TechEd Europe 2014. "EM-IL307 Troubleshooting Active Directory Replication Errors"
For hands-on practice troubleshooting AD lingering objects : check out my lab from TechEd Europe 2014. " EM-IL400 Troubleshooting Active Directory Lingering Objects "
12/8/2015 Update: This lab is now available from TechNet Virtual labs here .
Finally, if you would like access to a hands-on lab for in-depth lingering object troubleshooting; let us know in the comments.
Thank you,
Justin Turner and A. Conner
Update 2014/11/20 – Added link to TechEd Lingering objects hands-on lab
Update 2014/12/17 – Added text to indicate the lack of support in LOL for cleanup of Windows Server 2003 DCs
Update 2015/12/08 – Added link to new location of Lingering Object hands-on
lab
Update 2016/08/01 – Updated LoL download
link
Update 2017/10/09 – Added download link for released version of the tool on the Microsoft Download Center
http://aka.ms/msftLoL