ADAMSync + (AD Recycle Bin OR searchFlags) = “FUN”

This post has been republished via RSS; it originally appeared at: Ask the Directory Services Team articles.

First published on TechNet on Jan 09, 2013

Hello again ADAMSyncers! Kim Nichols here again with what promises to be a fun and exciting mystery solving adventure on the joys of ADAMSync and AD Recycle Bin (ADRB) for AD LDS. The goal of this post is two-fold:

1. Explain AD Recycle Bin for AD LDS and how to enable it


2. Highlight an issue that you may experience if you enable AD Recycle Bin for AD LDS and use ADAMSync

I'll start with some background on AD Recycle Bin for AD LDS and then go through a recent mind-boggling scenario from beginning to end to explain why you may not want (or need) to enable AD Recycle Bin if you are planning on using ADAMSync.

Hold on to your hats!

AD Recycle Bin for ADLDS

If you're not familiar with AD Recycle Bin and what it can do for you, check out Ned's prior blog posts or the content available on TechNet.

• Active Directory Recycle Bin in Windows Server 2008 R2


• The AD Recycle Bin: Understanding, Implementing, Best Practices, and Troubleshooting


• Active Directory Recycle Bin Step-by-Step Guide


• 
  
  Advanced AD DS Management Using Active Directory Administrative Center (Level 200)
  
  
  
  
  • 
    
    Lots of new features in Windows Server 2012 AD Administrative Center in regard to AD Recycle Bin
    
    
  
  
  
  

The short version is that AD Recycle Bin is a feature added in Windows Server 2008 R2 that allows Administrators to recover deleted objects without restoring System State backups and performing authoritative restores of those objects.

Requirements for AD Recycle Bin

To enable AD Recycle Bin (ADRB) for AD DS your forest needs to meet some basic requirements:

1. Have extended your schema to Windows Server 2008 R2.


2. Have only Windows Server 2008 R2 DC's in your forest.


3. Raise your domain(s) functional level to Windows Server 2008 R2.


4. Raise your forest's functional level to Windows Server 2008 R2.

What you may not be aware of is that AD LDS has this feature as well. The requirements for implementing ADRB in AD LDS are the same as AD DS although they are not as intuitive for AD LDS instances.

Schema must be Windows Server 2008 R2

If your AD LDS instance was originally built as an ADAM instance, then you may or may not have extended the schema of your instance to Windows Server 2008 R2. If not, upgrading the schema is a necessary first step in order to support ADRB functionality.

To update your AD LDS schema to Windows Server 2008 R2, run the following command from your ADAM installation directory on your AD LDS server:

Ldifde.exe –i –f MS-ADAM-Upgrade-2.ldf –s server:port –b username domain password –j . -$ adamschema.cat

You'll also want to update your configuration partition:

ldifde –i –f ms-ADAM-Upgrade-1.ldf –s server:portnumber –b username domain password –k –j . –c "CN=Configuration,DC=X" #configurationNamingContext

Information on these commands can be found on TechNet:

• Appendix B: Upgrading from ADAM to AD LDS


• 
  
  Requirements for Active Directory Recycle Bin
  
  
  
  

Decommission any Windows Server 2003 ADAM servers in the Replica set

In an AD DS environment, ADRB requires that all domain controllers in the forest be running Windows Server 2008 R2. Translating this to an AD LDS scenario, all servers in your replica set must be running Windows Server 2008 R2. So, if you've been hanging on to those Windows Server 2003 ADAM servers for some reason, now is the time to decommission them.

LaNae 's blog " How to Decommission an ADAM/ADLDS server and Add Additional Servers " explains the process for removing a replica member. The process is pretty straightforward and just involves uninstalling the instance, but you will want to check FSMO role ownership, overall instance health, and application configurations before blindly uninstalling. Now is not the time to discover applications have been hard-coded to point to your Windows Server 2003 server or that you've been unknowingly been having replication issues.

Raise the functional level of the instance

In AD DS, raising the domain and forest functional levels is easy; there's a UI -- AD Domains and Trusts. AD LDS doesn't have this snap-in, though, so it is a little more complicated. There's a good KB article ( 322692 ) that details the process of raising the functional levels of AD and gives us insight into what we need to do raise our AD LDS functional level since we can't use the AD Domains and Trusts MMC.

AD LDS only has the concept of forest functional levels. There is no domain functional level in AD LDS. The forest functional level is controlled by the msDS-Behavior-Version attribute on the CN=Partitions object in the Configuration naming context of your AD LDS instance.

Simply changing the value of msDS-Behavior-Version from 2 to 4 will update the functional level of your instance from Windows Server 2003 to Windows Server 2008 R2. Alternatively, you can use Windows PowerShell to upgrade the functional level of your AD LDS instance. For AD DS, there is a dedicated Windows PowerShell cmdlet for raising the forest functional level called Set-ADForestMode, but this cmdlet is not supported for AD LDS. To use Windows PowerShell to raise the functional level for AD LDS, you will need to use the Set-ADObject cmdlet to specify the new value for the msDS-Behavior-Version attribute.

To raise the AD LDS functional level using Windows PowerShell, run the following command (after loading the AD module):

Set-ADObject -Identity <path to Partitions container in Configuration Partition of instance> -Replace @{'msds-Behavior-Version'=4} -Server <server:port>

For example in my environment, I ran:

Set-ADObject -Identity 'CN=Partitions,CN=Configuration,CN={A1D2D2A9-7521-4068-9ACC-887EDEE90F91}' -Replace @{'msDS-Behavior-Version'=4} -Server 'localhost:50000'

As always, before making changes to your production environment:

1. Test in a TEST or DEV environment


2. Have good back-ups


3. Verify the general health of the environment (check replication, server health, etc)

Now we're ready to enable AD Recycle Bin!

Enabling AD Recycle Bin for AD LDS

For Windows Server 2008 R2, the process for enabling ADRB in AD LDS is nearly identical to that for AD DS. Either Windows PowerShell or LDP can be used to enable the feature. Also, there is no UI for enabling ADRB for AD LDS in Windows Server 2008 R2 or Windows Server 2012. Windows Server 2012 does add the ability to enable ADRB and restore objects through the AD Administrative Center for AD DS (you can read about it here ), but this UI does not work for AD LDS instances on Windows Server 2012.

Once the feature is enabled, it cannot be disabled. So, before you continue, be certain you really want to do this. (Read this whole post to help you decide.)

The ADRB can be enabled in both AD DS and AD LDS using a PowerShell cmdlet, but the syntax is slightly different between the two. The difference is fully documented in TechNet .

In my lab, I used the PowerShell cmdlet to enable the feature rather than using LDP. Below is the syntax for AD LDS:

Enable-ADOptionalFeature 'recycle bin feature' -Scope ForestOrConfigurationSet -Server <server:port> -Target <DN of configuration partition>

Here's the actual cmdlet I used and a screenshot of the output. The cmdlet asks you confirm that you want to enable the feature since this is an irreversible process .

You can verify that the command worked by checking the msDS-EnabledFeature attribute on the Partitions container of the Configuration NC of your instance.

Seemed like a good idea at the time. . .

Now, on to what prompted this post in the first place.

Once ADRB is enabled, there is a change to how deleted objects are handled when they are removed from the directory. Prior to enabling ADRB when an object is deleted, it is moved to the Deleted Objects container within the application partition of your instance (CN=Deleted Objects, DC=instance1, DC=local or whatever the name of your instance is) and most of the attributes are deleted. Without Recycle Bin enabled, a user object in the Deleted Object container looks like this in LDP:

After enabling ADRB, a deleted user object looks like this in LDP:

Notice that after enabling ADRB, givenName , displayName , and several other attributes including userPrincipalName (UPN) are maintained on the object while in the Deleted Objects container. This is great if you ever need to restore this user: most of the data is retained and it's a pretty simple process using LDP or PowerShell to reanimate the object without the need to go through the authoritative restore process. But, retaining the UPN attribute specifically can cause issues if ADAMSync is being used to synchronize objects from AD DS to AD LDS since the userPrincipalName attribute must be unique within an AD LDS instance.

In general, the recommendation when using ADAMSync, is to perform all user management (additions/deletions) on the AD DS side of the sync and let the synchronization process handle the edits in AD LDS. There are times, though, when you may need to remove users in AD LDS in order to resolve synchronization issues and this is where having ADRB enabled will cause problems.

For example:

Let's say that you discover that you have two users with the same userPrincipalName in AD and this is causing issues with ADAMSync: the infamous ATT_OR_VALUE_EXISTS error in the ADAMSync log.

====================================================

Processing Entry: Page 67, Frame 1, Entry 64, Count 1, USN 0 Processing source entry <guid=fe36238b9dd27a45b96304ea820c82d8> Processing in-scope entry fe36238b9dd27a45b96304ea820c82d8.

Adding target object CN=BillyJoeBob,OU=User Accounts,dc=fabrikam,dc=com. Adding attributes: sourceobjectguid, objectClass, sn, description, givenName, instanceType, displayName, department, sAMAccountName, userPrincipalName, Ldap error occurred. ldap_add_sW: Attribute Or Value Exists. Extended Info: 0000217B: AtrErr: DSID-03050758, #1:

0: 0000217B: DSID-03050758, problem 1006 (ATT_OR_VALUE_EXISTS), data 0, Att 90290 (userPrincipalName)

. Ldap error occurred. ldap_add_sW: Attribute Or Value Exists. Extended Info: 0000217B: AtrErr: DSID-03050758, #1:

0: 0000217B: DSID-03050758, problem 1006 (ATT_OR_VALUE_EXISTS), data 0, Att 90290 (userPrincipalName)

===============================================

Upon further inspection of the users, you determine that at some point a copy was made of the user's account in AD and the UPN was not updated. The old account is not needed anymore but was never cleaned up either. To get your ADAMSync working, you:

1. Delete the user account that synced to AD LDS.


2. Delete the extra account in AD (or update the UPN on one of the accounts).


3. Try to sync again

BWAMP!

The sync still fails with the ATT_OR_VALUE_EXISTS error on the same user. This doesn't make sense, right? You deleted the extra user in AD and cleaned up AD LDS by deleting the user account there. There should be no duplicates. The ATT_OR_VALUE_EXISTS error is not an ADAMSync error. ADAMSync is making LDAP calls to the AD LDS instance to create or modify objects. This error is an LDAP error from the AD LDS instance and is telling you already have an object in the directory with that same userPrincipalName . For what it's worth, I've never seen this error logged if the duplicate isn't there. It is there; you just have to find it!

At this point, it's not hard to guess where the duplicate is coming from, since we've already discussed ADRB and the attributes maintained on deletion. The duplicate userPrincipalName is coming from the object we deleted from the AD LDS instance and is located in the Deleted Objects container. The good news is that LDP allows you to browse the container to find the deleted object. If you've never used LDP before to look through the Deleted Objects container, TechNet provides information on how to browse for deleted objects via LDP .

It's great that we know why we are having the problem, but how do we fix it? Now that we're already in this situation, the only way to fix it is to eliminate the duplicate UPN from the object in CN=Deleted Objects. To do this:

1. Restore the deleted object in AD LDS using LDP or PowerShell


2. After the object is restored, modify the UPN to something bogus that will never be used on a real user


3. Delete the object again


4. Run ADAMSync again

Now your sync should complete successfully!

Not so fast, you say . . .

So, I was feeling pretty good about myself on this case. I spent hours figuring out ADRB for AD LDS and setting up the repro in my lab and proving that deleting objects with ADRB enabled could cause ATT_OR_VALUE_EXISTS errors during ADAMSync. I was already patting myself on the back and starting my victory lap when I got an email back from my customer stating the msDS-BehaviorVersion attribute on their AD LDS instance was still set to 2.

Huh?!

I'll admit it, I was totally confused. How could this be? I had LDP output from the customer's AD LDS instance and could see that the userPrincipalName attribute was being maintained on objects in the Deleted Objects container. I knew from my lab that this is not normal behavior when ADRB is disabled. So, what the heck is going on?

I know when I'm beat, so decided to use one of my "life lines" . . . I emailed Linda Taylor . Linda is an Escalation Engineer in the UK Directory Services team and has been working with ADAM and AD LDS much longer than I have. This is where I should include a picture of Linda in a cape because she came to the rescue again!

Apparently, there is more than one way for an attribute to be maintained on deletion. The most obvious was that ADRB had been enabled. The less obvious requires a better understanding of what actually happens when an object is deleted. Transformation into a Tombstone documents this process in more detail. The part that is important to us is:

• 
  
  All attribute values are removed from the object, with the following exceptions:
  
  
  
  
  • nTSecurityDescriptor , attributeID , attributeSyntax , dNReferenceUpdate , dNSHostName , flatName , governsID , groupType , instanceType , lDAPDisplayName , legacyExchangeDN , mS-DS-CreatorSID , mSMQOwnerID , nCName , objectClass , distinguishedName , objectGUID , objectSid , oMSyntax , proxiedObjectName , name , replPropertyMetaData , sAMAccountName , securityIdentifier , sIDHistory , subClassOf , systemFlags , trustPartner , trustDirection , trustType , trustAttributes , userAccountControl , uSNChanged , uSNCreated , whenCreated attribute values are retained.
  
  
  • In AD LDS, the msDS-PortLDAP attribute is also retained.
  
  
  • The attribute that equals the rdnType of the object (for example, cn for a user object) is retained.
  
  
  • Any attribute that has fPRESERVEONDELETE flag set in its searchFlags is retained, except objectCategory and sAMAccountType , which are always removed, regardless of the value of their searchFlags .
  
  
  
  

The Schema Management snap-in doesn't allow us to see attributes on attributes, so to verify the value of searchFlags on the userPrincipalName attribute we need to ADSIEdit or LDP.

WARNING: Modifying the schema can have unintended consequences. Please be certain you really need to do this before proceeding and always test first!

By default, the searchFlags attribute on userPrincipalName should be set to 0x1 (INDEX).

My customer's searchFlags attribute was set to 0x1F (31 decimal) = (INDEX |CONTAINER_INDEX |ANR |PRESERVE_ON_DELETE |COPY).

Apparently these changes to the schema had been made to improve query efficiency when searching on the userPrincipalName attribute.

Reminder: Manually modifying the schema in this way is not something you should doing unless are certain you know what you are doing or have been directed to by Microsoft Support.

The searchFlags attribute is a bitwise attribute containing a number of different options which are outlined here . This attribute can be zero or a combination of one or more of the following values:

To remove the PRESERVE_ON_DELETE flag, we subtracted 8 from customer's value of 31, which gave us a value of 23 (INDEX | CONTAINER | ANR | COPY).

Once we removed the PRESERVE_ON_DELETE flag, we created and deleted a test account to confirm our modifications changed the tombstone behavior of the userPrincipalName attribute. UPN was no longer maintained!

Mystery solved!! I think we all deserve a Scooby Snack now!

Nom nom nom!

Lessons learned

1. ADRB is a great feature for AD. It can even be useful for AD LDS if you aren't synchronizing with AD. If you are synchronizing with AD, then the benefits of ADRB are limited and in the end it can cause you more problems than it solves.


2. Manually modifying the schema can have unintended consequences.


3. PowerShell for AD LDS is not as easy as AD


4. AD Administrative Center is for AD and not AD LDS


5. LDP Rocks!

This wraps up the "More than you really ever wanted to know about ADAMSync, ADRB & searchFlags" Scooby Doo edition of AskDS. Now, go enjoy your Scooby Snacks!

- Kim "That Meddling Kid" Nichols