Improving Migrations Using Data Consistency Scoring

This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.


As you may have seen from the Microsoft 365 roadmap item, the O365 migration team has been working on improvements to the way we detect inconsistencies or data loss during migrations or moves.

We’ve historically used a configurable parameter called the ‘Bad Item Limit’ to define the number of items that you, the admin, are okay with dropping during migrations. We allow the admin to use their discretion to set this bad item limit to as low or as high as they feel comfortable with. On the service side, we can see that many migrations hit a large number of ‘bad items’ which are simply inconsistencies in metadata that the end-user may not even notice. This then resulted in many admins running their migrations with a very high Bad Item Limit by default. The problem is that the current implementation has limited capabilities to alert the admin when there are bad items that the end-user will notice, or significant amounts of truly ‘bad items’.

As our experience around migrations has grown over time, we’ve learned to distinguish between ‘expected’ and ‘unexpected’ inconsistencies and have built functionality to expose this to admins. We call this mechanism Data Consistency Scoring or DCS. Based on the number and type of data inconsistencies we detect, your migration will be categorized as Perfect, Good, Investigate, or Poor.

Migrations that end up in the Investigate bucket would require additional admin approval (self-approval via the UI or cmdlet) for completion. Migrations marked as Poor cannot be completed without escalating to support. By doing this, we are taking the guessing out of the “How many bad items am I OK with?” equation. We never had an official recommendation on what to set your ‘bad items’ limits to, and we are hoping this helps to deal with ambiguity that resulted.

Now that the DCS mechanism is fully rolled out, any new migration/migration batch that is started without a value set for the Bad Item Limit (-BadItemLimit parameter) or Large Item Limit (-LargeItemLimit parameter) will use the new DCS method. The Bad Item Limit mechanism will still be available to use and overrides DCS whenever explicitly specified as we want to allow you time to modify your scripts to work with the new DCS method.

But the long-term goal is to eventually do away with Bad Item Limit and Large Item Limit altogether.

For more details, take a look at the official documentation and guidelines for DCS! Let us know what you think!

O365 Migration Team

REMEMBER: these articles are REPUBLISHED. Your best bet to get a reply is to follow the link at the top of the post to the ORIGINAL post! BUT you're more than welcome to start discussions here:

This site uses Akismet to reduce spam. Learn how your comment data is processed.