Making Purview Data Quality a Little Less Complicated
Data Quality can become complicated very quickly. Once you start dealing with hundreds of tables and thousands of columns, asking data stewards to manually create & maintain DQ rules isn’t really scalable.
So I wanted to make the process as simple as possible for the people actually configuring the rules.
The approach I took is metadata-driven and centered around Data Products in Microsoft Purview. This article is solely focused on the Fabric Lakehouse tables in mind.
Start with the Data Product
Using Purview APIs, the framework retrieves the assets belonging to a Data Product and identifies the Lakehouse tables and their columns.
That metadata is then used to generate an Excel workbook.
The workbook becomes the interface for the data steward.
Instead of writing rules or dealing with APIs, they simply use predefined dropdowns to choose whether a particular DQ rule should be enabled or not for a column.
For example:
No scripting or API knowledge required.
Existing rules are visible too
The workbook also reflects the current state.
If a rule already exists for a column, the corresponding cell is highlighted. The steward can then choose to remove that rule using the dropdown.
And if they need the rule again later, they simply change the dropdown back.
The framework takes care of creating or removing the rule through the Purview APIs.
This makes the workbook more than an input file , it becomes a simple DQ configuration interface.
Alerts without making the configuration complicated
I also wanted alert configuration to be part of the same experience.
The workbook can have an additional column where the steward can provide the email IDs of users who should receive alerts.
A standard threshold can be used for triggering notifications.
One important consideration here is that alerts are configured at the asset level, not at an individual column/user level.
So if the workbook contains something like:
the framework consolidates those users and configures the alert for the Customer asset with all three recipients.
This is important because the alerting model is common to the asset as a whole.
The steward can therefore configure the recipients alongside the DQ rules without having to understand how the underlying alerting mechanism works.
What about scheduling?
Scheduling is deliberately kept separate from the workbook.
The framework can create and manage the DQ rules, but deciding when and how frequently the data should be scanned depends on the individual data product and the team’s requirements.
That requires a one-time discussion between the data steward/owner and the data team to understand the appropriate schedule and set up recurring scans.
Get sarav s’s stories in your inbox
Join Medium for free to get updates from this writer.
Once that setup is done, the recurring execution can happen without the steward having to manually trigger the process every time.
So the responsibilities remain fairly clear:
Steward: Decide what should be checked and who should be notified.
Data team: Set up the appropriate recurring scan schedule.
Framework: Discover metadata, create/remove rules and configure alerts.
Why Data Products instead of Domains?
The framework can technically work with assets added directly to a domain as well.
But there is a practical problem.
At the domain level, there could be a very large number of assets for a steward to go through. That makes the configuration experience tedious. Whereas data products provide a much more focused boundary.
Instead of asking:
“Which of these hundreds of assets in the domain need DQ?”
we can ask:
“What quality rules should be applied to the assets in this Data Product?”
That’s why I chose Data Products as the configuration boundary for this implementation.
Keeping the changes traceable
There was one more problem to solve.
Excel is great for making the configuration user-friendly, but it isn’t ideal for tracking exactly what changed when the file is committed to a repository.
We wanted the configuration to follow a proper DevOps workflow with version control, PRs and review.
So the workbook is converted into JSON before being stored in the DevOps repository.
This gives us a much cleaner representation of the configuration changes.
The overall flow becomes:
Purview APIs
↓
Data Product → Tables → Columns
↓
Generate Excel workbook
↓
Steward selects rules & alert recipients
↓
Excel → JSON
↓
DevOps Repository
↓
PR & Review
↓
Pipeline triggered after PR completion
↓
Purview APIs create/remove DQ rules & configure alerts
Now we have a simple interface for the user, while still getting proper change tracking behind the scenes.
End goal
The idea wasn’t to build another complicated DQ tool. It was to make the process easier to configure and maintain.
The steward doesn’t need to know how the rules are created, how the APIs work, or how the configuration gets deployed.
They just need to decide: Should this rule apply to this column?
And, if required: Who should be notified?
Everything else is automated.
The principle is quite simple:
Use Purview to discover the data.
Use the workbook to capture human decisions.
Use DevOps to track those decisions.
Use APIs and pipelines to automate the execution.
That gives us a Data Quality framework that is user-friendly, maintainable and traceable, without trying to automate the decisions that actually require business context.
