Your Data Product’s Trust Badge Is Probably Wrong
Data marketplaces often stamp trust labels on data products once and never revisit them. This design proposes evidence-based labels that are refreshed daily.
Nobody chooses their own credit score. A lender calculates it from your repayment history, and it moves when that history changes. Miss a few payments, and it drops. Pay down the debt, and it slowly climbs back. Banks lend against it every day without phoning anyone to check that it still holds.
Trust labels on data products usually work the other way round. Someone picks a tier when the product is onboarded, and nothing ever recalculates it. Two years on, a product can still be marked Certified after its owner has moved teams, its quality checks were switched off during a migration, and its contract has drifted away from what the pipeline delivers. The label was right when it was set, and nothing has checked it since.
This article describes a design that treats a data product’s trust tier more like a credit score. The organisation calculates the tier from evidence it already holds, recalculates it nightly or weekly, and uses it to decide what each product can be used for. People only get involved when a decision needs a person.
What a trust tier is
A trust tier is a label on a data product that tells users how much they can rely on it and what they may use it for. A product that is fine for an analyst’s exploratory work may not be fit to feed a regulatory return. The tier is how the marketplace distinguishes those two cases.
The model here has three tiers. Each has its own criteria, and the tiers stack: to hold a tier, a product must meet that tier’s criteria and those of every tier below it.
- Governed. The product has an owner, known sources and consumers, a support route, access and privacy controls, and a complete data catalog entry. Fit for exploration, prototyping and internal reporting that nothing critical depends on.
- Reliable. It also has an approved data contract, measured data quality, and freshness targets it meets. Fit for internal analytics, cross-team work, and feeding downstream pipelines and applications.
- Certified. It is also monitored in operation, traceable to its sources, protected against breaking changes and described in a semantic model. Fit for regulatory, financial, high-risk and external use.
A product that misses any Governed criterion holds no tier, and the marketplace shows it as Not qualified.
The tier appears next to the product’s lifecycle state (Active, Suspended, Deprecated, or Retired) but remains separate from it. The lifecycle state says whether the product is available. A product can be Certified and Deprecated at the same time.
Why a static tier fails
A tier applied by hand at onboarding is a judgement made when we know the least about the product. It has no track record yet, so there is no history of quality scores or on-time deliveries to review, and whoever picks the tier mostly relies on the owner’s description. Owners naturally want the highest tier they can get, and nothing checks their answers against the catalog, the quality platform or the access controls, so two similar products can end up on different tiers depending on who filled in the form. After that, the tier stays put while the product changes. Owners leave, source systems are replaced, quality rules lapse and deliveries start arriving late, and nobody can say how far the label has drifted.
Because nothing re-checks it, the first sign of a problem usually comes from a consumer who has already been affected, through a wrong figure in a board pack, a feed that missed its deadline, or personal data seen by people who should not have seen it.
A static tier also cannot explain itself. Ask why a product is Certified and the usual answer is that someone set it that way. No evidence is on record, so each audit starts from nothing.
The obvious fix is a reviewer who signs off on every tier change, and the first draft of this design included one. It traded one problem for two. Products queued for sign-off, and the same evidence could get a different answer depending on who reviewed it.
The 13 criteria behind the tiers
Each criterion is written as a check against a system that already holds the facts, so most of them ask nothing of the owner except keeping those systems up to date. The descriptions below closely match the rules the engine applies.
Governed: criteria 1 to 6
1. Accountable owner. The owner field names one person, not a team or a shared mailbox, and that person is a current employee.
2. Identified systems of record. The product names every source system it draws on, and each is onboarded and approved in the catalog.
3. Known consumers and use cases. At least one consuming team or application is registered, along with its use case.
4. Support route. Record a support channel and expected response time, and have the owner confirm someone monitors the channel.
5. Access, classification and privacy. Set a confidentiality class, do not make anything open to all staff, flag personal data, and record a masking policy wherever personal data is present.
6. Catalog and marketplace registration. The catalog entry is published rather than draft, with an ID, owner, description, classification and access route.
Reliable: criteria 7 to 9
7. Approved data contract. The contract is under version control, passes schema checks, and the current owner has approved the current version.
8. Data quality checks defined and measured. Every critical data element in the contract has at least one quality rule, and the latest scores are visible.
9. Latency and freshness targets. The contract includes targets, monitoring shows they were met over a recent window, and a breach alert is enabled.
Certified: criteria 10 to 13
10. Operational quality monitoring. Quality checks run on a schedule, their history is kept, and a failed run opens an incident for the owner.
11. Glossary alignment and lineage. Every critical data element links to an approved business glossary term, and lineage traces each system of record through to the product.
12. Change management and compatibility. A change and deprecation policy is linked, and a compatibility check must pass before merging any contract change.
13. Semantic model. Publish a semantic model or certified metric set, link it to the product, and ensure at least one BI tool queries it.
Eleven of the thirteen are checked entirely by machine. For the support route (4) and change management (12), the owner adds a short attestation in the catalog next to the automated check, and renews it only when the channel or the policy changes.
Three decisions that shaped the design
Evidence over assertion
Ownership is checked against the data catalog and Employee Information, access against the identity and access management (IAM) system, quality against the data quality platform, and the contract against the contract repository. Owners no longer fill in forms that repeat what those systems already record, and nobody has to take an owner’s word over evidence available from an authoritative source.
Get Rotimi Ademola’s stories in your inbox
Join Medium for free to get updates from this writer.
The criteria live in a versioned policy file that the evaluation engine runs. Changing a rule means releasing a new version of the file, with a transition period before it applies to existing products.
People only by exception
Nobody approves runs, promotions or demotions. A person steps in only when an owner asks to accept a known risk (an exception) or believes a check got it wrong (a dispute). Any other gap is ordinary data work: the owner fixes it at the source, and the next run picks it up.
A component inside the marketplace
The proposed certification capability would be built as a component of the Data Marketplace instead of a standalone service, because nothing outside the marketplace needs to call it. The marketplace’s pages would show the tier, and its access-request workflow would check it by calling the component in-process, so no separate service would need to be deployed or run. The daily sweep would run as its own scheduled job, so it does not slow marketplace requests. Other systems, including the data catalog, would learn about tier changes through events the component publishes.
How it works
An assessment starts in one of four ways:
- Onboarding. The data catalog raises an event when a product is registered.
- Publication. The marketplace raises an event when a product, or a new version of one, is published.
- The daily sweep. Overnight, the system assesses every onboarded product that has not been retired.
- On demand. An owner can request a run, and the contract’s CI pipeline can request a dry run before merging a change.
Every run then does the same work. It collects fresh evidence for all 13 criteria and updates each criterion’s status. It works out the tier, applying any demotion that has fallen due or any promotion the evidence now supports. It sends the owner a gap report and publishes the result.
Only the evidence collectors talk to source systems through one adapter per source, so a new evidence source means a new adapter and no change to the engine. The evaluation engine turns check results into criterion statuses and statuses into a tier. The evidence store is append-only and keeps every check result, every certificate and a snapshot of the evidence behind each decision. Anything that needs the tier reads it through the component’s interface or its events, never from the store directly.
A newly onboarded product usually ends its first run as Not qualified, because its catalog entry is still a draft and fails criterion 6. The gap report tells the owner what to finish. Publication starts its own run, so a listing carries a tier from day one. Until that run finishes, it shows “Assessment pending”, and certification never holds up publication. The sweep reads the full product list from the catalog rather than relying on events, so a lost event delays the first assessment by at most a day.
Tiers that move both ways
A product holds the highest tier where every mandatory criterion for that tier and every tier below is passing, at risk, or covered by an approved exception.
Promotion happens straight away. The first run that finds all of a higher tier’s criteria passing moves the product up, with no queue, and a product can climb more than one tier in a single run. Checks measured over a trailing window, such as freshness (9) and operational quality monitoring (10), need a track record before they pass.
Demotion is slower on purpose. When a criterion starts failing, it is marked at risk and a grace period starts. The tier stays, the badge shows it is at risk, and the owner is told what to fix. If the criterion is still failing when the grace period ends, the run demotes the product. When a collector cannot reach its source, the last result stands for up to three days before the criterion is marked at risk, so a platform outage alone never demotes anything.
A demotion can skip tiers. If a Certified product's owner leaves without a replacement, it fails criterion 1 and, when the grace period runs out, drops to Not qualified.
Every tier change writes a certificate naming the run and its evidence snapshot, so anyone can audit the change without signing it. Reaching Reliable also sets a certify-by date, and a product that passes that date without reaching Certified is flagged as overdue.
Using the tier to control access
The tier also decides who gets access. When someone requests access, the marketplace checks the use they declare against the product’s tier. Internal analytics on a Governed product gets a warning the requester must acknowledge. Regulatory, financial and external use of anything below Certified is refused.
The daily sweep also finds existing consumers whose use has ended up above a product’s tier after a demotion, and flags them to the owner and to governance with a date to sort it out. Enforcement can start in warn-only mode and move to blocking once owners have had time to adjust.
Where people still fit
People still make the decisions that need judgement. The rule is that nobody decides anything about their own product.
Exceptions. Sometimes a gap cannot be closed in time, and the business still needs the product. The owner raises an exception recording the criterion, the reason, a compensating control, a remediation plan and an expiry date. A named risk acceptor, who cannot be the product’s owner, approves or declines it. Only criteria 7 to 13 can be excepted. Criteria 1 to 6 cannot, because a product without an owner, known sources or access controls is not yet a data product.
Every active exception appears on the badge, and consumers registered for regulatory, financial or high-risk use are told when one is raised. The owner and the risk acceptor are reminded 30 days before it expires. At expiry, the criterion returns to its live result, which continues to be checked, and a failure starts the normal grace period.
Disputes. Sometimes the owner believes a check result is wrong. The governance lead decides within five working days, and the grace period is paused while the dispute is open. An upheld dispute fixes the check or its collector, so every product gets the correction.
The governance lead owns the policy file, grace periods, and time windows, but cannot set a tier by hand. Every tier change comes from a run.
Rolling it out in five phases
Each phase is useful on its own. The first phase still relies on owners for evidence that no system collects yet, which is the same weakness as the hand-set tier described earlier. The difference is that owners record the facts behind each criterion, the engine calculates the tier from them, and every run rechecks them. Automation then replaces those owner-recorded facts with checks against source systems, one tier at a time, until the owner’s word is needed only for the two attestations.
- Go live. Onboarding and publication events and the daily sweep start runs, and the tier is written to the catalog. Checks that are not yet automated read evidence the owner records in the catalog.
- Automate Governed (criteria 1 to 6), with collectors for the catalog, Employee Information, IAM and the marketplace. Grace periods and demotion switch on here.
- Automate Reliable (criteria 7 to 9), with collectors for the contract repository, the data quality platform and observability tools. Every Reliable product now gets a certify-by date.
- Automate Certified (criteria 10 to 13) and open the exception register. After this, only criteria 4 and 12 need an owner attestation.
- Enforce the usage rules in access requests, warn only at first, then block.
To see whether it is working, watch the share of evidence collected without manual input and the median time from Reliable to Certified. The most telling number is new high-risk access granted on products below Certified, which should be zero.
Before go-live, the grace period, the time allowed between Reliable and Certified, and who may accept an exception’s risk each need a named policy owner.
What the first nightly run shows
The first sweep is the most informative day of the project. It assesses every product in the marketplace against its evidence and shows how many of today’s hand-set tiers that evidence supports.
The number may be lower than people expect. It is still worth having, because for the first time it is a measurement rather than an assumption.
Lenders trust a credit score because nobody can award it to themselves, and it changes when the facts do. A trust tier can earn the same confidence. If your marketplace still sets a tier at onboarding and leaves it there, start with the triggers and the checks you can already automate, then let the first nightly run show you where you stand.
