Prologue: Between Law and Chaos

Hello and thanks for reading! Today I want to discuss Data Quality (DQ) and its maturity. How often do you feel that one not_null test for a 50-column-wide table is not enough? Do you think that's okay? Can it be acceptable at one stage but require more maturity later? And if a Data Governance (DG) team faces resistance, what should they do?

One more question. How would you visualize a DG team if I ask you? Would it be Themis, a goddess of justice with her scales in perfect equilibrium? But she is inactive even if worshippers ignore her decrees. Or would it be Nemesis, a goddess of righteous vengeance — who reminds us that mercy without accountability courts chaos?

Usually, in the Data area, Data Governance begins as Themis but must sometimes invoke Nemesis to protect the sanctity of our digital world. Let's take a look at levels of DQ maturity and then move to contracts, data providing patterns and sanctions when they are necessary.

Special thanks to Theo Gough for his inspiration, thoughtful discussions, and the groundwork he laid in defining these maturity levels.

Themis's Scales

I want to suggest three levels of Data Quality maturity (it can be a starting point for your vision) for three table tiers (low, medium and high, e.g. low tier might contain staging data and high tier includes core business metrics consumed in dashboards). As Themis weighted each soul's deeds, this model assesses a data asset and applies different rules to it.

Required

This is a baseline: all data assets which bring any value have to follow such rules. These rules set guardrails for where data control can begin. This level is applicable for medium and high tier tables and ideal for low.

  • Completeness: Table has: name, description, data‑owner
  • Validity: Data access restrictions align with DG team defined restrictions
  • Integrity: Mandatory DQ tests: is_unique for unique columns; not_null for mandatory ones
  • Timeliness: Freshness interval for ingested tables

Reasonable

Once the scales balance, Themis demands more nuance — lineage, glossary links, and column‑level stewardship — so teams can navigate by the stars, not just the horizon. Also, this level aligns with data maturity level, when a medium tier table becomes a high one. This level is applicable for high tier tables and ideal for medium and low.

Builds on Required, plus:

  • Completeness: Table has lineage from predecessor. Columns have column descriptions, owners and glossary ties.
  • Integrity: Mandatory DQ tests: is_unique for Primary Key (PK) (include composite PK)

Recommended

Imagine how Themis orders from Hephaestus automated tests for dates, numbers, orphaned records; lineage back to the first spark; and multi‑dimensional freshness metrics. And all of these items Hephaestus packs in our Recommended level. And this level is ideal for high and medium table tiers.

Builds on Reasonable, plus:

  • Consistency: Mandatory DQ tests: DATE/NUMBER boundary tests
  • Integrity: Mandatory DQ tests: Foreign Key orphan checks
  • Timeliness: Update‑duration, freshness targets
None
Data Quality maturity levels.

Once you've achieved Recommended or Reasonable maturity, you can formalize expectations via Data Contracts.

Raise the Bar

Data Contracts as Sacred Oaths

Since ancient times people wanted to formalize their relations. Initially it was an oath then a ceremony when one person says an oath surrounded by witnesses. Luckily nowadays we have digital signatures and the Internet. In the Data world people invented Open Data Contract Standard (ODCS) and modern tools for rich visualization of the contracts.

Data Contracts, stewarded by your Governance team, stand as ceremonial oaths. They formalize relations between Data Providers and Data Consumers. If a Data Provider wants to change something in e.g. data structure, they have to notify consumers and in advance create a new version of the contract. Data Contracts stand for:

  • Obligations: Producers must uphold quality; consumers must honor lineage
  • Enforcement: Breaking the covenant triggers Production Incidents, much like Nemesis's swift arrow

When all counterparties honor these contracts, data flow works smoothly and no lightning bolts are needed.

None
Data Contracts tab in Datahub.

At Just Eat Takeaway.com, we use Datahub reflecting our Data Contracts. Additionally, there you can link your assertions from Great Expectations or dbt-tests to the contract and visualize the quality of your data.

None
Assertions from Great Expectations which can be linked to a data contract.

Data Pattern as the Shield

Do you know how to avoid bad data in your reports? Data Professionals can give their stakeholders shields that both defend and repel data issues. Three simple steps and your stakeholders won't receive wrong numbers in their reports.

I'm talking about The Write‑Audit‑Publish (WAP) pattern and it works as that shield:

  1. Write data into a staging or sandbox area
  2. Audit it against your maturity‑level tests
  3. Publish only if all checks pass; otherwise, notify stakeholders
None
Protect your Data Consumers from incorrect data with a WAP pattern.

No flawed data breaches your defenses; any transgression triggers an alert and sanctions if you prescribe them.

When Mercy Fails: Nemesis's Sanctions

Sometimes Themis's gentle balance cannot be a guardrail. In that case the Data Governance team has to define a list of sanctions and strictly apply them. As Nemesis — the avenger — has a set of tools, DG team also have tools to prevent rule violation:

  • Production Incident: Automatically raise an incident when a DQ test fails
  • Production Incident Escalation: Automatically revoke user access to tables that fail their DQ tests, until remediation occurs
  • Access Control: Dynamically adjust permissions based on pipeline health and test outcomes
  • Dynamic Permissions: Withdraw privileges when someone constantly breaks the rules
  • Pull Request Justice: Mandate DG sign‑off on any schema or metadata changes, rejecting Pull Requests that violate standards
  • Pre‑commit Hooks: Enforce specific tools (e.g. dbt‑checkpoint) or custom linting hooks so poor metadata never even hits your CI/CD pipeline

By using these simple measures, Data Governance has options, to stay a blind scale which searches for a balance, or to become the uncompromising hand of Nemesis, or find a good position in between and slide along that gradient between the two poles, depending on the case or environment.

Epilogue: Building Idyll on Earth

In an ideal scenario, data quality tests never fail and the entire organization trusts its Data. But this dream never comes true and we need to build standards, guardrails and a vigilance system. In the end, every piece of work is a collaboration between people and they can act as Themis as well as Nemesis.

Embrace this duality. Let Themis guide your standards, and let Nemesis protect things that were created and maintained with great difficulty. In doing so, you'll ensure that your organization's data heritage endures — in legend and in practice.

Thanks for reading and stay tuned. See you in the next article.

Want to come work with us at Just Eat Takeaway.com? Check out our open roles.