Blog 7: Data Quality as Code — Expectations, Deequ & DLT

Tags: data-quality great-expectations deequ dlt

Why Data Quality Must Be Code

Manual DQ checks rot. Dashboards go stale. Alerts get ignored. The only sustainable model is DQ as code: versioned, tested, and enforced in CI/CD.

DQ Enforcement Flow

The Three Frameworks Tested

Great Expectations (GCP / Azure)

import great_expectations as ge
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = spark.read.format("bigquery").option("table", "poc.silver.clean_orders").load().toPandas()
gdf = ge.from_pandas(df)
gdf.expect_column_values_to_not_be_null("order_id")
gdf.expect_column_values_to_be_between("amount", min_value=0)
gdf.expect_column_values_to_be_unique("order_id")
result = gdf.validate()
assert result["success"], "DQ checks failed"

Pros: Rich expectation library, cross-platform, HTML reports
Cons: Slow on large data (pandas conversion), verbose config

Deequ (AWS)

from pydeequ.checks import Check, CheckLevel
from pydeequ.verification import VerificationSuite
from pyspark.sql import SparkSession
spark = SparkSession.builder \
    .config("spark.jars.packages", "com.amazon.deequ:deequ:2.0.4-spark-3.3") \
    .getOrCreate()
df = spark.read.format("iceberg").load("glue_catalog.poc_silver.clean_orders")
result = VerificationSuite(spark).onData(df).addCheck(
    Check(spark, CheckLevel.Error, "orders")
      .isComplete("order_id")
      .isUnique("order_id")
      .isNonNegative("amount")
).run()
assert result.status == "Success"

Pros: Spark-native, scales to billions of rows, constraint suggestions
Cons: Spark-only, less active community

Get CuriousTechie’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

DLT Expectations (Databricks)

@dlt.table(name="silver_orders")
@dlt.expect_or_drop("valid_order_id", "order_id IS NOT NULL")
@dlt.expect_or_drop("positive_amount", "amount > 0")
@dlt.expect("recent", "order_ts >= current_timestamp() - INTERVAL 30 DAYS")
def silver_orders():
    return dlt.read_stream("bronze_orders").dropDuplicates(["order_id"])

Pros: Zero-config, integrated with pipeline, auto-quarantine
Cons: Databricks-only

DQ Enforcement in CI/CD

The Four DQ Dimensions

1. Completeness — no unexpected nulls
2. Uniqueness — business keys unique
3. Validity — values in expected range
4. Freshness — data is recent (SLA check)

Recommendation

• Databricks-native? Use DLT expectations — the simplest path
• Spark-heavy on AWS? Use Deequ — scales best
• Multi-cloud or Python-first? Use Great Expectations — most portable

What’s Next

Blog 8 goes deep on governance — RBAC, lineage, and data marketplaces across vendors.