[DE] Catalog as a Code: A DataHub Metadata CI/CD Pipeline Inspired by dbt

“A story about building a CI/CD pipeline where a single GitHub PR automatically syncs both your BigQuery table schema and the descriptions written in a yaml spec file to DataHub. Inspired by dbt’s documentation compile pattern, I adopted a ‘Catalog as a Code’ principle and designed a 4-stage pipeline (Validate → Recipe → Ingest → Publish) that preserves legacy metadata while rolling out incrementally. In an era where the reliability of DataHub MCP and NL2SQL AI agents depends heavily on catalog quality, this is also foundational work for an AI-native data environment.”

⚠️ All datasets, tables, and column names shown in this post are personally created samples and are not related to any actual metrics, business data, or strategic direction of my current employer.

Table of Contents

1. Background: Why We Needed Catalog Automation
2. The Goal: A "Catalog as a Code" Principle
3. Designing the yaml Spec — A 3-State Policy
4. Overall Architecture
5. Stage-by-Stage Implementation
    - ① Validate — yaml schema validation
    - ② Recipe — Dynamic ingestion recipe generation
    - ③ Ingest — BigQuery → DataHub schema sync
    - ④ Publish — Merge-and-deploy yaml descriptions
6. Security & Notification — Cloud Armor & Slack
7. Directory Structure
8. Impact and What's Next

1. Background: Why We Needed Catalog Automation

The company I work for has spent years managing DW metadata — tables, columns, lineage — through DataHub, and with the recent arrival of DataHub MCP, it has become an even more valuable asset because our AI agents can now tap into it effectively. But despite having such a strong data catalog, one thing had always bothered me: every table and column description had to be written manually. In other words, DW deployment and metadata deployment were decoupled, which made the following scenarios very easy to fall into:

• A new table ships but sits in production with no description.
• An existing table’s logic or schema changes, but its description is left as legacy.

On top of that, the pace of decision-making at our organization has accelerated dramatically, and with our NL2SQL Slack bot and company-wide use of DataHub MCP, the case for automation kept getting stronger. So in this post, I want to share how I tackled the problem — a story about building a CI/CD pipeline where a single GitHub PR deploys DW code and DataHub metadata together.

2. The Goal: A “Catalog as a Code” Principle

Coming from a dbt background, the idea I had in mind was simple. I wanted to replicate the dbt documentation compile pattern for our own stack.

When modifying a DW SQL file (.sql), the yaml file in the same directory is modified together.
When the PR is opened, DataHub is updated automatically.

I expected this approach to pay off in three ways:

• Metadata is version-controlled alongside the code.
• Metadata can be reviewed during the code review process.
• Metadata is deployed together with the code, eliminating manual updates.

3. Designing the yaml Spec — A 3-State Policy

I placed each yaml file in the same directory as its SQL model, using the same filename:

dags/sql/
├── fact_events.sql # SQL query
└── fact_events.yml # metadata spec

The yaml file follows this structure:

version: 1
table:
  dataset_id: dataset
  name: fact_events
  description: |
    Table containing user behavior logs
columns:
  - name: date
    description: |
      **[Partition Filter]** Event date
  - name: user_id
    description: |
      **[Clustered by]** User identifier
  - name: session_id # no description key → preserve existing DataHub value

One design decision I spent real time on: how should each possible state of description in the yaml file map to DataHub? I split it into three cases, each with its own behavior.

The reason I allowed case 3 — what I call a “state inconsistency” — was to roll out Catalog as a Code incrementally. Our catalog was fairly mature, but the Bronze Layer still had many tables with missing or legacy specs. If I had enforced a strict “no yaml description → block the PR” policy from day one, our deployment velocity would have tanked for weeks, possibly months. At the same time, adopting Catalog as a Code was urgent in terms of both operational speed and asset management. So I chose the principle of “reflect only what is explicitly written; leave untouched what is not mentioned.” This way, we preserve the descriptions that were manually maintained in DataHub while gradually expanding yaml-based management, starting with new tables.

4. Overall Architecture

When a PR is opened, GitHub Actions runs a 4-stage pipeline.

5. Stage-by-Stage Implementation

① Validate: yaml Schema Validation

The first stage of the pipeline is to verify that the yaml file is well-formed.

Here’s the core of the validation logic:

def _validate_yml(path: Path) -> tuple[dict, list[str]]:
    """Read a single yml file and validate its schema."""
    errs: list[str] = []

    # 1) Is the YAML parseable?
    try:
        doc = yaml.load(path.read_text(encoding="utf-8"))
    except Exception as e:
        return {}, [f"YAML parse failed: {e}"]

    # 2) Is version: 1 declared?
    if doc.get("version") != 1:
        errs.append("`version: 1` not specified")

    # 3) Is there a table block?
    t = doc.get("table")
    if not isinstance(t, dict):
        return doc, errs + ["`table:` block missing"]

    # 4) Are table.dataset_id and table.name present?
    if not t.get("dataset_id"):
        errs.append("`table.dataset_id` is empty")
    if not t.get("name"):
        errs.append("`table.name` is empty")

    # 5) Is the columns block a list?
    cols = doc.get("columns")
    if not isinstance(cols, list):
        errs.append("`columns:` is not a list")

    # 6) Check for duplicate column names
    else:
        unique_names: set[str] = set()
        for i, col in enumerate(cols):
            name = col.get("name")
            if name in unique_names:
                errs.append(f"duplicate column: `{name}`")
            else:
                unique_names.add(name)

    return doc, errs

Tables that pass validation are saved to targets.json and handed off to the next stage:

[
  {
    "yml_path": "dags/sql/fact_events.yml",
    "dataset_id": "dataset-id",
    "table_name": "fact_events"
  }
]

② Recipe: Dynamic Ingestion Recipe Generation

DataHub needs an ingestion recipe to pull BigQuery table schemas (column names, types, and so on). But scanning every table every time is far too slow. So in this stage, I generate a recipe that targets only the tables changed in the PR. (A separate full-ingestion pipeline already runs on a daily schedule, so this targeted approach doesn’t sacrifice coverage.)

The template file (bigquery-recipe.template.yml) looks like this:

source:
  type: bigquery
  config:
    project_ids:
      - bv-edw
    include_table_lineage: false    # disabled for scan speed
    include_usage_statistics: false # disabled for scan speed
    profiling:
      enabled: false                # disabled for scan speed
    table_pattern:
      allow:
        __TABLE_PATTERN_ALLOW__     # placeholder dynamically populated from targets.json

sink:
  type: datahub-rest
  config:
    server: '${DATAHUB_GMS_URL}'
    token: '${DATAHUB_GMS_TOKEN}'

The __TABLE_PATTERN_ALLOW__ placeholder is replaced with the actual table patterns:

    table_pattern:
      allow:
        - '^project-id\\.dataset-id\\.fact_events$'
        - '^project-id\\.dataset-id\\.dim_users$'

③ Ingest: BigQuery → DataHub Schema Sync

Now we take the generated recipe (recipe.yml) and sync BigQuery table schemas into DataHub.

The usual way to run this is datahub ingest -c recipe.yml, but I opted to call the Python API (Pipeline.create) directly instead. Going through the CLI means you may end up installing and loading subcommand dependencies you don't actually need — which in a CI/CD context makes you more exposed to dependency conflicts and extra overhead. By calling the Python API directly, I can load only the modules ingestion actually requires, which I found to be a much better fit for a CI/CD environment.

def cmd_ingest(args: argparse.Namespace) -> int:
    """Read the recipe YAML and run the DataHub ingestion pipeline."""
    from datahub.ingestion.run.pipeline import Pipeline

    pipeline = Pipeline.create(config)
    pipeline.run()
    pipeline.raise_from_status()

④ Publish: yaml description → DataHub

The final stage publishes the descriptions from the yaml files via the DataHub API.

Here’s the core of the publishing logic. One important detail: DataHub splits schema metadata into schemaMetadata (the system-sourced aspect populated by ingestion) and editableSchemaMetadata (the aspect humans edit via the UI or API). I only update the latter. That way, I never overwrite the schema information (types, partitions, etc.) brought in by stage ③ Ingest, and I can manage descriptions safely in isolation.

def cmd_publish(args: argparse.Namespace) -> int:
    """Deploy descriptions from the yml file to DataHub via the API."""

    for target in targets:

        # Read the yml file
        doc = yaml.load((ROOT / target["yml_path"]).read_text())

        # 1) Publish the table description
        if "description" in doc.get("table", {}):
            table_desc = doc["table"]["description"].strip()
            client.emit(MetadataChangeProposalWrapper(
                entityUrn=urn,
                aspect=EditableDatasetPropertiesClass(description=table_desc)
            ))

        # 2) Publish column descriptions
        cols_with_desc = [col for col in all_cols if "description" in col]
        if cols_with_desc:
            # 2.1. Fetch all existing column descriptions from DataHub
            existing = client.get_aspect(entity_urn=urn, aspect_type=EditableSchemaMetadataClass)
            merged = {normalize(f.fieldPath): f for f in existing.editableSchemaFieldInfo}

            # 2.2. Overwrite only columns that have a description key in yaml
            for col in cols_with_desc:
                merged[col["name"]] = EditableSchemaFieldInfoClass(
                    fieldPath=col["name"],
                    description=col["description"].strip()
                )

            # 2.3. Push the merged result back to DataHub
            client.emit(MetadataChangeProposalWrapper(
                entityUrn=urn,
                aspect=EditableSchemaMetadataClass(
                    editableSchemaFieldInfo=list(merged.values()),
                    created=AuditStampClass(
                        time=int(time.time() * 1000),
                        actor="urn:li:corpuser:datahub-ci"  # recorded under the CI/CD account
                    ),
                )
            ))

6. Security & Notification — Cloud Armor & Slack

Cloud Armor

Our DataHub instance is protected by GCP Cloud Armor (WAF) so that it’s only accessible from our corporate VPN. To let GitHub Actions reach DataHub, I configured authentication via a custom header.

def _create_datahub_client():
    ci_key = os.environ.get("DATAHUB_CI_KEY")
    return DataHubGraph(DatahubClientConfig(
        server=server,
        token=token,
        extra_headers={"x-datahub-ci-key": ci_key} if ci_key else None,
    ))

Slack Notification

When a deployment completes, the result is reported back to Slack. I deliberately post this as a reply to the existing PR notification thread so that the context stays together.

• On success: show the list of synced tables.
• On failure: show which stage failed and the detailed error.

7. Directory Structure

airflow/
├── .github/
│   ├── datahub/
│   │   ├── sync.py                          # core sync script
│   │   ├── datasets.yml                     # list of sync target directories
│   │   └── bigquery-recipe.template.yml     # ingestion recipe template
│   └── workflows/
│       └── datahub-metadata-sync.yaml       # GitHub Actions workflow
│
└── dags/sql/
    ├── dw1/                 # ✓ sync target
    │   ├── fact_events.sql
    │   └── fact_events.yml
    ├── dw2/                 # ✓ sync target
    └── dw3/                 # ✓ sync target
    └── other-purposes/      # ✗ not a sync target

One thing I had to be careful about: some SQL files under dags/sql/ are not actually DW queries. To handle that, I manage the list of sync target directories explicitly in datasets.yml. Adding a new directory to the sync scope later is as simple as appending it to that file.

# .github/datahub/datasets.yml
- dw1
- dw2
- dw3

8. Impact and What’s Next

Impact

After rolling out this Catalog as a Code CI/CD pipeline, we started seeing the following benefits:

Get Joshua Kim’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

(1) Catalogs became a “development artifact” rather than an “administrative task.”

Previously, when creating or modifying a table, code deployment and catalog updates were two separate jobs. Now they’re handled in a single PR, which dramatically reduces the human error and state-management overhead that used to come from manual work. More importantly, metadata is now reviewed alongside the code — that shift in the team’s review culture is probably the biggest organizational change.

(2) Vendor lock-in on the catalog tool is significantly reduced.

Because the source of truth for metadata now lives in yaml files in our Git repository, migrating to a different catalog tool down the road would just mean swapping out the GitHub Workflow code. As an organization, we’ve gained meaningful optionality in our tech stack.

(3) Claude Skills can now take over metadata authoring itself, to a much greater extent.

Once we had yaml as a structured input format, I was able to get much more leverage out of a Claude Skill (/datahub-metadata) that scans SQL files, BigQuery MCP, and other sources to auto-draft yaml specs.

---
name: datahub-metadata
description: Analyzes the SQL of a given table and automatically drafts DataHub table and column definitions.
argument-hint: [table names (comma-separated for multiple)]
allowed-tools: Read Write ...
---

# DataHub Metadata Registration

## Arguments
## Output
## Instructions
### Step 1: Collect table information
### Step 2: Check existing DataHub metadata
### Step 3: Generate metadata draft
### Step 4: Present draft and confirm with user
### Step 5: Create yml spec file
### Step 6: Report results

What’s Next

Of course, this pipeline is far from finished. Two main pieces of homework remain.

(1) The yaml file is not yet a complete source of truth.

The current policy is “reflect only what’s explicitly written in yaml; leave what isn’t mentioned alone.” This was an intentional choice to support incremental rollout, but in the long run, the ideal end state is to have every spec managed in yaml and then enforce a strict “no yaml, no deploy” policy. That’s the direction I want this CI/CD pipeline to eventually converge toward.

(2) Advanced metadata beyond descriptions is still out of scope.

This pipeline focuses on table and column descriptions, but DataHub’s real power arguably lies in lineage, ownership, governance tags, and data quality metrics. If we can bring those under declarative yaml management too, we’ll be a step closer to “Catalog as a Code” in its fullest sense. That’s the next item on my list.

Closing Thoughts

For a long time, data catalogs were treated as a kind of accessory — something that made data environments “cleaner.” But in a world where DataHub MCP and NL2SQL have become part of everyday workflows, catalog accuracy is now a direct variable in the quality of AI agent responses. For an AI to give good answers, the catalog that fuels it has to be alive and accurate first.

The principle of “code = documentation” is no longer a nice-to-have for pretty data environments. It’s the fundamentals that a data team needs to master in the AI-native era. This pipeline was a first step in planting those fundamentals inside the organization, and being able to deliver this first chapter of an experience I’d wanted to build with my team is a real source of pride.

Thanks for reading. If you’re working through similar challenges, I’d love to hear from you — feel free to reach out any time.