<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Joachim Hodana on Medium]]></title>
        <description><![CDATA[Stories by Joachim Hodana on Medium]]></description>
        <link>https://medium.com/@joachimhodana?source=rss-27db4b2773fb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*WaEIFb06ReeYPkye0X8uOQ.jpeg</url>
            <title>Stories by Joachim Hodana on Medium</title>
            <link>https://medium.com/@joachimhodana?source=rss-27db4b2773fb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 14:02:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@joachimhodana/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[You Don’t Need Big Data to Make Data-Driven Decisions]]></title>
            <link>https://blog.dataengineerthings.org/you-dont-need-big-data-to-make-data-driven-decisions-86f3c0f80e88?source=rss-27db4b2773fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/86f3c0f80e88</guid>
            <category><![CDATA[dagster]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[clickhouse]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Joachim Hodana]]></dc:creator>
            <pubDate>Wed, 29 Jul 2026 00:00:31 GMT</pubDate>
            <atom:updated>2026-08-17T15:54:06.291Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Most companies think a data platform is for enterprises with terabytes and dedicated data teams. It is not.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BdB2TB7SSkpmEQm5_n_vIg.png" /></figure><p>There is a story about Ronald Read, a janitor and gas station attendant from Vermont who died in 2014 and left behind an $8 million estate. He built it by investing small amounts consistently over decades. No hedge fund. No inheritance. No financial team. Just discipline and a system.</p><p>The reaction when people hear this story is usually the same: that is the exception, not the rule. You need capital to invest seriously. You need scale to make it worth the effort.</p><p>The same assumption exists in data. Companies think they need terabytes of data, a dedicated data team and an enterprise warehouse contract before a data platform makes sense. So they wait. They run on spreadsheets and gut feeling until they are big enough to justify the investment. By then, they have made years of decisions without the evidence that would have made those decisions better.</p><p>The assumption is wrong in investing. It is equally wrong in data.</p><h3>What a data platform actually is</h3><p>Strip away the marketing and a data platform is a system that collects information from multiple sources, puts it in one place and makes it queryable. That is it.</p><p>The enterprise version of this involves petabytes of clickstream data, dozens of engineers and a warehouse bill that could fund a small startup. That version exists because enterprises have petabytes of clickstream data and dozens of engineers to manage it.</p><p>A small company has different data. Not less valuable data. Different data.</p><p>A three-person consultancy has contract revenue, project costs, team time, content performance across LinkedIn and Medium, newsletter subscribers and website traffic from Google Search Console. None of these sources individually tells the full story.</p><p>A client that looks profitable on revenue alone looks different when project costs and time are in the same query. A LinkedIn post that gets strong engagement looks different when you can see whether it actually drove traffic to the article it referenced.</p><p>The question is not whether the data is big enough. The question is whether having it in one place would change any decisions.</p><h3>What we built and what it costs</h3><p>Northgrain is a three-person data engineering consultancy. We built our own data platform because the alternative, five disconnected tabs and manual reconciliation every time a business question came up, was costing more in time than the platform costs to run.</p><p>The stack:</p><ul><li>Dagster for orchestration</li><li>dltHub for ingestion</li><li>self-hosted ClickHouse for storage</li><li>dbt for transformation</li></ul><p>The full platform runs on a VPS costing about $20 per month.</p><p>The sources include Google Search Console, Google Analytics 4, LinkedIn analytics, Medium stats, Resend newsletter data, client contracts, project expenses and team costs.</p><p>The reason for each stack choice was practical. Dagster’s asset-based model maps naturally onto keeping specific datasets fresh rather than treating every workflow as an isolated scheduled job. dltHub is Python-native and integrates cleanly without a separate managed service. ClickHouse on a modest VPS handles our data volumes without becoming the constraint and keeps sensitive business data under our control.</p><p>None of these are exotic choices. They are the same tools we recommend to clients, running on real operational data. Every architectural decision gets tested before it gets recommended.</p><h3>What the data shows that spreadsheets did not</h3><p>The clearest example is the relationship between LinkedIn and Medium.</p><p>LinkedIn posts that reference an article drive a measurable traffic spike on Medium within 24 to 48 hours of posting. The inverse is not true. Medium articles do not drive LinkedIn engagement in any consistent way.</p><p>Without the data connected, this relationship is invisible. With it, the content strategy becomes clear: LinkedIn is the distribution layer, Medium is the depth layer. Posts that perform well on LinkedIn justify the time investment in a full article. The platform made this visible without manual reconciliation across separate tabs.</p><p>LinkedIn reached 549,000 impressions over 90 days, up 28,369% compared with the prior period.</p><figure><img alt="LinkedIn content performance over the 90 days ending 3 August 2026." src="https://cdn-images-1.medium.com/max/876/1*4-URNzoNi8cQ8vh9iKjysg.png" /><figcaption><em>LinkedIn content performance over the 90 days ending 3 August 2026.</em></figcaption></figure><p>Medium tracked that growth month by month. May 2026 showed 596 views and 276 reads. June reached 2,600 views and 1,400 reads, roughly four times the views and five times the reads in a single month. July settled at 916 views and 365 reads, still above the May baseline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hggrVLxOxhk-MtzMVBKRmQ.png" /><figcaption><em>Medium performance from May to July 2026. June followed the strongest wave of LinkedIn distribution.</em></figcaption></figure><p>The content classification model reinforces this. Every piece of content gets scored on format type, estimated production time, engagement rate and traffic movement. The output ranks formats by the return they produce relative to the time they consume.</p><p>Long-form technical articles with code examples consistently outperform opinion pieces on read-to-view ratio. That insight changed how production time gets allocated.</p><p>On the business side, contract profitability looks different when time, expenses and revenue are in the same query layer. A project that generates strong revenue but consumes disproportionate team time shows up differently from one that is smaller but more efficient. That visibility now informs which types of work to prioritise and which to price differently.</p><h3>The actual barrier to entry</h3><p>The reason most small companies do not have a data platform is not cost. A self-hosted stack running on $20 per month of infrastructure is available to any company with a technical person who can set it up. The reason is the assumption that it is not worth it yet.</p><p>That assumption has a compounding cost. Every month of decisions made without connected data is a month where the relationship between LinkedIn and Medium is invisible, contract profitability is estimated rather than calculated and content strategy is based on feel rather than evidence.</p><p>The platform does not need to be sophisticated to be valuable. It needs to exist.</p><p>Ronald Read did not wait until he had enough money to make investing worth it. He invested what he had consistently and let the system do the work. The parallel for data is not perfect, but the logic is the same: waiting for scale before building the system means the system is never there when the early decisions are made. Those decisions do not get cheaper to unmake later.</p><h3>What this looks like outside a consultancy</h3><p>The stack above reflects the needs of a consultancy tracking content, contracts and costs. The principle applies more broadly.</p><p>A five-person SaaS company has product usage data, subscription revenue, support tickets and churn. Connecting those sources answers questions that no single source can: which features correlate with retention, which support issues precede cancellations and which acquisition channels produce the highest lifetime value customers.</p><p>A small e-commerce brand has order data, ad spend, email performance and inventory. Connected, they answer which campaigns produce profitable customers rather than just customers.</p><p>The data is already there. Most small companies generate enough operational data to support real analytical decisions. The gap is not the data. It is the system that connects it.</p><h3>Where to start</h3><p>The mistake most small companies make when they decide to build is starting with the infrastructure. The right starting point is the question.</p><p>What decision, if you had better information, would you make differently?</p><p>Start there. Identify which sources contain the relevant data. Build the minimum pipeline that connects them. Answer the question. Then ask the next one.</p><p>A platform built this way, question by question, stays grounded in decisions rather than growing into infrastructure that nobody uses. It also stays cheap. The $20 per month VPS running ClickHouse started with two data sources and one question. The question got answered. The platform grew from there.</p><p>The enterprise version of this costs millions. The small-company version costs about twenty dollars a month and the discipline to build it before it feels necessary.</p><h3>“Okay, but this probably takes months and costs a fortune”</h3><p>It does not have to.</p><p>The ingestion pipelines, transformation layer and scoring models described above can fit into a fixed-scope implementation. A Northgrain Data Engineering Sprint is a 3 to 6 week engagement that delivers one defined outcome, validates it and hands it over.</p><p>If the questions in this article sound familiar and you want them answered about your own business rather than ours, <a href="https://northgraindata.com/services/data-engineering/data-engineering-sprint">the Data Engineering Sprint</a> is designed for that situation.</p><p><a href="https://northgraindata.com/">Data Engineering Consultancy | Northgrain Data</a></p><p><em>Originally published at </em><a href="https://northgraindata.com/blog/you-dont-need-big-data-to-make-data-driven-decisions"><em>https://northgraindata.com</em></a><em> on July 29, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86f3c0f80e88" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/you-dont-need-big-data-to-make-data-driven-decisions-86f3c0f80e88">You Don’t Need Big Data to Make Data-Driven Decisions</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enterprise-Ready Airflow: What to Think About Before Writing Your First DAG]]></title>
            <link>https://medium.com/lortech-solutions-technical-blog/enterprise-ready-airflow-what-to-think-about-before-writing-your-first-dag-f161dc1b5d32?source=rss-27db4b2773fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/f161dc1b5d32</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Joachim Hodana]]></dc:creator>
            <pubDate>Thu, 23 Jul 2026 13:21:52 GMT</pubDate>
            <atom:updated>2026-07-23T13:21:52.475Z</atom:updated>
            <content:encoded><![CDATA[<p><em>The decisions you make in week one determine whether your pipeline project scales gracefully or collapses under its own weight twelve months later</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ytjcJxlESkHrEQsoDsqsvA.png" /></figure><p>There is a particular kind of Airflow project that most data engineers have either lived through or heard about. It starts cleanly: a handful of DAGs, a clear purpose, a small team. Then requirements change. New data sources appear. Stakeholders want more pipelines, faster. Someone adds a workaround here, a copy-pasted DAG there. Within a year, the codebase is a maze of duplicated logic, inconsistent naming, no monitoring, and tests that exist only in theory. Deploying a change feels like defusing a bomb.</p><p>The frustrating part is that most of this is avoidable. Not by writing perfect code from day one, but by making a small number of structural decisions before writing the first DAG. This article is about those decisions.</p><h3>Requirements Will Change, Faster Than You Expect</h3><p>Every Airflow project I have seen starts with assumptions that turn out to be wrong. The pipeline that was supposed to run once a day now needs to run hourly. The source that was supposed to be stable starts changing its schema. The team that was three people becomes ten, and suddenly “everyone just knows how things work” stops being a viable documentation strategy.</p><p>This is not a failure of planning. It is the nature of data engineering in organizations that are actually using their data. The question is not how to prevent requirements from changing. It is how to build a system that absorbs change without requiring a rewrite every time.</p><p>The answer is abstraction, and the time to introduce it is before you have ten DAGs that all do things slightly differently.</p><p>The most common trap is writing DAGs as self-contained scripts. Each DAG defines its own connection handling, its own retry logic, its own notification behavior. It works fine until you need to change how retries work across all pipelines, and now you are making the same change in forty places.</p><p>Instead, treat your Airflow project as a Python package with real software engineering practices. Extract shared behavior into base classes and utility functions. Define a standard DAG factory that encodes your defaults. When requirements change, you change the factory, and every DAG that uses it gets the update for free.</p><p>A minimal example of what this looks like in practice:</p><pre># pipelines/factory.py<br>from airflow import DAG<br>from datetime import datetime<br>from pipelines.callbacks import notify_on_failure<br><br>DEFAULT_ARGS = {<br>        &quot;owner&quot;: owner,<br>        &quot;retries&quot;: 2,<br>        &quot;retry_delay&quot;: timedelta(minutes=5),<br>        &quot;on_failure_callback&quot;: notify_on_failure,<br>    }<br><br>def create_dag(<br>    dag_id: str,<br>    schedule: str,<br>    owner: str,<br>    tags: list[str],<br>    **kwargs<br>) -&gt; DAG:<br>    default_args = kwargs.get(&#39;default_args&#39;, DEFAULT_ARGS)<br>    return DAG(<br>        dag_id=dag_id,<br>        schedule=schedule,<br>        start_date=datetime(2024, 1, 1),<br>        default_args=default_args,<br>        tags=tags,<br>        catchup=False,<br>        **kwargs,<br>    )</pre><p>Every DAG in your project calls create_dag(). Retry logic lives in one place. Failure callbacks live in one place. When you need to add query tagging, or change the default retry count, or swap out the notification system, you change one file.</p><p>This feels like unnecessary abstraction when you have two DAGs. It feels like a lifesaver when you have fifty.</p><h3>SLA</h3><p>Nobody wants to spend a project kickoff talking about SLAs. It feels premature, bureaucratic, and removed from the actual work of building things. Skip it, and you will have a much worse conversation six months later when a stakeholder asks why a pipeline failed silently for three days and nobody noticed.</p><p>SLA in the context of Airflow is not just “how fast does the pipeline need to run.” It is a set of agreements that drive concrete engineering decisions. Before writing the first DAG, it is worth nailing down at least the following:</p><p><strong>What is the acceptable latency for each pipeline?</strong> A pipeline that feeds a real-time dashboard has different requirements than one that populates a weekly report. This determines scheduling, timeout settings, and how aggressively you need to optimize.</p><p><strong>What is the acceptable failure rate?</strong> Some pipelines can tolerate occasional failures and catch up on the next run. Others cannot miss a single execution. This determines your retry strategy, alerting thresholds, and whether you need backfill logic.</p><p><strong>Who needs to be notified when something breaks, and how quickly?</strong> “The team will notice” is not an SLA. Define a concrete escalation path: which failures go to Slack immediately, which generate a ticket, which require a page. This conversation forces stakeholders to actually prioritize.</p><p><strong>What counts as a breach?</strong> A DAG that runs but produces incorrect output is worse than a DAG that fails loudly. Agreeing on what “working correctly” means before you build the pipeline is much easier than agreeing on it after an incident.</p><p>Airflow has native SLA miss functionality — you can set sla on individual tasks and Airflow will call a callback when a task exceeds its expected duration. This is worth wiring up from the start, because adding SLA monitoring to a running production pipeline is one of those things that always gets deprioritized.</p><pre>from datetime import timedelta<br><br>with create_dag(&quot;orders_pipeline&quot;, schedule=&quot;@hourly&quot;, owner=&quot;data-eng&quot;, tags=[&quot;orders&quot;]) as dag:<br>    transform = PythonOperator(<br>        task_id=&quot;transform_orders&quot;,<br>        python_callable=transform_orders,<br>        sla=timedelta(minutes=30),  # alert if this task takes longer than 30 min<br>    )</pre><p>Having these conversations early also helps with a subtler problem: expectation management. Stakeholders who helped define the SLA are stakeholders who understand why an incident happened and what it means. Stakeholders who were never consulted tend to assume the system should be perfect.</p><h3>Monitoring</h3><p>An Airflow DAG that fails silently is worse than no DAG at all, because it creates the illusion that the pipeline is running. Monitoring is not optional in a production environment. The question is only how to implement it well.</p><p>The starter pattern is to wire failure callbacks to Slack and email at the DAG factory level, so every pipeline gets notifications without any per-DAG configuration. The callback receives a context object from Airflow that contains everything you need to build a useful alert.</p><pre># pipelines/callbacks.py<br>from airflow.hooks.base import BaseHook<br>from slack_sdk import WebClient<br><br>def notify_on_failure(context):<br>    dag_id = context[&quot;dag&quot;].dag_id<br>    task_id = context[&quot;task_instance&quot;].task_id<br>    execution_date = context[&quot;execution_date&quot;]<br>    log_url = context[&quot;task_instance&quot;].log_url<br>    client = WebClient(token=BaseHook.get_connection(&quot;slack_default&quot;).password)<br>    client.chat_postMessage(<br>        channel=&quot;#data-alerts&quot;,<br>        text=(<br>            f&quot;:red_circle: *Pipeline failure*\n&quot;<br>            f&quot;*DAG:* `{dag_id}`\n&quot;<br>            f&quot;*Task:* `{task_id}`\n&quot;<br>            f&quot;*Run:* `{execution_date}`\n&quot;<br>            f&quot;&lt;{log_url}|View logs&gt;&quot;<br>        ),<br>    )</pre><p>A few things make the difference between monitoring that is actually useful and monitoring that people learn to ignore. First, every alert should include a direct link to the logs. An alert that requires three clicks before you can see what went wrong creates friction that leads to alerts being dismissed. Second, alerts should be routed by severity and ownership. A failure in a pipeline owned by the finance team should go to the finance team’s channel, not a generic #data-alerts channel that nobody reads during the day. Third, retry noise is a real problem. If a task retries twice before succeeding, you do not want two failure alerts followed by a success alert. Configure your callbacks to only fire on the final failure, after retries are exhausted.</p><p>Beyond failure alerts, consider tracking DAG duration over time. A pipeline that used to run in ten minutes and now takes forty minutes is not failing, but something is wrong. Airflow exposes this data via its metadata database, and it is worth building a simple dashboard or periodic check that flags pipelines whose runtime is drifting upward.</p><h3>Testing</h3><p>Testing Airflow pipelines is more environment-dependent than testing most software, and pretending otherwise leads to advice that does not generalize. A team running Airflow on Astronomer has different options than a team running it on MWAA or a self-hosted Kubernetes cluster. Rather than prescribing a specific framework, here are the principles that hold across environments.</p><p><strong>Test your Python, not Airflow.</strong> The most reliable tests are the ones that test the business logic inside your operators and callables in complete isolation, with no Airflow runtime involved at all. If your transform_orders function contains logic worth testing, write a unit test for that function. Mock the database connections. Assert on the output. These tests are fast, stable, and environment-independent.</p><p><strong>Test DAG structure separately from DAG execution.</strong> Airflow provides utilities to import and inspect DAG objects without running them. Use these to assert that a DAG has the expected tasks, the expected dependencies, and the expected configuration. This catches a wide class of bugs before deployment without needing a running Airflow environment.</p><pre># tests/test_dag_structure.py<br>from airflow.models import DagBag<br><br>def test_orders_dag_loads():<br>    dagbag = DagBag(dag_folder=&quot;dags/&quot;, include_examples=False)<br>    assert &quot;orders_pipeline&quot; in dagbag.dags<br>    assert len(dagbag.import_errors) == 0<br><br>def test_orders_dag_task_count():<br>    dagbag = DagBag(dag_folder=&quot;dags/&quot;, include_examples=False)<br>    dag = dagbag.dags[&quot;orders_pipeline&quot;]<br>    assert len(dag.tasks) == 3</pre><p><strong>Be honest about integration tests.</strong> Full end-to-end tests that run DAGs against real infrastructure are valuable, but they are also slow, fragile, and expensive to maintain. In most teams, the right answer is a small number of carefully chosen integration tests that cover critical paths, not comprehensive coverage of every DAG. Define the boundary clearly so the team knows what the test suite does and does not guarantee.</p><h3>Linting</h3><p>Code review catches bugs, but it does not scale as a mechanism for enforcing consistency. As a team grows, “we agreed on the conventions” becomes “we sort of remember the conventions” becomes “everyone does it slightly differently.” Linting automates the enforcement so that conventions do not depend on reviewers remembering to check.</p><p>For Airflow projects, which are Python at their core, the tooling is mature and well worth the setup cost.</p><p><strong>Ruff</strong> is the current standard for Python linting and formatting. It is significantly faster than flake8 or pylint, covers a broad set of rules, and handles both linting and formatting in a single tool. For an Airflow project, the key rules to enable include import order enforcement, unused import detection, and complexity checks that flag DAGs or callables that are getting too large.</p><p><strong>Pre-commit hooks</strong> run linting automatically before each commit. The value here is not just catching issues earlier — it is removing the need to remember to run the linter manually. A developer who sets up pre-commit once never has to think about it again.</p><pre># .pre-commit-config.yaml<br>repos:<br>  - repo: https://github.com/astral-sh/ruff-pre-commit<br>    rev: v0.4.4<br>    hooks:<br>      - id: ruff<br>        args: [--fix]<br>      - id: ruff-format</pre><p>One honest caveat here: pre-commit integrations do not always play well with every Airflow deployment environment. Teams have encountered issues with specific hook versions conflicting with the Python environment that Airflow runs in, leading to hooks that either fail to install or behave unpredictably. The solution is to be conservative with hook versions, pin everything explicitly, and test the full pre-commit setup in a clean environment before rolling it out to the team. If a specific hook causes persistent problems, remove it and run that check in CI instead — a linter that runs in CI but not locally is still far better than a linter that runs nowhere.</p><p>The goal is not to have the most comprehensive linting setup. It is to have a setup that is stable, that the team actually uses, and that enforces the subset of rules that matter most for your codebase.</p><h3>Putting It Together</h3><p>None of these decisions are complicated in isolation. The challenge is that they all need to happen before the project has momentum, when it feels like you are slowing down to do things that could be deferred.</p><p>The cost of deferring them is not obvious until it is too late. An Airflow project without a DAG factory is a project where changing default retry behavior requires touching every DAG file. A project without defined SLAs is a project where every incident becomes a negotiation about expectations. A project without monitoring is a project where failures are discovered by users before they are discovered by the team.</p><p>Build the structure before you need it. The first week is the cheapest time to do it. And the teams that do it consistently are the ones who are still shipping quickly two years in, while everyone else is talking about the rewrite.</p><p><a href="https://joachimhodana.com/">Joachim Hodana - Software &amp; Data Engineer</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f161dc1b5d32" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lortech-solutions-technical-blog/enterprise-ready-airflow-what-to-think-about-before-writing-your-first-dag-f161dc1b5d32">Enterprise-Ready Airflow: What to Think About Before Writing Your First DAG</a> was originally published in <a href="https://medium.com/lortech-solutions-technical-blog">Lortech Solutions Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From 2.3GB to Under 600MB: A Real Docker Optimization Case Study]]></title>
            <link>https://levelup.gitconnected.com/from-2-3gb-to-under-600mb-a-real-docker-optimization-case-study-e68b45aeae93?source=rss-27db4b2773fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/e68b45aeae93</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[nextjs]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Joachim Hodana]]></dc:creator>
            <pubDate>Fri, 10 Jul 2026 16:19:55 GMT</pubDate>
            <atom:updated>2026-07-10T16:19:55.858Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Three services, three different problems, one shared root cause.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fSXbVaocCl-2AEVbSYLJuw.png" /></figure><p>We did not set out to optimize our Docker images. We set out to deploy faster, and the images were in the way.</p><p>Our CI pipeline was taking twelve minutes to build and push three containers. Pull times on fresh nodes were embarrassing. And when someone finally ran docker images and added up the numbers, we were looking at over eight gigabytes of container images for an application that, at its core, serves an API, processes a job queue, and renders a Next.js frontend. Nothing exotic. No justification for those numbers.</p><p>This is the story of how we fixed it, service by service, and what we learned about why Docker images get fat in the first place.</p><h3>The Architecture</h3><p>Before the numbers make sense, here is what we are actually running.</p><p>The system has three main services. The <strong>Go API</strong> handles all HTTP traffic — authentication, business logic, REST endpoints. The <strong>Queue Worker</strong> is also Go, consuming jobs from RabbitMQ: emails, async processing tasks, anything that should not block an HTTP response. The <strong>Next.js Client</strong> is the frontend, server-side rendered, talking to the API.</p><p>Supporting infrastructure: PostgreSQL for persistence, Redis for caching and sessions, RabbitMQ for the job queue.</p><pre>┌─────────────────────────────────────────────────────────────┐<br>│                        USER / BROWSER                        │<br>└───────────────────────────┬─────────────────────────────────┘<br>                            │ HTTPS<br>                            ▼<br>┌─────────────────────────────────────────────────────────────┐<br>│                    Next.js Client                           │<br>│                  node:20-bookworm-slim                       │<br>│              SSR · static assets · rewrites                 │<br>└───────────────────────────┬─────────────────────────────────┘<br>                            │ HTTP (internal)<br>                            ▼<br>┌─────────────────────────────────────────────────────────────┐<br>│                       Go API (core)                         │<br>│              gcr.io/distroless/static-debian12              │<br>│           REST · auth · business logic · mailer             │<br>└──────────┬─────────────────────────┬────────────────────────┘<br>           │ publishes jobs          │ reads/writes<br>           ▼                         ▼<br>┌──────────────────┐      ┌──────────────────────────────────┐<br>│    RabbitMQ      │      │           PostgreSQL              │<br>│   job queue      │      │           persistence             │<br>└──────────┬───────┘      └──────────────────────────────────┘<br>           │ consumes<br>           ▼<br>┌─────────────────────────────────────────────────────────────┐<br>│                    Go Queue Worker                          │<br>│              gcr.io/distroless/static-debian12              │<br>│         emails · async tasks · background jobs             │<br>└───────────────────────────┬─────────────────────────────────┘<br>                            │ reads/writes<br>                            ▼<br>                 ┌──────────────────┐<br>                 │      Redis       │<br>                 │ cache · sessions │<br>                 └──────────────────┘</pre><p>Three containers. All running production workloads. All starting from a place of unnecessary size.</p><h3>The Problem, Before Any Fixes</h3><p>Go API(golang:1.24.2): Single-stage ~860 MB<br>Queue Worker(golang:1.24.2): Multi-stage to Alpine ~180 MB<br>Next.js Client(node:23.10): Single-stage ~1.1–1.3 GB</p><p>Total: north of two gigabytes. The Go API was the most egregious — it was shipping the entire Go toolchain into production. The Client was carrying dev dependencies and source code that have no business being in a running container.</p><h3>Service 1: Go API — Single-Stage to Distroless</h3><p>The original Dockerfile for the API was a single stage. Build the binary, run the binary, in the same image:</p><pre>FROM golang:1.24.2<br>WORKDIR /app/src<br>COPY go.* ./<br>RUN go mod download &amp;&amp; go mod verify<br>COPY . .<br>RUN go build -o /app/main ./cmd/main.go<br>RUN chmod +x start.sh<br>EXPOSE 8080<br>CMD [&quot;./start.sh&quot;]</pre><p>The golang:1.24.2 image includes the Go compiler, the standard library sources, build tools, and a full Debian OS. All of that ends up in the final image, even though at runtime you only need the compiled binary and a handful of static assets.</p><p>The fix is a multi-stage build that uses golang:1.24.2 only for compilation, then throws it away:</p><pre>FROM golang:1.24.2 AS builder<br>WORKDIR /src<br>COPY go.mod go.sum ./<br>RUN go mod download &amp;&amp; go mod verify<br>COPY . .<br>RUN CGO_ENABLED=0 GOOS=linux go build \<br>    -trimpath \<br>    -ldflags=&quot;-s -w&quot; \<br>    -o /out/api ./cmd/main.go<br><br>FROM gcr.io/distroless/static-debian12:nonroot<br>WORKDIR /app<br>COPY --from=builder /out/api /app/api<br>COPY --from=builder /src/static /app/static<br>COPY --from=builder /src/internal/mailer/templates /app/internal/mailer/templates<br>EXPOSE 8080<br>CMD [&quot;/app/api&quot;]</pre><p>Three things are happening here that did not happen before.</p><p>The builder stage compiles with -trimpath and -ldflags=&quot;-s -w&quot;. The -s -w flags strip the symbol table and DWARF debug information from the binary. That alone reduces binary size by 20–30%. -trimpath removes local filesystem paths from the binary, which matters for reproducibility and security.</p><p>CGO_ENABLED=0 produces a statically linked binary — no dependency on shared C libraries. This is what makes the next part possible.</p><p>The runtime stage is gcr.io/distroless/static-debian12:nonroot. Distroless images are Google&#39;s answer to the question: what is the absolute minimum a container needs to run a static binary? The answer is a handful of certificates, timezone data, and almost nothing else. No shell. No package manager. No utilities. The :nonroot variant runs as a non-root user by default, which is a security improvement that comes for free.</p><p>The final image contains the binary and the static assets. Nothing else.</p><p><strong>Result: ~860 MB → ~60–80 MB. Around 90% reduction.</strong></p><h3>Service 2: Queue Worker — Alpine to Distroless</h3><p>The worker already had a multi-stage build. Someone had done the right thing structurally, but the runtime image was Alpine:</p><pre>FROM golang:1.24.2 AS builder<br>WORKDIR /build<br>COPY go.mod go.sum ./<br>RUN go mod download<br>COPY . .<br>WORKDIR /build/cmd/queue-worker<br>RUN CGO_ENABLED=0 GOOS=linux go build -o queue-worker<br><br>FROM alpine:3.21<br>RUN addgroup -S app &amp;&amp; adduser -S -G app app<br>USER app<br>WORKDIR /app<br>COPY --from=builder /build/cmd/queue-worker/queue-worker .<br>COPY --from=builder /build/internal/mailer/templates ./templates<br>COPY --from=builder /build/static ./static<br>CMD [&quot;./queue-worker&quot;]</pre><p>Alpine is a reasonable choice and much smaller than a full Debian image. But for a statically compiled Go binary, it is still carrying more than necessary — a shell, a package manager, BusyBox utilities. None of which the worker ever uses.</p><p>The change is small: swap Alpine for Distroless, add the binary size flags:</p><pre>FROM golang:1.24.2 AS builder<br>WORKDIR /build<br>COPY go.mod go.sum ./<br>RUN go mod download<br>COPY . .<br>WORKDIR /build/cmd/queue-worker<br>RUN CGO_ENABLED=0 GOOS=linux go build \<br>    -trimpath \<br>    -ldflags=&quot;-s -w&quot; \<br>    -o queue-worker<br><br>FROM gcr.io/distroless/static-debian12:nonroot<br>WORKDIR /app<br>COPY --from=builder /build/cmd/queue-worker/queue-worker .<br>COPY --from=builder /build/internal/mailer/templates /app/internal/mailer/templates<br>COPY --from=builder /build/static /app/static<br>CMD [&quot;./queue-worker&quot;]</pre><p>The asset paths also got cleaned up — templates and static now live under /app/ consistently, which matters when the binary constructs paths at runtime.</p><p><strong>Result: ~180 MB → ~50–70 MB. About 60–70% reduction, on top of work that had already been done.</strong></p><p>The broader lesson here: having a multi-stage build does not mean the runtime image is optimal. Alpine is not the end of the road for Go services.</p><h3>Service 3: Next.js Client — The Hardest One</h3><p>The frontend was the most complex case, because Next.js is not a statically compiled binary. It requires a Node runtime, and the relationship between build-time and runtime dependencies is less clean.</p><p>The original Dockerfile was a single stage that built and served from the same image:</p><pre>FROM node:23.10<br>WORKDIR /app<br>COPY package*.json ./<br>RUN npm install<br>COPY . .<br>ARG NEXT_PUBLIC_API_URL<br># ... more ARGs and ENVs ...<br>RUN npm run build<br>EXPOSE 3000<br>CMD [&quot;npm&quot;, &quot;run&quot;, &quot;start&quot;]</pre><p>This image contains the source code, all dev dependencies, the build artifacts, and a full Node 23 OS image. Starting production with npm run start means the container has everything it needed to build the app sitting in layers that never get used again.</p><p>The fix required two changes working together. First, enabling Next.js standalone output in next.config.ts:</p><pre>const nextConfig: NextConfig = {<br>  output: &quot;standalone&quot;,<br>  // ...<br>};</pre><p>Standalone mode tells Next.js to produce a self-contained server bundle in .next/standalone — a minimal Node server with only the production dependencies it actually uses, no node_modules directory needed. This is Next.js doing the equivalent of static linking: bundling just what it needs.</p><p>Then, a three-stage Dockerfile that takes advantage of it:</p><pre>FROM node:20-bookworm-slim AS deps<br>WORKDIR /app<br>COPY package.json package-lock.json ./<br>RUN npm ci<br><br>FROM node:20-bookworm-slim AS builder<br>WORKDIR /app<br>COPY --from=deps /app/node_modules ./node_modules<br>COPY . .<br>ARG NEXT_PUBLIC_API_URL<br>ARG NEXT_PUBLIC_GOOGLE_CLIENT_ID<br>ARG NEXT_PUBLIC_POSTHOG_KEY<br>ARG NEXT_PUBLIC_APP_ENV<br># ... other build args ...<br>ENV NEXT_TELEMETRY_DISABLED=1<br>RUN npm run build<br><br>FROM node:20-bookworm-slim AS runner<br>WORKDIR /app<br>ENV NODE_ENV=production<br>ENV NEXT_TELEMETRY_DISABLED=1<br>COPY --from=builder /app/.next/standalone ./<br>COPY --from=builder /app/.next/static ./public/_next/static<br>COPY --from=builder /app/public ./public<br>EXPOSE 3000<br>CMD [&quot;node&quot;, &quot;server.js&quot;]</pre><p>The runner stage copies three things from the builder: the standalone bundle, the static assets, and the public directory. The source code never enters the runner. The dev dependencies never enter the runner. The full node_modules directory never enters the runner.</p><p>node:20-bookworm-slim replaced node:23.10. The slim variant is significantly smaller than the full image. Node 20 is the current LTS and appropriate for production; Node 23 was a cutting-edge release that brought size without meaningful benefit for this workload.</p><p>NEXT_TELEMETRY_DISABLED=1 is a small but worthwhile addition — it prevents Next.js from sending telemetry during the build and at runtime.</p><p><strong>Result: ~1.1–1.3 GB → ~350–450 MB. About 60–70% reduction.</strong></p><p>The Next.js reduction is smaller in percentage terms than the Go services, because there is a real minimum size for a Node.js runtime that you cannot easily escape. The standalone output gets you as close to that minimum as Next.js allows.</p><h3>The Full Picture</h3><p><strong>Total: ~2.1–2.3 GB -&gt; ~460–600 MB </strong>(~75%)</p><p>Beyond the numbers, the security posture improved considerably. Distroless images have no shell, which means there is nothing to exec into if a container is compromised — a meaningful reduction in attack surface for production services. Running as nonroot by default removes another category of risk.</p><p>CI build times dropped as well, mostly from the smaller push and pull sizes. A 60 MB image moves across a network in a fraction of the time an 860 MB image does.</p><h3>What to Take Away</h3><p>Each service needed a different approach because each had a different root cause.</p><p>The Go API was shipping its build environment into production. The fix was separating build from runtime and choosing the most minimal runtime possible for a static binary.</p><p>The Queue Worker had already done the separation but stopped at a runtime image that was still carrying unnecessary tools. The fix was going one step further.</p><p>The Next.js Client was building and serving from the same stage, carrying source code and dev dependencies into production. The fix was enabling standalone mode and using a proper multi-stage build that only ships what the server actually runs.</p><p>The common thread: each image was larger than it needed to be because nobody had explicitly asked what the image needed to do its job at runtime, and what it absolutely did not need. That question is worth asking before you write the first line of any Dockerfile.</p><p><a href="https://joachimhodana.com/">Joachim Hodana - Software &amp; Data Engineer</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e68b45aeae93" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/from-2-3gb-to-under-600mb-a-real-docker-optimization-case-study-e68b45aeae93">From 2.3GB to Under 600MB: A Real Docker Optimization Case Study</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enterprise dbt Project Setup: A Starter Kit That Scales]]></title>
            <link>https://medium.com/lortech-solutions-technical-blog/enterprise-dbt-project-setup-a-starter-kit-that-scales-f8764f086f20?source=rss-27db4b2773fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8764f086f20</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[dbt]]></category>
            <dc:creator><![CDATA[Joachim Hodana]]></dc:creator>
            <pubDate>Tue, 07 Jul 2026 07:25:27 GMT</pubDate>
            <atom:updated>2026-07-07T07:25:27.360Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Everything you need to ship a production-ready dbt project from day one</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jej9eVkZWqtTkJBtoctBQA.png" /></figure><p>Every dbt project starts the same way. Someone creates a new repo, runs dbt init, and thinks: &quot;We&#39;ll add proper structure later.&quot; Six months pass. The project has grown, three more engineers are contributing, there&#39;s no consistent naming convention, no CI pipeline, no metadata validation, and nobody really knows which models are safe to change. The technical debt is real, and now fixing it requires a weekend nobody wants to give up.</p><p>We’ve seen this pattern enough times to stop accepting it as inevitable. So we built a starter kit that bakes in enterprise-grade practices from the very first commit. This article walks through what’s in it, why each piece exists, and how it saves you from the most common pitfalls when scaling a dbt project in an organization.</p><p>The template is available on GitHub. Clone it, adapt it to your stack, and skip the painful lessons.</p><h3>The Folder Structure That Won’t Haunt You Later</h3><p>The first decision a dbt project makes is how to organize models. Most tutorials show a flat models/ folder. That works fine for ten models. At a hundred, it becomes a navigational nightmare.</p><p>The starter kit uses a strict three-layer structure:</p><pre>project/<br>  models/<br>    staging/       # source-aligned, one model per source table<br>    intermediate/  # reusable business logic<br>    marts/         # final facts and dimensions consumed by BI tools<br>  macros/<br>  analyses/codegen/<br>  tests/unit/<br>  seeds/<br>  snapshots/</pre><p>The staging layer is about faithfulness to the source. Each model in staging/ maps one-to-one with an upstream table, does light cleaning (casting types, renaming columns, filtering deleted records), and nothing else. No business logic lives here.</p><p>The intermediate layer is where business logic becomes reusable. If you find yourself writing the same join in three different marts, it belongs in intermediate/. This layer is often skipped on smaller projects, and that is precisely when it starts being missed.</p><p>The marts layer is the public API of your data warehouse. These are the models your analysts and BI tools actually touch. They should be stable, well-documented, and tested.</p><p>This separation matters for onboarding. A new engineer joining the team can look at the folder structure and immediately understand how data flows through the system. That clarity has real value.</p><p>Keep in mind this is the most basic separation. You probably should adjust it to your business needs. For example each source could use it’s own folder inside of staging layer.</p><h3>Metadata Contracts</h3><p>This is the part of the starter kit that gets the strongest reaction. When I tell people that every model must have contract.enforced: true in its config, alongside a data_type and description on every column, the immediate response is usually: &quot;That&#39;s a lot of overhead.&quot;</p><p>It is not overhead. It is load-bearing structure.</p><p>Here is what happens without contracts: an engineer renames a column in a staging model, the marts that depend on it fail silently or — worse — start returning nulls, and three days later a stakeholder notices the numbers look wrong. The root cause takes an afternoon to find.</p><p>With contract.enforced: true, dbt will refuse to build a model if its actual schema does not match the declared schema. The failure is loud, immediate, and actionable.</p><pre># stg_example_orders.yml<br>models:<br>  - name: stg_example_orders<br>    config:<br>      contract:<br>        enforced: true<br>    columns:<br>      - name: order_id<br>        description: &quot;Unique identifier for each order.&quot;<br>        data_type: varchar<br>      - name: order_date<br>        description: &quot;Date the order was placed.&quot;<br>        data_type: date</pre><p>But the starter kit goes further. A Pydantic validation script runs in CI and checks that:</p><ul><li>every .sql file in models/ has a corresponding .yml file</li><li>every .yml file uses version: 2</li><li>every column has both a name, a description, and a data_type</li><li>contract.enforced is set to true</li></ul><p>If any of these checks fail, the CI pipeline fails. There is no way to merge a model that skips documentation. This is the kind of guardrail that feels annoying the first week and becomes invisible by the second month, because everyone just does it.</p><h3>Linting and Pre-commit: Catch Issues Before They Hit the Repo</h3><p>Code review should focus on logic, not formatting. Formatting debates in pull requests are a waste of everyone’s time, and they leave room for real issues to slip through because reviewers are distracted by indentation.</p><p>The starter kit uses pre-commit hooks to enforce consistency before code ever reaches a pull request. The .pre-commit-config.yaml wires together four tools:</p><p><strong>SQLFluff</strong> handles SQL formatting and style. It is configured with the Snowflake dialect and the dbt templater, which means it understands Jinja and can lint your actual dbt models rather than treating {{ ref(&#39;...&#39;) }} as a syntax error.</p><p><strong>Ruff</strong> handles Python. The accompanying pyproject.toml manages the project as a Python package using uv, which makes dependency management reproducible across machines and CI environments.</p><p><strong>yamllint</strong> catches malformed YAML before it causes confusing dbt parse errors. A missing space after a colon in a schema file is the kind of bug that wastes twenty minutes.</p><p><strong>gitleaks</strong> scans for secrets. Warehouse credentials and API keys should never be committed. This is non-negotiable in an enterprise context, and having it run automatically means you never have to rely on “we trust developers to remember.”</p><p>Running pre-commit install after cloning the repo is a one-time step. After that, every commit is automatically validated before it leaves the developer&#39;s machine.</p><h3>CI/CD Workflows That Actually Run dbt</h3><p>Pre-commit catches local issues. CI catches everything that slips through and enforces standards across the entire team, including contributors who have not set up pre-commit locally.</p><p>The starter kit includes four GitHub Actions workflows:</p><p><strong>lint.yml</strong> runs ruff, SQLFluff, and yamllint on every pull request. Fast feedback, no warehouse required.</p><p><strong>validate-model-metadata.yml</strong> runs the Pydantic validation script. This is a separate job because it is distinct from linting — it is checking semantic correctness of your documentation, not code style.</p><p><strong>dbt-build.yml</strong> runs dbt debug followed by dbt build against a Snowflake CI environment. This is the workflow that actually compiles and runs your models to prove the build succeeds. Credentials are passed via environment variables; the included profiles/profiles.yml shows exactly which variables are expected.</p><p><strong>gitleaks.yml</strong> scans the full repository history for secrets. This runs on push to main and on pull requests, because catching a committed secret on the branch is better than catching it after merge.</p><p>Together these four workflows mean that nothing can be merged to main unless the SQL lints, the YAML is valid, the metadata is complete, and the dbt build succeeds against a real warehouse. That is a strong guarantee.</p><h3>Two Macros Every Project Needs</h3><p>Of all the things in the starter kit, these two macros are the ones I see omitted most often on teams that later regret it.</p><p><strong>generate_schema_name</strong> controls how dbt constructs schema names in the warehouse. Without this macro, dbt uses its default behavior, which means every developer writing to their personal development environment risks colliding with the production schema if environment variables are not set perfectly. The macro in the starter kit enforces a clear convention: in production, models land in their configured schema; in development, they land in a developer-specific prefix so that dbt build in a dev environment never touches production data.</p><pre>{% macro generate_schema_name(custom_schema_name, node) %}<br>  {# <br>    Production:<br>      - Use folder-level +schema (custom_schema_name) when defined<br>      - Otherwise fall back to target.schema (e.g. dbt_analytics)<br>    Non-prod (dev, ci, etc.):<br>      - Always use target.schema from the profile (e.g. dbt_joachimhodana),<br>        so everything builds into your personal dataset.<br>  #}<br>  {% if target.name in [&#39;prod&#39;, &#39;dev&#39;, &#39;qa&#39;, &#39;ci&#39;] %}<br>    {{ custom_schema_name or target.schema }}<br>  {% else %}<br>    {{ target.schema }}<br>  {% endif %}<br>{% endmacro %}</pre><p><strong>query_tag</strong> sets a Snowflake query tag on every query dbt runs. This means that when you open Snowflake&#39;s query history, you can see exactly which dbt model triggered a query, which environment it ran in, and which layer it belongs to. This is observability you will use constantly when debugging performance issues or tracking warehouse costs by layer.</p><p>The macro is applied via a pre-hook in dbt_project.yml at the layer level, so staging queries are tagged differently from mart queries with no per-model configuration required.</p><h3>Testing Strategy: Unit Tests and dbt-expectations</h3><p>The default dbt tests — not_null and unique — are a starting point, not a strategy. They will catch obvious data quality failures but miss the subtler ones: wrong calculations, unexpected distributions, rows that should exist but do not.</p><p>The starter kit includes dbt-expectations as a package dependency. This library ports many of the Great Expectations test patterns into dbt-native YAML. You can assert that a column&#39;s values fall within a specific range, that the row count of a model is above a minimum threshold, or that a column matches a regex pattern.</p><pre>- name: order_amount<br>  tests:<br>    - dbt_expectations.expect_column_values_to_be_between:<br>        min_value: 0<br>        max_value: 1000000</pre><p>The starter kit also includes an example unit test. Unit tests in dbt let you test a model’s logic in isolation by providing mock input data and asserting what the output should look like. This is especially valuable for intermediate models with complex logic — you can test edge cases without needing a full data refresh.</p><h3>The Packages Worth Adding From Day One</h3><p>Four packages come pre-configured in project/packages.yml:</p><p><strong>dbt-labs/codegen</strong> generates YAML schema files from existing models. When you inherit a legacy project with no documentation, this is the fastest way to create a baseline you can then enrich.</p><p><strong>dbt-labs/dbt_utils</strong> provides a collection of commonly needed SQL utilities: surrogate key generation, date spine creation, pivot helpers. You will use these sooner than you expect.</p><p><strong>calogica/dbt_expectations</strong> was covered above. Advanced data quality testing that goes well beyond the built-in tests.</p><p><strong>Matts52/dbt_orphan</strong> is the one most teams have not heard of. It identifies and optionally drops warehouse tables that dbt no longer manages — the leftover debris from renamed or deleted models. The starter kit includes a commented-out on-run-end hook showing how to wire it in. Enabling it in production is a one-line change.</p><h3>Get the Template</h3><p>Clone the repository and you will have a working project skeleton with all the tooling configured and ready to run:</p><pre>git clone https://github.com/lortechsolutions/dbt-starter-kit<br>cd dbt-starter-kit<br>uv sync<br>cd project<br>dbt debug</pre><p>Before you start building, there are four things to adapt to your organization: update CODEOWNERS with the right GitHub handles, fill in your Snowflake credentials in the environment variables expected by profiles.yml and rename the dbt project in dbt_project.yml</p><p>Everything else — the folder structure, the CI workflows, the macros, the validation script, the pre-commit hooks — is ready to use as-is.</p><p>The goal of this template is not to be prescriptive about every detail of how you build your models. It is to eliminate the decisions that every enterprise dbt project eventually makes the same way, and to make the good defaults automatic rather than aspirational.</p><p><a href="https://joachimhodana.com/">Joachim Hodana - Software &amp; Data Engineer</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8764f086f20" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lortech-solutions-technical-blog/enterprise-dbt-project-setup-a-starter-kit-that-scales-f8764f086f20">Enterprise dbt Project Setup: A Starter Kit That Scales</a> was originally published in <a href="https://medium.com/lortech-solutions-technical-blog">Lortech Solutions Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What breaks your Snowflake pipeline at 100M+ rows]]></title>
            <link>https://blog.dataengineerthings.org/what-breaks-your-snowflake-pipeline-at-100m-rows-1801c3f763f5?source=rss-27db4b2773fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/1801c3f763f5</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Joachim Hodana]]></dc:creator>
            <pubDate>Fri, 03 Jul 2026 15:51:21 GMT</pubDate>
            <atom:updated>2026-07-03T15:51:21.392Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N-clOhif7qATf2ukie_VPQ.png" /></figure><p>Pipelines that process a few million rows rarely reveal architectural problems. Snowflake is fast, warehouses auto-suspend, dbt models finish in seconds, and everything looks fine. Then data volume crosses a threshold, usually somewhere between 50 and 200 million rows on a core fact table, and things that used to work start producing 10-minute model runs, credit spikes, and queries that time out or spill to disk. The same SQL, the same warehouse size, the same dbt config.</p><p>Nothing obviously broke. The pipeline just scaled into its own weaknesses.</p><p>This article covers five failure modes that surface consistently at this scale and what to do about each one.</p><h3>How Snowflake stores data, and why it matters at scale</h3><p>Before the failure modes make sense, the storage model does.</p><p>All data in Snowflake tables is automatically divided into micro-partitions, which are contiguous units of storage. Snowflake collects and records clustering metadata for each micro-partition created during the loading process, then leverages this clustering information to avoid unnecessary scanning of micro-partitions during querying.</p><p>Each micro-partition contains between 50 MB and 500 MB of uncompressed data. While partitions are not defined directly, query performance can be influenced through date-based filtering in incremental models and by aligning incremental logic with natural partitioning.</p><p>The key concept is pruning: when a query has a WHERE clause on a well-clustered column, Snowflake reads only the micro-partitions that can contain matching rows and skips the rest. On a 100M row table that is well clustered on event_date, a query filtering on a single day might scan 0.5% of the data. On a poorly clustered table, the same query scans everything.</p><p>At small scale, the difference between scanning 100% and scanning 1% is a few seconds. At 100M+ rows, it is the difference between a 10-second query and a 15-minute one.</p><h3>Failure mode 1: dbt incremental models scanning the full table</h3><p>The most common problem at this scale is a dbt incremental model that is supposed to process only new rows but ends up scanning the entire target table on every run.</p><p>The naive incremental config looks like this:</p><pre>-- models/fct_events.sql<br>{{ config(<br>    materialized=&#39;incremental&#39;,<br>    unique_key=&#39;event_id&#39;,<br>    incremental_strategy=&#39;merge&#39;<br>) }}<br><br>select<br>    event_id,<br>    user_id,<br>    event_type,<br>    event_date,<br>    amount<br>from {{ ref(&#39;stg_events&#39;) }}<br>{% if is_incremental() %}<br>    where event_date &gt; (select max(event_date) from {{ this }})<br>{% endif %}</pre><p>The where clause filters the source, which is correct. The problem is the MERGE operation itself. Every MERGE in Snowflake starts by scanning the target table to find matching rows. Without a predicate on a clustered column in the join condition, Snowflake scans every micro-partition of fct_events on every run.</p><p>Every time a MERGE query runs, it starts by scanning the target table. This is one of the most time-consuming steps. To reduce the time spent scanning data, the target table needs to be filtered on a column it is well clustered on. This enables query pruning, which prevents Snowflake from scanning unnecessary micro-partitions.</p><p>The fix is to add a date predicate to the merge condition, so Snowflake can prune the target table before joining:</p><pre>-- models/fct_events.sql<br>{{ config(<br>    materialized=&#39;incremental&#39;,<br>    unique_key=&#39;event_id&#39;,<br>    incremental_strategy=&#39;merge&#39;,<br>    cluster_by=[&#39;event_date&#39;]<br>) }}<br><br>select<br>    event_id,<br>    user_id,<br>    event_type,<br>    event_date,<br>    amount<br>from {{ ref(&#39;stg_events&#39;) }}<br>{% if is_incremental() %}<br>    where event_date &gt; (select max(event_date) from {{ this }})<br>{% endif %}</pre><p>And in the dbt_project.yml, pushing the date predicate into the merge condition via a post-filter:</p><pre>-- or use a custom incremental predicate (dbt 1.4+)<br>{{ config(<br>    materialized=&#39;incremental&#39;,<br>    unique_key=&#39;event_id&#39;,<br>    incremental_strategy=&#39;merge&#39;,<br>    cluster_by=[&#39;event_date&#39;],<br>    incremental_predicates=[<br>        &quot;dbt_internal_dest.event_date &gt;= dateadd(day, -3, current_date)&quot;<br>    ]<br>) }}</pre><p>incremental_predicates tells dbt to add that condition to the MERGE&#39;s ON clause, which allows Snowflake to prune the target table. Combined with the cluster_by, the merge now touches only the last 3 days of micro-partitions instead of the entire table.</p><p>A 100-million row table that was often joined on USER_ID had slow joins that frequently spilled to disk because the data was not clustered. By clustering that table on USER_ID, one heavy join went from spilling and taking 10+ minutes to running in 2 minutes with no spill, a 5x speedup. The clustering ensured that matching rows shared partitions, making the join operation much more efficient.</p><h3>Failure mode 2: clustering degradation over time</h3><p>Natural clustering is the order data arrives in Snowflake. If events are loaded chronologically, event_date has good natural clustering initially. Over time, as incremental loads, late-arriving data, and DML operations add rows, that natural ordering degrades. Micro-partitions that used to contain data from a single day start containing data from multiple months. Pruning stops working.</p><p>Over time, as new data is inserted or updated, the natural ordering of micro-partitions degrades. Filtering gets slower, and Snowflake must scan more partitions than necessary. Clustering fixes that. It physically reorders the data by one or more columns so that related rows are grouped together inside partitions.</p><p>The first step is measuring clustering health before assuming it is a problem:</p><pre>-- check clustering information on a table<br>select system$clustering_information(&#39;analytics.fct_events&#39;, &#39;(event_date)&#39;);<br><br>-- check clustering history and cost<br>select<br>    to_date(start_time)     as date,<br>    table_name,<br>    sum(credits_used)       as credits<br>from snowflake.account_usage.automatic_clustering_history<br>where start_time &gt; dateadd(day, -30, current_timestamp())<br>group by 1, 2<br>order by 3 desc;</pre><p>The output of system$clustering_information includes average_depth, which is the average number of micro-partitions that overlap for the specified column. A depth close to 1 means good clustering. A depth of 5 or higher means Snowflake has to read 5 partitions to find data it should find in 1.</p><p>Clustering keys are not intended for all tables due to the costs of initially clustering the data and maintaining the clustering. Clustering is optimal when queries require the fastest possible response times regardless of cost, or when the table is very large and queries are selective and benefit from the same clustering key.</p><p>For large fact tables that are queried frequently on date columns, automatic clustering is usually worth the cost. For dbt models, the config is:</p><pre>{{ config(<br>    materialized=&#39;table&#39;,<br>    cluster_by=[&#39;event_date&#39;, &#39;event_type&#39;]<br>) }}</pre><p>For tables where automatic clustering is too expensive, a scheduled reclustering job run weekly or after large loads achieves most of the benefit at lower cost:</p><pre>-- run after large batch loads<br>alter table analytics.fct_events recluster;</pre><h3>Failure mode 3: exploding joins</h3><p>An exploding join happens when join logic is flawed, causing the number of rows to multiply unexpectedly. This is often due to a missing or incorrect join condition, or joining on columns that are not unique, resulting in an unintentional cartesian product. The Query Profile provides a clear visual clue: if the row count jumps from 100k on one side of a join to 100M on the other, an exploding join has been found.</p><p>At small table sizes, an exploding join produces a slow query. At 100M+ rows, it produces an out-of-memory error, massive disk spillage, or a query that runs for hours before timing out. The credit consumption can be extreme.</p><p>The most common source is a many-to-many relationship that was treated as one-to-many:</p><pre>-- this looks correct but produces an explosion if order_id is not unique in order_items<br>select<br>    o.customer_id,<br>    sum(oi.amount) as total<br>from orders o<br>join order_items oi on o.order_id = oi.order_id<br>group by o.customer_id</pre><p>If order_items has 20 rows per order and orders has 5M rows, the join intermediate result is 100M rows before the GROUP BY. If order_id is also not unique in orders (due to a data quality issue or incorrect model), the explosion is multiplicative.</p><p>The diagnostic is in the Query Profile. Open the profile for a slow query, find the join operator, and check the row count on the output side vs. the input sides. If the output is dramatically larger than either input, there is an explosion.</p><p>The fix depends on the cause. For genuine many-to-many relationships, aggregate before joining:</p><pre>-- aggregate order_items first, then join<br>with order_totals as (<br>    select<br>        order_id,<br>        sum(amount) as total_amount<br>    from order_items<br>    group by order_id<br>)<br>select<br>    o.customer_id,<br>    ot.total_amount<br>from orders o<br>join order_totals ot on o.order_id = ot.order_id</pre><p>For data quality issues causing unexpected duplicates, add a deduplication CTE and a dbt test:</p><pre># schema.yml<br>models:<br>  - name: stg_orders<br>    columns:<br>      - name: order_id<br>        tests:<br>          - unique<br>          - not_null</pre><p>Catching the duplicate at the staging layer means the explosion never reaches the mart.</p><h3>Failure mode 4: disk spillage from undersized warehouses</h3><p>When Snowflake runs out of local memory and disk, it leans on remote storage, which is much slower and introduces network latency. By scaling up the warehouse, RAM and local disk double, giving queries a faster runway.</p><p>Spillage appears in the Query Profile as “bytes spilled to local storage” or “bytes spilled to remote storage.” Local spillage is slow. Remote spillage is very slow and also costs more in storage I/O.</p><p>The two operations that most commonly cause spillage at 100M+ rows are large sorts and hash joins where the build side exceeds available memory.</p><p>The diagnostic before changing warehouse size:</p><pre>-- find queries with significant spillage in the last 7 days<br>select<br>    query_id,<br>    query_text,<br>    warehouse_size,<br>    bytes_spilled_to_local_storage,<br>    bytes_spilled_to_remote_storage,<br>    total_elapsed_time / 1000                   as elapsed_seconds,<br>    credits_used_cloud_services<br>from snowflake.account_usage.query_history<br>where start_time &gt; dateadd(day, -7, current_timestamp())<br>  and (bytes_spilled_to_local_storage &gt; 0<br>       or bytes_spilled_to_remote_storage &gt; 0)<br>order by bytes_spilled_to_remote_storage desc<br>limit 20;</pre><p>Before scaling the warehouse, check whether the spillage is caused by query logic rather than warehouse capacity. A join that explodes from 5M to 500M rows will spill on any warehouse size, because the problem is the data volume being produced, not the warehouse’s memory. Fixing the join logic is cheaper than buying an XXXL warehouse.</p><p>For queries where the logic is correct and the warehouse is genuinely undersized, Airflow or dbt can use a larger warehouse only for the heavy models:</p><pre>-- dbt model config<br>{{ config(<br>    materialized=&#39;incremental&#39;,<br>    snowflake_warehouse=&#39;TRANSFORM_XL&#39;<br>) }}</pre><pre># Airflow: override warehouse per task<br>SnowflakeOperator(<br>    task_id=&quot;run_heavy_model&quot;,<br>    sql=&quot;alter session set warehouse = TRANSFORM_XL; call run_model();&quot;,<br>    snowflake_conn_id=&quot;snowflake_prod&quot;,<br>)</pre><p>This avoids paying for a large warehouse for the entire pipeline when only one or two models need it.</p><h3>Failure mode 5: workload contention from shared warehouses</h3><p>At small scale, running ETL, dbt transformations, and analyst queries on the same warehouse is fine. At 100M+ rows, a single dbt full-refresh or a heavy analyst query can saturate the warehouse and queue everything else.</p><p>High concurrency can lead to resource contention, causing performance bottlenecks. Separate warehouses for different workload types, such as ETL, reporting, and ad-hoc queries, prevent resource contention.</p><p>The practical separation that works for most teams:</p><pre>-- create separate warehouses per workload type<br>create warehouse transform_wh<br>    warehouse_size = &#39;LARGE&#39;<br>    auto_suspend = 120<br>    auto_resume = true<br>    comment = &#39;dbt transformations and batch ETL&#39;;<br><br>create warehouse reporting_wh<br>    warehouse_size = &#39;MEDIUM&#39;<br>    auto_suspend = 60<br>    auto_resume = true<br>    comment = &#39;BI tools and analyst queries&#39;;<br><br>create warehouse adhoc_wh<br>    warehouse_size = &#39;SMALL&#39;<br>    auto_suspend = 60<br>    auto_resume = true<br>    comment = &#39;ad-hoc exploration, dev, testing&#39;;</pre><p>In dbt, the warehouse is set per profile target or per model:</p><pre># profiles.yml<br>my_project:<br>  target: prod<br>  outputs:<br>    prod:<br>      type: snowflake<br>      warehouse: transform_wh<br>      # ...</pre><p>For Airflow-triggered Snowflake queries, the connection’s default warehouse handles routine tasks and specific operators override it for heavy ones. The key is that a long-running full-refresh on fct_events no longer blocks the dashboard that a VP is trying to open at 9am.</p><h3>The Query Profile is the starting point for all of this</h3><p>Every failure mode above is diagnosable through Snowflake’s Query Profile before any code changes. The pattern is the same each time: find the slow query in Snowflake’s query history, open the profile, identify which operator is consuming the most time or bytes, and trace it back to one of the failure modes above.</p><p>One of the first things to look at is the micro-partitions scanned. The profile shows partitions scanned out of total. If a query scanned 500 out of 500 partitions of a table, that is a red flag indicating no pruning was happening. On the other hand, scanning 5 out of 500 is great pruning, reading only 1% of data. If a join or group by step consumes most of the time, the join may be exploding data or not using a join key effectively. If a single operator takes 50% of the time, ask if that sort is necessary or if the data can be pre-sorted to avoid heavy in-memory sorts.</p><p>The profile turns a vague “the pipeline is slow” into a specific operator on a specific table with a specific cause. That is the right place to start before changing warehouse sizes, adding clustering keys, or rewriting SQL.</p><ul><li><a href="https://joachimhodana.com/">Joachim Hodana - Software &amp; Data Engineer</a></li><li><a href="https://betterdataengineer.joachimhodana.com/">Better Data Engineer</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1801c3f763f5" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/what-breaks-your-snowflake-pipeline-at-100m-rows-1801c3f763f5">What breaks your Snowflake pipeline at 100M+ rows</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Writing Airflow DAGs that don’t fight you in production]]></title>
            <link>https://blog.dataengineerthings.org/writing-airflow-dags-that-dont-fight-you-in-production-d49fd0d5ddf5?source=rss-27db4b2773fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/d49fd0d5ddf5</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[airflow]]></category>
            <dc:creator><![CDATA[Joachim Hodana]]></dc:creator>
            <pubDate>Tue, 23 Jun 2026 14:02:12 GMT</pubDate>
            <atom:updated>2026-06-23T14:02:12.493Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m5WG_62MsGCWVihis0F90g.png" /></figure><p>Most Airflow performance problems are not infrastructure problems. Adding more workers or tuning the scheduler helps, but the biggest gains almost always come from how DAGs are written. A poorly structured DAG file creates overhead every 30 seconds, on every parse, regardless of whether any tasks are running.</p><p>This article covers five patterns that make a concrete difference in production: avoiding expensive top-level code, using ShortCircuitOperator to skip work that does not need to happen, dynamic task mapping instead of copy-pasted operators, BranchPythonOperator for conditional paths, and deferred operators for long-running waits. Each one addresses a specific, common mistake.</p><h3>Pattern 1: keep top-level code cheap</h3><p>Airflow parses every file in the dags/ folder continuously, roughly every 30 seconds by default. Every line of code outside a task or operator definition runs on every parse, in the scheduler process, whether or not any DAG is currently running.</p><p>This is the most common source of unexplained Airflow slowness, and it looks harmless:</p><pre># dags/orders_pipeline.py - this runs every 30 seconds<br><br>from airflow.decorators import dag, task<br>from mypackage.config import load_pipeline_config<br>import pandas as pd  # heavy import<br>from sqlalchemy import create_engine<br><br># This hits the metadata DB on every parse<br>from airflow.models import Variable<br><br>tables = Variable.get(&quot;tables_to_process&quot;, deserialize_json=True)<br><br># This opens a DB connection on every parse<br>engine = create_engine(os.environ[&quot;SNOWFLAKE_CONN&quot;])<br>schema = engine.execute(&quot;select current_schema()&quot;).scalar()<br><br>@dag(schedule=&quot;@daily&quot;)<br>def orders_pipeline():<br>    ...</pre><p>Three problems here, each running 30 times per minute. Variable.get at the top level opens a connection to the Airflow metadata database on every parse. The create_engine call opens a connection to Snowflake on every parse. Heavy imports like pandas add significant load time to every parse cycle.</p><p>The fix is to move everything that is not DAG structure definition inside the tasks:</p><pre># dags/orders_pipeline.py  -- this is cheap to parse<br><br>from airflow.decorators import dag, task<br>from datetime import datetime<br><br>TABLES = [&quot;orders&quot;, &quot;customers&quot;, &quot;products&quot;]  # plain Python, free to parse<br><br>@dag(schedule=&quot;@daily&quot;, start_date=datetime(2024, 1, 1), catchup=False)<br>def orders_pipeline():<br>    @task()<br>    def get_config() -&gt; dict:<br>        from airflow.models import Variable  # import inside task<br>        return Variable.get(&quot;pipeline_config&quot;, deserialize_json=True)<br><br>    @task()<br>    def process_table(table: str, config: dict):<br>        import pandas as pd  # import inside task<br>        from sqlalchemy import create_engine<br>        import os<br>        engine = create_engine(os.environ[&quot;SNOWFLAKE_CONN&quot;])<br>        # ... processing logic<br><br>    config = get_config()<br><br>    for table in TABLES:<br>        process_table(table, config)<br><br>orders_pipeline()</pre><p>Using Airflow Variables in top-level code creates a connection to the metadata database of Airflow to fetch the value, which can slow down parsing and place extra load on the database. Moving it inside a task means it runs exactly once, when the task executes, not on every scheduler heartbeat.</p><p>A single complex DAG incurred a parsing time of 5 seconds before optimization. After applying performance enhancements, this parsing time was significantly reduced. At 30-second parse intervals, a 5-second parse time means the scheduler spends 17% of its time just reading one DAG file.</p><h3>Pattern 2: ShortCircuitOperator to skip work that does not need to happen</h3><p>A common pipeline pattern is to check whether new data exists before running a full load. Without ShortCircuitOperator, the naive implementation runs the check and then has no way to stop the downstream tasks from executing:</p><pre>@task()<br>def check_new_data() -&gt; bool:<br>    count = query_snowflake(&quot;select count(*) from raw.orders_staging&quot;)<br>    return count &gt; 0<br><br>@task()<br>def transform():<br>    ...  # runs regardless of check_new_data result<br><br>@task()<br>def load():<br>    ...  # runs regardless<br><br>check = check_new_data()<br>t = transform()<br>l = load()<br>check &gt;&gt; t &gt;&gt; l  # check result is ignored</pre><p>ShortCircuitOperator turns the boolean return value into flow control. When the callable returns False, all downstream tasks are skipped:</p><pre>from airflow.operators.python import ShortCircuitOperator<br>from airflow.decorators import dag, task<br>from datetime import datetime<br><br>@dag(schedule=&quot;@hourly&quot;, start_date=datetime(2024, 1, 1), catchup=False)<br>def orders_pipeline():<br>    check_new_data = ShortCircuitOperator(<br>        task_id=&quot;check_new_data&quot;,<br>        python_callable=lambda: query_snowflake(<br>            &quot;select count(*) &gt; 0 from raw.orders_staging&quot;<br>        ),<br>    )<br>    @task()<br>    def transform():<br>        ...<br>    @task()<br>    def load():<br>        ...<br>    @task()<br>    def notify():<br>        ...<br>    check_new_data &gt;&gt; transform() &gt;&gt; load() &gt;&gt; notify()<br>orders_pipeline()</pre><p>If check_new_data returns False, transform, load, and notify are all marked as skipped. No Snowflake compute is consumed, no downstream systems are called, and the DAG run shows as successful rather than failed.</p><p>ShortCircuitOperator accepts a Python function that returns True or False. If True is returned, the DAG continues. If False is returned, all downstream tasks are skipped. This is different from BranchPythonOperator, which chooses between multiple paths. ShortCircuitOperator is for the simpler case: either continue or stop entirely.</p><p>A real-world case where this matters: an hourly pipeline that processes vendor files. Files arrive once a day. Without ShortCircuitOperator, the pipeline spins up a warehouse and runs transforms 23 times a day against an empty staging table.</p><h3>Pattern 3: BranchPythonOperator for conditional paths</h3><p>When a pipeline needs to take different paths based on a condition, rather than simply continuing or stopping, BranchPythonOperator handles the routing. The callable returns the task_id of the next task to execute:</p><pre>from airflow.operators.python import BranchPythonOperator<br>from airflow.decorators import dag, task<br>from airflow.utils.trigger_rule import TriggerRule<br>from datetime import datetime<br><br>@dag(schedule=&quot;@daily&quot;, start_date=datetime(2024, 1, 1), catchup=False)<br>def orders_pipeline():<br>    def route_by_volume() -&gt; str:<br>        row_count = query_snowflake(&quot;select count(*) from raw.orders_staging&quot;)<br>        if row_count &gt; 1_000_000:<br>            return &quot;process_large_batch&quot;<br>        return &quot;process_standard_batch&quot;<br><br>    branch = BranchPythonOperator(<br>        task_id=&quot;route_by_volume&quot;,<br>        python_callable=route_by_volume,<br>    )<br><br>    @task(task_id=&quot;process_large_batch&quot;)<br>    def process_large_batch():<br>        # use larger warehouse, parallel processing<br>        ...<br><br>    @task(task_id=&quot;process_standard_batch&quot;)<br>    def process_standard_batch():<br>        # standard warehouse<br>        ...<br><br>    @task(trigger_rule=TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS)<br>    def notify():<br>        ...<br><br>    large = process_large_batch()<br>    standard = process_standard_batch()<br>    branch &gt;&gt; [large, standard]<br>    [large, standard] &gt;&gt; notify()<br>orders_pipeline()</pre><p>The trigger_rule on notify is important. By default, a task only runs if all upstream tasks succeeded. Since BranchPythonOperatoralways skips one path, notify would be skipped too without an explicit trigger rule. NONE_FAILED_MIN_ONE_SUCCESSmeans: run if at least one upstream task succeeded and none failed.</p><p>The BranchPythonOperator executes a task created using a Python function. This task returns the task ID of the next task to be run. Based on that, the next task is executed, and hence the subsequent path to be followed in the pipeline is decided.</p><h3>Pattern 4: dynamic task mapping instead of copy-pasted operators</h3><p>A DAG that loads ten tables often ends up with ten nearly identical task definitions. Each one is a separate entry in the DAG file, parsed individually, and maintained separately when something changes:</p><pre># the pattern to avoid<br>load_orders = SnowflakeOperator(task_id=&quot;load_orders&quot;, sql=&quot;call load_table(&#39;orders&#39;)&quot;, ...)<br>load_customers = SnowflakeOperator(task_id=&quot;load_customers&quot;, sql=&quot;call load_table(&#39;customers&#39;)&quot;, ...)<br>load_products = SnowflakeOperator(task_id=&quot;load_products&quot;, sql=&quot;call load_table(&#39;products&#39;)&quot;, ...)<br># ... seven more</pre><p>Dynamic task mapping generates these at runtime from a list, keeping the DAG file short and the task graph consistent:</p><pre>from airflow.decorators import dag, task<br>from airflow.providers.snowflake.operators.snowflake import SnowflakeOperator<br>from datetime import datetime<br><br>TABLES = [&quot;orders&quot;, &quot;customers&quot;, &quot;products&quot;, &quot;inventory&quot;, &quot;returns&quot;, &quot;events&quot;]<br><br>@dag(schedule=&quot;@daily&quot;, start_date=datetime(2024, 1, 1), catchup=False)<br>def load_pipeline():<br>    @task()<br>    def get_tables_to_load() -&gt; list[str]:<br>        # can also query a config table or API<br>        return TABLES<br>    load = SnowflakeOperator.partial(<br>        task_id=&quot;load_table&quot;,<br>        snowflake_conn_id=&quot;snowflake_prod&quot;,<br>    ).expand(<br>        sql=get_tables_to_load().map(lambda t: f&quot;call load_table(&#39;{t}&#39;)&quot;)<br>    )<br><br>load_pipeline()</pre><p>The real power of dynamic task mapping is when the list is not static. If the tables to load are read from a config table or an API, the DAG automatically adjusts on each run without any code changes:</p><pre>@task()<br>def get_active_tables() -&gt; list[str]:<br>    return query_snowflake(&quot;&quot;&quot;<br>        select table_name<br>        from pipeline_config.tables<br>        where is_active = true<br>        order by load_priority<br>    &quot;&quot;&quot;)<br><br>SnowflakeOperator.partial(<br>    task_id=&quot;load_table&quot;,<br>    snowflake_conn_id=&quot;snowflake_prod&quot;,<br>).expand(<br>    sql=get_active_tables().map(lambda t: f&quot;call load_table(&#39;{t}&#39;)&quot;)<br>)</pre><p>Adding a new table to the pipeline now means inserting a row into a config table, not editing a DAG file.</p><h3>Pattern 5: deferred operators for long-running waits</h3><p>A common pattern in data pipelines is triggering an external job and then waiting for it to complete: a Snowflake stored procedure, a dbt Cloud run, a Spark job. The naive implementation uses a sensor that polls on a fixed interval and holds a worker slot for the entire wait:</p><pre>from airflow.sensors.python import PythonSensor<br><br># this holds a worker slot for 2 hours while the job runs<br>wait_for_job = PythonSensor(<br>    task_id=&quot;wait_for_snowflake_job&quot;,<br>    python_callable=check_job_status,<br>    poke_interval=60,  # check every minute<br>    timeout=7200,<br>)</pre><p>With ten pipelines each waiting on a 2-hour job, ten worker slots are permanently occupied doing nothing but checking a status endpoint once a minute. Under CeleryExecutor or KubernetesExecutor this is expensive. Under LocalExecutor it can exhaust all available workers.</p><p>Deferred operators solve this by releasing the worker slot during the wait and resuming only when the job completes:</p><pre>from airflow.providers.snowflake.operators.snowflake import SnowflakeOperator<br><br># SnowflakeOperator with deferrable=True releases the worker while waiting<br>run_procedure = SnowflakeOperator(<br>    task_id=&quot;run_stored_procedure&quot;,<br>    snowflake_conn_id=&quot;snowflake_prod&quot;,<br>    sql=&quot;call transform_orders()&quot;,<br>    deferrable=True,  # releases worker slot during execution<br>)</pre><p>Not all operators support deferrable mode, but the major cloud providers and warehouse connectors do as of Airflow 2.2+. For operators that do not, wrapping the wait in an asyncio-based sensor achieves the same result:</p><pre>from airflow.sensors.base import BaseSensorOperator<br>from airflow.triggers.temporal import TimeDeltaTrigger<br>from datetime import timedelta<br><br>class DeferrableJobSensor(BaseSensorOperator):<br>    def execute(self, context):<br>        if not self.poke(context):<br>            self.defer(<br>                trigger=TimeDeltaTrigger(timedelta(seconds=60)),<br>                method_name=&quot;execute&quot;,<br>            )<br><br>    def poke(self, context) -&gt; bool:<br>        return check_job_status(self.job_id)</pre><p>The worker starts the sensor, checks the status, and if the job is not done, defers itself. The triggerer process takes over and wakes the task when the timer fires. The worker slot is free in the meantime.</p><h3>Where these patterns overlap</h3><p>The five patterns here address different layers of the same problem: Airflow spending resources on work that either does not need to happen or could happen more efficiently.</p><p>Top-level code optimization reduces scheduler overhead before any task runs. ShortCircuitOperator and BranchPythonOperatorprevent unnecessary task execution during a run. Dynamic task mapping keeps DAG files maintainable as pipelines grow. Deferred operators reduce worker slot consumption during waits.</p><p>None of these require changing infrastructure or tuning airflow.cfg. They are DAG design decisions, which means they are also the easiest category of optimization to apply to an existing codebase one DAG at a time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d49fd0d5ddf5" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/writing-airflow-dags-that-dont-fight-you-in-production-d49fd0d5ddf5">Writing Airflow DAGs that don’t fight you in production</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[6 tools that keep your dbt project clean]]></title>
            <link>https://blog.dataengineerthings.org/6-tools-that-keep-your-dbt-project-clean-650c89535e34?source=rss-27db4b2773fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/650c89535e34</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[code-quality]]></category>
            <dc:creator><![CDATA[Joachim Hodana]]></dc:creator>
            <pubDate>Sun, 07 Jun 2026 14:52:01 GMT</pubDate>
            <atom:updated>2026-06-07T14:52:01.701Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>There is more to dbt linting than SQLFluff</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q16c9aObf6mKle-fjLzQlA.png" /></figure><p><em>Disclosure: dbt-doctor is a tool I built and maintain. All other tools in this article are independent open-source projects.</em></p><p>As a dbt project grows, so does the number of things that can quietly go wrong. A model without tests, a source referenced the wrong way, SQL that nobody on the team formats consistently, a mart exposed to dashboards with no declared contract. None of these break a dbt run. They accumulate silently until someone spends half a day debugging something that a linter would have caught in seconds.</p><p>The dbt ecosystem has several tools for this. They solve overlapping but distinct problems, and understanding what each one actually does saves the time of setting up the wrong one.</p><h3>SQLFluff</h3><p>SQLFluff is the de facto standard for SQL linting. Its architecture is built on a parsing engine that converts SQL into an abstract syntax tree, enabling deep, context-aware analysis of SQL structures rather than simple regex-based checks. It handles Jinja templating through a dbt-specific templater, which means it understands {{ ref() }}, {{ source() }}, and macros without choking on them.</p><p>What it catches: keyword capitalization, indentation, trailing commas, line length, unused CTEs, implicit column references. Things that make SQL hard to read across a team.</p><p>Setup requires a .sqlfluff config file at the project root:</p><pre>[sqlfluff]<br>dialect = snowflake<br>templater = dbt<br>max_line_length = 120<br><br>[sqlfluff:templater:dbt]<br>project_dir = .<br>profiles_dir = ~/.dbt</pre><p>The key limitation is that SQLFluff needs to compile the project to handle Jinja, which means it needs a working profiles.yml and database credentials. Implementation at scale in dbt environments can present challenges around initial setup for large, established codebases and performance considerations for extensive projects.</p><p>For teams on dbt Cloud, SQLFluff is available directly in the Studio IDE and can be configured to lint changed SQL files compared to the last deferred production state as part of CI.</p><pre>pip install sqlfluff sqlfluff-templater-dbt<br>sqlfluff lint models/<br>sqlfluff fix models/ --force</pre><p><a href="https://www.sqlfluff.com/">SQLFluff</a></p><h3>dbt-checkpoint</h3><p>dbt-checkpoint is a collection of pre-commit hooks for dbt projects. Tests are written in Python and configured via .pre-commit-config.yaml. Provided the dbt artifacts have already been generated, it does not need access to the underlying database. The hooks execute when a new commit is made, making it designed to run only as part of pre-commit.</p><p>What it catches: models without descriptions, columns without tests, sources without freshness, YML files that reference models which do not exist.</p><pre># .pre-commit-config.yaml<br>repos:<br>  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint<br>    rev: v2.0.0<br>    hooks:<br>      - id: check-model-has-tests<br>        args: [&quot;--test-cnt&quot;, &quot;1&quot;]<br>      - id: check-model-has-description<br>      - id: check-source-has-freshness<br>      - id: check-script-ref-and-source</pre><p>The pre-commit design is both its strength and its limitation. It runs automatically before every commit, which means governance is enforced at the point where it costs the least to fix something. But it cannot run in dbt Cloud, since dbt Cloud only executes dbt commands, not pre-commit hooks.</p><p><a href="https://github.com/dbt-checkpoint/dbt-checkpoint">GitHub - dbt-checkpoint/dbt-checkpoint: :fishing_pole_and_fish: List of `pre-commit` hooks to ensure the quality of your `dbt` projects.</a></p><h3>dbt-project-evaluator</h3><p>dbt-project-evaluator automates critical checks across five key areas: modelling, testing, documentation, structure, and performance. Unlike the other tools here, it is a dbt package, which means it installs inside the project and runs as a set of dbt models that materialize findings into the warehouse.</p><p>Tests are written in SQL files using a combination of Jinja and SQL, configured via dbt_project.yml and seed files. It requires a connection to the underlying database and is designed to run both in CI and during active development.</p><p>What it catches: sources not referenced by any staging model, staging models joining directly to other staging models, models with no primary key test, DAG structures that violate the expected layer hierarchy.</p><pre># packages.yml<br>packages:<br>  - package: dbt-labs/dbt_project_evaluator<br>    version: 0.8.0</pre><pre>dbt deps<br>dbt build --select package:dbt_project_evaluator</pre><p>The warehouse dependency is the significant trade-off. Running structural checks requires a live database connection, which means it does not work offline or in environments without credentials. dbt-project-evaluator powers the recommendations seen in dbt Explorer, so teams on dbt Cloud get a subset of its checks without any additional setup.</p><p><a href="https://github.com/dbt-labs/dbt-project-evaluator">GitHub - dbt-labs/dbt-project-evaluator: This package contains macros and models to find DAG issues automatically</a></p><h3>dbt-score</h3><p>dbt-score, maintained by Picnic Technologies, is a Python-based CLI linter that reads your manifest.json and assigns each model a score from 0 to 10 based on metadata quality: missing descriptions, absent owners, undocumented columns, models without tests. It requires no database connection and no dbt run, just a manifest.json.</p><p>The scoring approach is the differentiating design choice. Rather than blocking a PR with a hard failure, dbt-score gives each model a grade, making it possible to track improvement over time and prioritize which models need attention first.</p><pre>pip install dbt-score<br>dbt parse  # generates manifest.json<br>dbt-score lint</pre><p>Output looks like:</p><pre>stg_orders        A (10/10)<br>stg_customers     B (7.5/10)<br>fct_orders        C (5/10)   ← missing description, 2 untested columns<br>dim_products      F (2/10)   ← no tests, no description, no owner</pre><p>Custom rules are fully supported, which makes it extensible for teams with project-specific metadata requirements.</p><p><a href="https://dbt-score.picnic.tech/">dbt-score</a></p><h3>dbt-bouncer</h3><p>dbt-bouncer contains all the tests that can be run by dbt-checkpoint, dbt-project-evaluator, and dbt-score in a single tool, operating on dbt artifacts rather than source files or a live database. It reads the compiled manifest.json and catalog.json and applies configurable rules.</p><p>The positioning is explicitly as a superset: one config file, one command, covers the governance surface that previously required three separate tools. The trade-off is that it requires running dbt parse or dbt compile first to generate the artifacts, and it cannot catch issues that only manifest at query execution time.</p><pre># dbt-bouncer.yml<br>manifest-checks:<br>  - name: check_model_has_tests_by_name<br>    min_count: 1<br>  - name: check_model_description_populated<br>  - name: check_source_has_freshness_populated</pre><p><a href="https://github.com/godatadriven/dbt-bouncer">GitHub - godatadriven/dbt-bouncer: Configure and enforce conventions for your dbt project.</a></p><h3>dbt-doctor</h3><p>The newest entry in this space takes a different approach. dbt-doctor runs offline, requires no compilation, no warehouse connection, and no artifacts. This one is different as it does not use Python at all (you need Node.js). One command from the project root:</p><pre>npx dbt-doctor@latest</pre><p>It produces a health score from 0 to 100 with findings grouped across SQL quality (via SQLFluff if installed), documentation, naming conventions, and architecture issues. The checks cover the issues that most consistently produce silent production incidents: missing contracts on mart models, sources referenced with ref() instead of source(), incremental models without on_schema_change, missing unique_key, undocumented models, sources without freshness.</p><p>The score format is particularly useful when communicating project health to non-technical stakeholders. A number between 0 and 100 with a label — Great, Needs work, or Critical — is easier to present to a business audience than a list of YAML violations. It gives engineering conversations about technical debt a shape that product managers and engineering leads can engage with without reading linting output.</p><p>It also ships as a GitHub Action for CI integration, posting findings as a sticky comment on every PR:</p><pre>- name: dbt Doctor<br>  uses: joachimhodana/dbt-doctor@dbt-doctor@0.2.0<br>  with:<br>    diff: main<br>    github-token: ${{ secrets.GITHUB_TOKEN }}</pre><p>The tool is early. The check library is combination of dbt-score, dbt-checkpoint and other (see <a href="https://dbt-doctor.joachimhodana.com/docs/tool-parity">tool parity</a> in docs). The repo is at <a href="https://github.com/joachimhodana/dbt-doctor">github.com/joachimhodana/dbt-doctor</a> — worth starring if the direction is interesting.</p><p><a href="https://dbt-doctor.joachimhodana.com/">dbt Doctor</a></p><h3>How they fit together</h3><p>None of these tools are mutually exclusive, and most production dbt projects benefit from combining them.</p><p>SQLFluff handles SQL style and syntax but needs database access for Jinja compilation. dbt-checkpoint runs as a pre-commit hook without a database, catching governance issues before they are committed. dbt-project-evaluator goes deepest on structural DAG issues but requires a live warehouse connection. dbt-score reads the manifest and grades each model on metadata quality without touching the database. dbt-bouncer consolidates most of what the previous three do into artifact-based validation. dbt-doctor is the only one that needs neither a database nor generated artifacts, making it the fastest option for a first pass on an unfamiliar project — and the most readable option when presenting project health outside the engineering team.</p><p>A practical starting point: dbt-checkpoint for pre-commit governance, dbt-project-evaluator and dbt-score as complementary layers as the team builds confidence. Add dbt-bouncer when consolidation starts to matter. Run dbt-doctor when onboarding a new project or making the case for technical debt work, for generic scoring for non-technical stakeholders and initial CI run.</p><ul><li><a href="https://joachimhodana.com/">Joachim Hodana - Software &amp; Data Engineer</a></li><li><a href="https://betterdataengineer.joachimhodana.com/">Better Data Engineer</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=650c89535e34" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/6-tools-that-keep-your-dbt-project-clean-650c89535e34">6 tools that keep your dbt project clean</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 dbt mistakes I see in every startup]]></title>
            <link>https://blog.dataengineerthings.org/5-dbt-mistakes-i-see-in-every-startup-c638ed24b95d?source=rss-27db4b2773fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/c638ed24b95d</guid>
            <category><![CDATA[data-build-tool]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Joachim Hodana]]></dc:creator>
            <pubDate>Thu, 04 Jun 2026 03:50:32 GMT</pubDate>
            <atom:updated>2026-06-04T03:50:32.162Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*96lNldySqiRJA_OczioY0Q.png" /></figure><p>There is a pattern to how dbt projects fail. Not the obvious failures — a model that breaks, a test that catches a null — but the slower kind, where the project works fine for six months and then becomes progressively harder to maintain, trust, or run without burning Snowflake credits. The mistakes that cause this are not complicated. They show up in almost every early-stage data team, and they are all fixable.</p><h3>1. Running a full dbt run in CI on every pull request</h3><p>The most common CI setup for a dbt project looks like this in a GitHub Actions workflow:</p><pre># .github/workflows/ci.yml<br>jobs:<br>  dbt_ci:<br>    runs-on: ubuntu-latest<br>    steps:<br>      - name: Run dbt<br>        run: dbt run &amp;&amp; dbt test</pre><p>This runs every model in the project, rebuilds everything from scratch in a temporary schema, and runs every test. On a project with 50 models, this takes several minutes and costs warehouse credits on every PR. On a project with 200 models, it becomes a 20-minute block on the development workflow and eventually people stop waiting for it.</p><p>The fix is slim CI using --state and --defer. Instead of rebuilding everything, dbt compares the current branch&#39;s manifest.jsonagainst the last successful production run and only builds what actually changed:</p><pre># .github/workflows/ci.yml<br>jobs:<br>  dbt_ci:<br>    runs-on: ubuntu-latest<br>    steps:<br>      - name: Download production manifest<br>        run: |<br>          # fetch manifest.json from last prod run<br>          # e.g. from S3, GCS, or dbt Cloud API<br>          aws s3 cp s3://your-bucket/dbt/prod/manifest.json ./prod/manifest.json<br>      - name: Run slim CI<br>        run: |<br>          dbt run \<br>            --select state:modified+ \<br>            --defer \<br>            --state ./prod/ \<br>            --target ci<br>          dbt test \<br>            --select state:modified+ \<br>            --defer \<br>            --state ./prod/ \<br>            --target ci</pre><p>state:modified+ selects only models that changed in this PR, plus all their downstream dependencies. --defer tells dbt to resolve unmodified upstream ref() calls using production tables instead of rebuilding them in the CI schema. A PR that touches one staging model and one mart now runs two models and their tests, not the whole project.</p><p>dbt defer lets you run and test only the models you changed, while dbt resolves upstream ref() calls using artifacts from another environment, usually production. It works by pointing dbt at a production manifest.json, so ref() calls for upstream models use existing production relations instead of rebuilding them.</p><p>The production manifest.json needs to be stored somewhere accessible from CI. The simplest approach is uploading it to S3 or GCS at the end of every successful production run:</p><pre># at the end of the production dbt job<br>aws s3 cp ./target/manifest.json s3://your-bucket/dbt/prod/manifest.json</pre><p>On dbt Cloud, this is built-in: CI jobs can defer to the production environment without any manual artifact management.</p><h3>2. No data contracts on mart models</h3><p>Most dbt projects have tests. not_null, unique, accepted_values — the standard schema.yml setup. These tests catch data quality issues after a model builds. What they do not catch is a structural change to a model that breaks downstream consumers before anyone notices.</p><p>A common scenario: an analytics engineer renames customer_id to id in dim_customers to align with a naming convention. The model builds. The tests pass. Three dashboards that query dim_customers.customer_id break silently, and the data team finds out two days later when someone notices wrong numbers in a report.</p><p>One team had a critical model that fed revenue reports. A tiny upstream change, a decimal precision tweak, caused a 7% discrepancy in reported numbers. Nobody noticed for two weeks. A simple dbt contract would have caught it during the first build.</p><p>dbt model contracts, available since v1.5, enforce schema at build time. If the model’s output does not match the declared contract, the build fails before anything is written to the warehouse:</p><pre># models/marts/dim_customers.yml<br>models:<br>  - name: dim_customers<br>    config:<br>      contract:<br>        enforced: true<br>    columns:<br>      - name: customer_id<br>        data_type: varchar<br>        constraints:<br>          - type: not_null<br>          - type: unique<br>      - name: email<br>        data_type: varchar<br>      - name: created_at<br>        data_type: timestamp<br>      - name: country_code<br>        data_type: varchar</pre><p>With enforced: true, if someone renames customer_id to id in the SQL, dbt run fails immediately with a contract violation error. The broken change never reaches production.</p><p>A model’s contract defines the shape of the returned dataset. If the model’s logic or input data does not conform to that shape, the model does not build. Data tests are a more flexible mechanism for validating the content of your model after it is built.</p><p>Contracts are not necessary on every model. Staging models change frequently during development and adding contracts too early creates maintenance overhead. The right place to enforce them is on mart models and any model consumed by dashboards, APIs, or other teams — the models where a structural change has an immediate downstream impact.</p><h3>3. Incremental models without on_schema_change</h3><p>Incremental models are how most teams handle large fact tables in dbt. The standard config:</p><pre>{{ config(<br>    materialized=&#39;incremental&#39;,<br>    unique_key=&#39;event_id&#39;<br>) }}<br><br>select<br>    event_id,<br>    user_id,<br>    event_type,<br>    created_at<br>from {{ ref(&#39;stg_events&#39;) }}<br>{% if is_incremental() %}<br>    where created_at &gt; (select max(created_at) from {{ this }})<br>{% endif %}</pre><p>This works until the source schema changes. A new column appears in stg_events. The incremental model runs, the SELECT * or explicit column list now includes the new column, but the target table does not have that column. The merge or insert silently drops the new column, or fails with a schema mismatch error, depending on the database.</p><p>The fix is on_schema_change, which tells dbt what to do when the model&#39;s output schema diverges from the existing table:</p><pre>{{ config(<br>    materialized=&#39;incremental&#39;,<br>    unique_key=&#39;event_id&#39;,<br>    on_schema_change=&#39;append_new_columns&#39;<br>) }}</pre><p>The three meaningful options are fail (stop immediately, which is the right default for contracted models), append_new_columns (add new columns without touching existing ones), and sync_all_columns (add new columns and remove deleted ones). For most fact tables, append_new_columns is the right choice: it handles upstream additions without risk of accidentally dropping columns that downstream models depend on.</p><p>Without on_schema_change, schema drift accumulates silently. The incremental model runs successfully for months, and then one day a downstream query fails because a column that was supposed to be there is missing.</p><h3>4. Using ref() in sources instead of source()</h3><p>This one is subtle but has consequences. Early dbt projects often mix ref() and source() in ways that blur the line between raw source data and transformed models:</p><pre>-- models/staging/stg_orders.sql<br>-- wrong: using ref() to reference a raw table loaded by an external tool<br>select * from {{ ref(&#39;raw_orders&#39;) }}</pre><p>When raw_orders is a table loaded by Fivetran, Airbyte, or a custom ingestion pipeline, it should be declared as a source, not referenced as a model:</p><pre># models/sources.yml<br>sources:<br>  - name: raw<br>    schema: raw<br>    tables:<br>      - name: orders<br>        loaded_at_field: _fivetran_synced<br>        freshness:<br>          warn_after: {count: 6, period: hour}<br>          error_after: {count: 24, period: hour}</pre><pre>-- models/staging/stg_orders.sql<br>-- correct<br>select * from {{ source(&#39;raw&#39;, &#39;orders&#39;) }}</pre><p>The practical consequences of getting this wrong are significant. dbt source freshness only checks tables declared as sources. If raw tables are referenced via ref(), there is no freshness monitoring, and a pipeline where ingestion stopped 12 hours ago will run without any warning until someone notices stale data in a dashboard. Source lineage in dbt docs also breaks: the dependency graph shows model-to-model edges instead of the correct source-to-model edges, which makes it impossible to trace data back to its origin.</p><h3>5. No separation between dev, staging, and prod schemas</h3><p>The default dbt setup writes everything to a single schema. In development, engineers run models that overwrite production tables because profiles.yml is misconfigured or everyone uses the same target:</p><pre># profiles.yml -- the dangerous version<br>my_project:<br>  target: prod  # everyone defaults to prod<br>  outputs:<br>    prod:<br>      schema: analytics</pre><p>Every dbt run by any engineer on any branch writes to analytics.dim_customers, analytics.fct_orders, and so on. A half-finished model with broken logic overwrites the table that the BI tool queries.</p><p>The correct setup uses per-engineer schemas in development and a shared schema only in production:</p><pre># profiles.yml<br>my_project:<br>  target: dev<br>  outputs:<br>    dev:<br>      type: snowflake<br>      schema: &quot;dbt_{{ env_var(&#39;DBT_USER&#39;, &#39;dev&#39;) }}&quot;  # dbt_alice, dbt_bob, etc.<br>      warehouse: DEV_WH<br>    ci:<br>      type: snowflake<br>      schema: &quot;dbt_ci_{{ env_var(&#39;PR_NUMBER&#39;, &#39;local&#39;) }}&quot;  # dbt_ci_123<br>      warehouse: CI_WH<br>    prod:<br>      type: snowflake<br>      schema: analytics<br>      warehouse: TRANSFORM_WH</pre><p>Each engineer writes to their own isolated schema in dev. CI runs write to a per-PR schema that is cleaned up after the PR merges. Production is the only environment that writes to analytics.</p><p>This also means the --target flag must be set explicitly in CI and production Airflow jobs, not left to the default:</p><pre># in CI<br>dbt run --select state:modified+ --defer --state ./prod/ --target ci<br><br># in Airflow production job<br>dbt run --target prod</pre><p>Without this separation, development and production share the same tables and the same warehouse. A long dbt run --full-refresh by a developer during business hours contends with dashboard queries on the production warehouse, and any schema mistake in dev lands directly in production data.</p><h3>The common thread</h3><p>All five mistakes share the same root cause: a setup that works for one engineer on a small project does not scale to a team working on a production data product. Full CI rebuilds, no contracts, missing on_schema_change, confused source declarations, and shared dev/prod schemas are all reasonable shortcuts when the project is new. They become the source of the most painful debugging sessions as the project grows.</p><p>None of these require a large refactor. Each one is a configuration change or a small addition to schema.yml. The earlier they are fixed, the less technical debt accumulates around them.</p><ul><li><a href="https://joachimhodana.com/">Joachim Hodana - Software &amp; Data Engineer</a></li><li><a href="https://betterdataengineer.joachimhodana.com/">Better Data Engineer</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c638ed24b95d" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/5-dbt-mistakes-i-see-in-every-startup-c638ed24b95d">5 dbt mistakes I see in every startup</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DuckDB for data engineers]]></title>
            <link>https://blog.dataengineerthings.org/duckdb-for-data-engineers-8e885367fcd1?source=rss-27db4b2773fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e885367fcd1</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[duckdb]]></category>
            <dc:creator><![CDATA[Joachim Hodana]]></dc:creator>
            <pubDate>Tue, 19 May 2026 18:54:55 GMT</pubDate>
            <atom:updated>2026-05-19T18:54:55.857Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>You are already processing files, testing SQL, and validating data locally. You are just doing it with more infrastructure than necessary.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4HlwyhSfmizTqSZHfGL8qw.png" /></figure><p>There is a category of problems in data engineering that are too small for Spark, too slow for Pandas, and too annoying to spin up a Snowflake warehouse for. Reading a 2GB Parquet file to check its schema. Running a quick aggregation on a CSV dump from a vendor. Testing a complex SQL transformation locally before deploying it to production. For years, the answer to all of these was “just use Pandas” or “just connect to the warehouse.” DuckDB is a better answer to most of them.</p><p>DuckDB is an in-process analytical database. It runs inside the Python process, requires no server, no Docker container, and no configuration. It is columnar, vectorized, and fast enough that on a laptop it outperforms Pandas on most analytical workloads above a few hundred megabytes. It speaks SQL, reads Parquet and CSV natively, and integrates with Pandas and Polars DataFrames without copying data. The installation is a single pip install.</p><p>This article covers four use cases where DuckDB earns its place in a data engineering workflow.</p><h3>Use case 1: replacing Pandas for local file processing</h3><p>The standard pattern for processing a large CSV or Parquet file in Python looks something like this:</p><pre>import pandas as pd<br><br>df = pd.read_parquet(&quot;orders.parquet&quot;)<br>result = (<br>    df[df[&quot;status&quot;] == &quot;completed&quot;]<br>    .groupby(&quot;customer_id&quot;)<br>    .agg(total_spend=(&quot;amount&quot;, &quot;sum&quot;), order_count=(&quot;order_id&quot;, &quot;count&quot;))<br>    .sort_values(&quot;total_spend&quot;, ascending=False)<br>    .head(100)<br>)</pre><p>This works until the file is large enough that read_parquet loads more data into memory than the machine has available. Pandas reads the entire file before any filtering happens. A 4GB Parquet file with 10 columns, where only 2 columns are needed, still loads all 10 into memory.</p><p>DuckDB does not work this way. It pushes predicates and column selection down into the file read, so only the data that is actually needed gets loaded:</p><pre>import duckdb<br><br>result = duckdb.sql(&quot;&quot;&quot;<br>    select<br>        customer_id,<br>        sum(amount)    as total_spend,<br>        count(order_id) as order_count<br>    from read_parquet(&#39;orders.parquet&#39;)<br>    where status = &#39;completed&#39;<br>    group by customer_id<br>    order by total_spend desc<br>    limit 100<br>&quot;&quot;&quot;).df()</pre><p>The .df() at the end converts the result to a Pandas DataFrame. For files that fit comfortably in memory, the performance difference is modest. For files that do not, DuckDB continues to work while Pandas runs out of memory or starts swapping to disk.</p><p>DuckDB also reads multiple files in a single query using glob patterns, which is useful for partitioned datasets:</p><pre>result = duckdb.sql(&quot;&quot;&quot;<br>    select<br>        date_trunc(&#39;month&#39;, order_date) as month,<br>        sum(amount)                     as revenue<br>    from read_parquet(&#39;data/orders/year=2024/month=*/*.parquet&#39;)<br>    group by 1<br>    order by 1<br>&quot;&quot;&quot;).df()</pre><p>No loop over files, no pd.concat, no intermediate DataFrames.</p><h3>Use case 2: querying DataFrames with SQL</h3><p>Sometimes a DataFrame already exists in memory, either from an upstream transformation or from a source that Pandas handles well. DuckDB can query it directly without any data transfer:</p><pre>import pandas as pd<br>import duckdb<br><br>orders = pd.read_csv(&quot;orders.csv&quot;)<br>customers = pd.read_csv(&quot;customers.csv&quot;)<br>result = duckdb.sql(&quot;&quot;&quot;<br>    select<br>        c.country,<br>        count(distinct o.customer_id)   as customer_count,<br>        sum(o.amount)                   as total_revenue,<br>        avg(o.amount)                   as avg_order_value<br>    from orders o<br>    join customers c on o.customer_id = c.id<br>    where o.status = &#39;completed&#39;<br>      and o.order_date &gt;= &#39;2024-01-01&#39;<br>    group by c.country<br>    having sum(o.amount) &gt; 10000<br>    order by total_revenue desc<br>&quot;&quot;&quot;).df()</pre><p>DuckDB references the Python variables orders and customers directly in the SQL query. There is no to_sql() call, no SQLite connection, no temporary table. The DataFrame objects are scanned in place.</p><p>This pattern is useful when a transformation is naturally expressed in SQL but the surrounding code is Python. Rather than translating the SQL logic into a chain of Pandas operations, the SQL runs as-is and the result comes back as a DataFrame.</p><h3>Use case 3: local testing of dbt and warehouse SQL</h3><p>One of the more expensive habits in data engineering is using a Snowflake or BigQuery warehouse as a development environment. Every dbt run during development compiles and executes against the warehouse, which costs money, requires a network connection, and is slower than running locally.</p><p>DuckDB has a dbt adapter that allows dbt models to run locally against DuckDB instead of the production warehouse. The setup requires two things: the adapter and a local profile.</p><pre>pip install dbt-duckdb</pre><pre># profiles.yml<br>my_project:<br>  target: dev<br>  outputs:<br>    dev:<br>      type: duckdb<br>      path: /tmp/dev.duckdb<br>    prod:<br>      type: snowflake<br>      account: &quot;{{ env_var(&#39;SNOWFLAKE_ACCOUNT&#39;) }}&quot;<br>      # ... rest of Snowflake config</pre><p>With this setup, dbt run --target dev executes against a local DuckDB file. Models that use standard SQL and dbt macros run without modification. The developer iterates locally, then runs against Snowflake only when the logic is ready.</p><p>For seeding local development with realistic data, DuckDB reads directly from S3 if credentials are available:</p><pre>import duckdb<br><br>conn = duckdb.connect(&quot;/tmp/dev.duckdb&quot;)<br>conn.execute(&quot;install httpfs; load httpfs;&quot;)<br>conn.execute(&quot;&quot;&quot;<br>    create or replace table raw_orders as<br>    select *<br>    from read_parquet(&#39;s3://data-lake/raw/orders/2024-01/*.parquet&#39;)<br>    limit 100000<br>&quot;&quot;&quot;)</pre><p>This pulls a representative sample from the data lake into the local DuckDB file, which dbt then uses as its source. The workflow becomes: sample once, iterate locally, deploy when ready.</p><h3>Use case 4: lightweight data quality checks</h3><p>Running data quality checks against a warehouse is the same problem as running dbt locally: it requires a live connection, costs money, and is slower than it needs to be for checks that could run against a local file.</p><p>DuckDB makes it practical to run checks against Parquet files directly, before data reaches the warehouse:</p><pre>import duckdb<br>from dataclasses import dataclass<br><br>@dataclass<br>class CheckResult:<br>    check: str<br>    passed: bool<br>    failing_rows: int<br><br>def run_quality_checks(path: str) -&gt; list[CheckResult]:<br>    conn = duckdb.connect()<br>    checks = {<br>        &quot;no_null_order_ids&quot;: &quot;&quot;&quot;<br>            select count(*) from read_parquet(?)<br>            where order_id is null<br>        &quot;&quot;&quot;,<br>        &quot;positive_amounts&quot;: &quot;&quot;&quot;<br>            select count(*) from read_parquet(?)<br>            where amount &lt;= 0<br>        &quot;&quot;&quot;,<br>        &quot;valid_status&quot;: &quot;&quot;&quot;<br>            select count(*) from read_parquet(?)<br>            where status not in (&#39;pending&#39;, &#39;completed&#39;, &#39;cancelled&#39;)<br>        &quot;&quot;&quot;,<br>        &quot;no_future_dates&quot;: &quot;&quot;&quot;<br>            select count(*) from read_parquet(?)<br>            where order_date &gt; current_date<br>        &quot;&quot;&quot;,<br>    }<br>    results = []<br>    for name, sql in checks.items():<br>        failing = conn.execute(sql, [path]).fetchone()[0]<br>        results.append(CheckResult(<br>            check=name,<br>            passed=failing == 0,<br>            failing_rows=failing<br>        ))<br>    return results<br><br>results = run_quality_checks(&quot;data/orders_2024_01.parquet&quot;)<br>for r in results:<br>    status = &quot;PASS&quot; if r.passed else f&quot;FAIL ({r.failing_rows} rows)&quot;<br>    print(f&quot;{r.check}: {status}&quot;)</pre><p>This runs without a warehouse connection, without Spark, and without loading the file into a Pandas DataFrame. For files in the hundreds of megabytes to low gigabytes range, it completes in seconds.</p><p>The same pattern integrates into an Airflow task that runs before the load step, rejecting files that fail quality checks before they reach the warehouse:</p><pre>from airflow.decorators import dag, task<br>from airflow.exceptions import AirflowFailException<br>import duckdb<br><br>@dag(schedule=&quot;@daily&quot;)<br>def orders_pipeline():<br>    @task()<br>    def validate_file(path: str):<br>        conn = duckdb.connect()<br>        failing = conn.execute(&quot;&quot;&quot;<br>            select count(*) from read_parquet(?)<br>            where order_id is null or amount &lt;= 0<br>        &quot;&quot;&quot;, [path]).fetchone()[0]<br>        if failing &gt; 0:<br>            raise AirflowFailException(<br>                f&quot;Quality check failed: {failing} invalid rows in {path}&quot;<br>            )<br><br>    @task()<br>    def load_to_snowflake(path: str):<br>        ...<br>    path = &quot;s3://data-lake/raw/orders/{{ ds }}.parquet&quot;<br>    validate_file(path) &gt;&gt; load_to_snowflake(path)</pre><h3>What DuckDB is not</h3><p>DuckDB is an in-process database. It runs in one process, which means it does not scale horizontally. For datasets that exceed available RAM on a single machine, it will struggle in the same way Pandas does, just at a higher threshold. For genuinely large-scale processing, Spark or warehouse-native compute is still the right tool.</p><p>It is also not a replacement for a production warehouse. There is no access control, no sharing between processes, no query history, and no concurrent writes. The use cases above are all local or single-process workloads: development, validation, exploration, lightweight transformation before a load step.</p><p>Within those boundaries, it is one of the more useful tools to have in a data engineering workflow, precisely because it requires almost no setup and covers a category of problems that previously required either a full warehouse connection or a Pandas workaround.</p><p>If this was useful, I write about practical data engineering patterns at</p><p><a href="https://betterdataengineer.joachimhodana.com/">Better Data Engineer</a></p><p>tools, code, and the things that actually come up in production pipelines. More about my work at</p><p><a href="https://joachimhodana.com/">Joachim Hodana - Software &amp; Data Engineer</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e885367fcd1" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/duckdb-for-data-engineers-8e885367fcd1">DuckDB for data engineers</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Data Engineers Should Care About functools]]></title>
            <link>https://blog.dataengineerthings.org/why-data-engineers-should-care-about-functools-4364fc128861?source=rss-27db4b2773fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/4364fc128861</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[airflow]]></category>
            <dc:creator><![CDATA[Joachim Hodana]]></dc:creator>
            <pubDate>Fri, 24 Apr 2026 20:04:02 GMT</pubDate>
            <atom:updated>2026-04-24T20:04:02.147Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>lru_cache, partial, reduce, and the Airflow trap nobody warns you about.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4dwCe_EI7o9YQZtybiNTWA.png" /></figure><p>There is a pattern that shows up in almost every data pipeline codebase. Somewhere in the DAG definitions there is a helper dictionary that caches API responses. In another file, the same Snowflake operator is copy-pasted twelve times with slightly different parameters. In a third place, a reduce-like aggregation is written manually with a for loop and a running total. None of this is wrong, exactly. It just means the engineer did not know that functools already solved these problems.</p><p>functools is part of the Python standard library and is almost universally underused in data engineering codebases. This article covers the four parts of it that come up most often in pipeline work: lru_cache, partial, cache, and reduce. It also covers one specific place where lru_cache behaves unexpectedly in Airflow, which is worth knowing before running into it in production.</p><h3>lru_cache: memoization without the boilerplate</h3><p>The manual version looks like this:</p><pre>_schema_cache = {}<br><br>def get_table_schema(table_name: str) -&gt; dict:<br>    if table_name not in _schema_cache:<br>        _schema_cache[table_name] = fetch_schema_from_snowflake(table_name)<br>    return _schema_cache[table_name]</pre><p>This works. It also requires maintaining a module-level dictionary, remembering to check it, and writing a new version of this pattern for every function that needs caching. lru_cache replaces all of it:</p><pre>from functools import lru_cache<br><br>@lru_cache(maxsize=128)<br>def get_table_schema(table_name: str) -&gt; dict:<br>    return fetch_schema_from_snowflake(table_name)</pre><p>The decorator handles the cache dictionary, the lookup, and the write. maxsize=128 means the cache holds the 128 most recently used results and evicts older ones when it fills up. Setting maxsize=None disables eviction entirely, which is only safe when the number of distinct inputs is small and known in advance.</p><p>In data engineering, lru_cache earns its place in a few specific situations.</p><p><strong>Schema and metadata lookups.</strong> Fetching a table schema or column list from a warehouse is an I/O call. If the same schema is needed multiple times within a task, caching it avoids repeated round trips:</p><pre>from functools import lru_cache<br>import snowflake.connector<br><br>@lru_cache(maxsize=64)<br>def get_column_names(table: str, schema: str) -&gt; tuple[str, ...]:<br>    conn = snowflake.connector.connect(...)<br>    cursor = conn.cursor()<br>    cursor.execute(f&quot;describe table {schema}.{table}&quot;)<br>    return tuple(row[0].lower() for row in cursor.fetchall())</pre><p>The return type is a tuple rather than a list because lru_cache requires all arguments and return values to be hashable. Lists are not hashable, tuples are.</p><p><strong>Config and mapping lookups.</strong> Any function that reads from a static config file or builds a lookup table from a fixed source is a good candidate:</p><pre>from functools import lru_cache<br>import json<br>from pathlib import Path<br><br>@lru_cache(maxsize=None)<br>def load_country_currency_map() -&gt; dict:<br>    with open(Path(__file__).parent / &quot;config/currencies.json&quot;) as f:<br>        return json.load(f)</pre><p>Because the file does not change during a pipeline run, maxsize=None is appropriate here.</p><h3>The Airflow trap</h3><p>lru_cache on a function inside an Airflow DAG file behaves differently from anywhere else in Python, and the difference matters.</p><p>Airflow parses every DAG file continuously, roughly every 30 seconds by default. Each parse runs the top-level code in the file. A function decorated with lru_cache at the module level will have its cache populated during parsing, in the scheduler process. When a task actually executes on a worker, it runs in a separate process that does not share memory with the scheduler. The cache that was built during parsing is not available at execution time.</p><p>This means code like this does not work the way it appears to:</p><pre># dags/my_dag.py  -- this does not do what it looks like<br><br>from functools import lru_cache<br>from airflow.decorators import dag, task<br><br>@lru_cache(maxsize=32)<br>def get_pipeline_config(env: str) -&gt; dict:<br>    # This runs in the scheduler process during parsing.<br>    # Workers never see this cache.<br>    return fetch_config_from_api(env)<br><br>@dag(schedule=&quot;@daily&quot;)<br>def my_pipeline():<br>    @task()<br>    def process():<br>        config = get_pipeline_config(&quot;production&quot;)  # cache miss every time on workers<br>        ...</pre><p>Beyond cache misses, there is a more serious problem. The scheduler parses DAG files to build the task graph, not to execute business logic. Expensive calls at parse time, cached or not, run every 30 seconds and add load to whatever external system they query. Airflow’s own codebase removed lru_cache from class methods in version 2.9 for related reasons.</p><p>The correct pattern is to move the cached function inside the task, where it runs in the worker process and the cache persists for the lifetime of that task execution:</p><pre>from airflow.decorators import dag, task<br><br>@dag(schedule=&quot;@daily&quot;)<br>def my_pipeline():<br>    @task()<br>    def process():<br>        from functools import lru_cache<br>        @lru_cache(maxsize=32)<br>        def get_pipeline_config(env: str) -&gt; dict:<br>            return fetch_config_from_api(env)<br>        config = get_pipeline_config(&quot;production&quot;)<br>        ...</pre><p>This is less elegant than a module-level decorator, but it reflects how Airflow actually executes code. For configuration that is loaded once and reused across multiple function calls within a single task, this pattern avoids redundant I/O without the footgun of cross-process cache sharing.</p><h3>cache: the simpler version</h3><p>Python 3.9 introduced functools.cache, which is equivalent to lru_cache(maxsize=None). When the input space is bounded and eviction is not needed, it is cleaner:</p><pre>from functools import cache<br><br>@cache<br>def get_exchange_rate(base: str, target: str) -&gt; float:<br>    return fetch_rate_from_api(base, target)</pre><p>The practical difference from lru_cache is that cache never evicts entries. For a function called with a large or unbounded set of distinct inputs, this will grow indefinitely. For functions called with a small, fixed set of inputs, such as a handful of currency pairs or environment names, it is the right choice.</p><h3>partial: operator factories in Airflow</h3><p>partial creates a new function by fixing some arguments of an existing one. The most common data engineering use case is avoiding copy-pasted operators with slightly different parameters.</p><p>A DAG that loads multiple tables from S3 to Snowflake often ends up looking like this:</p><pre>load_orders = S3ToSnowflakeOperator(<br>    task_id=&quot;load_orders&quot;,<br>    s3_bucket=&quot;data-lake&quot;,<br>    snowflake_conn_id=&quot;snowflake_prod&quot;,<br>    stage=&quot;raw_stage&quot;,<br>    table=&quot;orders&quot;,<br>    file_format=&quot;PARQUET&quot;,<br>)<br><br>load_customers = S3ToSnowflakeOperator(<br>    task_id=&quot;load_customers&quot;,<br>    s3_bucket=&quot;data-lake&quot;,<br>    snowflake_conn_id=&quot;snowflake_prod&quot;,<br>    stage=&quot;raw_stage&quot;,<br>    table=&quot;customers&quot;,<br>    file_format=&quot;PARQUET&quot;,<br>)<br><br># ... six more of these</pre><p>partial extracts the shared arguments into a factory:</p><pre>from functools import partial<br>from airflow.providers.snowflake.transfers.s3_to_snowflake import S3ToSnowflakeOperator<br><br>snowflake_loader = partial(<br>    S3ToSnowflakeOperator,<br>    s3_bucket=&quot;data-lake&quot;,<br>    snowflake_conn_id=&quot;snowflake_prod&quot;,<br>    stage=&quot;raw_stage&quot;,<br>    file_format=&quot;PARQUET&quot;,<br>)<br><br>load_orders = snowflake_loader(task_id=&quot;load_orders&quot;, table=&quot;orders&quot;)<br>load_customers = snowflake_loader(task_id=&quot;load_customers&quot;, table=&quot;customers&quot;)<br>load_products = snowflake_loader(task_id=&quot;load_products&quot;, table=&quot;products&quot;)</pre><p>The shared arguments are defined once. Adding a new table is a single line. Changing the connection ID or stage name is a single edit rather than a find-and-replace across the file.</p><p>partial also composes cleanly with loops for dynamically generated tasks:</p><pre>from functools import partial<br><br>TABLES = [&quot;orders&quot;, &quot;customers&quot;, &quot;products&quot;, &quot;inventory&quot;, &quot;events&quot;]<br>snowflake_loader = partial(<br>    S3ToSnowflakeOperator,<br>    s3_bucket=&quot;data-lake&quot;,<br>    snowflake_conn_id=&quot;snowflake_prod&quot;,<br>    stage=&quot;raw_stage&quot;,<br>    file_format=&quot;PARQUET&quot;,<br>)<br><br>load_tasks = [<br>    snowflake_loader(task_id=f&quot;load_{table}&quot;, table=table)<br>    for table in TABLES<br>]</pre><h3>reduce: explicit aggregations</h3><p>reduce applies a function cumulatively to a sequence, collapsing it to a single value. It is less commonly needed than lru_cache or partial, but it shows up in a specific data engineering pattern: building a composed transformation from a list of individual steps.</p><p>The manual version of a multi-step transformation looks like this:</p><pre>df = remove_nulls(df)<br>df = normalize_timestamps(df)<br>df = cast_numeric_columns(df)<br>df = drop_duplicates(df)</pre><p>With reduce, the transformation steps become data:</p><pre>from functools import reduce<br><br>TRANSFORMS = [<br>    remove_nulls,<br>    normalize_timestamps,<br>    cast_numeric_columns,<br>    drop_duplicates,<br>]<br><br>df = reduce(lambda acc, fn: fn(acc), TRANSFORMS, df)</pre><p>The practical value is that TRANSFORMS can be assembled dynamically based on configuration or the source table&#39;s properties:</p><pre>from functools import reduce<br><br>def build_transform_pipeline(source: str) -&gt; list:<br>    transforms = [remove_nulls, normalize_timestamps]<br>    if source == &quot;legacy_erp&quot;:<br>        transforms.append(fix_legacy_encoding)<br>    if source in (&quot;orders&quot;, &quot;returns&quot;):<br>        transforms.append(cast_numeric_columns)<br>    transforms.append(drop_duplicates)<br>    return transforms<br><br>def apply_transforms(df, source: str):<br>    return reduce(lambda acc, fn: fn(acc), build_transform_pipeline(source), df)</pre><p>This pattern is useful when the transformation logic varies by source and the alternative is a long chain of if source == ...blocks scattered through the processing function.</p><p>One honest caveat: reduce can make code harder to read if the function passed to it is not trivial. When the accumulation logic is simple and the steps are named functions, it reads cleanly. When the lambda grows complex, a for loop is clearer.</p><h3>Where functools does not help</h3><p>lru_cache only works with hashable arguments. DataFrames, dictionaries, and lists cannot be used as cache keys. A function that takes a DataFrame and returns a transformed version of it cannot be memoized with lru_cache without wrapping the input somehow, at which point the overhead often outweighs the benefit.</p><p>partial fixes positional arguments by order, which can produce confusing errors if the underlying function&#39;s signature changes. It is best used with keyword arguments exclusively, as shown in the examples above.</p><p>reduce has no early exit. If a transformation step produces an empty DataFrame and subsequent steps should be skipped, reduce will run them anyway. A for loop with an explicit break condition is the right tool in that case.</p><p>None of these are reasons to avoid functools. They are reasons to understand what each tool actually does before reaching for it.</p><h3>The broader pattern</h3><p>The thread connecting lru_cache, partial, cache, and reduce is that they all make implicit structure explicit. The cache dictionary becomes a decorator. The shared operator arguments become a factory. The transformation sequence becomes a list.</p><p>This is the same shift that Pydantic makes with data contracts: moving things that were buried in imperative code into a form that is readable, testable, and reusable. functools does it for functions themselves.</p><ul><li><a href="https://joachimhodana.com/">Joachim Hodana - Software &amp; Data Engineer</a></li><li><a href="https://betterdataengineer.joachimhodana.com/">Better Data Engineer</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4364fc128861" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/why-data-engineers-should-care-about-functools-4364fc128861">Why Data Engineers Should Care About functools</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>