<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[dbt for Analytics - Medium]]></title>
        <description><![CDATA[dbt for Analytics is a technical publication focused on modern analytics engineering, data transformation, platform architecture, governance, observability, performance optimization, and AI-powered analytics. We publish practical, experience-driven for engineers - Medium]]></description>
        <link>https://medium.com/dbt-for-analytics?source=rss----822562fe37b4---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>dbt for Analytics - Medium</title>
            <link>https://medium.com/dbt-for-analytics?source=rss----822562fe37b4---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 06:59:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/dbt-for-analytics" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Engineering Production-Ready dbt Python Models]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/dbt-for-analytics/engineering-production-ready-dbt-python-models-3171ca636bdc?source=rss----822562fe37b4---4"><img src="https://cdn-images-1.medium.com/max/1536/1*WqxqS8_jQg53l0TDm0qX_w.png" width="1536"></a></p><p class="medium-feed-snippet">A Deep Dive into Execution Architecture, Incremental Processing, Performance Optimization and Enterprise Data Engineering</p><p class="medium-feed-link"><a href="https://medium.com/dbt-for-analytics/engineering-production-ready-dbt-python-models-3171ca636bdc?source=rss----822562fe37b4---4">Continue reading on dbt for Analytics »</a></p></div>]]></description>
            <link>https://medium.com/dbt-for-analytics/engineering-production-ready-dbt-python-models-3171ca636bdc?source=rss----822562fe37b4---4</link>
            <guid isPermaLink="false">https://medium.com/p/3171ca636bdc</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Sendoa Moronta]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 09:28:17 GMT</pubDate>
            <atom:updated>2026-11-17T09:28:18.931Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Dashboards as Code, Part 1: Getting Started with dbt Charts]]></title>
            <link>https://medium.com/dbt-for-analytics/dashboards-as-code-part-1-getting-started-with-dbt-charts-7e80502b8acd?source=rss----822562fe37b4---4</link>
            <guid isPermaLink="false">https://medium.com/p/7e80502b8acd</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[dbt-charts]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[dbt]]></category>
            <dc:creator><![CDATA[Muhammad Azhar]]></dc:creator>
            <pubDate>Wed, 11 Nov 2026 09:53:51 GMT</pubDate>
            <atom:updated>2026-11-11T09:53:50.666Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bXOnRww7_oIzo4eh6IBulQ.png" /></figure><p><strong>What is this series?</strong> I am learning dbt Charts and writing it up as I go. Each post will be based on a working board you can clone and run.</p><p><strong>Who this is for:</strong> you use dbt, you build dashboards somewhere else, and you are curious whether this is worth your time. On your part, I assume understanding of SQL and dbt; I explain YAML only where dbt Charts does something unusual with it.</p><h3>What dbt Charts is</h3><p>It is dbt Labs’ open-source language for dashboards. You write a board (read: dashboard) as a YAML file. The tool turns it into charts and renders them as a web page, PNG, SVG, or a PDF. The package is dbt-charts and the command is dct.</p><p>One rule explains most of the design:</p><blockquote>The query decides what data appears. The chart decides how it looks. Aggregation stays in the query.</blockquote><p>There are no calculated fields in the chart layer. Every number on the canvas comes from a line of SQL you can point at. Some chart features do perform calculations; more on those in later posts.</p><p><strong>If you come from Tableau,</strong> dropping a measure onto a shelf aggregates it for you, and a mark’s grain depends on what else is on the shelves. Here the grain is fixed in SQL, and the chart cannot change it behind you.</p><h3>Getting started</h3><p>The lab runs on dbt’s jaffle shop (a fictional sandwich shop with 100 customers and 99 orders). Clone the repo mentioned below to set up your dbt project; it loads the jaffle shop data for you.</p><pre>git clone https://github.com/imagineazhar/dbt-charts-learning.git<br>cd dbt-charts-learning<br>git checkout p1<br>./setup.sh</pre><p>It builds the sample data locally and wires up the boards.</p><p>Then start the preview server:</p><pre>source .venv/bin/activate<br>cd lab/jaffle_shop &amp;&amp; dct serve</pre><p>Open the URL it prints and go to /lessons/m01_first_board/.</p><p>If you would rather look before installing anything, <a href="https://play.dbtcharts.com/">play.dbtcharts.com</a> runs boards in the browser. You can use its built-in AI to create dummy data to practice on. I would recommend this.</p><p>You can also follow the <a href="https://docs.dbtcharts.com/guides/installation/#server-settings">official docs </a>for installation.</p><h3>What a board is</h3><p>A board is one YAML file that renders as one page. It holds four kinds of things:</p><ul><li>queries: — named SQL, one result set each</li><li>charts: — named charts, each reading one query</li><li>a layout — exactly one of rows:, cols:, grid: or tabs:</li><li>optional extras — title:, variables:, theme:, style:</li></ul><p>Boards (charts and dashboards) live in charts/ inside your dbt project. A file called dbt_charts.yml at the project root anchors everything: it holds the sources: registry, which is the set of named connections a board is allowed to read from, and dct walks up from wherever you run it to find that file. Drop a new .yml under charts/ and dct serve picks it up.</p><h3>Where it sits in your pipeline</h3><p>Downstream of dbt, in the slot a BI tool normally occupies:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*RU467x2b4JZQvIOdZ-nSLg.png" /></figure><p>A board points at a model, not a table, so validation can check it against dbt&#39;s manifest before touching a database.</p><h3>Creating your first chart</h3><p>The smallest thing that works is one query, one chart, one row:</p><pre>title: &quot;Orders per week&quot;<br><br>queries:<br>  orders_by_week:<br>    sql: |<br>      SELECT date_trunc(&#39;week&#39;, order_date) AS week,<br>             COUNT(*) AS orders<br>      FROM {{ ref(&#39;orders&#39;) }}<br>      GROUP BY 1<br>      ORDER BY 1<br>    source: jaffle<br><br>charts:<br>  weekly_orders:<br>    query: orders_by_week<br>    type: line<br>    x: week<br>    y: orders<br><br>rows:<br>  - weekly_orders</pre><p>Save that as charts/first.yml and check it:</p><pre>dct validate charts/first.yml</pre><p>Validation needs no database. It checks the schema, every chart-to-query reference. Then run dct serve and reload the page.source: jaffle names a connection from dbt_charts.yml.</p><p>Add each new chart only once the previous one validates. That is dct’s own advice in <a href="https://docs.dbtcharts.com/">dct docs getting-started</a>and it holds up.</p><h3>What’s on a chart</h3><p>A chart is a binding: one query, one type, and columns mapped to channels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*X20Jeu90g39waa7x30w55g.png" /></figure><p>Every channel takes a <strong>column name</strong>, never an expression: if a value needs changing, change the SQL.</p><p>Add a second chart and a layout, and you have a simple dashboard.</p><pre>title: &quot;Module 1: first board on jaffle shop&quot;<br><br>queries:<br>  orders_by_week:<br>    sql: |<br>      SELECT date_trunc(&#39;week&#39;, order_date) AS week,<br>             COUNT(*) AS orders<br>      FROM {{ ref(&#39;orders&#39;) }}<br>      GROUP BY 1<br>      ORDER BY 1<br>    source: jaffle<br><br>  orders_by_status:<br>    sql: |<br>      SELECT status, COUNT(*) AS orders<br>      FROM {{ ref(&#39;orders&#39;) }}<br>      GROUP BY 1<br>      ORDER BY 2 DESC<br>    source: jaffle<br><br>charts:<br>  weekly_orders:<br>    query: orders_by_week<br>    type: line<br>    title: &quot;Orders per week&quot;<br>    x: week<br>    y: orders<br><br>  status_mix:<br>    query: orders_by_status<br>    type: bar<br>    title: &quot;Orders by status&quot;<br>    x: status<br>    y: orders<br><br>rows:<br>  - text: |<br>      Two charts, two queries, one layout.<br>  - cols:<br>      - weekly_orders<br>      - status_mix</pre><p>rows: Stacks items vertically; a cols: block sits them side by side. Charts appear by name, so layout is only for arrangement. A text: block is the one exception; it is a published Markdown. Whatever you write there lands on the page, in the PNG and in the PDF.</p><h3>Rendering boards and charts</h3><p>Two ways. dct serve is the live preview you edit against. dct render writes a file:</p><pre>dct render charts/first.yml --format png<br>dct render charts/first.yml --format pdf --output reports/orders.pdf<br>dct render charts/*.yml --format svg --output renders/{stem}.svg</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oVFp3-J7SOcJcs1Ebyy83A.png" /><figcaption>Rendered chart</figcaption></figure><h3>What’s next in the series</h3><p>Ten more posts, each with a board in the repo:</p><ul><li><strong>P2 — Different chart types, and the shapes that aren’t types.</strong> Why asking for a bullet chart returns an error instead of a chart.</li><li><strong>P3 — Layout:</strong> rows, cols, tabs and grid, and what the layout tree cannot express.</li><li><strong>P4 — Filters you can read:</strong> variables wired into SQL.</li><li><strong>P5 — Encoding a house style</strong> in meta.yml.</li><li><strong>P6 — Layers, small multiples,</strong> and the first chart I could not build.</li><li><strong>P7 — Drill-down</strong> without a BI tool.</li><li><strong>P8 — Reviewing a dashboard in a pull request.</strong></li><li><strong>P9 — </strong>I asked an agent to build the same board.</li><li><strong>P10 — Shipping boards:</strong> PNG, SVG, PDF, MkDocs.</li><li><strong>P11 — </strong>From decision brief to board.</li></ul><p>The board from this post is in <a href="https://github.com/imagineazhar/dbt-charts-learning">the repo</a>, tagged. p1.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e80502b8acd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dbt-for-analytics/dashboards-as-code-part-1-getting-started-with-dbt-charts-7e80502b8acd">Dashboards as Code, Part 1: Getting Started with dbt Charts</a> was originally published in <a href="https://medium.com/dbt-for-analytics">dbt for Analytics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[dbt-osmosis: Automation for Schema and Documentation Management in dbt]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/dbt-for-analytics/dbt-osmosis-automation-for-schema-and-documentation-management-in-dbt-70ecfec3442a?source=rss----822562fe37b4---4"><img src="https://cdn-images-1.medium.com/max/1536/0*Fgsp4bEkQIKfLqm7" width="1536"></a></p><p class="medium-feed-snippet">How dbt-osmosis automates schema management, enforces documentation standards and scales governance for modern analytics engineering teams.</p><p class="medium-feed-link"><a href="https://medium.com/dbt-for-analytics/dbt-osmosis-automation-for-schema-and-documentation-management-in-dbt-70ecfec3442a?source=rss----822562fe37b4---4">Continue reading on dbt for Analytics »</a></p></div>]]></description>
            <link>https://medium.com/dbt-for-analytics/dbt-osmosis-automation-for-schema-and-documentation-management-in-dbt-70ecfec3442a?source=rss----822562fe37b4---4</link>
            <guid isPermaLink="false">https://medium.com/p/70ecfec3442a</guid>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[documentation]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[dbt]]></category>
            <dc:creator><![CDATA[Sendoa Moronta]]></dc:creator>
            <pubDate>Fri, 06 Nov 2026 13:48:25 GMT</pubDate>
            <atom:updated>2026-11-06T13:48:26.716Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[dbt Charts Part 3: Three dbt Charts errors that only appear at render time.]]></title>
            <link>https://medium.com/dbt-for-analytics/dbt-charts-part-3-three-dbt-charts-errors-that-only-appear-at-render-time-09072f464a91?source=rss----822562fe37b4---4</link>
            <guid isPermaLink="false">https://medium.com/p/09072f464a91</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[dbt]]></category>
            <dc:creator><![CDATA[Muhammad Azhar]]></dc:creator>
            <pubDate>Fri, 06 Nov 2026 12:56:37 GMT</pubDate>
            <atom:updated>2026-11-06T12:56:35.819Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ddm9pfW8YvSW0cqZ8Qm_Aw.png" /></figure><p>Three mistakes passed dct validate and then failed dct render. The two commands check different things, and the difference decides where you catch a mistake.</p><p>In this tutorial, we will split a line by a column, control the time axis and the number labels, and put a skewed measure on a log scale. You will also know which mistakes dct validate it catches and which only dct render does.</p><h3>Build: one line per status.</h3><p>The first chart counts orders per week and splits the line by status.</p><pre>queries:<br>  orders_by_week_status:<br>    sql: |<br>      SELECT date_trunc(&#39;week&#39;, order_date) AS week,<br>             status,<br>             COUNT(*) AS orders<br>      FROM {{ ref(&#39;orders&#39;) }}<br>      WHERE order_date &lt; (SELECT date_trunc(&#39;week&#39;, MAX(order_date)) FROM {{ ref(&#39;orders&#39;) }})<br>      GROUP BY 1, 2<br>      ORDER BY 1<br>    source: jaffle<br><br>charts:<br>  weekly_orders_by_status:<br>    type: line<br>    query: orders_by_week_status<br>    x: week<br>    y: orders<br>    color: status</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HWhfMbbe9Hj6bj9s2vT44g.png" /></figure><p>color: status gives one series per value in that column. The render has five lines: completed, placed, shipped, returned and return_pending. dct labels each line at its right end, in the line&#39;s colour, instead of drawing a legend.</p><p>The WHERE drops the final, partial week. That belongs in the query, where a reviewer can see it and disagree. A chart that quietly skipped the last week would hide the same decision.</p><p>Only completed is a full line. The other four are short runs, because dct leaves a gap where a status has no orders in a week. It does not draw a zero.</p><p>The chart is accurate and misleading. A status is where an order is now, not a kind of order. Old orders are completedMarch orders are shipped, the newest are placed. Splitting time by status mixes when an order was placed with how far it has gotten.</p><h3>Build: the time axis</h3><p>dct reads the grain from the dates in the x column. After I set time_unit: yearweek and a label format, the ticks read Jan 01, Feb 05, Mar 05.</p><pre>   style:<br>      axis_x:<br>        time_unit: yearweek<br>        labels:<br>          format: &quot;%b %d&quot;</pre><p>That is five weeks between ticks, so there are fewer labels than points.</p><h3>Build: a log axis for lifetime value.</h3><p>The second chart ranks the 62 customers by lifetime value, with the rank on x. The values run from about 1 to 100.</p><pre> cltv_log:<br>    type: line<br>    query: customer_value_rank<br>    x: customer_rank<br>    y: customer_lifetime_value<br>    style:<br>      axis_y:<br>        labels:<br>          format: &quot;$,.0f&quot;<br>        scale:<br>          continuous:<br>            type: log</pre><p>A log axis spaces values by multiples instead of equal steps. This one is labelled 1, 2, 3, 5, 10, 20, 30, 50, and 100, with uneven grid lines.</p><h3>What the tool told me</h3><p>Three mistakes. All three passed dct validate. All three stopped dct render.</p><ol><li><strong>A coarser grain than the data.</strong> I set time_unit: yearmonth on the weekly query:</li></ol><pre>ERR-GAP-FILL-BUCKET-COLLISION  Rows with &#39;week&#39; values datetime.datetime(2018, <br>1, 1, 0, 0) and datetime.datetime(2018, 1, 8, 0, 0) both collapse to the <br>&#39;yearmonth&#39; bucket &#39;2018-01-01&#39; (status=&#39;completed&#39;). Aggregate to yearmonth <br>grain in the query before rendering.</pre><p>Several weekly rows fell into one month. dct will not pick one and hide the rest. The fix is in the query: group by month first. The chart never aggregates.</p><p><strong>2. A number format on a date axis.</strong> I set style.axis_x.labels.format: &quot;$,.0f&quot;:</p><pre>ERR-LABEL-FORMAT-AXIS-MISMATCH  style.axis_x.labels.format (&#39;$,.0f&#39;) cannot be <br>read on this axis: &#39;week&#39; holds non-numeric tick labels. This axis is temporal <br>— d3&#39;s number grammar doesn&#39;t read over dates. Author a time spec here (e.g. <br>&quot;%b %Y&quot;), or set style.time_format instead.</pre><p>The format has to match what the axis holds. The docs say a number axis takes a number format, a date axis takes a time format, and a text axis takes none.</p><p><strong>3. A zero under a log scale.</strong> I added a row of 0 to the lifetime-value query:</p><pre>ERR-LOG-SCALE-REQUIRES-POSITIVE-DATA  Chart &#39;cltv_log&#39;: column <br>&#39;customer_lifetime_value&#39; has a value &lt;= 0, but axis_y.scale.type: log requires<br>strictly positive data; a log domain is undefined at and below zero. Filter out<br>the non-positive rows, or drop the log scale.</pre><h3>Why validate passed</h3><p>Each error depends on something dct validate does not read: the rows, or the data type of an axis column. Validate checks the YAML and the names. Render checks the YAML against the data. A clean validate tells you the board is well formed, not that it will draw.</p><p>One message did come from validate. dct guessed from the field name that customer_lifetime_value is money, and warned that the axis format was the default compact one:</p><pre>WARN-LIKELY-CURRENCY-OR-PERCENT-MISSING-FORMATTER <br>cltv_log: Chart &#39;cltv_log&#39;:<br>field &#39;customer_lifetime_value&#39; looks like currency but the y-axis format is<br>&#39;.3~s&#39;.<br>Fix: Set `style.axis_y.labels.format` to a currency format (e.g. `$,.2f`) or a<br>percent format (e.g. `.1%`) to match the field&#39;s meaning.</pre><p>Setting labels.format: &quot;$,.0f&quot; cleared it. This check looks at the encoding, not the data, so validate can run it.</p><h3>Where it falls short</h3><p><strong>Gaps, not zeros.</strong> dct leaves a gap where a series has no rows, and draws no zero.</p><p><strong>Warnings that read only the model.</strong> Both queries read models that end in select *, so validate warns that it could not check my column names (WARN-DBT-MODEL-COLUMNS-UNRESOLVED). Those references went unchecked at validate.</p><h3>Next</h3><p>Next is layout: how rows, cols, grid, and tabs place the charts on a page. The board from this post is in <a href="https://github.com/imagineazhar/dbt-charts-learning">the repo</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=09072f464a91" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dbt-for-analytics/dbt-charts-part-3-three-dbt-charts-errors-that-only-appear-at-render-time-09072f464a91">dbt Charts Part 3: Three dbt Charts errors that only appear at render time.</a> was originally published in <a href="https://medium.com/dbt-for-analytics">dbt for Analytics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reprocessing BigQuery at Scale: The Hidden Failure Modes of Incremental dbt Pipelines]]></title>
            <link>https://medium.com/dbt-for-analytics/reprocessing-bigquery-at-scale-the-hidden-failure-modes-of-incremental-dbt-pipelines-b913cf3fd847?source=rss----822562fe37b4---4</link>
            <guid isPermaLink="false">https://medium.com/p/b913cf3fd847</guid>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <dc:creator><![CDATA[Sendoa Moronta]]></dc:creator>
            <pubDate>Tue, 27 Oct 2026 09:38:45 GMT</pubDate>
            <atom:updated>2026-10-27T09:38:45.650Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>Designing deterministic backfills, safe partition replacement, and dependency-aware recovery in production data platforms.</em></blockquote><p>Most data pipelines are designed around one assumption: new data arrives, transformations run, and the resulting tables become more complete.</p><p>Production systems eventually break that assumption.</p><p>A business rule changes. An upstream source corrects six months of historical records. A dimension receives retroactive updates. An ingestion bug silently corrupts a subset of partitions. Suddenly, a pipeline that was designed to process yesterday’s data must reconstruct a portion of its past.</p><p>The obvious solution is to rerun the affected dbt models.</p><p>The dangerous assumption is that rerunning them will restore the correct state.</p><p>In BigQuery, incremental materializations, partition replacement, late-arriving events, concurrent ingestion, and downstream dependencies introduce failure modes that are not immediately visible. A job can finish successfully while leaving stale records behind. A backfill can produce correct source-level results while leaving downstream aggregates inconsistent. A retry can be technically successful without being operationally safe.</p><p>The problem is not how to execute a historical transformation.</p><p><strong>The problem is how to replace historical state without introducing a new inconsistency.</strong></p><p>This article explores a production-oriented approach to that problem, using dbt and BigQuery.</p><h3>1. Reprocessing is a state transition, not a query execution</h3><p>Consider a fact table partitioned by event_date:</p><pre>CREATE TABLE analytics.fact_orders<br>PARTITION BY event_date<br>CLUSTER BY customer_id AS<br>SELECT<br>    event_date,<br>    customer_id,<br>    order_id,<br>    amount<br>FROM analytics.stg_orders;</pre><p>The table contains two years of historical data. An incremental dbt model processes new events every day.</p><p>Now imagine that a defect in the order attribution logic is discovered. The affected records span 90 historical days.</p><p>A naive solution is to execute the model again with a date filter.</p><p>That approach leaves several questions unanswered:</p><ul><li>What happens to records that existed before the fix but are no longer valid?</li><li>What happens if an affected day now produces zero rows?</li><li>What happens if new corrections arrive during the replay?</li><li>Which downstream models must be recalculated?</li><li>What happens if the execution fails after updating some of the affected partitions?</li><li>How do we know that a retry produces exactly the same final state?</li></ul><h3>2. Choose the correct replacement strategy</h3><p>BigQuery offers different ways to update an existing table. The choice of incremental strategy determines both the cost profile and the failure modes.</p><h4>Strategy A: MERGE</h4><p>A typical dbt incremental model uses a unique key and a MERGE operation to update existing records and insert new ones.</p><pre>{{ config(<br>    materialized=&#39;incremental&#39;,<br>    unique_key=&#39;order_id&#39;,<br>    incremental_strategy=&#39;merge&#39;<br>) }}<br><br>SELECT<br>    order_id,<br>    event_date,<br>    customer_id,<br>    amount<br>FROM {{ ref(&#39;stg_orders&#39;) }}<br><br>{% if is_incremental() %}<br>WHERE event_date &gt;= DATE(&#39;{{ var(&quot;replay_start&quot;) }}&#39;)<br>  AND event_date &lt; DATE(&#39;{{ var(&quot;replay_end_exclusive&quot;) }}&#39;)<br>{% endif %}</pre><p>This is useful when the source produces individual record changes and the model must preserve unaffected records.</p><p>However, a MERGE that only updates matching records and inserts new ones does not automatically delete records that disappeared from the source.</p><p>Suppose order ORD-123 was present yesterday but is excluded by the corrected business logic today. If the replay does not explicitly delete it, the old record can remain in the target.</p><p>There is another subtlety: the unique_key configuration is not a uniqueness constraint enforced by BigQuery. Duplicate source keys can produce errors or unexpected results, depending on the operation and the shape of the source data.</p><p>For a correct merge-based replay, you must define how deletions are represented and ensure that the source is unique at the intended grain.</p><p>Partition pruning also requires care. Restricting the target side of a MERGE can reduce scanning, but if the restriction excludes an existing matching record, the intended update may become an insert instead. Target-side pruning is safe only when the matching and replacement semantics remain valid.</p><h4>Strategy B: insert_overwrite</h4><p>For date-partitioned facts, rebuilding complete partitions is often a better fit.</p><pre>{{ config(<br>    materialized=&#39;incremental&#39;,<br>    incremental_strategy=&#39;insert_overwrite&#39;,<br>    partition_by={<br>        &#39;field&#39;: &#39;event_date&#39;,<br>        &#39;data_type&#39;: &#39;date&#39;,<br>        &#39;granularity&#39;: &#39;day&#39;<br>    },<br>    cluster_by=[&#39;customer_id&#39;]<br>) }}<br><br>SELECT<br>    event_date,<br>    customer_id,<br>    SUM(amount) AS total_amount<br>FROM {{ ref(&#39;stg_orders&#39;) }}<br>WHERE event_date &gt;= DATE(&#39;{{ var(&quot;replay_start&quot;) }}&#39;)<br>  AND event_date &lt; DATE(&#39;{{ var(&quot;replay_end_exclusive&quot;) }}&#39;)<br>GROUP BY 1, 2</pre><p>This example illustrates the transformation. A production implementation must also define how the affected partitions are selected and replaced.</p><p>The crucial difference is that the target is not updated record by record. The old contents of the selected partitions are replaced by the newly computed results.</p><p>That gives us a useful invariant:</p><p><strong>A replay must replace the complete state of every affected partition, not merely append the rows produced by the replay.</strong></p><p>Partition replacement is particularly attractive when a business date is the natural unit of recomputation and the complete source for that date is available.</p><p>It is less appropriate when corrections span arbitrary records, the partition boundary does not match the business impact, or reconstructing a complete partition is prohibitively expensive.</p><h4>Strategy C: full refresh</h4><p>A full refresh can be appropriate when a model is small, its logic changes fundamentally, or its complete historical state must be rebuilt.</p><p>It is rarely the best default for a large production fact table.</p><p>Rebuilding a multi-terabyte table to correct a few dozen partitions increases cost and execution time. It can also create unnecessary operational pressure on downstream consumers.</p><p>The decision should be based on the smallest unit of state that can be reconstructed correctly.</p><pre>| Strategy           | Best fit                          | Main risk                            |<br>|--------------------|-----------------------------------|--------------------------------------|<br>| `MERGE`            | Record-level corrections          | Stale records and incorrect matching |<br>| `insert_overwrite` | Complete partition reconstruction | Incorrect partition selection        |<br>| Full refresh       | Small models or global rebuilds   | —                                    |</pre><p>Cost and operational disruption</p><p>The important distinction is not simply performance.</p><p>It is whether the chosen strategy can express the intended correction semantics.</p><h3>3. The empty-partition trap</h3><p>This is one of the most dangerous failure modes in historical reprocessing.</p><p>Suppose a partition for 2026-03-14 contains 120 orders. A corrected business rule determines that none of those orders should remain in the target.</p><p>The reconstructed query returns zero rows for that date.</p><p>What happens when dbt uses dynamic partition replacement?</p><p>The dynamic insert_overwrite strategy identifies partitions from the data produced by the temporary model result. If 2026-03-14 does not appear in that result, the partition might not be included in the replacement set.</p><p>The old 120 orders can survive.</p><p>The model execution may succeed. The SQL may be valid. The output may be correct for every row it contains.</p><p>The final table is still wrong.</p><p>This is the distinction between two statements:</p><ul><li>The transformation produced no rows for a date.</li><li>The transformation was not intended to modify that date.</li></ul><p>They are not equivalent.</p><p>A robust replay mechanism must carry the intended replacement set independently of the rows returned by the transformation.</p><h4>Use an explicit replay manifest</h4><p>A replay manifest makes the intended mutation explicit.</p><pre>CREATE TABLE IF NOT EXISTS ops.replay_manifest (<br>    replay_id STRING,<br>    model_name STRING,<br>    partition_date DATE,<br>    logic_version STRING,<br>    source_watermark TIMESTAMP,<br>    status STRING,<br>    started_at TIMESTAMP,<br>    completed_at TIMESTAMP<br>)<br>PARTITION BY partition_date<br>CLUSTER BY replay_id, model_name;</pre><p>A replay might create one record per model and partition:</p><pre>replay_id  | model_name       | partition_date | status<br>-----------|------------------|----------------|----------<br>R-20261027 | fact_orders      | 2026-03-14     | PENDING<br>R-20261027 | fact_orders      | 2026-03-15     | PENDING<br>R-20261027 | fact_orders      | 2026-03-16     | PENDING</pre><p>The manifest is not merely a logging table. It is the authoritative description of the intended replay.</p><p>It records which partitions must change even when their correct result is empty.</p><p>For a bounded replay, the partition list should be generated from the requested date range or from a deterministic impact analysis, not inferred exclusively from the transformed data.</p><h4>Make the replacement set explicit in dbt</h4><p>The BigQuery adapter supports static partition selection for insert_overwrite. For example, a generated model configuration can contain:</p><pre>{{<br>    config(<br>        materialized=&#39;incremental&#39;,<br>        incremental_strategy=&#39;insert_overwrite&#39;,<br>        partition_by={<br>            &#39;field&#39;: &#39;event_date&#39;,<br>            &#39;data_type&#39;: &#39;date&#39;,<br>            &#39;granularity&#39;: &#39;day&#39;<br>        },<br>        partitions=[<br>            &quot;DATE &#39;2026-03-14&#39;&quot;,<br>            &quot;DATE &#39;2026-03-15&#39;&quot;,<br>            &quot;DATE &#39;2026-03-16&#39;&quot;<br>        ]<br>    )<br>}}</pre><p>In a real implementation, these values should be generated from the replay manifest through controlled dbt/Jinja configuration or a dedicated orchestration layer. The dates above are illustrative.</p><p>Static partition replacement allows the intended partitions to be specified independently of the temporary table’s distinct partition values. This is the key property that protects against the empty-partition failure.</p><p>There is one important operational detail: static partition configuration is part of the compiled model configuration. A runtime manifest does not automatically become a dbt partitions list. Your orchestration must explicitly pass or generate the intended values and validate the compiled SQL before execution.</p><p>For very large replay ranges, a separate SQL-driven publication step may be more practical than generating an enormous static list.</p><h3>4. Event time is not ingestion time</h3><p>A replay window is only as good as the logic used to identify affected data.</p><p>Consider an event with:</p><pre>event_timestamp:     2026-03-14 10:30:00 UTC<br>ingestion_timestamp: 2026-10-27 08:12:00 UTC</pre><p>The event belongs to a historical business date, but it arrived today.</p><p>A pipeline that only processes recent event dates will never include it.</p><p>This is why ingestion time and business time must be modelled separately.</p><ul><li>event_timestamp determines when the business event occurred.</li><li>ingestion_timestamp determines when the platform received the record.</li></ul><p>Neither timestamp is a universal substitute for the other.</p><p>A watermark based only on MAX(event_timestamp) can miss late-arriving records, especially if the source does not guarantee ordered delivery.</p><p>A rolling three-day lookback reduces the risk for bounded lateness, but it does not solve corrections that can arrive months after the original event.</p><h4>Maintain an explicit change-to-partition mapping</h4><p>For sources that support CDC, retain the original business date and the corrected business date when an event is moved.</p><p>For example:</p><pre>order_id | old_event_date | new_event_date<br>---------|----------------|---------------<br>ORD-123  | 2026-03-14     | 2026-03-15</pre><p>Both dates are affected.</p><p>If the old date is not rebuilt, the previous representation can survive. If the new date is not rebuilt, the corrected representation is missing.</p><p>The same principle applies when a dimension changes the classification of existing facts. The affected partition set must reflect the business impact of the change, not just the arrival date of the correction.</p><p>For mutable sources, a replay should also define a source cutoff or snapshot boundary. Otherwise, the same replay ID may observe different inputs on successive attempts.</p><p>A practical replay identity can be modelled as:</p><pre>replay_id<br>model_name<br>partition_date<br>logic_version<br>source_watermark<br>input_snapshot_id</pre><p>This creates a reproducible description of what the replay was supposed to calculate.</p><p>If the source cannot provide a consistent snapshot, the replay process must explicitly accept eventual convergence or introduce a staging snapshot of its own.</p><h3>5. A replay manifest is a lightweight control plane</h3><p>Once the affected partition set is explicit, the next step is to treat replay execution as a state machine.</p><p>A useful lifecycle is:</p><pre>PENDING<br>   |<br>   v<br>RUNNING<br>   |<br>   v<br>VALIDATING<br>   |<br>   v<br>READY_TO_PUBLISH<br>   |<br>   v<br>PUBLISHED</pre><p>Failure states should distinguish between computation failures, validation failures and publication failures.</p><p>For each replay unit, the orchestration layer should know:</p><ul><li>Which partitions are being rebuilt.</li><li>Which source cutoff is being used.</li><li>Which version of the transformation is expected.</li><li>Whether the output has been validated.</li><li>Whether publication has occurred.</li><li>Whether downstream dependencies have been reconciled.</li></ul><p>A retry must not blindly rerun every operation. It must inspect the last committed state and resume from the appropriate boundary.</p><p>For example, if computation and validation succeeded but publication failed, the replay may only need to repeat publication. If publication succeeded but the orchestration process crashed before recording success, the process should verify the target state before deciding what to do next.</p><p>This is why replay status alone is insufficient. The status transition must be reconciled with the actual state of the target table.</p><h4>Idempotency is more than deduplication</h4><p>A common definition of idempotency is that executing the same operation twice has the same effect as executing it once.</p><p>For replay systems, the intended property is:</p><pre>[<br>  R(R(T,S,v),S,v)=R(T,S,v)<br>]</pre><p>Here, (R) is the replay operation, (T) is the initial target state, (S) is the defined source snapshot, and (v) is the transformation version.</p><p>If the same replay is executed twice with the same inputs, the final target state should be identical.</p><p>That requires more than a unique replay ID. The transformation must be deterministic enough for the required correctness guarantees, and the publication step must replace state rather than accumulate duplicate output.</p><p>If a model depends on current exchange rates, mutable dimensions or non-versioned reference data, the source snapshot must include those dependencies or document why they are intentionally read at execution time.</p><p>A replay ID identifies the operation. It does not make the operation idempotent by itself.</p><h3>6. BigQuery atomicity does not make a multi-model replay atomic</h3><p>BigQuery provides atomicity for individual DML statements. A successful partition-replacement statement does not expose a partially applied version of that statement.</p><p>However, a replay involving multiple dbt models is not automatically one atomic operation.</p><p>Imagine this dependency chain:</p><pre>stg_orders<br>    |<br>    v<br>fact_orders<br>    |<br>    v<br>daily_customer_sales<br>    |<br>    v<br>customer_monthly_summary</pre><p>The historical rebuild of fact_orders may succeed while daily_customer_sales still contains values calculated from the old fact table.</p><p>If consumers query both models during that interval, they can observe inconsistent business state.</p><p>The same problem occurs if the first 20 partitions are published and the process fails before the remaining 70 are replaced.</p><p>The correct response depends on the consistency requirements of the platform.</p><h4>Option A: Publish one partition at a time</h4><p>This is suitable when each partition is independently consumable and downstream processing is coordinated by partition.</p><p>For every partition:</p><ol><li>Rebuild the complete partition.</li><li>Validate its output.</li><li>Replace the target partition.</li><li>Trigger or schedule downstream recomputation for that partition.</li><li>Mark the partition complete only after the required checks succeed.</li></ol><p>This reduces the size of each publication unit, but it does not guarantee a consistent 90-day view throughout the entire replay.</p><h4>Option B: Stage and publish a replay batch</h4><p>When consumers must not observe partially reconstructed historical state, use a staging and publication design.</p><p>The process is:</p><ol><li>Build the complete replay output into an isolated staging table or version.</li><li>Validate every affected partition.</li><li>Publish the validated results using a controlled DML operation or an appropriately designed table-swap strategy.</li><li>Recompute dependent state or switch consumers to a consistent published version.</li></ol><p>For a single target table, a carefully constructed MERGE or partition-replacement operation can provide an atomic publication boundary for that statement. BigQuery multi-statement transactions may also help where the required operations and transaction constraints are supported.</p><p>However, a collection of dbt model executions should not be assumed to share one transaction. Replacing multiple tables, coordinating downstream jobs and recording external orchestration state requires a separate consistency design.</p><p>A staging table alone does not solve this problem. The publication protocol is what determines which state consumers can observe.</p><h3>7. Reprocessing a parent model does not repair its descendants</h3><p>This is one of the most frequently underestimated aspects of historical recovery.</p><p>Suppose fact_orders is corrected for March. The model daily_customer_sales is derived from it, and customer_monthly_summary is derived from the daily aggregation.</p><p>Rebuilding fact_orders does not retroactively update the materialized contents of either downstream table.</p><p>dbt knows the dependency graph, but dependency selection is not the same as historical impact analysis.</p><p>A downstream model may need a wider replay window than its parent.</p><p>For example:</p><ul><li>A daily aggregation may need the same affected dates.</li><li>A monthly aggregation may need the complete affected months.</li><li>A rolling 30-day metric may need a larger input window than the dates being published.</li><li>A slowly changing dimension may affect facts across an open-ended historical range.</li></ul><p>Consider a 30-day rolling measure:</p><pre>[<br>  M_d=\sum_{i=d-29}^{d} x_i<br>]</pre><p>If the value of (x_d) changes, the affected outputs include dates (d) through (d+29), assuming the metric is calculated over the preceding 30 calendar days.</p><p>Rebuilding only the partition corresponding to (d) leaves downstream results stale.</p><p>This is not a dbt-specific defect. It follows directly from the semantics of the transformation.</p><h4>Model the impact function explicitly</h4><p>For every materialized model, define the relationship between changed input dates and affected output dates.</p><p>For a simple daily aggregation:</p><pre>[<br>  I_{\text{daily}}(D)=D<br>]</pre><p>For a trailing 30-day metric, a change on date (d) affects:</p><pre>[<br>  I_{\text{rolling30}}({d})=[d,d+29]<br>]</pre><p>For monthly aggregates, the affected output partitions are the months intersecting the impacted input dates.</p><p>This information can be encoded in orchestration metadata or model-specific macros. It should not be inferred solely from the SQL file’s dependencies.</p><p>A replay planner can then propagate the impact set through the dbt DAG, expanding or transforming the set of affected partitions according to each model’s semantics.</p><p>The result is a more accurate replay plan than simply selecting all downstream nodes and applying the original date range to each one.</p><h3>8. Concurrency: the replay is competing with the live pipeline</h3><p>Historical replay rarely happens in a quiet environment.</p><p>The daily ingestion pipeline continues running. New events arrive. Scheduled dbt jobs may write to the same destination. Another operator might start a second replay.</p><p>This creates several potential problems:</p><ul><li>Two operations attempt to modify overlapping partitions.</li><li>A replay reads an incomplete or changing upstream dataset.</li><li>A live incremental run publishes newer data before an older replay overwrites it.</li><li>An orchestration retry repeats a publication whose outcome was already committed.</li></ul><p>BigQuery’s DML concurrency controls and conflict handling help protect data integrity, but they do not define the intended business ordering between live ingestion and replay operations.</p><p>That ordering belongs to the pipeline design.</p><h4>Use explicit ownership of affected partitions</h4><p>For each replay, establish an ownership boundary covering the target model and the affected partitions.</p><p>The orchestration layer should reject, queue or coordinate overlapping operations rather than assuming that BigQuery will resolve the business conflict correctly.</p><p>A replay can use a lock or lease keyed by model and partition range. The lock must support expiration and recovery so that a crashed worker cannot permanently block future work.</p><p>The live pipeline must follow a compatible protocol. Locking only the replay while allowing the regular incremental model to publish to the same partitions is insufficient.</p><p>There is also a source-consistency question.</p><p>If ingestion continues while the replay runs, define whether the replay:</p><ul><li>Uses a fixed source snapshot and lets later arrivals be processed separately.</li><li>Includes all changes received up to a specified cutoff.</li><li>Repeats reconciliation until the affected source and target converge.</li></ul><p>The first approach is often easiest to make reproducible. The third can be appropriate for systems with frequent late-arriving corrections, but requires explicit convergence criteria.</p><p>Never let the effective source boundary depend on when an individual SQL statement happens to execute.</p><h3>9. Cost control: optimize the replay, not just the final query</h3><p>A historical replay can be logically correct and still be operationally unacceptable.</p><p>BigQuery cost depends on the amount of data scanned, the transformations performed, the materialization strategy and the amount of data written.</p><p>A common mistake is to restrict the output dates without restricting the upstream work.</p><p>For example:</p><pre>SELECT<br>    event_date,<br>    customer_id,<br>    SUM(amount) AS total_amount<br>FROM {{ ref(&#39;stg_orders&#39;) }}<br>WHERE event_date &gt;= DATE(&#39;2026-03-01&#39;)<br>  AND event_date &lt; DATE(&#39;2026-04-01&#39;)<br>GROUP BY 1, 2</pre><p>This filter is useful only if the upstream table is partitioned compatibly and the predicate allows BigQuery to prune the relevant partitions.</p><p>If the source model is a view over an unpartitioned table, or if the transformation wraps the partition column in a way that prevents effective pruning, the scan can be much larger than expected.</p><p>The same applies to joins. A replay of one month of orders may still scan an enormous customer-history table if the join cannot be pruned.</p><p>For large replays, evaluate the following independently:</p><p><strong>Source pruning.</strong> Filter the earliest accessible source before expensive joins and aggregations.</p><p><strong>Target pruning.</strong> Ensure the publication statement touches only the intended partitions when the replacement semantics permit it.</p><p><strong>Clustering.</strong> Use clustering columns that align with common filtering and join patterns, while remembering that clustering is not a substitute for partition pruning.</p><p><strong>Intermediate materialization.</strong> Consider whether expensive source transformations should be staged once and reused across the replay.</p><p><strong>Batch sizing.</strong> Replaying a year as one operation may be efficient in some environments, but smaller deterministic batches can improve recovery, observability and cost control.</p><p><strong>Dry runs and execution plans.</strong> Estimate the bytes processed and inspect the compiled SQL before launching an expensive historical rebuild.</p><p>One subtle trade-off deserves attention: many small replay batches can repeatedly scan the same upstream data. Conversely, one large batch can increase resource consumption and make recovery less granular.</p><p>The optimal batch size is therefore not simply the smallest possible date range. It is the unit that balances source-scan efficiency, publication safety and recoverability.</p><h3>10. Validate the reconstructed state, not just the job status</h3><p>A successful BigQuery job proves that the statement completed. It does not prove that the business state is correct.</p><p>For each replayed partition, validation should cover several dimensions.</p><h4>Structural validation</h4><p>Confirm that the partition contains the expected schema, types and non-null fields.</p><h4>Key-level validation</h4><p>If the model has a defined grain, test that the corresponding key is unique.</p><pre>SELECT<br>    event_date,<br>    order_id,<br>    COUNT(*) AS row_count<br>FROM analytics.fact_orders<br>WHERE event_date &gt;= DATE(&#39;2026-03-01&#39;)<br>  AND event_date &lt; DATE(&#39;2026-04-01&#39;)<br>GROUP BY 1, 2<br>HAVING COUNT(*) &gt; 1;</pre><p>This query should return no rows if (event_date, order_id) is the intended unique grain.</p><h4>Reconciliation</h4><p>Compare source and target counts, totals, and other business invariants at the appropriate grain.</p><p>For example:</p><pre>SELECT<br>    event_date,<br>    SUM(amount) AS total_amount,<br>    COUNT(DISTINCT order_id) AS distinct_orders<br>FROM analytics.fact_orders<br>WHERE event_date &gt;= DATE(&#39;2026-03-01&#39;)<br>  AND event_date &lt; DATE(&#39;2026-04-01&#39;)<br>GROUP BY 1;</pre><p>Aggregate checks are useful but insufficient on their own. Two incorrect datasets can have the same total amount.</p><p>Where the requirements justify it, compare canonicalized row representations or deterministic hashes, taking care to define null handling, numeric precision, timestamp normalization and column ordering.</p><h4>Explicitly validate empty partitions</h4><p>This check is particularly important for partition replacement.</p><p>If a date was expected to be empty after replay, verify that it actually contains zero rows in the published table.</p><p>Do not omit empty dates from validation just because they do not appear in the transformation output.</p><p>The replay manifest provides the expected partition set, so validation can iterate over that set rather than over the partitions observed in the target.</p><h4>Validate downstream invariants</h4><p>After rebuilding a fact table, validate the corresponding aggregates and dependent models.</p><p>The validation contract should be defined by the model’s grain and business meaning, not simply by whether dbt reports a successful execution.</p><h3>11. Putting the pieces together</h3><p>A production replay can be organized into a small number of explicit phases.</p><pre>               Replay Request<br>                       |<br>                       v<br>             Impact Analysis / DAG<br>                       |<br>                       v<br>                Replay Manifest<br>                       |<br>                       v<br>           Source Snapshot / Cutoff<br>                       |<br>                       v<br>              Partition Rebuild<br>                       |<br>                       v<br>             Output Validation<br>                       |<br>                       v<br>             Controlled Publication<br>                       |<br>                       v<br>            Downstream Reprocessing<br>                       |<br>                       v<br>             Final Reconciliation<br>                       |<br>                       v<br>                  COMPLETED</pre><p>Each phase has a separate responsibility.</p><p>The impact analysis determines which partitions must change, including downstream partitions whose results depend on the corrected data.</p><p>The manifest records the intended work independently of the output produced by the transformation.</p><p>The source cutoff establishes which input state is being reconstructed.</p><p>The rebuild calculates the complete replacement state for each affected partition.</p><p>Validation verifies the candidate output before publication.</p><p>Publication replaces the target state using explicit replacement semantics.</p><p>Finally, downstream reprocessing and reconciliation establish that the overall dependency chain has converged.</p><p>This architecture does not require a complex platform on day one. A dbt project, a small BigQuery control table and an orchestration layer can implement the core concepts.</p><p>The important part is that the control plane makes the intended mutation explicit.</p><h3>12. The production checklist</h3><p>Before launching a historical replay, I would want clear answers to these questions:</p><ul><li>Is the affected partition set explicit and reproducible?</li><li>Is the source snapshot or ingestion cutoff defined?</li><li>Does the selected strategy remove records that are no longer valid?</li><li>Can the process replace partitions whose correct result contains zero rows?</li><li>Are the target’s partition and clustering definitions compatible with the replay?</li><li>Can a retry safely repeat computation and publication?</li><li>Are overlapping replay and live-write operations coordinated?</li><li>Has the downstream impact window been calculated for every materialized model?</li><li>Are outputs validated before publication?</li><li>Can operators identify which partitions were rebuilt, validated and published?</li><li>Is the replay cost estimated and monitored?</li><li>Can a failed replay resume without leaving the system permanently inconsistent?</li></ul><p>If any of these questions has no clear answer, the process is not yet a fully controlled replay mechanism.</p><h3>Final thoughts</h3><p>Incremental pipelines are optimized for normal operation. Historical reprocessing tests the assumptions that normal operation allows us to ignore.</p><p>The hardest failures are not always SQL errors or failed jobs. They are successful executions that leave stale records, miss empty partitions, use inconsistent source versions or fail to propagate corrections through the dependency graph.</p><p>BigQuery partitioning and dbt incremental strategies provide the mechanisms to build efficient replay systems. They do not automatically provide the operational semantics required for safe historical recovery.</p><p>Those semantics must be designed explicitly.</p><p><strong>The key principle is simple: define the intended state transition independently of the data produced by the transformation, then validate and publish that transition under controlled conditions.</strong></p><p>Once reprocessing is treated as a first-class operation — with an explicit impact set, a reproducible source boundary, deterministic replacement semantics and downstream reconciliation — it becomes far easier to reason about failures, retry safely and control costs.</p><p>That is the difference between a pipeline that can rerun historical data and a platform that can reliably recover its historical state.</p><blockquote>🙌 Found this helpful?</blockquote><blockquote>A few claps help more people discover it.<br>If you’re interested in Data Mesh and modern data architecture, consider following me for more insights.</blockquote><blockquote>Thanks for reading!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eKEE3ThHNCfdWI4bPu8fHg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b913cf3fd847" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dbt-for-analytics/reprocessing-bigquery-at-scale-the-hidden-failure-modes-of-incremental-dbt-pipelines-b913cf3fd847">Reprocessing BigQuery at Scale: The Hidden Failure Modes of Incremental dbt Pipelines</a> was originally published in <a href="https://medium.com/dbt-for-analytics">dbt for Analytics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Logging in dbt: How to Standardize Logs Across Your Project with custom macro]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/dbt-for-analytics/logging-in-dbt-how-to-standardize-logs-across-your-project-with-custom-macro-c23c444761e5?source=rss----822562fe37b4---4"><img src="https://cdn-images-1.medium.com/max/1536/0*v1zq0aY6orqpdJRz" width="1536"></a></p><p class="medium-feed-snippet">Learn how to implement consistent, clear, and actionable logging in dbt using macros</p><p class="medium-feed-link"><a href="https://medium.com/dbt-for-analytics/logging-in-dbt-how-to-standardize-logs-across-your-project-with-custom-macro-c23c444761e5?source=rss----822562fe37b4---4">Continue reading on dbt for Analytics »</a></p></div>]]></description>
            <link>https://medium.com/dbt-for-analytics/logging-in-dbt-how-to-standardize-logs-across-your-project-with-custom-macro-c23c444761e5?source=rss----822562fe37b4---4</link>
            <guid isPermaLink="false">https://medium.com/p/c23c444761e5</guid>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[observability]]></category>
            <dc:creator><![CDATA[Sendoa Moronta]]></dc:creator>
            <pubDate>Fri, 09 Oct 2026 12:05:14 GMT</pubDate>
            <atom:updated>2026-10-09T12:05:16.139Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[dbt Charts Part 2: How to Build 9 Chart Types in dbt Charts]]></title>
            <link>https://medium.com/dbt-for-analytics/how-to-build-9-chart-types-in-dbt-charts-e5a6df64f71a?source=rss----822562fe37b4---4</link>
            <guid isPermaLink="false">https://medium.com/p/e5a6df64f71a</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[dbt-charts]]></category>
            <dc:creator><![CDATA[Muhammad Azhar]]></dc:creator>
            <pubDate>Mon, 05 Oct 2026 13:39:20 GMT</pubDate>
            <atom:updated>2026-11-04T11:42:08.657Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qNeTi0v1ARBV13EmdUXXnA.png" /></figure><p><a href="https://medium.com/@imagineazhar/dashboards-as-code-part-1-getting-started-with-dbt-charts-7e80502b8acd">Post 1 </a>gave you a line chart and a bar chart. A real dashboard needs different chart types. This post builds nine: bar, pie, donut, line, area, scatter, histogram, KPI, and table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rADJCdsO8X4dsqIssKAsSw.png" /></figure><p>dct has 16 authorable types. These nine cover most of what a first dashboard needs. Maps, layers, and small multiples come later in the series.</p><p>Each section starts with the query, then the chart.</p><h3>Bar: one number per category</h3><pre>queries:<br>  payment_mix:<br>    sql: |<br>      SELECT payment_method, SUM(amount) AS total_amount<br>      FROM {{ ref(&#39;stg_payments&#39;) }}<br>      GROUP BY 1<br>      ORDER BY 2 DESC<br>    source: jaffle<br><br>charts:<br>  payment_bar:<br>    query: payment_mix<br>    type: bar<br>    title: &quot;Payment method mix&quot;<br>    x: payment_method<br>    y: total_amount</pre><p>One row per bar. x is the category, y is the number. That is the whole chart. Here is what it draws:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4EConWEuKQE22RqlSwXyBA.png" /></figure><p>I did not say orientation; a bar with a text category on x is horizontal by default. The value axis runs along the bottom.</p><h3>Turn it into a vertical bar chart</h3><p>I want columns, so I say so. Add one key under the chart:</p><pre> style:<br>      orientation: vertical</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tGJoOf3asOt7XSPm2gZhdA.png" /></figure><p>The category is still x and the number is still y. Only the direction changed. dct also moved the value axis to the right and drew gridlines. Those are defaults, and the next step removes them.</p><h3>Remove the axis, label the bars</h3><p>If the bars carry their own numbers, the value axis is redundant. Three additions under style: :</p><pre> style:<br>      orientation: vertical<br>      number_format: currency_whole<br>      axis_y:<br>        title:  { visible: false }<br>        labels: { visible: false }<br>        ticks:  { visible: false }<br>        line:   { visible: false }<br>        grid:   { visible: false }<br>      marks:<br>        bar:<br>          labels:<br>            visible: true</pre><p>number_format: currency_whole formats every number on the chart as whole dollars.</p><p>There is no single switch for the axis. It has five parts, and you hide each one. axis_y always means the measure axis.</p><p>marks.bar.labels.visible: true prints each value on its bar. The labels default to white text inside the top of the bar and pick up number_format, so they read $871.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tkDB9MEzCJPWBxRZ6N9G8A.png" /></figure><h3>Turn it sideways on purpose</h3><p>A ranking is where horizontal earns its place: the names get room. This chart needs a new query, one row per customer, ten rows:</p><pre>queries:<br>  top_customers:<br>    sql: |<br>      SELECT first_name || &#39; &#39; || last_name AS customer,<br>             number_of_orders,<br>             customer_lifetime_value<br>      FROM {{ ref(&#39;customers&#39;) }}<br>      WHERE customer_lifetime_value IS NOT NULL<br>      ORDER BY customer_lifetime_value DESC<br>      LIMIT 10<br>    source: jaffle<br><br>charts:<br>  top_customers_bar:<br>    query: top_customers<br>    type: bar<br>    title: &quot;Top 10 customers by lifetime value&quot;<br>    x: customer<br>    y: customer_lifetime_value<br>    style:<br>      orientation: horizontal<br>      number_format: currency_whole</pre><p>x and y are unchanged: the category, then the number. orientation: horizontal makes the intent visible to the next person reading the file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vv4y-kQkIoj2gZNK6zyKuQ.png" /></figure><p>The largest customer is at the top because the query says ORDER BY ... DESC. The chart draws rows in the order it receives them. The value axis, axis_y, is drawn along the bottom and reads $0, $50, $100 because of number_format.</p><h3>Stack the bars</h3><p>Orientation was one key that changed how a bar looks. stack is another. Add a color: column, and the docs say dct draws the groups side by side. Set style.stack: zero , and they stack:</p><pre> weekly_status_stacked:<br>    query: orders_by_week_status   # week, status, orders<br>    type: bar<br>    x: week<br>    y: orders<br>    color: status<br>    style:<br>      stack: zero</pre><p>The query returns one row per week and status. color splits the bars by status. stack: zero piles them up from zero.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BgivWPLa619C_ltAZxhXAg.png" /></figure><p>dct labels the colours at the right edge instead of drawing a legend box (not the best choice as it reads as an axis).</p><p>This is the point of the section. “Stacked bar” and “horizontal bar” are not chart types. They are bar with different style: keys.</p><h3>Pie: shares of a whole</h3><pre>payment_pie:<br>    query: payment_mix<br>    type: pie<br>    title: &quot;Payment method mix (pie)&quot;<br>    theta: total_amount<br>    color: payment_method</pre><p>Same query as the bar. A pie swaps x and y for theta, the angle each slice takes, and color, which names the slices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XMYqZgQ2emSdvrdhQvWEjg.png" /></figure><p>I set no labels. Each slice still reads percentage, name, and value.</p><h3>Donut: a pie with a hole</h3><p>To turn the pie into a donut, add one key:</p><pre>style:<br>      inner_radius: 0.6</pre><p>inner_radius is the size of the hole as a share of the pie&#39;s radius. The docs give a range of 0 to 1, where 0 is a solid pie.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LN1u3yJPIf5tjtPKg_lsMQ.png" /></figure><p>Two things happened. The hole opened, and a number appeared in it: the sum of total_amount, labelled Total Amount after the column. I did not specify either.</p><p>type: donut is a shorthand for the same thing. If you rendered a type: pie with inner_radius: 0.6 and a bare type: donut, you will get the same results.</p><p>The label is the part you are most likely to change, and total: does that:</p><pre>total:<br>      label: Total</pre><p>Here is the finished donut:</p><pre>payment_donut:<br>    query: payment_mix<br>    type: donut<br>    title: &quot;Payment method mix&quot;<br>    theta: total_amount<br>    color: payment_method<br>    total:<br>      label: Total</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TUA2Za1yzPL-M_JqtQTJmQ.png" /></figure><p>The percentages, slice labels, and centred total came by default.</p><h3>Line: one number over time</h3><pre>queries:<br>  revenue_by_week:<br>    sql: |<br>      SELECT date_trunc(&#39;week&#39;, order_date) AS week,<br>             SUM(amount) AS revenue<br>      FROM {{ ref(&#39;orders&#39;) }}<br>      WHERE order_date &lt; (SELECT date_trunc(&#39;week&#39;, MAX(order_date)) FROM {{ ref(&#39;orders&#39;) }})<br>      GROUP BY 1<br>      ORDER BY 1<br>    source: jaffle<br><br>charts:<br>  revenue_line:<br>    query: revenue_by_week<br>    type: line<br>    title: &quot;Weekly revenue&quot;<br>    x: week<br>    y: revenue<br>    style:<br>      number_format: currency_whole</pre><p>One row per point. x is the date and y is the number, exactly as on a bar chart, with dates in place of categories.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zyVohcEsHmIBsM2V5RZlZg.png" /></figure><p>Look at the value axis. Its lowest label is 100, <em>and there is no 0</em>. The axis does not start at zero. Nothing in the YAML says so. dct chose the range to fit the data.</p><h3>Area: the same line, filled</h3><p>Change one word: Same query, same x and y.</p><pre> revenue_area:<br>    query: revenue_by_week<br>    type: area<br>    title: &quot;Weekly revenue, filled&quot;<br>    x: week<br>    y: revenue<br>    style:<br>      number_format: currency_whole</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vs6InKJA2rpbJMrZrXKNEw.png" /></figure><p>The axis now starts at $0. On the line chart, your eye follows the swings. On the area chart, it reads the height of the whole shape, so a zero baseline matters. Choose based on the question you are answering.</p><h3>Scatter: two measures, one dot per row</h3><pre>queries:<br>  customer_value:<br>    sql: |<br>      SELECT number_of_orders, customer_lifetime_value<br>      FROM {{ ref(&#39;customers&#39;) }}<br>      WHERE customer_lifetime_value IS NOT NULL<br>    source: jaffle<br><br>charts:<br>  clv_scatter:<br>    query: customer_value<br>    type: scatter<br>    title: &quot;Lifetime value vs number of orders&quot;<br>    x: number_of_orders<br>    y: customer_lifetime_value<br>    style:<br>      number_format: currency_whole</pre><p>The query returns one row per customer, not a summary. Scatter is the first chart here that wants row-level data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*muSgyWxiJGN-P5qEQPOHGg.png" /></figure><p>Order count is a whole number, so the dots line up in columns and the chart reads more like a strip plot. The docs list color: and size: for a third and fourth variable. I didn’t test here.</p><h3>Histogram:</h3><pre>queries:<br>  order_amounts:<br>    sql: |<br>      SELECT amount<br>      FROM {{ ref(&#39;orders&#39;) }}<br>      WHERE amount IS NOT NULL<br>    source: jaffle<br><br>charts:<br>  amount_histogram:<br>    query: order_amounts<br>    type: histogram<br>    title: &quot;Order amount distribution&quot;<br>    x: amount</pre><p>This is the odd one. The query returns raw rows, one per order, and the chart does the binning. There is no y-axis. dct chose ten-dollar bins and counted them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AQiRyXe7skmYzKchpjcNKw.png" /></figure><p>The vertical axis is titled Count. This is default behaviour. dct added it because the chart is counting rows.</p><p>It breaks the rule that aggregation belongs to the query. It is one of the few places the chart layer computes. I did not find how to set the bin width, so I am not claiming there is a way.</p><h3>KPI: one row, one number</h3><pre>queries:<br>  order_summary:<br>    sql: |<br>      SELECT COUNT(*) AS total_orders,<br>             SUM(amount) AS total_revenue,<br>             COUNT(DISTINCT customer_id) AS total_customers<br>      FROM {{ ref(&#39;orders&#39;) }}<br>    source: jaffle<br><br>charts:<br>  kpi_revenue:<br>    query: order_summary<br>    type: kpi<br>    label: &quot;Total revenue&quot;<br>    value: total_revenue<br>    style:<br>      value:<br>        format: currency_whole</pre><p>A KPI needs exactly one row. value: is a bare column name, never SUM(...). Three tiles are three charts reading the same query and picking a different column.</p><p>Two details differ from every other chart. The header is label:, and title: is rejected. The number format lives at style.value.format, not number_format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-gY6n43JBR2SFkD1KY-uug.png" /></figure><h3>Table: every column, in query order</h3><pre>top_customers_table:<br>    query: top_customers<br>    type: table<br>    title: &quot;Top 10 customers by lifetime value&quot;<br>    style:<br>      columns:<br>        customer:<br>          label: Customer<br>        number_of_orders:<br>          label: Orders<br>          align: right<br>        customer_lifetime_value:<br>          label: Lifetime value<br>          format: currency_whole<br>          align: right</pre><p>A table shows every column the query returns, in the order it returns them. style.columns Only styles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bNmme4MYX94i8wmrYTJaxg.png" /></figure><p><strong>Table currency formats.</strong> In the table above, format: currency_whole puts a $ on the first row only. Rows two to ten show a bare number. First row holds two &lt;tspan&gt;s ($, 99), and every other cell holds one (65, 64, 57). The same alias works on every point in a bar or scatter. I have not tested other aliases, trailing symbols, or kpi and spark_bar.</p><p>dct cannot draw: funnel, gauge, waterfall, sankey, treemap, and more.</p><h3>Next</h3><p>The board at the top of this post holds all nine chart types, thirteen charts in total, on one page. This post built the charts but didn’t explain how to set up a grid. That is the next one: how to organise a board, and how rows, cols, grid, and tabs each lay charts out. The board from this post is in <a href="https://github.com/imagineazhar/dbt-charts-learning">the repo</a>, tagged p2.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5a6df64f71a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dbt-for-analytics/how-to-build-9-chart-types-in-dbt-charts-e5a6df64f71a">dbt Charts Part 2: How to Build 9 Chart Types in dbt Charts</a> was originally published in <a href="https://medium.com/dbt-for-analytics">dbt for Analytics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[dbt Charts: What Happens When Dashboards Become Code]]></title>
            <link>https://medium.com/dbt-for-analytics/dbt-charts-what-happens-when-dashboards-become-code-6f39e691ad26?source=rss----822562fe37b4---4</link>
            <guid isPermaLink="false">https://medium.com/p/6f39e691ad26</guid>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Sendoa Moronta]]></dc:creator>
            <pubDate>Mon, 05 Oct 2026 09:19:48 GMT</pubDate>
            <atom:updated>2026-10-05T09:19:50.150Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong><em>How dbt Charts brings dashboards into Git, CI/CD, lineage, impact analysis, and AI-driven analytics</em></strong></blockquote><p>For years, the data stack has been moving in one direction.</p><p>Data transformations became code.</p><p>Tests became code.</p><p>Infrastructure became code.</p><p>Data contracts became code.</p><p>But the final layer (the dashboard) often remained somewhere else.</p><p>A developer could make a change to a dbt model, open a BI tool, manually update a visualization, publish it, and hope that nothing downstream had been missed.</p><p>dbt Charts takes a different approach.</p><p>Instead of treating a dashboard as something you configure inside a BI application, it treats the dashboard itself as a software artifact: YAML, SQL, references, layouts and visualizations stored in Git and developed alongside the dbt project.</p><p>That sounds like a change in syntax.</p><p>It is actually a change in architecture.</p><h3>From BI artifacts to executable definitions</h3><p>A traditional BI workflow often looks roughly like this:</p><pre>dbt models<br>    |<br>    v<br>warehouse<br>    |<br>    +------------------+<br>                       |<br>                       v<br>                  BI platform<br>                       |<br>              +--------+--------+<br>              |                 |<br>           Dashboard        Dashboard<br>           definition       metadata</pre><p>The transformation layer is version-controlled.</p><p>The dashboard often isn’t.</p><p>Even when the BI platform provides versioning, the dashboard’s definition usually lives in a different system, with different deployment semantics and a different dependency model.</p><p>dbt Charts moves that definition into the project:</p><pre>dbt_project/<br>│<br>├── models/<br>│   ├── staging/<br>│   ├── intermediate/<br>│   └── marts/<br>│<br>├── tests/<br>│<br>├── macros/<br>│<br>└── charts/<br>    ├── sales.yml<br>    ├── operations.yml<br>    └── executive.yml</pre><p>The chart definition is now another artifact that can be committed, reviewed, diffed and deployed.</p><p>The dbt Charts documentation describes boards as YAML files that define queries, charts and layout, with the resulting dashboards rendered from those definitions.</p><p>That immediately changes the development model.</p><p>Instead of:</p><pre>Open BI tool<br>    ↓<br>Find dashboard<br>    ↓<br>Edit visualization<br>    ↓<br>Publish</pre><p>we can have:</p><pre>Edit YAML<br>    ↓<br>Run validation<br>    ↓<br>Open Pull Request<br>    ↓<br>Review diff<br>    ↓<br>CI<br>    ↓<br>Merge<br>    ↓<br>Deploy</pre><p>And that is where dbt Charts becomes much more interesting.</p><h3>A dashboard is just another data product</h3><p>Consider a simple revenue dashboard.</p><p>The underlying dbt model might be:</p><pre>-- models/marts/monthly_revenue.sql<br><br>select<br>    date_trunc(&#39;month&#39;, order_date) as month,<br>    sum(revenue) as revenue,<br>    count(distinct customer_id) as customers<br>from {{ ref(&#39;fct_orders&#39;) }}<br>group by 1<br>order by 1</pre><p>The dashboard query can consume that model directly:</p><pre>title: &quot;Revenue Overview&quot;<br>source: db<br>queries:<br>  monthly_revenue:<br>    sql: |<br>      select<br>          month,<br>          revenue,<br>          customers<br>      from {{ ref(&#39;monthly_revenue&#39;) }}<br>      order by month</pre><p>The visualization is then declared separately:</p><pre>charts:<br>  revenue_trend:<br>    title: &quot;Monthly Revenue&quot;<br>    query: queries.monthly_revenue<br>    type: line<br>    x: month<br>    y: revenue</pre><p>And the layout references the chart:</p><pre>rows:<br>  - title: &quot;Revenue&quot;<br>    charts:<br>      revenue_trend: ...<br>    cols:<br>      - revenue_trend</pre><p>The important architectural detail is the separation:</p><pre>  DATA<br>    |<br>    v<br>dbt model<br>    |<br>    v<br>  query<br>    |<br>    v<br>  chart<br>    |<br>    v<br>  layout<br>    |<br>    v<br> dashboard</pre><p>The SQL answers:</p><blockquote><em>What data should exist?</em></blockquote><p>The chart answers:</p><blockquote><em>How should that data be represented?</em></blockquote><p>The layout answers:</p><blockquote><em>How should those visualizations be composed into an interface?</em></blockquote><p>This separation makes the dashboard definition deterministic and inspectable rather than hiding its logic behind a visual editor.</p><p>The current dbt Charts syntax explicitly models queries, charts and layout as separate concepts.</p><h3>SQL remains the data interface</h3><p>One of the things I find particularly interesting is that dbt Charts doesn’t try to hide SQL behind a semantic drag-and-drop abstraction.</p><p>You can write the query explicitly:</p><pre>queries:<br>  revenue_by_region:<br>    sql: |<br>      select<br>          region,<br>          sum(revenue) as revenue<br>      from {{ ref(&#39;fct_orders&#39;) }}<br>      where order_date &gt;= current_date - interval &#39;12 months&#39;<br>      group by region<br>      order by revenue desc</pre><p>Then describe the visualization:</p><pre>charts:<br>  regional_revenue:<br>    title: &quot;Revenue by Region&quot;<br>    query: queries.revenue_by_region<br>    type: bar<br>    x: region<br>    y: revenue</pre><p>This has a useful property:</p><p><strong>the visualization does not become a second transformation layer.</strong></p><p>The query produces a well-defined relation.</p><p>The chart consumes that relation.</p><p>That means the boundary is explicit:</p><pre>SQL<br> |<br> | columns<br> | rows<br> v<br>Chart specification<br> |<br> | visual encoding<br> v<br>Rendered visualization</pre><p>This is particularly important in data teams because the SQL remains inspectable by engineers.</p><p>You can review the query independently from the visualization.</p><h3>Git changes the unit of review</h3><p>This is where the “BI as code” idea becomes operationally interesting.</p><p>Imagine that someone changes:</p><pre>sum(revenue) as revenue</pre><p>to:</p><pre>sum(net_revenue) as revenue</pre><p>The corresponding dbt model can be reviewed in a Pull Request.</p><p>But what about the dashboards depending on that column?</p><p>With a dashboard definition living in Git, the dependency is no longer hidden inside a separate application.</p><p>The PR can contain:</p><pre>models/marts/monthly_revenue.sql<br>charts/revenue.yml</pre><p>and the reviewer can inspect both.</p><p>More importantly, dbt Charts provides tooling for understanding the impact of changes.</p><p>The dct impact command can inspect compiled SQL across boards and identify dashboards referencing a particular column, including references resolved through CTEs, aliases and subqueries. It can also report cases where the dependency cannot be determined rather than silently treating them as safe.</p><p>For example:</p><pre>dct impact customer_id</pre><p>or, when the column name is common:</p><pre>dct impact id --table customers</pre><p>This changes the question from:</p><blockquote><em>“Which dashboards do I think might use this column?”</em></blockquote><p>to:</p><blockquote><em>“Which dashboard artifacts does the project know depend on this column?”</em></blockquote><p>That is a much more useful question for a data platform.</p><h3>Impact analysis is more important than visualization</h3><p>This is probably the feature I would pay the most attention to as a Data Architect.</p><p>Consider this dependency chain:</p><pre>raw_orders<br>     |<br>     v<br>stg_orders<br>     |<br>     v<br>fct_orders<br>     |<br>     +------------------+<br>     |                  |<br>     v                  v<br>customer_metrics    revenue_metrics<br>     |                  |<br>     v                  v<br>customer board      revenue board</pre><p>Now imagine renaming:</p><pre>customer_id</pre><p>to:</p><pre>customer_key</pre><p>The obvious problem is the dbt DAG.</p><p>The less obvious problem is everything consuming the resulting relations.</p><p>With dashboards represented as code, the dependency graph becomes closer to:</p><pre>            dbt<br>             |<br>  +----------+----------+<br>  |                     |<br>  v                     v<br>models                 charts<br>  |                     |<br>  +----------+----------+<br>             |<br>             v<br>         dashboard</pre><p>The dashboard is no longer an external artifact at the end of the pipeline.</p><p>It becomes another consumer in the same software lifecycle.</p><h3>CI for dashboards</h3><p>Once dashboards are code, CI becomes possible.</p><p>A minimal pipeline can start with:</p><pre>dct validate charts/ --strict</pre><p>The CLI provides validation for YAML structure and cross-references without requiring the database for basic validation. It can also execute or inspect queries when warehouse-level validation is required.</p><p>A more complete pipeline could look like:</p><pre>name: Validate Analytics<br>steps:<br>  - name: Validate dbt<br>    run: dbt build --select state:modified+<br>  - name: Validate dashboards<br>    run: dct validate charts/ --strict<br>  - name: Check impact<br>    run: dct impact customer_id --json<br>  - name: Render<br>    run: |<br>      for board in charts/*.yml; do<br>        dct render &quot;$board&quot; \<br>          --format html \<br>          --output &quot;dist/$(basename &quot;$board&quot; .yml).html&quot;<br>      done</pre><p>The exact deployment architecture will obviously depend on the environment, but the important point is that a dashboard now participates in the same automation model as the rest of the data project.</p><p>The CLI supports validation, rendering, query inspection, impact analysis and local serving, which makes this workflow possible without requiring a graphical BI editor in the development loop.</p><h3>Rendering becomes a build step</h3><p>This leads to another interesting consequence.</p><p>A dashboard can be treated as a build artifact.</p><p>For example:</p><pre>dct render charts/sales.yml \<br>  --format html \<br>  --output dist/sales.html</pre><p>Or render other formats when needed.</p><p>That means the conceptual pipeline becomes:</p><pre>YAML<br>  |<br>  v<br>Validation<br>  |<br>  v<br>SQL<br>  |<br>  v<br>Warehouse<br>  |<br>  v<br>Chart rendering<br>  |<br>  v<br>Dashboard artifact</pre><p>This is remarkably similar to a traditional application build.</p><p>Source code:</p><pre>.py / .ts / .java<br>       |<br>       v<br>     build<br>       |<br>       v<br>   executable</pre><p>Analytics application:</p><pre>SQL + YAML<br>       |<br>       v<br>     build<br>       |<br>       v<br> dashboard</pre><p>The dashboard stops being something that only exists inside a browser session.</p><p>It becomes reproducible.</p><h3>The local development loop</h3><p>The developer experience is also intentionally CLI-oriented.</p><p>A project can be initialized with:</p><pre>dct init</pre><p>and dashboards can be served locally with:</p><pre>dct serve</pre><p>The server watches the board definitions and provides browser-based previews while the YAML is edited.</p><p>So the workflow becomes:</p><pre>                     ┌───────────────┐<br>                     │     Git       │<br>                     └───────┬───────┘<br>                             |<br>                         charts/*.yml<br>                             |<br>              +--------------+--------------+<br>              |                             |<br>              v                             v<br>        dct validate                   dct serve<br>              |                             |<br>              v                             v<br>             CI                         Browser</pre><p>This is a much more natural workflow for engineers than switching continuously between an IDE, SQL editor and BI application.</p><h3>What about the semantic layer?</h3><p>This is where I think teams need to be precise.</p><p>dbt Charts does not mean:</p><blockquote><em>“SQL is the semantic layer.”</em></blockquote><p>That’s a dangerous conclusion.</p><p>You still need to decide where business definitions belong.</p><p>For example:</p><pre>revenue</pre><p>could mean:</p><pre>sum(gross_revenue)</pre><p>or:</p><pre>sum(gross_revenue - discounts - refunds)</pre><p>or something considerably more complex.</p><p>Moving the visualization into YAML doesn’t automatically solve metric governance.</p><p>Instead, the architecture can be thought of as:</p><pre>                  Business definitions<br>                           |<br>                           v<br>                    dbt / semantic layer<br>                           |<br>                           v<br>                      SQL relation<br>                           |<br>                           v<br>                      dbt Charts<br>                           |<br>                           v<br>                     Visualization</pre><p>That distinction matters.</p><p>dbt Charts is primarily addressing the visualization and dashboard-definition layer.</p><p>It doesn’t eliminate the need for governed metric definitions.</p><h3>Where AI becomes genuinely interesting</h3><p>There is another reason this architecture matters now.</p><p>AI agents are becoming increasingly capable of writing code.</p><p>They are considerably less naturally suited to operating arbitrary graphical interfaces.</p><p>Compare these two requests:</p><blockquote><em>“Create a dashboard showing monthly revenue by region.”</em></blockquote><p>In a traditional BI system, an agent potentially has to understand a proprietary dashboard model and manipulate an application state.</p><p>With dbt Charts, the target artifact can be text:</p><pre>queries:<br>  revenue:<br>    sql: |<br>      select<br>          date_trunc(&#39;month&#39;, order_date) as month,<br>          region,<br>          sum(revenue) as revenue<br>      from {{ ref(&#39;fct_orders&#39;) }}<br>      group by 1, 2<br><br>charts:<br>  revenue_by_region:<br>    title: &quot;Monthly Revenue by Region&quot;<br>    query: queries.revenue<br>    type: line<br>    x: month<br>    y: revenue<br>    series: region</pre><p>An agent can generate it.</p><p>A human can read it.</p><p>Git can diff it.</p><p>CI can validate it.</p><p>And the same agent can modify it after receiving feedback.</p><p>This is a very different model from:</p><pre>User → AI → mysterious dashboard state</pre><p>It becomes:</p><pre>User<br>  |<br>  v<br>AI agent<br>  |<br>  v<br>YAML + SQL<br>  |<br>  v<br>Git diff<br>  |<br>  v<br>Validation<br>  |<br>  v<br>Dashboard</pre><p>dbt Charts explicitly positions YAML as an AI-friendly representation and provides CLI/MCP tooling intended for AI-assisted workflows.</p><p>The important part isn’t that AI can write YAML.</p><p>AI can write many things.</p><p>The important part is that <strong>the generated artifact remains reviewable</strong>.</p><h3>A better mental model</h3><p>I would therefore avoid thinking about dbt Charts as:</p><pre>&quot;another BI tool&quot;</pre><p>A more useful model is:</p><pre>DATA PLATFORM<br>Sources<br>   |<br>   v<br>Transformations<br>   |<br>   v<br>Governed models<br>   |<br>   v<br>Queries<br>   |<br>   v<br>Visualization definitions<br>   |<br>   v<br>Dashboard</pre><p>with Git and CI surrounding the whole thing:</p><pre>┌──────────── Git / CI ────────────┐<br>│                                  │<br>│   Models → Queries → Charts      │<br>│       ↓         ↓        ↓       │<br>│   Validation  Tests    Impact    │<br>│                                  │<br>└──────────────────────────────────┘</pre><p>That is the real shift.</p><p>Not:</p><blockquote><em>dashboards written in YAML instead of clicked into existence.</em></blockquote><p>But:</p><blockquote><em>dashboards becoming part of the software lifecycle of the data platform.</em></blockquote><h3>What I would watch carefully</h3><p>This architecture also introduces new questions.</p><p>Who owns the dashboard definitions?</p><p>Who reviews visualization changes?</p><p>Where do metric definitions live?</p><p>How are access controls enforced?</p><p>How do you promote dashboards across environments?</p><p>How do you prevent a dashboard query from becoming an ungoverned transformation layer?</p><p>How do you test visualizations semantically rather than syntactically?</p><p>Those questions don’t disappear because the dashboard is now code.</p><p>In fact, they become more explicit.</p><p>And that is probably a good thing.</p><h3>The interesting part isn’t the chart</h3><p>dbt Charts is easy to describe as a YAML-based dashboarding tool.</p><p>That undersells what is interesting about it.</p><p>The more important change is that it brings a previously semi-managed artifact into the same engineering lifecycle as the data models that feed it.</p><p>You can now think about analytics as:</p><pre>             CODE<br>               |<br>      +--------+---------+<br>      |                  |<br>      v                  v<br>   DATA LOGIC       PRESENTATION<br>      |                  |<br>      v                  v<br>    dbt SQL          Chart YAML<br>      |                  |<br>      +--------+---------+<br>               |<br>               v<br>              Git<br>               |<br>               v<br>              CI<br>               |<br>               v<br>           Deployment</pre><p>And that creates something BI has historically struggled with:</p><p><strong>a reproducible, reviewable and machine-readable definition of the analytical interface.</strong></p><p>The chart itself is almost the least interesting part.</p><p>The interesting part is what happens when the dashboard becomes code.</p><p>Because once that happens, you can review it.</p><p>Test it.</p><p>Version it.</p><p>Generate it.</p><p>Analyze its dependencies.</p><p>And eventually let agents modify it without turning your BI layer into an opaque black box.</p><p>That is a much bigger architectural shift than changing how we draw a bar chart.</p><blockquote>🙌 Found this helpful?</blockquote><blockquote>A few claps help more people discover it.<br>If you’re interested in Data Mesh and modern data architecture, consider following me for more insights.</blockquote><blockquote>Thanks for reading!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eKEE3ThHNCfdWI4bPu8fHg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f39e691ad26" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dbt-for-analytics/dbt-charts-what-happens-when-dashboards-become-code-6f39e691ad26">dbt Charts: What Happens When Dashboards Become Code</a> was originally published in <a href="https://medium.com/dbt-for-analytics">dbt for Analytics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Column-Level Lineage and Data Quality in dbt with SQLGlot and Elementary]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/dbt-for-analytics/linaje-de-columna-y-calidad-de-datos-en-dbt-con-sqlglot-elementary-36f0c80b8421?source=rss----822562fe37b4---4"><img src="https://cdn-images-1.medium.com/max/1536/1*sOkoWIUV6zPSfIXiVJV9Bg@2x.jpeg" width="1536"></a></p><p class="medium-feed-snippet">Enhancing Data Observability with Column-Level Lineage and Automated Quality Monitoring</p><p class="medium-feed-link"><a href="https://medium.com/dbt-for-analytics/linaje-de-columna-y-calidad-de-datos-en-dbt-con-sqlglot-elementary-36f0c80b8421?source=rss----822562fe37b4---4">Continue reading on dbt for Analytics »</a></p></div>]]></description>
            <link>https://medium.com/dbt-for-analytics/linaje-de-columna-y-calidad-de-datos-en-dbt-con-sqlglot-elementary-36f0c80b8421?source=rss----822562fe37b4---4</link>
            <guid isPermaLink="false">https://medium.com/p/36f0c80b8421</guid>
            <category><![CDATA[elementary]]></category>
            <category><![CDATA[sqlglot]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[dbt]]></category>
            <dc:creator><![CDATA[Sendoa Moronta]]></dc:creator>
            <pubDate>Mon, 28 Sep 2026 09:01:02 GMT</pubDate>
            <atom:updated>2026-09-28T09:01:03.366Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Migrating dbt Core Projects to dbt Fusion with the Official dbt Agent Skill: An Engineering…]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/dbt-for-analytics/migrating-dbt-core-projects-to-dbt-fusion-with-the-official-dbt-agent-skill-an-engineering-d741f153d536?source=rss----822562fe37b4---4"><img src="https://cdn-images-1.medium.com/max/1536/1*0QKDpNw9k6u5_88GFj0y1w.png" width="1536"></a></p><p class="medium-feed-snippet">Beyond a Version Upgrade: Understanding an AI-Assisted Migration to dbt Fusion</p><p class="medium-feed-link"><a href="https://medium.com/dbt-for-analytics/migrating-dbt-core-projects-to-dbt-fusion-with-the-official-dbt-agent-skill-an-engineering-d741f153d536?source=rss----822562fe37b4---4">Continue reading on dbt for Analytics »</a></p></div>]]></description>
            <link>https://medium.com/dbt-for-analytics/migrating-dbt-core-projects-to-dbt-fusion-with-the-official-dbt-agent-skill-an-engineering-d741f153d536?source=rss----822562fe37b4---4</link>
            <guid isPermaLink="false">https://medium.com/p/d741f153d536</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[analytics-engineering]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[dbt]]></category>
            <dc:creator><![CDATA[Sendoa Moronta]]></dc:creator>
            <pubDate>Thu, 17 Sep 2026 10:20:12 GMT</pubDate>
            <atom:updated>2026-09-17T10:20:14.114Z</atom:updated>
        </item>
    </channel>
</rss>