<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Santosh Mehtre on Medium]]></title>
        <description><![CDATA[Stories by Santosh Mehtre on Medium]]></description>
        <link>https://medium.com/@santosh_mehtre?source=rss-958f880d0a9d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*JXW6n2T4SOgw8yyerefXNw.jpeg</url>
            <title>Stories by Santosh Mehtre on Medium</title>
            <link>https://medium.com/@santosh_mehtre?source=rss-958f880d0a9d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 04:23:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@santosh_mehtre/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data in Motion or Data at Rest: A Practical Framework for When to Test Your Data]]></title>
            <link>https://medium.com/@santosh_mehtre/data-in-motion-or-data-at-rest-a-practical-framework-for-when-to-test-your-data-d044311273ad?source=rss-958f880d0a9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d044311273ad</guid>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[governance]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Santosh Mehtre]]></dc:creator>
            <pubDate>Wed, 10 Jun 2026 00:27:09 GMT</pubDate>
            <atom:updated>2026-06-10T00:27:09.510Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>Most teams put their data quality checks in the wrong place. Here is how I decide where each test actually belongs.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DAL6Co9XoO_rZ89cG71sfg.jpeg" /><figcaption>Data in motion/rest ?</figcaption></figure><h4><strong>The decision nobody wants to make at the wrong time</strong></h4><p>Two weeks to launch. The data product on the screen feeds a number an executive will say out loud on an earnings call — which means a wrong value here does not get quietly patched, it gets <em>reported</em>, to people who do not forgive reported numbers. Events stream in from a dozen sources, a Spark job reshapes them, and the result lands in a warehouse table the entire company is about to build on top of.</p><p>Then the staff engineer asks the question that stops the room cold. <em>Where do we put the quality check?</em></p><p>One camp leans in: test it <strong>in motion</strong>— interrogate every event the instant it lands, reject the bad ones at the door, never let a corrupt record cross the threshold. The other pushes back hard: naive — test it <strong>at rest</strong>, let the data settle, then check the full table where you can actually see the duplicates, the totals that do not add up, the trend that went sideways at 2 a.m. Voices rise. Both engineers are senior. Both have scar tissue. Both are convinced. And the clock does not care, because in two weeks this ships either way.</p><p>What makes it genuinely hard is that whoever loses is not just wrong in a meeting — they are signing the team up for a specific future pain: either bad data blocked so aggressively that good pipelines stall, or bad data waved through so politely that it surfaces six months later as a silently wrong dashboard nobody catches until a customer does.</p><p>I have been both engineers in that room. The hard-won lesson is that “in motion or at rest” is the wrong thing to be shouting about. The right question is quieter: <em>which check is this — and therefore where does it belong?</em> There is a framework for answering that every time, and once you have it, the argument dissolves.</p><h4><strong>Two places to test: motion versus rest</strong></h4><p>Before the framework, we need clean definitions — because half of these arguments are really two people using the same words to mean different things. Let me ground it in something every one of us has dealt with: tax. Say each order in your system carries a tax_amount, and the business cannot afford to get it wrong.</p><p><strong>Data in motion </strong>is data on its way somewhere — an order event flowing through a queue, a record moving down a streaming pipeline, a row mid-transformation between source and destination. Testing in motion means inspecting each record <em>as it passes</em>, at a boundary, before it is allowed to continue. The defining trait is that nothing has landed yet. So for our tax field, a test in motion asks the questions you can answer one record at a time: is tax_amount present, is it a non-negative number, is it less than the order total? A malformed order arrives at the door, you check its papers, and you turn it away before it ever steps inside.</p><p><strong>Data at rest</strong> is data that has already arrived and is sitting in a table, a file, a partition — stored, queryable, whole. Testing at rest means running checks <em>after </em>it has landed, with the whole population in front of you. This is where the interesting tax questions live. Does the total tax collected today reconcile with what finance computed independently? Are there two tax entries for the same order ID? Did the effective tax rate jump from 8% to 18% at 2 a.m. because somebody fat-fingered a config? None of these can be answered at the door — you need the full table even to ask them.</p><p>Notice the trade already forming. In motion, you can <em>prevent</em>, but you see one record at a time. At rest, you can see <em>everything</em>, but the data is already inside. Hold that tension, because it is the whole framework.</p><h4><strong>Why teams default to the wrong place</strong></h4><p>Here is the uncomfortable pattern I keep seeing: when in doubt, teams test at rest. Almost everything becomes a check that runs against the warehouse table after the data has landed — not because anyone reasoned their way there, but because testing at rest is the path of least resistance.</p><p>The reasons are entirely human. At-rest checks are just SQL, and everyone on a data team can write SQL. The data is already sitting there — nothing to intercept, no stream to tap, no producer to coordinate with. And the tooling pushes the same way: dbt tests, warehouse assertions, scheduled quality jobs all run naturally on tables. So the warehouse quietly becomes where all quality lives, by default rather than by design.</p><p>The cost is subtle but real. A check at rest can only <em>detect,</em> by the time it runs, the bad tax value has already landed and very likely already been read by something downstream. You have built a smoke detector that, by construction, only goes off once the room is full of smoke. That is not useless — you do want to know the room is on fire — but you have given up the one thing motion offered: the chance to stop it at the door.</p><h4><strong>The framework: four questions</strong></h4><p>When the room is arguing, it is because everyone is debating <em>the answer</em> before agreeing on <em>the questions</em>. So here are the four I ask of any check, in order. Each one pulls the test toward motion or toward rest, and together they settle it.</p><p><strong>1. Can the check even be answered one record at a time?</strong> This is the first filter, because it overrides everything else. “Is tax_amountnon-negative?” needs nothing but the record in hand — it <em>can </em>live in motion. “Does today’s total tax reconcile with finance’s number?” is meaningless for a single row; it needs the whole population and therefore <em>must </em>live at rest. A surprising number of arguments end right here: the check simply cannot go where someone wants it to.</p><p><strong>2. Do you need to block, or only to know?</strong> If a bad record must never reach consumers — a negative tax that would corrupt a customer invoice — you need to <em>prevent</em>, and only motion can stop a record at the door. If you merely need to be alerted that something drifted, detection at rest is enough. Blocking is powerful, but it has teeth: a check in motion that is too strict will halt your pipeline at 2 a.m. over a record that was merely unusual, not wrong.</p><p><strong>3. How fast does acting late get expensive?</strong> Some bad data is cheap to catch late — you find it at rest in the morning, fix the row, move on. Other bad data compounds: it lands, a dozen downstream jobs read it, dashboards cache it, a model trains on it. The faster the blast radius grows, the more the check earns its place in motion, before the spread begins.</p><p><strong>4. What does it cost to recompute?</strong> If reprocessing a bad partition is a cheap, idempotent backfill, you can afford to catch problems at rest and rerun. If your pipeline is expensive, stateful, or hard to replay, letting bad data land is a costly mistake — and paying for a check in motion up front is the bargain.</p><p>Put them together and the rule falls out on its own. <strong>Per-record, must-block, fast-blast-radius, expensive-to-redo → test it in motion. Cross-dataset, detect-and-alert, cheap-to-rerun → test it at rest.</strong> Most real checks answer cleanly once you ask in this order — and the two engineers who were shouting discover they were each describing a <em>different</em> check the whole time.</p><h4><strong>What belongs in motion</strong></h4><p>Apply the four questions and a clear category emerges. Tests in motion are the ones that are answerable per record, that you need to <em>block</em>, where late detection is expensive. In practice that means three families of checks.</p><p><strong>Structural and schema checks.</strong>Is the record even shaped correctly — required fields present, types right, the contract honoured? This is the renamed-column class of failure, and it belongs at the door. A malformed record is never going to become valid by sitting in a table.</p><p><strong>Per-record sanity rules.</strong>The single-row invariants: tax_amountis non-negative, is a number, does not exceed the order total, the currency is one you support. None of these need neighbours to be evaluated, and every one of them is cheaper to reject now than to hunt down later.</p><p><strong>Boundary freshness and provenance.</strong> Did this event arrive from a source you trust, within the window you expect? Checks that guard the edge of your system belong on the edge.</p><p>In a PySpark pipeline, an in-motion check is often a split driven by a named rule set: valid records flow on, bad ones are routed away before they can land.</p><pre>from pyspark.sql import functions as F<br><br># in-motion rule set: each rule is named, so a failure tells you WHICH one broke<br>RULES = {<br>    &quot;tax_present&quot;:      F.col(&quot;tax_amount&quot;).isNotNull(),<br>    &quot;tax_non_negative&quot;: F.col(&quot;tax_amount&quot;) &gt;= 0,<br>    &quot;tax_within_total&quot;: F.col(&quot;tax_amount&quot;) &lt;= F.col(&quot;order_total&quot;),<br>    &quot;currency_known&quot;:   F.col(&quot;currency_code&quot;).isin(&quot;USD&quot;, &quot;EUR&quot;, &quot;GBP&quot;),<br>}<br><br># apply every rule in one place; tag each record with the list of rules it failed<br>failed = F.array_remove(<br>    F.array(*[F.when(~cond, F.lit(name)) for name, cond in RULES.items()]),<br>    None,<br>)<br>checked = events.withColumn(&quot;_failed_rules&quot;, failed)<br><br>clean    = checked.filter(F.size(&quot;_failed_rules&quot;) == 0).drop(&quot;_failed_rules&quot;)<br>rejected = checked.filter(F.size(&quot;_failed_rules&quot;) &gt; 0)   # keeps WHY it failed<br><br>rejected.writeStream.format(&quot;delta&quot;).option(&quot;path&quot;, &quot;/quarantine/tax&quot;).start()<br>clean.writeStream.format(&quot;delta&quot;).option(&quot;path&quot;, &quot;/curated/orders&quot;).start()</pre><p>The pattern is the important part, not the syntax. The rules live in one named dictionary, so adding a check is a one-line change and a quarantined record carries failed_rules telling you exactly <em>which </em>rule it broke — no guessing at 2 a.m. The bad tax record never reaches the curated table; it is set aside the instant it fails, where you can inspect it without it polluting anything downstream. That is the whole promise of testing in motion — the problem is contained at the doorway, not discovered in the morning.</p><h4><strong>What belongs at rest</strong></h4><p>Now the other category: checks the framework pushes the other way — ones that <em>cannot </em>be answered a record at a time, where detect-and-alert is enough and a rerun is cheap. These need the whole population in the room.</p><p><strong>Cross-record and cross-dataset checks</strong> — uniqueness, reconciliation, referential integrity. “Are there two tax entries for one order?” or “does our total tax match finance’s ledger?” are invisible to a single record and obvious to a full table.</p><p><strong>Aggregate and statistical checks </strong>— counts, sums, distributions, day-over-day drift. The effective tax rate jumping from 8% to 18% is wrong on no single row; the <em>aggregate</em> betrays the bug.</p><p>These are what SQL was made for:</p><pre>-- only answerable with the full table(s) in front of you<br>-- duplicate tax entries per order<br>SELECT order_id, COUNT(*) AS n<br>FROM curated.orders<br>GROUP BY order_id HAVING COUNT(*) &gt; 1;<br><br>-- our tax total reconciles with finance&#39;s independent ledger, within tolerance<br>SELECT o.tax_total, f.tax_total<br>FROM (SELECT SUM(tax_amount)   AS tax_total FROM curated.orders     WHERE order_date  = CURRENT_DATE) o<br>CROSS JOIN<br>     (SELECT SUM(tax_collected) AS tax_total FROM finance.tax_ledger WHERE ledger_date = CURRENT_DATE) f<br>WHERE ABS(o.tax_total - f.tax_total) &gt; 1.00;   -- fails if off by more than a dollar</pre><p>In a dbt or scheduled-job setup, each query is a test that fails loudly when it returns rows. The reconciliation is the clearest case for why rest exists at all — it compares two entire datasets, and no per-record check in motion could ever catch a quiet drift between them. That is the strength of testing at rest: not prevention, but the panoramic view that catches what no single record could reveal.</p><h4><strong>Stop choosing, start layering</strong></h4><p>So walk back into that design review. The two senior engineers are still arguing, and now you can hand them the resolution: they were never actually disagreeing. One was describing a per-record, must-block check — `tax_amount` cannot be negative — which belongs in motion. The other was describing a cross-dataset reconciliation against finance — which belongs at rest. Both checks should exist. They just live in different places, doing different jobs.</p><p>That is the real answer to “motion or rest.” It is not a choice; it is a sequence. Validate each record at the door so the obvious garbage never lands. Then test the assembled dataset once it has, so the failures only the full picture reveals still get caught. Motion contains, rest reconciles. A mature pipeline does both, and stops pretending it has to pick.</p><p>If you want somewhere to start, take your single most painful data incident and ask the four questions of it. Could the check have run per record? Did you need to block? How fast did it spread? Was the rerun expensive? The answers will tell you, without a shouting match, exactly where that test should have lived — and where to put the next one.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d044311273ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[git-town: The Tool That Made Git Click for Me]]></title>
            <link>https://medium.com/@santosh_mehtre/git-town-the-tool-that-made-git-click-for-me-cb7b5012a78d?source=rss-958f880d0a9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb7b5012a78d</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[developer-tools]]></category>
            <dc:creator><![CDATA[Santosh Mehtre]]></dc:creator>
            <pubDate>Sun, 24 May 2026 16:38:44 GMT</pubDate>
            <atom:updated>2026-05-24T16:38:44.004Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0EV9Skbhp02hlqbOuGgUDA.jpeg" /></figure><p><em>A short, honest review of the only git tool I have stuck with — and why every developer should at least try it.</em></p><p>Let me be honest. For years, git intimidated me.</p><p>Not the basics — add, commit, push, pull — those were fine. But the moment things got real? Stacked branches, syncing with main, rebasing without losing work, cleaning up after a messy PR review — every one of those was a minute of staring at the terminal, googling the exact flag, copy-pasting from Stack Overflow, and hoping I did not destroy my branch.</p><p>I am not a junior engineer. I have been doing this for over a decade. But git’s branch model has always felt like a tax I paid every single day, just to get back to actually writing code.</p><p>Then I found <strong><em>git-town</em></strong>. And I stopped paying that tax.</p><h4><strong>What Actually Changed</strong></h4><p>git-town did not replace git. That is the first thing to understand. It sits <strong>on top</strong> of git, and it draws a clean line between two kinds of work I used to mash together in my head.</p><p>Branches and pull requests are one kind of work. Editing code and committing changes is another. Plain git tries to handle both, and that is exactly why it feels overwhelming for so many of us.</p><p>git-town lets you split them cleanly:</p><p>- <strong>git-town handles branches </strong>— creating them, syncing them with main, stacking them on top of each other, opening PRs, shipping them, deleting them when done.</p><p>- <strong>Plain git handles your actual work</strong> — add, commit, push, pull, status, log, diff. The same commands you already know.</p><p>And once branches become first-class, things that are painful in plain git suddenly become trivial.</p><p>Want a <strong>child branch on top of your current one</strong>— a stacked PR that depends on the previous one? git town append child-feature. Done. The new branch knows its parent and how to sync in the right order.</p><p>Need to keep your whole stack of three or four branches up to date with main, all at once? git town sync — all. One command. Every branch in the stack, in the correct order, automatically. No cherry-picking. No rebase chains. No <strong>wait, which branch was I on?</strong></p><p>Once that split clicked in my head, the mental load dropped overnight. I stopped fearing rebases. I just typed git town hack my-feature, wrote my code, and let the tool handle the rest.</p><h4><strong>The Five Commands I Use Every Day</strong></h4><p>Here is my entire branch workflow now. Five commands. That is it.</p><p>bash</p><p># Start a new feature (branches off latest main automatically)</p><p>git town hack add-paymnet-flow</p><p># Keep your branch fresh (pulls main, rebases your branch, pushes)</p><p>git town sync</p><p># Jump between branches (interactive picker that shows the branch tree)</p><p>git town switch</p><p># Open a pull request (browser opens, correct base branch already filled in)</p><p>git town propose</p><p># Done — merge into parent, delete branch locally and remotely, sync main</p><p>git town ship</p><p>hackto start. sync whenever I want to be safe. switch when I have a stack going. propose when I am ready for review. ship when it is merged.</p><p>The genius is in what is <strong>not</strong> here. No git checkout -b. No git pull — rebase. No git push — set-upstream origin HEAD. No remembering which branch is the parent of which one. The tool knows the answer to every one of those questions, so I do not have to.</p><p>And the one I quietly love the most: <strong>git town undo</strong>. Made a mistake? Synced the wrong branch? Shipped too early? git town undo reverses the last git-town operation cleanly. It is the safety net that lets me move fast without thinking twice.</p><h4><strong>Try It</strong></h4><p>If any of this resonates — if branches feel like a daily tax, if rebasing makes you nervous, if your git push muscle memory has more flags than commands — give git-town a week on a side project.</p><p>bash</p><p># macOS</p><p>brew install git-town</p><p>git town config setup # one-time interactive wizard</p><p>git town hack first-feature # you are ready to go</p><p>That is the whole onboarding. One install, one setup wizard, one branch.I have not typed git checkout -b, git pull — rebase, or git push — set-upstream in months. I do not miss them. You will not either.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb7b5012a78d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[# dbt Was Good. SQLMesh Is Better. Here Is Why.]]></title>
            <link>https://medium.com/@santosh_mehtre/dbt-was-good-sqlmesh-is-better-here-is-why-aef0d0d92f6a?source=rss-958f880d0a9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/aef0d0d92f6a</guid>
            <category><![CDATA[analytics-engineering]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[modern-data-stack]]></category>
            <category><![CDATA[sqlmesh]]></category>
            <dc:creator><![CDATA[Santosh Mehtre]]></dc:creator>
            <pubDate>Sat, 23 May 2026 23:17:38 GMT</pubDate>
            <atom:updated>2026-05-23T23:17:38.159Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UPN90siedSjZa6Qf1QCfCQ.jpeg" /></figure><p><em>After shipping both to production, the verdict is not even close on the things that matter.</em></p><p>Let me tell you honestly. When I first started with dbt, I was excited. Everyone in the data world was talking about it. So I jumped in.</p><p>And then I spent two weeks just learning the vocabulary.</p><p>ref(), source()models, seeds, snapshots, materializations — table or view or incremental or ephemeral? Schema YAMLs, profiles, targets, packages, macros. Before writing one useful transformation, I had to internalize a whole framework. None of it was hard. There was just a lot of it. And every new engineer who joined the team went through the same two-week ramp-up only.</p><p>I made peace with that. Onboarding tax — you pay it once, life moves on.</p><p>But here is the part that really broke me. <strong>dbt is stateless.</strong>In any lower environment, the tool does not remember what changed since last run. If I touched one model, I could not just test that one model — I had to run the whole DAG end-to-end, every single time, burning warehouse credits and hours of engineer patience to verify a two-line change.</p><blockquote>A modern transform tool in 2026 that cannot tell me what changed since last run? No.</blockquote><p>Then I tried SQLMesh. State-aware by design. Virtual environments that let me test only what I changed, in seconds. Free, open-source lineage in the box — no dbt Cloud subscription needed. I am not going back.</p><h4><strong>So What Are We Comparing?</strong></h4><p>Quick level-set for anyone new to either tool.</p><p><strong>dbt</strong> is the transform layer that defined the modern data stack. SQL plus Jinja, models as .sqlfiles, lineage through ref() and source(). dbt Labs built it, and it made analytics engineering a real discipline. Your warehouse is the compute, dbt is the framework that turns raw tables into reliable, tested, version-controlled data products. For five years, it was the default. Rightly so.</p><p><strong>SQLMesh </strong>is the next-generation transform tool from Tobiko Data. SQL-first like dbt, but the philosophy is different. It uses a plan and apply workflow like Terraform — describe the change, see the impact, then apply. It is built on SQLGlot, which means it actually parses your SQL instead of just templating strings. Virtual environments, state awareness, automatic backfills, and column-level lineage all ship in the box — free, native, no paid tier needed.</p><p>This is not a dbt-is-bad article. dbt did its job, beautifully. SQLMesh is just solving the next set of problems — the ones that hurt once your dbt project crosses 200 models.</p><h4><strong>Virtual Environments: The Feature That Changes Everything</strong></h4><p>This is the feature that, by itself, justifies trying SQLMesh.</p><p>Remember the problem I described at the top? In dbt, when you want a dev environment, you build it. Properly build it. dbt run --target dev walks your DAG and materializes every model into your dev schema. For a small project, fine. For a 200-model project on Snowflake or BigQuery, you are looking at long minutes — sometimes hours — and a real warehouse bill, every single time.</p><p>So what do most teams do? They cut corners. They share one “dev” environment across the whole team. They run — select my_model+ and hope they caught every downstream. They stop testing properly because testing properly is too expensive.</p><p>SQLMesh fixes this with virtual environments. The idea is simple but the implementation is elegant. When you run sqlmesh plan dev, it does not rebuild your tables. It creates a new schema where everything is a <strong>view pointing to existing production tables</strong>, except for the models you actually changed. Those get materialized in the background. Everything else stays as a cheap view over prod data.</p><p># dbt — build a dev environment</p><p>$ dbt run — target dev</p><p>… 47 models, 12 minutes</p><p># SQLMesh — create a dev environment</p><p>$ sqlmesh plan dev</p><p>virtual environment ready in seconds. Only changed models recomputed.</p><p>A new engineer can spin up their own dev environment in seconds. CI can create one per pull request without anyone flinching at the bill. You can test only what you changed — exactly what I wanted from dbt in the first place.</p><p><em>This is not a nice-to-have feature. Once you work this way, going back feels primitive.</em></p><p><strong>SQLMesh’s Execution Plan Is Much Better Than dbt’s</strong></p><p>Here is a concrete example from BigQuery that made me appreciate SQLMesh more than any marketing page could.</p><p>Take a daily-partitioned table — say, an orders table partitioned on order_date. You want partition-overwrite loads: every run wipes a day’s partition and writes it fresh. Standard pattern. Both tools can do it. Watch what each one actually sends to the warehouse.</p><p><strong>dbt with </strong><strong>insert_overwrite:</strong></p><p>sql:</p><p>{{ config(</p><p>materialized=’incremental’,</p><p>incremental_strategy=’insert_overwrite’,</p><p>partition_by={‘field’: ‘order_date’, ‘data_type’: ‘date’}</p><p>) }}</p><p>select order_id, customer_id, order_total, order_date</p><p>from {{ source(‘raw’, ‘orders’) }}</p><p>where order_date = current_date()</p><p><strong><em>First run</em></strong>, when the partition does not exist yet, dbt does the simple thing:</p><p>sql:</p><p>INSERT INTO analytics.orders SELECT * FROM raw.orders WHERE …;</p><p><strong><em>Second run</em></strong>, when the same partition needs to be overwritten, dbt does a four-step dance:</p><p>sql:</p><p>1. Stage everything into a temp table</p><p>CREATE TABLE analytics.orders__tmp AS SELECT * FROM raw.orders WHERE …;</p><p>2. Delete the partition from the target</p><p>DELETE FROM analytics.orders WHERE order_date = current_date();</p><p>3. Insert from temp into target</p><p>INSERT INTO analytics.orders SELECT * FROM analytics.orders__tmp;</p><p>4. Drop the temp table</p><p>DROP TABLE analytics.orders__tmp;</p><p>Two different execution plans for what is functionally the same load. Extra scans, extra storage, extra cost — every time the partition already exists.</p><p><strong>SQLMesh, same scenario, every single run:</strong></p><p>sql:</p><p>DELETE FROM analytics.orders WHERE order_date BETWEEN @start_ds AND @end_ds;</p><p>INSERT INTO analytics.orders SELECT * FROM raw.orders WHERE …;</p><p>Two queries. No temp table. No conditional branching. Same plan first run, hundredth run, thousandth run — exactly the pattern BigQuery’s own documentation recommends for partition overwrites.</p><blockquote>dbt’s approach works. But “<strong>works</strong>” and “<strong>efficient</strong>” are not the same thing.</blockquote><p>On a table that runs every hour with partition overwrites, that temp-table dance is burning BigQuery slot-hours you did not need to burn. SQLMesh just does the right thing — once, consistently, the way the warehouse actually wants it done.</p><h4><strong>SQLMesh Actually Understands Your SQL</strong></h4><p>Here is a thing most people do not realize about dbt: it does not understand SQL. Not really. Jinja is a templating engine. It takes your .sql file, expands the ref()and source() macros into table names, and ships the resulting string to the warehouse. If your SQL is broken, the warehouse tells you. At runtime. After your CI has already spent ten minutes building the upstream models.</p><p>SQLMesh is built on SQLGlot — a real SQL parser. When you run sqlmesh plan, the tool reads your model, parses it into an AST, walks the column references, checks them against the upstream schemas it already knows about, and tells you exactly what is going to break. Before any query touches the warehouse.</p><p>Concrete example. You rename customer_idto cust_idin a staging model. In dbt, you find out which downstream models broke when their next run fails — or worse, when somebody notices the dashboard is wrong. In SQLMesh, you find out the moment you type sqlmesh plan. Every downstream model that references the old column lights up in the plan output, with line numbers.</p><blockquote>This is the difference between “I think my change works” and “I know my change works.”</blockquote><h4><strong>Where dbt Still Wins (and Why You Might Stay)</strong></h4><p>I owe you honesty. SQLMesh is not pure upside.</p><p><strong>Ecosystem:</strong> dbt has a decade of head start. The package registry, the adapters for every warehouse you can name, the integrations with every BI tool and data-quality vendor — that is real. SQLMesh is still catching up. If your stack depends on five dbt packages and three vendor integrations that only ship dbt adapters, that friction is not going away tomorrow.</p><p><strong>Hiring:</strong> Every analytics engineer on the market knows dbt. The number who have used SQLMesh in production is still small. Your next hire is going to need ramp-up on something they have not seen before.</p><p><strong>Community:</strong> Stack Overflow answers, Discourse threads, YouTube tutorials, conference talks — there is a decade of compounding documentation for dbt. SQLMesh’s own docs are good, but Google still helps you faster with dbt.</p><p><strong>Maturity and trust:</strong> dbt has been in the market for a long time. When something breaks in production at midnight, the odds are very good that someone else hit the exact same problem two years ago and posted the fix. There is community support waiting for any concern or query you can throw at it. SQLMesh’s community is growing fast and the team behind it is responsive, but that kind of accumulated trust takes years to build. You feel the gap the first time you Google an obscure error and get back three results instead of three hundred.</p><p>So my honest advice: if your team is happy and productive on dbt today, do not migrate just because SQLMesh is technically cooler. Migration is real work. The cost only pays back if the pain points I described above are actually pain points for <em>you</em>.</p><h4><strong>So Where Does That Leave You?</strong></h4><p>Two scenarios, two clear answers.</p><p><strong>Starting a new data project today? </strong>Pick SQLMesh. Honestly. You will spend a fraction of the time on incremental gotchas, you will get a real dev environment in seconds instead of hours, and you will catch breakages before they ever hit production. The ecosystem gap is real but shrinking fast, and you will not be carrying years of dbt habits to unlearn later.</p><p><strong>Already running a healthy dbt project?</strong> Do not migrate everything tomorrow. Pick one domain — one logical slice of your warehouse, ideally one with painful incremental loads or expensive dev rebuilds — and pilot SQLMesh there. Run both for a quarter. Compare warehouse bills. Compare iteration speed. Compare how many production incidents each tool absorbs versus causes. Let the numbers decide.</p><p>The trajectory matters more than today’s feature parity. dbt is improving. SQLMesh is improving faster, and from a stronger architectural starting point. That direction is what made me commit. And that is why I am not going back.</p><p>If you have your own war stories — dbt or SQLMesh — drop them in the comments. The more honest, battle-tested takes we have out in the open, the better decisions everyone gets to make.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aef0d0d92f6a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>