dbt on Databricks: Strategic Asset or Redundant Control Plane?

Apache Spark 4.1 retired the portability argument of dbt on 16th of December 2025. The decision is worth re-running, not reversing.

On that date Apache Spark 4.1 shipped with Declarative Pipelines under Apache 2.0 [1]. Databricks had contributed the framework to Apache six months earlier, built on what it learned from Delta Live Tables and rewritten as a Spark-native open-source API [2]. Five and a half months later, on 1 June 2026, Fivetran closed its merger with dbt Labs [3].

So the bottom line for a Databricks-first platform: dbt is still good software but the reason most teams chose it is mostly gone: portability.

The architecture facts and the ownership facts moved in the same fourteen months. Only the upper track changes the technical decision. Sources: [1][2][3][15][16][28]

dbt Was Adopted as a Hedge Against the Platform

The decision usually runs like this. Databricks owns the compute and the catalog. dbt owns the transformations. Ask a platform lead why and the answer is rarely about features, because feature for feature the comparison has always been close. The answer is about ownership: keep the modeling logic in a tool no cloud vendor controls, and the platform stays movable.

That was correct in 2024. Databricks had a declarative transformation framework, Delta Live Tables (DLT), and it ran on Databricks and nowhere else. Writing pipelines in it meant writing them into the platform. dbt’s SQL was portable by construction: models compile to SQL and adapters swap underneath, so the business rules can survive a migration the platform doesn’t.

One boundary was fixed from the start. Databricks documentation puts it plainly: dbt does not extract or load data, and focuses on the transformation step only [4]. Everything upstream stayed somebody else’s job. That was a fair division when the alternative was a proprietary framework.

Portability was the argument platform leads made. Analysts made a different one: dbt brought software engineering to people who write SQL and nothing else. Modularity was given through refs and macros, testing as a first-class citizen, generated documentation, clean separation between development and production. None of it required learning Python. SDP (Spark Declarative Pipelines) asks for a decorator and a CLI while dbt asks for Jinja and a SELECT.

Buying both with some deployment friction was a good compromise and the friction was bounded and well documented.

Portability Stopped being dbt’s to Sell

Declarative Pipelines are defined in the pyspark.pipelines module, installed with pip install pyspark[pipelines], and driven by a spark-pipelines CLI that scaffolds and dry-runs a project [5]. The authoring model is a decorator over a function that returns a DataFrame: @dp.materialized_view for batch, @dp.table for streaming [5]. None of it is Databricks property as it ships in Apache Spark.

Databricks states the consequence in its own documentation, under one governing clause:

“Because pipelines build on SDP rather than proprietary APIs, the transformation code you write stays portable” [6]

Databricks also lists what its runtime adds on top of Apache SDP, and those additions are exactly what doesn’t travel: AUTO CDC for slowly changing dimensions, data quality expectations, the queryable event log, update flows, continuous mode [6]. Most production pipelines use at least one. AUTO CDC and data quality expectations are the reason a team reaches for a pipeline framework instead of a scheduled notebook. And the guarantee runs one direction only. Open-source code runs unmodified on Databricks [7].

Scope is where the native path wins, and the win is the vendor’s rather than the open-source project’s. Sources: [4][5][8]

The scope line moved as well. Auto Loader runs inside a Lakeflow pipeline definition, so a single declaration can carry raw files from cloud storage through to curated tables [8]. The caveat belongs in the same sentence: Auto Loader is a Databricks feature, not part of open-source SDP. The end-to-end version is the vendor’s. The portable version still starts after the data has landed, which is where dbt starts too.

Streaming sits on the same line. dbt doesn’t have a streaming engine of its own. dbt Labs describes the tool as batch-oriented, with jobs that have a defined start and end time [9], and the microbatch strategy narrows the window to an hour rather than to a stream [10]. The dbt-databricks adapter can declare a Databricks streaming table [11]. dbt declares it; Databricks runs it.

So, categorical portability, the version where the logic simply isn’t the vendor’s, stopped being something only dbt could offer. What replaced it is a graded trade with a line a team can draw on purpose: a pipeline using AUTO CDC is a Databricks pipeline, and one that avoids it is a Spark pipeline that happens to run on Databricks. Drawing that line costs design effort. It’s also sharper than the line most teams can currently draw around a dbt project, because dbt’s portability was always partly aspirational. SQL dialects diverge, and a model written against one warehouse rarely lands on another without cross-database macros doing the reconciling.

Containment, not comparison. The open-source subset is real, and every feature a team adopts from the outer ring narrows it. Sources: [5][6][7]

The Case That Still Holds for dbt

The native path has constraints a dbt project doesn’t, and they’re specific enough to impact your design. A pipeline dataset can be defined only once, which makes it the target of a single operation across all pipelines [22]. Materialized views don’t support Delta time travel, and incremental refresh works only on serverless compute; everything else fully recomputes [22][23]. Some constraints travel with the portable code rather than belonging to Databricks: PIVOT is unsupported in SDP SQL, and dataset functions must return a DataFrame [5].

Documentation is the strongest card dbt still holds. dbt docs generate produces a static site with model descriptions, tests and a lineage graph, built from the project and versioned alongside it [24]. Databricks doesn't have an equivalent artifact. It has a live one instead: column-level lineage in Unity Catalog [25], a pipeline graph in the pipelines UI, and generally available AI-generated table and column comments [26]. Coverage isn't the difference. One is defined in code and reviewable in a pull request; the other is runtime metadata held in the catalog, if you don’t build your own metadata repo and belonging pipelines.

Scarcity of SDP skilled workforce is the weakest argument in the set. The SDP surface is a handful of decorators over functions that return DataFrames; an engineer who writes PySpark reads a pipeline the same afternoon, and the transferable skill underneath is Spark, which the market has in volume. What dbt has is harder to replace: more than 100,000 teams running it in production every week, a billion downloads, a decade of accumulated convention, and a population of analysts who write SQL and won’t write Python [27]. That’s a workflow asset rather than a scarce syntax, and it’s a legitimate reason to stay.

Get Dr. Christian Koch’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Neither purist position is the best available answer. In May 2026 the dbt-databricks adapter added Unity Catalog metric views as a materialization [21]. Metrics can be authored in dbt and land in Unity Catalog Business Semantics, announced generally available on 2 April 2026 and included with the platform, while dbt’s own semantic layer requires a Starter or Enterprise account [28][29]. Where dbt’s already in production, that pattern is preferable to a migration.

Accounting for the Extra Layer

The core question isn’t whether dbt is good software, nor is it about rushing into an immediate migration. It is much more specific: What concrete value does maintaining an extra tool layer alongside Databricks actually provide today?

Whether a team decides to keep dbt or transition away from it, the technical trade-offs generally fall into three categories:

• When the rationale is Portability: True portability now comes with narrow boundaries. Since Apache Spark 4.1, open-source pipelines only remain portable across systems if they actively avoid vendor-specific features like automated CDC or built-in data quality expectations.
• When the rationale is Workflow and Documentation: This remains a strong argument. If dbt provides analysts with a familiar SQL environment, version-controlled documentation, and reliable deployment workflows, that represents tangible value.
• When there is no clear rationale: Without a distinct purpose, dbt stops acting as just a transformation layer and becomes a redundant second control plane — introducing extra permission management, release friction, and conflicting vendor documentation without an offsetting benefit.

When evaluating an architecture against current technical realities, clarity is key. If a team can’t summarize in a single sentence what this dual-tool setup buys them, then you better figure it out soon.

References

1. [1] Apache Spark — “Spark Release 4.1.0” (2025).
2. [2] Databricks — “Bringing Declarative Pipelines to the Apache Spark Open Source Project” (2025).
3. [3] dbt Labs — “Fivetran + dbt Labs Complete Merger to Create the Data Infrastructure for Trusted AI Agents” (2026).
4. [4] Databricks — “Connect to dbt Core” (2026).
5. [5] Apache Spark — “Spark Declarative Pipelines Programming Guide, 4.1.0” (2025).
6. [6] Databricks — “Apache Spark Declarative Pipelines” (2026).
7. [7] Databricks — “Lakeflow pipelines Python language reference” (2026).
8. [8] Databricks — “Load data with Lakeflow pipelines” (2026).
9. [9] dbt Labs — “How we handle real-time data” (2026).
10. [10] dbt Labs — “About microbatch incremental models” (2026).
11. [11] dbt Labs — “Databricks configurations” (2026).
12. [12] BigDATAwire — “What the Fivetran-dbt Merger Means for the Data Ecosystem” (2025).
13. [13] dbt Labs — “dbt Core v2.0 Is Here” (2026).
14. [14] dbt Labs — “dbt Fusion engine availability” (2026).
15. [15] Fivetran — “Fivetran Acquires Tobiko Data” (2025).
16. [16] Fivetran — “Fivetran Contributes SQLMesh to the Linux Foundation” (2026).
17. [17] TechTarget — “Fivetran, dbt Labs Complete Merger to Form Data Layer for AI” (2026).
18. [18] Databricks — “Use dbt transformations in a Databricks job” (2026).
19. [19] dbt Labs — “Python models” (2026).
20. [20] dbt Labs — “dbt-core v1.3.0 release notes” (2022).
21. [21] Databricks — “dbt-databricks CHANGELOG” (2026).
22. [22] Databricks — “Lakeflow pipelines limitations” (2026).
23. [23] Databricks — “Incremental refresh for materialized views” (2026).
24. [24] dbt Labs — “Build and view your docs” (2026).
25. [25] Databricks — “Capture and view data lineage using Unity Catalog” (2026).
26. [26] Databricks — “Platform release notes, June 2024: AI-generated comments GA” (2024).
27. [27] dbt Labs — “Fivetran + dbt 2.0: What Comes Next” (2026).
28. [28] Databricks — “Redefining Semantics at the Data Layer for the Future of BI and AI” (2026).
29. [29] dbt Labs — “dbt Semantic Layer” (2026).

Content, including text and images, was created with the assistance of AI.