<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Cloud Native Consulting - Medium]]></title>
        <description><![CDATA[All about delivering timely, accurate data to drive results - Medium]]></description>
        <link>https://medium.com/cloud-native-consulting?source=rss----669804cd394f---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Cloud Native Consulting - Medium</title>
            <link>https://medium.com/cloud-native-consulting?source=rss----669804cd394f---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:59:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/cloud-native-consulting" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I Ran Every Common Spark Declarative Pipelines (SDP) Ingestion Pattern So You Don’t Have To]]></title>
            <link>https://medium.com/cloud-native-consulting/i-ran-every-common-spark-declarative-pipelines-sdp-ingestion-pattern-so-you-dont-have-to-a753d747d329?source=rss----669804cd394f---4</link>
            <guid isPermaLink="false">https://medium.com/p/a753d747d329</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[delta-live-tables]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Filip Pastuszka]]></dc:creator>
            <pubDate>Mon, 05 Oct 2026 07:16:01 GMT</pubDate>
            <atom:updated>2026-10-05T07:16:01.574Z</atom:updated>
            <content:encoded><![CDATA[<p>Spark Declarative Pipelines (SDP) has a lot of concepts: streaming tables, materialized views, flows, AUTO CDC, snapshots, change data feed. The docs explain each one well. What they don&#39;t tell you is which one to use for the table you actually have.</p><p>So I took the most common ways data lands in a landing table and built the bronze table for each. Every example below has the landing data, the code, and the real output after two pipeline runs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6Ii77pVCJCI9Lxz6sspMvg.jpeg" /></figure><h3>First, check how the landing table is written</h3><p>This decides everything else.</p><pre>DESCRIBE HISTORY landing.customers;<br>-- operation = WRITE (mode: Append)      -&gt; appended<br>-- operation = WRITE (mode: Overwrite)   -&gt; overwritten<br>-- operation = MERGE / UPDATE / DELETE   -&gt; merged in place</pre><h3>The four building blocks</h3><ul><li><strong>Materialized view</strong> (@dp.materialized_view): recomputed from the source on every run. Current state only, no memory.</li><li><strong>Streaming table</strong> (@dp.table over readStream): reads only rows added since the last run.</li><li>AUTO CDC (dp.create_auto_cdc_flow): applies inserts, updates and deletes on a key. SCD1 or SCD2.</li><li>AUTO CDC FROM SNAPSHOT (dp.create_auto_cdc_from_snapshot_flow): compares full snapshots on a key. SCD1 or SCD2.</li></ul><p>AUTO CDC and AUTO CDC FROM SNAPSHOT are Databricks features. Check your platform if you&#39;re on open-source Spark.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Iwe8b1x42OGZePr6OyPKw.jpeg" /></figure><h3>1. Full snapshot → current state</h3><p>The landing table is overwritten every night.</p><p><strong>Flow:</strong> landing.customers_full_snapshot → materialized view → bronze.customers_current</p><pre>-- landing.customers_full_snapshot, Day 0<br>+-----------+-----+------+<br>|customer_id| name|  city|<br>+-----------+-----+------+<br>|          1| Anna|Warsaw|<br>|          2|  Ben|Berlin|<br>|          3|Chloe| Paris|<br>+-----------+-----+------+<br><br>-- landing.customers_full_snapshot, Day 1 (INSERT OVERWRITE)<br>+-----------+----+------+<br>|customer_id|name|  city|<br>+-----------+----+------+<br>|          1|Anna|Warsaw|<br>|          2| Ben|Munich|<br>|          4| Dan|  Oslo|<br>+-----------+----+------+</pre><pre>@dp.materialized_view(name=&quot;bronze.customers_current&quot;)<br>def customers_current():<br>    return spark.read.table(&quot;landing.customers_full_snapshot&quot;)</pre><pre>-- bronze.customers_current, after Day 0<br>+-----------+-----+------+<br>|customer_id| name|  city|<br>+-----------+-----+------+<br>|          1| Anna|Warsaw|<br>|          2|  Ben|Berlin|<br>|          3|Chloe| Paris|<br>+-----------+-----+------+<br><br>-- bronze.customers_current, after Day 1<br>+-----------+----+------+<br>|customer_id|name|  city|<br>+-----------+----+------+<br>|          1|Anna|Warsaw|<br>|          2| Ben|Munich|<br>|          4| Dan|  Oslo|<br>+-----------+----+------+</pre><p><strong>Gotchas</strong></p><ul><li>No history. After Day 1 there’s no trace of Chloe. Need it? See scenario 3.</li><li>Don’t stream this table. The overwrite fails the stream, and with skipChangeCommits every overwrite is skipped, so bronze stays on Day 0.</li></ul><h3>2. Append-only events → append</h3><p>Payments, clicks, logs. Rows are added, never changed.</p><p><strong>Flow:</strong> landing.payments → stream → bronze.payments</p><pre>-- landing.payments, Day 0<br>+----------+-----------+------+-------------------+<br>|payment_id|customer_id|amount|            paid_at|<br>+----------+-----------+------+-------------------+<br>|     P-001|          1|120.00|2026-09-01 09:14:00|<br>|     P-002|          2| 35.50|2026-09-01 11:02:00|<br>|     P-003|          1| 80.00|2026-09-01 17:45:00|<br>+----------+-----------+------+-------------------+<br><br>-- landing.payments, Day 1 (2 rows appended)<br>+----------+-----------+------+-------------------+<br>|payment_id|customer_id|amount|            paid_at|<br>+----------+-----------+------+-------------------+<br>|     P-001|          1|120.00|2026-09-01 09:14:00|<br>|     P-002|          2| 35.50|2026-09-01 11:02:00|<br>|     P-003|          1| 80.00|2026-09-01 17:45:00|<br>|     P-004|          4| 60.00|2026-09-02 08:30:00|<br>|     P-005|          2| 15.00|2026-09-02 13:10:00|<br>+----------+-----------+------+-------------------+</pre><pre>@dp.table(name=&quot;bronze.payments&quot;)<br>def payments():<br>    return (<br>        spark.readStream.table(&quot;landing.payments&quot;)<br>        .withColumn(&quot;_ingested_at&quot;, F.current_timestamp())<br>    )</pre><pre>-- bronze.payments, after Day 0<br>+----------+-----------+------+-------------------+-----------------------+<br>|payment_id|customer_id|amount|            paid_at|           _ingested_at|<br>+----------+-----------+------+-------------------+-----------------------+<br>|     P-001|          1|120.00|2026-09-01 09:14:00|2026-10-03 13:06:22.609|<br>|     P-002|          2| 35.50|2026-09-01 11:02:00|2026-10-03 13:06:22.609|<br>|     P-003|          1| 80.00|2026-09-01 17:45:00|2026-10-03 13:06:22.609|<br>+----------+-----------+------+-------------------+-----------------------+<br><br>-- bronze.payments, after Day 1<br>+----------+-----------+------+-------------------+-----------------------+<br>|payment_id|customer_id|amount|            paid_at|           _ingested_at|<br>+----------+-----------+------+-------------------+-----------------------+<br>|     P-001|          1|120.00|2026-09-01 09:14:00|2026-10-03 13:06:22.609|<br>|     P-002|          2| 35.50|2026-09-01 11:02:00|2026-10-03 13:06:22.609|<br>|     P-003|          1| 80.00|2026-09-01 17:45:00|2026-10-03 13:06:22.609|<br>|     P-004|          4| 60.00|2026-09-02 08:30:00|2026-10-03 13:08:13.465|<br>|     P-005|          2| 15.00|2026-09-02 13:10:00|2026-10-03 13:08:13.465|<br>+----------+-----------+------+-------------------+-----------------------+</pre><p>_ingested_at shows the Day 1 run only read the two new rows.</p><p><strong>Gotchas</strong></p><ul><li>Only for tables that are really append-only. An UPDATE, DELETE or MERGE that changes existing rows fails the stream. skipChangeCommits hides the error and drops those changes.</li><li>A MERGE that only inserts is fine.</li></ul><h3>3. Full snapshot → history</h3><p>Same overwritten table as scenario 1, but now we keep history.</p><p><strong>Flow:</strong> landing.customers_full_snapshot → AUTO CDC FROM SNAPSHOT → bronze.customers_history</p><pre>-- landing.customers_full_snapshot, Day 0<br>+-----------+-----+------+<br>|customer_id| name|  city|<br>+-----------+-----+------+<br>|          1| Anna|Warsaw|<br>|          2|  Ben|Berlin|<br>|          3|Chloe| Paris|<br>+-----------+-----+------+<br><br>-- landing.customers_full_snapshot, Day 1 (INSERT OVERWRITE)<br>+-----------+----+------+<br>|customer_id|name|  city|<br>+-----------+----+------+<br>|          1|Anna|Warsaw|<br>|          2| Ben|Munich|<br>|          4| Dan|  Oslo|<br>+-----------+----+------+</pre><pre>dp.create_streaming_table(&quot;bronze.customers_history&quot;)<br><br>dp.create_auto_cdc_from_snapshot_flow(<br>    target=&quot;bronze.customers_history&quot;,<br>    source=&quot;landing.customers_full_snapshot&quot;,<br>    keys=[&quot;customer_id&quot;],<br>    stored_as_scd_type=2,<br>)</pre><pre>-- bronze.customers_history, after Day 0<br>+-----------+-----+------+-----------------------+--------+<br>|customer_id| name|  city|             __START_AT|__END_AT|<br>+-----------+-----+------+-----------------------+--------+<br>|          1| Anna|Warsaw|2026-10-03 13:06:16.531|    NULL|<br>|          2|  Ben|Berlin|2026-10-03 13:06:16.531|    NULL|<br>|          3|Chloe| Paris|2026-10-03 13:06:16.531|    NULL|<br>+-----------+-----+------+-----------------------+--------+<br><br>-- bronze.customers_history, after Day 1<br>+-----------+-----+------+-----------------------+-----------------------+<br>|customer_id| name|  city|             __START_AT|               __END_AT|<br>+-----------+-----+------+-----------------------+-----------------------+<br>|          1| Anna|Warsaw|2026-10-03 13:06:16.531|                   NULL|<br>|          2|  Ben|Berlin|2026-10-03 13:06:16.531|2026-10-03 13:08:10.766|<br>|          2|  Ben|Munich|2026-10-03 13:08:10.766|                   NULL|<br>|          3|Chloe| Paris|2026-10-03 13:06:16.531|2026-10-03 13:08:10.766|<br>|          4|  Dan|  Oslo|2026-10-03 13:08:10.766|                   NULL|<br>+-----------+-----+------+-----------------------+-----------------------+</pre><p>Chloe’s row is closed, not removed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Px3rmKX3GJ_u7rEm_TJbZg.jpeg" /></figure><p><strong>Gotchas</strong></p><ul><li>__START_AT / __END_AT are pipeline run times, not when Ben actually moved.</li><li>Skip a run and two days of changes become one version.</li><li>History starts on the day you deploy.</li></ul><h3>4. Snapshot history → history with business dates</h3><p>The source keeps every snapshot with a snapshot_date.</p><p><strong>Flow:</strong> landing.customers_snapshot_history → function, one date at a time → AUTO CDC FROM SNAPSHOT → bronze.customers_history_by_date</p><pre>-- landing.customers_snapshot_history, Day 0<br>+-------------+-----------+-----+------+<br>|snapshot_date|customer_id| name|  city|<br>+-------------+-----------+-----+------+<br>|   2026-09-01|          1| Anna|Warsaw|<br>|   2026-09-01|          2|  Ben|Berlin|<br>|   2026-09-01|          3|Chloe| Paris|<br>+-------------+-----------+-----+------+<br><br>-- landing.customers_snapshot_history, Day 1 (new snapshot appended)<br>+-------------+-----------+-----+------+<br>|snapshot_date|customer_id| name|  city|<br>+-------------+-----------+-----+------+<br>|   2026-09-01|          1| Anna|Warsaw|<br>|   2026-09-01|          2|  Ben|Berlin|<br>|   2026-09-01|          3|Chloe| Paris|<br>|   2026-09-02|          1| Anna|Warsaw|<br>|   2026-09-02|          2|  Ben|Munich|<br>|   2026-09-02|          4|  Dan|  Oslo|<br>+-------------+-----------+-----+------+</pre><pre>def next_snapshot(last_version):<br>    snapshots = spark.read.table(&quot;landing.customers_snapshot_history&quot;)<br>    dates = snapshots.select(&quot;snapshot_date&quot;)<br>    if last_version is not None:<br>        dates = dates.filter(F.col(&quot;snapshot_date&quot;) &gt; F.lit(last_version))<br>    next_date = dates.agg(F.min(&quot;snapshot_date&quot;)).first()[0]<br>    if next_date is None:<br>        return None<br>    return snapshots.filter(F.col(&quot;snapshot_date&quot;) == next_date).drop(&quot;snapshot_date&quot;), next_date<br><br><br>dp.create_streaming_table(&quot;bronze.customers_history_by_date&quot;)<br>dp.create_auto_cdc_from_snapshot_flow(<br>    target=&quot;bronze.customers_history_by_date&quot;,<br>    source=next_snapshot,<br>    keys=[&quot;customer_id&quot;],<br>    stored_as_scd_type=2,<br>)</pre><pre>-- bronze.customers_history_by_date, after Day 0<br>+-----------+-----+------+----------+--------+<br>|customer_id| name|  city|__START_AT|__END_AT|<br>+-----------+-----+------+----------+--------+<br>|          1| Anna|Warsaw|2026-09-01|    NULL|<br>|          2|  Ben|Berlin|2026-09-01|    NULL|<br>|          3|Chloe| Paris|2026-09-01|    NULL|<br>+-----------+-----+------+----------+--------+<br><br>-- bronze.customers_history_by_date, after Day 1<br>+-----------+-----+------+----------+----------+<br>|customer_id| name|  city|__START_AT|  __END_AT|<br>+-----------+-----+------+----------+----------+<br>|          1| Anna|Warsaw|2026-09-01|      NULL|<br>|          2|  Ben|Berlin|2026-09-01|2026-09-02|<br>|          2|  Ben|Munich|2026-09-02|      NULL|<br>|          3|Chloe| Paris|2026-09-01|2026-09-02|<br>|          4|  Dan|  Oslo|2026-09-02|      NULL|<br>+-----------+-----+------+----------+----------+</pre><p>The version the function returns ends up in __START_AT / __END_AT. Here: business dates.</p><p><strong>Gotchas</strong></p><ul><li>SDP keeps calling the function until it returns None. Return each snapshot once, oldest first.</li><li>Versions must only go up. A late snapshot for an earlier date is ignored.</li></ul><h3>5. Merged mirror → current state, only changed rows</h3><p>An ingestion tool (Fivetran, Airbyte, …) keeps landing as a copy of the source and applies changes with MERGE. A plain stream fails here, so we read the change data feed: Delta&#39;s log of inserted, updated and deleted rows.</p><p><strong>Flow:</strong> landing.customers_mirror → change feed → view → AUTO CDC → bronze.customers_from_cdf</p><pre>-- landing.customers_mirror, Day 0<br>+-----------+-----+------+<br>|customer_id| name|  city|<br>+-----------+-----+------+<br>|          1| Anna|Warsaw|<br>|          2|  Ben|Berlin|<br>|          3|Chloe| Paris|<br>+-----------+-----+------+<br><br>-- landing.customers_mirror, Day 1 (one MERGE: update Ben, delete Chloe, insert Dan)<br>+-----------+----+------+<br>|customer_id|name|  city|<br>+-----------+----+------+<br>|          1|Anna|Warsaw|<br>|          2| Ben|Munich|<br>|          4| Dan|  Oslo|<br>+-----------+----+------+</pre><pre>SELECT customer_id, name, city, _change_type, _commit_version<br>FROM table_changes(&#39;landing.customers_mirror&#39;, 2);  -- the Day 1 MERGE</pre><pre>+-----------+-----+------+----------------+---------------+<br>|customer_id| name|  city|    _change_type|_commit_version|<br>+-----------+-----+------+----------------+---------------+<br>|          2|  Ben|Berlin| update_preimage|              2|<br>|          2|  Ben|Munich|update_postimage|              2|<br>|          3|Chloe| Paris|          delete|              2|<br>|          4|  Dan|  Oslo|          insert|              2|<br>+-----------+-----+------+----------------+---------------+</pre><p>Anna isn’t in the feed. She didn’t change.</p><pre>@dp.temporary_view<br>def customers_mirror_cdf_v():<br>    return (<br>        spark.readStream.option(&quot;readChangeFeed&quot;, &quot;true&quot;)<br>        .table(&quot;landing.customers_mirror&quot;)<br>        .filter(F.col(&quot;_change_type&quot;) != &quot;update_preimage&quot;)<br>    )<br><br>dp.create_streaming_table(&quot;bronze.customers_from_cdf&quot;)<br>dp.create_auto_cdc_flow(<br>    target=&quot;bronze.customers_from_cdf&quot;,<br>    source=&quot;customers_mirror_cdf_v&quot;,<br>    keys=[&quot;customer_id&quot;],<br>    sequence_by=&quot;_commit_version&quot;,<br>    apply_as_deletes=F.expr(&quot;_change_type = &#39;delete&#39;&quot;),<br>    except_column_list=[&quot;_change_type&quot;, &quot;_commit_version&quot;, &quot;_commit_timestamp&quot;],<br>    stored_as_scd_type=1,<br>)</pre><pre>-- bronze.customers_from_cdf, after Day 0<br>+-----------+-----+------+<br>|customer_id| name|  city|<br>+-----------+-----+------+<br>|          1| Anna|Warsaw|<br>|          2|  Ben|Berlin|<br>|          3|Chloe| Paris|<br>+-----------+-----+------+<br><br>-- bronze.customers_from_cdf, after Day 1<br>+-----------+----+------+<br>|customer_id|name|  city|<br>+-----------+----+------+<br>|          1|Anna|Warsaw|<br>|          2| Ben|Munich|<br>|          4| Dan|  Oslo|<br>+-----------+----+------+</pre><p>Same result as scenario 1, but t<strong>he flow processed three rows instead of the whole table</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b8QxnozL80mUxlZNSSqd_w.jpeg" /></figure><p><strong>Gotchas</strong></p><ul><li>Turn on delta.enableChangeDataFeed before the changes happen. It isn&#39;t retroactive (always-on feature in beta now).</li><li>Without except_column_list, _change_type and friends end up in bronze.</li><li>Only worth it if landing is merged. On an overwritten table the feed contains every row.</li><li>Need history? Set stored_as_scd_type=2.</li></ul><h3>6. Source-maintained SCD2 → current state</h3><p>The source tracks history itself with valid_from, valid_to and is_current.</p><p><strong>Flow:</strong> landing.customers_scd2 → materialized view → bronze.customers_current_from_scd2</p><pre>-- landing.customers_scd2, Day 0<br>+-----------+-----+------+----------+----------+----------+<br>|customer_id| name|  city|valid_from|  valid_to|is_current|<br>+-----------+-----+------+----------+----------+----------+<br>|          1| Anna|Warsaw|2026-01-10|9999-12-31|      true|<br>|          2|  Ben|Berlin|2026-03-05|9999-12-31|      true|<br>|          3|Chloe| Paris|2026-05-20|9999-12-31|      true|<br>+-----------+-----+------+----------+----------+----------+<br><br>-- landing.customers_scd2, Day 1 (old versions closed, new ones inserted)<br>+-----------+-----+------+----------+----------+----------+<br>|customer_id| name|  city|valid_from|  valid_to|is_current|<br>+-----------+-----+------+----------+----------+----------+<br>|          1| Anna|Warsaw|2026-01-10|9999-12-31|      true|<br>|          2|  Ben|Berlin|2026-03-05|2026-09-02|     false|<br>|          2|  Ben|Munich|2026-09-02|9999-12-31|      true|<br>|          3|Chloe| Paris|2026-05-20|2026-09-02|     false|<br>|          4|  Dan|  Oslo|2026-09-02|9999-12-31|      true|<br>+-----------+-----+------+----------+----------+----------+</pre><pre>@dp.materialized_view(name=&quot;bronze.customers_current_from_scd2&quot;)<br>def customers_current_from_scd2():<br>    return (<br>        spark.read.table(&quot;landing.customers_scd2&quot;)<br>        .filter(&quot;is_current&quot;)<br>        .select(&quot;customer_id&quot;, &quot;name&quot;, &quot;city&quot;, &quot;valid_from&quot;)<br>    )</pre><pre>-- bronze.customers_current_from_scd2, after Day 0<br>+-----------+-----+------+----------+<br>|customer_id| name|  city|valid_from|<br>+-----------+-----+------+----------+<br>|          1| Anna|Warsaw|2026-01-10|<br>|          2|  Ben|Berlin|2026-03-05|<br>|          3|Chloe| Paris|2026-05-20|<br>+-----------+-----+------+----------+<br><br>-- bronze.customers_current_from_scd2, after Day 1<br>+-----------+----+------+----------+<br>|customer_id|name|  city|valid_from|<br>+-----------+----+------+----------+<br>|          1|Anna|Warsaw|2026-01-10|<br>|          2| Ben|Munich|2026-09-02|<br>|          4| Dan|  Oslo|2026-09-02|<br>+-----------+----+------+----------+</pre><p><strong>Gotchas</strong></p><ul><li>Closing an old version is an UPDATE, so a plain stream fails here too.</li><li>Want the full history in bronze? Drop the filter.</li></ul><h3>Cheat sheet</h3><ul><li><strong>Appended</strong> → streaming table.</li><li><strong>Overwritten</strong>, current state → materialized view.</li><li><strong>Overwritten</strong>, history → AUTO CDC FROM SNAPSHOT.</li><li><strong>Snapshots with a date column</strong> → AUTO CDC FROM SNAPSHOT with a function.</li><li><strong>Merged in place</strong> → materialized view, or change feed + AUTO CDC for big tables or history.</li><li><strong>SCD2 from the source</strong> → materialized view.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TzOj1Suxiw8pMbbJKaGLwg.jpeg" /></figure><p>Happy coding!</p><p>Filip</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a753d747d329" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloud-native-consulting/i-ran-every-common-spark-declarative-pipelines-sdp-ingestion-pattern-so-you-dont-have-to-a753d747d329">I Ran Every Common Spark Declarative Pipelines (SDP) Ingestion Pattern So You Don’t Have To</a> was originally published in <a href="https://medium.com/cloud-native-consulting">Cloud Native Consulting</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Being the Human ETL: How to Let Business Users Load CSVs into Databricks (Self-Service)]]></title>
            <link>https://medium.com/cloud-native-consulting/stop-being-the-human-etl-how-to-let-business-users-load-csvs-into-databricks-self-service-80c5088ab491?source=rss----669804cd394f---4</link>
            <guid isPermaLink="false">https://medium.com/p/80c5088ab491</guid>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Filip Pastuszka]]></dc:creator>
            <pubDate>Mon, 27 Jul 2026 07:58:07 GMT</pubDate>
            <atom:updated>2026-07-27T07:58:06.329Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Volumes, YAML contracts, DQ validation, and append-only versioning — the trade-offs behind user-data ingestion pipeline in Databricks.</em></p><p>Meet Jake. Jake is a data engineer. Every Monday, a friendly person from Finance emails Jake a CSV — this week’s forecast. Jake downloads it, eyeballs it, commits it into the repo as a seed file, opens a PR, waits for CI, merges, and deploys. The number lands in a table. Everyone’s happy.</p><p>Then Tuesday happens. Finance found a typo. New CSV. Same dance.</p><p>Then a <em>second</em> team hears Jake can “get their file in” and starts emailing him too.</p><p>Jake is no longer a data engineer. Jake is a human ETL pipeline with a calendar full of “quick favors.” And Jake is starting to wonder whether he should have taken over his father’s dry-cleaning business after all.</p><p>Sound familiar (minus the dry-cleaning)?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q1F3KuLmwOP8g2SLFeVR3A.png" /><figcaption>From inbox chaos to a governed table — without an engineer in the middle.</figcaption></figure><p>This article is about how I got Jake out of the loop — for one of my clients — by building a <strong>governed self-service upload surface</strong>: business users drop their own CSVs into Databricks, get instant feedback on whether the file is any good, and the data lands in a versioned, audited table without an engineer touching anything. Safely</p><h3>⚡️ TL;DR</h3><ul><li>Stakeholders emailing CSVs to engineers is a bottleneck, a security risk, and an audit black hole.</li><li>The fix is a <strong>self-service surface</strong>, not another one-off script. I used <strong>Databricks Volumes</strong> as the drop zone.</li><li>Files are validated against a <strong>YAML data contract</strong> using <strong>DQX</strong> <em>before</em> they’re allowed anywhere near a table.</li><li>Re-uploads <strong>append a new version</strong> (never overwrite); downstream reads a thin “latest” view.</li><li>Every row carries <strong>metadata</strong> (who uploaded it, when, which version) — so “who put this in production?” finally has an answer.</li><li>The whole thing is <strong>pure Python</strong>, triggered by a Databricks job the stakeholders run themselves.</li></ul><p>By the end you’ll be able to design your own version of this — and I’ll be honest about where it’s heavier than I’d like.</p><h3>1. You are the bottleneck</h3><p>Let’s name what’s actually wrong with the email-a-CSV workflow, because it’s more than “it’s annoying.”</p><ul><li><strong>Engineering bottleneck.</strong> Nothing moves without a human. Your ingestion throughput is capped by how fast an engineer can babysit files.</li><li><strong>Security.</strong> Sensitive business data is now sitting in inboxes, in Downloads folders, in Slack threads. Nobody signed off on that.</li><li><strong>No audit trail.</strong> Six months later: “where did <em>this</em> number come from?” The honest answer is “an email Jake can’t find anymore.”</li><li><strong>Deployment-coupled turnaround.</strong> If the file goes in via a commit, then correcting a value means a PR, a review, and a deploy. That’s a lot of ceremony for a typo.</li></ul><p>The temptation is to write a script that watches a folder and “just loads the files.” Don’t. A script solves the typing, not the <em>governance</em>. What you actually need is a surface where users can serve themselves <strong>and</strong> the platform enforces the rules.</p><h3>2. What “good” looks like</h3><p>Before choosing any tech, let’s write down what we’re actually optimizing for. Every later decision gets checked against this list.</p><p><strong>Must-haves:</strong></p><ul><li>★ <strong>File discovery</strong> — scan storage automatically and pick up new files.</li><li>★ <strong>Atomic per-file ingest</strong> — one bad file must not poison the others.</li><li>★ <strong>Per-file error handling</strong> — validate each file on its own merits.</li><li>★ <strong>Per-file user feedback</strong> — the uploader learns <em>this file</em> passed or failed, and why.</li><li>★ <strong>Schema enforcement via a contract</strong> — a declared, expected shape; no free-for-all.</li><li>★ <strong>Audit trail</strong> — who uploaded what, and when.</li><li>★ <strong>No engineer in the steady-state loop</strong> — after setup, engineers go do engineering.</li><li>★ <strong>Auto archiving</strong> — ingested files get to be saved for audit purposes.</li><li><strong>★ Schema evolution without downtime </strong>— adding or renaming a column should not impact the existing solution.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K4PQyic-HcvT3zwmx7-31A.png" /><figcaption>The non-negotiables: what any self-service ingestion must do.</figcaption></figure><h3>3. Where do the files actually go in?</h3><p>This is the biggest fork in the road: <strong>what’s the drop zone?</strong> Get this wrong and everything downstream inherits the pain.</p><p>Here are the contenders — one sentence on what each is <em>good at</em>, and the one thing that <em>rules it out</em>:</p><ul><li><strong>Databricks Volumes (chosen)</strong> — native to the platform, best permission model, almost nothing to build or maintain; the UX is functional rather than pretty.</li><li><strong>Custom upload web app</strong> — the nicest stakeholder UX (inline preview + validation before submit), but now you own an app forever: auth, infra, on-call — biggest upfront investment and maintenance cost.</li><li><strong>Cloud object storage drop (e.g. S3)</strong> — rock-solid and cheap, but a cloud console is not a business-user surface — not very user friendly plus I would rather not have my business users roam around AWS console and raw storage.</li><li><strong>Corporate file sync (SharePoint / OneDrive)</strong> — familiar office UX, but per-tenant connector plumbing and ongoing permission overhead.</li><li><strong>Git repo + PRs</strong> — a perfect audit trail, but stakeholders won’t (and shouldn’t) write pull requests. Not to mention poor access control mangement. You see all of the data or none of it — finance should NOT see HR data and the other way around.</li><li><strong>Email-to-storage</strong> —still requires some development plus more things to break. Also, does not provide really good (timely) user feedback.</li><li><strong>Managed file connectors</strong> — powerful, but overkill and costly for a weekly CSV drop.</li></ul><p><strong>What I picked, and why:</strong> <strong>Databricks Volumes.</strong> If your stakeholders already live in Databricks, Volumes give you a drag-and-drop drop zone, a native Unity Catalog permission model, and <em>nothing new to run</em>. The UX is a little engineer-y (it’s the Catalog Explorer, not a bespoke portal), but that’s a price I’ll happily pay to avoid owning a web app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*joczR4xTLd4M2nE5MI0E_g.png" /><figcaption>Every drop zone has a catch. Volumes had the fewest.</figcaption></figure><blockquote><strong><em>Stakeholders / users not on Databricks?</em></strong><em> <br>Then this calculus changes. If your users have never seen a Databricks workspace, a custom app or a plain object-storage drop may win instead — or it might be the nudge you need to onboard them. The rest of this article assumes they’re already in Databricks.</em></blockquote><h3>4. The architecture at a glance (One user-triggered job)</h3><p>Purpose / what this section covers:</p><ul><li>Introduce the <strong>single, linear, user-triggered flow</strong>: the user uploads a file, then triggers <strong>one job</strong> that does it all — <strong>discover → validate → ingest to bronze </strong><strong>_hist (+ archive)</strong>.</li><li>No scheduled background ingest, no second pipeline. One upload, one button, one job.</li><li>Why one job (not a split validate/ingest): self-service lives on immediate end-to-end feedback — “did my file make it in?” A single run answers that (pass = in bronze + archived; fail = quarantined + reason).</li><li>Foreshadow that stakeholders trigger their own runs (control + UX).</li></ul><p>This is how it looks like end-to-end:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*D773NYQ8C3HBxM-OwDR3Ug.png" /><figcaption>The whole pipeline is a single job the user triggers themselves.</figcaption></figure><h3>5. Storage layout: two volumes, one contract</h3><p>Here’s the hierarchy. Generic names, but the shape is what matters:</p><pre>user_defined_inputs/                 (catalog)<br>└── finance_data/                     (schema)<br>    ├── inbox_volume/                 (volume — read/write for the group)<br>    │   └── inbox/                    ← users upload here<br>    └── archive_volume/               (volume — read-only for the group)<br>        └── processed/                ← system-written, successfully ingested</pre><p>Two design choices worth calling out:</p><ul><li><strong>A separate </strong><strong>archive_volume, not a subfolder.</strong> Once a file is processed, it moves somewhere users can <em>read</em> but not <em>touch</em>. That gives you a clean, tamper-resistant record of exactly what was ingested — great for replay and audits.</li><li><strong>One volume per stakeholder group.</strong> This one’s not really a choice; it’s a Unity Catalog constraint. Permissions are granted at the <strong>volume level, not the folder level</strong>. So if you want Finance and HR isolated, they need separate volumes. Plan for it up front.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lD-bWCCEKVDfqN3AthFUeg.png" /><figcaption>Two volumes: write to the inbox, read-only on the archive.</figcaption></figure><h3>6. Provisioning and access (Terraform + groups, not people)</h3><p>Two things need setting up per group: the <strong>storage</strong> and the <strong>access</strong>. I’ll keep this short — you don’t need my Terraform code, you want the ideas.</p><p><strong>Provisioning, kept DRY.</strong> Volumes and grants are created declaratively with <strong>Terraform</strong> (over the CI/CD-plus-CLI alternatives, including Databricks Asset Bundles — Terraform is the source of truth). The bit worth stealing: a <strong>single reusable YAML</strong> lists your schemas, files, and contracts, and it feeds <em>both</em> the provisioning <em>and</em> the runtime job. One place to change when you onboard a new dataset.</p><pre>schemas:<br>  - uc_schema: finance_data<br>    files:<br>      - file_type: expense_forecast<br>        target_table: expense_forecast_hist<br>        schema_version: 1<br>        notify:<br>          - finance-data-owners@example.com</pre><p>Honest caveat: “one YAML, two consumers” is only DRY if you actually keep it as one source of truth. The moment provisioning and runtime read <em>different</em> configs, you’ve bought yourself drift.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LIjroghKJwK3co21mom5zw.png" /><figcaption>One config feeds both Terraform and the job; access goes to groups, not people.</figcaption></figure><p><strong>Access: groups, not people.</strong> You <em>can</em> grant access to individual users. Don’t — it doesn’t scale and it rots the moment someone changes teams. Grant to <strong>groups</strong>, managed through your IdP (Okta/Entra + SCIM) with an access-request flow. The cost is group sprawl; the rule of thumb is <em>enough groups to isolate, few enough to manage</em>.</p><p>The grant shape is simple: stakeholders get <strong>write on the inbox</strong>, <strong>read on the archive and intested tables</strong>, and they get trigger access to the ingestion job. That’s all.</p><pre>GRANT WRITE VOLUME ON VOLUME user_defined_inputs.finance_data.inbox_volume<br>  TO `finance_data_uploaders`;<br>GRANT READ VOLUME  ON VOLUME user_defined_inputs.finance_data.archive_volume<br>  TO `finance_data_uploaders`;</pre><h3>7. File rules that make everything else possible</h3><p>Conventions feel like bureaucracy until you realize they’re load-bearing. A few rules here make versioning, discovery, and validation <em>possible</em> downstream.</p><ul><li><strong>CSV only.</strong> Not Excel. Excel isn’t Spark-native, and the moment you accept .xlsx you inherit type-inference surprises, hidden formulas, and &quot;why is this number now a date&quot; bugs. Ask for a CSV export. (Yes, users will grumble. They&#39;ll survive.)</li><li><strong>A declared delimiter and decimal separator.</strong> Half the world writes 1,000.50 and the other half writes 1.000,50. Pick one, put it in the contract.</li><li><strong>A naming convention: </strong><strong>&lt;file_type&gt;_&lt;date&gt;.csv.</strong> For example expense_forecast_20260101.csv. The <strong>stem</strong> (expense_forecast) is stable across re-uploads and tells discovery which contract to apply. The <strong>date</strong> is the <em>business effective date</em>, not the upload date — that distinction matters more than you&#39;d think.</li></ul><p>That stable stem is the hook the whole pipeline hangs on: discovery matches files to a contract by prefix, and re-uploading the same stem is what triggers versioning rather than a mess.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5p4PrZLScZPED6OHIPOGDw.png" /><figcaption>A filename that does three jobs: route, date, and version.</figcaption></figure><h3>8. Batch vs Auto Loader</h3><p>If you’ve spent any time in Databricks, your instinct here is “surely this is a job for <strong>Auto Loader</strong> — it’s the native, streaming, file-discovery thing.” It’s a fair instinct. It’s also, at this scale, the wrong one.</p><p>Let’s score it against the must-haves from section #2:</p><ul><li><strong>Atomic per-file ingest?</strong> Auto Loader treats incoming files as <em>one stream</em>. We want per-file semantics — this file passed, that file didn’t.</li><li><strong>Per-file exception handling?</strong> Same problem. A stream doesn’t naturally give you “file 3 of 5 is bad, the rest are fine.”</li><li><strong>Per-file notification?</strong> We want to tell <em>this uploader</em> about <em>their</em> file. Streams think in micro-batches, not uploads.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P5RoX-ZImrVeDmQedM5jvg.png" /><figcaption>Auto Loader is great — just not when every file needs its own verdict.</figcaption></figure><p><strong>What I picked, and why:</strong> plain <strong>Batch</strong>. Self-service is fundamentally a per-file experience, and Batch gives me per-file control for free. Auto Loader’s genuine wins — automatic discovery, archiving, corrupt-record capture — are either low-priority here or cheap to replicate in a few lines of Python.</p><p><strong>When I’d revisit this:</strong> if the volume becomes continuous and high-throughput — hundreds of files an hour instead of a handful a week — Auto Loader’s streaming model starts to earn its keep. For a weekly forecast drop, it’s over-engineering.</p><h3>9. Validation: the gate to bronze</h3><p>This is the heart of the article, and the part I feel most strongly about. So let me plant the flag early: <strong>nothing reaches bronze without passing a contract.</strong> No contract, no entry. Implemented using <a href="https://databrickslabs.github.io/dqx/">Databricks’ DQX</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V_PytG_aOy4dMqeyOOxceA.png" /><figcaption>The gate: match a contract, run DQX, then pass to bronze or quarantine.</figcaption></figure><p>A <strong>data contract</strong> is one YAML file that declares a dataset’s expected schema <em>and</em> its data-quality rules. It’s maintained centrally and <strong>exposed read-only</strong> so users can self-check <em>before</em> they upload — no more guessing what “the right format” means.</p><pre>- table: expense_forecast_hist<br>  file_name_date_format: &quot;%Y%m%d&quot;<br>  schema_version: 1<br>  schema:<br>    - name: cost_center<br>      type: string<br>      nullable: false<br>    - name: amount<br>      type: decimal<br>      nullable: false<br>    - name: period<br>      type: date<br>      nullable: false<br>  dq_checks:<br>    - check: is_not_null<br>      column: cost_center<br>    - check: is_positive<br>      column: amount</pre><p>Now, <em>how</em> do we enforce it? Here’s where I part ways with the “obvious” answer.</p><p><a href="https://databrickslabs.github.io/dqx/">DQX</a> in pure Python<strong>.</strong> One process. One language. Validate the file in memory against the contract using <strong>DQX</strong>, and if it passes, write it straight to bronze. One pre-write gate, low latency, and it works for any file without spinning up a per-file project.</p><p>Here’s the shape of it. First, <strong>discovery</strong> — scan the inbox, keep the CSVs, match each to a contract by its stem:</p><pre>def discover(inbox_path, files_config):<br>    for path in list_files(inbox_path):<br>        if extension(path) not in ALLOWED_FORMATS:      # csv only<br>            skip(path, reason=&quot;format&quot;); continue<br>        contract = match_by_prefix(path, files_config)   # &quot;&lt;file_type&gt;_*.csv&quot;<br>        if not contract:<br>            skip(path, reason=&quot;unknown file type&quot;); continue<br>        yield WorkItem(path=path, contract=contract,<br>                       uploaded_at=modified_time(path))   # FIFO per file_type</pre><p>Then the <strong>gate</strong> itself — apply the contract’s checks and split the good rows from the bad:</p><pre>checks = build_dqx_checks(contract)                 # from dq_checks[] above<br>df = spark.read.csv(path, header=True, schema=contract.spark_schema)<br><br>valid_df, quarantine_df = DQEngine(ws).apply_checks_and_split(df, checks)<br>if quarantine_df.count():<br>    return Result(passed=False, errors=collect_reasons(quarantine_df))<br>return Result(passed=True, df=valid_df)</pre><p>The outcomes are deliberately simple: <strong>pass</strong> → the rows are written to bronze and the file is archived; <strong>fail</strong> → the file is quarantined and the reason is handed back to the uploader. And schema enforcement stays mandatory even if we auto-detect the CSV dialect to make life easier — being <em>nice</em> about formatting is fine, being <em>loose</em> about schema is not.</p><h3>10. Re-uploads: append, don’t overwrite</h3><p>Real life: someone uploads expense_forecast_20260101.csv, then realizes a number was wrong and uploads it <em>again</em> with the same name. What should happen?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FEjXg9NfSceLHejTgHuuNQ.png" /><figcaption>Re-uploads append a new version; the “latest” view sees only v2.</figcaption></figure><p>The wrong answer is “overwrite.” Overwriting destroys history and makes audits impossible.</p><p>The right answer: <strong>append every load</strong> with an incremented _source_file_version, and never overwrite. Version 1 stays. Version 2 lands next to it. The full history is queryable forever — for audits, for replay, for &quot;what did we think the forecast was back in January?&quot;</p><p>Downstream doesn’t want to think about versions, though, so it reads a thin <strong>“latest version” view</strong>:</p><pre>SELECT * FROM expense_forecast_hist<br>QUALIFY ROW_NUMBER() OVER (<br>  PARTITION BY _source_file ORDER BY _source_file_version DESC<br>) = 1;</pre><p>_source_file_version is computed at ingest time — and because the archive keeps every processed file, it can be recomputed <em>deterministically</em> if you ever rebuild the table from scratch. History that you can reconstruct is history you can trust.</p><h3>11. Schema evolution: versioned tables vs one big Delta table</h3><p>This is the most debatable decision in the whole design, so I’ll present it as a genuine fork rather than pretend there’s one true answer.</p><p>Schemas change. Finance adds a column. What now?</p><p><strong>Option A — explicit versioned tables (chosen).</strong> When the schema changes meaningfully, you create expense_forecast_hist_v2 alongside _v1, with its own contract. Downstream migrates <em>deliberately</em>, on its own schedule.</p><p><strong>Option B — one big Delta table with schema evolution.</strong> Turn on mergeSchema, add a _schema_version column, and let the table absorb changes in place.</p><p>The trade-off, honestly:</p><ul><li>Option B is <strong>less boilerplate</strong> up front. But it’s <strong>risky</strong>: you can’t cleanly tell “this column didn’t exist yet” from “this value was missing,” replay gets harder, type and nullability changes can be unsafe, and a schema bump forces an <em>immediate</em> downstream refactor whether you’re ready or not.</li><li>Option A is <strong>more tables</strong>, yes — but every migration is <strong>safe and decoupled</strong>. Nothing breaks by surprise.</li></ul><p>I also rejected the extremes: letting stakeholders manage their own schemas (chaos), overwriting in place with no contract update (silent corruption), and auto-bumping versions on any change (over-engineering that hides real decisions).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4M0_vcJqh2E4e1jNv-z9iA.png" /><figcaption>The schema changed, and nothing downstream broke. On purpose. We only retire v1 (and replace it with v2) when the silver layer is ready for it.</figcaption></figure><p>The migration playbook with Option A is boring, which is the compliment: get a sample of the new format → add a v2 contract → ingest once → repoint downstream → retire v1. The repoint is a one-liner, and that one-liner is <em>exactly</em> what the version split buys you:</p><pre>-- silver now reads v2 instead of v1<br>CREATE OR REPLACE VIEW silver.expense_forecast AS<br>SELECT * FROM expense_forecast_hist_v2;   -- was ..._v1</pre><h3>12. Metadata columns: the audit trail you’ll thank yourself for</h3><p>Every row that lands in bronze carries a set of system columns, all _-prefixed by convention so they never collide with real data. This is the difference between &quot;a table&quot; and &quot;a table you can <em>trust in production</em>.&quot;</p><ul><li>_source_file — the file this row came from.</li><li>_ingestion_timestamp — when we processed it.</li><li>_source_modified_by — <strong>who uploaded it</strong> (from Unity Catalog audit / volume metadata).</li><li>_source_file_version — the re-upload version from §10.</li><li>_schema_version — which contract version this row conforms to.</li><li>_source_system — where it originated.</li></ul><p>The natural lineage key is (_source_file, _source_file_version) — that pair uniquely identifies any load.</p><p>And the star of the show is _source_modified_by. That single column answers the question that used to require an archaeological dig through inboxes: <strong>&quot;who put this number into production?&quot;</strong> That&#39;s the audit-trail win over email, in one column.</p><h3>13. Closing the loop: feedback and failure</h3><p>Feedback isn’t a nice-to-have in self-service — it <em>is</em> the service. If a user can’t tell whether their upload worked, you haven’t replaced the email workflow, you’ve just hidden it.</p><p><strong>When a file is validated,</strong> feedback fires two ways: an inline job-run summary (per-file ✅/❌) and (optionally) ane mail to the dataset’s notify list if something failed. It fires at the user-triggered validation, and again if a later scheduled ingest ever meets a bad file that slipped through.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9WrAw07GgUbZkXty40acUQ.png" /><figcaption>Per-file pass/fail in the run, plus (optionally) an email (or Slack/Teams) when something’s wrong.</figcaption></figure><p><strong>When something fails <em>after</em> a file is accepted</strong> (i.e. beyond validation), the principle is simple: <strong>consumers never see a half-populated table.</strong> Bronze rows still land and stay queryable by version; the previous good silver/gold stays exactly as-is until the next successful run; and the failure surfaces through standard job alerts. Nobody downstream gets a surprise.</p><h3>14. The code structure</h3><p>You don’t need my exact code, but you do need the <em>shape</em>:</p><pre>self_service_csv/<br>  configs/<br>    contracts.yaml        # schemas[] -&gt; files[] -&gt; schema + dq_checks<br>    constants.py<br>  helpers/<br>    filesystem.py         # list / move files on the volume<br>    logging.py<br>    notify.py             # job-run summary + email on failure<br>  app/<br>    scanner.py            # discover + match files to a contract<br>    validator.py          # DQX: apply contract checks, split pass/quarantine<br>    writer.py             # append to bronze (_hist) + archive<br>    orchestrator.py       # scan -&gt; validate -&gt; write/quarantine -&gt; notify<br>    main.py               # stakeholder-triggered entrypoint (param: uc_schema)</pre><p>The orchestrator is the whole story in about ten lines — discover, then for each file validate, write-or-quarantine, and notify:</p><pre>def run(uc_schema):<br>    items = sorted(discover(inbox_for(uc_schema), config_for(uc_schema)))<br>    for item in items:<br>        result = validate(item.path, item.contract)     # DQX gate (§9)<br>        if result.passed:<br>            append_to_bronze(result.df, item)            # + metadata cols (§12)<br>            archive(item)<br>        else:<br>            quarantine(item, result.errors)<br>            notify(item.contract.notify, result.errors)<br>    return summary(items)</pre><p>And the entrypoint is deliberately dumb: one generic, parameterized job per group serves a single stakeholder group. It triggers the generic/template job by passsing a parameter.</p><pre># Databricks job param -&gt; one job serves every group<br>args = parse_args()            # --uc-schema finance_data<br>run(uc_schema=args.uc_schema)</pre><p>One generic/template job. One stakeholder-facing job. Many stakeholder groups.</p><h3>15. Trade-offs, and what I’d reconsider</h3><p>I promised honesty, so here’s where this design is heavier than I’d like:</p><ul><li><strong>Config lives in two mindsets.</strong> Provisioning and runtime <em>should</em> share one YAML — but keeping that discipline is on you. Lose it and you get drift.</li><li><strong>Group management is real overhead.</strong> Groups-not-people is the right call, but somebody still curates the groups and the access requests.</li><li><strong>Explicit versioning is toil.</strong> _v2 tables are safe, but safe isn&#39;t free — every bump is a small migration.</li><li><strong>The more dynamic the files, ther harder it is to manage. </strong>It takes and investment to setup a stakeholder group, volume, jobs, contracts. This setup works great if you set it once and keep reusing it. If it requires constant updates, it may defeat the purpose.</li></ul><p>And what I’d keep an eye on:</p><ul><li><strong>Auto Loader</strong>, the moment volume goes continuous and high-throughput.</li><li><strong>Auto CSV-dialect detection</strong>, to smooth the stakeholder UX (while keeping schema enforcement strict).</li><li><strong>Consolidating the contract format</strong>, so there’s genuinely one shape everywhere.</li></ul><p>None of these are dealbreakers. They’re the price of a system that’s governed instead of ad-hoc — and I’d pay it again.</p><h3>16. Takeaways</h3><p>If you remember five things, remember these:</p><ol><li><strong>Give users a governed surface, not another script.</strong> The surface is the product; Volumes made it cheap.</li><li><strong>A contract is the gate.</strong> Nothing reaches bronze without passing it — and expose the contract so users can self-check.</li><li><strong>Append, never overwrite.</strong> Version re-uploads and let downstream read a “latest” view.</li><li><strong>Metadata is the audit trail.</strong> One _source_modified_by column answers &quot;who did this?&quot; forever (reusing UC system tables).</li><li><strong>Let stakeholders trigger their own runs.</strong> That’s the line between self-service and “email, but slower.”</li></ol><p>The stack here was Databricks-flavored, but the spine is portable: a drop zone + a contract-driven validation gate + append-only versioning + rich metadata + user-triggered jobs. Build that on whatever platform you’re on and you’ll get Jake his afternoons back.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EMIdlUYmZKWusz4bqyfPIA.png" /><figcaption>The whole thing in one line: drop → contract-gated job → versioned bronze.</figcaption></figure><p>How have you solved the human-ETL problem on your stack? I’d genuinely like to know — tell me in the comments.</p><p>Happy loading, and may you never be the human ETL.</p><p><em>Filip</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80c5088ab491" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloud-native-consulting/stop-being-the-human-etl-how-to-let-business-users-load-csvs-into-databricks-self-service-80c5088ab491">Stop Being the Human ETL: How to Let Business Users Load CSVs into Databricks (Self-Service)</a> was originally published in <a href="https://medium.com/cloud-native-consulting">Cloud Native Consulting</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Connecting to a Storage Account using Databricks Unity Catalog in Azure]]></title>
            <link>https://medium.com/cloud-native-consulting/connecting-to-a-storage-account-using-databricks-unity-catalog-in-azure-db08f0b0afc9?source=rss----669804cd394f---4</link>
            <guid isPermaLink="false">https://medium.com/p/db08f0b0afc9</guid>
            <category><![CDATA[databricks-unity-catalog]]></category>
            <category><![CDATA[unity-catalog]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Filip Pastuszka]]></dc:creator>
            <pubDate>Mon, 04 May 2026 06:31:01 GMT</pubDate>
            <atom:updated>2026-05-04T06:31:01.569Z</atom:updated>
            <content:encoded><![CDATA[<p>This is a bare-minimum tutorial for getting access to a Storage Account from a Databricks workspace using Unity Catalog.</p><h3>How access works</h3><p>Every connection request goes through a two-step verification process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N_lCr06uzHRIOidgdDZdGQ.jpeg" /><figcaption>2-step auth verification process when trying to reach Storage Account.</figcaption></figure><ol><li><strong>User permission</strong> — the user needs at least READ permission in Unity Catalog on the target External Location. <em>(Answers: does Jake have access to this storage?)</em></li><li><strong>Databricks service permission</strong> — Databricks needs access to the Storage Account via a Databricks Access Connector backed by a Managed Identity. <em>(Answers: does the Databricks service have access to this storage?)</em></li></ol><p>If either condition is not met, the connection request will fail.</p><h3>How to connect</h3><h4>Prerequisites:</h4><p>(I would be releasing a quick, Terraform-backed way soon, so stay tuned :) )</p><ul><li>Azure Storage Account</li><li>Databricks Workspace</li><li>Databricks Access Connector + Managed Identity</li><li>Unity Catalog metastore already set up and attached to the workspace</li><li>Admin rights on UC &amp; Azure/Storage Account</li></ul><h4>Steps</h4><p><strong>1. Assign the Storage Account role to the Access Connector</strong></p><p>Grant the Access Connector’s Managed Identity the Storage Blob Data Contributor role on the Storage Account (or Storage Blob Data Reader if read-only access is sufficient).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ReeBuDhfxtKHtMIbp7TwYw.png" /></figure><p><strong>2. Create a Storage Credential in Unity Catalog</strong> <em>(admin permissions required)</em></p><p><strong>Option 1 — UI:</strong></p><p>Catalog → Settings → Credentials → Create Credential</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*DdSyUUqrDgNSOhOwG33sEQ.png" /></figure><p><strong>Option 2 — Databricks CLI:</strong></p><pre># https://docs.databricks.com/api/azure/workspace/storagecredentials/create<br>databricks storage-credentials create \<br>  --json &#39;{<br>    &quot;name&quot;: &quot;&lt;credential-name&gt;&quot;,<br>    &quot;azure_managed_identity&quot;: {<br>      &quot;access_connector_id&quot;: &quot;&lt;ACCESS_CONNECTOR_RESOURCE_ID&gt;&quot;<br>    }<br>  }&#39;</pre><p><strong>Option 3 — Terraform:</strong></p><pre>resource &quot;databricks_storage_credential&quot; &quot;unity_catalog&quot; {<br>  name    = &quot;&lt;storage-credential-name&gt;&quot;<br>  comment = &quot;Managed identity credential managed by Terraform&quot;<br><br>  azure_managed_identity {<br>    access_connector_id = data.terraform_remote_state.foundation.outputs.databricks_access_connector_id<br>  }<br><br>  depends_on = [module.databricks]<br>}</pre><p><strong>3. Create an External Location pointing to the Storage Account</strong></p><pre>CREATE EXTERNAL LOCATION IF NOT EXISTS my_external_location<br>  URL &#39;abfss://&lt;storage-container&gt;@&lt;storage-account&gt;.dfs.core.windows.net&#39;<br>  WITH (STORAGE CREDENTIAL `&lt;credential-name&gt;`);</pre><p><strong>4. Grant permissions to a user group</strong></p><pre>GRANT READ FILES, WRITE FILES<br>  ON EXTERNAL LOCATION my_external_location<br>  TO `data_engineers`;</pre><p><strong>5. Test connection</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SrRn80OKLMbMX9pzfrCJ8Q.png" /></figure><h3>How to do it securely</h3><p>If you create one credential per environment and grant access to a group without any further isolation, that group ends up with cross-environment access:</p><ul><li>development can reach production</li><li>pre-production can reach development</li><li>etc.</li></ul><p>That is clearly not what you want. Each environment should only be able to access its own storage. This is where <strong>Workspace Bindings</strong> come in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lmv-JVAv1zLqxT7MEelFqA.jpeg" /><figcaption>Ensuring that UC Credential (and associated Access Connector) can only access allowed environement.</figcaption></figure><p>By default, a newly created credential is available across your entire Azure tenant. You need two additional CLI commands to lock it down.</p><p><strong>Step 1 — Create the credential</strong> (same as before):</p><p>In the previous section we created credential programatically.</p><pre># 1) Create credential<br># https://docs.databricks.com/api/azure/workspace/storagecredentials/create<br>databricks storage-credentials create \<br>  --json &#39;{<br>    &quot;name&quot;: &quot;&lt;credential-name&gt;&quot;,<br>    &quot;azure_managed_identity&quot;: {<br>      &quot;access_connector_id&quot;: &quot;&lt;ACCESS_CONNECTOR_RESOURCE_ID&gt;&quot;<br>    }<br>  }&#39;</pre><p>After that step this our credential was created. Great!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_WhAmI7hbvtdCVQR7pW9Gg.png" /></figure><p>Only problem is that our credential is available for every person in our Azure galaxy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*c9OyXay-hNnFg2D0uGCqIg.png" /></figure><p><strong>Step 2 — Isolate the credential</strong> (disable access from all workspaces by default):</p><pre># https://docs.databricks.com/api/azure/workspace/storagecredentials/update<br>databricks storage-credentials update &lt;credential-name&gt; \<br>  --isolation-mode ISOLATION_MODE_ISOLATED \<br>  --profile &lt;profile-name&gt;</pre><p><strong>Step 3 — Bind the credential to a specific workspace</strong> (only this workspace can now use it):</p><pre># https://docs.databricks.com/api/azure/workspace/workspacebindings/updatebindings<br>databricks workspace-bindings update-bindings storage_credential &lt;credential-name&gt; \<br>  --profile &lt;profile-name&gt; \<br>  --json &#39;{<br>    &quot;add&quot;: [<br>      {<br>        &quot;workspace_id&quot;: &lt;workspace-id&gt;,<br>        &quot;binding_type&quot;: &quot;BINDING_TYPE_READ_WRITE&quot;<br>      }<br>    ]<br>  }&#39;</pre><p>After these three steps, the credential is scoped to a single workspace and cannot be used from any other. Each environment gets its own Access Connector, credential, and workspace binding — keeping dev, pre-prod, and prod fully isolated from each other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8QelBRtAntf736XqsASvTg.png" /></figure><p>After these three steps, the credential is scoped to a single workspace and cannot be used from any other. Each environment gets its own Access Connector, credential, and workspace binding — keeping dev, pre-prod, and prod fully isolated from each other.</p><p>Happy coding!</p><p><em>Filip</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db08f0b0afc9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloud-native-consulting/connecting-to-a-storage-account-using-databricks-unity-catalog-in-azure-db08f0b0afc9">Connecting to a Storage Account using Databricks Unity Catalog in Azure</a> was originally published in <a href="https://medium.com/cloud-native-consulting">Cloud Native Consulting</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Azure Databicks Authentication: Unity Catalog, Service Principal (SPN), and Managed Identity (MI)]]></title>
            <link>https://medium.com/cloud-native-consulting/azure-databicks-authentication-unity-catalog-service-principal-spn-and-managed-identity-mi-9e46cb5963df?source=rss----669804cd394f---4</link>
            <guid isPermaLink="false">https://medium.com/p/9e46cb5963df</guid>
            <category><![CDATA[azure-databricks]]></category>
            <category><![CDATA[unity-catalog]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[databricks-unity-catalog]]></category>
            <dc:creator><![CDATA[Filip Pastuszka]]></dc:creator>
            <pubDate>Mon, 30 Mar 2026 06:41:00 GMT</pubDate>
            <atom:updated>2026-03-30T06:41:00.469Z</atom:updated>
            <content:encoded><![CDATA[<p>If you’re working with Azure Databricks and need to connect to a Storage Account, you have three options: Service Principal (SPN), Managed Identity (MI), and Unity Catalog (UC). They are not equal — and the gap between them matters more than most tutorials let on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0fN7J5-rr19bv2RmdIdMGQ.jpeg" /></figure><p>This article walks through all three, ordered from newest to oldest. Unity Catalog is the recommended approach today. SPN and MI are still widely used in the wild, so understanding them is worth your time — but go in knowing they are legacy patterns.</p><pre>Evolution of Databricks storage authentication<br>─────────────────────────────────────────────<br>SPN (legacy)  →  Managed Identity (legacy)  →  Unity Catalog (recommended)<br>  secrets          no secrets                    no secrets + fine-grain access control<br>  no governance    no governance                 full governance</pre><h3>Unity Catalog — The Recommended Way</h3><p>Unity Catalog does more than govern who can query which table. It sits in the middle of every storage request and acts as the single enforcement point between your users and your data lake. No request reaches storage without passing through it first.</p><h4>How the authentication flow works</h4><pre>User query (SELECT / LIST / READ FILES)<br>        │<br>        ▼<br>Unity Catalog permission check<br>  ├── ✗ denied → request stops here, storage is never touched<br>  └── ✓ authorised<br>              │<br>              ▼<br>      Access Connector for Azure Databricks<br>      (dedicated Azure ARM resource)<br>              │<br>              ▼<br>      System-assigned Managed Identity<br>      obtains OAuth token from Azure AD<br>      — no secret, no SPN, nothing to leak —<br>              │<br>              ▼<br>      ADLS Gen2 / Blob Storage</pre><p>The key insight: if Unity Catalog says no, the request never reaches storage. The user cannot attempt to hit the blob endpoint through Databricks. There is no credential to steal, because authentication happens at the Azure infrastructure level — not in your code.</p><h4>The Access Connector is not a Managed Identity</h4><p>People often conflate the two. The Access Connector is a dedicated Azure ARM resource. The Managed Identity is baked into it automatically — same lifecycle, not a separate resource you manage.</p><pre>Azure Resource Group<br>│<br>├── Databricks Workspace<br>│<br>└── Access Connector for Azure Databricks   ← ARM resource you create<br>        └── system-assigned Managed Identity ← auto-created, same lifecycle<br>                └── RBAC role on Storage Account<br>                        e.g. Storage Blob Data Contributor</pre><p>When the Access Connector is deleted, the identity goes with it. You never touch the Managed Identity directly.</p><h4>Setup steps</h4><p>1.Grant the Access Connector access to your storage<strong><br></strong>In Azure Portal, go to your Storage Account → IAM → Add role assignment:<br><strong>Role: </strong>Storage Blob Data Contributor<br><strong>Assign to:</strong> Access Connector for Azure Databricks (your resource)</p><p>2. Create a storage credential in Unity Catalog. You can do this via the UI (Catalog → External Data → Credentials) or API call. I called mine: my_created_credential.</p><p>3. Create an external location pointing to your container:</p><pre>CREATE EXTERNAL LOCATION my_external_location<br>URL &#39;abfss://sa-container-lakehousemasterydev@salakehousemasterydev.dfs.core.windows.net/&#39;<br>WITH CREDENTIAL my_created_credential;</pre><p>4. Grant a user access</p><pre>-- Grant raw file access (this is a demo, we just want to get minimal access)<br>GRANT READ FILES ON EXTERNAL LOCATION my_external_location TO `&lt;my-email&gt;@gmail.com`;</pre><p>5. Test</p><pre>dbutils.fs.ls(&quot;abfss://sa-container-lakehousemasterydev@salakehousemasterydev.dfs.core.windows.net/&quot;)<br><br>&gt;&gt;&gt; []  # empty list means access is working</pre><h4>A common trap: the system credential</h4><p>When Unity Catalog is set up, Databricks auto-creates a credential for its own internal metastore storage. It looks like a normal credential but is locked to Databricks’ own internal path:</p><pre>SHOW STORAGE CREDENTIALS;<br><br>-- dbx_lakehousemastery_dev_7405605994753309  ← UC internal, do not use<br>-- my_created_credential                      ← yours, use this one</pre><p>Using the system credential for your own storage will give you this error:</p><pre>PERMISSION_DENIED: The credential &#39;dbx_lakehousemastery_dev_7405605994753309&#39;<br>is a workspace default credential that is only allowed to access data in:<br>abfss://unity-catalog-storage@dbstoragefzs3pqydictam.dfs.core.windows.net/...</pre><h4>Benefits:</h4><ul><li>Fine-grained access control: table, column, and row-level permissions per user</li><li>No secrets to manage: nothing to rotate, nothing to leak</li><li>Full data lineage: UC tracks where data came from and who accessed it</li><li>Centralised governance: one control plane across all workspaces</li></ul><h4>Drawbacks:</h4><ul><li>More setup complexity: metastore, credentials, external locations, grants</li><li>Requires UC-enabled workspace: not available on all legacy workspace configurations</li></ul><h3>Legacy Approaches</h3><p>The following two methods are still in use across many organisations. Understanding them is valuable — both to work with existing setups and to appreciate why Unity Catalog exists.</p><h3>Service Principal (SPN)</h3><p>A Service Principal is an Azure AD application identity. You create it, give it a secret, and use that secret in your code to authenticate against storage.</p><pre>Your Notebook / Cluster config<br>        │<br>        ├── tenant_id<br>        ├── app_id        ← registered in Azure AD<br>        └── client_secret ← the credential that can leak<br>                │<br>                ▼<br>        login.microsoftonline.com  (public endpoint)<br>                │<br>                ▼<br>        OAuth token<br>                │<br>                ▼<br>        Storage Account</pre><h4>Setup steps</h4><p><strong>1.</strong> Register an Enterprise App in Azure AD and note the Application (client) ID.</p><p><strong>2.</strong> Create a client secret: App Registrations → your app → Certificates &amp; secrets → New client secret. Copy it immediately — you will not see it again.</p><p><strong>3.</strong> Grant the SPN a role on your container or storage account (Storage Blob Data Contributor at minimum).</p><p><strong>4.</strong> Configure Spark to use the SPN:</p><pre>storage_account = &quot;salakehousemasterydev&quot;<br>tenant_id       = &quot;678b9a92-*****-9sb87ecbas45&quot;<br>spn_app_id      = &quot;s7f93kd0-*****-0273sdk72ksa&quot;  # store in Key Vault using DBX Secret Scope<br>spn_secret      = &quot;*****&quot;  # store in Key Vault using DBX Secret Scope<br><br>spark.conf.set(f&quot;fs.azure.account.auth.type.{storage_account}.dfs.core.windows.net&quot;, &quot;OAuth&quot;)<br>spark.conf.set(f&quot;fs.azure.account.oauth.provider.type.{storage_account}.dfs.core.windows.net&quot;,<br>               &quot;org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider&quot;)<br>spark.conf.set(f&quot;fs.azure.account.oauth2.client.id.{storage_account}.dfs.core.windows.net&quot;, spn_app_id)<br>spark.conf.set(f&quot;fs.azure.account.oauth2.client.secret.{storage_account}.dfs.core.windows.net&quot;, spn_secret)<br>spark.conf.set(f&quot;fs.azure.account.oauth2.client.endpoint.{storage_account}.dfs.core.windows.net&quot;,<br>               f&quot;https://login.microsoftonline.com/{tenant_id}/oauth2/token&quot;)</pre><p>5. Test:</p><pre>dbutils.fs.ls(&quot;abfss://sa-container-lakehousemasterydev@salakehousemasterydev.dfs.core.windows.net/&quot;)<br><br>&gt;&gt;&gt; []  # empty list means access is working</pre><p>The Azure AD token endpoint is public. If `tenant_id`, `app_id`, and `client_secret` leak — from an attack, a git commit, a screensho, unless you’ve explicitly added network restrictions (like storage firewalls or private endpoints), those credentials can be used from anywhere on the internet. Your VPN does nothing here.</p><pre>What an attacker needs:         Where they can use it:<br>──────────────────────────────  ───────────────────────────────────<br>tenant_id + app_id + secret  →  any machine, anywhere on the internet<br>                                login.microsoftonline.com is public<br>                                storage endpoint is public<br>                                VPN protects neither</pre><h4>Benefits:</h4><ul><li>Simple to understand — familiar OAuth2 client credentials flow</li><li>Widely documented — many years of community knowledge</li></ul><h4>Drawbacks:</h4><ul><li>Credentials can leak — exposed in notebooks, logs, git history</li><li>No fine-grain control — cluster access = data access</li><li>Secret rotation required — ecrets expire; rotation is operational overhead</li><li>Bypasses Unity Catalog — no lineage, no table-level permissions</li></ul><h3>Managed Identity (MI)</h3><p>Managed Identity removes the secret entirely. Instead of your code holding a credential, the Azure infrastructure authenticates on behalf of the compute resource — in this case, the Databricks cluster.</p><pre>Databricks Cluster<br>        │<br>        └── Managed Identity attached to cluster<br>                │<br>                ▼  (infrastructure-level auth, no secret in code)<br>        Azure AD<br>                │<br>                ▼<br>        OAuth token<br>                │<br>                ▼<br>        Storage Account</pre><h4>Setup steps</h4><p><strong>1.</strong> Assign a Managed Identity to your Databricks cluster (or use the workspace-level identity).</p><p><strong>2.</strong> Grant that identity the appropriate role on your storage account (i.e. Storage Blob Data Contributor).</p><p><strong>3.</strong> Configure Spark:</p><pre>tenant_id     = &quot;678b9a92-*****-9sb87ecbas45&quot;<br>mi_client_id  = &quot;9skasd73-*****-9djgfr72sl0k&quot;<br><br>spark.conf.set(&quot;fs.azure.account.auth.type&quot;, &quot;OAuth&quot;)<br>spark.conf.set(&quot;fs.azure.account.oauth.provider.type&quot;,<br>               &quot;org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider&quot;)<br>spark.conf.set(&quot;fs.azure.account.oauth2.msi.tenant&quot;, tenant_id)<br>spark.conf.set(&quot;fs.azure.account.oauth2.client.id&quot;, mi_client_id)</pre><p>4. Test:</p><pre>dbutils.fs.ls(&quot;abfss://sa-container-lakehousemasterydev@salakehousemasterydev.dfs.core.windows.net/&quot;)<br><br>&gt;&gt;&gt; []  # empty list means access is working</pre><p>Managed Identity eliminates the credential leak problem. There is no secret in your code, nothing to rotate, and nothing to accidentally commit to git. That is a meaningful improvement over SPN.</p><p><strong>But the governance problem remains unchanged:</strong></p><pre>SPN:               cluster access = storage access (secret in code)<br>Managed Identity:  cluster access = storage access (no secret, same outcome)<br>Unity Catalog:     cluster access ≠ storage access (UC checks per user, per table)</pre><p>Anyone who can run code on the cluster can read the storage. There is no per-user, per-table, or per-column control. For a personal project or a small trusted team, this may be acceptable. For an organisation with multiple teams and sensitivity levels, it is not.</p><h4>Benefits:</h4><ul><li>No secrets to manage — nothing to rotate or leak</li><li>Simpler than SPN — no app registration or secret storage needed</li></ul><p>Drawbacks:</p><ul><li>No fine-grain control — cluster access = data access, same as SPN</li><li>Bypasses Unity Catalog — no lineage, no table-level permissions</li></ul><h3>Choosing the Right Approach</h3><pre>Situation                                      Recommended approach<br>─────────────────────────────────────────────────────────────────────<br>New workspace, greenfield setup                Unity Catalog<br>Existing UC-enabled workspace                  Unity Catalog<br>Multi-team org, mixed data sensitivity         Unity Catalog (non-negotiable)<br>Legacy workspace, UC not available             Managed Identity<br>Quick personal project, no org requirements    Managed Identity<br>You inherited SPN-based setup                  Migrate to UC when possible</pre><p>If you are starting from scratch, use Unity Catalog. The additional setup cost is paid once. The governance, lineage, and security benefits compound over time.</p><p>Happy coding!</p><p>Filip</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e46cb5963df" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloud-native-consulting/azure-databicks-authentication-unity-catalog-service-principal-spn-and-managed-identity-mi-9e46cb5963df">Azure Databicks Authentication: Unity Catalog, Service Principal (SPN), and Managed Identity (MI)</a> was originally published in <a href="https://medium.com/cloud-native-consulting">Cloud Native Consulting</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[13 Mistakes to Avoid in Databricks]]></title>
            <link>https://medium.com/cloud-native-consulting/13-mistakes-to-avoid-in-databricks-ae489abd3b11?source=rss----669804cd394f---4</link>
            <guid isPermaLink="false">https://medium.com/p/ae489abd3b11</guid>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Filip Pastuszka]]></dc:creator>
            <pubDate>Mon, 23 Feb 2026 08:41:01 GMT</pubDate>
            <atom:updated>2026-02-23T08:41:00.890Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bgqj4r2BOAWHbJAz7hkwEw.png" /></figure><p>Databricks is like a well-oiled machine when it’s set up properly. Smooth. Predictable. Almost suspiciously boring.</p><p>But I often see teams — whether it’s because they’re taking shortcuts, don’t know any better, don’t have enough time, or simply don’t care enough — make small setup erros that seem completely innocent at the time.</p><p>And they are innocent. At first.</p><p>Until they start compounding. Until they quietly introduce risk. Until bugs creep in. Until costs go up. Until your team slows down because nobody quite understands what’s going on anymore.</p><p>None of these mistakes look dramatic on day one. That’s the tricky part.</p><p>Let’s have a look at the top offenders.</p><h3>Identity &amp; Access</h3><h4>1. Using personal user accounts in production</h4><p><strong>Issue:</strong> Your colleague goes on holiday, leaves the company, or changes roles. Jobs they owned break or disappear. Permissions vanish. No bueno.</p><p><strong>Better alternative:</strong> Use service principals (OAuth) or managed identities. Bind jobs to a principal scoped just for that workload — no single point of human failure.</p><pre># Example: Use a service principal to run a job<br>dbutils.notebook.run(&quot;/Jobs/my_job&quot;, 60, {&quot;user&quot;: dbutils.secrets.get(scope=&quot;prod&quot;, key=&quot;sp_user&quot;)})</pre><h4>2. Over-privileged service principals</h4><p><strong>Issue:</strong> One “super” SP with admin rights for everything. Easy shortcut that can lead to issues. Accidental table exposure or cluster misconfig happens fast.</p><p><strong>Better alternative:</strong> Scope permissions to the minimum needed per workload. Follow least-privilege principles.</p><h3>Data Governance</h3><h4>3. Ignoring Unity Catalog &amp; data governance</h4><p><strong>Issue:</strong> Relying on ad-hoc workspace ACLs leads to inconsistent permissions, manual tracking, and audit hell.</p><p><strong>Better alternative:</strong> Centralize governance with Unity Catalog. Apply table-level ACLs and automate permission policies across clusters and notebooks.</p><pre># Example: Grant read access to analytics team<br>grant:<br>  principal: &quot;analytics_team&quot;<br>  privileges: [&quot;SELECT&quot;]<br>  on: &quot;delta.my_table&quot;</pre><h3>Secrets &amp; Credentials</h3><h4>4. Hardcoding credentials in notebooks or scripts</h4><p><strong>Issue:</strong> Credentials in notebooks = instant security risk. Rotations, leaks, and breaches are inevitable.</p><p><strong>Better alternative:</strong> Use Databricks Secrets or a vault. Never store passwords, keys, or tokens in code.</p><pre># BAD<br>db_user = &quot;user&quot;<br>db_password = &quot;pass...&quot;<br><br># GOOD<br>db_user = secrets.get(scope=&quot;prod&quot;, key=&quot;db_user&quot;)<br>db_password = secrets.get(scope=&quot;prod&quot;, key=&quot;db_password&quot;)</pre><h3>Cluster &amp; Cost Management</h3><h4>5. Leaving clusters idle or using all-purpose clusters for production</h4><p><strong>Issue:</strong> High cloud bills. Idle clusters cost money; all-purpose clusters mix workloads → instability.</p><p><strong>Better alternative:</strong> Use job clusters for production jobs. Auto-terminate idle clusters. Pin library versions.</p><pre>spark_version: &quot;13.1.x-scala2.12&quot;<br>auto_terminate: 30  # minutes<br>libraries:<br>  - pypi: pandas==2.1.0</pre><h3>Environment &amp; Code Hygiene</h3><h4>6. No environment separation</h4><p><strong>Issue:</strong> Production, staging, dev mashed together → accidental overwrites, job conflicts, unpredictable results.</p><p><strong>Better alternative:</strong> Separate environments per workload/stage. Clearly defined dev, staging, production.</p><p>via CI/CD pipelines.</p><pre># config.yaml<br>environments:<br>  dev: &quot;abfss://dev.devsa.dfs.core.windows.net/&quot;<br>  staging: &quot;abfss://staging.stagingsa.dfs.core.windows.net/&quot;<br>  prod: &quot;abfss://prod.prodsa.dfs.core.windows.net/&quot;</pre><h4>7. No version control configured, code lives in Databricks only</h4><p><strong>Issue:</strong> Workspace-only code storage = no history, no rollback, no blame assignment &lt;wink&gt;. Debugging becomes painful.</p><p><strong>Better alternative:</strong> Git integration. Branch per feature, PRs for reviews, merge to main when ready. What I like to use:</p><ul><li>local development (+ ad-hoc testing in dev)</li><li>open PR deploys to dev</li><li>merge to main deploys to staging/pre-prod</li><li>tag/release deploys to prod</li></ul><h4>8. Inconsistent table naming / no layers</h4><p><strong>Issue:</strong> Hard to navigate, debug, or onboard. One person’s “raw” might be another person’s “bronze.”</p><p><strong>Better alternative:</strong> Adopt a table layering convention:</p><ul><li>bronze → silver → gold, OR</li><li>staging → integration → datamart OR</li><li>ingestion → processing → consumption</li></ul><p>you can call it whatever you want, just be consistent.</p><h4>9. Over-reliance on notebooks</h4><p><strong>Issue:</strong> Everything in one cell, no modularity, notebooks used as orchestrators. Easy to start, low complexity but scaling and testing becomes painful as you grow.</p><p><strong>Better alternative:</strong> Modularize notebooks: use helper scripts, config files, and constants. Treat notebooks like applications, not scripts.</p><pre># helper.py<br>def clean_data(df):<br>    return df.dropna().withColumnRenamed(&quot;old&quot;, &quot;new&quot;)</pre><pre># main notebook<br>from helpers import clean_data<br>df_clean = clean_data(df_raw)</pre><h3>Deployment &amp; Testing</h3><h4>10. Manual job/cluster/script changes via UI</h4><p><strong>Issue:</strong> Clicling through UI may be fun but it’s not scalable and it does not help to maintain your app. Each change should go through version control.</p><p><strong>Better alternative:</strong> Use IaC / Databricks Asset Bundles. Deploy reproducible changes via code.</p><h4>11. No data quality or logic tests</h4><p><strong>Issue:</strong> Bad data propagates silently. Logic errors compound.</p><p><strong>Better alternative:</strong> It’s a bit of a broad topic, but generally mature teams should absolutely have:</p><ul><li>DQ tests on the data itself (daily runs)</li><li>Unit tests while developing &amp; deploying via CI/CD</li><li>Table Performance &amp; Storage</li></ul><p>If in doubt, use Beyonce’s rule of thumb:</p><blockquote>If you like it, then you shoulda put a test on it</blockquote><h4>12. Poor table optimization</h4><p><strong>Issue:</strong> No partitioning, no Z-order, skipped vacuuming, ignoring file size → slow queries, high storage costs.</p><p><strong>Better alternative:</strong> Partition appropriately, optimize with Z-order/liquid clustering, vacuum old files, monitor file size.</p><pre>VACUUM silver.mytable RETAIN 168 HOURS;<br>OPTIMIZE silver.mytable ZORDER BY (user_id);</pre><h3>Visibility &amp; Monitoring</h3><h4>13. No cost or usage audits</h4><p><strong>Issue:</strong> Costs grow silently. Teams surprised by bills or cluster abuse.</p><p><strong>Better alternative:</strong> Implement dashboards, alerts, and regular audits. Make cost and usage visible to responsible owners.</p><h3>Summary</h3><p>Databricks is great — but pitfalls are easy to hit:</p><ul><li>Use proper identities, least-privilege service principals, and central governance.</li><li>Never hardcode secrets; pin cluster versions.</li><li>Separate environments, version control everything, and modularize notebooks.</li><li>Automate deployment, test your logic, and check your data.</li><li>Optimize tables and maintain visibility on costs.</li></ul><p>Happy coding!<br> Filip</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae489abd3b11" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloud-native-consulting/13-mistakes-to-avoid-in-databricks-ae489abd3b11">13 Mistakes to Avoid in Databricks</a> was originally published in <a href="https://medium.com/cloud-native-consulting">Cloud Native Consulting</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Calculating the real cost of a Databricks Environment]]></title>
            <link>https://medium.com/cloud-native-consulting/calculating-the-real-cost-of-a-databricks-environment-8958619330b8?source=rss----669804cd394f---4</link>
            <guid isPermaLink="false">https://medium.com/p/8958619330b8</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Filip Pastuszka]]></dc:creator>
            <pubDate>Mon, 06 Oct 2025 06:01:57 GMT</pubDate>
            <atom:updated>2025-10-06T06:01:56.588Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i-xgRIBXIfZXJocfgk7v6A.png" /></figure><h3>⚡️ TL;DR</h3><p>Databricks costs go far beyond compute. You’re paying for networking, storage, security, and governance, even if you never spin up a single Spark job. If you’re running workloads on Azure/AWS, expect your <strong>Databricks bill</strong> to include:</p><ul><li><strong>Compute (clusters or serverless):</strong> your biggest cost driver</li><li><strong>Networking (NAT, IPs, Private Link):</strong> at least €30 of a static cost</li><li><strong>Storage (ADLS Gen2/S3):</strong> cheap per GB, but watch out for small files &amp; Delta history</li><li><strong>Key Vault / Secrets:</strong> almost free but worth tracking</li><li><strong>Unity Catalog / Governance:</strong> free feature, but compute behind it isn’t</li></ul><p>And yes — you can absolutely optimize it (we’ll get there).</p><h3>Why Databricks Costing Deserves More Attention</h3><p>Databricks pricing is often misunderstood. Most engineers think in terms of <em>“my cluster costs X per hour”</em> — and forget the rest of the iceberg (no pun intended) sitting under the waterline.</p><p>Networking, storage, and even governance quietly contribute to your monthly invoice.</p><p>This article breaks down every cost component in a <strong>typical Azure Databricks environment</strong> — what it is, who pays for it, and how to keep it from spiraling out of control.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*Y8V4mzI-kuKT-e9OxNryCw.png" /></figure><h3>⚙️ Compute — The Core of Databricks Costs</h3><p>This is where the magic happens (and most of the money goes).</p><h4>Classic Compute (Clusters)</h4><p>Your total cost comes from three main components:</p><ul><li>VM/EC2 cost (actual servers)</li><li>DBU cost (Databricks)</li><li>Storage and network I/O</li></ul><p>Optional: enable <strong>Photon</strong> acceleration to reduce DBU consumption for SQL workloads.</p><p><strong>Formula:</strong><br> <em>Total = (DBUs × DBU rate) + (VMs × compute cost) + (storage/network I/O)</em></p><h4>Serverless Compute</h4><p>If you use <strong>serverless</strong>, Databricks manages the infrastructure for you.<br> You just pay per second of actual compute time — no idle costs.</p><p>Includes:</p><ul><li>Serverless SQL Warehouses</li><li>Serverless Jobs / Notebooks</li><li>Delta Live Tables (DLT)</li></ul><p>✅ Tip: Serverless can save <strong>30–50%</strong> for short or bursty jobs since there’s no idle time.</p><h3>🏗️ The Control Plane &amp; Networking Layer</h3><p>Your workspace is the command center — the part of Databricks that lives in your Azure subscription. It’s where clusters are spun up, jobs triggered, and data sent flying across virtual networks.</p><p><strong>Common cost components:</strong></p><ul><li><strong>NAT Gateway:</strong> Handles outbound traffic to the internet. Billed hourly and per GB.</li><li><strong>Public IP addresses (optional):</strong> Used for workspace or cluster access. Charged per hour.</li><li><strong>VNet Peering / Private Link:</strong> Secure connection between workspace and data VNet. Adds data transfer fees.</li><li><strong>Azure Load Balancer:</strong> Required for Private Link setups. Billed hourly.</li><li><strong>Managed Services (Control Plane):</strong> Databricks-managed infra. Included in DBU cost.</li></ul><p>✅ Tip: For idle workspaces, static networking costs can quietly eat up significant amount of resources at scale so terminate them when not needed (don’t ask me how I learned that).</p><h3>📦 Storage — ADLS Gen2 (or S3)</h3><p>Databricks doesn’t own your data storage — you do. For example Azure bills you directly for ADLS Gen2 usage.</p><p>Here’s what contributes to cost:</p><ul><li><strong>Storage (per GB/month):</strong> main cost for storing tables, logs, checkpoints</li><li><strong>Hierarchical Namespace (HN):</strong> required for folder-like structure</li><li><strong>Transactions / API Calls:</strong> each read/write to Delta tables counts</li><li><strong>Snapshots &amp; Versions:</strong> Delta table history adds storage overhead</li><li><strong>External Tables:</strong> queried via Unity Catalog, billed as normal Azure storage</li></ul><p>✅ Tip: Run VACUUM regularly and compact small files. Both small files and long Delta history inflate cost faster than you’d expect.</p><h3>🔐 Security &amp; Secrets — Azure Key Vault</h3><ul><li>Key Vault stores credentials and secrets securely.</li><li>You’re charged per secret operation (read/write) and per stored secret.</li><li>The cost is negligible — fractions of a euro — but still worth tracking for completeness.</li></ul><h3>Other Components Worth Knowing</h3><ul><li><strong>Unity Catalog:</strong> free to use, but compute that queries data still costs DBUs.</li><li><strong>Model Serving / MLflow:</strong> charged per serving minute in DBUs.</li><li><strong>Instance Pools:</strong> pre-warmed VMs for faster startup; you pay for idle pool time.</li></ul><h3>Example: Realistic Azure Scenario (per workspace)</h3><p>Let’s say we have a small data team with:</p><ul><li>Scenario in <strong>Azure (West Europe)</strong>:</li><li>1 workspace</li><li>3 daily batch jobs, 1 hour each, using <strong>classic Databricks compute</strong></li><li>Each job writes 10 GB to ADLS Gen2</li><li>3 secrets stored in <strong>Azure Key Vault</strong></li></ul><p>Here’s a rough guestimate (based on <strong>West Europe</strong> pricing):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/1*-lc4Bm-8SQYU47pz1fBo8A.png" /></figure><h4>⚙️ Compute</h4><ul><li><strong>Cluster:</strong> Standard_D4ds_v5 (4 vCPU, 16 GB)</li><li><strong>Size:</strong> 1 driver + 4 workers → 5 nodes</li><li><strong>DBU per cluster-hour:</strong> 10 DBU (for the whole cluster)</li><li><strong>DBU rate:</strong> ~€0.28 / DBU-hour</li><li><strong>VM cost:</strong> ~€0.23 / node-hour × 5 nodes = €1.15/hour</li><li><strong>Hourly cost: </strong>DBU: 10 × €0.28 = €2.80, VM: €1.15 =&gt; <strong>Total ≈ €3.95 / hour</strong></li><li><strong>Monthly cost:</strong> 3 jobs × 1 hour/day × 30 days = 90 hours, 90 × €3.95 ≈ <strong>€355 / month</strong></li></ul><p>And that’s <em>per workspace</em>. If you replicate this setup across dev, test, and prod — multiply accordingly.</p><h4>🗄️ Storage — ADLS Gen2</h4><ul><li>3 jobs × 10 GB × 30 days = ~900 GB/month</li><li>Storage ~€0.02 / GB → <strong>~€18/month</strong></li><li>Transaction / Delta history costs may add a few euros</li></ul><h4>🌐 Networking — NAT Gateway &amp; VNet</h4><ul><li><strong>NAT Gateway resource cost:</strong> ~€0.045/hour × 720 hrs ≈ €32/month</li><li><strong>Data processed:</strong> ~€0.045 / GB × 900 GB ≈ €40/month (if traffic goes through NAT)</li><li><strong>Public IP / VNet Peering:</strong> ~€5/month</li></ul><p><strong>Total networking:</strong> €77/month (worst case)<br> ✅ Could be lower if traffic stays inside Private Link or VNet.</p><h4>🔐 Security &amp; Secrets</h4><ul><li>3 secrets in Key Vault: ~€0.10/month</li><li>Unity Catalog: free (compute billed separately)</li></ul><h3>💰 Total Estimated Monthly Cost (Per Workspace)</h3><ul><li>Compute: <strong>€355</strong></li><li>Storage: €18</li><li>Networking: €77</li><li>Key Vault: €0.10</li></ul><p><strong>≈ €450/month</strong></p><p>And that’s <em>per workspace</em>. If you replicate this setup across dev, test, and prod — multiply accordingly.</p><h3>🧩 Hidden Costs to Keep an Eye On</h3><ul><li>Idle clusters burning DBUs overnight.</li><li>Delta table versioning ballooning storage.</li><li>“Temporary” workspaces that never get deleted.</li></ul><h3>🧾 Final Thought — How to Keep Your Databricks Bill Happy</h3><p>A few battle-tested tricks:</p><ol><li>Use auto-termination — don’t trust engineers to manually shut clusters.</li><li>Prefer serverless for short or spiky workloads.</li><li>Enable Photon — it often pays for itself in speed.</li><li>Monitor Delta tables — clean small files, set retention.</li><li>Centralize cost monitoring using cost manager, i.e. Azure Cost Management.</li></ol><p>Happy coding,</p><p>Filip</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8958619330b8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloud-native-consulting/calculating-the-real-cost-of-a-databricks-environment-8958619330b8">Calculating the real cost of a Databricks Environment</a> was originally published in <a href="https://medium.com/cloud-native-consulting">Cloud Native Consulting</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Full Reloads to Smart Loads: Incremental Loading in Spark]]></title>
            <link>https://medium.com/cloud-native-consulting/from-full-reloads-to-smart-loads-incremental-loading-in-spark-829c92fbfdfd?source=rss----669804cd394f---4</link>
            <guid isPermaLink="false">https://medium.com/p/829c92fbfdfd</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Filip Pastuszka]]></dc:creator>
            <pubDate>Mon, 18 Aug 2025 05:46:11 GMT</pubDate>
            <atom:updated>2025-08-18T05:47:07.454Z</atom:updated>
            <content:encoded><![CDATA[<h3>From Full Reloads to Smart Loads: 3 Scenarios for Incremental Loading in Spark</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4B9J-VOiJHynxYqAPnfuRQ.jpeg" /></figure><p>Processing a full dataset every time can feel like drinking from a firehose — fast, overwhelming — plus your cloud bill may cry for mercy. Incremental loading — picking only the new or changed records — keeps your pipelines lean, efficient, and just a touch fun.</p><h3>Understand the nature of your upstream source:</h3><p>Before deciding your approach, answer these questions:</p><ul><li>Can new <strong>records arrive late</strong> (i.e., records from last week may be loaded today)?</li><li>Can <strong>past records change </strong>(be upserted)?</li><li>Does the source support <strong>change flags</strong> (e.g. load_datetime)?</li><li>What is your <strong>primary key</strong> and <strong>timestamp column</strong>?</li></ul><p>Your strategy will depend on these answers. Let’s explore common scenarios.</p><h3>Scenario A: Streaming Ingestion (for real-time or near-real-time)</h3><p>If your data arrives continuously, streaming ingestion is the easiest way to ensure only new records are loaded.</p><p><strong>Tools</strong>: Kafka, Kinesis, Azure Event Hubs, or cloudFiles (Autoloader).</p><p><strong>Benefits:</strong></p><ul><li>Ideal when <strong>new data arrives continuously</strong>.</li><li>Handles <strong>deduplication</strong> and incremental ingestion with <strong>checkpoints</strong>.</li><li>With Autoloader, you can set up <strong>schema evolution</strong>, <strong>triggered or continuous modes</strong>, and scale easily in cloud environments.</li></ul><p><strong>Best Practices</strong>:</p><ul><li>Use trigger(once=True) for incremental batch-style streaming.</li><li>Partition by event date for scalability.</li><li>Store ingestion_timestamp for lineage and reprocessing.</li></ul><h3>Scenario B: Append-Only Data (no retroactive changes)</h3><p>If records cannot change retroactively, you can simply process new rows since the last load — which is not very realistic but can happen.</p><p><strong>Ideal for</strong>: Logs, events, telemetry, incremental exports.</p><ul><li>In that case we can query the target table for the latest events since last processing, using i.e. event_load_dt:</li></ul><p>Upstream table:</p><pre>+--------------------------+--------------------------+--------+----------+---------+<br>|edl_load_dt               |order_dt                  |order_id|product_cd|price_eur|<br>+--------------------------+--------------------------+--------+----------+---------+<br>|2025-08-16 08:12:05.001234|2025-08-15 04:36:54.841235|1       |5984      |160.49   |<br>|2025-08-16 08:12:05.001234|2025-08-15 04:36:54.841235|1       |3905      |54.49    |<br>|2025-08-16 08:12:05.001234|2025-08-15 01:05:05.378596|2       |1031      |29.99    |<br>|2025-08-16 08:12:05.001234|2025-08-15 07:37:34.136758|3       |9053      |99.99    |<br>+--------------------------+--------------------------+--------+----------+---------+</pre><p>Create initial orders table:</p><pre>CREATE OR REPLACE TABLE orders AS<br>SELECT * <br>FROM upstream_table</pre><p>let’s assume that our upstream_table has a new row:</p><pre>+--------------------------+-------------------------+--------+----------+---------+<br>|edl_load_dt               |order_dt                 |order_id|product_cd|price_eur|<br>+--------------------------+-------------------------+--------+----------+---------+<br>|2025-08-17 00:12:05.005832|2025-08-15 07:19:06.75625|4       |5834      |19.99    |<br>+--------------------------+-------------------------+--------+----------+---------+</pre><p>Insert only new records based on MAX(edl_load_dt):</p><pre>INSERT INTO orders<br>SELECT *<br>FROM upstream_table<br>WHERE edl_load_dt &gt; (SELECT MAX(edl_load_dt) FROM orders);</pre><p>Query the orders table:</p><pre>%sql<br>SELECT * FROM orders</pre><p>Result:</p><pre>+--------------------------+--------------------------+--------+----------+---------+<br>|edl_load_dt               |order_dt                  |order_id|product_cd|price_eur|<br>+--------------------------+--------------------------+--------+----------+---------+<br>|2025-08-16 13:39:32.606303|2025-08-15 04:36:54.841235|1       |5984      |160.49   |<br>|2025-08-16 13:39:32.606333|2025-08-15 04:36:54.841235|1       |3905      |54.49    |<br>|2025-08-16 13:39:32.606346|2025-08-15 01:05:05.378596|2       |1031      |29.99    |<br>|2025-08-16 13:39:32.606358|2025-08-15 07:37:34.136758|3       |9053      |99.99    |<br>|2025-08-16 21:39:32.60637 |2025-08-15 07:19:06.75625 |4       |5834      |19.99    |<br>+--------------------------+--------------------------+--------+----------+---------+</pre><p><strong>Pro tip:</strong></p><ul><li>Schedule periodic backfills to correct for latency (if exists)</li></ul><p><strong>Pros</strong>:</p><ul><li>Simple, cheap, fast.</li></ul><p><strong>Caveats</strong>:</p><ul><li>Assumes no retroactive changes.</li><li>Susceptible to missing late-arriving records if not backfilled.</li><li>May cause small file problems — consider periodic re-compaction.</li></ul><h3>Scenario C: Mutable Data (retroactive changes possible)</h3><p>When data can change in the past, you need to reconcile old and new rows. The EDL_LOAD_DT column may not always exist or be reliable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/0*yKn-Rc3fJe4u_QnK.png" /><figcaption>Credit: <a href="https://delta.io/blog/2023-02-14-delta-lake-merge/">https://delta.io/blog/2023-02-14-delta-lake-merge/</a></figcaption></figure><p><strong>Option 1: Delta MERGE INTO (Full Scan)</strong></p><p>Our upstream_table initially:</p><pre>+--------------------------+--------------------------+--------+----------+---------+----------+<br>|edl_load_dt               |order_dt                  |order_id|product_cd|price_eur|order_date|<br>+--------------------------+--------------------------+--------+----------+---------+----------+<br>|2025-08-16 08:12:05.001234|2025-08-15 04:36:54.841235|1       |5984      |160.49   |2025-08-15|<br>|2025-08-16 08:12:05.001234|2025-08-15 04:36:54.841235|1       |3905      |54.49    |2025-08-15|<br>|2025-08-16 08:12:05.001234|2025-08-15 01:05:05.378596|2       |1031      |29.99    |2025-08-15|<br>|2025-08-16 08:12:05.001234|2025-08-15 07:37:34.136758|3       |9053      |99.99    |2025-08-15|<br>+--------------------------+--------------------------+--------+----------+---------+----------+</pre><p>Create initial orders table — we also paritioned by order_date:</p><pre>CREATE OR REPLACE TABLE orders<br>PARTITIONED BY (order_date) -- we partition here<br>AS<br>SELECT *<br>FROM upstream_table;</pre><p>Let’s merge upstream_table into orders table:</p><pre>MERGE INTO orders target<br>USING upstream_table source<br>ON target.order_id = source.order_id<br>AND target.product_cd = source.product_cd<br>AND target.order_dt = source.order_dt<br>AND target.order_date = source.order_date --partition column for partition pruning!<br><br>--WHEN MATCHED THEN UPDATE SET * <br>--WHEN NOT MATCHED BY SOURCE THEN DELETE -- both statements would significantly show the job<br>WHEN NOT MATCHED THEN INSERT *</pre><h4>Option 2: Delta MERGE INTO with already_processed flag</h4><p>Look at the new condition here — we use WHERE already_processed = FALSE to ensure we only take records from upstream_table that have NOT been processed yet:</p><pre>MERGE INTO orders target<br>USING (<br>    SELECT * <br>    FROM upstream_table<br>    WHERE already_processed = FALSE --key phrase here<br>) AS source<br>ON target.order_id = source.order_id<br>AND target.product_cd = source.product_cd<br>AND target.order_dt = source.order_dt<br>AND target.order_date = source.order_date --partition column for partition pruning!<br><br>--WHEN MATCHED THEN UPDATE SET *<br>WHEN NOT MATCHED THEN INSERT *<br>-- WHEN NOT MATCHED BY SOURCE THEN DELETE</pre><p>Now time to update upstream_table with already_processed=TRUE flag:</p><pre>UPDATE upstream_table<br>SET already_processed = TRUE<br>WHERE already_processed = FALSE<br>  AND EXISTS (<br>        SELECT 1<br>        FROM orders o<br>        WHERE o.order_id = upstream_table.order_id<br>          AND o.product_cd = upstream_table.product_cd<br>          AND o.order_dt = upstream_table.order_dt<br>          AND o.order_date = upstream_table.order_date<br>  );</pre><h4><strong>Tips for Delta MERGE:</strong></h4><ul><li>Allows us to update previous records</li><li>Partition by date for pruning and make sure to add that to merge condition = if in doubt use EXPLAIN MERGE INTO ... and check the physical plan — look for <strong>partition filters</strong> applied before the merge.</li><li>Use ZORDER to optimize joins (OPTIMIZE target ZORDER BY (id))</li><li>Skip unchanged records with hash_diff, already_processed, whenever possible.</li></ul><h3>Designing for Incremental Loading</h3><p><strong>Choose your strategy:</strong></p><ol><li><strong>Streaming ingestion</strong>: Kafka or Autoloader — catch data as it flows</li><li><strong>Append-only, immutable data</strong>: Filter source on max(edl_load_dt) + occasional backfills — quick and safe — also not very realistic.</li><li><strong>Mutable data</strong>: Delta MERGE with partition pruning + upsert already_processed = TRUE — the careful cleanup option.</li></ol><p>Mastering these moves ensures you load only new or changed records efficiently, keeping pipelines lean, cost-effective, and a little fun.</p><p>Happy coding!</p><p>Filip</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=829c92fbfdfd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloud-native-consulting/from-full-reloads-to-smart-loads-incremental-loading-in-spark-829c92fbfdfd">From Full Reloads to Smart Loads: Incremental Loading in Spark</a> was originally published in <a href="https://medium.com/cloud-native-consulting">Cloud Native Consulting</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Used a Nokia 3310 for a Week — Here’s What I Learned]]></title>
            <link>https://medium.com/cloud-native-consulting/i-used-a-nokia-3310-for-a-week-heres-what-i-learned-575342bd3336?source=rss----669804cd394f---4</link>
            <guid isPermaLink="false">https://medium.com/p/575342bd3336</guid>
            <category><![CDATA[mindfulness]]></category>
            <category><![CDATA[smartphones]]></category>
            <category><![CDATA[lifestyle]]></category>
            <category><![CDATA[digital]]></category>
            <category><![CDATA[productivity]]></category>
            <dc:creator><![CDATA[Filip Pastuszka]]></dc:creator>
            <pubDate>Tue, 10 Jun 2025 06:02:14 GMT</pubDate>
            <atom:updated>2025-06-10T06:02:13.626Z</atom:updated>
            <content:encoded><![CDATA[<h3>I Used a Nokia 3310 Instead of My iPhone for a Week — Here’s What Happened</h3><p>Yes, you read that right. I voluntarily turned off my iPhone and went full “analog” for seven days, swapping my sleek smartphone for a chunky Nokia 3310 (v2).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*A_ANiRnTJmP_5Uet4k63cA.png" /></figure><p>If you’re imagining a tech detox, you’re half right. It was more like stepping out of a Tesla and climbing onto a rusty scooter. It still gets you from point A to B — but forget heating, Bluetooth, GPS, or… a camera that works.</p><h3>Why would I do this to myself?</h3><p>Honestly? Curiosity. Balance. And maybe a bit of masochism.</p><p>We’re on autopilot with our phones. They ping, we respond. They buzz, we check. It’s compulsive. Like caffeine, social media, or doomscrolling before bed. Sometimes the only way to notice how wired in you are is to <em>cut the cord completely</em>.</p><h3>TL;DR: Here’s the Short Version</h3><ul><li>Smartphones are <strong>amazing tools</strong>, not enemies.</li><li>Going offline was initially <strong>liberating</strong>, later just <strong>annoying</strong>.</li><li>I missed music, maps, messages, notes and daily utility apps.</li><li>I didn’t become a tech-free monk — just more intentional.</li><li>Now, I use my phone with better boundaries and appreciation.</li></ul><h3>Preparation: The Digital Cleanse Begins</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FEaNmpbrr3QA%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DEaNmpbrr3QA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FEaNmpbrr3QA%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/1d66b8756507143cfb061d9ffd423d96/href">https://medium.com/media/1d66b8756507143cfb061d9ffd423d96/href</a></iframe><p>Before Day 1, I made a few key moves:</p><ul><li>Set an <em>“I’m offline this week”</em> status on WhatsApp.</li><li>Pulled SIM cards from my smartphones and deactivated WiFi on them.</li><li>Allowed laptop and internet for <strong>work only</strong> — no social media, no YouTube, no rabbit holes.</li></ul><h3>The Rules</h3><ul><li>No smartphone use (unless totally essential, like alarms).</li><li>No internet-connected devices.</li><li>No modern messaging apps or social platforms.</li><li>Yes to laptops — for <em>work only</em>.</li></ul><h3>Day 1 — Liberation Day</h3><p>Waking up without the glow of my screen felt oddly refreshing. I was free. Present. Even smug. I shared the challenge with friends — who either laughed or said, “Why!?”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6O7FLiu3uCUVmAJNm9fxiQ.png" /><figcaption>This was my WhatsApp picture for the week.</figcaption></figure><p>I went through the day <em>without feeling the itch</em> to check messages. Even walking to the gym felt peaceful. No headphones meant I actually noticed birds. And people. And how much time I all of the sudden had.</p><p><strong>Rating</strong>: 🌟🌟🌟🌟🌟 Off the leash</p><h3>Day 2 — My Phone Holds 5 Photos, Total</h3><p>That’s not a joke. Five pictures — <strong>total</strong>. Suddenly, every snapshot had to <em>earn</em> its place. No random memes. No 47 angles of your lunch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/240/1*JiE1ZCspvJRJNvIkF-hCkA.jpeg" /><figcaption>Photo of my gym training plan at max resolution.</figcaption></figure><p>Also, I had no music during workouts, no podcasts during cooking. Biking in silence was… weird. But I did catch snippets of real life — conversations, car horns, laughter. It made me feel connected in an old-school way.</p><p><strong>Rating</strong>: 🌟🌟🌟 Slightly less cute</p><h3>Day 3 — Lost in Boxtel</h3><p>Without Google Maps, I planned routes like it was 1999. A trip outside Amsterdam nearly ended in disaster until I asked a stranger for directions. Twice.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fh9omnhs4Hyw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dh9omnhs4Hyw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fh9omnhs4Hyw%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f2f7e82a7c5be2f6d1fd3674b7ff9888/href">https://medium.com/media/f2f7e82a7c5be2f6d1fd3674b7ff9888/href</a></iframe><p>Also, texting on a T9 keyboard is medieval. I started ignoring messages because replying was <em>work</em>.</p><p><strong>Rating</strong>: 🌟🌟 Retro struggle is real</p><h3>Day 4 — Present but Disconnected</h3><p>Not using WhatsApp meant missing out on invites, updates and conversations. Group chats? Dead to me. Want to send a pic? Good luck with MMS (if that even works anymore).</p><p>I began craving not dopamine — but <strong>connection</strong>. The real benefit of smartphones isn’t distraction, it’s staying in sync with your world and people you care about.</p><p><strong>Rating</strong>: 🌟🌟🌟 Emotionally unplugged</p><h3>Day 5 — Mild Breakdown</h3><p>The novelty wore off. I was over it.</p><p>I forgot my wallet because I usually use Apple Pay. I couldn’t check class schedules. I couldn’t send a location to a freind. At one point I genuinely considered bartering my phone for a sandwich when I had no way to pay.</p><p>Still, I pushed through. Barely.</p><p>Snake became my main source of entertainment. I broke my personal record: 117 points. Sadly, a useless life skill.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*g4q_8R9NbA7imouOnJ_rvA.png" /></figure><p><strong>Rating</strong>: 🌟 Painfully analog</p><h3>Day 6 — The Beach Epiphany</h3><p>Spent the day at the beach — ironically only possible because a friend <em>with a smartphone</em> handled the car rental, directions, and payments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H6nqCu6D4UnLPdjzLAFVmQ.jpeg" /></figure><p>I realized smartphones aren’t evil. They’re not distractions — they’re <strong>infrastructure</strong>. They <em>run</em> your day. Logistics, money, communication, entertainment. Everything.</p><p><strong>Rating</strong>: 🌟🌟🌟🌟 Smart friends are better</p><h3>Day 7 — Sweet, Sweet Digital Reunion</h3><p>At 12:01, I turned my iPhone back on.</p><p>I blasted music. Replied to a week’s worth of messages. Checked in on friends. It felt… incredible. But also calmer than before. I wasn’t frantically refreshing anything. Just using the tool when needed.</p><h3>What I’ll Keep (and Ditch) Going Forward</h3><h3>✅ Start</h3><ul><li><strong>Permanent Focus Mode</strong>: iPhone’s new “Intelligent Breakthrough &amp; Silencing” mode lets AI decide if a notification is truly worth it. Spoiler: most aren’t.</li></ul><h3>🔁 Continue</h3><ul><li>Communication, navigation, emails, banking.</li><li>Music: It’s my mood modulator. Helps me work, move, feel.</li><li><a href="https://www.forestapp.cc/">Forest App</a>: I love planting virtual trees for every focus session.</li><li>No social media: IG, YouTube, TikTok.</li></ul><h3>❌ Stop</h3><ul><li>Podcasts &amp; commentary overload: I no longer need constant noise.</li><li>Mindless scrolling: If I’m bored, I <em>stay bored</em>. It’s healthy.</li></ul><h3>Final Thoughts: It’s Not the Phone, It’s the Pattern</h3><p>We love to blame smartphones for our attention span and dopamine addiction. But really, <strong>they’re not the villain</strong> — our habits are.</p><p>This challenge made me appreciate just how powerful, helpful, and — yes — <em>essential</em> these tiny machines are. But also how easy it is to let them take over.</p><p>Now, me and my phone? We’re on speaking terms again. Just with a little more respect.</p><p><strong>Would I recommend trying this?</strong><br> Absolutely. Even just for a few days. Not to punish yourself — but to see how deep the rabbit hole goes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=575342bd3336" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloud-native-consulting/i-used-a-nokia-3310-for-a-week-heres-what-i-learned-575342bd3336">I Used a Nokia 3310 for a Week — Here’s What I Learned</a> was originally published in <a href="https://medium.com/cloud-native-consulting">Cloud Native Consulting</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Are You Deploying Databricks Code the Right Way? Meet Asset Bundles (+video).]]></title>
            <link>https://medium.com/cloud-native-consulting/are-you-deploying-databricks-code-the-right-way-meet-asset-bundles-video-4e557f1863a5?source=rss----669804cd394f---4</link>
            <guid isPermaLink="false">https://medium.com/p/4e557f1863a5</guid>
            <category><![CDATA[databricks-asset-bundles]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Filip Pastuszka]]></dc:creator>
            <pubDate>Mon, 02 Jun 2025 05:51:33 GMT</pubDate>
            <atom:updated>2025-06-02T05:51:33.628Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d_aSZUnxaNbo5rQASqj8cQ.png" /></figure><p>Databricks Asset Bundles might just be the missing piece in your deployment workflow — especially if you’re tired of managing job clusters, notebooks, models, and configurations manually across dev, staging, and production environments.</p><p>In this post, I’ll walk you through how <strong>Databricks Asset Bundles</strong> work, how they simplify CI/CD for your projects, and show a full <strong>demo</strong> of deploying from development to pre-production — effortlessly.</p><blockquote><em>🎥 Want to watch instead?</em></blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FPk1hOE36TqA%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DPk1hOE36TqA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FPk1hOE36TqA%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6ec6ffa1f7654c04b041f6f94e6d6cab/href">https://medium.com/media/6ec6ffa1f7654c04b041f6f94e6d6cab/href</a></iframe><h3>🚀 What Are Databricks Asset Bundles?</h3><p>Asset Bundles let you define and manage your Databricks jobs, clusters, and other resources as code — in a consistent, version-controlled way. Think of them as the glue between your <strong>feature development</strong> and your <strong>automated CI/CD pipelines</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*YXj-AV4k8HxSWGvXhvItSg.jpeg" /><figcaption>Left to right</figcaption></figure><p>Instead of managing configurations manually or with Terraform + Databricks APIs, Asset Bundles abstract away the complexity into a <strong>developer-friendly CLI tool</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RTEcsWoBAmWtAW8rMpyMAw.jpeg" /><figcaption>If TF and Databricks APIs had a baby…</figcaption></figure><p>Use cases:</p><ul><li>Manage job and cluster definitions, notebooks, and code assets</li><li>Move code safely between environments (dev → staging → prod)</li><li>Integrate with Git and your CI pipeline</li></ul><h3>🧪 Typical Use Case: From Feature to Production</h3><p>Let’s imagine you’re developing a new feature, and it needs:</p><ul><li>Updated job logic</li><li>A new cluster size</li><li>A couple of new tables or views</li></ul><p>With Asset Bundles:</p><ol><li>You develop and test in your dev workspace.</li><li>Once tested, you merge changes into your main branch.</li><li>Your CI pipeline uses the databricks bundle deploy command to push changes to pre-prod or prod.</li></ol><p>That’s it, simple as that.</p><h3>🛠️ Getting Started — Step-by-Step Demo</h3><p>Here’s what I did in my demo:</p><h3>1. Set Up Two Workspaces</h3><p>One for <strong>development</strong>, one for <strong>pre-production</strong>. I used Terraform to create them, but you can also do it manually via Azure/AWS/GCP.</p><p>Want to see how to automate workspace setup with Terraform? Check out my post on that here:</p><p><a href="https://medium.com/cloud-native-consulting/effortlessly-manage-databricks-workspaces-with-terraform-4a75a6bdf4dd">Effortlessly manage Databricks workspaces with Terraform</a></p><h3>2. Install the Databricks CLI</h3><p>For macOS:</p><pre>brew install databricks</pre><p>Then log into both environments using:</p><pre>databricks auth login --host &lt;workspace_url&gt;</pre><h3>3. Initialize a Bundle</h3><pre>databricks bundle init</pre><p>Finish init by choosing:</p><ul><li>Template: Python, SQL, DBT, …</li><li>Catalog: I used hive_metastore for simplicity</li><li>Profile: Your dev workspace</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i8F_4FmdMUF2u0SgdwRFCA.png" /></figure><h3>4. Modify the Bundle</h3><p>Make any changes you want — e.g., add a new SQL view, edit a job definition, or change cluster parameters. For this demo, I:</p><ul><li>Created two views</li><li>Added a new one called new_table</li><li>Updated the job name</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/1*BsVurASjb8t2vHI0dFuuZQ.png" /></figure><h3>5. Deploy to Development</h3><pre>databricks bundle deploy -t dev</pre><p>And that’s it — the job, views, and resources appear in the dev workspace. Fast, clean, and versioned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*RRE1O-8GlDEmadTA19-SOQ.png" /></figure><h3>6. Deploy to Pre-Production</h3><pre>databricks bundle deploy -t preprod</pre><p>This moves your <strong>tested changes</strong> to a staging-like environment. Production deployments would work exactly the same — likely triggered by a Git tag or merge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/1*IjfmvqnCYKz-vBYaMlz65Q.png" /></figure><p>Please note that my job does not have [dev filippatuszka] prefix because it’s deployed in productionmode.</p><h3>🤖 Bonus: Real-World Considerations</h3><p>In a full setup, you’d also want to:</p><ul><li><strong>Use service principals or short-lived tokens</strong> for secure, non-user deployments.</li><li><strong>Automate with CI/CD</strong>: Trigger deployments from GitHub Actions or Azure DevOps.</li><li><strong>Handle access control</strong>: Restrict write access in prod, allow devs freedom in dev.</li></ul><h3>💬 Wrapping Up</h3><p>Databricks Asset Bundles simplify multi-environment management <strong>massively</strong>. Once you set up your bundles and pipeline, deployments become reliable, reproducible, and hands-off.</p><p>Happy coding!</p><p>Filip</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e557f1863a5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloud-native-consulting/are-you-deploying-databricks-code-the-right-way-meet-asset-bundles-video-4e557f1863a5">Are You Deploying Databricks Code the Right Way? Meet Asset Bundles (+video).</a> was originally published in <a href="https://medium.com/cloud-native-consulting">Cloud Native Consulting</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Overwriting My Tables! 3 ways to fix it in Databricks]]></title>
            <link>https://medium.com/cloud-native-consulting/stop-overwriting-my-tables-3-ways-to-fix-it-in-databricks-daa3f52323ee?source=rss----669804cd394f---4</link>
            <guid isPermaLink="false">https://medium.com/p/daa3f52323ee</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Filip Pastuszka]]></dc:creator>
            <pubDate>Wed, 14 May 2025 04:29:43 GMT</pubDate>
            <atom:updated>2025-05-14T04:29:42.979Z</atom:updated>
            <content:encoded><![CDATA[<p>Have you ever worked on a data engineering project where there was <strong>no clear way to isolate your development work</strong> from other engineers? It might look something like this:</p><p>Business as usual in our main development data environment…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dI5YzjBq0z21iBlqkbqrhA.jpeg" /></figure><p>Then Jake starts developing a new feature. He modifies tables: <strong>Silver_1</strong> and <strong>Gold_1</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RYVwzcOMpY7-yI2SPrnedA.jpeg" /></figure><p>Simultaneously, Paul begins working on another feature and modifies <strong>Silver_2</strong> and <strong>Gold_1</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u3IW_1_RSTYW-mH93VXsKw.jpeg" /></figure><p>Later, Jake spends two frustrating hours debugging his changes, only to realize that his logic was perfectly fine — <strong>Paul had overwritten Gold_1</strong> in the background.</p><p>Jake starts to wonder whether he should have taken over his father’s dry-cleaning business after all. Maybe this life of silent table conflicts wasn’t meant for him.</p><p>Sounds familar, minus the dry-cleaning?</p><h3>There has to be a better way</h3><p>Enters isolated data schemas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/0*Rk1q-jqP7kowBSiR.gif" /></figure><p>The goal is simple: <strong>developers should be able to work independently without stepping on each other’s toes.</strong> Once they’re done, they can safely merge their changes into the main environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*BroQZ9Ccb2ha3tON134J_w.jpeg" /></figure><p>In the visual above, both Jake and Paul work in <strong>isolated environments</strong>. They make changes, test confidently, and merge clean code into main branch. Jake’s father is proud.</p><h3>How to achieve schema isolation in Databricks?</h3><p>Here are <strong>three practical ways</strong> to isolate development environments in Databricks.</p><ol><li><strong>Use Databricks Asset Bundles (Recommended)</strong></li></ol><p>Databricks engineers saw this coming — and built a solution into <a href="https://docs.databricks.com/aws/en/dev-tools/bundles/"><strong>Databricks Asset Bundles</strong></a>. All you need is a development mode in your databricks.yml.</p><pre># databricks.yml<br><br>targets:<br>  dev_user:<br>    # The default target uses &#39;mode: development&#39; to create a development copy.<br>    # - Deployed resources get prefixed with &#39;[dev my_user_name]&#39;<br>    # - Any job schedules and triggers are paused by default.<br>    # See also https://docs.databricks.com/dev-tools/bundles/deployment-modes.html.<br>    <br>    mode: development # &lt;&lt;&lt;- magic line!<br><br>    default: true<br>    workspace:<br>      host: https://adb-1592122XXXXXXXXX.15.azuredatabricks.net</pre><p>Then deploy your development bundle:</p><pre>databricks bundle deploy -t dev_user<br><br>&gt;&gt;&gt; Uploading bundle files to /Workspace/Users/filipxxxxxxxxxxxx@gmail.com/.bundle/cncdemo/dev_user/files...<br>&gt;&gt;&gt; Deploying resources...<br>&gt;&gt;&gt; Updating deployment state...<br>&gt;&gt;&gt; Deployment complete!</pre><p>And my demo workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*BKHQOOBKC_K7ch94sVUong.png" /></figure><p>test_table created as a part of this workflow is now part of my own personal schema, not interfering with anyone else’s work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*NcyrNXSQCqKavTd3qkpSDQ.png" /></figure><p>2. <strong>Use DBT with Dynamic Schema Prefixing</strong></p><p>If you’re using DBT, isolating schemas is easy with this simple tweak in profiles.yml:</p><p>schema: &quot;{{ var(&#39;schema_prefix&#39;, &#39;&#39;) ~ &#39;my_default_schema&#39; }}&quot;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*JSqOKSnOtFV4VPG03bkvJA.png" /></figure><p>Now run:</p><p>dbt run -t $DBT_ENV — vars ‘{schame_prefix: filip_}’,</p><p>Your tables will be created under: filip_my_default_schema</p><p>If you omit the variable and run:</p><p>dbt run -t $DBT_ENV,</p><p>schema would default to: my_default_schema. Simple as that.</p><p>3. <strong>Use environment variables or user context</strong></p><p>For teams running custom jobs or notebooks, you can isolate data based on <strong>who runs the code</strong> using the built-in context API.</p><pre># for notebook users:<br>dbutils.notebook.entry_point.getDbutils().notebook().getContext().userName().get()<br>&gt;&gt;&gt; &#39;filipxxxxxxxxxxxxx@gmail.com&#39;<br><br># for all other users:<br>spark.sql(&#39;select current_user() as user&#39;).collect()[0][&#39;user&#39;]<br>&gt;&gt;&gt; &#39;filipxxxxxxxxxxxxx@gmail.com&#39;</pre><p>Now you can dynamically prefix your schemas or tables based on the active user — giving each engineer a private workspace.</p><h3>✨ Bonus: impact assessment option unlocked!</h3><p>With isolated schemas, you unlock an added benefit:<br> You can <strong>validate your data changes independently</strong> before merging to production. That means fewer bugs, safer deployments, and more confidence in your work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NEKC8pDI7SncXsCmI_ah6g.jpeg" /></figure><h3>Summary</h3><p>We explored three ways to create isolated development environments in Databricks:</p><ol><li><strong>Databricks Asset Bundles</strong> — Native and powerful (recommended)</li><li><strong>DBT schema prefixing</strong> — Lightweight and CI/CD-friendly</li><li><strong>User-based schema logic</strong> — Great for notebooks or custom apps</li></ol><p>Personally, I recommend using <strong>Databricks Asset Bundles</strong> if you’re already invested in the Databricks ecosystem. But for more flexibility or integration with tools like DBT, options 2 and 3 work just as well.</p><p><strong>Happy coding, and may your tables never clash again.</strong><br> — <em>Filip</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=daa3f52323ee" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloud-native-consulting/stop-overwriting-my-tables-3-ways-to-fix-it-in-databricks-daa3f52323ee">Stop Overwriting My Tables! 3 ways to fix it in Databricks</a> was originally published in <a href="https://medium.com/cloud-native-consulting">Cloud Native Consulting</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>