<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nevenka Lukic on Medium]]></title>
        <description><![CDATA[Stories by Nevenka Lukic on Medium]]></description>
        <link>https://medium.com/@nenamilkov?source=rss-10235c107f8c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*8D8eURKbaWufpkhQxs7jjg.jpeg</url>
            <title>Stories by Nevenka Lukic on Medium</title>
            <link>https://medium.com/@nenamilkov?source=rss-10235c107f8c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 02:07:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nenamilkov/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Designing Idempotent Data Pipelines]]></title>
            <link>https://medium.com/towards-data-engineering/designing-idempotent-data-pipelines-0017fc4e8a74?source=rss-10235c107f8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/0017fc4e8a74</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[duckdb]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Nevenka Lukic]]></dc:creator>
            <pubDate>Thu, 16 Jul 2026 10:57:50 GMT</pubDate>
            <atom:updated>2026-07-16T10:57:50.566Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Article 4 in the single-engineer data stack series. </em><a href="https://medium.com/towards-data-engineering/duckdb-the-death-of-small-scale-spark-e86ac6aedf23?sharedUserId=nenamilkov"><em>Article 1</em></a><em> swapped Spark for DuckDB. </em><a href="https://pythondataengineering.net/projects/polars-vs-pandas-production-pipelines"><em>Article 2 </em></a><em>swapped Pandas for Polars. </em><a href="https://changedatacapture.net/cdc-vs-full-refresh-etl/guide"><em>Article 3</em></a><em> swapped full-refresh ETL for log-based CDC. This one answers the question that haunts all three: what happens when any of these pipelines fails halfway and retries?</em></p><p>This article was originally published on <a href="https://pythondataengineering.net/blog/designing-idempotent-data-pipelines">Python Data Engineering.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T7RDqNggm2JDNQmkYVHnXg.jpeg" /></figure><h3>The Retry Nobody Thinks About</h3><p>Every pipeline fails eventually. Not might fail. Will fail. A network blip, a timeout, a worker that gets killed mid-batch. I have stopped being surprised by this. What surprised me for years was how rarely anyone asked the follow-up question.</p><p>The follow-up question is not whether your pipeline will fail. It is what happens the second time it runs.</p><p>Most engineers design for the happy path and let the orchestrator handle retries as an afterthought. Airflow retries the task. Dagster retries the task. dlt resumes from the last offset. All of that infrastructure assumes one thing without ever saying it out loud: that running the same logic twice on the same input produces the same result as running it once.</p><p>If that assumption is false, every retry is a small chance of corruption. Not a crash. Corruption. The kind that does not throw an error, does not page anyone, and sits quietly in a revenue table for months before someone notices the numbers do not add up.</p><p>This article is about making that assumption true. Not through better monitoring. Through pipeline design.</p><h3>What Non-Idempotent Pipelines Actually Cost</h3><p>I want to ground this in numbers before making any architectural claims, because vague warnings about “data quality” do not move anyone to action.</p><p>The diagram below summarizes three named incidents at very different scales.</p><figure><img alt="Three cards showing real data incidents. Card 1: An e-commerce analytics pipeline overcounted purchases by about 2x for five months because duplicate events lacked a shared deduplication key. Card 2: Kargo’s analytics pipeline failed for three days, becoming a $500,000 problem. Card 3: Citibank mistakenly wired about $900 million, 100 times the intended amount, in 2020 due to an operational re-execution error; about $500 million was never recovered." src="https://cdn-images-1.medium.com/max/1024/1*Hder4di7ogu7Eyo05BS1Ow.png" /><figcaption>Three documented incidents where unsafe retries or unsafe re-execution produced measurable financial and operational damage.</figcaption></figure><p>A widely cited analytics case describes an e-commerce team whose tracking showed roughly 4,200 purchase events in a month against 2,100 actual transactions in their payment processor. A clean 2x inflation, running undetected for five months. The cause was ordinary: a browser pixel and a server-side conversion API both firing without a shared key to recognize duplicates, plus mobile clients retrying on flaky connections.</p><p>Andy Owens, VP of Analytics at Kargo, described a pipeline failure in Monte Carlo’s “How Kargo Prevents Six-Figure Data Quality Issues.” His words: “One of our main pipelines failed because an external partner didn’t feed us the right information to process the data. That was a three day issue that became a $500,000 problem.”</p><p>And then there is Citibank. On August 11, 2020, an operator intending to send Revlon’s $7.8 million interest payment instead wired the full $900 million outstanding loan principal, roughly 100 times the intended amount. Ten investment firms kept more than $500 million of it under a legal defense a court initially upheld. This is not a CDC pipeline story. It is the cleanest illustration anywhere of why anything that touches money must be safe against accidental re-execution.</p><p>The pattern across all three: nobody designed for malice. Nobody made an obviously bad decision in the moment. The systems simply did exactly what they were told, more than once, and nothing stopped them.</p><h3>The Property That Makes Retries Safe</h3><p>Here is the concept that should have been taught to me on day one and was not. An operation is idempotent when applying it multiple times has the same effect as applying it once. In notation: f(f(x)) equals f(x).</p><p>I spent years treating this as an academic detail from API design, the reason <strong><em>PUT</em></strong> and <strong><em>DELETE</em></strong> are safe to retry but a naive <strong><em>POST /charges</em></strong> is not. It took me much longer to see that the exact same property is what separates a pipeline you can trust from one you have to babysit.</p><p>The architectural insight worth sitting with: true exactly-once delivery across a network is, for practical purposes, impossible to guarantee. Even Kafka’s exactly-once machinery reduces, on the consuming side, to at-least-once delivery. A consumer can always reprocess after a crash or a rollback. The entire industry has converged on a different, achievable pattern instead: at-least-once delivery, paired with idempotent processing.</p><p>This reframes the whole problem. You stop trying to guarantee each event arrives exactly once, which is genuinely hard. You start guaranteeing that it does not matter if an event arrives twice, which is a property you can design into a single SQL statement.</p><p>The diagram below makes the contrast concrete.</p><figure><img alt="Two-row diagram. Top row shows a naive append pipeline: batch fails halfway through writing, retry reinserts all rows including ones already written, target table now contains duplicate rows. Bottom row shows an idempotent merge pipeline: batch fails halfway through writing, retry re-applies the same merge statement keyed on a stable identifier, target table converges to the identical correct state regardless of how many times the retry runs." src="https://cdn-images-1.medium.com/max/1024/1*sbZ_cS4bHat0WO04cfLA8w.png" /><figcaption>The same failed batch retried twice, once against a naive append pipeline and once against an idempotent merge pipeline.</figcaption></figure><p>This connects directly back to Article 3. PostgreSQL’s own documentation states the deal plainly: the position of a replication slot persists only at checkpoint, so after a crash the slot can return to an earlier point in the log, and recent changes will be sent again. The database is telling you, explicitly, that it will redeliver. Your job is to make redelivery harmless.</p><h3>Idempotent MERGE Patterns In DuckDB And dbt</h3><p>The good news is that the pattern from Article 3 already does most of the work. The bad news is that “most” is not “all,” and the missing part is where the bugs live.</p><p><strong>DuckDB’s MERGE INTO</strong></p><p>The setup is the same as the rest of this series. If you have not already, <strong><em>uv add duckdb</em></strong> gets you everything you need, no separate installation step.</p><p>The keyed upsert from the CDC article is idempotent by construction, as long as two conditions hold. The match key has to be a stable identifier, the same logical row produces the same key every time. And the update has to be deterministic given the source row, no <strong><em>now()</em></strong>, no incrementing counters.</p><pre>MERGE INTO dim_customers AS target<br>USING staging_customers AS source<br>ON target.customer_id = source.customer_id<br>WHEN MATCHED THEN<br>    UPDATE SET<br>        status     = source.status,<br>        updated_at = source.updated_at<br>WHEN NOT MATCHED THEN<br>    INSERT (customer_id, status, updated_at)<br>    VALUES (source.customer_id, source.status, source.updated_at);</pre><p>Re-run this with the same staging data and every matched row updates to values it already holds. Net change: zero. That is the property you want.</p><p>One thing worth knowing if you are running this against Iceberg tables, which v1.5.3 supports directly. DuckDB’s own documentation notes that MERGE INTO on Iceberg uses merge-on-read semantics and writes positional deletes. A retry of an identical merge is logically idempotent, the queryable result converges correctly, but it is not physically idempotent. Every retry creates a new snapshot and new delete files, even when nothing actually changed. On a laptop-scale pipeline this is fine. At production scale, budget for table maintenance.</p><p><strong>dbt’s Incremental Strategies</strong></p><p>dbt’s own documentation is unusually candid about this: incremental models are the easiest place to accidentally break idempotence. Three strategies, two of them safe.</p><p><strong><em>merge</em></strong> with a correct <strong>unique_key</strong> gives you an idempotent upsert. <strong><em>delete+insert</em></strong> with the same key does the same thing through a different mechanism. <strong><em>append</em></strong>, the default if you forget to specify a strategy, gives you neither. It inserts every row the model returns with no deduplication at all.</p><p>I have seen this exact mistake more than once: an incremental model with no <strong><em>unique_key</em></strong> set, working fine in testing because the pipeline never failed in testing, then duplicating every row in production the first time a retry happened. The fix costs one line of configuration. The bug costs a cleanup project.</p><h3>Where The CDC Pipeline From Article 3 Breaks</h3><p>This is the section I want to be precise about, because the pg_replication plus DuckDB MERGE INTO pattern from the previous article is mostly idempotent, and mostly is exactly the gap that bites people.</p><p>A CDC change event carries an operation type: insert, update, delete, or the initial snapshot read. The naive assumption is that applying each event through MERGE handles everything correctly. Three specific edge cases say otherwise.</p><p>The diagram below shows the most common failure: an out-of-order replay overwriting newer data with older data.</p><figure><img alt="Two-row flow diagram. Top: a naive replay where an update at log position 5 is applied, the consumer crashes, then replay resends an older update from position 1. A naive merge overwrites the newer value with the older one. Bottom: the same replay, but a version guard only applies updates with a higher log position, so the stale replay is ignored and the newer value is preserved." src="https://cdn-images-1.medium.com/max/1024/1*5N7IBUXFWTVJ-80AFhU-Bw.png" /><figcaption>An out-of-order CDC replay after a crash, shown with and without a version guard on the merge statement.</figcaption></figure><p>First, out-of-order delivery. If a consumer crashes and replays an older change after a newer one already landed, a blind merge overwrites correct data with stale data. The fix is a version guard on the merge condition itself.</p><pre>MERGE INTO target t<br>USING changes s ON t.id = s.id<br>WHEN MATCHED AND s._lsn &gt; t._lsn THEN<br>    UPDATE SET<br>        data = s.data,<br>        _lsn = s._lsn    -- persist the log position so future replays are guarded<br>WHEN NOT MATCHED THEN<br>    INSERT (id, data, _lsn)<br>    VALUES (s.id, s.data, s._lsn);</pre><p>Second, duplicate or replayed delete events. A delete that gets re-applied after a row has already been re-inserted can wrongly remove a row that should exist. Deletes need the same version guard as updates, not a blanket “remove anything not present” rule unless you are working from a full snapshot.</p><p>Third, and this one surprised me when I first hit it: a crash mid-transaction causes the entire transaction to replay from its beginning, not just the unconfirmed part. Your sink has to tolerate seeing a whole transaction’s worth of events twice, not just one stray row.</p><p>The practical fix for all three: a merge keyed on a stable identifier, with an explicit version or log-position guard on every write branch, including deletes. That single change is what turns “mostly idempotent” into actually idempotent.</p><h3>Deduplication And Checkpoint Design</h3><p>Stripe’s idempotency key is the reference point everyone in this space eventually rediscovers independently. The mechanism: a client generates a key, sends it with a mutating request, and on any failure retries with the identical key. Stripe stores the result of the first request and returns it for any later request carrying the same key.</p><p>The data pipeline version of this is the same idea in different clothes. Your match key, whether a natural key or a content hash, plays the role of the idempotency key. Your checkpoint or offset store, the replication slot’s log position from Article 3, plays the role of Stripe’s key store.</p><p>Three deduplication patterns cover almost every case I encounter. <strong><em>ROW_NUMBER()</em></strong> partitioned by key and ordered by a timestamp, keeping only the latest row, works well as long as the ordering has no ties. Content hashing, computing a hash of the meaningful business columns, catches duplicates even when the source system reissues a row with a different envelope. Deterministic identifiers, UUIDv5 generated from a stable namespace and natural key rather than random UUIDv4, mean reimporting the same data produces the same IDs automatically.</p><p>The pattern I now default to in any pipeline touching money or counts that matter: hash the business key, guard every write with a version check, and test the retry before the bug ever has a chance to ship.</p><h3>The Engineer Who Trusts Their Retries</h3><p>There is a specific kind of confidence that comes from knowing a pipeline can fail safely. Not the confidence of believing it will not fail. The confidence of not caring much when it does.</p><p>This week, take the pipeline in your stack you trust the least, the one you check on manually after every run. Run it twice on the same input, deliberately. With uv, that is <strong><em>uv run pipeline.py</em></strong> twice in a row against the same source data. Compare the output. If anything differs, you have found exactly where the next incident is waiting.</p><p>I should have run that test years before I did. The fix, almost every time, was smaller than the anxiety the missing fix had been causing me.</p><p>DuckDB gave you a compute layer that does not need a cluster. Polars gave you a transformation layer that does not need pandas. CDC gave you an ingestion layer that does not need a nightly reload. Idempotency is what lets you trust all three when they fail, because they will. The engineer who designs for that failure in advance is not being pessimistic. They are the only one in the room not hoping nothing goes wrong.</p><p>I write weekly about data engineering: real benchmarks, working code, and honest takes on the tools worth your time. <strong><em>Follow</em></strong> me here if that sounds useful.</p><p><em>Sources: Monte Carlo, “How Kargo Prevents Six-Figure Data Quality Issues,” 2024. PostgreSQL documentation, “Logical Decoding Concepts,” 2025. Stripe API documentation, “Idempotent Requests,” 2025. DuckDB documentation, “MERGE INTO,” 2025–2026. dbt documentation, “About Incremental Models,” 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0017fc4e8a74" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/designing-idempotent-data-pipelines-0017fc4e8a74">Designing Idempotent Data Pipelines</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CDC Explained: Why Full Refresh ETL Is Dying]]></title>
            <link>https://medium.com/towards-data-engineering/cdc-explained-why-full-refresh-etl-is-dying-3175e29d9697?source=rss-10235c107f8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/3175e29d9697</guid>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[duckdb]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Nevenka Lukic]]></dc:creator>
            <pubDate>Sat, 04 Jul 2026 08:12:53 GMT</pubDate>
            <atom:updated>2026-07-04T08:12:53.451Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>This is the third article in a trilogy on the modern single-engineer data stack. The first covered</em><a href="https://medium.com/@yourusername/duckdb-the-death-of-small-scale-spark"><em> DuckDB replacing small-scale Spark</em></a><em>. The second covered</em><a href="https://pythondataengineering.net/projects/polars-vs-pandas-production-pipelines"><em> Polars replacing Pandas in production pipelines</em></a><em>. This one closes the ingestion layer.</em></h4><p>This article was originally published on <a href="https://changedatacapture.net/cdc-vs-full-refresh-etl/guide"><strong>Change Data Capture</strong></a><strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0GI1Gne1QPk_gIdSyTsSIQ.png" /></figure><h3>The Default Nobody Questions</h3><p>There is a pipeline running somewhere right now that truncates a table, reads every row from the source database, and reloads the entire thing. It has been doing this every night for three years. Nobody remembers writing it. Nobody questions it. It works, mostly, and mostly is good enough.</p><p>I know this pipeline. I have written versions of it. It is the first thing you reach for because it is simple, it is self-healing, and it requires almost no operational knowledge to maintain. You can explain it to a stakeholder in one sentence: we copy the whole table every night.</p><p>The problem appears slowly. The table grows. The window stretches. The source database starts complaining about the read load at 2 AM. Someone adds a retry loop. Someone else adds a timeout. The pipeline that used to take eight minutes now takes ninety. Nothing broke. It just quietly became expensive and fragile.</p><p>Full refresh ETL is not wrong. It is carrying assumptions from a different era, when tables had fifty thousand rows and nobody needed data fresher than yesterday morning. The question is whether those assumptions still match the workloads you are actually running.</p><p>For most production pipelines above a few million rows, they do not.</p><p>If you have not read the first two articles in this series, the short version is this. DuckDB gives you a compute layer that replaces Spark for pipelines under a hundred gigabytes. Polars gives you a transformation layer that replaces pandas for file-based ETL above one gigabyte. CDC is what feeds both of them with data that is actually fresh.</p><h3>What Full Refresh Actually Costs</h3><p>The costs of full refresh are real and measurable. They do not appear in a single incident report. They accumulate silently across every run.</p><p>The comparison below maps the four dimensions where full refresh and log-based CDC diverge most sharply.</p><figure><img alt="Comparison table with four rows. Row one: Compute cost, full refresh reprocesses all rows every run, CDC reads only changed rows near-zero marginal cost. Row two: Data freshness, full refresh tied to batch cadence hours of lag, CDC sub-10-second latency. Row three: Delete capture, full refresh timestamp queries miss deletes, CDC captures hard deletes as op equals d events. Row four: Schema evolution, full refresh auto-infers but re-reads everything, CDC propagates additive changes automatically." src="https://cdn-images-1.medium.com/max/1024/1*1cRR0YzkSCL7s5iHJvoFSA.png" /><figcaption>Full refresh versus log-based CDC across compute cost, data freshness, delete capture, and schema evolution handling</figcaption></figure><p><strong>Compute Cost</strong></p><p>Every full refresh reprocesses data that did not change. Airbyte estimates that teams moving from full refresh to incremental and CDC pipelines commonly cut cloud data warehouse costs by thirty to sixty percent. That is not a performance optimization. It is the elimination of waste that was invisible because the bill arrived at the end of the month.</p><p>A financial sector case study published by Raymond Consulting Group cut ETL runtime from three hours to under one hour after switching from full to incremental loads. Some models saw three to five times speed improvements on the same hardware, without any infrastructure changes.</p><p><strong>Data Freshness</strong></p><p>Full refresh ties your data’s freshness to the batch cadence. A nightly job means your downstream consumers work with data that is up to twenty-four hours old. For most reporting use cases, that is acceptable. For anything touching inventory, pricing, fraud signals, or operational dashboards, it is not.</p><p>Log-based CDC delivers sub-ten-second latency from database commit to downstream availability. Shopify’s CDC platform, built on Debezium and Kafka, achieves p99 latency under ten seconds from MySQL insertion to event availability across more than a hundred shards.</p><p><strong>Delete Capture</strong></p><p>This is the failure mode I have seen cause the most confusion. A timestamp-based incremental job uses a query like WHERE updated_at &gt; last_run. A deleted row has no updated timestamp. It is simply gone. The pipeline never sees it. The destination table retains rows that no longer exist in the source, silently, indefinitely.</p><p>Log-based CDC captures hard deletes as explicit events with op = &#39;d&#39;. The downstream system knows a row was removed. It can act on that information. Full refresh cannot offer this at all.</p><p><strong>Schema Evolution</strong></p><p>A full refresh re-infers the schema every run, which means new columns appear automatically. That is the one genuine advantage. The cost is that you re-read everything to get it. Log-based CDC propagates additive changes through the event stream without re-reading historical data. Destructive changes, drops, renames, and type changes, require coordination. Neither approach handles schema drift for free, but CDC’s approach scales.</p><h3>How CDC Works Structurally</h3><p>Before any tools, the mechanism matters. I spent a long time treating CDC as a fancy name for “incremental loading.” It is structurally different, and understanding why changes how you approach the problem.</p><p>The diagram below shows exactly where the two approaches diverge.</p><figure><img alt="Two-column diagram. Left shows full refresh: source database, full table scan, TRUNCATE, bulk INSERT all rows, downstream consumers. Red labels mark read load on source and hours of inconsistency. Right shows log-based CDC: source database WAL, replication slot tracking offset, changed rows only to staging, MERGE INTO target, downstream consumers. Green labels mark under one percent CPU overhead and sub-ten-second latency." src="https://cdn-images-1.medium.com/max/1024/1*P1dOlGxovHKNwrQ-uihEcQ.png" /><figcaption>Full table read and truncate-reload cycle on the left versus WAL-based incremental capture on the right.</figcaption></figure><p>PostgreSQL writes every committed transaction to its Write-Ahead Log before applying changes to data files. This is the durability mechanism that allows crash recovery. When you set wal_level = logical, PostgreSQL records the additional row-level metadata needed to decode changes from the WAL, without adding load to the write path.</p><p>Debezium and dlt’s PostgreSQL replication source both read from this log via a replication slot. The slot tracks the last consumed position as an LSN, a Log Sequence Number, which is a byte offset into the WAL. The database knows exactly which changes have been consumed and retains the necessary WAL segments until the consumer confirms receipt.</p><p>Each change event carries an op field. The value is c for insert, u for update, d for delete, and r for a row read during the initial snapshot. Your destination receives not just the new state of the row but the operation type, the timestamp of the commit, and the position in the log. This is information that a full refresh cannot produce.</p><p><strong>The Three CDC Approaches And Their Trade-Offs</strong></p><p>Not all CDC implementations read the transaction log. The three approaches differ sharply in overhead, capability, and operational complexity.</p><p>Log-based CDC reads the WAL directly. It adds under one percent CPU overhead to the source database at twenty thousand writes per second, captures deletes, preserves exact event ordering, and delivers sub-second to single-digit-second latency. It requires database configuration: wal_level = logical, a replication slot, and appropriate permissions.</p><p>Trigger-based CDC installs database triggers that fire on every write and record changes to a shadow table. It is portable across database types and requires no special configuration. At high write volumes it adds twenty to thirty percent overhead to the write path, which is a real cost on a busy production database.</p><p>Query-based CDC polls a high-watermark column like updated_at. It is the simplest approach and requires no database changes. It cannot capture deletes, has ten to sixty-second polling latency, adds read load proportional to poll frequency, and silently misses rows where the timestamp was not updated.</p><p>The article you are reading is about log-based CDC. It is the standard above roughly ten thousand writes per second, and it is what Shopify, Zalando, and Netflix run in production.</p><h3>The Stack That Makes It Practical</h3><p>Two years ago, building a Python CDC pipeline meant wiring together Kafka, Debezium Kafka Connect, a schema registry, and a custom consumer. The infrastructure cost was real. I looked at that stack and decided my pipelines were small enough to live with full refresh for a while longer.</p><p>That calculation has changed.</p><p>The diagram below shows the two tiers available to you today.</p><figure><img alt="Two-row pipeline diagram. Top row Tier 1: PostgreSQL WAL to dlt pg_replication to DuckDB staging to MERGE INTO target. Labels: pure Python, no Java, no Kafka, Postgres only. Bottom row Tier 2: PostgreSQL WAL to Debezium Kafka Connect to Redpanda broker to multiple consumers including DuckDB and Iceberg. Labels: multi-consumer, durable replay, graduation path for ten or more downstream systems." src="https://cdn-images-1.medium.com/max/1024/1*DAhkKGWxWZn-vbr7dxyx3A.png" /><figcaption>dlt pg_replication into DuckDB for single-source pipelines, graduating to Debezium plus Redpanda for multi-consumer production scale.</figcaption></figure><p><strong>Tier 1: dlt Replication Into DuckDB</strong></p><p>For PostgreSQL sources, dlt ships a verified pg_replication source that reads the WAL directly using psycopg2 and PostgreSQL&#39;s built-in pgoutput plugin. No Java. No Kafka. No connector configuration files. Pure Python, runnable today with two commands.</p><p>This is the path I recommend for teams starting with CDC. It covers the majority of use cases, it is operationally simple, and it grows into Tier 2 without changing your downstream DuckDB MERGE logic.</p><p><strong>Tier 2: Debezium Plus Redpanda</strong></p><p>When you need more than one downstream consumer reading the same change stream, or when you need durable replay, or when your sources extend beyond PostgreSQL, the standard production stack is Debezium Kafka Connect plus a broker. Redpanda is Kafka API-compatible, deploys as a single binary without ZooKeeper or a JVM, and is the simplest path to a Kafka-compatible broker for teams that do not already run Kafka. The downstream consumer still applies changes with DuckDB MERGE INTO, so your pipeline logic stays identical.</p><p>The decision rule is concrete: if fewer than ten downstream systems need the same change stream, Tier 1 is simpler and sufficient. Above ten consumers, or when replay and durability become hard requirements, graduate to Tier 2.</p><h3>The Code</h3><p>Setup is three commands. Everything in this section covers the Tier 1 path: dlt pg_replication into DuckDB, pure Python, no Java, no Kafka. The Tier 2 architecture with Debezium and Redpanda is covered in the diagram above and serves as a reference for when you need multiple consumers or durable replay.</p><pre># Create project and add the stack<br>uv init cdc-pipeline<br>cd cdc-pipeline<br>uv add &quot;dlt[duckdb]&quot; duckdb psycopg2-binary<br>uv run pipeline.py</pre><p><strong>Step 1: Enable Logical Replication On PostgreSQL</strong></p><p>For local development, use Docker with the configuration pre-applied:</p><pre># docker-compose.yml<br>services:<br>  postgres:<br>    image: postgres:16<br>    environment:<br>      POSTGRES_USER: postgres<br>      POSTGRES_PASSWORD: postgres<br>      POSTGRES_DB: appdb<br>    ports:<br>      - &quot;5432:5432&quot;<br>    command: &gt;<br>      postgres<br>        -c wal_level=logical<br>        -c max_replication_slots=4<br>        -c max_wal_senders=4</pre><p>On a self-managed database, run these as a superuser and restart:</p><pre>ALTER SYSTEM SET wal_level = &#39;logical&#39;;<br>ALTER SYSTEM SET max_replication_slots = 4;<br>ALTER SYSTEM SET max_wal_senders = 4;<br>-- Then restart PostgreSQL</pre><p><strong>Step 2: Configure dlt Replication</strong></p><pre># pipeline.py<br>import dlt<br>from pg_replication import replication_resource<br>from pg_replication.helpers import init_replication<br><br>CREDENTIALS = &quot;postgresql://postgres:postgres@localhost:5432/appdb&quot;<br>SLOT_NAME   = &quot;cdc_pipeline_slot&quot;<br>PUB_NAME    = &quot;cdc_pipeline_pub&quot;<br><br>pipeline = dlt.pipeline(<br>    pipeline_name=&quot;cdc_pipeline&quot;,<br>    destination=&quot;duckdb&quot;,<br>    dataset_name=&quot;staging&quot;,<br>)<br><br># Initial snapshot: captures all rows present before replication begins<br>snapshot = init_replication(<br>    slot_name=SLOT_NAME,<br>    pub_name=PUB_NAME,<br>    schema_name=&quot;public&quot;,<br>    table_names=[&quot;orders&quot;, &quot;customers&quot;, &quot;products&quot;],<br>    credentials=CREDENTIALS,<br>    persist_snapshots=True,<br>    reset=True,<br>)<br>load_info = pipeline.run(snapshot)<br>print(load_info)<br><br># Ongoing changes: streams inserts, updates, and deletes from the WAL<br># dlt tracks the last consumed LSN in the replication slot automatically<br>changes = replication_resource(SLOT_NAME, PUB_NAME, credentials=CREDENTIALS)<br>load_info = pipeline.run(changes)<br>print(load_info)</pre><p>Each loaded table in DuckDB staging carries the _dlt_load_id, _dlt_id, and a deleted_ts column. Rows with a non-null deleted_ts are deletes. Rows with a null deleted_ts are inserts or updates.</p><p><strong>Step 3: Apply CDC Events With DuckDB MERGE INTO</strong></p><p>This is the centerpiece. MERGE INTO was introduced in DuckDB v1.4.0 LTS in September 2025 and extended to Iceberg tables in v1.5.3 in May 2026. If you read the first article in this series, you used a version of this pattern in the Iceberg REST catalog pipeline. The same statement that applied upserts against a lakehouse table there now applies CDC semantics here: upsert the row if it exists or is new, delete it if the source deleted it.</p><pre>import duckdb<br><br>con = duckdb.connect(&quot;analytics.duckdb&quot;)<br><br># Attach the pipeline&#39;s DuckDB file to read from its raw staging layer<br>con.execute(&quot;ATTACH &#39;cdc_pipeline.duckdb&#39; AS cdc (READ_ONLY)&quot;)<br><br># Create target table if it does not exist<br>con.execute(&quot;&quot;&quot;<br>    CREATE TABLE IF NOT EXISTS orders (<br>        order_id    INTEGER PRIMARY KEY,<br>        customer_id INTEGER,<br>        status      VARCHAR,<br>        amount      DECIMAL(10,2),<br>        updated_at  TIMESTAMP<br>    )<br>&quot;&quot;&quot;)<br><br># Apply CDC events: upsert updated or inserted rows, remove deleted rows<br># deleted_ts is non-null for deletes in dlt&#39;s staging layer<br>con.execute(&quot;&quot;&quot;<br>    MERGE INTO orders AS target<br>    USING (<br>        SELECT *<br>        FROM cdc.staging_staging.orders<br>        WHERE _dlt_load_id = (<br>            SELECT MAX(_dlt_load_id) FROM cdc.staging_staging.orders<br>        )<br>    ) AS source<br>    ON target.order_id = source.order_id<br>    WHEN MATCHED AND source.deleted_ts IS NOT NULL THEN<br>        DELETE<br>    WHEN MATCHED AND source.deleted_ts IS NULL THEN<br>        UPDATE SET<br>            customer_id = source.customer_id,<br>            status      = source.status,<br>            amount      = source.amount,<br>            updated_at  = source.updated_at<br>    WHEN NOT MATCHED AND source.deleted_ts IS NULL THEN<br>        INSERT (order_id, customer_id, status, amount, updated_at)<br>        VALUES (source.order_id, source.customer_id, source.status,<br>                source.amount, source.updated_at)<br>&quot;&quot;&quot;)<br><br>print(&quot;CDC events applied.&quot;)<br>con.close()</pre><p>What just happened: you read only the rows that changed since the last run, applied inserts, updates, and deletes in a single statement, and wrote the result to a DuckDB table that is immediately queryable. No truncate. No full reload. No maintenance window.</p><h3>Where Full Refresh Still Wins</h3><p>CDC is not free. I want to be precise about that, because the pattern I have described above has real failure modes that a full refresh does not.</p><p>A replication slot that stalls can cause uncontrolled WAL growth on the source database. PostgreSQL retains WAL segments until the slot confirms consumption, so a stuck consumer means the database disk fills up. Zalando encountered this with low-activity databases and had to contribute upstream fixes to Debezium. You need to monitor pg_replication_slots and alert on confirmed_flush_lsn versus restart_lsn drift.</p><p>Schema evolution requires coordination. Adding a nullable column is safe. Renaming a column is not. PostgreSQL’s logical decoding does not propagate DDL as a separate event, so a rename appears downstream as a drop plus an add. On a full refresh pipeline, the same rename is handled automatically on the next run. On a CDC pipeline, it is a coordinated migration.</p><p>With those trade-offs stated clearly, full refresh remains the right choice in four situations. Small reference tables under a few million rows where the reload cost is trivial. Append-only sources like event logs where you only need to add rows, not update or delete them. Sources that do not expose a transaction log, many SaaS APIs and some managed databases fall into this category. And teams without the operational capacity to monitor replication slots and coordinate schema migrations.</p><p>Even mature CDC deployments run periodic full refreshes as reconciliation safety nets. The two approaches are not mutually exclusive.</p><h3>The Engineer You Are Becoming</h3><p>Three articles into this series, the picture is complete.</p><p>DuckDB gave you a compute layer that replaces Spark for any pipeline under a hundred gigabytes, without a cluster, without a JVM, and with full Iceberg write support so your output tables are readable by every engine in your stack. Polars gave you a transformation layer that replaces pandas for file-based ETL above one gigabyte, using lazy evaluation, multi-core execution, and Arrow memory to finish in seconds what used to take minutes. CDC gives you an ingestion layer that replaces nightly full refreshes with a continuous, low-overhead stream of exactly what changed, captured in order, with deletes included.</p><p>This week, take the oldest and most expensive full refresh job in your stack and ask four questions. How big is the table it reads? How much of the data changes between runs? Does the downstream system ever need to know about deletes? Could you tolerate sub-ten-second latency instead of overnight?</p><p>If the answers are large, small fraction, yes, and yes, you have your migration target.</p><p>The data engineer who asks these questions before defaulting to the nightly reload is doing something more than optimizing a pipeline. They are taking ownership of the hidden costs in their stack, the compute waste, the staleness, the silent delete blindspot, and replacing them with a structural solution that scales without changing.</p><p>I should have asked these questions earlier than I did. The infrastructure was simpler than I thought, and the gains were larger than I expected.</p><p>I write weekly about data engineering: real benchmarks, working code, and honest takes on the tools worth your time. Follow me here if that sounds useful.</p><p><em>Sources: Shopify Engineering, “Change Data Capture at Shopify,” 2021. Zalando Engineering, “Fabric Event Streams,” December 2025. Gunnar Morling, “Five Advantages of Log-Based Change Data Capture,” Debezium blog, 2018. Airbyte, “Incremental Load vs Full Load ETL,” 2024. dlt documentation, “PostgreSQL replication source,” 2025. DuckDB v1.4.0 release post, duckdb.org, September 2025. DuckDB v1.5.3 Iceberg release post, duckdb.org, May 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3175e29d9697" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/cdc-explained-why-full-refresh-etl-is-dying-3175e29d9697">CDC Explained: Why Full Refresh ETL Is Dying</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Polars vs Pandas in Production Pipelines]]></title>
            <link>https://blog.devgenius.io/polars-vs-pandas-in-production-pipelines-20b468a05eb7?source=rss-10235c107f8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/20b468a05eb7</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[polar]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Nevenka Lukic]]></dc:creator>
            <pubDate>Sun, 14 Jun 2026 14:38:51 GMT</pubDate>
            <atom:updated>2026-06-18T11:05:10.840Z</atom:updated>
            <content:encoded><![CDATA[<h4>The tool that made Python data engineering accessible is quietly becoming the thing that slows it down.</h4><p>This article was originally published on <a href="https://pythondataengineering.net/projects/polars-vs-pandas-production-pipelines">Python Data Engineering.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hjGjinhy6i-UKDtv_jhaog.png" /></figure><h3>The Default That Is Slowing You Down</h3><p>Most Python data engineers learned pandas first. I was one of them as well. It was the standard, it worked, and it kept working well enough that nobody questioned it. That is exactly the problem.</p><p>Pandas was designed in 2008 for a different kind of data problem. It assumes you want immediate results from every operation, that a single CPU core is enough, and that data fits comfortably in RAM. Those assumptions held for years. They hold less and less as pipelines grow.</p><p>The question is not whether pandas is good. I know it is. I used it for years and it served me well. The question is if it still matches the shape of the workloads you are actually running.</p><p>For file-based ETL above roughly 1 GB, the answer is increasingly no. Not because pandas is broken, but because the ecosystem has produced something structurally better for that specific job. That tool is Polars. And the gap between the two is not marginal.</p><h3>Why Polars Is Structurally Faster</h3><p>Before any benchmark numbers, the mechanism matters. I spent a long time treating Polars as “pandas but faster” before I understood why it is actually different. Once I did, the decision about when to use it became obvious. Understanding why it is faster tells you exactly when the advantage applies and when it does not.</p><p>The diagram below shows the two execution models side by side.</p><figure><img alt="Two-column architecture diagram. Left column shows pandas execution: Python GIL blocks parallel threads, NumPy arrays stored row-adjacent in memory, every operation executes immediately and produces an intermediate copy, single core active during computation. Right column shows Polars execution: Rust core with no GIL, Apache Arrow columnar memory layout, lazy plan built and optimized before any data is read, all available CPU cores active simultaneously via morsel-driven parallelism." src="https://cdn-images-1.medium.com/max/1024/1*M8EzODk8glAk6O3kWhgEHQ.png" /><figcaption>Single-threaded eager execution in pandas versus multi-threaded lazy execution in Polars.</figcaption></figure><p><strong>The Pandas Execution Model</strong></p><p>Pandas runs on NumPy, which is single-threaded for most operations. Python’s Global Interpreter Lock prevents parallel execution even when multiple threads are available. Every operation runs immediately, which means intermediate results are created and stored in memory whether you need them or not. A chain of five transformations on a 5 GB DataFrame creates five intermediate copies. Memory usage compounds fast.</p><p>String columns make this worse. By default, pandas stores strings as NumPy object arrays: each value is a Python object pointer, consuming roughly 67 bytes per unique string versus 4 bytes of overhead in columnar storage. I first noticed this when a pipeline that should have used 3 GB of RAM was consistently consuming 9 GB. The culprit was a single string column on a wide DataFrame. A column of 1 million 18-character strings uses around 75 MB in pandas object dtype. The same column in Arrow format uses around 22 MB.</p><p><strong>The Polars Execution Model</strong></p><p>Polars is written in Rust and runs outside the GIL entirely. It stores data in Apache Arrow columnar format: each column is a contiguous block of typed memory, cache-friendly and SIMD-ready. When you call scan_parquet, no data is read yet. Polars builds a logical execution plan and optimizes it before touching a single byte: predicates are pushed down to the scan level, unused columns are pruned before reading, and joins are reordered for efficiency.</p><p>Execution is morsel-driven. Each CPU thread takes a chunk of input, processes it independently with its own local state, and merges results at the end. There is no lock contention during computation. All cores run simultaneously.</p><p>The practical consequence is something I measured myself, and the difference is not subtle. A pipeline that touches a 10 GB Parquet file in pandas reads all 10 GB, processes it sequentially on one core, and creates intermediate copies along the way. The same pipeline in Polars reads only the columns and row groups it needs, processes them in parallel across all cores, and never materializes an intermediate result it does not have to.</p><h3>The Benchmark Evidence</h3><p>Three named tests establish the range of the gap.</p><p>The table below summarizes the results across different scales and workload types.</p><figure><img alt="Table with three rows and five columns. Headers are Benchmark, Workload, Polars result, Pandas result, and Source. Row one: PDS-H scale factor 10, all 22 TPC-H queries on 10 GB CSV, Polars streaming 3.89 seconds, pandas 365.71 seconds, 94x gap, Polars official benchmark May 2025, AWS c7a.24xlarge." src="https://cdn-images-1.medium.com/max/1024/1*IJnrjHlrWaybTbrPbTecqw.png" /><figcaption>Three independent benchmarks comparing Polars and pandas across query pipelines, production ETL migration, and a scheduling workload.</figcaption></figure><p><strong>The PDS-H Benchmark</strong></p><p>The Polars team published updated PDS-H results in May 2025, run on an AWS c7a.24xlarge instance with 96 vCPUs and 192 GB of RAM. The benchmark runs all 22 TPC-H-derived queries against a 10 GB CSV dataset. Polars streaming finished in 3.89 seconds. Pandas 2.2.3 with PyArrow dtypes enabled finished in 365.71 seconds. That is a 94x gap on a full multi-step analytical pipeline.</p><p>At scale factor 100 (100 GB), pandas was excluded entirely. Its single-threaded execution and lack of a query optimizer caused out-of-memory failures before completing the benchmark. Polars streaming finished in 23.94 seconds.</p><p>This is a vendor-run benchmark and I want to be clear about that. Treat it as directional, not as a universal promise. Independent tests on modest hardware consistently show smaller gaps, typically 5 to 22x on individual operations. The headline multiple emerges when the query optimizer and multi-threading compound across a full pipeline.</p><p><strong>The Production Evidence</strong></p><p>Check Technologies, a Dutch mobility provider, migrated all 100 plus Airflow DAGs from pandas to Polars over a single sprint. Their motivation was not performance benchmarks. It was repeated out-of-memory errors on their most data-intensive pipelines. The migration took under two weeks and delivered a 3.3x speed improvement on the most problematic DAG, roughly 2x gains across almost all others, and a 25 percent reduction in cloud infrastructure cost. Paul Duvenage, Senior Data Engineer at Check, described the migration as straightforward once the team shifted to the declarative expression API.</p><p>DB Systel, a Deutsche Bahn subsidiary, rewrote a train-scheduling reprocessing job in Polars 0.20. The job previously ran in 96 minutes. The Polars version ran in 5.5 minutes. That is a 17.5x improvement on a real production workload, not a synthetic benchmark.</p><p>The counterintuitive finding: on small data, the gap largely disappears and sometimes reverses. Polars’ query optimizer adds planning overhead that exceeds the compute savings when the dataset is under a few hundred megabytes. For quick scripts on small frames, pandas is faster to write and fast enough to run.</p><h3>Setting Up And Running Both</h3><p>The setup is one command.</p><pre># Create project and add Polars<br>uv init polars-pipeline<br>cd polars-pipeline<br>uv add polars pyarrow      # standard install<br>uv run pipeline.py</pre><p>No compilation. No system dependencies. Polars ships as a pre-built wheel with the Rust runtime included.</p><p>⚠️ One thing worth knowing before you start: on Apple Silicon (M-series Macs), the standard polars wheel triggers a CPU compatibility warning and recommends the runtime-compatible variant.</p><pre># Apple Silicon: use this instead<br>uv add &quot;polars[rtcompat]&quot; pyarrow</pre><p>On Linux and Intel Macs the standard install is fine. On M-series hardware, polars[rtcompat] avoids the warning and ensures the correct SIMD instructions are used for your processor.</p><p><strong>Side-By-Side: The Same Operation In Both Libraries</strong></p><p>When I first looked at Polars code, the syntax felt unfamiliar. The expression API takes a day to get used to. After that day, I found it more readable than the pandas equivalent, not less. The code below runs an identical ETL transformation: read a Parquet file, filter rows, aggregate by category, and sort the result. First in pandas, then in Polars LazyFrame.</p><pre># pandas version<br>import pandas as pd<br>import time<br><br>start = time.perf_counter()<br><br>df = pd.read_parquet(&quot;sales.parquet&quot;)<br>result = (<br>    df[df[&quot;revenue&quot;] &gt; 1000]<br>    .groupby(&quot;category&quot;)<br>    .agg(<br>        total_revenue=(&quot;revenue&quot;, &quot;sum&quot;),<br>        avg_price=(&quot;price&quot;, &quot;mean&quot;),<br>        order_count=(&quot;order_id&quot;, &quot;count&quot;),<br>    )<br>    .sort_values(&quot;total_revenue&quot;, ascending=False)<br>    .reset_index()<br>)<br><br>print(f&quot;pandas: {time.perf_counter() - start:.3f}s&quot;)<br>print(result)</pre><pre># Polars version — lazy execution with predicate and projection pushdown<br>import polars as pl<br>import time<br><br>start = time.perf_counter()<br><br>result = (<br>    pl.scan_parquet(&quot;sales.parquet&quot;)          # no data read yet<br>    .filter(pl.col(&quot;revenue&quot;) &gt; 1000)         # pushed into the scan<br>    .group_by(&quot;category&quot;)<br>    .agg(<br>        total_revenue=pl.col(&quot;revenue&quot;).sum(),<br>        avg_price=pl.col(&quot;price&quot;).mean(),<br>        order_count=pl.col(&quot;order_id&quot;).count(),<br>    )<br>    .sort(&quot;total_revenue&quot;, descending=True)<br>    .collect()                                 # execution happens here<br>)<br><br>print(f&quot;Polars: {time.perf_counter() - start:.3f}s&quot;)<br>print(result)</pre><p>The structural difference is in scan_parquet versus read_parquet. The pandas version reads the entire file immediately. The Polars version builds a plan and reads only the rows and columns required to satisfy the filter and aggregation. On a 1 GB file with 20 columns where you need 3, Polars may read less than 20 percent of the bytes pandas reads. That gap is what I feel every time I time a pipeline: pandas is already reading when Polars is still planning, but Polars finishes first.</p><p><strong>Streaming For Larger-Than-Memory Data</strong></p><p>When your dataset exceeds available RAM, add engine=”streaming” to collect. Polars processes the data in batches called morsels, spilling to disk adaptively and never holding the full dataset in memory at once.</p><pre>result = (<br>    pl.scan_parquet(&quot;large_dataset/*.parquet&quot;)<br>    .filter(pl.col(&quot;status&quot;) == &quot;active&quot;)<br>    .group_by(&quot;region&quot;)<br>    .agg(pl.col(&quot;amount&quot;).sum())<br>    .sort(&quot;amount&quot;, descending=True)<br>    .collect(engine=&quot;streaming&quot;)              # out-of-core execution<br>)</pre><p>For writing directly to disk without materializing in memory at all, use sink_parquet:</p><pre>(<br>    pl.scan_parquet(&quot;raw/*.parquet&quot;)<br>    .filter(pl.col(&quot;error_code&quot;).is_null())<br>    .sink_parquet(&quot;clean/output.parquet&quot;)     # streams to disk in batches<br>)</pre><p><strong>Production Guardrails</strong></p><p>Set thread count explicitly in production environments to avoid contention with other processes:</p><pre>import os<br>os.environ[&quot;POLARS_MAX_THREADS&quot;] = &quot;8&quot;       # MUST be set before importing polars<br>import polars as pl                           # Polars initializes its thread pool at<br>                                              # import time — setting this after import<br>                                              # has no effect</pre><p>Pin your Polars version in pyproject.toml. The API is stable at 1.x but minor versions introduce new features that change behavior on edge cases. Lock it, test in a container matching your production OS, and upgrade deliberately.</p><h3>Where Pandas Still Wins</h3><p>Polars is the better tool for file-based ETL on data above 1 GB. I still use pandas every week. There are three situations where I reach for it without hesitation.</p><p>The framework below maps the decision by data size and workload type.</p><figure><img alt="Three-column card layout. Left card labeled Polars in blue: data above 1 GB, file-based ETL, group-by and join heavy pipelines, larger-than-memory workloads, production data pipelines where speed and cost matter. Verdict: scan then collect. Middle card labeled Evaluate Both in amber: data from 500 MB to 2 GB, mixed workloads, teams with existing pandas code, pipelines that feed both ML models and dashboards. Verdict: Polars for compute, pandas for the final step." src="https://cdn-images-1.medium.com/max/1024/1*UPKJZ3yG5f87WGfTFfEizg.png" /><figcaption>Tool selection by data size and workload type, with the ML ecosystem constraint mapped explicitly.</figcaption></figure><p><strong>The ML Ecosystem Constraint</strong></p><p>Most ML libraries accept pandas DataFrames as their native input format. scikit-learn, statsmodels, and many plotting libraries expect pandas either explicitly or by default. Rewriting your entire stack to avoid a single .to_pandas() call is not a trade-off worth making.</p><p>The practical pattern I use in every pipeline that feeds a model is the hybrid approach: Polars for the expensive compute work, then convert at the final step.</p><pre># Polars for the heavy lifting<br>features = (<br>    pl.scan_parquet(&quot;events/*.parquet&quot;)<br>    .group_by(&quot;user_id&quot;)<br>    .agg([<br>        pl.col(&quot;session_duration&quot;).mean().alias(&quot;avg_session&quot;),<br>        pl.col(&quot;purchase&quot;).sum().alias(&quot;total_purchases&quot;),<br>        pl.col(&quot;event_date&quot;).max().alias(&quot;last_seen&quot;),<br>    ])<br>    .collect(engine=&quot;streaming&quot;)<br>)<br><br># Zero-copy conversion at the boundary<br>import sklearn<br>X = features.to_pandas()                     # Arrow-backed, near-instant</pre><p>The conversion is near-instant because both Polars and recent pandas share the Apache Arrow memory format. No data is copied when both sides use Arrow types.</p><p><strong>What Pandas 3.0 Changed</strong></p><p>Pandas 3.0, released January 2026, made PyArrow-backed strings the default for string columns and made Copy-on-Write the only execution mode. These changes reduce memory usage on string-heavy datasets by up to 70 percent and eliminate a class of silent mutation bugs.</p><p>These are real improvements and I want to give them credit. They close the I/O and memory gap meaningfully. But they do not close the multi-core execution gap. Pandas aggregations remain single-threaded and eager with no query optimizer in any version. If your bottleneck is compute rather than memory, the 3.0 update does not change the calculus.</p><h3>The Decision You Need To Make This Week</h3><p>Three signals mean a pipeline is ready to migrate. I have hit all three of them at different points, and each time I waited longer than I should have before switching.</p><p>First: it runs out of memory on production-sized data, or you have added chunking logic to work around memory limits. Second: it takes minutes where it should take seconds, and profiling shows that pandas groupby or join operations dominate the runtime. Third: it runs on a machine with multiple CPU cores that sit idle during execution.</p><p>Any one of those signals is enough. All three together mean the migration pays back in the first sprint.</p><p>The starting point is narrow and concrete. Pick one pipeline, the slowest or the one that OOMs most often. Rewrite only the compute-heavy portion in Polars LazyFrame, keep the pandas boundary at the output for anything feeding into ML or plotting, and benchmark on production-sized data before promoting. If you do not see at least a 3x improvement on a dataset above 1 GB, the bottleneck is probably not the DataFrame library.</p><p>The broader shift is in how you think about tool selection. Pandas is not legacy. It is a precise instrument for specific work. Polars is a precise instrument for different work. Using the right one for each job is not a migration project. It is just engineering discipline applied to your own stack.</p><p>I learned this later than I should have. The data engineer who reaches for the tool that fits the workload, rather than the tool they learned first, ships faster, spends less on compute, and builds pipelines that do not need rewriting when the data gets bigger.</p><p><em>I write weekly about </em><strong><em>data engineering</em></strong><em>: real benchmarks, working code, and honest takes on the tools worth your time. </em><strong><em>Follow me here</em></strong><em> if that sounds useful.</em></p><p><em>Sources: Polars team, “Updated PDS-H Benchmark Results,” pola.rs, May 2025. Paul Duvenage, Check Technologies engineering blog, 2024. DB Systel engineering blog, 2024. Nahrstedt et al., “An Empirical Study on the Energy Usage and Performance of Pandas and Polars,” EASE 2024. Polars 1.0 announcement, pola.rs, July 2024. Pandas 3.0 release notes, pandas.pydata.org, January 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20b468a05eb7" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/polars-vs-pandas-in-production-pipelines-20b468a05eb7">Polars vs Pandas in Production Pipelines</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DuckDB: The Death Of Small-Scale Spark]]></title>
            <link>https://medium.com/towards-data-engineering/duckdb-the-death-of-small-scale-spark-e86ac6aedf23?source=rss-10235c107f8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/e86ac6aedf23</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[duckdb]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Nevenka Lukic]]></dc:creator>
            <pubDate>Wed, 10 Jun 2026 09:39:43 GMT</pubDate>
            <atom:updated>2026-06-10T09:39:43.626Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>While everyone debates the future of distributed compute, a single binary is quietly replacing Spark clusters for most of the data pipelines in most of the stacks you will actually encounter.</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X2ADZ9IcNbCOh0pSUDCIEA.png" /></figure><p>There is a Spark job running somewhere right now on a managed cluster. It reads 8 GB of Parquet, runs three aggregations, and writes a result table. The whole thing takes four minutes. Two of those minutes are executor negotiation, JVM warmup, and shuffle staging. The actual computation takes under thirty seconds.</p><p>Nobody questions it. The cluster was already there. Spark was the standard. So the pipeline stays.</p><p>This is a story about that decision, and the cost of never revisiting it.</p><h3>The Reflex That Is Costing You Time</h3><p>Data engineers learn Spark early and apply it everywhere. The mental model locks in fast: large data requires distributed compute, and distributed compute means Spark. That logic was sound for a long time. It is eroding now, and the erosion is measurable.</p><p>The question worth asking before any new pipeline is not “how do I run this on Spark?” The right question is “does this job actually need a cluster?”</p><p>For most pipelines processing under 50 GB on a single developer machine, the honest answer is no. The cluster adds JVM overhead, executor spin-up latency, shuffle staging cost, and configuration complexity to a problem that a single binary could finish in seconds. The engineer pays the distributed tax without receiving the distributed benefit.</p><p>The threshold is not a fixed law. A 16 GB laptop hits memory pressure around 20 GB of data. A 128 GB VM with NVMe scratch space handles well past 100 GB. The number depends on your machine. The principle does not.</p><p><strong><em>This article is about two things.</em></strong> <strong>First</strong>, the concrete evidence that DuckDB outperforms Spark on small and medium data by a significant margin. <strong>Second</strong>, the practical setup to run your own pipelines, including a working ETL from Parquet through to an Iceberg lakehouse, using a single Python process and no cluster.</p><p>It is also honest about where Spark wins. Because it does.</p><h3>Why DuckDB Is Structurally Faster On One Machine</h3><p>Before the benchmarks, the mental model matters. Speed claims without mechanism are just marketing.</p><p>DuckDB is not a server. It is not a framework. It is a library. It runs inside your Python process, your R session, or your command line. There is no TCP connection between your code and the query engine, no result serialization, no driver protocol overhead. The engine lives in the same memory space as your script, which eliminates an entire category of latency that distributed systems carry as a permanent structural cost.</p><p>The diagram below shows exactly where the cost difference lives.</p><figure><img alt="Side-by-side architecture diagram. Left: DuckDB runs as a library inside the Python process with no network boundary, passing through a vectorized engine, buffer manager, and directly to Parquet or Iceberg files. Right: Spark routes through a driver JVM, cluster manager, multiple executors, and a shuffle layer. Red annotations mark 185,639 context switches and 16,303 CPU migrations from Digital Turbine’s March 2025 perf stat measurement." src="https://cdn-images-1.medium.com/max/1024/1*nM41Y_bW8LHYO9ZkoWDgzQ.png" /><figcaption>In-process versus cluster execution model across the full stack from query entry point to storage.</figcaption></figure><p>On the DuckDB side, there are no moving parts to start before computation begins. On the Spark side, there are several.</p><h4><strong>The Engine Itself</strong></h4><p>DuckDB processes data in columnar batches called vectors, roughly 1,024 rows at a time. Each vector is sized to maximize CPU cache utilization, which lets the processor use SIMD instructions to operate on entire columns at once instead of row by row. Vectorized execution dramatically improves cache locality and reduces memory-access overhead compared with row-at-a-time models.</p><p>Thread management follows a morsel-driven model. Each thread takes its own segment of input and maintains its own local hash table or sorted run, with no lock contention during the compute phase. The combine step runs across all cores after each thread finishes. The result is full CPU utilization with near-zero coordination overhead.</p><p>When data exceeds available RAM, DuckDB’s buffer manager spills to disk adaptively. Grouping, sorting, joining, and windowing all support out-of-core execution. Hannes Mühleisen demonstrated this live at PyData Amsterdam 2025: a 265 GB dataset with 6 billion rows, running COUNT DISTINCT, completed in 46 seconds using only 2 GB of memory.</p><p>No cluster required. No executors to negotiate. Startup overhead is typically negligible compared with Spark, where executor negotiation alone adds tens of seconds to minutes on managed clusters.</p><p>The key insight is this: DuckDB’s in-process design does not just remove infrastructure complexity. It removes entire categories of latency that Spark treats as the cost of doing business at scale. For workloads that fit one machine, that cost is pure waste.</p><h3>The Benchmark Evidence</h3><p>Claims without data are opinion. Here are the numbers, with named sources and methodology.</p><p>The table below summarizes four independent tests across a range of scales and workload types.</p><figure><img alt="Benchmark table with four rows. Row 1: DataExpert.io, up to 23 GB on a 16 GB laptop, COUNT DISTINCT, DuckDB up to 100x faster, DuckDB wins. Row 2: Digital Turbine, 40 GiB group-by, DuckDB 3.99s and 196 MB versus Spark 16.39s and 1.65 GB, DuckDB wins. Row 3: DataExpert.io, 1 TB on MotherDuck Mega, under 17 seconds, DuckDB wins. Row 4: Miles Cole LakeBench, 127 GB ELT on Fabric, Spark 3.5x faster at 32 cores, Spark wins." src="https://cdn-images-1.medium.com/max/1024/1*x2hSbgKbuhGK_ke-Akzg1w.png" /><figcaption>Wall-clock time, peak memory, and CPU behavior across four independent tests ranging from 23 GB to 1 TB.</figcaption></figure><h4><strong>The Laptop Test: Up To 100x On Sub-20 GB</strong></h4><p>Matt Martin and Zach Wilson at DataExpert.io benchmarked DuckDB against vanilla Spark on a 16 GB RAM laptop across seven synthetic Parquet datasets up to 23 GB. The workload was COUNT DISTINCT on a random string column, designed to force full file scans with no optimization shortcuts. DuckDB was faster in every single run, including the 23 GB dataset that exceeded the machine’s RAM. Their published conclusion: on the tested COUNT DISTINCT workloads, DuckDB was up to roughly 100 times faster than Apache Spark on sub-20 GB datasets.</p><p>Matt’s working rule from that benchmark: default to DuckDB, switch to Spark when data exceeds approximately 20 GB on a 16 GB machine. Not a hard data limit. A memory pressure threshold.</p><h4><strong>The Production Pipeline: Measured With perf stat</strong></h4><p>Digital Turbine’s engineering team ran a more rigorous test in March 2025, using Linux perf stat to capture CPU behavior, not just wall-clock time. The workload was a group-by aggregation on 40 GiB of Parquet across 52 files and 94 columns, on a 16-core, 64 GB cloud VM.</p><p>DuckDB finished in 3.99 seconds. Spark took 16.39 seconds. The memory and CPU numbers tell a sharper story than the 2.5x speed gap alone. DuckDB consumed 196 MB of peak RSS. Spark consumed 1.65 GB. DuckDB triggered 6,726 context switches and 130 CPU migrations. Spark triggered 185,639 context switches and 16,303 CPU migrations.</p><p>Those numbers are JVM warmup, garbage collection, and a distribution-oriented execution plan imposing real cost on a job that never needed distribution. Digital Turbine replaced this pipeline with a DuckDB Kubernetes CronJob. The Spark version was a 100-line Scala project with an sbt build. The DuckDB replacement was six lines of Python and a pip install.</p><h4><strong>The Honest Ceiling: Where Spark Wins Back</strong></h4><p>Miles Cole’s LakeBench study on Microsoft Fabric, published in June 2025 and fully open-sourced, is the most rigorous independent test and the most important for calibrating expectations.</p><p>At 140 MB and 1.2 GB compressed, single-node engines beat Spark cleanly. At 12.7 GB, Spark with a Native Execution Engine was fastest, with DuckDB competitive. At 127 GB compressed, Spark ran 3.5 to 6 times faster than DuckDB depending on core count. DuckDB ran out of memory at 16 cores on that test.</p><p>The cost dimension from endjin’s parallel Fabric study adds context: the cheapest Spark run costs more than five times the cheapest DuckDB run, even at scales where Spark wins on wall-clock time. Speed and cost are separate questions, and they sometimes have different answers.</p><p>The evidence points to a clear threshold. Below roughly 50 GB on a standard developer machine, DuckDB wins on speed, cost, and operational simplicity. Above 100 GB compressed, Spark with a native vectorized engine pulls ahead on speed, though at significantly higher infrastructure cost. Between those two points, you should benchmark on your actual data.</p><h3>Setting Up Your Environment In VS Code</h3><p>The setup takes under five minutes. That fact is itself an argument.</p><p><strong>Install With uv</strong></p><p>uv is the current default Python package manager for engineers who care about speed and reproducibility. Install it once, then initialize a project:</p><pre># Install uv if you do not have it<br>curl -LsSf https://astral.sh/uv/install.sh | sh<br><br># Create the project<br>uv init duckdb-pipeline<br>cd duckdb-pipeline<br><br># Add dependencies<br>uv add duckdb pandas pyarrow<br><br># Run any script with the managed environment<br>uv run any_script.py</pre><p>No JVM. No cluster configuration. No executor memory tuning. DuckDB is approximately 60 MB with zero external dependencies. uv resolves and locks the environment in milliseconds, and uv run guarantees your script always uses the correct versions without manual activation.</p><h4><strong>VS Code Extension</strong></h4><p>Install the DuckDB extension by ChuckJonas from the VS Code marketplace. It gives you a live query panel, a schema browser, direct file preview for Parquet and CSV, and server-side pagination for large result sets. You can open any <strong><em>.parquet</em></strong> file and run SQL against it immediately, without writing a script first. This is how exploratory work should feel.</p><h4><strong>Project Structure</strong></h4><p>A clean DuckDB project sits like this:</p><pre>duckdb-pipeline/<br>├── .venv/                          # Managed by uv<br>├── data/<br>│   ├── raw/<br>│   │   ├── sales.parquet           # Generated by setup_a/generate_data.py<br>│   │   └── sales_incremental.parquet  # Generated by setup_b/generate_incremental.py<br>│   └── processed/<br>│       └── sales_summary.parquet   # Output of setup_a/pipeline_local.py<br>├── setup_a/                        # Local Parquet pipeline (no infrastructure)<br>│   ├── generate_data.py            # Generates 2M synthetic sales rows<br>│   ├── pipeline_local.py           # Transform, aggregate, write output<br>│   └── inspect_plan.py             # EXPLAIN ANALYZE query plan<br>├── setup_b/                        # Iceberg REST catalog pipeline (requires Docker)<br>│   ├── generate_incremental.py     # Generates 200k incremental rows<br>│   ├── pipeline_iceberg.py         # Initial load into Iceberg table<br>│   ├── merge_iceberg.py            # MERGE INTO upsert<br>│   └── timetravel_iceberg.py       # Snapshot inspection and time travel<br>├── sql/                            # Reusable SQL transform files<br>├── pyproject.toml                  # uv-managed dependencies<br>└── uv.lock                         # Locked dependency graph</pre><h4><strong>Production Guardrails</strong></h4><p>Set explicit resource limits before touching any large dataset. DuckDB’s defaults are permissive by design, and an unguarded aggregation on a large join can exhaust memory without warning.</p><pre>import duckdb<br><br>con = duckdb.connect(&quot;analytics.duckdb&quot;)<br><br>con.execute(&quot;SET memory_limit = &#39;12GB&#39;&quot;)<br>con.execute(&quot;SET temp_directory = &#39;/tmp/duckdb_spill&#39;&quot;)<br>con.execute(&quot;SET max_temp_directory_size = &#39;50GB&#39;&quot;)<br>con.execute(&quot;SET threads = 8&quot;)           # or: SET threads = system_threads<br>con.execute(&quot;SET preserve_insertion_order = false&quot;)</pre><p>Set <strong><em>memory_limit</em></strong> to roughly 75 percent of available RAM. Set <strong><em>temp_directory</em></strong> to your fastest disk. The <strong><em>threads</em></strong> value of 8 is a conservative default for shared machines. On a dedicated box, use <strong><em>SET threads = system_threads</em></strong> to let DuckDB claim every core automatically. These guardrails are the difference between a graceful spill and an out-of-memory failure mid-job.</p><h3>A Real ETL Pipeline: From Parquet To Iceberg</h3><p>This section contains two complete, runnable setups. The first is fully local, no cloud credentials required. The second connects DuckDB to an Apache Iceberg REST catalog, runs a MERGE INTO upsert, and uses time travel to inspect the snapshot history.</p><p>Both setups use identical DuckDB patterns. The difference is where the data lives and how DuckDB commits its writes.</p><h3>Setup A: Local Parquet Pipeline</h3><h4><strong>Generate Sample Data</strong></h4><pre># generate_data.py<br>import duckdb<br><br>con = duckdb.connect()<br><br>con.execute(&quot;&quot;&quot;<br>    COPY (<br>        SELECT<br>            (random() * 1000000)::INTEGER          AS order_id,<br>            [&#39;electronics&#39;,&#39;clothing&#39;,&#39;food&#39;,&#39;books&#39;][<br>                floor(random() * 4 + 1)::INTEGER]  AS category,<br>            [&#39;US&#39;,&#39;DE&#39;,&#39;FR&#39;,&#39;BR&#39;,&#39;JP&#39;][<br>                floor(random() * 5 + 1)::INTEGER]  AS country,<br>            (random() * 500 + 5)::DECIMAL(10,2)    AS amount,<br>            (random() * 0.3)::DECIMAL(5,4)          AS discount_rate,<br>            NOW() - INTERVAL (random() * 365) DAY  AS order_date<br>        FROM range(2000000)<br>    )<br>    TO &#39;data/raw/sales.parquet&#39;<br>    (FORMAT parquet, COMPRESSION zstd)<br>&quot;&quot;&quot;)<br><br>con.close()</pre><h4><strong>Transform, Aggregate, And Write</strong></h4><pre># pipeline_local.py<br>import duckdb<br>import time<br><br>con = duckdb.connect(&quot;analytics.duckdb&quot;)<br>con.execute(&quot;SET memory_limit = &#39;12GB&#39;&quot;)<br>con.execute(&quot;SET threads = 8&quot;)<br><br>start = time.perf_counter()<br><br># DuckDB reads Parquet directly. No import step.<br># Note on the window function: DuckDB evaluates aggregate expressions inside<br># OVER() after the GROUP BY is resolved, so SUM(amount) in the RANK() clause<br># refers to the already-grouped per-category-country-month sum, not raw rows.<br>con.execute(&quot;&quot;&quot;<br>    CREATE OR REPLACE TABLE sales_summary AS<br>    SELECT<br>        category,<br>        country,<br>        DATE_TRUNC(&#39;month&#39;, order_date)              AS month,<br>        COUNT(*)                                      AS total_orders,<br>        SUM(amount)                                   AS gross_revenue,<br>        SUM(amount * (1 - discount_rate))             AS net_revenue,<br>        AVG(discount_rate)                            AS avg_discount,<br>        RANK() OVER (<br>            PARTITION BY category<br>            ORDER BY SUM(amount) DESC<br>        )                                             AS revenue_rank<br>    FROM read_parquet(&#39;data/raw/sales.parquet&#39;)<br>    WHERE order_date &gt;= CURRENT_DATE - INTERVAL 90 DAY<br>    GROUP BY 1, 2, 3<br>&quot;&quot;&quot;)<br><br>elapsed = time.perf_counter() - start<br>print(f&quot;Transform complete in {elapsed:.3f}s&quot;)<br><br># Write output to Parquet with Zstandard compression<br>con.execute(&quot;&quot;&quot;<br>    COPY sales_summary<br>    TO &#39;data/processed/sales_summary.parquet&#39;<br>    (FORMAT parquet, COMPRESSION zstd, ROW_GROUP_SIZE 100000)<br>&quot;&quot;&quot;)<br><br># Verify<br>result = con.execute(&quot;&quot;&quot;<br>    SELECT category, COUNT(DISTINCT country) AS markets,<br>           SUM(net_revenue) AS total_net_revenue<br>    FROM sales_summary<br>    GROUP BY category<br>    ORDER BY total_net_revenue DESC<br>&quot;&quot;&quot;).fetchdf()<br><br>print(result)<br>con.close()</pre><h4><strong>Inspect The Query Plan</strong></h4><p>Before any pipeline goes near production, check what the optimizer is doing:</p><pre>con.execute(&quot;PRAGMA enable_profiling&quot;)<br><br>plan = con.execute(&quot;&quot;&quot;<br>    EXPLAIN ANALYZE<br>    SELECT category, SUM(amount)<br>    FROM read_parquet(&#39;data/raw/sales.parquet&#39;)<br>    WHERE country = &#39;US&#39;<br>    GROUP BY category<br>&quot;&quot;&quot;).fetchall()<br><br>for row in plan:<br>    print(row[1])</pre><p>Look for <strong><em>PARQUET_SCAN</em></strong> at the base of the plan with a <strong><em>Filters</em></strong> annotation. That confirms the filter is pushed into the scan, not applied after reading every row.</p><h3>Setup B: Iceberg REST Catalog Pipeline</h3><p>This is the argument that changes the scope of what DuckDB is. Since v1.4.0 LTS in September 2025, DuckDB writes to Apache Iceberg tables with full ACID semantics. Since v1.5.3 in May 2026, it supports MERGE INTO upserts against Iceberg tables connected to REST catalogs. The table you write is immediately readable by Spark, Trino, or Flink. The format is the contract, not the engine.</p><h4><strong>Start The Local Catalog (Lab Setup)</strong></h4><pre># Clone the DuckDB Iceberg test scripts<br>git clone https://github.com/duckdb/duckdb-iceberg.git<br>cd duckdb-iceberg<br><br># Start a local REST catalog on port 8181 and MinIO on port 9000<br>docker compose -f scripts/docker-compose.yml up -d</pre><p>For production on AWS, replace the local catalog attachment below with your S3 Tables ARN and call <strong><em>load_aws_credentials()</em></strong> instead of the manual secret.</p><h4><strong>Connect, Create, And Load</strong></h4><pre># pipeline_iceberg.py<br>import duckdb<br><br>con = duckdb.connect()<br><br>con.execute(&quot;INSTALL iceberg; LOAD iceberg&quot;)<br>con.execute(&quot;INSTALL httpfs;  LOAD httpfs&quot;)<br><br># Configure storage credentials (MinIO for local lab)<br>con.execute(&quot;&quot;&quot;<br>    CREATE OR REPLACE SECRET minio_secret (<br>        TYPE s3,<br>        KEY_ID    &#39;admin&#39;,<br>        SECRET    &#39;password&#39;,<br>        ENDPOINT  &#39;127.0.0.1:9000&#39;,<br>        URL_STYLE &#39;path&#39;,<br>        USE_SSL   false<br>    )<br>&quot;&quot;&quot;)<br><br># Attach the Iceberg REST catalog.<br># The empty string &#39;&#39; is the warehouse name for this local catalog.<br># For named catalogs or cloud providers, replace &#39;&#39; with the warehouse identifier.<br># Verified against DuckDB v1.5.3 Iceberg extension docs (May 2026).<br>con.execute(&quot;&quot;&quot;<br>    ATTACH &#39;&#39; AS lakehouse (<br>        TYPE          iceberg,<br>        CLIENT_ID     &#39;admin&#39;,<br>        CLIENT_SECRET &#39;password&#39;,<br>        ENDPOINT      &#39;http://127.0.0.1:8181&#39;<br>    )<br>&quot;&quot;&quot;)<br><br>con.execute(&quot;CREATE SCHEMA IF NOT EXISTS lakehouse.analytics&quot;)<br><br>con.execute(&quot;&quot;&quot;<br>    CREATE TABLE IF NOT EXISTS lakehouse.analytics.sales_summary (<br>        category     VARCHAR,<br>        country      VARCHAR,<br>        month        TIMESTAMP,<br>        total_orders BIGINT,<br>        net_revenue  DECIMAL(18,2)<br>    )<br>&quot;&quot;&quot;)<br><br># Initial load from local Parquet<br>con.execute(&quot;&quot;&quot;<br>    INSERT INTO lakehouse.analytics.sales_summary<br>    SELECT<br>        category,<br>        country,<br>        DATE_TRUNC(&#39;month&#39;, order_date)        AS month,<br>        COUNT(*)                                AS total_orders,<br>        SUM(amount * (1 - discount_rate))       AS net_revenue<br>    FROM read_parquet(&#39;data/raw/sales.parquet&#39;)<br>    GROUP BY 1, 2, 3<br>&quot;&quot;&quot;)<br><br>print(&quot;Initial load complete&quot;)</pre><h4><strong>Upsert With MERGE INTO</strong></h4><pre># Stage incremental data from a new Parquet file<br>con.execute(&quot;&quot;&quot;<br>    CREATE OR REPLACE TEMP TABLE incremental AS<br>    SELECT<br>        category,<br>        country,<br>        DATE_TRUNC(&#39;month&#39;, order_date)        AS month,<br>        COUNT(*)                                AS total_orders,<br>        SUM(amount * (1 - discount_rate))       AS net_revenue<br>    FROM read_parquet(&#39;data/raw/sales_incremental.parquet&#39;)<br>    GROUP BY 1, 2, 3<br>&quot;&quot;&quot;)<br><br># MERGE INTO: update existing rows, insert new ones.<br># Verified: MERGE INTO against Iceberg REST catalog tables was announced<br># in the official DuckDB v1.5.3 release post (duckdb.org, May 29 2026).<br>con.execute(&quot;&quot;&quot;<br>    MERGE INTO lakehouse.analytics.sales_summary AS target<br>    USING incremental AS source<br>        ON  target.category = source.category<br>        AND target.country  = source.country<br>        AND target.month    = source.month<br>    WHEN MATCHED THEN<br>        UPDATE SET<br>            total_orders = target.total_orders + source.total_orders,<br>            net_revenue  = target.net_revenue  + source.net_revenue<br>    WHEN NOT MATCHED THEN<br>        INSERT (category, country, month, total_orders, net_revenue)<br>        VALUES (source.category, source.country, source.month,<br>                source.total_orders, source.net_revenue)<br>&quot;&quot;&quot;)<br><br>print(&quot;Upsert complete&quot;)</pre><h4><strong>Time Travel And Snapshot Inspection</strong></h4><p>Iceberg stores every committed write as an immutable snapshot. Each snapshot has a <strong><em>snapshot_id</em></strong> (a unique integer assigned at commit time) and a <strong><em>sequence_number</em></strong> (the monotonically increasing version counter). Time travel lets you query the table as it existed at any past snapshot, which is the primary mechanism for debugging bad loads, auditing state before a MERGE, and recovering from accidental deletes.</p><pre># Inspect the full snapshot history<br>snapshots = con.execute(&quot;&quot;&quot;<br>    SELECT * FROM iceberg_snapshots(&#39;lakehouse.analytics.sales_summary&#39;)<br>&quot;&quot;&quot;).fetchdf()<br><br># sequence_number: version counter (1, 2, 3...)<br># snapshot_id:     unique commit identifier used for AT (VERSION =&gt; ...)<br># timestamp_ms:    wall-clock time of the commit<br>print(snapshots[[&quot;sequence_number&quot;, &quot;snapshot_id&quot;, &quot;timestamp_ms&quot;]])<br><br># Query the table as it existed before the upsert<br># Sort ascending so iloc[0] is the oldest snapshot (initial load), not the newest<br>first_snapshot = snapshots.sort_values(&quot;sequence_number&quot;)[&quot;snapshot_id&quot;].iloc[0]<br><br>historical = con.execute(f&quot;&quot;&quot;<br>    SELECT category, SUM(net_revenue) AS revenue<br>    FROM lakehouse.analytics.sales_summary<br>        AT (VERSION =&gt; {first_snapshot})<br>    GROUP BY category<br>    ORDER BY revenue DESC<br>&quot;&quot;&quot;).fetchdf()<br><br>print(&quot;State before upsert:&quot;)<br>print(historical)<br><br>con.close()</pre><p>That is a full incremental pipeline against a shared Iceberg table on object storage, with ACID commits, snapshot history, and time travel, from a single Python process with no cluster. The table is immediately readable by any Iceberg-compatible engine. You did not need Spark to get here.</p><h3>Where Spark Still Wins</h3><p>This argument has a ceiling. Saying it plainly is what separates a practitioner’s take from a sales pitch.</p><p>The decision framework below maps workload scale against operational requirements so you can locate your job before picking your tool.</p><figure><img alt="Three-column decision card. Left card, DuckDB in yellow: 100 MB to roughly 100 GB, use for single-engineer pipelines, ad-hoc queries, ELT on Parquet and Iceberg, and cost-sensitive jobs. Middle card, Evaluate Both in amber: 10 GB to 130 GB overlap zone, benchmark on real data and compare cost. Right card, Apache Spark in red: above 100 GB or when multi-node fault tolerance, elastic scaling, or Kafka Structured Streaming is required." src="https://cdn-images-1.medium.com/max/1024/1*w61KiOTItwDh54EX7HLuqw.png" /><figcaption>Tool selection by data size and operational requirements, based on 2025 benchmark thresholds from DataExpert.io, LakeBench, and endjin.</figcaption></figure><p>Spark wins in four specific situations.</p><p>When data is genuinely larger than one machine can hold. DuckDB runs on a single node. If your dataset spans terabytes distributed across storage nodes and the job requires multi-node shuffles, DuckDB has no mechanism for that. Spark was purpose-built for exactly this shape of problem.</p><p>When you need node-level fault tolerance. A DuckDB job that fails, fails completely. Spark has task lineage and can restart failed work from the last checkpoint. For long-running jobs on preemptible or spot compute, that resilience has real operational value.</p><p>When the workload involves live event streams. DuckDB processes files and tables. Spark Structured Streaming consumes Kafka topics with exactly-once semantics and stateful windowing. These are different problem shapes, and no amount of DuckDB optimization changes that.</p><p>When your team already runs Spark with a native vectorized engine. Databricks Photon and Microsoft Fabric’s Native Execution Engine eliminate most of the JVM overhead that makes vanilla Spark slow on small data. Miles Cole’s LakeBench showed Spark with NEE winning at 12.7 GB and running 3.5x faster at 127 GB. If you already pay for those engines, the gap from 10 GB upward is much narrower than the vanilla benchmarks suggest.</p><p>The threshold is not a fixed number. On a 16 GB laptop, the practical DuckDB ceiling is around 20 GB of data. On a 128 GB VM with NVMe scratch space, it extends past 100 GB compressed. The signal to move is not data size. The signal is when your largest intermediates consistently exceed RAM and spill heavily, or when the job genuinely needs multi-node distribution.</p><p>The mistake is not using Spark. The mistake is reaching for it without asking whether the job needs it.</p><h3>The Engineer You Are Becoming</h3><p>There is a version of this story where DuckDB is just a convenience for local development. That version missed what happened in 2025 and 2026.</p><p>DuckDB with Iceberg write support means you produce tables that Spark, Trino, and Flink can read. You run MERGE INTO upserts against a shared lakehouse on S3 without a cluster. You inspect snapshot history and time travel to any previous state from a Python script. Watershed serves 75,000 daily DuckDB queries in production. Okta processed 7.5 trillion records across 130 million files using concurrent DuckDB instances on AWS Lambda, handling spikes from 1.5 TB to 50 TB per day. Digital Turbine replaced a 100-line Scala project with six lines of Python.</p><p>The open format is the contract. The engine is a choice you can make per workload.</p><p>This week, find the three smallest regular Spark jobs in your stack. Check the actual data size and the actual compute cost. If any of them process under 50 GB, run the same transformation in DuckDB locally and time it. The measurement will make the decision obvious.</p><p>The deeper shift is in how you frame the question. Choosing the right compute layer for the workload is not a performance optimization. It is basic engineering judgment. A cluster for a 10 GB job is not wrong, exactly. It is just carrying a tool that is heavier than the problem requires.</p><p>You are becoming the engineer who asks the right question before picking the tool. Does this job need a cluster? Most of the time, for most of the pipelines in most of the stacks you will encounter, the answer is no.</p><p>DuckDB is what you reach for when the answer is no.</p><h4><em>Sources:</em></h4><p><em>Matt Martin and Zach Wilson, “DuckDB vs Spark Benchmark,” DataExpert.io, September 2025.</em></p><p><em>Matt Martin and Zach Wilson, “I Processed 1 TB with DuckDB in Less Than 30 Seconds,” DataExpert.io, December 2025.</em></p><p><em>Digital Turbine Engineering Blog, “DuckDB vs Apache Spark,” March 2025.</em></p><p><em>Miles Cole, “Small Data Showdown ‘25,” LakeBench, June 2025, github.com/milescole/lakebench.endjin,</em></p><p><em>“DuckDB vs Spark on Microsoft Fabric,” 2025. DuckDB v1.4.0 and v1.5.3 release posts, duckdb.org.</em></p><p><em>Watershed, “How Watershed Uses DuckDB,” watershed.com, 2024.</em></p><p><em>MotherDuck, “15+ Companies Using DuckDB in Production,” motherduck.com, 2024.</em></p><p><em>Mark Raasveldt and Hannes Mühleisen, “DuckDB: An Embeddable Analytical Database,” SIGMOD 2019.</em></p><p><em>Hannes Mühleisen, PyData Amsterdam 2025 keynote.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e86ac6aedf23" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/duckdb-the-death-of-small-scale-spark-e86ac6aedf23">DuckDB: The Death Of Small-Scale Spark</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Is the Data Engineer’s Power Tool, Not Their Replacement]]></title>
            <link>https://medium.com/towards-data-engineering/ai-is-the-data-engineers-power-tool-not-their-replacement-b5e902de5614?source=rss-10235c107f8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5e902de5614</guid>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Nevenka Lukic]]></dc:creator>
            <pubDate>Mon, 01 Jun 2026 16:35:19 GMT</pubDate>
            <atom:updated>2026-06-01T16:35:19.815Z</atom:updated>
            <content:encoded><![CDATA[<h4>While everyone argues about replacement, the role is quietly becoming the most important job in every AI-driven organization</h4><figure><img alt="Bar chart and stat cards showing data engineering job growth projections for 2026, including 110% WEF forecast and 37% of DE time spent on AI projects" src="https://cdn-images-1.medium.com/max/1024/1*Mj3Zv43BeSoeKdXeKS6j4Q.png" /></figure><h3>The Noise Around AI And Why It Misses The Point</h3><p>Every few months, a new headline arrives. “AI will automate data engineering.” “LLMs can write your pipelines.” “Your job is changing forever.”</p><p>Most of it is noise. Not because it is wrong, but because it is asking the wrong question.</p><p>The conversation keeps circling around replacement. Whether AI will take the job, shrink the headcount, or make the role irrelevant. That framing is a distraction. It keeps engineers in a defensive posture when they should be in an offensive one.</p><p>The right question is simpler and more useful: what does AI let you build today that you could not build six months ago?</p><p>That question has a concrete answer. It points to specific tools, specific architectural patterns, and specific categories of work that are now inside the reach of a single data engineer that used to require a team. This article is about that answer. No speculation. No career anxiety. Just what is actually shipping, what it does, and how to use it.</p><h3>The Numbers That Actually Matter</h3><p>Before diving into tooling, it is worth grounding the conversation in real data, because the data is unambiguous.</p><figure><img alt="Stat cards and bar chart comparing data engineering AI adoption from 2024 to 2025. Data team headcount growth rose from 14% to 40%. Daily AI tool usage rose from 30% to 80%. DE time spent on AI projects grew from 19% in 2023 to 37% in 2025, projected to reach 61% by 2027." src="https://cdn-images-1.medium.com/max/1024/1*ZEmt12dxfXRw1niszSW2ow.png" /><figcaption>Data engineering team growth and AI tool adoption, 2024 vs 2025. Sources: MIT Technology Review and Snowflake, dbt Labs, WEF.</figcaption></figure><p>According to a 2025 survey of 400 senior technology executives conducted by MIT Technology Review and Snowflake, data engineers now spend 37% of their working time on AI projects. In 2023, that figure was 19%. By 2027, the projection is 61%. That is not a role in decline. That is a role being pulled toward the center of every organization’s most important work.</p><p>The dbt Labs 2025 State of Analytics Engineering report, which surveyed 459 practitioners, found that 80% now use AI tools in their daily workflows, up from 30% the prior year. Forty percent of data teams grew in headcount in 2025, compared to 14% the year before. The World Economic Forum’s Future of Jobs Report 2025 projects 110% growth for Big Data Specialists by 2030, the single fastest-growing role on its list.</p><p>The same MIT and Snowflake survey found that 77% of leaders say data engineering workloads are increasing, and 72% describe data engineers as integral to the business. Among companies with revenue above $10 billion, that figure rises to 86%.</p><p>None of this looks like a role under threat. It looks like a role under pressure to expand.</p><p>The pressure is real. The opportunity inside it is bigger.</p><h3>LLM-Assisted Development: Writing Less Code, Shipping More Value</h3><p>The most immediate change to the daily workflow is in how data engineers write code. The tools here are not experimental. They are in general availability across every major platform, and they are grounded in catalog metadata rather than raw row data, which is what makes them safe to use inside governed enterprise environments.</p><p>They fall into two distinct groups. The first group lives inside your warehouse. The second operates at the architecture level, across your entire repo and stack.</p><p><strong>Warehouse-Native Copilots</strong></p><p>These tools share one critical property: they are grounded in your actual catalog, schemas, and governance rules before they generate a single line of code. They never touch row-level data. They are already inside the platform you work in, which means zero friction to adopt.</p><p><strong>dbt Copilot</strong> generates SQL models, YAML configurations, tests, semantic models, and documentation from natural language prompts. A Developer Agent mode can autonomously refactor and validate models end to end. Writing a new staging model with tests and documentation drops from an hour to under ten minutes.</p><p><strong>Databricks Assistant in Agent Mode</strong>, which shipped in December 2025, plans multi-step workflows, retrieves assets from Unity Catalog, generates and executes code, and fixes errors automatically. Banco Bradesco’s data engineering team reported a 50% reduction in coding time after adoption. That is a measurable change in how much a team can ship per sprint.</p><p><strong>Snowflake Cortex Code</strong> builds entire dbt project skeletons from a single prompt, including sources.yml, staging models, silver layer transformations, and unit tests, while remaining fully aware of your existing schemas, governance rules, and compute constraints. As of April 2026, over 50% of Snowflake customers are actively using it.</p><p><strong>Gemini in BigQuery</strong> handles SQL generation from natural language comments, partitioning and clustering recommendations, Python code generation, and cross-table exploration through BigQuery Data Canvas.</p><p><strong>External Ai Development Tools</strong></p><p>The second group operates above the warehouse. These tools work at the repo and architecture level, across your entire project rather than a single model or query.</p><p><strong>Claude Code</strong> with MCP servers has become the standard for senior data engineers doing architecture-level work. The pattern is practical: encode your project conventions in a CLAUDE.md file, connect an MCP server to your warehouse or dbt project, and let the agent handle the structural work while you focus on the design decisions that actually require judgment. <strong>Cursor</strong> handles fast IDE-native iteration, with multi-agent workflows that make individual file edits and cross-file refactors significantly faster.</p><p><strong>The Takeaway</strong></p><p>Warehouse-native tools handle the routine work: models, tests, documentation, SQL generation. External tools handle the structural decisions: migrations, architecture reviews, repo-wide refactors. Together they recover the hours that used to disappear into boilerplate. Condé Nast’s senior director of data engineering reported saving 16 hours per project sprint after adopting dbt Cloud. That time does not disappear. It gets redirected toward the work that matters more.</p><p>That speed comes with a specific responsibility. AI generates code confidently, including confidently wrong code. A model that passes its own generated tests but introduces a fan-out join inflating row counts by 3x will sail through a weak CI pipeline and land in production. The data engineer’s job has shifted from writing logic to auditing it. This makes rigorous unit testing, schema contracts, and CI/CD validation more important now than they were before AI tooling existed, not less.</p><p>The cost dimension deserves equal attention. An autonomous agent with warehouse access and no guardrails is a budget incident waiting to happen. A poorly scoped query generated by Cortex Code or Databricks Assistant running a SELECT * across a petabyte-scale table can consume hours of compute credits in seconds. Data engineers are now financial guardians as much as they are architects. Warehouse timeout policies, query governors, slot quotas, and cost-per-query monitoring are not optional add-ons to an AI-assisted workflow. They are the foundation it runs on.</p><h3>Agentic Observability: From Incident Reports To Automatic Triage</h3><p>Data quality incidents have always followed the same pattern. Something breaks. An alert fires. An engineer spends two hours pulling logs, tracing lineage, checking upstream sources, and ruling out false positives before identifying the actual cause. Then they fix it and write a post-mortem.</p><p>AI is not eliminating that work. It is compressing the investigation loop from hours to minutes.</p><p>To understand how, it helps to think about observability as three distinct layers. Detection is the layer that spots anomalies automatically by learning the statistical distribution of your data over time. Triage is the layer that investigates root cause without manual intervention, running parallel hypotheses across data, system, and code dimensions simultaneously. Validation is the layer that enforces explicit quality rules at the pipeline level, catching known failure modes before they reach production. Different tools operate at different layers, and a mature observability architecture needs all three.</p><h4>Detection And Triage</h4><p><strong>Monte Carlo’s Monitoring Agent</strong>, generates threshold recommendations by analyzing sampled data, metadata, and query logs across your entire pipeline graph. It has achieved a 60% recommendation acceptance rate across thousands of suggestions. More importantly, its <strong>Troubleshooting Agent</strong>, which landed in Q2 2025, launches parallel sub-agents that simultaneously test hypotheses across data, system, code, and model dimensions. Rather than following a sequential investigation path, it runs the entire diagnostic tree at once.</p><p>Roche’s data platform team described an experience that most data engineers will recognize: incidents that previously required days of investigation now resolve in minutes.</p><p>Understanding why this matters requires knowing what the actual distribution of incidents looks like. Monte Carlo’s own telemetry from monitoring over 11 million tables in 2025 shows that pipeline execution faults cause 26.2% of data incidents, real-world variation causes 20%, and intentional changes like backfills cause 14.2%. That means roughly 34% of fired alerts are not bugs.</p><p>They are noise. An engineer chasing those alerts manually is losing hours every week to problems that do not exist.</p><figure><img alt="Donut chart showing the distribution of data incident causes across 11 million monitored tables in 2025. Pipeline execution faults account for 26.2%, real-world data variation 20%, intentional changes such as backfills 14.2%, alert noise with no underlying issue 34%, and other causes 5.6%." src="https://cdn-images-1.medium.com/max/1024/1*oEaAOyBnptLeUwMDmz4MTg.png" /><figcaption>Distribution of data incident causes across 11 million monitored tables. Source: Monte Carlo, 2025.</figcaption></figure><h4>Validation</h4><p><strong>Soda AI</strong> ships Record-Level Anomaly Detection with proprietary algorithms that produce 70% fewer false positives than Facebook Prophet and can scan one billion rows in 64 seconds. <strong>Great Expectations</strong> now supports AI-generated expectation suites through its ExpectAI feature in GX Cloud, as well as natural language interfaces built on top of the open-source library.</p><p>The practical production pattern combines both layers deliberately. Rule-based validation handles deterministic quality checks: known constraints, schema expectations, freshness SLOs. LLM-based semantic validation handles the nuanced, cross-field consistency checks that explicit rules cannot reach. Neither replaces the other. Together they cover the full failure surface.</p><p>The data engineer’s job in this layer is not disappearing. It is shifting from manual triage to designing the monitoring architecture that agents operate within.</p><h3>The Semantic Layer: The Architecture AI Actually Needs</h3><p>This is the section that matters most for anyone serious about building AI-ready data infrastructure.</p><p>Only 7% of enterprises say their data is completely AI-ready, according to a March 2026 report by Cloudera and Harvard Business Review Analytic Services. Global enterprise AI investment surpassed $684 billion in 2025. Yet over 80% of that spend failed to deliver its intended business value, and MIT’s Project NANDA found that roughly 95% of generative AI pilots show no measurable impact on profit and loss.</p><p>The bottleneck is not the model. It is not the compute. It is the absence of a governed semantic layer sitting between raw warehouse tables and the systems that consume them.</p><p>Here is the concrete problem. An AI agent querying a raw schema for “monthly recurring revenue” might hit three different tables with three different definitions, each built by a different team at a different time. It picks one and returns a confident, wrong answer. The number goes into a deck. The executive makes a decision on bad data. The diagram below shows exactly this failure, and what the governed alternative looks like.</p><figure><img alt="Architecture diagram comparing two approaches to AI data access. On the left, an AI agent queries three raw tables with conflicting definitions of monthly recurring revenue, producing wrong answers. On the right, all consumers including AI agents, dashboards, and notebooks query a single governed semantic layer via MCP, returning consistent results. Tools shown include dbt Semantic Layer, Snowflake Semantic Views, Databricks Unity Catalog, Cube, and AtScale." src="https://cdn-images-1.medium.com/max/1024/1*jYYEpR_-ZYkrpuWnS5zM2w.png" /><figcaption>AI data access without a semantic layer versus with one. Every governed consumer returns the same answer.</figcaption></figure><p>The fix is straightforward in principle. monthly_recurring_revenue is defined once, in versioned code, with governance built in. Every consumer, whether a dashboard, a notebook, or an AI agent, queries that single definition and returns the same number.</p><p>The tooling for this is mature and available today. <strong>dbt Semantic Layer with MetricFlow</strong> is open source and integrates natively with dbt projects. <strong>Snowflake Semantic Views</strong> and its <strong>Open Semantic Interchange</strong> standard allow metric definitions to be shared across tools and vendors. <strong>Databricks Unity Catalog Business Semantics</strong> and <strong>Metric Views</strong> provide the same capability within the Lakehouse. <strong>Cube</strong> and <strong>AtScale</strong> offer headless, universal semantic layers that sit on top of any warehouse.</p><p>The 2025 breakthrough that tied all of this together is the <strong>Model Context Protocol</strong>. MCP is the bridge between LLM agents and governed metric definitions. Instead of querying raw tables directly, agents query semantic definitions through a standardized interface. They get the right number every time, with governance enforced at the layer that matters. AtScale’s MCP Server, for example, brought natural language access to governed metrics directly inside Slack and Google Meet, without exposing a single raw table to the agent.</p><p>Promethium-cited research found that semantic layers reduce generative AI data errors by 66%. That number represents the difference between an AI product that earns trust and one that gets abandoned after the third wrong answer.</p><p>The data engineer who defines these metrics, maintains their governance, and exposes them correctly through MCP is doing the work that determines whether the organization’s AI investment pays off.</p><h3>Building For AI Consumers: RAG Pipelines And Feature Stores</h3><p>Two categories of data engineering work have grown dramatically in scope over the past two years. Neither of them is traditional ETL. Both of them are fundamentally data engineering problems.</p><p><strong>Rag Ingestion Pipeline</strong></p><p>The first is <strong>RAG ingestion pipelines</strong>. Retrieval-Augmented Generation is now the dominant enterprise LLM application pattern. Databricks reports that over 60% of organizations are actively developing AI-powered retrieval tools. The architecture has stabilized into a repeatable pattern:</p><ul><li><strong>Parse</strong> unstructured documents with Unstructured.io or Apache Tika</li><li><strong>Chunk</strong> into semantically coherent segments using LangChain or LlamaIndex splitters, with 512 to 1024 token windows and 10 to 20% overlap</li><li><strong>Embed</strong> with Voyage-3-large for highest quality, or OpenAI text-embedding-3-small for convenience</li><li><strong>Index</strong> into a vector store like Qdrant, Pinecone, or Databricks Vector Search</li><li><strong>Retrieve</strong> using hybrid dense-plus-sparse search with a Cohere reranker on top</li></ul><figure><img alt="Two-row flowchart showing a production RAG pipeline architecture. The offline indexing pipeline runs from document sources through parse with Unstructured.io, chunk with 512 to 1024 token windows, embed with Voyage-3-large, and index into a vector store such as Qdrant or Pinecone, with lineage tracking and quality monitoring applied throughout. The online query pipeline runs from user query through retrieve using hybrid dense and sparse search, rerank with Cohere rerank-v3, and generate with an" src="https://cdn-images-1.medium.com/max/1024/1*wncdAs5NQ8pP3Y275Pt9WQ.png" /><figcaption>Production RAG architecture: offline indexing pipeline and online query pipeline, with lineage and quality monitoring throughout.</figcaption></figure><p>The offline indexing pipeline is governed infrastructure. It needs lineage tracking, freshness monitoring, schema validation on chunk metadata, and access controls on the resulting index. Every one of those requirements is a data engineering problem. The source happens to be a document corpus and the destination a vector index, but the discipline is identical to any production ETL pipeline you have built before.</p><p><strong>Retrieval And Ranking Layer</strong></p><p>The online query pipeline is where architecture choices have the most direct impact on answer quality. Hybrid search combines dense vector similarity with sparse keyword matching, which means the retrieval layer catches both semantically related content and exact terminology that dense-only search misses. That gap translates directly into fewer irrelevant chunks reaching the LLM, shorter context windows, and more accurate answers at lower inference cost.</p><p>Chunking strategy matters for the same reason. Fixed-size windows split sentences and ideas at arbitrary boundaries, breaking the coherence of the retrieved chunk before it ever reaches the model. Semantic chunking preserves meaning by splitting at natural conceptual boundaries, which is why it consistently outperforms fixed-size approaches on unstructured narrative text. Embedding model choice affects the architecture upstream. Voyage-3-large supports 32,000-token context windows compared to 8,000 for OpenAI’s text-embedding-3-small, which means longer documents can be embedded as coherent units rather than forcing aggressive chunking that introduces fragmentation early in the pipeline.</p><p>A reranker like Cohere rerank-v3 sits at the top of the retrieval layer, re-scoring the top retrieved chunks before they reach the model. It is a small addition to the pipeline that consistently improves final answer quality by filtering out chunks that are topically related but contextually irrelevant to the specific query.</p><p><strong>Feature Stores And Ml Pipeline Infrastructure</strong></p><p>The second category is <strong>feature stores and ML pipeline infrastructure</strong>. The central problem that feature stores solve is training and serving skew, which affects approximately 40% of production ML models. When the feature values used to train a model differ from the feature values served at inference time, model performance degrades silently. The data engineer builds the pipelines that prevent that from happening.</p><p>DoorDash’s Fabricator, presented at the Databricks Data and AI Summit in 2024, manages 1,400 daily batch jobs and 2.2 trillion feature values. After adopting Fabricator’s declarative feature engineering approach, DoorDash accelerated their feature development lifecycle by more than 2 times and scaled feature volumes nearly 10 times. The entire system is a data engineering artifact. It just feeds a recommendation model instead of a dashboard.</p><p><strong>Reliability Starts At The Contract Layer</strong></p><p>Both RAG pipelines and feature stores break silently when upstream data changes without warning. A data contract is the mechanism that prevents that. It is a formal, versioned agreement between a data producer and its consumers that covers schema, semantics, quality rules, freshness SLOs, and change policy. Stored in Git next to the producing code, validated in CI, and monitored in production, it is the difference between a reliable AI product and one that degrades invisibly every time a backend team renames a column.</p><p>Soda AI now generates data contracts automatically from sample data. A July 2025 arXiv paper demonstrated that fine-tuned LLMs can generate valid JSON Schema and Avro contracts from data samples, reducing the manual authoring workload by over 70%.</p><h3>What The Role Actually Looks Like Now</h3><p>The data engineering role is not disappearing. It is moving upstream.</p><p>Tristan Handy, founder and CEO of dbt Labs, said in March 2025: “I think it will be hard to compare data engineering in 2024 and data engineering in 2028 and say those are the same things.” Chris Child, VP of Product for Data Engineering at Snowflake, put it plainly: “Over time, the role of the data engineer will be less about writing code for every single pipeline and more about managing the infrastructure that AI agents are operating in.”</p><p>That shift is concrete and measurable. The engineers who are gaining leverage right now are doing three specific things. They are defining semantic metrics once, in governed, versioned code, so that every AI agent and every dashboard returns the same answer. They are building RAG ingestion pipelines and feature stores as first-class data engineering infrastructure with lineage, quality monitoring, and access controls. And they are using LLM-assisted development tools not to avoid thinking, but to eliminate the repetitive structural work that used to consume most of a sprint.</p><p>The engineers who are falling behind are doing the opposite. They are writing every pipeline from scratch. They are chasing observability alerts manually. They are exposing raw tables directly to AI tools and wondering why the answers are wrong.</p><p>The gap between those two groups is widening fast.</p><p>Here is the practical starting point. Enable dbt Copilot, Databricks Assistant, Snowflake Cortex Code, or Gemini in BigQuery, whichever platform you work on, and use it for one full sprint of model development. Track the time you recover. Take that time and use it to define five core business metrics in your semantic layer and expose them through MCP. Then pick one internal document corpus and build a governed RAG ingestion pipeline with Elementary or Monte Carlo monitoring on the output.</p><p>At the end of it, you will have done something more important than shipped a pipeline. You will have started the transition into what the data engineer role is actually becoming.</p><p>Three things, stated plainly.</p><p>You are the <strong>semantic layer owner</strong>. The single person responsible for defining business logic once, in governed versioned code, so that every AI agent, every dashboard, and every analyst returns the same answer.</p><p>You are the <strong>AI system architect</strong>. The person who designs the RAG pipelines, feature stores, and context engineering patterns that determine whether the organization’s AI investment produces reliable outputs or confident hallucinations.</p><p>You are the <strong>governance engineer</strong>. The person who enforces data contracts, monitors pipeline health, controls access to sensitive data, and ensures that autonomous agents operate within cost and quality boundaries.</p><p>These are not future job titles. They are the responsibilities landing on data engineers right now, in production environments, at companies that are serious about making AI work.</p><p>The tool is in your hands. The question is what you build with it.</p><p><em>Sources: MIT Technology Review and Snowflake, “Redefining Data Engineering in the Age of AI,” 2025. dbt Labs, “2025 State of Analytics Engineering.” World Economic Forum, “Future of Jobs Report 2025.” Monte Carlo, “Data Quality Statistics from Monitoring 11 Million Tables,” 2025. Databricks, “An AI-First Approach to Data Engineering with Lakeflow and Agent Bricks,” 2025. Cloudera and Harvard Business Review Analytic Services, March 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5e902de5614" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/ai-is-the-data-engineers-power-tool-not-their-replacement-b5e902de5614">AI Is the Data Engineer’s Power Tool, Not Their Replacement</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learn AI Programming from Scratch: A Fast, Local Python Setup]]></title>
            <link>https://blog.devgenius.io/learn-ai-programming-from-scratch-a-fast-local-python-setup-1b920c537714?source=rss-10235c107f8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b920c537714</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Nevenka Lukic]]></dc:creator>
            <pubDate>Sun, 03 May 2026 16:06:01 GMT</pubDate>
            <atom:updated>2026-05-03T16:06:01.234Z</atom:updated>
            <content:encoded><![CDATA[<p>When I decided to learn machine learning and modern AI, I noticed that most tutorials rely on heavy, bloated software suites like Anaconda. I wanted to build something leaner, faster, and completely under my control.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gxg4T4uCwPD0nY0ZsCXHPQ.jpeg" /><figcaption>Local Developer Environment</figcaption></figure><p>I wanted to:</p><ul><li>Learn modern AI programming using lightweight, native tools.</li><li>Understand the underlying mechanics of neural networks, rather than just calling an API.</li><li>Build a structured path from basic Python coding to running local Large Language Models (LLMs).</li><li>Create a reproducible, open-source learning environment that anyone could use.</li></ul><p>By the end of this project, I had a complete, four-module learning workspace that:</p><ul><li>Refreshes foundational Python syntax and data structures.</li><li>Explores and cleans datasets locally using Jupyter Notebooks, pandas, and NumPy.</li><li>Trains a custom PyTorch digit-recognizing neural network from scratch.</li><li>Runs a local Natural Language Processing (NLP) sentiment analysis pipeline.</li></ul><h4>Tools &amp; Libraries I Used</h4><ul><li><strong>Python</strong> and <strong>uv</strong> for lightning-fast, lightweight environment management.</li><li><strong>pandas</strong> and <strong>NumPy</strong> for data cleaning, vectorization, and matrix math.</li><li><strong>Matplotlib</strong> and <strong>Seaborn</strong> for data visualization.</li><li><strong>PyTorch</strong> for building, training, and testing deep learning models.</li><li><strong>Hugging Face (</strong><strong>transformers)</strong> for loading pre-trained NLP models.</li></ul><h4>What I Learned</h4><ul><li><strong>Lightweight is better.</strong> Building an environment from scratch using uv instead of Anaconda makes the workspace incredibly fast and prevents dependency bloat.</li><li><strong>Fundamentals matter.</strong> You cannot skip straight to Transformers without first understanding data manipulation in pandas and tensor operations in PyTorch.</li><li><strong>Hardware quirks are real.</strong> I learned how to troubleshoot deep OS-level memory crashes (like the dreaded macOS “bus error”) by disabling specific features like safetensors when running large models locally.</li><li><strong>Demystifying AI.</strong> Building an end-to-end NLP pipeline from scratch removes the “magic” behind modern AI and shows exactly how tokenization and attention mechanisms work.</li></ul><h4>Who is this for?</h4><p>This repository is a great fit for:</p><ul><li>Beginners who want a rock-solid, step-by-step foundation in Python.</li><li>Self-taught developers or data professionals looking to transition into machine learning.</li><li>Anyone curious about how modern AI models actually work under the hood.</li><li>Developers who want a clean, fast, local AI environment without installing heavy software suites.</li></ul><h4>Final Thoughts</h4><p>Sometimes the best way to understand complex AI concepts is to strip away the heavy abstractions and build everything from the ground up. Creating this repository didn’t just teach me how to write PyTorch or Hugging Face code; it gave me total confidence and control over my local development environment.</p><p>If you want to learn machine learning, don’t be afraid to start small and local. You will be surprised how quickly the pieces fit together once you start building.</p><p><strong>Project Repository:</strong> <a href="https://www.google.com/search?q=https://github.com/your-username/your-repo-name">github.com/your-username/your-repo-name</a></p><p>If you need to practice Python more, I highly recommend this book:</p><p>“<a href="https://mybook.to/JwiH4">Python Data Engineering Resources</a>”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b920c537714" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/learn-ai-programming-from-scratch-a-fast-local-python-setup-1b920c537714">Learn AI Programming from Scratch: A Fast, Local Python Setup</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enterprise Data Pipeline Controls Checklist: Security, Auditability, and Compliance]]></title>
            <link>https://medium.com/towards-data-engineering/enterprise-data-pipeline-controls-checklist-security-auditability-and-compliance-b1b11f48cf95?source=rss-10235c107f8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1b11f48cf95</guid>
            <category><![CDATA[data-security]]></category>
            <category><![CDATA[compliance]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <dc:creator><![CDATA[Nevenka Lukic]]></dc:creator>
            <pubDate>Tue, 21 Apr 2026 11:41:39 GMT</pubDate>
            <atom:updated>2026-04-21T11:41:39.452Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g9-6xXXGdqsMEVfQY48l2Q.png" /></figure><p>This article will share a practical SOC 2 controls checklist for data pipelines. We’ll go over data regulation, access, networking, encryption, data integrity, and other important areas. We will also explain what to focus on when choosing a pipeline platform.</p><p>The principles we discuss can be applied to any tool you choose when creating a secure, compliant pipeline.</p><h3>Data Pipelines Then and Now</h3><p>Data pipelines used to be something that happened behind the scenes. Scheduling an extract to load data into a warehouse late at night, so that the rest of the organization could use it the next day, rarely required much explanation. Today, these pipelines feed operational systems, trigger business logic, and move sensitive information across various environments in real time. And as soon as you have a pipeline that processes customer data or financial transactions, it becomes part of your regulated compliance scope. As such, it has to be audited eventually.</p><p>The problem is, many teams continue to treat compliance as something to be addressed only after the pipeline has been built. They make sure everything’s working first, and only then do they try to pass an audit. As a result, the controls implemented in the pipeline are often full of gaps. Access is too broad, encryption is inconsistent, and there are no records of who made changes to the systems or when. Once auditors start asking questions about compliance, the development team is forced into panic mode trying to prove something that has been documented poorly (if at all).</p><h3>SOC 2 Controls for Data Pipelines</h3><p>SOC 2 is based on five Trust Services Criteria: security, availability, processing integrity, confidentiality, and privacy. Security is the only one that is absolutely required, but many organizations that deal with regulated information include processing integrity and confidentiality as part of their audit.</p><p>Three areas are typically under the most scrutiny on the pipeline.</p><ul><li><strong>Security (CC6, CC7)</strong> focuses on who has access to the data and how operations are monitored.</li><li><strong>Processing integrity</strong> examines whether all the received data is complete and accurate.</li><li><strong>Change management (CC8)</strong> evaluates if all infrastructure alterations have gone through a formal review process.</li></ul><p>Another important distinction is the difference between Type I and Type II reports. Type I is a snapshot that verifies specific controls existed on a particular date, whereas Type II covers a period of six to twelve months in order to prove that give controls operated consistently during the particular timeframe.</p><p>For pipeline systems, Type II is more important because auditors’ primary goal is to verify that encryption, access controls, and logging were operational every day, not just on the day of the audit.</p><p>Since teams often confuse the different SOC report types, the table below will provide a comparison:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ywyFROphV8LlYb9XzLb83w.png" /></figure><p>When auditing pipelines, you may be expected to provide evidence of compliance upon request, including access logs, configuration details for encryption, deployment records, and change approvals.</p><h3>Identity and Access as the First Control</h3><p>Shared logins are still common in data tooling. The whole team shares a single database credential, or everyone uses the same service account to deploy pipelines. In these cases, it become difficult to determine who has made a particular change to the data or the environment. It also means you can’t revoke access for one person without affecting everybody else.</p><p>Centralized authentication via Single Sign-On (SSO) resolves the tracking issue. Organizations that use an identity provider such as Okta or Azure AD can enforce multi-factor authentication (MFA) and password policies in one location. For example, Estuary uses SAML 2.0 to enable SSO and supports Just-In-Time provisioning to create user accounts automatically when users log in for the first time. This removes the manual onboarding process, and in case a user leaves the company, we don’t end up with an orphan account.</p><p>The second problem refers to what users are actually allowed to do, and it requires granular access control. Role-based access combined with prefix scoping will allow you to restrict visibility and actions. For example, the admin manages the entire tenant, whereas a read-only user can only see the pipeline status and log activity but can’t make any changes.</p><p>Prefix-based scoping also allows each department (like marketing or finance) to only view collections that fall under their prefix, such as yourCompany/marketing/ and yourCompany/finance/. This is how you ensure that the minimum required privileges are in place.</p><h3>Lock Down Your Network</h3><p>Perimeter-based security, also known as the “castle and moat” model, assumes that everything within the network can be trusted. That assumption does not hold in environments with cloud services, remote access, and third-party integrations, though. The zero-trust model is based on a different principle: all connections have to be validated, no matter where they have originated.</p><p>One of the best ways to create a successful pipeline is to separate your control plane from your data plane. The control plane is responsible for configuration and user management, and the data plane is where the actual data processing occurs. When these two components are properly isolated, gaining access to the management interface will not provide access to the underlying data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AAVps6GFaClLAGpop9xReA.png" /><figcaption>Network Connectivity Paths</figcaption></figure><p>For organizations in regulated industries, standard cloud deployments do not satisfy strict data residency requirements. Private deployments address this by maintaining the data plane within a dedicated network, while the control plane is being managed by the platform provider.</p><p>BYOC (Bring Your Own Cloud) takes this one step further. Namely, in this model, the data plane runs on your cloud account, while the provider manages the platform. Your data remains within your network and storage.</p><h3>Encryption That Is Not Optional</h3><p>Encryption should be the standard way of protecting all data. Data in transit should use TLS 1.2 or higher on every connection, whereas data at rest should be encrypted with AES-256 across all storage layers.</p><p>For the majority of organizations, cloud-managed encryption keys should be enough. However, enterprises that are governed by strict regulations often need customer-managed keys through a cloud KMS service. This approach gives the organization complete authority over how keys are rotated, how access is granted, and how they are revoked. Auditors will typically look for evidence of this capability in a Type II report.</p><p>However, encryption alone does not fully address the risk that personally identifiable information (PII) and other sensitive data carry. In many pipeline architectures, identifiers such as email addresses, Social Security numbers, or health records pass through several processing stages before reaching their final destination. Each stage where this data remains in cleartext increases the likelihood of exposure.</p><p>A practical way to reduce that risk is to mask or <a href="https://docs.estuary.dev/features/redaction/">redact</a> sensitive fields as early as possible, ideally at capture. This will limit the number of systems and people who ever see the raw data and will make downstream access control simpler. Instead of managing PII exposure across multiple destinations, you handle it once at ingestion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OvDEd-nN2DbikagLGQlfnA.png" /><figcaption>Redact PII early to limit its exposure</figcaption></figure><p>Building custom encryption or skipping it for internal traffic adds unnecessary risks. Internal networks are breached and custom cryptographic implementations often contain bugs, so it’s safer to rely on the encryption provided by your cloud and platform and confirm that it’s consistently enabled everywhere.</p><h3>Accurate Data Even When Things Change</h3><p>Under SOC 2, it is required that data is complete, accurate, and processed only by authorized operations. In financial or medical context, a missing transaction or a duplicate record can easily turn into a regulatory problem.</p><p>Ensuring <a href="https://estuary.dev/blog/exactly-once-delivery/">exactly-once delivery</a> in distributed systems is difficult. Network retries and failures can easily lead to duplicate writes if the system is not designed carefully. Platforms that support exactly-once semantics typically rely on checkpointing and idempotent operations. If a connector crashes, it resumes from the last recorded checkpoint and will not skip or duplicate any data. In financial systems, where a duplicated withdrawal is unacceptable, exactly-once delivery is a baseline requirement.</p><p>Schema drift is another common reason why pipelines fail. An application team may rename a column or change a data type without notifying the data team, which will cause downstream systems to break.</p><p>Platforms that separate write schemas, which validate data on entry, from read schemas, which govern consumption, are able to handle many of these changes. For example, if a source adds a new column, it can be discovered automatically, or if a field widens from integer to float, the cast can be handled safely.</p><p>However, narrowing changes that could cause data loss still requires manual intervention since you want these changes to produce a visible failure instead of quietly causing corruption in your downstream data.</p><h3>Audits Without Manual Effort</h3><p>An audit trail should be able to provide answers to the four key questions:</p><ol><li>Who performed the action?</li><li>What action was taken?</li><li>When did it occur?</li><li>Where in the system did it occur?</li></ol><p>(As a side note, imagine if your team had to gather this information manually every time an auditor asked for it. Preparing for reviews could take weeks of their time, which could otherwise be spent on productive work.)</p><p>Monitoring and auditing are often treated as the same thing, but they serve completely different purposes. Monitoring is about operational health. It tells you what is happening right now, if jobs are running, whether error rates are spiking, or if the throughput is where you expect it to be. Auditing, on the other hand, looks backward to find out whether the controls have been working consistently over time. As you can assume, this requires durable records, not just dashboards.</p><p>Estuary exposes two internal collections that help with this, ops/yourCompany/logs and ops/yourCompany/stats. The logs collection captures runtime events and lifecycle changes, such as when a capture starts, fails, or emits an error. The stats collection records transaction level metrics, including bytes read, bytes written, and document counts. Both are read-only and immutable, which makes them usable as audit evidence.</p><p>Teams can export this information to a SIEM or store it long-term to meet retention requirements. If auditors need direct visibility, read-only roles with prefix scoping allow them to inspect the system without changing anything. Using the flowctl CLI, they can query logs and metrics for a defined time range instead of asking the team to manually assemble screenshots and reports. And voila! What could have taken days of back and forth can actually be resolved with a few targeted queries.</p><h3>Change Management Through Code</h3><p>While Estuary’s History tab records the user who made a change, the time when it was made, and a diff highlighting what changed, these logs alone do not guarantee that it was actually authorized, tested, and properly documented, as required by SOC 2 C8.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6AcsNaUiOZ_8XEsDnGTG2g.png" /><figcaption>Connector History Tab</figcaption></figure><p>Auditors want to see evidence of a <strong>controlled</strong> change process, not just a record of what happened. Treating pipeline definitions as code strengthens this control. When captures, materializations, and collections are defined in specification files managed through Git, every change will have a commit hash, a timestamp, an author, and a reviewer associated with it.</p><p>Below is a simple example of an Estuary specification:</p><pre>captures:<br>  acmeCo/source-postgres:<br>    endpoint:<br>      connector:<br>        image: ghcr.io/estuary/source-postgres:dev<br>        config:<br>          address: &lt;host&gt;:&lt;port&gt;<br>          database: postgres<br>          user: flow_capture<br>          password: &lt;password&gt;<br>    bindings:<br>      - resource:<br>          stream: shipments<br>          namespace: public<br>          syncMode: incremental<br>        target: acmeCo/postgres_shipments</pre><p>If a change is needed, it goes through a pull request. A second engineer reviews it, and CI uses flowctl to validate the YAML and run tests. This way, the person making the change is not the only one who reviews it, and nothing reaches production until the entire process is complete.</p><p>Git also saves the history, and the benefits of this feature become clear over time. For example, after several months, you can clearly demonstrate who made changes, who approved them, and when they were deployed. This is exactly the kind of evidence auditors look for in a Type II review; plus, if something breaks, you can just compare versions or roll back to the previous state.</p><h3>Full SOC2 Compliance Checklist</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*fc-JZa8Mu6hZ8VISZY-pGQ.jpeg" /></figure><h3>Compliance Starts With the Design</h3><p>Now pipelines reside within your regulatory boundaries. They handle both regulated data and production workflow for your business, which puts them squarely within audit scope.</p><p>When the initial design includes all the essential processes, such as access control, data encryption, data integrity, audit log requirements, and structured change management, audits become relatively easy. Without them in place, teams can spend weeks recreating history.</p><p>This advantage becomes even clearer if you choose a platform like Estuary, which implements these controls from the get-go. Explore how Estuary helps teams build secure, audit-ready data pipelines without adding compliance work later.</p><h3>Frequently Asked Questions</h3><h3>Why do data pipelines fall under SOC 2 audit scope?</h3><p>Pipelines today move sensitive customer data, financial records, and PII across environments in real time. The moment they start processing regulated data, they fall into your compliance boundary. Auditors will primarily focus on access controls, encryption, and logging here, just as they would for any other production infrastructure.</p><h3>Why are shared credentials a compliance risk in data tooling?</h3><p>Individual accountability is a baseline expectation in compliance. When a team shares credentials or accounts, you can’t trace who made specific changes. Plus, revoking access in such cases affects everyone in the chain.</p><h3>Why is exactly-once so important for audits?</h3><p>Exactly-once delivery means that every record is processed only once, no skips and duplicates. This promise is quite hard to keep, though, and you have to handle it on the infrastructure level. In sensitive industries (like healthcare, finance, or any other industry that handles sensitive data), duplicates are both a data quality and regulatory risk. When issues arise, your infrastructure has to be able to mark them as failures and allow for manual interventions, while logging every step.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1b11f48cf95" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/enterprise-data-pipeline-controls-checklist-security-auditability-and-compliance-b1b11f48cf95">Enterprise Data Pipeline Controls Checklist: Security, Auditability, and Compliance</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cloud Sovereignty: Navigating BYOC, Private Deployments, and Public SaaS]]></title>
            <link>https://medium.com/towards-data-engineering/cloud-sovereignty-navigating-byoc-private-deployments-and-public-saas-95a459e6e0e9?source=rss-10235c107f8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/95a459e6e0e9</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[saas]]></category>
            <category><![CDATA[network-security]]></category>
            <category><![CDATA[data-sovereignty]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Nevenka Lukic]]></dc:creator>
            <pubDate>Wed, 15 Apr 2026 03:09:41 GMT</pubDate>
            <atom:updated>2026-04-15T03:09:41.724Z</atom:updated>
            <content:encoded><![CDATA[<p>Data sovereignty is no longer a mere policy concern. In fact, it has become a design constraint.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T_srHvPKUqXakmiUpbT7oA.png" /></figure><p>Public SaaS tools make moving data from a database into a public application remarkably easy. A user simply signs up for an account and links their database, and voila! It begins. There is no configuration required, no need to host servers, and no maintenance to worry about. The simplicity of this approach makes it the most convenient option for small projects and low-risk data.</p><p>However, as a business grows and the data they store becomes more sensitive, hosting that data in someone else’s cloud starts to feel risky. This is typically the time when security and compliance teams get involved. They ask practical questions about access, logging, network paths, and data residency. These are all reasonable concerns, but many tools do not offer clear answers to them.</p><p>At that point, running everything by yourself often seems like the safest solution. And sure, it should solve some of these problems, but it will also create new ones. For instance, self-hosting means owning the infrastructure, managing upgrades, responding to failures, and maintaining on-call support, and most teams underestimate how much ongoing work this requires.</p><p>In an attempt to overcome this issue, deployment models such as BYOC and private deployments have become increasingly common. <a href="https://estuary.dev/">Estuary</a> is built to support these realities, as it lets teams choose how much control they need, without forcing them to operate and maintain everything themselves.</p><p>This article explores the trade-offs between public SaaS, private deployments, BYOC, and self-hosted models and discusses how teams can choose the right level of control without sacrificing speed.</p><h3>Key Takeaways</h3><ul><li>Data sovereignty has become a design requirement.</li><li>Public SaaS prioritizes speed, whereas BYOC and private deployments focus on control.</li><li>More flexibility is achievable across deployment models by separating the control plane from the data plane.</li><li>Private connectivity reduces risk compared to public internet-based data movement.</li><li>The right deployment model depends on regulation, scale, and operational maturity.</li></ul><h3>Choosing a Data Deployment Model: Speed vs. Control</h3><p>Each deployment model solves a different problem, and none of them is perfect. The table below compares their characteristics side by side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fCdb3xyoiqoFB4jcdEDuog.png" /><figcaption>Data Deployment model: Speed vs. Control, Comparison Table</figcaption></figure><p>This diagram further illustrates how these models differ in practice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ADIUvBjUY4n3yF1VHzRRcw.jpeg" /><figcaption>Deployment Model Comparison</figcaption></figure><h3>Public SaaS Deployment</h3><p>Public SaaS is built with speed in mind. The vendor takes care of everything, and you don’t have to worry about servers, storage, or upgrades. This allows for a very fast startup process and provides an easy way to develop your application.</p><p>However, you lose control since your data is running on an infrastructure you don’t own. For many teams, that’s an acceptable compromise. In regulated environments, though, it often isn’t.</p><p>Estuary’s public SaaS option can help teams who need to develop and test new ideas and concepts quickly before they commit to a more formal and controlled solution.</p><h3>Private Deployments (Private SaaS)</h3><p>In a <a href="https://docs.estuary.dev/getting-started/deployment-options/#private-deployment">private deployment</a>, Estuary runs a dedicated environment for you. It’s isolated from other customers, and you don’t have to manage the underlying infrastructure yourself.</p><p>This is a good solution for teams that need better isolation than shared SaaS but don’t have the resources to run their own cloud environment.</p><h3>Bring Your Own Cloud (BYOC)</h3><p><a href="https://docs.estuary.dev/private-byoc/byoc-deployments/">BYOC</a> focuses on where data processing happens and who controls the infrastructure.</p><p>The data plane runs on your cloud account, while Estuary manages the platform, and your data remains within your network and storage. You gain control over regions, access policies, and encryption without having to run everything yourself.</p><p>BYOC is ideal for companies that currently use the cloud to support their production systems and need greater assurances about data ownership.</p><h3>Self-Hosted Deployment</h3><p>A self-hosted deployment gives you complete control. Everything runs in your environment according to your policies. This kind of approach is sometimes necessary for systems with air-gapped or strict residency rules.</p><p>However, your team will be responsible for updates, monitoring, failures, and recovery. And as with any resource-intensive tasks, self-hosting can be difficult to maintain over long periods of time.</p><h3>What a BYOC Deployment Looks Like</h3><p>In modern approaches to cloud sovereignty, <a href="https://estuary.dev/blog/control-plane-vs-data-plane/">control plane is separated from the data plane</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i700kFfNZWUv9eAIqgSdzg.jpeg" /><figcaption>BYOK Deployment Overview</figcaption></figure><p>With Estuary, this split is consistent across public SaaS, private deployments, and BYOC. As a result, teams can apply the same security model regardless of how the system is deployed.</p><h3>Control Plane Responsibilities</h3><p>Estuary handles operations like orchestration, metadata, updates, monitoring, and system health and manages the control plane from its infrastructure.</p><p>The control plane only handles technical metadata, such as configurations, schemas, and system metrics. It tracks the movement of data between systems, but it never accesses or stores it, nor does it interact with record-level data. Sensitive information is kept entirely within the customer’s environment.</p><h3>Data Plane Responsibilities</h3><p>The data plane runs within the customer’s cloud account. It reads from source systems, transforms data, writes to destinations, and stores data in customer-managed storage. Since it uses the customer’s IAM roles and network policies, full control over access remains in the customer’s hands.</p><p>It also makes opting out easy. If you need to restrict or remove vendor access, it can be done at the infrastructure level without touching the data itself. Your data always stays in your environment.</p><p>Estuary also maintains a complete history of changes in customer-owned storage. This allows new destinations to be backfilled from existing data instead of re-reading production databases. As a result, operational load is reduced and the risk is minimized.</p><h3>Network Security and Private Connectivity Options</h3><p>Cloud sovereignty is harder to guarantee when data travels over the public internet. Even with a private data plane, using public IP addresses adds risk and creates management challenges. IP whitelists and firewall rules change, and the assumptions they are built on quickly become outdated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rVJXquWPaykP62UZ0z8mlg.png" /><figcaption>Network Connectivity Options</figcaption></figure><p>Estuary offers multiple options for storing data completely within your cloud provider’s network, including <a href="https://docs.estuary.dev/private-byoc/privatelink/">AWS PrivateLink, Azure Private Link, and GCP Private Service Connect</a>. These approaches limit exposure to the public internet, improve security, and reduce the effort required to manage IP-based access policies. Over time, private connectivity becomes more stable and easier to manage.</p><h3>When Each Deployment Model Makes Sense</h3><p>The right option depends on how sensitive your data is, how quickly you need to move, and what level of operational work your team can reasonably handle.</p><ul><li><strong>Public SaaS:</strong> This is the top choice when speed matters most. It works well for early-stage projects, prototypes, and low-risk data where convenience outweighs strict control. If you want to get something running quickly without having to worry about infrastructure, this is usually the best place to start.</li><li><strong>Private deployments:</strong> If public SaaS deployments don’t meet your needs, but you’re not ready to host your own solutions yet, then private deployments are just the right fit. They are a good option when a team needs isolation and clear boundaries around security but doesn’t want to deal with the day-to-day management of cloud infrastructure.</li><li><strong>Bring Your Own Cloud (BYOC):</strong> When data is part of your production systems and ownership really matters, BYOC is the right solution. Teams choose it when they need control over access, networking, regions, and encryption, or when customer requirements block the use of shared SaaS. It offers strong guarantees while avoiding the cost and complexity of full self-hosting.</li><li><strong>Self-hosted deployment:</strong> Self-hosting is usually reserved for the strictest environments. This includes air-gapped systems, highly regulated workloads, or cases where data cannot leave a specific network under any circumstances. It provides maximum control, but also requires the most time, expertise, and ongoing operational effort.</li></ul><p>In practice, many teams don’t choose just one model forever. Most will start with a public SaaS to move fast, and then move into either a private deployment or BYOC as their data becomes more critical. The goal isn’t always maximum control, but the <strong>right level</strong> of control for where your business is today.</p><h3>Operational Trade-Offs of BYOC and Private Deployments</h3><p>Both BYOC and private deployments offer more control than public SaaS. However, they differ in day-to-day operational demands.</p><h3>Private Deployments</h3><p>Private deployments save time. Estuary handles the environment for you, and you get much stronger isolation than you would with a regular SaaS. It’s the right choice for teams that need to stay in control but don’t want to spend their whole day managing cloud servers.</p><h3>BYOC</h3><p>BYOC, on the other hand, assumes a certain level of cloud expertise. Customers manage networking, IAM, and infrastructure costs, while Estuary handles the data movement platform that runs on top. This model offers strong control but also requires a certain degree of comfort with cloud operations.</p><p>Here are some situations when BYOC might be a good fit for you:</p><ul><li><strong>Production is the priority:</strong> Your data needs high speed and low latency to support real-time applications.</li><li><strong>You work under heavy regulations:</strong> You are in a field like healthcare, finance, or government, where data must be stored within your country or region.</li><li><strong>Your data is massive:</strong> You are moving large amounts of data daily and want to use your existing cloud discounts to lower your overall costs when using SaaS.</li><li><strong>You need full security control:</strong> You would like to manage all aspects of your security, including encryption, zero-trust models, and IAM policies, in order to make sure you never hand over your unencrypted payload to a third party.</li><li><strong>Your team has the cloud skills:</strong> While Estuary manages software updates and scaling, your team is comfortable managing the cloud environment (VPC and networking) it lives in.</li></ul><h3>Why Teams Choose Them</h3><p>Customer requirements are the common reason why teams choose either of these approaches. For example, a company may have everything in place to close an important large enterprise deal, but then they learn that the SaaS solution they are currently using does not satisfy the customer’s expectation on how they would like their data processed. The customer wants to see where the data is being sent, who has access to it, and how it’s isolated. Rather than having to rebuild pipelines or switch to different tools, the team can simply move the data plane into their own cloud to satisfy customer requirements.</p><p>If none of this sounds like your situation, public SaaS is likely the faster way to get started. Alternatively, a private deployment can give you that middle ground of isolation without the extra infrastructure work.</p><h3>Pricing</h3><p>Another thing to note is that pricing is typically divided. You pay your cloud vendor for the storage and compute you use, and you pay Estuary for the platform. Since the data plane stays in your account, you can use the same monitoring and cost tools you already have.</p><h3>Sovereignty as a Spectrum</h3><p>Even though people often see sovereignty as a one-time decision, it actually changes based on your data and the amount of work your team can realistically manage to pull off.</p><p>If you want to move fast, public SaaS is the easiest way to start, as you won’t have to worry about the infrastructure at all. However, if you need your data to be isolated, then self-hosting is the best option. Keep in mind that it requires more effort to run, though.</p><p>BYOC and private deployment fall right in the middle here. They are designed for teams that need to own their data but don’t want to become full-time infrastructure operators.</p><p>Estuary is built to grow with you. You can start fast with their public SaaS, then switch to private data planes or a BYOC setup as your needs grow. Many teams use a mix of both depending on how sensitive their data is.</p><p>Remember, security and sovereignty are non-negotiable with sensitive information. While you’re choosing the best managed service for your needs, you should also keep in mind that your team should be able to trust the setup. Strike that balance, and you can stop worrying about the plumbing, and start focusing on the building part of the process instead.</p><h3>FAQ</h3><h3>What is data sovereignty and how does it differ from data residency?</h3><p>Data sovereignty refers to who controls the data and which laws apply to it. Data residency is a narrower term and focuses only on where the data is physically stored. You can meet the residency requirements and still not meet the sovereignty requirements if a third party controls access or processing.</p><h3>When does a team need BYOC instead of public SaaS or private deployments?</h3><p>Think of those situations when security checks could delay a deal or when customers want clear answers about access, isolation, and ownership; so basically, when the data is used in production systems or when customers need a clearer assurance. And that’s exactly the kind of control BYOC provides. Plus, the best part is, it won’t make your team handle every bit of infrastructure themselves.</p><h3>How does Bring Your Own Cloud (BYOC) help with compliance and regulatory requirements for sensitive data?</h3><p>With BYOC, data processing and storage stay entirely in the customer’s cloud account. This allows teams to apply their own IAM policies, encryption, logging, and regional controls. Since record-level data never leaves their environment, compliance boundaries are much easier to define and explain.</p><h3>How does Estuary support different data sovereignty requirements?</h3><p>Estuary separates the control plane from the data plane across all deployment models. Teams can start with public SaaS, then move to private deployments or BYOC as requirements change, without rebuilding pipelines. This allows organizations to increase control over time while keeping the same platform and workflows.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95a459e6e0e9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/cloud-sovereignty-navigating-byoc-private-deployments-and-public-saas-95a459e6e0e9">Cloud Sovereignty: Navigating BYOC, Private Deployments, and Public SaaS</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Enterprises Need More Than Data Pipelines]]></title>
            <link>https://medium.com/@nenamilkov/why-enterprises-need-more-than-data-pipelines-44485182f7da?source=rss-10235c107f8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/44485182f7da</guid>
            <category><![CDATA[estuary]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-architecture]]></category>
            <dc:creator><![CDATA[Nevenka Lukic]]></dc:creator>
            <pubDate>Mon, 02 Mar 2026 10:27:33 GMT</pubDate>
            <atom:updated>2026-03-02T10:27:33.642Z</atom:updated>
            <content:encoded><![CDATA[<p>I originally wrote a deeper technical breakdown on Estuary’s blog. This version is adapted for Medium readers, with a focus on the core ideas.<br>Full article: <a href="https://estuary.dev/blog/why-enterprises-need-data-movement-layer/">https://estuary.dev/blog/why-enterprises-need-data-movement-layer/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wV3IXlsBRi-4rwjo1ELNNQ.png" /></figure><p>Most data teams don’t start with complexity. They start with a simple goal: move data from one system to another so someone can use it.</p><p>As a data engineer, I’ve built many of these pipelines. Extract, transform, load. It works. Until it doesn’t.</p><p>Over time, new tools are added. New databases appear. Business teams want data in more places, faster. What started as a few clean integrations turns into a web of dependencies that is harder to reason about and even harder to maintain.</p><p>At some point, the issue stops being individual pipelines. It becomes the architecture behind them.</p><h3>The Combinatorial Problem</h3><p>Traditional data architecture is point-to-point. Every source connects directly to every destination.</p><p>If you have N sources and M destinations, you end up with N × M pipelines. Five sources and three destinations already require fifteen separate integrations. Adding one more destination means building five more extraction paths, each reading from production systems.</p><p>Each pipeline has its own schema mappings, retry logic, and monitoring. A single column rename can break multiple jobs at once. If a job fails silently, downstream systems drift out of sync. Recovery often means re-querying production data.</p><p>Operational databases feel this pressure. Multiple jobs read the same tables independently. CPU usage rises. I/O increases. Teams start coordinating extraction windows to avoid affecting user-facing applications.</p><p>None of this happens because one pipeline is poorly written. It happens because the system grows faster than the model behind it.</p><h3>Change Data Capture Helps, But It Is Not Enough</h3><p>Change Data Capture improves how we read from sources. Instead of repeatedly querying tables, CDC reads the database transaction log and emits inserts, updates, and deletes as events. This reduces load and allows near real-time data movement.</p><p>But CDC alone does not solve durability or coordination.</p><p>If changes are streamed but not stored as a reliable history, backfills remain difficult. If a downstream system fails weeks later, the only option may be re-extracting data from production. If logic changes, replay becomes manual and risky.</p><p>Efficient capture is important. Durable infrastructure is more important.</p><h3>The Data Movement Layer Model</h3><p>A data movement layer changes the structure of the system.</p><p>Instead of building pipelines between every pair of systems, you capture changes once, store them as durable history, and let destinations consume independently. Each consumer tracks its own position in the stream.</p><p>Real-time systems can process events immediately. Analytical systems can prioritize completeness and consistency. Both rely on the same authoritative record.</p><p>Adding a new destination does not require touching the source. Backfills come from stored history, not from production queries. Failures are handled through replay rather than manual repair.</p><p>The shift is architectural. Data movement becomes shared infrastructure instead of glue code between tools.</p><h3>Operational Impact</h3><p>In a pipeline-based model, engineers spend time building and fixing integrations. As the number of systems grows, maintenance grows with it.</p><p>In a data movement layer model, sources are configured once. Destinations subscribe. Complexity no longer increases for every new connection. Source load becomes predictable. Schema changes affect fewer components. Historical replay becomes a normal operation, not an emergency.</p><p>Pipelines are still useful for small systems with limited needs.</p><p>But when data powers analytics, operational systems, and AI at the same time, enterprises need infrastructure that treats data movement as durable and stateful by design.</p><h3>Why This Matters to Me as a Data Engineer</h3><p>In real projects, I’ve seen teams spend weeks building integrations that quietly break a few months later. A column changes. A job fails overnight. Someone has to re-run data from production. It becomes normal to live in maintenance mode.</p><p>The problem usually isn’t bad engineering. It’s the architecture. When every new system requires another direct connection, complexity grows faster than the team. You don’t feel it at first, but you pay for it later.</p><p>Good data work should create clarity and speed, not constant repair work. If the foundation is right, engineers can focus on building useful systems instead of chasing broken pipelines.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44485182f7da" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Real-Time Coding Interview Platform With Antigravity and Vibe Coding]]></title>
            <link>https://medium.com/@nenamilkov/building-a-real-time-coding-interview-platform-with-antigravity-and-vibe-coding-d8114341a958?source=rss-10235c107f8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8114341a958</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[developer]]></category>
            <category><![CDATA[vibe-coding]]></category>
            <category><![CDATA[antigravity]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Nevenka Lukic]]></dc:creator>
            <pubDate>Wed, 10 Dec 2025 14:13:39 GMT</pubDate>
            <atom:updated>2025-12-10T14:13:39.259Z</atom:updated>
            <content:encoded><![CDATA[<p>For Homework 2 of the AI Dev Tools Zoomcamp by DataTalksClub, we built an end-to-end application with AI to create a modern platform for online coding interviews. I started by using ChatGPT to generate the initial implementation prompt and then vibe coded the rest with Antigravity.</p><p>The idea was to build a collaborative tool that makes technical interviews smoother by offering a shared coding environment for interviewers and candidates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*54PE60DP9qFowztJxYAdrg.jpeg" /></figure><h3>Key Features</h3><ul><li><strong>Create and share interview links</strong> to invite candidates</li><li><strong>Real-time collaborative editing</strong> across multiple users</li><li><strong>Instant updates</strong> visible to everyone connected</li><li><strong>Syntax highlighting</strong> for multiple programming languages</li><li><strong>Safe code execution in the browser</strong> without touching the server</li></ul><p>To bring everything together, I ended up using React + Vite + Tailwind on the frontend, Express.js and Socket.IO on the backend, and Pyodide (Python compiled to WebAssembly) for safe in-browser code execution.</p><h3>Real-Time Collaboration with Socket.IO</h3><p>To enable live collaborative editing, the backend uses Express.js together with Socket.IO. Each interview session gets its own room, and all users inside that room receive updates instantly whenever someone edits code, switches languages, or runs a program.</p><p>The server listens for events such as code-change, language-change, and code-output, and broadcasts those updates to everyone else in the same room. This keeps the editor fully synchronized without storing any state on the backend.</p><p>A shortened version of the server logic looks like this:</p><pre>io.on(’connection’, (socket) =&gt; {<br><br>    socket.on(’join-room’, (roomId) =&gt; {<br><br>        socket.join(roomId);<br><br>    });<br><br>    socket.on(’code-change’, ({ roomId, code }) =&gt; {<br><br>        socket.to(roomId).emit(’code-update’, code);<br><br>    });<br><br>    socket.on(’language-change’, ({ roomId, language }) =&gt; {<br><br>        socket.to(roomId).emit(’language-update’, language);<br><br>    });<br><br>    socket.on(’code-output’, ({ roomId, output }) =&gt; {<br><br>        socket.to(roomId).emit(’output-update’, output);<br><br>    });<br><br>});</pre><p>The server doesn’t evaluate code or store user data. Its only job is to relay messages between clients. This keeps the backend simple, fast, and safe while still delivering real-time updates across multiple participants.</p><h3>Collaborative Editor and In-Browser Code Execution</h3><p>The shared coding environment is built using a custom editor component in React powered by CodeMirror 6, with syntax highlighting for both JavaScript and Python. Every keystroke is broadcast in real time through Socket.IO so that all connected participants see updates instantly.</p><p>For execution, JavaScript runs directly in the browser inside an isolated eval wrapper, while Python is executed through <strong>Pyodide</strong> (Python compiled to WebAssembly). This keeps execution entirely client-side and avoids running untrusted code on the server.</p><p>A simplified excerpt from the editor component shows how the execution logic works:</p><pre>const runCode = async () =&gt; {<br><br>    setIsRunning(true);<br><br>    setOutput([]);<br><br>    if (language === ‘javascript’) {<br><br>        const log = [];<br><br>        const originalLog = console.log;<br><br>        console.log = (...args) =&gt; log.push(args.join(’ ‘));<br><br>        try {<br><br>            await eval(`(async () =&gt; { ${code} })()`);<br><br>        } catch (e) {<br><br>            log.push(`Error: ${e.message}`);<br><br>        } finally {<br><br>            console.log = originalLog;<br><br>            setOutput(log);<br><br>            socket.emit(’code-output’, { roomId, output: log });<br><br>        }<br><br>    }<br><br>    if (language === ‘python’) {<br><br>        if (!pyodideRef.current) {<br><br>            setOutput([’Pyodide is loading...’]);<br><br>            return;<br><br>        }<br><br>        const pythonOutput = [];<br><br>        pyodideRef.current.setStdout({<br><br>            batched: (msg) =&gt; pythonOutput.push(msg)<br><br>        });<br><br>        try {<br><br>            await pyodideRef.current.runPythonAsync(code);<br><br>        } catch (e) {<br><br>            pythonOutput.push(`Error: ${e.message}`);<br><br>        }<br><br>        setOutput(pythonOutput);<br><br>        socket.emit(’code-output’, { roomId, output: pythonOutput });<br><br>    }<br><br>    setIsRunning(false);<br><br>};</pre><p>This approach ensures that all code runs safely in the browser without involving the backend, while still enabling collaborative execution feedback across all participants.</p><h3>Deployment</h3><p>The app runs as a single Docker container that handles both the frontend and backend. I deployed it on <strong>Render</strong>, which made the process easy.</p><p>Because I’m using the <strong>free tier</strong>, the instance goes to sleep when there’s no traffic. If you open the app and it seems slow at first, that’s normal. Render needs time to spin the service back up. Once it wakes, everything works normally.</p><p>Try it here:</p><p><a href="https://coding-interview-app-gwly.onrender.com/">https://coding-interview-app-gwly.onrender.com</a></p><p>If the first request takes a bit longer (Render says up to ~50 seconds), that just means the container was idle.</p><h3>What I Learned</h3><ul><li>Real-time collaboration isn’t as intimidating as it looks.</li><li>WebAssembly unlocks powerful browser-only execution.</li><li>Integration tests catch issues early even in small projects.</li><li>A single-container deployment keeps things simple and portable.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*GfuL2GwmpAkNtbxW0aMs8w.gif" /></figure><h3>Final Thoughts</h3><p>This homework project turned into a great full-stack exercise: real-time communication, in-browser Python execution, Docker, deployment, and a modern frontend stack. Vibe coding most of the project, made the whole experience even more enjoyable.</p><p>And a quick shoutout to Antigravity: don’t forget to <strong>commit after finishing each part</strong>, <strong>run tests</strong>, and of course, <strong>write those integration tests</strong>, otherwise the whole room might see chaos instead of code!</p><p>If you want to explore the app yourself, here’s the link again:</p><p><a href="https://coding-interview-app-gwly.onrender.com/">https://coding-interview-app-gwly.onrender.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8114341a958" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>