<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by chakshu_salgotra on Medium]]></title>
        <description><![CDATA[Stories by chakshu_salgotra on Medium]]></description>
        <link>https://medium.com/@chakshu-salgotra?source=rss-7afde1b096a7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*SCr35b-OM1h8sSo4q3BmXg.jpeg</url>
            <title>Stories by chakshu_salgotra on Medium</title>
            <link>https://medium.com/@chakshu-salgotra?source=rss-7afde1b096a7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 15:53:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@chakshu-salgotra/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Part XXII — Caching Is Not Free: Persist Levels and the Memory You Lose]]></title>
            <link>https://pub.towardsai.net/part-xxii-caching-is-not-free-persist-levels-and-the-memory-you-lose-57196d16144d?source=rss-7afde1b096a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/57196d16144d</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[spark-internals]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[performance-tuning]]></category>
            <category><![CDATA[memory-management]]></category>
            <dc:creator><![CDATA[chakshu_salgotra]]></dc:creator>
            <pubDate>Sun, 15 Nov 2026 12:01:01 GMT</pubDate>
            <atom:updated>2026-11-15T12:01:01.755Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>The fastest way to make a Spark job slower is to cache everything “just in case.” Here is the engine-level accounting of what `.cache()` actually costs — and the persist discipline that gets the speed back.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_smTSaShvBpbHe4GBMjx_A.png" /></figure><p><a href="https://chakshu-salgotra.medium.com/how-spark-reads-parquet-2333c7fa728c">Part I</a> | <a href="https://chakshu-salgotra.medium.com/spark-shuffle-partitioning-under-the-hood-of-a-wide-transformation-521907cd5f1c">Part II</a><strong> | </strong><a href="https://medium.com/@chakshu-salgotra/spark-catalyst-optimizer-internals-2b762ce0b2e2">Part III</a> <strong>| </strong><a href="https://pub.towardsai.net/jobs-stages-tasks-how-spark-actually-decomposes-your-query-e50bc8ed768e">Part IV</a> <strong>| </strong><a href="https://pub.towardsai.net/how-spark-manages-memory-the-unified-memory-model-spills-and-aqe-f37f21799fc6?sk=c91e30bf8b55b332b5611b7585b0b883">Part V</a> <strong>| </strong><a href="https://pub.towardsai.net/part-vi-the-spark-join-strategies-broadcast-vs-52a2048203e6?sk=aec7100814c2dbac55341898fb4f0686">Part VI<strong> </strong></a><strong>| </strong><a href="https://pub.towardsai.net/part-vii-data-skew-in-spark-why-199-tasks-finish-in-40-seconds-and-one-runs-for-3-hours-7daef3b1de44?sk=2262448f82bf04acda6dea9bfdeca857">Part VII</a> | <a href="https://pub.towardsai.net/part-viii-batch-vs-streaming-in-spark-one-sql-engine-two-execution-contracts-a4e888cb4748?sk=e8f00a5249b4a1ce3aa992f614e76705">Part VIII </a>| <a href="https://pub.towardsai.net/part-ix-the-anatomy-of-a-spark-application-what-the-driver-executors-and-cluster-manager-ffa2bba60dce?sk=d66767c2ca186aaab54860931975741e">Part IX</a> | <a href="https://pub.towardsai.net/part-x-z-ordering-and-data-clustering-explained-why-your-partitioned-table-still-scans-4b109e016c80?sk=dc7cfcb72d4bd523b6b5ca35f79f2b16">Part X </a>| <a href="https://pub.towardsai.net/part-xi-rdd-vs-dataframe-vs-dataset-why-your-type-safe-refactor-made-spark-3x-slower-99bdba6f7eaa?source=friends_link&amp;sk=3cef58b143ef9d20917ac93e62731ecc">Part XI</a> | <a href="https://chakshu-salgotra.medium.com/da093c04b481?source=friends_link&amp;sk=1f9b68eee7c3c07c233d280b625b5fab">Part XII </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xiii-internal-vs-external-tables-the-drop-table-that-deleted-40tb-4fffae7f740c">Part XIII </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xiv-broadcast-variables-the-driver-bottleneck-nobody-profiles-87189bb691ed">Part XIV</a> | <a href="https://medium.com/towards-artificial-intelligence/part-xv-adaptive-query-execution-what-aqe-actually-rewrites-at-runtime-f010f22ad3fa">Part XV</a> | <a href="https://medium.com/towards-artificial-intelligence/part-xvi-whole-stage-codegen-the-tungsten-trick-behind-2x-speedups-6e0011a07944">Part XVI </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xvii-checkpointing-vs-lineage-the-fault-tolerance-trade-you-misconfigure-f5b1a5892d68">Part XVII</a> | <a href="https://medium.com/towards-artificial-intelligence/part-xviii-the-small-files-tax-compaction-strategies-that-dont-stall-593e9a3abe2c">Part XVIII</a><strong> </strong>| <a href="https://medium.com/towards-artificial-intelligence/e7b1a25cd7a1">Part XIX</a> | <a href="https://medium.com/towards-artificial-intelligence/part-xx-delta-lakes-transaction-log-time-travel-vacuum-and-the-files-that-never-die-fc60f6de0360"><strong>Part XX</strong></a>| <a href="https://medium.com/towards-artificial-intelligence/part-xxi-streaming-state-watermarks-state-stores-and-the-146gb-checkpoint-92f5687d60ac">Part XXI</a>| <strong>Part XXII</strong></p><h3>The Night the “Optimization” Doubled Our Runtime</h3><p>At 1:42 AM on a Tuesday, the revenue-aggregation pipeline breached its SLA for the first time in eleven months. The job was unglamorous but load-bearing: 3.4 TB of raw order events on S3, a nightly rollup feeding every finance dashboard in the company, running on 24 r5.2xlarge nodes with a steady-state runtime of 41 minutes at roughly $63 per run. It had been boring in the best possible way.</p><p>The week before, a well-meaning engineer had “optimized” it. The diff was almost comically simple: .cache() after every major transformation — nine cached DataFrames, one per stage, on the theory that nothing should ever be computed twice. Code review waved it through. Caching is free performance, right?</p><p>The first run after the change took 2 hours and 17 minutes. Two stages failed outright when YARN killed executors for exceeding their memory limits. The Spark UI told the real story: average GC pause per task had gone from 0.3 seconds to 11 seconds, the BlockManager had evicted 78% of cached blocks before any downstream stage read them, and the driver was spending more time serializing cache bookkeeping than scheduling work. We had spent the entire executor memory budget caching data nobody would ever read twice, and the garbage collector was charging us interest by the minute. Cost per run: $212. SLA: breached. Pager: mine.</p><p>The rollback took four minutes. The lesson took longer to sink in: .cache() is not a performance hint. It is a memory allocation with a garbage-collection bill attached, and the engine will happily let you bankrupt yourself.</p><p>So: what does a cached DataFrame actually cost, where does it live, and when is persisting data the right call versus pure overhead?</p><h3>The Mechanic Underneath: The BlockManager Rents You Memory by the Byte</h3><p>Every cached partition lives in the <strong>BlockManager</strong>, Spark’s per-executor storage service — the same service that holds shuffle blocks and broadcast variables. On the JVM heap, memory is split by the <strong>UnifiedMemoryManager</strong>: after a 300 MB reserved region, spark.memory.fraction (default 0.6) of the heap becomes one unified region shared by execution (shuffles, joins, sorts) and storage (cache). Inside that region, spark.memory.storageFraction (default 0.5) draws a soft line — storage is guaranteed half the unified region, and it may <em>borrow</em> the rest, but execution memory can <strong>evict cached blocks</strong> whenever it needs the space.</p><p>That last clause is the whole game. Your cache is not a reservation; it is a tenant with no lease. A big shuffle in a later stage can — and will — evict your carefully cached DataFrame block by block, LRU-first, with no warning and no ceremony. (Part V of this series covered this memory split in depth; here is where it bites.)</p><p>And what you store matters more than where. MEMORY_ONLY, the default behind .cache(), stores <strong>deserialized Java objects</strong> — each row as a graph of objects on the heap. A 4-byte integer becomes a 16-byte Integer plus object header; a modest row can inflate 2–5x versus its serialized size. You are not caching 3 GB of data. You are caching 3 GB of data wearing a 12 GB object costume, and the garbage collector has to walk that costume on every major GC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H-_82NeMZDDP-vi2tqUYKw.png" /></figure><h3>MEMORY_ONLY Is a Bet You Usually Lose</h3><p>The default storage level is the worst one for most production workloads, which is a remarkable design choice until you remember it was optimized for demos. Deserialized objects give you the fastest possible read — no deserialization cost on access — at the price of maximum footprint and maximum GC pressure. On heaps above ~32 GB the GC pauses on a fully cached, object-heavy RDD can exceed the time you saved by not recomputing.</p><p>The rule is blunt: use MEMORY_ONLY only when the cached dataset is small, reused many times, and the cluster has memory to waste. That combination is rarer than the tutorials suggest. In our incident, nine DataFrames cached with MEMORY_ONLY consumed a combined 41 GB of heap across the cluster to hold roughly 14 GB of actual data — and the GC was spending 11 seconds per task just traversing it.</p><h3>Serialization Trades CPU for Survival: Why _SER Exists</h3><p>Appending _SER to a storage level stores <strong>serialized bytes</strong> instead of objects — one compact byte array per partition instead of a thicket of objects. The trade is explicit: you pay deserialization CPU every time you read the cache, and you gain 2–5x smaller footprints plus dramatically lighter GC, because the collector sees a handful of byte arrays instead of millions of objects.</p><p>Two details most teams miss. First, the serializer matters: the default Java serializer is slow and bloated; <strong>Kryo</strong> (spark.serializer=org.apache.spark.serializer.KryoSerializer) typically cuts serialized size by another 2–10x and is the difference between _SER being a win and a wash. Second, serialized caching composes well with whole-stage codegen (Part XVI&#39;s topic): generated code operating on Tungsten binary rows deserializes less and copies less, so the CPU tax of _SER shrinks exactly where the engine is fastest.</p><h3>Eviction Is Not Failure, It’s Policy: LRU and the Unpersist You Forgot</h3><p>When execution memory needs space, the BlockManager evicts cached blocks <strong>least-recently-used first</strong>. This is not an error — it is the documented policy — but it turns sloppy caching into a special kind of waste: you pay the full cost of writing blocks to memory (and to disk, for _AND_DISK levels), and then the engine throws them away before the second read you cached them for.</p><p>The discipline has two halves. <strong>Cache lazily and deliberately</strong>: call .persist() with an explicit level, then force materialization with an action <em>on your terms</em> — a .count() right after persisting — so the cache is populated once, up front, instead of trickling in mid-stage while downstream tasks are already waiting. <strong>Unpersist aggressively</strong>: the moment the last consumer finishes, release the blocks. A cached DataFrame that outlives its last reader is not a cache; it is a memory leak with a Spark UI entry.</p><h3>If You Read It Once, Caching Is Pure Overhead</h3><p>Here is the decision rule, and it fits in one sentence: <strong>persist a dataset only if it is read more than once AND expensive to recompute.</strong> Both conditions. A DataFrame consumed by a single downstream stage gains nothing from caching — you pay serialization, memory, and GC, and you read it exactly as many times as you would have without the cache. A cheap-to-recompute frame (a filter over an already-cached parent) gains nothing either; recomputation is faster than the cache bookkeeping.</p><p>In the incident diff, five of the nine cached DataFrames had exactly one consumer. They were pure overhead — memory spent to avoid recomputation that was never going to happen.</p><h3>Off-Heap Changes the Game but Not the Rules</h3><p>spark.memory.offHeap.enabled moves storage outside the JVM heap into Tungsten&#39;s binary format, which largely sidesteps GC — the collector never sees off-heap bytes. It is the right answer when GC pauses dominate, but it does not change the accounting: off-heap memory is still finite, still shared with execution, and still evictable. It converts a GC problem into a capacity problem. The decision rule above still applies; off-heap just makes the &quot;expensive to recompute&quot; side of the ledger cheaper to satisfy.</p><h3>Executor Memory, Annotated</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DPPybp6w4W62ByQYx0MaDw.png" /></figure><h3>The Fix, and What It Cost</h3><pre># Naive: cache everything &quot;just in case&quot; — 9 DataFrames, MEMORY_ONLY default.<br># raw = spark.read.parquet(&quot;s3://lake/orders/&quot;).cache()<br># cleaned = raw.filter(...).cache()<br># enriched = cleaned.join(...).cache()<br># ... six more ...<br># Result: 41 min -&gt; 2h17m, 11s GC pauses, 78% of blocks evicted unread.</pre><pre># Optimized: persist ONE frame — read twice, expensive to build — serialized,<br># materialized up front, released the moment its last consumer finishes.<br>from pyspark import StorageLevel</pre><pre>base = (<br>    spark.read.parquet(&quot;s3://lake/orders/&quot;)<br>    .filter(&quot;order_date &gt;= current_date() - interval 90 days&quot;)<br>    .repartition(512, &quot;order_date&quot;)  # stable layout: cache once, reuse the partitioning<br>)<br># MEMORY_AND_DISK_SER: serialized bytes survive memory pressure by spilling;<br># Kryo keeps the byte footprint small (set spark.serializer accordingly).<br>base.persist(StorageLevel.MEMORY_AND_DISK_SER)<br>base.count()  # materialize NOW, on your terms — not trickling in mid-stage</pre><pre>agg_a = base.groupBy(&quot;region&quot;).agg(...)     # consumer 1<br>agg_b = base.join(dims, &quot;sku&quot;).groupBy(...) # consumer 2</pre><pre>agg_a.write.mode(&quot;overwrite&quot;).parquet(&quot;s3://lake/revenue_by_region/&quot;)<br>agg_b.write.mode(&quot;overwrite&quot;).parquet(&quot;s3://lake/revenue_by_sku/&quot;)<br>base.unpersist()  # last consumer done: give the memory back immediately</pre><p>The numbers after the fix, same cluster, same data: wall-clock <strong>2h17m → 38 minutes</strong> (three minutes <em>faster</em> than the pre-cache baseline, because the stable repartitioning also fixed a skew-adjacent shuffle). Cached footprint: 14 GB of objects → <strong>3.1 GB of serialized bytes</strong>. Average GC pause: 11s → 0.4s. Evicted-before-read blocks: 78% → 0%. Executor losses: two per run → zero. Cost per run: <strong>$212 → $58</strong> — and the 3 AM pager went back to sleep.</p><h3>The Catch: Persist Is Not Durability, and DISK_ONLY Is a Trap</h3><p>Three honest counterweights. First, <strong>persist is not durability</strong>: cached blocks die with the application. If you need data to survive a driver restart, that is checkpoint() (Part XVII&#39;s territory), which writes to reliable storage at the cost of breaking lineage. Persist is a within-application optimization, nothing more.</p><p>Second, DISK_ONLY looks like a safety net and behaves like a tar pit: every read pays full disk I/O plus deserialization, and on cloud object stores the &quot;disk&quot; is often network-attached anyway. It is occasionally right for enormous, rarely-read frames — but measure it, because recomputation is frequently faster.</p><p>Third, <strong>caching interacts with skew</strong> (Part VII’s playbook): persisting a skewed frame bakes the skew into every downstream consumer. Fix the partitioning first — as the repartition(512, &quot;order_date&quot;) above does — then cache the balanced result. A cache of a bad layout is just a fast way to repeat a bad layout.</p><h3>Takeaways</h3><ul><li><strong>.cache() is MEMORY_ONLY:</strong> deserialized Java objects, 2–5x inflation, maximum GC pressure. It is the default, not the recommendation.</li><li><strong>The decision rule is two conditions:</strong> persist only if the dataset is read more than once AND expensive to recompute. One consumer means pure overhead.</li><li><strong>Serialize to survive:</strong> _SER levels trade a little CPU for 2–5x smaller footprints and GC the collector can actually traverse — pair with Kryo.</li><li><strong>Your cache has no lease:</strong> execution memory evicts storage LRU-first. Cache lazily, materialize with an action on your terms, and unpersist() the moment the last consumer finishes.</li><li><strong>Off-heap moves the problem:</strong> it kills GC pauses but not capacity limits or eviction. The accounting still applies.</li><li><strong>Fix the layout before you cache it:</strong> persisting a skewed or badly partitioned frame just makes every downstream stage repeat the mistake faster.</li></ul><p>Cache is a loan from the executor, and the garbage collector collects the interest.</p><p>If this saved you a 3 AM pager incident, <a href="https://medium.com/@chakshu-salgotra">follow me on Medium</a> for Twice a week for the next 3 Months of Advanced Data Engineering &amp; AI. Let’s connect on <a href="https://www.linkedin.com/in/chakshu-salgotra/">LinkedIn</a> and <a href="https://x.com/chakshusalgotra">X</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57196d16144d" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/part-xxii-caching-is-not-free-persist-levels-and-the-memory-you-lose-57196d16144d">Part XXII — Caching Is Not Free: Persist Levels and the Memory You Lose</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part XXI — Streaming State: Watermarks, State Stores, and the 146GB Checkpoint]]></title>
            <link>https://pub.towardsai.net/part-xxi-streaming-state-watermarks-state-stores-and-the-146gb-checkpoint-92f5687d60ac?source=rss-7afde1b096a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/92f5687d60ac</guid>
            <category><![CDATA[ai-watermarking]]></category>
            <category><![CDATA[state-management]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[structured-streaming]]></category>
            <category><![CDATA[spark-streaming]]></category>
            <dc:creator><![CDATA[chakshu_salgotra]]></dc:creator>
            <pubDate>Sun, 08 Nov 2026 17:01:01 GMT</pubDate>
            <atom:updated>2026-11-08T17:01:01.908Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Every Structured Streaming query is a stateful program wearing a batch costume. The watermark is the only thing standing between your state store and unbounded growth — and when it is misconfigured, the checkpoint doesn’t save you. It becomes the incident.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NI49DIHj9oPKzPfuneOdzw.png" /></figure><p><a href="https://chakshu-salgotra.medium.com/how-spark-reads-parquet-2333c7fa728c">Part I</a> | <a href="https://chakshu-salgotra.medium.com/spark-shuffle-partitioning-under-the-hood-of-a-wide-transformation-521907cd5f1c">Part II</a><strong> | </strong><a href="https://medium.com/@chakshu-salgotra/spark-catalyst-optimizer-internals-2b762ce0b2e2">Part III</a> <strong>| </strong><a href="https://pub.towardsai.net/jobs-stages-tasks-how-spark-actually-decomposes-your-query-e50bc8ed768e">Part IV</a> <strong>| </strong><a href="https://pub.towardsai.net/how-spark-manages-memory-the-unified-memory-model-spills-and-aqe-f37f21799fc6?sk=c91e30bf8b55b332b5611b7585b0b883">Part V</a> <strong>| </strong><a href="https://pub.towardsai.net/part-vi-the-spark-join-strategies-broadcast-vs-52a2048203e6?sk=aec7100814c2dbac55341898fb4f0686">Part VI<strong> </strong></a><strong>| </strong><a href="https://pub.towardsai.net/part-vii-data-skew-in-spark-why-199-tasks-finish-in-40-seconds-and-one-runs-for-3-hours-7daef3b1de44?sk=2262448f82bf04acda6dea9bfdeca857">Part VII</a> | <a href="https://pub.towardsai.net/part-viii-batch-vs-streaming-in-spark-one-sql-engine-two-execution-contracts-a4e888cb4748?sk=e8f00a5249b4a1ce3aa992f614e76705">Part VIII </a>| <a href="https://pub.towardsai.net/part-ix-the-anatomy-of-a-spark-application-what-the-driver-executors-and-cluster-manager-ffa2bba60dce?sk=d66767c2ca186aaab54860931975741e">Part IX</a> | <a href="https://pub.towardsai.net/part-x-z-ordering-and-data-clustering-explained-why-your-partitioned-table-still-scans-4b109e016c80?sk=dc7cfcb72d4bd523b6b5ca35f79f2b16">Part X </a>| <a href="https://pub.towardsai.net/part-xi-rdd-vs-dataframe-vs-dataset-why-your-type-safe-refactor-made-spark-3x-slower-99bdba6f7eaa?source=friends_link&amp;sk=3cef58b143ef9d20917ac93e62731ecc">Part XI</a> | <a href="https://chakshu-salgotra.medium.com/da093c04b481?source=friends_link&amp;sk=1f9b68eee7c3c07c233d280b625b5fab">Part XII </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xiii-internal-vs-external-tables-the-drop-table-that-deleted-40tb-4fffae7f740c">Part XIII </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xiv-broadcast-variables-the-driver-bottleneck-nobody-profiles-87189bb691ed">Part XIV</a> | <a href="https://medium.com/towards-artificial-intelligence/part-xv-adaptive-query-execution-what-aqe-actually-rewrites-at-runtime-f010f22ad3fa">Part XV</a> | <a href="https://medium.com/towards-artificial-intelligence/part-xvi-whole-stage-codegen-the-tungsten-trick-behind-2x-speedups-6e0011a07944">Part XVI </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xvii-checkpointing-vs-lineage-the-fault-tolerance-trade-you-misconfigure-f5b1a5892d68">Part XVII</a> | <a href="https://medium.com/towards-artificial-intelligence/part-xviii-the-small-files-tax-compaction-strategies-that-dont-stall-593e9a3abe2c">Part XVIII</a><strong> </strong>| <a href="https://medium.com/towards-artificial-intelligence/e7b1a25cd7a1">Part XIX</a> | <a href="https://medium.com/towards-artificial-intelligence/part-xx-delta-lakes-transaction-log-time-travel-vacuum-and-the-files-that-never-die-fc60f6de0360"><strong>Part XX</strong></a>| <strong>Part XXI</strong></p><h3>The 146GB Checkpoint That Ate the Fraud Pipeline</h3><p>The page came at 2:47 AM, and the symptom made no sense at first: fraud.dedup_orders — a Structured Streaming query whose entire job is deduplicating order events — had fallen 93 minutes behind. Not failed. Not erroring. Just… walking, while 38,000 events per second piled up behind it. The consumer lag graph looked like a ski slope.</p><p>The query was simple enough to be boring. Kafka in, deduplicate on order_id, Delta Lake out. Someone, at some point, had set the watermark to 7 days — &quot;to be safe,&quot; the commit message said, &quot;so we never miss a late retry.&quot; The upstream payment service does retry failed webhooks for up to 20 minutes. Seven days of safety margin for a 20-minute retry SLA.</p><p>Here is what seven days of watermark on a 38k-events-per-second firehose means arithmetically: every unique order_id seen in the last seven days must be kept in state, because a duplicate could theoretically still arrive. That&#39;s roughly 1.4 billion live keys. The state store — RocksDB, thankfully, or we&#39;d have died much earlier — had swollen to <strong>146GB across 200 partitions</strong>. Each two-minute micro-batch spent 41 seconds just writing state changelog deltas to the checkpoint directory, and that number was climbing every day.</p><p>Then came the deploy. A routine config change, a routine restart. Restarting a Structured Streaming query means reloading the latest state snapshot and replaying the changelog since that snapshot — and our snapshot-plus-changelog replay took <strong>38 minutes</strong>. When the executors finally came back, two of them OOM’d during state restoration, the query restarted again, and the lag went from 93 minutes to “we’ll tell you when it’s over.”</p><p>Part VIII of this series covered the batch-vs-streaming execution contract — one SQL engine, two very different promises. This is the chapter where the streaming side of that contract sends its bill: <strong>in Structured Streaming, state is not a feature of the query. State is the query.</strong> Everything else is plumbing.</p><p>So: where does streaming state actually live, what is supposed to bound it, and what does “exactly-once” really cost when the bound is wrong?</p><h3>The Mechanic Underneath: A Streaming Query Is a Versioned Key-Value Store With a SQL Frontend</h3><p>Strip away the DataFrame API and this is what’s running: for every stateful operator in your query, Spark maintains a <strong>versioned key-value store</strong>, partitioned the same way as your shuffle partitions, versioned by micro-batch ID. Each micro-batch reads the previous version, applies its updates, and writes a new version. The SQL you wrote — the aggregation, the dedup, the join — is a polite fiction over get, put, and delete on that store.</p><p>Three components implement it:</p><ul><li><strong>The StateStoreProvider.</strong> The default, HDFSBackedStateStoreProvider, keeps the working set in JVM heap and spills deltas to the checkpoint directory. The alternative, RocksDBStateStoreProvider (via spark.sql.streaming.stateStore.providerClass), keeps state in an off-heap LSM tree on local disk. Same API, radically different ceiling: heap versus disk.</li><li><strong>The watermark.</strong> Defined as <em>max event time observed so far minus allowed lateness</em>. It is the query’s statement about the future — “I do not expect data older than this” — and the engine uses it as a license to evict state it believes can never be needed again.</li><li><strong>The checkpoint.</strong> Offsets, state snapshots, state changelog deltas, and the commit log, all in the checkpoint directory. It is what makes restart possible and exactly-once semantics achievable.</li></ul><p>The operators that keep state: streaming aggregations, dropDuplicates, stream-stream joins, and flatMapGroupsWithState/mapGroupsWithState. If your query has none of these, congratulations — you have no state problem. Our dedup query had exactly one, and one was enough.</p><h3>Watermarks Don’t Filter Data; They License Eviction</h3><p>This is the most misunderstood line in the Structured Streaming documentation, so let’s be blunt: <strong>a watermark does not filter your data. It authorizes the engine to forget.</strong> Data older than the watermark isn’t “processed late” — for stateful operators, it’s dropped silently (or, more precisely, the state that would have matched it is already gone).</p><p>The watermark only moves forward, and it’s computed as the <em>minimum</em> across partitions — one straggler partition with old event times pins the watermark for the entire query. And critically: <strong>without a watermark, stateful operators never evict anything.</strong> A dropDuplicates(&quot;order_id&quot;) with no watermark is an infinite key set with extra steps. Our 7-day watermark was technically an eviction policy — it evicted keys after seven days — but on a 38k/sec firehose, &quot;evict after seven days&quot; is barely distinguishable from &quot;never.&quot;</p><p>The sizing rule is not complicated, which is what makes getting it wrong so embarrassing: <strong>the watermark’s allowed lateness should match the upstream lateness SLA, not your anxiety.</strong> Our upstream retried for 20 minutes. The correct watermark was 20 minutes (plus a small buffer), which would have kept roughly 46 million keys instead of 1.4 billion. The 7-day setting wasn’t safety; it was a 500x state multiplier purchased with someone else’s disk.</p><p>Stream-stream joins deserve a special warning here, because they bite differently: a stream-stream join requires watermarks on <strong>both</strong> sides, and each side’s state is evicted by its own watermark. Leave one side unwatermarked and that side’s state grows forever while the other side stays trim — the asymmetry makes it look like a “mystery” growth problem when it’s really a missing three-line config.</p><h3>RocksDB vs In-Memory: Where the 146GB Actually Lives</h3><p>Had we been on the default HDFSBackedStateStoreProvider, this incident would have been an OOM at 3 AM instead of a slowdown — the entire 146GB working set would have needed JVM heap, and no executor heap is 146GB. RocksDB moved the ceiling from heap to local disk, which is why the query degraded gracefully instead of dying. Give the LSM tree its due: it bought us months.</p><p>But “graceful degradation” still degrades. Here’s where the 146GB bit us, component by component:</p><ul><li><strong>Checkpoint write amplification.</strong> Every micro-batch writes changelog deltas for every changed key to the checkpoint directory, plus periodic full snapshots. At 1.4B keys with steady churn, our 2-minute micro-batches spent 41 seconds on state I/O alone — over a third of the batch interval burned before any business logic ran.</li><li><strong>Restart replay.</strong> A restart loads the latest snapshot and replays every changelog delta since it. Thirty-eight minutes of replay before the query processed a single new event — during which lag kept accumulating.</li><li><strong>Compaction debt.</strong> RocksDB’s LSM tree needs periodic compaction, and spark.sql.streaming.stateStore.rocksdb.compactOnCommit exists because without it, read amplification during state lookups grows with every uncompacted level. We had it at the default, on executors whose local disks were already 80% full of state.</li></ul><p>The state store is not a black box you can ignore once you’ve picked RocksDB. It’s a database you’re now operating — with its own storage engine, its own compaction, and its own failure modes. The fraud query’s state directory had become the largest “database” in our infrastructure, and nobody was monitoring it as one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WCG6nHu6e2xymHP7QKvhIg.png" /></figure><p>On restart, the arrow runs backward: the query reloads the checkpoint, replays deltas onto the state store, and only then resumes. That backward arrow is the 38 minutes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E7KIyoHeIAtpwNoJm_atgw.png" /></figure><h3>Exactly-Once Is an End-to-End Property, and the State Store Sends the Bill</h3><p>Let’s settle what “exactly-once” actually means in Structured Streaming, because the phrase gets thrown around like a guarantee and it’s really a contract with four signatories:</p><ol><li><strong>A replayable source</strong> (Kafka offsets can be re-read; a socket stream cannot).</li><li><strong>Deterministic transformations</strong> (your business logic, Spark’s responsibility once you behave).</li><li><strong>An idempotent sink</strong> (writing to Delta with foreachBatch + MERGE on the batch ID, so a retried batch doesn&#39;t double-write).</li><li><strong>A durable checkpoint</strong> (offsets + state + commit log, so a restart resumes from a consistent point).</li></ol><p>Spark delivers 2 and the machinery for 4. You supply 1 and 3. Miss any one and you have at-least-once with extra steps — which is exactly what most “exactly-once” pipelines quietly are.</p><p>And here is the cost nobody puts in the design doc: exactly-once is purchased with <strong>write amplification and restart latency, both proportional to state size.</strong> The changelog deltas, the snapshots, the 38-minute replay — that’s the premium on the exactly-once insurance policy, and it’s denominated in state bytes. Our 7-day watermark didn’t just waste disk; it multiplied the cost of every guarantee the query made. Shrink the state and every downstream cost — checkpoint I/O, restart time, compaction debt — shrinks with it. There is no cheaper exactly-once optimization than needing less state.</p><p>One more contractual fine print: <strong>the checkpoint directory is part of the query’s semantics.</strong> Change the state schema — the key columns, the aggregation, the watermark column — and you cannot resume from the old checkpoint; you need a new checkpoint location, which means reprocessing from the source’s earliest retained offset. We learned to treat checkpoint paths as immutable infrastructure, versioned alongside the query code.</p><h3>The Fix, and What It Cost</h3><p>The naive version is the one that paged us. The optimized version starts from the upstream SLA and treats the state store as the database it is:</p><pre># Naive: &quot;be safe&quot; with a 7-day watermark on the default in-memory state store.<br># (events<br>#   .withWatermark(&quot;event_ts&quot;, &quot;7 days&quot;)   # 500x more state than the 20-min retry SLA needs<br>#   .dropDuplicates(&quot;order_id&quot;))           # default HDFSBackedStateStoreProvider: heap is the ceiling</pre><pre># Optimized: watermark sized to the upstream retry SLA, state moved off-heap.<br>spark.conf.set(<br>    &quot;spark.sql.streaming.stateStore.providerClass&quot;,<br>    &quot;org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider&quot;,<br>)  # off-heap LSM: the ceiling moves from executor heap to local disk<br>spark.conf.set(<br>    &quot;spark.sql.streaming.stateStore.rocksdb.compactOnCommit&quot;, &quot;true&quot;<br>)  # pay compaction incrementally instead of as read-amplification debt</pre><pre>deduped = (<br>    events<br>    .withWatermark(&quot;event_ts&quot;, &quot;20 minutes&quot;)<br>    # upstream retries for 20 min; anything later is dropped BY DESIGN, not by accident<br>    .dropDuplicates(&quot;order_id&quot;)<br>)</pre><p>Two operational companions, both non-optional:</p><ul><li><strong>State parallelism matched to key cardinality.</strong> We rebalanced to 400 state partitions so no single RocksDB instance held a disproportionate key range — Part VII’s skew playbook applies to state stores too, because a state store is just a partitioned key space.</li><li><strong>State monitoring as a first-class metric.</strong> We added a StreamingQueryListener scraping stateOperators metrics from each progress event — numRowsTotal, memoryUsedBytes — into our dashboards with an alert at 2x the post-fix baseline. The state store is now the largest database we operate; it gets database-grade observability.</li></ul><p>The quantified results, two weeks later:</p><ul><li><strong>State size: 146GB → 11GB.</strong> 1.4B live keys → ~46M, matching the 20-minute watermark window.</li><li><strong>Processing lag: 93 minutes → 35 seconds</strong> at the same 38k events/sec input.</li><li><strong>Checkpoint write per micro-batch: 41s → 3s.</strong> State I/O stopped being the bottleneck.</li><li><strong>Restart time: 38 minutes → 5 minutes.</strong> The backward arrow got short.</li></ul><h3>The Catch: A Watermark Is a Heuristic, and Heuristics Drop Data</h3><p>The honest counterweight, because every number above came with a trade:</p><ul><li><strong>Shrinking the watermark changes dedup semantics.</strong> Going from 7 days to 20 minutes means a duplicate arriving 21 minutes late is now processed as a new order. That is a <em>business</em> decision wearing an engineering costume — we got written sign-off from the fraud team before deploying it, and you should too.</li><li><strong>Late data dies silently.</strong> There is no dead-letter queue, no warning, no metric that says “I dropped 12,000 late events this hour” unless you build one. If late data matters to you, add a side-output or a counter on the watermark gap before you tighten anything.</li><li><strong>State schema changes need a new checkpoint.</strong> New key, new aggregation, new watermark column — the old checkpoint is unreadable to the new query plan. Version your checkpoint paths with your query code and plan the reprocessing window.</li><li><strong>RocksDB is not magic disk.</strong> It needs fast local SSDs, headroom for compaction (we keep 40% free on state volumes now), and memory budgets of its own (spark.sql.streaming.stateStore.rocksdb.*). On spinning disks or cramped volumes, it compacts forever and your &quot;fix&quot; becomes a new incident.</li><li><strong>Exactly-once without an idempotent sink is fiction.</strong> If your sink can’t tolerate a retried batch, no watermark tuning in the world gives you exactly-once. The guarantee is end-to-end or it isn’t.</li></ul><h3>Takeaways</h3><ul><li><strong>Every stateful operator is a key-value store:</strong> aggregations, dedup, stream-stream joins, and flatMapGroupsWithState all persist versioned state per partition. If you can&#39;t name your query&#39;s stateful operators, you can&#39;t reason about its failure modes.</li><li><strong>No watermark, no eviction:</strong> state without a watermark grows until something breaks — usually the checkpoint, the restart, or the heap, in that order.</li><li><strong>Size the watermark to the SLA, not your anxiety:</strong> allowed lateness should mirror the upstream lateness contract. Every extra hour is a state multiplier on your highest-cardinality key space.</li><li><strong>RocksDB moves the ceiling from heap to disk:</strong> the default in-memory provider makes executor heap your state limit; RocksDB trades that for local-disk capacity and compaction ops. Pick deliberately.</li><li><strong>The checkpoint is part of the query’s contract:</strong> offsets, state snapshots, and the commit log are load-bearing. Version checkpoint paths with your code and never hand-edit them.</li><li><strong>Exactly-once is end-to-end and priced in state bytes:</strong> replayable source, deterministic logic, idempotent sink, durable checkpoint — and every byte of state multiplies the write amplification and restart cost of all four.</li><li><strong>Monitor state like production data:</strong> numRowsTotal and memoryUsedBytes per state operator, alerted at 2x baseline. The state store is a database you&#39;re operating whether you admit it or not.</li></ul><p>In streaming, state is the query. Everything else is just the plumbing that feeds it.</p><p>If this saved you a 3 AM pager incident, <a href="https://medium.com/@chakshu-salgotra">follow me on Medium</a> for Twice a week for the next 3 Months of Advanced Data Engineering &amp; AI. Let’s connect on <a href="https://www.linkedin.com/in/chakshu-salgotra/">LinkedIn</a> and <a href="https://x.com/chakshusalgotra">X</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92f5687d60ac" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/part-xxi-streaming-state-watermarks-state-stores-and-the-146gb-checkpoint-92f5687d60ac">Part XXI — Streaming State: Watermarks, State Stores, and the 146GB Checkpoint</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part XX — Delta Lake’s Transaction Log: Time Travel, VACUUM, and the Files That Never Die]]></title>
            <link>https://pub.towardsai.net/part-xx-delta-lakes-transaction-log-time-travel-vacuum-and-the-files-that-never-die-fc60f6de0360?source=rss-7afde1b096a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc60f6de0360</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[data-storage]]></category>
            <dc:creator><![CDATA[chakshu_salgotra]]></dc:creator>
            <pubDate>Mon, 02 Nov 2026 16:01:03 GMT</pubDate>
            <atom:updated>2026-11-02T16:01:03.418Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Your table never deletes anything. The Delta log is an append-only ledger, and every file you “removed” is still sitting on S3 until the log’s retention rules, the checkpoint cadence, and your VACUUM discipline all agree it can die. Misunderstand any one of the three and you get a $41,000-a-month storage bill — or worse, a restore that points at files which no longer exist.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mcPzPkd31wb4wJzmkBKk_Q.png" /></figure><p><a href="https://chakshu-salgotra.medium.com/how-spark-reads-parquet-2333c7fa728c">Part I</a> | <a href="https://chakshu-salgotra.medium.com/spark-shuffle-partitioning-under-the-hood-of-a-wide-transformation-521907cd5f1c">Part II</a><strong> | </strong><a href="https://medium.com/@chakshu-salgotra/spark-catalyst-optimizer-internals-2b762ce0b2e2">Part III</a> <strong>| </strong><a href="https://pub.towardsai.net/jobs-stages-tasks-how-spark-actually-decomposes-your-query-e50bc8ed768e">Part IV</a> <strong>| </strong><a href="https://pub.towardsai.net/how-spark-manages-memory-the-unified-memory-model-spills-and-aqe-f37f21799fc6?sk=c91e30bf8b55b332b5611b7585b0b883">Part V</a> <strong>| </strong><a href="https://pub.towardsai.net/part-vi-the-spark-join-strategies-broadcast-vs-52a2048203e6?sk=aec7100814c2dbac55341898fb4f0686">Part VI<strong> </strong></a><strong>| </strong><a href="https://pub.towardsai.net/part-vii-data-skew-in-spark-why-199-tasks-finish-in-40-seconds-and-one-runs-for-3-hours-7daef3b1de44?sk=2262448f82bf04acda6dea9bfdeca857">Part VII</a> | <a href="https://pub.towardsai.net/part-viii-batch-vs-streaming-in-spark-one-sql-engine-two-execution-contracts-a4e888cb4748?sk=e8f00a5249b4a1ce3aa992f614e76705">Part VIII </a>| <a href="https://pub.towardsai.net/part-ix-the-anatomy-of-a-spark-application-what-the-driver-executors-and-cluster-manager-ffa2bba60dce?sk=d66767c2ca186aaab54860931975741e">Part IX</a> | <a href="https://pub.towardsai.net/part-x-z-ordering-and-data-clustering-explained-why-your-partitioned-table-still-scans-4b109e016c80?sk=dc7cfcb72d4bd523b6b5ca35f79f2b16">Part X </a>| <a href="https://pub.towardsai.net/part-xi-rdd-vs-dataframe-vs-dataset-why-your-type-safe-refactor-made-spark-3x-slower-99bdba6f7eaa?source=friends_link&amp;sk=3cef58b143ef9d20917ac93e62731ecc">Part XI</a> | <a href="https://chakshu-salgotra.medium.com/da093c04b481?source=friends_link&amp;sk=1f9b68eee7c3c07c233d280b625b5fab">Part XII </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xiii-internal-vs-external-tables-the-drop-table-that-deleted-40tb-4fffae7f740c">Part XIII </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xiv-broadcast-variables-the-driver-bottleneck-nobody-profiles-87189bb691ed">Part XIV</a> | <a href="https://medium.com/towards-artificial-intelligence/part-xv-adaptive-query-execution-what-aqe-actually-rewrites-at-runtime-f010f22ad3fa">Part XV</a> | <a href="https://medium.com/towards-artificial-intelligence/part-xvi-whole-stage-codegen-the-tungsten-trick-behind-2x-speedups-6e0011a07944">Part XVI </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xvii-checkpointing-vs-lineage-the-fault-tolerance-trade-you-misconfigure-f5b1a5892d68">Part XVII</a> | <a href="https://medium.com/towards-artificial-intelligence/part-xviii-the-small-files-tax-compaction-strategies-that-dont-stall-593e9a3abe2c">Part XVIII</a><strong> </strong>| <a href="https://medium.com/towards-artificial-intelligence/e7b1a25cd7a1">Part XIX</a> | <strong>Part XX</strong></p><h3>The $41,000-a-Month DELETE That Deleted Nothing</h3><p>The page came at 2:58 AM on a Tuesday, and it wasn’t from a pipeline. It was from FinOps, in Slack, with a screenshot: our S3 spend had crossed $41,000 for the month, up from $9,400 five months earlier, and the line was still climbing. The accused was bronze.clickstream — 412 terabytes, fourteen months of raw click events, roughly 2.1 billion rows a day landing through a Structured Streaming MERGE.</p><p>Six months before that night, we had deleted 238 terabytes from that table. A GDPR erasure request had forced us to purge nine months of history containing raw PII. The DELETE ran for eleven hours, the row counts confirmed it, the auditors signed off. And the storage bill did not move. Not by a gigabyte.</p><p>We did what every team does: we blamed VACUUM. Then we checked, and VACUUM had been running green every Sunday for a year. VACUUM bronze.clickstream RETAIN 168 HOURS, dry-run clean, a few hundred gigabytes reclaimed each time, job healthy. The 238 terabytes were simply still there, on S3, billing us $0.023 per GB-month, while the table insisted they were gone.</p><p>At 3:40 AM, running on caffeine and indignation, the on-call engineer did the obvious thing. He disabled the retention safety check and ran the nuclear option:</p><pre>-- the 3:40 AM version of &quot;have you tried turning it off and on again&quot;<br>VACUUM bronze.clickstream RETAIN 0 HOURS</pre><p>It worked. Forty-one thousand dollars a month of dead files evaporated in ninety minutes. We went back to sleep feeling like heroes.</p><p>At 9:04 AM, finance ran the month-end close. Their reconciliation query reads the table TIMESTAMP AS OF the last day of the month — a time-travel read, thirty-one days back. It failed instantly with FileNotFoundException: a file referenced in the transaction log could not be found. The version they needed had been vacuumed out from under them at 4 AM. So had the training snapshot the ML team had pinned for a model refresh. Two teams, two broken reads, one very long incident review.</p><p>So: what does DELETE actually delete in Delta Lake, what is VACUUM legally allowed to remove, and why did 238 terabytes of &quot;deleted&quot; data survive six months of weekly vacuums — while thirty-one-day-old files died the moment we got impatient?</p><h3>The Mechanic Underneath: The Table Is a Ledger, Not a Directory</h3><p>Forget the parquet files for a minute. A Delta table is not a directory of data files. A Delta table is _delta_log/ — a directory of JSON files, each one a commit, each commit a list of <strong>actions</strong>. The data files are just the payload the ledger points at.</p><p>Every commit appends one file: 00000000000000000000.json, 00000000000000000001.json, and so on. Inside, the actions that matter:</p><ul><li><strong>add</strong>: this parquet file is now part of the table (path, size, partition values, file statistics).</li><li><strong>remove</strong>: this parquet file is no longer part of the table — with a deletionTimestamp and a dataChange flag. The file itself is untouched. It becomes a <strong>tombstone</strong>.</li><li><strong>protocol</strong>: the minimum reader/writer versions this table requires.</li><li><strong>metaData</strong>: schema, partition columns, table properties.</li><li><strong>txn</strong>: the streaming application ID and version, for idempotent micro-batch commits.</li><li><strong>commitInfo</strong>: who committed, when, with what operation.</li></ul><p>The state of the table at version N is the replay of every action from 0 to N. Nothing is ever mutated in place. A DELETE doesn&#39;t delete rows — it writes new parquet files without those rows and logs remove actions against the old ones. An UPDATE, a MERGE, an OPTIMIZE: all rewrites, all tombstones. The old files sit on S3, fully intact, until something with the authority to forget them comes along.</p><p>Because replaying millions of JSON files on every read would be absurd, Delta writes a <strong>checkpoint</strong> every 10 commits (controlled by delta.checkpointInterval): a parquet file summarizing the complete table state at that version, with _last_checkpoint pointing at the newest one. Reading version N means loading the nearest checkpoint at or below N and replaying only the JSON tail after it. That tail — the checkpoint cadence, the JSON retention, the replay cost — is where our second bill was hiding, as we&#39;ll see.</p><p>This is the whole mental model: <strong>the log is the table, the files are just evidence, and forgetting is a scheduled, retention-gated operation — never an immediate one.</strong></p><h3>Optimistic Concurrency: Every Commit Is a Conditional Write on a Version Number</h3><p>Writers don’t take locks. A writer reads the current version V, does its (possibly enormous) work against that snapshot, then attempts to write V+1.json as an atomic conditional put — it succeeds only if V is still the latest version. If another writer got there first, Delta runs conflict detection: two blind appends to different partitions can both commit (the loser retries against V+1); two MERGEs touching overlapping files cannot, and the loser gets a ConcurrentAppendException.</p><p>This is why the streaming MERGE into bronze.clickstream and the Sunday OPTIMIZE could coexist without corrupting anything — and also why our incident had a blast radius instead of a corruption. The 4 AM vacuum didn&#39;t corrupt the log; it deleted <em>evidence</em> the log still referenced. The ledger was perfect. The files were gone. Optimistic concurrency protects the log&#39;s integrity, not the physical files&#39; existence. Those are two different durability promises, and we had confused them.</p><p>The protocol action deserves a paranoid glance too. minReaderVersion and minWriterVersion gate which engine versions may touch the table, and modern Delta adds named table features — deletion vectors, column mapping, liquid clustering, v2 checkpoints. Bumping the protocol is a one-way door: old readers are locked out permanently. It has nothing to do with our incident, but it&#39;s the same class of mistake — a metadata decision with physical consequences that you only discover at 9:04 AM.</p><h3>Time Travel Is Log Replay, Not a Backup</h3><p>TIMESTAMP AS OF and VERSION AS OF feel like backups. They are not. There is no snapshot stored anywhere. A time-travel read resolves your timestamp to a version number, loads the nearest checkpoint at or below it, and replays the JSON tail — the exact same machinery as a current read, just stopping earlier.</p><p>The consequence is absolute: <strong>a time-travel read is only as durable as the log entries and the data files they reference.</strong> Keep the log for 30 days and you can time-travel 30 days — provided nobody vacuumed the files those old versions point at. Our finance query needed version history 31 days deep. Our vacuum at RETAIN 0 HOURS had deleted every file unreferenced by the <em>current</em> version, including every file the 31-day-old version needed. The ledger still described a table that could be reconstructed; the reconstruction was impossible. FileNotFoundException is the log telling the truth about files that no longer exist.</p><h3>VACUUM Can Only Delete What the Log Has Already Forgotten</h3><p>Here is the actual deletion pipeline, in order:</p><ol><li>A DELETE/MERGE/UPDATE logs remove actions with a deletionTimestamp. The files are now tombstones — logically dead, physically alive.</li><li>Tombstones are protected for delta.deletedFileRetentionDuration (default: 7 days). This is not a suggestion; it&#39;s the table property VACUUM consults.</li><li>VACUUM ... RETAIN N HOURS deletes data files that are referenced by <strong>no</strong> table version within the retention window — and only if they&#39;re older than the threshold. It also refuses RETAIN under 168 hours unless you explicitly disable spark.databricks.delta.retentionDurationCheck.enabled, because a query or a streaming micro-batch can legitimately take longer than you think.</li></ol><p>Now the root cause of the $41,000 bill: eighteen months earlier, a well-meaning predecessor had set two table properties “for audit safety”:</p><pre>&#39;delta.deletedFileRetentionDuration&#39; = &#39;interval 365 days&#39;,<br>&#39;delta.logRetentionDuration&#39;         = &#39;interval 365 days&#39;</pre><p>Every tombstone the GDPR delete created was legally protected for a year. Our weekly VACUUM RETAIN 168 HOURS would list the 238 terabytes of dead files, check each one&#39;s deletionTimestamp against the one-year tombstone retention, and walk away. It ran green because it was doing exactly what the table told it to do: nothing. The retention property outranks your urgency. <strong>VACUUM obeys the log, not your FinOps dashboard.</strong></p><p>And the 3:40 AM RETAIN 0 HOURS? It bypassed the only guardrail standing between &quot;delete unreferenced files&quot; and &quot;delete files that old-but-legitimate versions still reference.&quot; The default 168-hour floor exists precisely because time-travel readers, long-running queries, and concurrent writers can hold a logical reference to a version for days. Zero hours means zero patience for anyone reading the past.</p><h3>The Log Files Themselves Are the Second Bill</h3><p>While we were staring at the 238 terabytes, the _delta_log directory was running its own scam. bronze.clickstream commits roughly 3,400 times a day — streaming micro-batches, hourly OPTIMIZE runs, the occasional MERGE. With delta.logRetentionDuration set to 365 days, that is:</p><p><strong>3,400 commits/day × 365 days ≈ 1.24 million JSON files</strong> sitting in _delta_log/.</p><p>Every single read of that table — every dashboard refresh, every ad-hoc query — starts by listing _delta_log, reading _last_checkpoint, loading the checkpoint parquet, and replaying the JSON tail since the last checkpoint. Our tail was thousands of files long on a bad day. Query planning alone took 47 seconds before a single byte of data was touched, and S3 LIST requests — $0.005 per thousand — were a line item of their own. The ledger had become heavier than some of the tables it described.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GYx8dxYVCMy83GPLPlXWYg.png" /></figure><p>Checkpoints cap the replay, but they don’t cap the listing or the retention. The checkpoint interval (10) was fine; the 365-day log retention was the disease. A ledger you’re legally required to keep for a year is an archive, not a live table — and we were paying live-table prices to replay it on every query.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nCoQ8avzu1eNisfSg1tIfw.png" /></figure><h3>The Fix, and What It Cost</h3><p>The naive fix is what we did at 3:40 AM. The engine-aware fix starts from the actual SLAs — how far back anyone is <em>contractually</em> allowed to read — and tiers every retention knob to match:</p><pre># Naive: nuke everything unreferenced RIGHT NOW.<br># spark.conf.set(<br>#     &quot;spark.databricks.delta.retentionDurationCheck.enabled&quot;, &quot;false&quot;<br># )  # disables the 7-day guardrail — the guardrail exists for time-travel readers<br># spark.sql(&quot;VACUUM bronze.clickstream RETAIN 0 HOURS&quot;)<br># Result: 238TB freed, then FileNotFoundException on every read older than &quot;now&quot;.</pre><pre># Optimized: tier retention to the real SLAs, then vacuum honestly.<br>spark.sql(&quot;&quot;&quot;<br>  ALTER TABLE bronze.clickstream SET TBLPROPERTIES (<br>    &#39;delta.deletedFileRetentionDuration&#39; = &#39;interval 7 days&#39;,<br>    &#39;delta.logRetentionDuration&#39;         = &#39;interval 30 days&#39;<br>  )<br>&quot;&quot;&quot;)  # finance needs 30-day time travel; GDPR needs 7-day tombstone expiry, not 365</pre><pre># Always DRY RUN first, and read what it plans to delete:<br># spark.sql(&quot;VACUUM bronze.clickstream RETAIN 168 HOURS DRY RUN&quot;)<br>spark.sql(&quot;VACUUM bronze.clickstream RETAIN 168 HOURS&quot;)</pre><p>Two companion decisions, both non-optional:</p><ul><li><strong>Audit history moved out of the live table.</strong> The “audit safety” requirement was real, so we gave it a real answer: a monthly archived copy of the table in cheap storage, instead of a 365-day retention property strangling the production table.</li><li><strong>RESTORE discipline.</strong> RESTORE TABLE ... TO VERSION AS OF is metadata-only — it appends a new version re-adding old removed files. It can only resurrect files that still exist. Our runbook now states the restore horizon explicitly: 30 days, matching logRetentionDuration, verified by a monthly restore drill against a clone.</li></ul><p>The quantified results, thirty days later:</p><ul><li><strong>S3 spend: $41,000/month → $12,400/month.</strong> The 238TB of GDPR tombstones plus ~40TB of orphaned log and compaction debris, gone — legally this time.</li><li><strong>_delta_log file count: 1.24M JSON files → ~102k.</strong> Thirty days of history at 3,400 commits/day.</li><li><strong>Query planning time: 47s → 3s</strong> on the standard dashboard queries. Same data, same cluster.</li><li><strong>Month-end close time-travel query: green</strong>, and covered by the monthly restore drill.</li></ul><h3>The Catch: Retention Is a One-Way Door, and the Vacuum Doesn’t Know Your SLAs</h3><p>The honest counterweight, because the fix has sharp edges:</p><ul><li><strong>Lowering retention is irreversible for old versions.</strong> The moment tombstones age past the new threshold and VACUUM runs, those versions are unrecoverable — RESTORE becomes a FileNotFoundException factory. We lowered 365 days to 30 only after the archive copy existed and the stakeholders signed the horizon in writing.</li><li><strong>VACUUM cannot delete files referenced within retention, full stop.</strong> A 30-day time-travel SLA means a minimum of 30 days of dead files sitting on S3. That is the price of the feature, not a bug in the tool. Budget for it.</li><li><strong>Never point S3 lifecycle rules at Delta data directories.</strong> Lifecycle policies delete by object age with no knowledge of the log. A lifecycle rule expiring “old” parquet files will happily delete files the current version references, and the table will corrupt silently until the next read. Retention belongs in Delta properties, not in bucket policies.</li><li><strong>The 168-hour floor is load-bearing.</strong> We now treat any request to vacuum below 7 days the way we’d treat a request to delete the backups: possible, documented, and requiring two sign-offs. A streaming MERGE micro-batch retrying against a transient failure can legitimately need files older than your impatience.</li></ul><h3>Takeaways</h3><ul><li><strong>The log is the table:</strong> _delta_log/ defines every version; the parquet files are just evidence the ledger points at. Protect the ledger&#39;s integrity and the files&#39; existence as two separate promises.</li><li><strong>DELETE is a rewrite, not an erasure:</strong> every DELETE, UPDATE, and MERGE writes new files and tombstones the old ones. Storage is freed only when retention expires <em>and</em> VACUUM runs.</li><li><strong>VACUUM obeys the log, not your urgency:</strong> deletedFileRetentionDuration outranks your RETAIN clause. A 365-day tombstone retention makes weekly vacuums a ceremonial no-op.</li><li><strong>RETAIN 0 HOURS is a loaded gun:</strong> it deletes files that old-but-legitimate versions still reference. The 7-day guardrail exists because time-travel readers and long-running queries hold logical references for days.</li><li><strong>Time travel is only as durable as your retention:</strong> a 31-day-old read needs 31 days of log <em>and</em> 31 days of un-vacuumed files. Set logRetentionDuration from the restore SLA, not from vibes.</li><li><strong>The </strong><strong>_delta_log is a cost center too:</strong> commit-heavy tables with long log retention accumulate millions of JSON files, and every reader pays the LIST-and-replay tax. Size logRetentionDuration to the readership, not the auditors — archive separately.</li></ul><p>In Delta Lake, nothing is ever deleted — it is only forgotten on a schedule. Make sure you’re the one writing the schedule.</p><p>If this saved you a 3 AM pager incident, <a href="https://medium.com/@chakshu-salgotra">follow me on Medium</a> for Twice a week for the next 3 Months of Advanced Data Engineering &amp; AI. Let’s connect on <a href="https://www.linkedin.com/in/chakshu-salgotra/">LinkedIn</a> and <a href="https://x.com/chakshusalgotra">X</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc60f6de0360" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/part-xx-delta-lakes-transaction-log-time-travel-vacuum-and-the-files-that-never-die-fc60f6de0360">Part XX — Delta Lake’s Transaction Log: Time Travel, VACUUM, and the Files That Never Die</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part XIX — Spark on Kubernetes: Executor Sizing and Dynamic Allocation]]></title>
            <link>https://pub.towardsai.net/part-xix-spark-on-kubernetes-executor-sizing-and-dynamic-allocation-e7b1a25cd7a1?source=rss-7afde1b096a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7b1a25cd7a1</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[chakshu_salgotra]]></dc:creator>
            <pubDate>Fri, 30 Oct 2026 20:31:01 GMT</pubDate>
            <atom:updated>2026-10-30T20:31:01.552Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Kubernetes killed our executors every night for a week while the JVM insisted memory was fine. The JVM was telling the truth — it just wasn’t the memory Kubernetes was measuring.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6nT5uKb2wlo6gP-fye8tLA.png" /></figure><p><a href="https://chakshu-salgotra.medium.com/how-spark-reads-parquet-2333c7fa728c">Part I</a> | <a href="https://chakshu-salgotra.medium.com/spark-shuffle-partitioning-under-the-hood-of-a-wide-transformation-521907cd5f1c">Part II</a><strong> | </strong><a href="https://medium.com/@chakshu-salgotra/spark-catalyst-optimizer-internals-2b762ce0b2e2">Part III</a> <strong>| </strong><a href="https://pub.towardsai.net/jobs-stages-tasks-how-spark-actually-decomposes-your-query-e50bc8ed768e">Part IV</a> <strong>| </strong><a href="https://pub.towardsai.net/how-spark-manages-memory-the-unified-memory-model-spills-and-aqe-f37f21799fc6?sk=c91e30bf8b55b332b5611b7585b0b883">Part V</a> <strong>| </strong><a href="https://pub.towardsai.net/part-vi-the-spark-join-strategies-broadcast-vs-52a2048203e6?sk=aec7100814c2dbac55341898fb4f0686">Part VI<strong> </strong></a><strong>| </strong><a href="https://pub.towardsai.net/part-vii-data-skew-in-spark-why-199-tasks-finish-in-40-seconds-and-one-runs-for-3-hours-7daef3b1de44?sk=2262448f82bf04acda6dea9bfdeca857">Part VII</a> | <a href="https://pub.towardsai.net/part-viii-batch-vs-streaming-in-spark-one-sql-engine-two-execution-contracts-a4e888cb4748?sk=e8f00a5249b4a1ce3aa992f614e76705">Part VIII </a>| <a href="https://pub.towardsai.net/part-ix-the-anatomy-of-a-spark-application-what-the-driver-executors-and-cluster-manager-ffa2bba60dce?sk=d66767c2ca186aaab54860931975741e">Part IX</a> | <a href="https://pub.towardsai.net/part-x-z-ordering-and-data-clustering-explained-why-your-partitioned-table-still-scans-4b109e016c80?sk=dc7cfcb72d4bd523b6b5ca35f79f2b16">Part X </a>| <a href="https://pub.towardsai.net/part-xi-rdd-vs-dataframe-vs-dataset-why-your-type-safe-refactor-made-spark-3x-slower-99bdba6f7eaa?source=friends_link&amp;sk=3cef58b143ef9d20917ac93e62731ecc">Part XI</a> | <a href="https://chakshu-salgotra.medium.com/da093c04b481?source=friends_link&amp;sk=1f9b68eee7c3c07c233d280b625b5fab">Part XII </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xiii-internal-vs-external-tables-the-drop-table-that-deleted-40tb-4fffae7f740c">Part XIII </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xiv-broadcast-variables-the-driver-bottleneck-nobody-profiles-87189bb691ed">Part XIV</a> | <a href="https://medium.com/towards-artificial-intelligence/part-xv-adaptive-query-execution-what-aqe-actually-rewrites-at-runtime-f010f22ad3fa">Part XV</a> | <a href="https://medium.com/towards-artificial-intelligence/part-xvi-whole-stage-codegen-the-tungsten-trick-behind-2x-speedups-6e0011a07944">Part XVI </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xvii-checkpointing-vs-lineage-the-fault-tolerance-trade-you-misconfigure-f5b1a5892d68">Part XVII</a> | <a href="https://medium.com/towards-artificial-intelligence/part-xviii-the-small-files-tax-compaction-strategies-that-dont-stall-593e9a3abe2c">Part XVIII</a><strong> </strong>| <strong>Part XIX</strong></p><h3>The Executors That Kept Dying at 2 AM</h3><p>The page came at 2:14 AM on a Tuesday: the overnight ETL had failed. Again. The third night in a row, and the fourth time that week if you counted Sunday’s partial run. Each time the same signature — executors dying mid-stage with exit code 137, the job restarting from the last shuffle boundary, and the whole thing stretching from its usual 90 minutes to 5 hours and 12 minutes before finally limping across the finish line or giving up.</p><p>Here’s what made it maddening: the Spark UI showed nothing wrong. Executor heap usage hovered at 55–65%, GC looked healthy, no skew (Part VII’s playbook came back clean), no spill warnings. The driver was calm. By every metric Spark itself reported, the job was fine — and yet Kubernetes kept killing executors like it had a grudge.</p><p>The kill count told the real story: <strong>43 OOMKills in one night</strong> across 48 executors, each one forcing its shuffle outputs to be recomputed, each recompute triggering a retry storm that made the survivors work harder and die faster. We had configured each executor with 8 GB of heap, confident that was generous for the workload. Kubernetes disagreed, silently, at 2 AM, every night.</p><p>So: why does Kubernetes kill a JVM that reports healthy memory — and how do you size executors for the scheduler that’s actually in charge?</p><h3>The Mechanic Underneath: The Pod Is the Unit of Accounting</h3><p>Here is the mental model: <strong>the JVM manages heap; Kubernetes manages the cgroup; and the cgroup always wins.</strong> When you set spark.executor.memory=8g, you&#39;ve configured the JVM&#39;s heap ceiling — not the pod&#39;s memory footprint. The pod also carries everything off-heap: Netty shuffle buffers, direct byte buffers, native libraries, thread stacks, GC bookkeeping, and Spark&#39;s own memory overhead. Kubernetes sums <em>all</em> of it against the pod&#39;s memory limit, and when the total crosses the line, the OOM killer fires. Exit 137. No warning, no heap dump, no appeal.</p><p>This is why Spark’s metrics lied to us. The UI reports heap. The killer measures the cgroup. Our executors were running at 60% heap and 104% of their pod limit — healthy by Spark’s books, dead by Kubernetes’.</p><p>Part IX of this series broke down what the driver, executors, and cluster manager each own. On Kubernetes, add one more owner: the <strong>kubelet</strong>, which enforces the one limit Spark never sees.</p><h3>Memory Overhead Is Not a Tuning Knob — It’s Physics</h3><p>Spark knows about this gap, which is why spark.executor.memoryOverhead exists. The default is max(384MB, 10% of executor memory) — for our 8 GB executors, 800 MB of headroom above heap. The problem: off-heap usage doesn&#39;t scale politely with heap. Shuffle-heavy stages allocate large Netty buffers and direct memory that can spike well past 10%, especially with wide joins or big aggregations. Our 800 MB of overhead was being eaten by shuffle buffers alone during the join stage; the rest — thread stacks, native code, page cache pressure — pushed the pod over the limit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gToXirkDN9DW561yM36YPg.png" /></figure><p>The rule that actually holds: <strong>size the pod limit as heap + measured off-heap peak + margin, not heap + 10% and hope.</strong> The 10% default is a starting guess for average workloads. Shuffle-heavy ETL is not an average workload.</p><h3>Big Executors Are a Scheduling Tax</h3><p>Our original instinct had been fat executors: 12 cores, 32 GB heap each, fewer pods to manage. On YARN that instinct is often fine. On Kubernetes it’s a scheduling tax with three line items:</p><ul><li><strong>Bin-packing:</strong> a 12-core pod only schedules where 12 contiguous cores are free. On a busy cluster, fat pods sit in Pending while small pods flow around them. We watched executors wait 9 minutes for placement on nights the cluster was warm.</li><li><strong>Blast radius:</strong> one OOMKill on a 12-core executor loses 12 cores of shuffle output at once. The retry storm is proportional to executor size.</li><li><strong>Task granularity:</strong> Spark parallelizes at the task level; a 12-core executor needs 12 concurrent tasks to stay busy. Narrow stages and small partitions leave cores idle — the “97% idle cluster” failure mode from Part IX, wearing a Kubernetes costume.</li></ul><p>The sane band for K8s executors is boring on purpose: <strong>4–5 cores, 12–16 GB heap each.</strong> Small enough to schedule anywhere, small enough that a kill is a flesh wound, granular enough to keep cores fed.</p><p>Match the node pool to the pod shape while you’re at it. If your executors request 4 cores and 15 GB, a node with 16 cores and 64 GB fits exactly four of them with no stranded capacity. Nodes sized as clean multiples of your executor shape turn bin-packing from a scheduling lottery into arithmetic — and it’s the cheapest utilization win on this whole page.</p><h3>Dynamic Allocation on Kubernetes Is Not YARN’s Dynamic Allocation</h3><p>This is the trap that bites every YARN veteran. On YARN, dynamic allocation leans on the <strong>external shuffle service</strong> — executors can be killed and their shuffle data survives. On Kubernetes there is no external shuffle service by default. If the autoscaler (or idle timeout) kills an executor holding shuffle output, that output is gone, and every downstream stage recomputes it.</p><p>Spark 3.x added shuffle tracking on Kubernetes (spark.dynamicAllocation.shuffleTracking.enabled=true) as the answer: the driver tracks which executors hold shuffle data needed by running stages and refuses to decommission them. It works — but understand what it costs:</p><ul><li><strong>Executors pin longer.</strong> Shuffle-tracking executors can’t be reclaimed until their data is consumed, so your “dynamic” allocation is stickier than the name suggests.</li><li><strong>Idle timeout still applies</strong> (spark.dynamicAllocation.executorIdleTimeout, default 60s) — but only to executors with no useful state.</li><li><strong>Scale-down is lazy, scale-up is eager.</strong> New executors appear fast; old ones linger. Budget for the tail.</li></ul><p>The alternative — running a shuffle service as a DaemonSet on K8s — exists but is operationally heavy and rarely worth it. Shuffle tracking plus sensible min/max executor bounds covers nearly every workload.</p><p>One more K8s-specific wrinkle: scale-up requests go through the Kubernetes scheduler, not Spark’s. spark.dynamicAllocation.initialExecutors controls how many pods Spark requests at startup, but every scale-up beyond that is a pod-creation round trip — image pull, scheduling, JVM warm-up — typically 30–90 seconds before the executor registers. If your stages are short and bursty, that latency eats the benefit of elasticity entirely; you&#39;ll finish the stage on the initial executors before the reinforcements arrive. For bursty workloads, a higher initialExecutors with a tight idle timeout beats a low floor with aggressive scale-up.</p><h3>The Fix, and What It Cost</h3><pre># Naive: throw heap at the problem and hope the killer goes away.<br># The OOMKills continued — off-heap spikes don&#39;t respect heap settings.<br>conf = (<br>    SparkConf()<br>    .set(&quot;spark.executor.instances&quot;, &quot;48&quot;)<br>    .set(&quot;spark.executor.cores&quot;, &quot;12&quot;)            # fat executors, slow scheduling<br>    .set(&quot;spark.executor.memory&quot;, &quot;16g&quot;)          # bigger heap, same 10% overhead guess<br>    .set(&quot;spark.kubernetes.executor.limit.cores&quot;, &quot;12&quot;)<br>)</pre><pre># Optimized: right-size the pod, make overhead explicit, let allocation breathe.<br># Naive: 48 fat executors, 16g heap, default overhead, static allocation.<br># Optimized: small executors, measured overhead, shuffle-tracked dynamic allocation.<br>conf = (<br>    SparkConf()<br>    # Small executors: schedule fast, die cheap<br>    .set(&quot;spark.executor.cores&quot;, &quot;4&quot;)<br>    .set(&quot;spark.executor.memory&quot;, &quot;12g&quot;)<br>    # Overhead sized from MEASURED off-heap peaks, not the 10% default<br>    .set(&quot;spark.executor.memoryOverheadFactor&quot;, &quot;0.25&quot;)  # 3g overhead on 12g heap<br>    # Pod limit = heap + overhead + margin; the kubelet enforces the SUM<br>    .set(&quot;spark.kubernetes.memoryOverheadFactor&quot;, &quot;0.25&quot;)<br>    # Dynamic allocation with shuffle tracking (no external shuffle svc on K8s)<br>    .set(&quot;spark.dynamicAllocation.enabled&quot;, &quot;true&quot;)<br>    .set(&quot;spark.dynamicAllocation.shuffleTracking.enabled&quot;, &quot;true&quot;)  # don&#39;t kill shuffle holders<br>    .set(&quot;spark.dynamicAllocation.minExecutors&quot;, &quot;8&quot;)<br>    .set(&quot;spark.dynamicAllocation.maxExecutors&quot;, &quot;64&quot;)<br>    .set(&quot;spark.dynamicAllocation.executorIdleTimeout&quot;, &quot;120s&quot;)  # let stragglers finish<br>    # Keep shuffle partitions proportional to the smaller executors<br>    .set(&quot;spark.sql.shuffle.partitions&quot;, &quot;512&quot;)  # 64 exec x 4 cores x 2<br>)</pre><p>How we sized the overhead honestly: one canary run with the overhead factor at 0.4, watched the pod’s actual peak RSS via kubelet metrics, then set the factor to measured peak + 20% margin. No guessing.</p><p>Results, measured over the following two weeks:</p><ul><li><strong>OOMKills:</strong> 43/night → 0. Not reduced — eliminated.</li><li><strong>Job wall-clock:</strong> 5h12m (worst night) → 1h48m steady. The retry storms were the real cost, not the kills.</li><li><strong>Scheduling delay:</strong> 9-minute Pending waits → under 40 seconds. Small pods pack anywhere.</li><li><strong>Cluster utilization:</strong> 31% → 74% on the same node pool, because dynamic allocation actually released idle executors instead of holding 48 fat ones hostage.</li><li><strong>Cost:</strong> ~38% lower nightly spend on the same workload — fewer nodes held, no recompute storms.</li></ul><h3>The Catch: What Right-Sizing Doesn’t Fix</h3><ul><li><strong>Dynamic allocation and streaming mix badly.</strong> Structured Streaming wants stable executors; aggressive scale-down fights the micro-batch cadence. For streaming on K8s, prefer fixed executor counts and tune from there.</li><li><strong>Shuffle tracking isn’t free.</strong> The driver does more bookkeeping, and pinned executors mean your “dynamic” floor is higher than minExecutors suggests during shuffle-heavy phases. Watch it; don&#39;t assume.</li><li><strong>Spot and preemptible nodes are shuffle Russian roulette.</strong> A preemption is an OOMKill with better PR — shuffle data on that node is gone either way. If you run executors on spot, keep maxExecutors generous and accept recompute as a line item, or keep shuffle-critical stages on on-demand.</li><li><strong>The driver is still a single point of failure.</strong> All of this is executor hygiene. Driver sizing, driver pod disruption budgets, and driver retry policy are a separate incident waiting to happen — and the kubelet will kill an oversized driver just as happily.</li></ul><h3>Takeaways</h3><ul><li><strong>The kubelet measures the cgroup, not the heap:</strong> a JVM at 60% heap can still be OOMKilled if off-heap pushes the pod over its limit.</li><li><strong>Overhead is physics, not a default:</strong> max(384MB, 10%) is a starting guess — size it from measured pod RSS peaks plus margin.</li><li><strong>Small executors schedule faster and die cheaper:</strong> 4–5 cores and 12–16 GB heap beats fat pods on bin-packing, blast radius, and task granularity.</li><li><strong>No shuffle service on K8s by default:</strong> use shuffle tracking so dynamic allocation never kills an executor holding live shuffle data.</li><li><strong>Scale-up is eager, scale-down is lazy:</strong> budget for the sticky tail of shuffle-pinned executors.</li><li><strong>The retry storm is the real cost:</strong> one OOMKill is cheap; the recompute cascade it triggers is what turns 90 minutes into 5 hours.</li></ul><p>The scheduler is always in charge — the only question is whether you’ve sized for the one that’s actually enforcing the limits.</p><p>If this saved you a 3 AM pager incident, <a href="https://medium.com/@chakshu-salgotra">follow me on Medium</a> for Twice a week for the next 3 Months of Advanced Data Engineering &amp; AI. Let’s connect on <a href="https://www.linkedin.com/in/chakshusalgotra/">LinkedIn</a> and <a href="https://x.com/chakshusalgotra">X</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7b1a25cd7a1" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/part-xix-spark-on-kubernetes-executor-sizing-and-dynamic-allocation-e7b1a25cd7a1">Part XIX — Spark on Kubernetes: Executor Sizing and Dynamic Allocation</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part XVIII — The Small-Files Tax: Compaction Strategies That Don’t Stall]]></title>
            <link>https://pub.towardsai.net/part-xviii-the-small-files-tax-compaction-strategies-that-dont-stall-593e9a3abe2c?source=rss-7afde1b096a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/593e9a3abe2c</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[delta-lake]]></category>
            <dc:creator><![CDATA[chakshu_salgotra]]></dc:creator>
            <pubDate>Fri, 23 Oct 2026 12:31:01 GMT</pubDate>
            <atom:updated>2026-10-23T12:31:01.585Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Your lakehouse doesn’t slow down because the data grew. It slows down because the file count did — and every one of those files is a tax your query planner pays before it reads a single byte.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*no-z1FU_aQKTCWHPqp0AIQ.png" /></figure><p><a href="https://chakshu-salgotra.medium.com/how-spark-reads-parquet-2333c7fa728c">Part I</a> | <a href="https://chakshu-salgotra.medium.com/spark-shuffle-partitioning-under-the-hood-of-a-wide-transformation-521907cd5f1c">Part II</a><strong> | </strong><a href="https://medium.com/@chakshu-salgotra/spark-catalyst-optimizer-internals-2b762ce0b2e2">Part III</a> <strong>| </strong><a href="https://pub.towardsai.net/jobs-stages-tasks-how-spark-actually-decomposes-your-query-e50bc8ed768e">Part IV</a> <strong>| </strong><a href="https://pub.towardsai.net/how-spark-manages-memory-the-unified-memory-model-spills-and-aqe-f37f21799fc6?sk=c91e30bf8b55b332b5611b7585b0b883">Part V</a> <strong>| </strong><a href="https://pub.towardsai.net/part-vi-the-spark-join-strategies-broadcast-vs-52a2048203e6?sk=aec7100814c2dbac55341898fb4f0686">Part VI<strong> </strong></a><strong>| </strong><a href="https://pub.towardsai.net/part-vii-data-skew-in-spark-why-199-tasks-finish-in-40-seconds-and-one-runs-for-3-hours-7daef3b1de44?sk=2262448f82bf04acda6dea9bfdeca857">Part VII</a> | <a href="https://pub.towardsai.net/part-viii-batch-vs-streaming-in-spark-one-sql-engine-two-execution-contracts-a4e888cb4748?sk=e8f00a5249b4a1ce3aa992f614e76705">Part VIII </a>| <a href="https://pub.towardsai.net/part-ix-the-anatomy-of-a-spark-application-what-the-driver-executors-and-cluster-manager-ffa2bba60dce?sk=d66767c2ca186aaab54860931975741e">Part IX</a> | <a href="https://pub.towardsai.net/part-x-z-ordering-and-data-clustering-explained-why-your-partitioned-table-still-scans-4b109e016c80?sk=dc7cfcb72d4bd523b6b5ca35f79f2b16">Part X </a>| <a href="https://pub.towardsai.net/part-xi-rdd-vs-dataframe-vs-dataset-why-your-type-safe-refactor-made-spark-3x-slower-99bdba6f7eaa?source=friends_link&amp;sk=3cef58b143ef9d20917ac93e62731ecc">Part XI</a> | <a href="https://chakshu-salgotra.medium.com/da093c04b481?source=friends_link&amp;sk=1f9b68eee7c3c07c233d280b625b5fab">Part XII </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xiii-internal-vs-external-tables-the-drop-table-that-deleted-40tb-4fffae7f740c">Part XIII </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xiv-broadcast-variables-the-driver-bottleneck-nobody-profiles-87189bb691ed">Part XIV</a> | <a href="https://medium.com/towards-artificial-intelligence/part-xv-adaptive-query-execution-what-aqe-actually-rewrites-at-runtime-f010f22ad3fa">Part XV</a> | <a href="https://medium.com/towards-artificial-intelligence/part-xvi-whole-stage-codegen-the-tungsten-trick-behind-2x-speedups-6e0011a07944">Part XVI </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xvii-checkpointing-vs-lineage-the-fault-tolerance-trade-you-misconfigure-f5b1a5892d68">Part XVII</a> | <strong>Part XVIII</strong></p><h3>The 58-Minute Query That Read Nothing Useful</h3><p>The Slack message landed at 9:41 PM, eleven minutes into the nightly reporting window: <em>“finance dashboard is stuck again.”</em> Not failed — stuck. The aggregate query over our events Delta table had been creeping for months: 11 minutes in March, 24 in May, 41 in July. That night it hit 58 minutes, and the CFO&#39;s morning numbers were going to be late.</p><p>The table wasn’t that big. 2.9 TB, which is a rounding error on our cluster. But the file count was 3.8 million, median file size 2.4 MB — a graveyard of hourly micro-batch writes, backfills, and streaming checkpoints, each one dutifully committed as its own tiny Parquet file. The Spark UI told the story in one line: the job spent its first <strong>22 minutes</strong> doing nothing but listing and planning. Twenty-two minutes before a single task read data.</p><p>The driver was the casualty. To plan the query it had to fetch 3.8 million file statuses, hold them in memory, and prune against them. Driver GC pauses stretched to 40 seconds. Meanwhile every listing hammered S3: 3,800 LIST requests per full scan at 1,000 keys a pop, and we were listing that prefix several times a day across jobs — the LIST bill alone had quietly crossed $900/month, which is absurd money to spend on <em>finding</em> your data.</p><p>And the cruelest part: data skipping was dead. With 2.4 MB files, the min/max statistics on each file overlapped everything. A filter on event_date that should have pruned 90% of files pruned almost nothing, because every tiny file&#39;s date range bled into its neighbors&#39;. We were reading the whole table to answer a query that needed 3% of it.</p><p>So: where does file count actually bite in a lakehouse engine, and how do you compact millions of small files without stalling the readers who are querying them right now?</p><h3>The Mechanic Underneath: Every File Is a Row in Your Query Plan</h3><p>Here is the mental model that makes everything else click: <strong>a lakehouse query planner does not read data — it reads files, then reads data.</strong> Every file on the table is a unit of planning work. Listing it, fetching its statistics, evaluating it against partition and data-skipping predicates, assigning it to a task. That work scales with file count, not data volume. A 3 TB table in 3,000 files plans in seconds. The same 3 TB in 3.8 million files plans in tens of minutes and occasionally kills the driver trying.</p><p>This is why the small-files problem never shows up in benchmarks and always shows up in production. Benchmarks generate clean, big files. Production accumulates files the way a desk accumulates paper — one small write at a time, each one individually reasonable, collectively catastrophic.</p><h3>Listing Is a Distributed Systems Bill, Not a Free Lookup</h3><p>Object stores don’t have directories. A “listing” is a paginated API call returning 1,000 keys at a time, and the planner needs the <em>full</em> listing before it can prune. For our table: 3.8M files meant 3,800 LIST calls per scan, each with its own latency. Delta Lake’s transaction log helps — the log knows the current file set without listing the prefix — but the driver still materializes every active file’s metadata into the plan. The listing cost moved from S3 to driver memory, which is arguably worse: <strong>S3 LIST costs you dollars; driver-side file metadata costs you GC pauses and OOMs.</strong></p><p>The math that matters:</p><ul><li><strong>Plan-time is O(files).</strong> Doubling the file count roughly doubles planning latency, regardless of data size.</li><li><strong>Driver memory is O(files).</strong> Each file’s status and statistics live in the driver’s heap during planning. Millions of files is a multi-GB planning footprint before Spark reads anything.</li><li><strong>Task granularity follows files.</strong> One file is (roughly) one task’s minimum unit. Millions of tiny files mean millions of tiny tasks — scheduler overhead, tiny I/O, no vectorization benefit.</li></ul><h3>Data Skipping Dies When Min/Max Ranges Overlap</h3><p>Part X of this series covered Z-ordering and file-level statistics in depth; the short version: data skipping works when a file’s min/max range for a filtered column sits <em>outside</em> your predicate. A 128 MB file holding one hour of events has a tight range and skips cleanly. A 2.4 MB file holding ninety seconds of events has a range that overlaps its neighbors on every side — and with 3.8 million such files, nearly every file overlaps nearly every predicate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LX6CWcEPsvBKaFltdvyQ6w.png" /></figure><p><strong>Overlapping ranges are the tax; the files are just the receipt.</strong> Compaction’s real job isn’t reducing file count for its own sake — it’s restoring non-overlapping, tight min/max ranges so skipping works again.</p><h3>Compaction Is Bin-Packing With a Write-Amplification Bill</h3><p>OPTIMIZE (Delta) / rewrite_data_files (Iceberg) is bin-packing: read N small files, write M large ones at the target size (128 MB–1 GB is the sane band), commit atomically. The engine mechanics are straightforward; the economics are the interesting part:</p><ul><li><strong>Write amplification:</strong> compacting 3.8M files of 2.4 MB means <em>reading 2.9 TB and rewriting 2.9 TB</em>. You pay the full I/O twice, plus the compute to do it. Compaction is never free — it’s a loan you repay in faster reads.</li><li><strong>Z-order interplay:</strong> a plain bin-pack restores file size but not clustering. If you Z-order on the hot filter columns during compaction (see Part X), you fix size <em>and</em> skipping in one pass. Skip the Z-order and you’ll be compacting again in six months.</li><li><strong>Scheduling without stalling readers:</strong> compaction commits are atomic snapshot swaps, so readers never see a half-compacted table. The real contention is compute: a full-table OPTIMIZE hogs the cluster. The fix is incremental — compact only what drifted (more in The Fix).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tAbAXnuNXkU12HzxYG-Stg.png" /></figure><h3>The File-Count Budget: Sizing Compaction Before It Owns You</h3><p>Don’t compact by vibes. Run the budget:</p><ul><li><strong>Production rate:</strong> our table gained ~48,000 files/day (hourly micro-batches + streaming + backfills).</li><li><strong>Planning pain threshold:</strong> on our driver shape, planning stayed under 60 seconds up to ~150K files per query’s file set.</li><li><strong>Compaction capacity:</strong> our nightly window could rewrite ~400 GB/hour without starving production.</li></ul><p>48,000 files/day × 2.4 MB ≈ 115 GB/day of small files — well within a nightly incremental window. The mistake had been letting it compound for 18 months instead of draining it nightly. <strong>File count is a queue; compaction is the drain. Size the drain to the inflow, not to the backlog.</strong></p><p>One operational habit made this stick: we started emitting the table’s file count as a first-class metric, scraped nightly from the Delta log, with an alert at 2× the planning-pain threshold. File count had been invisible for 18 months because nobody was watching it — it wasn’t in any dashboard, so it never became anyone’s problem until it became everyone’s. The cheapest compaction strategy is the one that never lets the backlog form, and that starts with a number on a graph, not a cron job.</p><h3>The Fix, and What It Cost</h3><pre>-- Naive: blanket OPTIMIZE on the whole table every night.<br>-- Rewrites 2.9 TB nightly, 6+ hours, burns the compute budget,<br>-- and re-clusters data that was already fine.<br>OPTIMIZE events;</pre><pre>-- Optimized: compact only what drifted, predicate-filtered,<br>-- Z-ordered on the hot filter column, low concurrency.<br>-- Naive: OPTIMIZE events;  (whole table, nightly, no filter)<br>-- Optimized: target files below 64MB in recent partitions only<br>OPTIMIZE events<br>WHERE event_date &gt;= current_date() - INTERVAL 7 DAYS<br>  AND _metadata.file_size &lt; 67108864<br>ZORDER BY (event_date, user_id);</pre><pre># Naive: spark.conf.set(&quot;spark.databricks.delta.optimize.maxFileSize&quot;, 134217728)<br>#        then OPTIMIZE the entire table — one giant job, reader-visible<br>#        cluster contention, and a 6-hour rewrite of data that was fine.</pre><pre># Optimized: incremental, partition-scoped, off-peak, bounded rewrite<br># Schedule: nightly, 2 AM, max 4 concurrent compaction tasks<br>spark.conf.set(&quot;spark.databricks.delta.optimize.maxFileSize&quot;, &quot;134217728&quot;)  # 128MB target<br>spark.conf.set(&quot;spark.databricks.delta.optimize.minFileSize&quot;, &quot;67108864&quot;)   # only touch &lt;64MB files</pre><pre>spark.sql(&quot;&quot;&quot;<br>  OPTIMIZE events<br>  WHERE event_date &gt;= current_date() - INTERVAL 7 DAYS<br>  ZORDER BY (event_date, user_id)<br>&quot;&quot;&quot;)<br># Readers see atomic snapshot swaps; the 2 AM slot keeps production clear.</pre><p>The backfill took one weekend (the 18-month backlog had to drain once). After that, the nightly incremental ran in 25–40 minutes. Results:</p><ul><li><strong>File count:</strong> 3.8M → 41K (then held steady; the drain matches the inflow).</li><li><strong>Planning time:</strong> 22 minutes → 47 seconds.</li><li><strong>The finance query:</strong> 58 minutes → 9 minutes. Same data, same cluster.</li><li><strong>S3 LIST spend:</strong> ~$900/month → under $40/month.</li><li><strong>Data skipping restored:</strong> the event_date filter now prunes 94% of files instead of 11%.</li></ul><h3>The Catch: Compaction Has Its Own Bills</h3><ul><li><strong>Time travel and retention.</strong> Every compaction commit obsoletes old files. If your VACUUM retention is 7 days, you can no longer time-travel past a compaction boundary. Set retention deliberately; don’t discover this during an incident.</li><li><strong>Z-order decays.</strong> New writes are unclustered by definition. Compaction restores clustering, but if your hot filter column changes (new query pattern), the old Z-order is just expensive decoration. Re-evaluate the Z-order columns quarterly.</li><li><strong>Don’t compact the landing zone obsessively.</strong> The raw ingestion prefix is write-heavy and rarely queried directly. Compacting it nightly is pure write amplification with no read payoff. Compact the <em>serving</em> tables; leave the landing zone chunky.</li><li><strong>Streaming sources need the fix upstream.</strong> If a streaming job writes 2 MB files every trigger, no compaction schedule keeps up gracefully. Fix the trigger interval and foreachBatch output partitioning first — compaction is the drain, not the faucet.</li></ul><h3>Takeaways</h3><ul><li><strong>Plan-time is O(files), not O(bytes):</strong> a 3 TB table in 3,000 files plans in seconds; in 3.8M files it plans in 22 minutes.</li><li><strong>Driver memory is the real listing bill:</strong> S3 LIST costs dollars, but materializing millions of file statuses costs driver GC pauses and OOMs.</li><li><strong>Overlapping min/max ranges are the tax:</strong> tiny files poison data skipping because every file’s range bleeds into its neighbors’.</li><li><strong>Compaction is bin-packing with write amplification:</strong> you pay the full read+rewrite once to buy back every future query.</li><li><strong>Z-order during compaction, or do it twice:</strong> size alone doesn’t restore skipping; clustering does.</li><li><strong>Budget file count like a queue:</strong> size the nightly compaction drain to the daily file inflow, not to the backlog.</li></ul><p>Small files are the compound interest of lazy writes — invisible daily, brutal eventually.</p><p>If this saved you a 3 AM pager incident, <a href="https://medium.com/@chakshu-salgotra">follow me on Medium</a> for Twice a week for the next 3 Months of Advanced Data Engineering &amp; AI. Let’s connect on <a href="https://www.linkedin.com/in/chakshusalgotra/">LinkedIn</a> and <a href="https://x.com/chakshusalgotra">X</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=593e9a3abe2c" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/part-xviii-the-small-files-tax-compaction-strategies-that-dont-stall-593e9a3abe2c">Part XVIII — The Small-Files Tax: Compaction Strategies That Don’t Stall</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part XVII — Checkpointing vs Lineage: The Fault-Tolerance Trade You Misconfigure]]></title>
            <link>https://pub.towardsai.net/part-xvii-checkpointing-vs-lineage-the-fault-tolerance-trade-you-misconfigure-f5b1a5892d68?source=rss-7afde1b096a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5b1a5892d68</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[fault-tolerance]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[chakshu_salgotra]]></dc:creator>
            <pubDate>Thu, 22 Oct 2026 13:31:02 GMT</pubDate>
            <atom:updated>2026-10-22T13:31:02.831Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Spark’s lineage is the most elegant recovery mechanism in distributed computing — until your iterative job’s DAG grows so long that remembering how the data was built costs more than the data itself. Checkpointing cuts the chain. The question is what you pay for the scissors, and most teams get the answer wrong in both directions.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*irItcRWB1Ltwwweaji7l4g.png" /></figure><p><a href="https://chakshu-salgotra.medium.com/how-spark-reads-parquet-2333c7fa728c">Part I</a> | <a href="https://chakshu-salgotra.medium.com/spark-shuffle-partitioning-under-the-hood-of-a-wide-transformation-521907cd5f1c">Part II</a><strong> | </strong><a href="https://medium.com/@chakshu-salgotra/spark-catalyst-optimizer-internals-2b762ce0b2e2">Part III</a> <strong>| </strong><a href="https://pub.towardsai.net/jobs-stages-tasks-how-spark-actually-decomposes-your-query-e50bc8ed768e">Part IV</a> <strong>| </strong><a href="https://pub.towardsai.net/how-spark-manages-memory-the-unified-memory-model-spills-and-aqe-f37f21799fc6?sk=c91e30bf8b55b332b5611b7585b0b883">Part V</a> <strong>| </strong><a href="https://pub.towardsai.net/part-vi-the-spark-join-strategies-broadcast-vs-52a2048203e6?sk=aec7100814c2dbac55341898fb4f0686">Part VI<strong> </strong></a><strong>| </strong><a href="https://pub.towardsai.net/part-vii-data-skew-in-spark-why-199-tasks-finish-in-40-seconds-and-one-runs-for-3-hours-7daef3b1de44?sk=2262448f82bf04acda6dea9bfdeca857">Part VII</a> | <a href="https://pub.towardsai.net/part-viii-batch-vs-streaming-in-spark-one-sql-engine-two-execution-contracts-a4e888cb4748?sk=e8f00a5249b4a1ce3aa992f614e76705">Part VIII </a>| <a href="https://pub.towardsai.net/part-ix-the-anatomy-of-a-spark-application-what-the-driver-executors-and-cluster-manager-ffa2bba60dce?sk=d66767c2ca186aaab54860931975741e">Part IX</a> | <a href="https://pub.towardsai.net/part-x-z-ordering-and-data-clustering-explained-why-your-partitioned-table-still-scans-4b109e016c80?sk=dc7cfcb72d4bd523b6b5ca35f79f2b16">Part X </a>| <a href="https://pub.towardsai.net/part-xi-rdd-vs-dataframe-vs-dataset-why-your-type-safe-refactor-made-spark-3x-slower-99bdba6f7eaa?source=friends_link&amp;sk=3cef58b143ef9d20917ac93e62731ecc">Part XI</a> | <a href="https://chakshu-salgotra.medium.com/da093c04b481?source=friends_link&amp;sk=1f9b68eee7c3c07c233d280b625b5fab">Part XII </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xiii-internal-vs-external-tables-the-drop-table-that-deleted-40tb-4fffae7f740c">Part XIII </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xiv-broadcast-variables-the-driver-bottleneck-nobody-profiles-87189bb691ed">Part XIV</a> | <a href="https://medium.com/towards-artificial-intelligence/part-xv-adaptive-query-execution-what-aqe-actually-rewrites-at-runtime-f010f22ad3fa">Part XV</a> | <a href="https://medium.com/towards-artificial-intelligence/part-xvi-whole-stage-codegen-the-tungsten-trick-behind-2x-speedups-6e0011a07944">Part XVI </a>| <strong>Part XVII</strong></p><h3>The Job That Got Slower Every Single Iteration</h3><p>The entity-resolution job ran 120 iterations. Iteration 1 took 42 seconds. Iteration 30 took 6 minutes. By iteration 90, each pass was taking 14 minutes, and the whole thing was on track to blow past its 12-hour SLA window before iteration 110. Nobody had changed the code in weeks. The data volume was flat. The cluster was healthy. The job was simply, deterministically, getting slower every single time it looped — like a runner accumulating weights.</p><p>Then came the real incident. At 4:12 AM, on iteration 118 of 120, a spot instance was reclaimed and took two executors with it. Spark did exactly what it was designed to do: it recomputed the lost partitions from lineage. The problem was the lineage. To rebuild those partitions, it had to replay 117 iterations of joins and aggregations — every shuffle, every stage, all the way back to the raw input. Recovery took 6 hours and 40 minutes. The two lost executors cost us nearly seven hours of cluster time and a missed downstream SLA that paged three teams.</p><p>The post-mortem was one line long: <em>we had 118 iterations of lineage and zero checkpoints.</em> The DAG the driver was carrying had grown to tens of thousands of nodes. DAGScheduler was spending longer <em>planning</em> each iteration than the executors spent <em>running</em> it — planning cost grows with DAG depth, and we had built the deepest DAG in the company’s history one innocent loop at a time. The driver was also spending 40% of its time in garbage collection just holding the lineage graph (Part V’s memory lesson, applied to the driver instead of the heap).</p><p>So: when does remembering how your data was built become more expensive than the data itself — and what does it actually cost to forget on purpose?</p><h3>The Mechanic Underneath: Lineage Is a Promise, and Promises Compound</h3><p>Spark’s core fault-tolerance idea is disarmingly simple: instead of replicating data, remember <em>how it was built</em>. Every RDD and DataFrame carries its <strong>lineage</strong> — the graph of transformations from the source. Lose a partition, and Spark replays the lineage to rebuild exactly that partition. No replication overhead, no write-ahead cost during the happy path. For a ten-stage ETL, this is nearly free and brilliantly robust.</p><p>But lineage is a promise that compounds. In an iterative algorithm — PageRank, ALS, K-means, our entity-resolution loop — each iteration appends to the DAG. After <em>n</em> iterations, the lineage graph has depth <em>n</em>, and three costs grow with it. First, <strong>recovery cost</strong>: rebuilding a lost partition means replaying every stage back to the last materialization point, so recovery time is roughly <em>lineage depth × per-stage cost</em>. Our 6h40m recovery was not bad luck; it was arithmetic. Second, <strong>scheduling cost</strong>: the DAGScheduler traverses the DAG to plan each job, and planning time grows with graph size — at iteration 90, planning alone took minutes. Third, <strong>driver memory</strong>: the entire lineage graph lives on the driver, and tens of thousands of nodes with their metadata will eventually push the driver into GC hell. Lineage is free until the day it is the most expensive thing in your architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MhTmFsKQOOtkHtlhVBpnLw.png" /></figure><p><strong>Checkpointing</strong> is the deliberate decision to stop paying the compounding cost: materialize the data to stable storage (or executor memory), and <em>cut the lineage</em> at that point. Everything before the checkpoint is forgotten; recovery replays only from the checkpoint forward. You trade write cost now for bounded recovery cost later. The entire art of this topic is choosing where to cut, how often, and which kind of scissors to use — because each choice has a failure mode that teams discover at 4 AM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BHj_rs9nme8Sq-2ew9d1dQ.png" /></figure><h3>Reliable Checkpointing Cuts the Chain — For a Price</h3><p>RDD.checkpoint() (and Dataset.checkpoint()) writes the data to the <strong>checkpoint directory</strong> — HDFS, S3, any reliable filesystem you configure via sparkContext.setCheckpointDir() — and then truncates the lineage, pointing the RDD&#39;s parent at the checkpointed files instead of the full history. Recovery from a checkpoint replays at most the stages since the last cut. In our job, checkpointing every 10 iterations would have bounded recovery to 10 stages instead of 118 — turning 6h40m into roughly 25 minutes.</p><p>The price is write amplification. A reliable checkpoint is a full materialization: every byte of the checkpointed dataset gets written to durable storage, with all the serialization and I/O cost that implies. Checkpoint a 2TB iterative state every 10 iterations over 120 iterations and you have written 24TB to S3 that your lineage would never have written. On S3 specifically, the cost is worse than it looks — no atomic renames, high per-request latency — and checkpoint writes go through the same object-store path as everything else. Teams that checkpoint every iteration “to be safe” often discover their fault-tolerance mechanism is the slowest stage in the job. The checkpoint directory also needs lifecycle management: Spark does not clean up old checkpoints for RDDs automatically, and an unmonitored checkpoint dir is a slow-growing storage bill with your name on it.</p><h3>Local Checkpointing Is Faster and Lies About Durability</h3><p>localCheckpoint() — RDD.localCheckpoint() and Dataset.localCheckpoint() — cuts the lineage too, but materializes the data <strong>on the executors&#39; local disks</strong> (or in memory, depending on storage level) instead of reliable storage. It is dramatically faster than a reliable checkpoint: no HDFS/S3 round trip, just a local write. For iterative ML loops where the dominant cost is DAG growth rather than executor loss, this is often the right tool.</p><p>But read the contract carefully: local checkpoint data dies with the executor. If the executor holding your locally-checkpointed partition is lost, Spark falls back to recomputing from the <em>truncated</em> lineage — which still exists underneath, because local checkpointing does not delete the original lineage, it just stops <em>extending</em> it. In practice this means local checkpointing bounds scheduling cost and driver memory (the DAG stops growing) but only partially bounds recovery cost (the underlying lineage is still replayable, just not re-traversed for planning). It is a scheduler optimization wearing a fault-tolerance costume. Our 4 AM executor loss would still have hurt with local checkpoints — less, but not the 25-minute story. Know which guarantee you are actually buying.</p><h3>RDD Checkpointing Is Lazy; DataFrame Checkpointing Is Not</h3><p>A gotcha that has bitten every team exactly once: RDD.checkpoint() is <strong>lazy</strong> — it marks the RDD for checkpointing, but nothing is written until the next action triggers computation. Dataset.checkpoint() (the DataFrame API), by contrast, is <strong>eager</strong> — it triggers a job immediately and blocks until the checkpoint is materialized. Mix them in the same codebase without knowing this and you get either checkpoints that never materialize (RDD marked but no action follows before the driver moves on) or mysterious blocking jobs in the middle of your pipeline (DataFrame checkpoint firing eagerly inside a loop). The eager DataFrame behavior is usually what you want in an iterative loop — you want the cut to happen <em>now</em>, deterministically, every <em>k</em> iterations — but you must budget for it: each eager checkpoint inside the loop is a full extra job with its own stages. Count it in your iteration-time math or your &quot;optimization&quot; will look like a regression in the UI.</p><h3>Streaming Checkpoints Are a Different Animal</h3><p>Structured Streaming’s checkpointing (Part VIII’s territory) shares the name but not the mechanics. The <strong>checkpoint location</strong> (checkpointLocation option, or spark.sql.streaming.checkpointLocation) stores three things: source offsets, the write-ahead log of processed batches, and the serialized <strong>state store</strong> for stateful operations. Recovery replays from the last committed offset — exactly-once semantics depend on the sink supporting idempotent writes, which is why the docs pair checkpointing with foreachBatch idempotency patterns. Two sharp edges: checkpoint state is <strong>tied to your query&#39;s code</strong> — change the projection, the schema, or the stateful logic incompatibly and the old checkpoint becomes unloadable, forcing you to start from a fresh location (or face cryptic deserialization errors); and the default spark.sql.streaming.minBatchesToRetain (100) bounds how much history the checkpoint keeps, which interacts with late-data handling. Streaming checkpointing is not optional — without a checkpoint location, a streaming query cannot recover at all — but it is also not free: state serialization for large keyed state is a real, ongoing write cost on every trigger.</p><h3>The Fix, and What It Cost</h3><p>The naive version was our original loop — 120 iterations, pure lineage, no cuts:</p><pre># Naive: 120 iterations of pure lineage. DAG depth grows to 120, driver holds<br># the entire graph, and ONE executor loss replays ALL 118 prior iterations.<br># result = result.join(updates, &quot;id&quot;).groupBy(&quot;id&quot;).agg(...)  # in a loop, no checkpoint</pre><pre># Optimized: cut the lineage every 10 iterations with a RELIABLE checkpoint.<br># Recovery is now bounded to &lt;= 10 stages; driver DAG stays shallow.<br>sc.setCheckpointDir(&quot;s3://our-bucket/checkpoints/entity-resolve&quot;)  # reliable store</pre><pre>result = base_df<br>for i in range(120):<br>    result = result.join(updates, &quot;id&quot;).groupBy(&quot;id&quot;).agg(...)  # the iterative step<br>    if i % 10 == 0 and i &gt; 0:<br>        result = result.checkpoint()  # eager: materializes NOW, truncates lineage<br>        # checkpoint dir lifecycle: old cuts cleaned by retention job, not Spark</pre><p>For the tight inner loop of the ML variant we used localCheckpoint() instead — the DAG-growth problem dominated and executor loss was rare on our on-demand pool — reserving reliable checkpoints for every 30th iteration as the durability backstop. Two scissors, each used where its guarantee matches the risk.</p><p>The measured result on the next full run: <strong>per-iteration time stabilized at 55–70 seconds across all 120 iterations</strong> (vs. degrading from 42s to 14min), <strong>total runtime 11h20m → 2h35m</strong>, and a chaos test — killing two executors at iteration 100 — <strong>recovered in 22 minutes instead of 6h40m</strong>. The checkpoint writes added ~9 minutes of I/O across the whole run. Nine minutes of writes bought back nine hours of compounding lineage cost. That is the entire trade, expressed as a number.</p><h3>The Catch: Checkpointing Can Quietly Make Everything Worse</h3><p>Four ways the cure becomes the disease. First, <strong>checkpointing too often</strong>: every reliable checkpoint is a full write of the dataset. Checkpoint every iteration of a 2TB state and you have built a write-amplification machine that dwarfs the lineage cost you were avoiding — profile the write cost per checkpoint before choosing the interval. Second, <strong>checkpointing on slow storage</strong>: S3 checkpoint writes without S3A tuning (or worse, without fs.s3a committer settings) serialize your &quot;fast&quot; iterative loop behind object-store latency; the checkpoint dir belongs on fast storage or a tuned path. Third, <strong>localCheckpoint&#39;s false confidence</strong>: it does not bound recovery the way teams assume — the underlying lineage survives, and executor loss still replays it. If your post-mortem says &quot;but we had checkpoints,&quot; check <em>which kind</em>. Fourth, <strong>streaming checkpoint incompatibility</strong>: evolving a stateful streaming query&#39;s logic without migrating the checkpoint location produces failures that look like data corruption but are really version skew — treat checkpoint locations as versioned artifacts, not scratch space. Checkpointing is a sharp tool. Grip it by the handle.</p><h3>Takeaways</h3><ul><li><strong>Lineage cost compounds with DAG depth:</strong> recovery time ≈ depth × stage cost, scheduling time grows with graph size, and the driver holds all of it. Iterative jobs must cut the chain.</li><li><strong>Reliable checkpointing bounds recovery for a write price:</strong> every cut is a full materialization. Size the interval from measured write cost, not from superstition.</li><li><strong>Local checkpointing is a scheduler optimization, not durability:</strong> it stops DAG growth but the underlying lineage survives executor loss. Buy it for planning cost, not for recovery.</li><li><strong>RDD checkpointing is lazy, DataFrame checkpointing is eager:</strong> a marked-but-never-materialized RDD checkpoint protects nothing; an eager DataFrame checkpoint inside a loop is a full extra job per cut.</li><li><strong>Streaming checkpoints are versioned state:</strong> offsets, WAL, and serialized state stores. Change query logic incompatibly and the old checkpoint is unloadable — treat locations as versioned.</li><li><strong>Clean up after yourself:</strong> Spark does not retire old RDD checkpoints. An unmonitored checkpoint directory is a storage bill growing in the dark.</li></ul><p>Lineage remembers everything for free, until the remembering costs more than the data. Checkpointing is how you choose what to forget — deliberately, measurably, and before 4 AM chooses for you.</p><p>If this saved you a 3 AM pager incident, <a href="https://medium.com/@chakshu-salgotra">follow me on Medium</a> for Twice a week for the next 3 Months of Advanced Data Engineering &amp; AI. Let’s connect on <a href="https://www.linkedin.com/in/chakshusalgotra/">LinkedIn</a> and <a href="https://x.com/chakshusalgotra">X</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5b1a5892d68" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/part-xvii-checkpointing-vs-lineage-the-fault-tolerance-trade-you-misconfigure-f5b1a5892d68">Part XVII — Checkpointing vs Lineage: The Fault-Tolerance Trade You Misconfigure</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part XVI — Whole-Stage Codegen: The Tungsten Trick Behind 2x Speedups]]></title>
            <link>https://pub.towardsai.net/part-xvi-whole-stage-codegen-the-tungsten-trick-behind-2x-speedups-6e0011a07944?source=rss-7afde1b096a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e0011a07944</guid>
            <category><![CDATA[spark-internals]]></category>
            <category><![CDATA[tungsten]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[chakshu_salgotra]]></dc:creator>
            <pubDate>Wed, 21 Oct 2026 21:01:02 GMT</pubDate>
            <atom:updated>2026-10-21T21:01:02.442Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Your Spark job is not slow because of your data. It is slow because Spark is interpreting your query one virtual function call at a time — unless Tungsten’s code generator compiled the whole stage into a single tight loop. Here is how that compilation works, what silently disables it, and how to get the 2x back.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QhWajxFdgqw4xGZBf_vf5g.png" /></figure><p><a href="https://chakshu-salgotra.medium.com/how-spark-reads-parquet-2333c7fa728c">Part I</a> | <a href="https://chakshu-salgotra.medium.com/spark-shuffle-partitioning-under-the-hood-of-a-wide-transformation-521907cd5f1c">Part II</a><strong> | </strong><a href="https://medium.com/@chakshu-salgotra/spark-catalyst-optimizer-internals-2b762ce0b2e2">Part III</a> <strong>| </strong><a href="https://pub.towardsai.net/jobs-stages-tasks-how-spark-actually-decomposes-your-query-e50bc8ed768e">Part IV</a> <strong>| </strong><a href="https://pub.towardsai.net/how-spark-manages-memory-the-unified-memory-model-spills-and-aqe-f37f21799fc6?sk=c91e30bf8b55b332b5611b7585b0b883">Part V</a> <strong>| </strong><a href="https://pub.towardsai.net/part-vi-the-spark-join-strategies-broadcast-vs-52a2048203e6?sk=aec7100814c2dbac55341898fb4f0686">Part VI<strong> </strong></a><strong>| </strong><a href="https://pub.towardsai.net/part-vii-data-skew-in-spark-why-199-tasks-finish-in-40-seconds-and-one-runs-for-3-hours-7daef3b1de44?sk=2262448f82bf04acda6dea9bfdeca857">Part VII</a> | <a href="https://pub.towardsai.net/part-viii-batch-vs-streaming-in-spark-one-sql-engine-two-execution-contracts-a4e888cb4748?sk=e8f00a5249b4a1ce3aa992f614e76705">Part VIII </a>| <a href="https://pub.towardsai.net/part-ix-the-anatomy-of-a-spark-application-what-the-driver-executors-and-cluster-manager-ffa2bba60dce?sk=d66767c2ca186aaab54860931975741e">Part IX</a> | <a href="https://pub.towardsai.net/part-x-z-ordering-and-data-clustering-explained-why-your-partitioned-table-still-scans-4b109e016c80?sk=dc7cfcb72d4bd523b6b5ca35f79f2b16">Part X </a>| <a href="https://pub.towardsai.net/part-xi-rdd-vs-dataframe-vs-dataset-why-your-type-safe-refactor-made-spark-3x-slower-99bdba6f7eaa?source=friends_link&amp;sk=3cef58b143ef9d20917ac93e62731ecc">Part XI</a> | <a href="https://chakshu-salgotra.medium.com/da093c04b481?source=friends_link&amp;sk=1f9b68eee7c3c07c233d280b625b5fab">Part XII </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xiii-internal-vs-external-tables-the-drop-table-that-deleted-40tb-4fffae7f740c">Part XIII </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xiv-broadcast-variables-the-driver-bottleneck-nobody-profiles-87189bb691ed">Part XIV</a> | <a href="https://medium.com/towards-artificial-intelligence/part-xv-adaptive-query-execution-what-aqe-actually-rewrites-at-runtime-f010f22ad3fa">Part XV</a> | Part XVI</p><h3>The Query That Got 3x Slower After We “Optimized” It</h3><p>It was a Tuesday, which is worse than a 3 AM page because nobody expects a Tuesday. Our nightly clickstream ETL — 9.2TB of raw JSON events, parsed, normalized, and written to Delta — had been running in 41 minutes for months. We migrated it to a new cluster, bumped Spark from 3.3 to 3.5, and swapped our hand-rolled timestamp parser for a “cleaner” Scala UDF that handled three legacy date formats in one place. The next morning the job had taken 2 hours and 7 minutes. Nothing else changed. Same data volume, same cluster shape, same partitions.</p><p>The Spark UI told a story I have now seen a dozen times. The heavy stage — the one that parsed JSON, filtered bots, and projected 40 columns down to 12 — showed task deserialization fine, GC fine, no skew, no spill. Just CPU time that had tripled. The executors were burning cycles doing… nothing visible. No shuffle wait. No I/O wait. Pure, unexplained CPU burn on a stage that was supposed to be embarrassingly parallel and embarrassingly simple.</p><p>I pulled the plan with EXPLAIN. And there it was, in the most boring diff imaginable: the stage that used to read *(4) Project [ ... ] — note the asterisk — now read plain Project [ ... ]. No asterisk. No WholeStageCodegen. The stage had lost its compilation, and the culprit was our shiny new Scala UDF sitting inside the projection. One expression, unsupported by the code generator, had poisoned the entire stage back into the interpreted iterator loop. The &quot;cleaner&quot; code was costing us 86 minutes a night — roughly $4,100 a month in wasted compute at our cluster rates — because of a single asterisk in a query plan nobody checks.</p><p>So: what does that asterisk actually mean, what machine is doing the compiling, and what are all the quiet ways your pipeline can lose it?</p><h3>The Mechanic Underneath: Your Query Is a Tree, and Trees Can Be Compiled</h3><p>Every Spark SQL query you write becomes an <strong>expression tree</strong>. filter($&quot;status&quot; === &quot;click&quot;) is not a function call — it is a tree node EqualTo with two children, sitting inside a Filter operator node, sitting inside a Project operator node. In the classic <strong>Volcano iterator model</strong> that Spark inherited, each operator is an object with a next() method, and executing the query means calling next() down the tree, one row at a time, one operator at a time. A filter on top of a scan on top of a project means three virtual method dispatches <em>per row</em>. On 9.2TB — roughly 40 billion rows — that is hundreds of billions of virtual calls, each one defeating branch prediction and keeping your data bouncing between CPU caches instead of living in registers.</p><p><strong>Whole-stage codegen</strong> deletes that overhead in one move. Instead of interpreting the operator tree at runtime, Spark walks the tree <em>once</em> and generates a single Java method that does the whole pipeline: read a row, apply the filter, apply the projection, write the output — in one tight loop, no virtual dispatch, data staying in registers. That generated source is compiled in-memory by <strong>Janino</strong>, a lightweight Java compiler embedded in Spark, and the resulting bytecode is what your executors actually run. The * in the plan marks stages that got this treatment. This is the Tungsten project (Spark 2.0+) doing what it promised: treating your query like a compiler treats a program, not like an interpreter treats a script.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SJxaQ1OgamXBwgzRHY0xbg.png" /></figure><p>The key insight: codegen operates on <em>pipelines</em> of operators, not individual operators. Catalyst finds maximal subtrees of the physical plan that can be fused — typically everything between two shuffle boundaries — and generates one function per subtree. Your 40-column projection, your filter, your expression evaluation: all of it becomes straight-line code inside a single while (input.hasNext()) loop. That is where the 2x comes from. Not from better algorithms. From deleting the interpreter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*il9poAA2jCAAE_YEH8YQyA.png" /></figure><h3>The Volcano Model Is Elegant, and It Costs You a Virtual Call per Row</h3><p>The iterator model is beautiful computer science. Each operator implements next(): the scan&#39;s next() reads a row, the filter&#39;s next() loops until a row passes, the project&#39;s next() transforms it. Composition is trivial, and every database textbook teaches it. But elegance is not free. A virtual call per operator per row means the CPU cannot inline across operator boundaries, cannot keep the row in registers across the pipeline, and pays branch-misprediction penalties on every dispatch. For a pipeline of five operators over a billion rows, you are paying five billion virtual dispatches before you have done a single useful comparison. Tungsten&#39;s bet — correct, as it turned out — was that this dispatch tax, not your logic, dominates CPU-bound stages. When your stage is filter-and-project heavy with no shuffle, the interpreter <em>is</em> the bottleneck.</p><h3>Codegen Collapses the Pipeline Into a Single Function</h3><p>Here is what actually happens when Catalyst plans your query. After analysis and optimization, the physical planner looks for <strong>pipelining</strong> opportunities: chains of operators like Filter → Project → Filter that process rows one at a time without materializing intermediate results. It wraps each maximal chain in a WholeStageCodegenExec node — that is the * you see in EXPLAIN output, e.g. *(4) Project [...]. Then the code generator walks the expression trees inside and emits Java source: the filter predicate becomes an if statement, the projection becomes direct field assignments, null checks become explicit branches. Janino compiles it, the JVM JIT-compiles it <em>again</em> into native code, and your 40-billion-row stage runs as machine code with the row living in registers the whole way through. Two compilation layers, zero virtual calls. The measurable result on CPU-bound stages is routinely 1.5–2.5x — which is why losing the asterisk tripled our stage time: we did not just lose an optimization, we fell off a cliff back into the interpreter.</p><h3>One Bad Expression Poisons the Entire Stage</h3><p>This is the part that burns people. Codegen is all-or-nothing <em>per stage</em>. The generator supports a fixed set of expressions; if any single expression in the pipeline cannot be generated, the <strong>entire stage</strong> falls back to interpreted mode. The usual suspects:</p><ul><li><strong>Python UDFs</strong> (BatchEvalPython) — never codegen&#39;d. The data crosses a process boundary to Python per batch; there is nothing to compile.</li><li><strong>Scala/Java UDFs</strong> (ScalaUDF) — the generator cannot see inside your arbitrary function, so the whole stage falls back.</li><li>A handful of exotic expressions and <strong>complex UDT handling</strong> in older versions.</li><li>Extremely wide projections can hit generator limits (more on that below).</li></ul><p>How do you spot it? EXPLAIN (or the SQL tab in the UI): codegen&#39;d stages carry the * prefix — *(2) Filter, *(4) Project. A stage that fell back shows the operators bare, sometimes with an explicit CODEGEN_FALLBACK marker in newer versions, and the plan will show an InputAdapter where the whole-stage node would have been. The config spark.sql.codegen.wholeStage (default true) is the master switch, and spark.sql.codegen.fallback (default true) controls whether Spark silently falls back to interpreted mode instead of failing. Silent fallback is the default — which is exactly why our Tuesday was a surprise. The job did not error. It just got slow, politely.</p><h3>The JVM Has a 64KB Opinion About Your Generated Code</h3><p>Even when every expression is supported, the generated method can hit hard JVM limits. A Java method cannot exceed 64KB of bytecode, and a 200-column projection with complex expressions can generate a method that blows past it. Spark handles this with method splitting, governed by two configs you should know exist: spark.sql.codegen.hugeMethodLimit (default 65535, the bytecode-size threshold that triggers splitting) and spark.sql.codegen.maxFields (default 100, which caps generated fields per class before splitting into multiple classes). When you see a stage that <em>should</em> be codegen&#39;d but is not, and there is no UDF in sight, check the driver logs for codegen compilation warnings — oversized generated code is the usual second culprit after UDFs. The fix is the same shape as everything in this series: narrower projections (Part I&#39;s Parquet lesson rhymes here — read fewer columns, generate smaller methods).</p><h3>The Fix, and What It Cost</h3><p>The naive code was our “cleaner” timestamp parser — a Scala UDF registered once and used in the projection:</p><pre># Naive: a Scala UDF for legacy timestamp formats. One unsupported expression<br># poisons the ENTIRE stage back to the Volcano iterator loop - no more *(n).<br># spark.udf.register(&quot;parse_ts&quot;, lambda s: ...)  # ScalaUDF: codegen fallback</pre><pre># Optimized: built-in expressions only. Every expression here is codegen-safe,<br># so the whole Filter -&gt; Project pipeline compiles into ONE generated function.<br>parsed = (<br>    raw<br>    .filter(F.col(&quot;event_type&quot;) == &quot;click&quot;)                      # codegen: *(n) Filter<br>    .withColumn(<br>        &quot;event_ts&quot;,<br>        F.coalesce(                                               # codegen-safe branch<br>            F.to_timestamp(&quot;ts_raw&quot;, &quot;yyyy-MM-dd HH:mm:ss&quot;),<br>            F.to_timestamp(&quot;ts_raw&quot;, &quot;MM/dd/yyyy HH:mm&quot;),<br>            F.to_timestamp(&quot;ts_raw&quot;, &quot;yyyyMMddHHmmss&quot;),<br>        ),<br>    )<br>    .select(&quot;user_id&quot;, &quot;event_ts&quot;, &quot;url&quot;, &quot;session_id&quot;)          # narrow: small method<br>)</pre><pre>spark.conf.set(&quot;spark.sql.codegen.wholeStage&quot;, &quot;true&quot;)  # default true; verify, don&#39;t assume<br># EXPLAIN should show: *(4) Project [...], *(2) Filter [...]  (asterisks = compiled)</pre><p>Same semantics, three built-in format attempts instead of one opaque UDF. The plan regained its asterisks on every stage. The measured result on the next nightly run: <strong>stage time 2h07m → 38 minutes, total job 2h07m → 41 minutes</strong> — the full 86 minutes recovered, plus 3 from the narrower projection. CPU time on the stage dropped 68%. Monthly compute cost for the pipeline went from ~$6,200 back to ~$2,100. The UDF version was not wrong — it was uncompilable, and in Spark those are very different things.</p><h3>The Catch: Codegen Fixes the Interpreter, Not Your Job</h3><p>Three honest limits. First, codegen only accelerates <strong>CPU-bound, pipelined stages</strong>. If your stage is shuffle-bound (Part II) or skew-bound (Part VII), compiling the filter loop buys you almost nothing — the bottleneck is data movement, and no generated bytecode fixes a straggler partition. Second, codegen has a <strong>compile-time cost</strong>: on enormous plans with thousands of expressions, Janino compilation itself can take minutes and bloat driver memory. If your job has a suspiciously long planning phase, spark.sql.codegen.wholeStage=false is a legitimate diagnostic, not heresy. Third, generated code is <strong>harder to debug</strong> — stack traces point at GeneratedClass$...$ line numbers that map to nothing you wrote. When a codegen&#39;d stage throws, the first step is often disabling codegen to get a readable stack trace, fixing the logic, and re-enabling. The asterisk is a performance feature with a debuggability tax. Pay it knowingly.</p><h3>Takeaways</h3><ul><li><strong>The </strong><strong>* is the plan telling you it compiled:</strong> *(4) Project means one generated function; a bare Project means the Volcano interpreter loop. Read your EXPLAIN output like a compiler engineer.</li><li><strong>Fallback is per-stage and all-or-nothing:</strong> one unsupported expression — a Scala UDF, a Python UDF — poisons the entire pipeline back to interpreted mode.</li><li><strong>Silent fallback is the default:</strong> spark.sql.codegen.fallback=true means the job gets slow instead of failing. slowness without errors is the signature symptom.</li><li><strong>Built-ins are a performance feature:</strong> coalesce + to_timestamp with multiple formats does what your UDF did, and the generator can see through it.</li><li><strong>Generated methods hit JVM limits:</strong> 64KB per method; spark.sql.codegen.hugeMethodLimit and maxFields govern the splitting. Narrow projections help twice.</li><li><strong>Codegen is for CPU-bound stages:</strong> it will not save a shuffle-bound or skew-bound job. Fix the bottleneck you actually have (Parts II and VII).</li></ul><p>Codegen does not make your logic faster. It deletes the interpreter standing between your logic and the CPU — and one invisible expression can put the interpreter right back.</p><p>If this saved you a 3 AM pager incident, <a href="https://medium.com/@chakshu-salgotra">follow me on Medium</a> for Twice a week for the next 3 Months of Advanced Data Engineering &amp; AI. Let’s connect on <a href="https://www.linkedin.com/in/chakshusalgotra/">LinkedIn</a> and <a href="https://x.com/chakshusalgotra">X</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e0011a07944" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/part-xvi-whole-stage-codegen-the-tungsten-trick-behind-2x-speedups-6e0011a07944">Part XVI — Whole-Stage Codegen: The Tungsten Trick Behind 2x Speedups</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part XV — Adaptive Query Execution: What AQE Actually Rewrites at Runtime]]></title>
            <link>https://pub.towardsai.net/part-xv-adaptive-query-execution-what-aqe-actually-rewrites-at-runtime-f010f22ad3fa?source=rss-7afde1b096a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/f010f22ad3fa</guid>
            <category><![CDATA[query-optimization]]></category>
            <category><![CDATA[spark-internals]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[adaptive-query-execution]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[chakshu_salgotra]]></dc:creator>
            <pubDate>Tue, 20 Oct 2026 13:01:04 GMT</pubDate>
            <atom:updated>2026-10-20T13:01:04.491Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Spark plans your query before it has seen your data. AQE is the engine’s admission that the planner was guessing — and the machinery that lets the runtime rewrite the plan after every shuffle, using statistics that only exist once the bytes have actually moved.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pH9GBCwsA1f-FksCYLEimA.png" /></figure><p><a href="https://chakshu-salgotra.medium.com/how-spark-reads-parquet-2333c7fa728c">Part I</a> | <a href="https://chakshu-salgotra.medium.com/spark-shuffle-partitioning-under-the-hood-of-a-wide-transformation-521907cd5f1c">Part II</a><strong> | </strong><a href="https://medium.com/@chakshu-salgotra/spark-catalyst-optimizer-internals-2b762ce0b2e2">Part III</a> <strong>| </strong><a href="https://pub.towardsai.net/jobs-stages-tasks-how-spark-actually-decomposes-your-query-e50bc8ed768e">Part IV</a> <strong>| </strong><a href="https://pub.towardsai.net/how-spark-manages-memory-the-unified-memory-model-spills-and-aqe-f37f21799fc6?sk=c91e30bf8b55b332b5611b7585b0b883">Part V</a> <strong>| </strong><a href="https://pub.towardsai.net/part-vi-the-spark-join-strategies-broadcast-vs-52a2048203e6?sk=aec7100814c2dbac55341898fb4f0686">Part VI<strong> </strong></a><strong>| </strong><a href="https://pub.towardsai.net/part-vii-data-skew-in-spark-why-199-tasks-finish-in-40-seconds-and-one-runs-for-3-hours-7daef3b1de44?sk=2262448f82bf04acda6dea9bfdeca857">Part VII</a> | <a href="https://pub.towardsai.net/part-viii-batch-vs-streaming-in-spark-one-sql-engine-two-execution-contracts-a4e888cb4748?sk=e8f00a5249b4a1ce3aa992f614e76705">Part VIII </a>| <a href="https://pub.towardsai.net/part-ix-the-anatomy-of-a-spark-application-what-the-driver-executors-and-cluster-manager-ffa2bba60dce?sk=d66767c2ca186aaab54860931975741e">Part IX</a> | <a href="https://pub.towardsai.net/part-x-z-ordering-and-data-clustering-explained-why-your-partitioned-table-still-scans-4b109e016c80?sk=dc7cfcb72d4bd523b6b5ca35f79f2b16">Part X </a>| <a href="https://pub.towardsai.net/part-xi-rdd-vs-dataframe-vs-dataset-why-your-type-safe-refactor-made-spark-3x-slower-99bdba6f7eaa?source=friends_link&amp;sk=3cef58b143ef9d20917ac93e62731ecc">Part XI</a> | <a href="https://chakshu-salgotra.medium.com/da093c04b481?source=friends_link&amp;sk=1f9b68eee7c3c07c233d280b625b5fab">Part XII </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xiii-internal-vs-external-tables-the-drop-table-that-deleted-40tb-4fffae7f740c">Part XIII </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xiv-broadcast-variables-the-driver-bottleneck-nobody-profiles-87189bb691ed">Part XIV</a> | <strong>Part XV</strong></p><h3>The Job That Ran 4 Hours 47 Minutes Every Night for a Year</h3><p>Friday, 1:12 AM. The clickstream sessionization job — 1.8TB of raw events, keyed by session_id, shuffled, windowed, written out — finished in 4 hours 47 minutes. It had finished in roughly that time every night for a year. The on-call runbook said &quot;sessionization is slow, it&#39;s just big,&quot; and everyone believed it, because the Spark UI showed the same shape every night: 200 shuffle partitions, three of them glowing red, each holding ~78% of the data between them while 197 partitions finished in minutes and sat idle.</p><p>The standard playbook had been tried. More executors: no change — the three hot partitions still had to be processed serially by three tasks. spark.sql.shuffle.partitions=2000: worse, now there were 2,000 tiny files and the same three hot keys. Salting (Part VII&#39;s playbook): worked, but it meant rewriting the sessionization logic around salted keys and un-salting downstream, and nobody wanted to touch a pipeline that &quot;worked.&quot;</p><p>Then someone noticed a config line inherited from the Spark 2.4 migration, three years stale: spark.sql.adaptive.enabled=false. AQE had shipped in Spark 3.0, been enabled by default in 3.2, and this cluster had been carrying the opt-out like a vestigial organ. We flipped it on for the next run. The job finished in 52 minutes. Same data, same cluster, same query. The planner had been flying blind for a year, and the fix was deleting one line.</p><p>So: what does AQE actually rewrite at runtime, which knobs control each rewrite, and where does the adaptive machinery run out of road?</p><h3>The Mechanic Underneath: The Plan Is Re-Planned After Every Shuffle</h3><p>The static Spark planner (Part III’s Catalyst) optimizes your query once, before execution, using table statistics that are frequently stale, missing, or lies. It picks join strategies, partition counts, and build sides based on estimates. AQE replaces that single planning moment with a loop:</p><p><strong>Execute a stage → collect runtime statistics from the actual shuffle output → re-optimize the rest of the plan → repeat.</strong></p><p>The unit of adaptivity is the <strong>query stage</strong>: a subtree of the plan that ends at a shuffle boundary. AQE wraps each one in a QueryStageExec, runs it, and harvests real numbers — actual partition sizes, actual row counts — from the shuffle files that were just written. The remaining plan is then re-planned with truth instead of estimates. In the Spark UI&#39;s SQL tab you&#39;ll see it as AdaptiveSparkPlan with collapsed stages; each collapse point is a place where the plan changed its mind.</p><p>Three rewrites happen in that loop, each controlled by its own configs, each fixing a different way the static planner guesses wrong. They are: coalescing shuffle partitions, switching join strategies, and splitting skewed partitions.</p><h3>Coalescing: 200 Empty Partitions Were Never a Plan</h3><p>spark.sql.shuffle.partitions (default 200) is a static guess made before the query runs. Set it for your biggest shuffle and every small shuffle pays the tax: 200 partitions for a 50MB shuffle means 200 tasks, 200 files, and scheduling overhead that dwarfs the actual work. Set it for the small shuffle and the big one gets 200 giant partitions.</p><p>With spark.sql.adaptive.coalescePartitions.enabled=true (default), AQE measures the actual shuffle output after the stage completes and merges adjacent small partitions until each is close to spark.sql.adaptive.advisoryPartitionSizeInBytes (default <strong>64MB</strong>). Your 50MB shuffle becomes 1 partition. Your 1.8TB shuffle keeps its parallelism. One config, both problems gone.</p><p>The coalescing respects a floor: it won’t merge below the advisory size target, and spark.sql.adaptive.coalescePartitions.initialPartitionNum (defaulting to spark.sql.shuffle.partitions) caps how many it starts from. The practical upshot is that spark.sql.shuffle.partitions stops being a tuning knob you agonize over and becomes a rough upper bound. Set it generously; AQE trims.</p><h3>Join Switching: Sort-Merge at Plan Time, Broadcast at Runtime</h3><p>Part VI covered the static join strategy choice: broadcast hash join for small build sides, sort-merge for everything else. The static planner makes that call from estimated sizes. When the estimate is wrong — the “small” table is actually 40GB, or the “huge” table filtered down to 3MB — you’re locked into the wrong algorithm for the entire query.</p><p>AQE re-evaluates the join strategy after each stage using <strong>runtime statistics</strong>. The classic win: the planner chose sort-merge join because the build side was estimated at 2GB, but after filters and the actual shuffle the build side is 8MB. AQE converts the sort-merge join into a <strong>broadcast hash join</strong> mid-flight — no shuffle of the big table, no re-plan by you. The threshold it uses is spark.sql.adaptive.autoBroadcastJoinThreshold, which defaults to the familiar spark.sql.autoBroadcastJoinThreshold (10MB). It can also demote to a shuffled hash join where that fits.</p><p>This is the feature that quietly retires a whole class of BROADCAST hints. Part XIV showed what happens when a hint fossilizes a wrong size assumption; AQE&#39;s runtime join selection is the engine doing that decision with fresh data instead of your eighteen-month-old guess. It doesn&#39;t make broadcast free — the driver still collects (Part XIV&#39;s bottleneck is still real) — but it makes the <em>decision</em> correct.</p><h3>Skew Handling: Splitting the Partition the Planner Couldn’t See</h3><p>This is the one that fixed our sessionization job. AQE detects skew <strong>after a shuffle stage completes</strong>, by looking at actual partition sizes: a partition is skewed if it’s larger than spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes (default <strong>256MB</strong>) <em>and</em> larger than spark.sql.adaptive.skewJoin.skewedPartitionFactor (default <strong>5</strong>) times the median partition size. Both conditions — absolute and relative — must hold, which is why it doesn&#39;t fire on uniformly large shuffles.</p><p>When it fires, on a sort-merge or shuffled-hash join, AQE <strong>splits the skewed partition into sub-partitions</strong> and replicates the corresponding partition of the <em>other</em> side to each split. The hot key’s work fans out across many tasks instead of serializing on one. In our job, the three monster partitions — 78% of 1.8TB — were split into dozens of pieces, each paired with a replicated copy of the small side. The 4h47m wall clock was three tasks; the 52-minute wall clock was hundreds.</p><p>Two details matter. First, skew splitting only applies to <strong>sort-merge and shuffled-hash joins</strong> — if AQE already converted the join to broadcast, there’s nothing to split, because every executor holds the full build side anyway. Second, the split count derives from the skewed partition’s size relative to the advisory partition size, so advisoryPartitionSizeInBytes quietly controls skew handling granularity too.</p><h3>Reading an Adaptive Plan: Where the Rewrites Show Up</h3><p>AQE is invisible if you don’t know where to look, which is why teams run with it for years without realizing what it’s doing — or run <em>without</em> it for years without realizing what they’re missing.</p><p>In the Spark UI’s SQL tab, an adaptive query shows AdaptiveSparkPlan at the root, with stages that render <strong>collapsed</strong>: each collapsed node is a QueryStageExec whose final plan was chosen at runtime. Expand the final stage and you&#39;ll see AQEShuffleRead nodes where the static plan had plain Exchange — those are the reads that got coalesced or skew-split. The &quot;Details&quot; for a stage show the runtime statistics AQE collected: actual size, actual rows.</p><p>The fastest diagnostic: EXPLAIN the same query with spark.sql.adaptive.enabled true and false, and diff them. The non-adaptive plan is what Catalyst guessed. The adaptive plan&#39;s final form is what the data required. The distance between them is the value AQE added — and on our sessionization job, that distance was 200 static partitions versus 34 coalesced ones with three skew splits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5gJtc3rFp5jHzCCEYOz6Gg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bTlGb6W8q7369-0cEr6JgQ.png" /></figure><h3>The Fix, and What It Cost</h3><p>The naive state of the sessionization pipeline: AQE disabled by a three-year-old migration config, a static partition count tuned by superstition, and a manual salting design doc nobody wanted to implement.</p><pre># Naive: static planning with a fossilized opt-out. The planner guesses 200<br># partitions and sort-merge join from stale stats; skew is discovered by humans<br># at 1 AM instead of by the engine at runtime.<br>spark.conf.set(&quot;spark.sql.adaptive.enabled&quot;, &quot;false&quot;)  # carried over from the Spark 2.4 migration<br>spark.conf.set(&quot;spark.sql.shuffle.partitions&quot;, &quot;200&quot;)  # tuned once, wrong for every query shape</pre><pre>sessions = (<br>    events<br>    .groupBy(&quot;session_id&quot;)<br>    .agg(...)  # 3 partitions hold 78% of 1.8TB; 197 sit idle<br>)<br># Result: 4h47m nightly, three red partitions in the UI, &quot;just big&quot; in the runbook.</pre><p>The engine-aware fix: delete the opt-out, set the advisory size to match the workload, and let the runtime see the skew.</p><pre># Optimized: let the runtime plan with real statistics. No query rewrite needed.<br>spark.conf.set(&quot;spark.sql.adaptive.enabled&quot;, &quot;true&quot;)  # default since 3.2; we had opted out<br>spark.conf.set(&quot;spark.sql.adaptive.advisoryPartitionSizeInBytes&quot;, 128 * 1024 * 1024)  # 128MB: our files are large<br>spark.conf.set(&quot;spark.sql.adaptive.skewJoin.enabled&quot;, &quot;true&quot;)  # default true; explicit for the runbook<br># skewJoin.skewedPartitionThresholdInBytes=256MB, skewedPartitionFactor=5: defaults were fine</pre><pre>sessions = (<br>    events<br>    .groupBy(&quot;session_id&quot;)<br>    .agg(...)  # identical query text<br>)<br># AQE coalesced 200 -&gt; 34 partitions, skew-split the 3 hot ones, runtime: 52 minutes.</pre><p>No salting, no query rewrite, no logic changes downstream. The query text is byte-identical; only the planning loop changed.</p><p><strong>Before/after:</strong> 4h47m wall clock → <strong>52 minutes</strong>. Shuffle partitions 200 static → 34 coalesced with skew splits. EMR cost per run ~$96 → ~$18. The three red partitions in the UI are gone — not because the skew disappeared, but because the engine finally saw it and split it.</p><h3>The Catch: Where the Adaptive Machinery Runs Out of Road</h3><p>AQE is powerful the way a good autopilot is powerful: within its envelope. Outside it, you’re hand-flying again.</p><p>First, <strong>AQE needs a shuffle to learn from.</strong> The first query stage runs on static planning, always — there are no runtime statistics yet. A query with no shuffles gets zero adaptivity. A query whose only shuffle is at the very end gets its statistics too late to matter.</p><p>Second, <strong>skew handling only covers sort-merge and shuffled-hash joins.</strong> If your skew is in a groupBy aggregation or a window function, AQE watches it happen and does nothing. That&#39;s still Part VII territory: salt it yourself.</p><p>Third, <strong>runtime statistics cost scheduling.</strong> Each query stage boundary is a synchronization point — stages can’t overlap the way a fully pipelined static plan might. For queries with many small stages, the stage-scheduling overhead can eat the coalescing gains. It’s usually worth it. Usually is doing work in that sentence.</p><p>Fourth, <strong>AQE’s broadcast conversions still hit Part XIV’s bottleneck.</strong> Converting a sort-merge join to a broadcast hash join at runtime means the driver collects the build side <em>right then</em>. If the runtime statistics say the build side is 9GB and your driver has 8g of heap, AQE will cheerfully schedule a driver OOM. The adaptive threshold deserves the same scrutiny as the static one.</p><p>Fifth, <strong>join reordering still wants real table statistics.</strong> AQE can change strategies mid-flight, but the initial join <em>order</em> comes from Catalyst, and Catalyst without CBO stats is guessing. ANALYZE TABLE ... COMPUTE STATISTICS remains the cheapest performance work nobody does.</p><h3>Takeaways</h3><ul><li><strong>The static planner guesses; AQE measures</strong> — every rewrite is funded by runtime statistics collected at shuffle boundaries.</li><li><strong>Coalescing retires </strong><strong>spark.sql.shuffle.partitions as a tuning knob</strong> — set it as a generous upper bound and let the 64MB advisory size trim the rest.</li><li><strong>Runtime join switching is the anti-hint</strong> — it makes the broadcast decision with fresh data instead of your eighteen-month-old guess.</li><li><strong>Skew splits fire at 256MB and 5× median, on sort-merge/shuffle-hash joins only</strong> — aggregation skew is still yours to salt.</li><li><strong>Read the adaptive plan, not the query text</strong> — collapsed stages and AQEShuffleRead nodes are where the rewrites show up in the UI.</li><li><strong>ANALYZE TABLE is still the cheapest optimization</strong> — AQE fixes strategy, but join order still needs real statistics to start from.</li></ul><p>The most expensive query in your warehouse is the one the planner imagined and the data never matched.</p><p>If this saved you a 3 AM pager incident, <a href="https://medium.com/@chakshu-salgotra">follow me on Medium</a> for Twice a week for the next 3 Months of Advanced Data Engineering &amp; AI. Let’s connect on <a href="https://www.linkedin.com/in/chakshu-salgotra/">LinkedIn</a> and <a href="https://x.com/chakshusalgotra">X</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f010f22ad3fa" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/part-xv-adaptive-query-execution-what-aqe-actually-rewrites-at-runtime-f010f22ad3fa">Part XV — Adaptive Query Execution: What AQE Actually Rewrites at Runtime</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part XIV — Broadcast Variables: The Driver Bottleneck Nobody Profiles]]></title>
            <link>https://pub.towardsai.net/part-xiv-broadcast-variables-the-driver-bottleneck-nobody-profiles-87189bb691ed?source=rss-7afde1b096a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/87189bb691ed</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[broadcast-join]]></category>
            <category><![CDATA[performance-tuning]]></category>
            <category><![CDATA[spark-internals]]></category>
            <dc:creator><![CDATA[chakshu_salgotra]]></dc:creator>
            <pubDate>Wed, 14 Oct 2026 22:31:01 GMT</pubDate>
            <atom:updated>2026-10-14T22:31:01.881Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Broadcast joins are sold as free money: skip the shuffle, ship the small table everywhere. But “small” is measured against the driver’s heap, the collect has to survive a 300-second timeout, and every byte lands on the one node nobody profiles — until it dies at 2:47 AM.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zmuid0AZ78kQzdj3FnsovQ.png" /></figure><p><a href="https://chakshu-salgotra.medium.com/how-spark-reads-parquet-2333c7fa728c">Part I</a> | <a href="https://chakshu-salgotra.medium.com/spark-shuffle-partitioning-under-the-hood-of-a-wide-transformation-521907cd5f1c">Part II</a><strong> | </strong><a href="https://medium.com/@chakshu-salgotra/spark-catalyst-optimizer-internals-2b762ce0b2e2">Part III</a> <strong>| </strong><a href="https://pub.towardsai.net/jobs-stages-tasks-how-spark-actually-decomposes-your-query-e50bc8ed768e">Part IV</a> <strong>| </strong><a href="https://pub.towardsai.net/how-spark-manages-memory-the-unified-memory-model-spills-and-aqe-f37f21799fc6?sk=c91e30bf8b55b332b5611b7585b0b883">Part V</a> <strong>| </strong><a href="https://pub.towardsai.net/part-vi-the-spark-join-strategies-broadcast-vs-52a2048203e6?sk=aec7100814c2dbac55341898fb4f0686">Part VI<strong> </strong></a><strong>| </strong><a href="https://pub.towardsai.net/part-vii-data-skew-in-spark-why-199-tasks-finish-in-40-seconds-and-one-runs-for-3-hours-7daef3b1de44?sk=2262448f82bf04acda6dea9bfdeca857">Part VII</a> | <a href="https://pub.towardsai.net/part-viii-batch-vs-streaming-in-spark-one-sql-engine-two-execution-contracts-a4e888cb4748?sk=e8f00a5249b4a1ce3aa992f614e76705">Part VIII </a>| <a href="https://pub.towardsai.net/part-ix-the-anatomy-of-a-spark-application-what-the-driver-executors-and-cluster-manager-ffa2bba60dce?sk=d66767c2ca186aaab54860931975741e">Part IX</a> | <a href="https://pub.towardsai.net/part-x-z-ordering-and-data-clustering-explained-why-your-partitioned-table-still-scans-4b109e016c80?sk=dc7cfcb72d4bd523b6b5ca35f79f2b16">Part X </a>| <a href="https://pub.towardsai.net/part-xi-rdd-vs-dataframe-vs-dataset-why-your-type-safe-refactor-made-spark-3x-slower-99bdba6f7eaa?source=friends_link&amp;sk=3cef58b143ef9d20917ac93e62731ecc">Part XI</a> | <a href="https://chakshu-salgotra.medium.com/da093c04b481?source=friends_link&amp;sk=1f9b68eee7c3c07c233d280b625b5fab">Part XII </a>| <a href="https://medium.com/towards-artificial-intelligence/part-xiii-internal-vs-external-tables-the-drop-table-that-deleted-40tb-4fffae7f740c">Part XIII </a>| <strong>Part XIV</strong></p><h3>The Night the Driver Died Collecting a “Small” Table</h3><p>Tuesday, 2:47 AM. The daily revenue reconciliation job — the one finance opens before their standup — failed for the third night in a row. Same signature every time: the driver container got killed by YARN, java.lang.OutOfMemoryError: Java heap space, buried inside a BroadcastExchangeExec. Forty r5.2xlarge executors sat at 3% CPU while the single driver node thrashed, retried, and died again.</p><p>The join was textbook. A 1.9TB fact table of payment events joined against dim_merchants, the merchant dimension. When the pipeline was built eighteen months ago, that dimension was 3.2GB — comfortably &quot;small.&quot; A well-meaning engineer had added /*+ BROADCAST(merchants) */ to the query and moved on. Nobody revisited it. Merchant onboarding had since grown the dimension to 14.6GB, and nobody noticed because the broadcast hint doesn&#39;t care how big the table is. It just collects.</p><p>Three failed attempts, 6 hours 12 minutes of wall clock, roughly $412 in EMR spend to produce zero rows — 97% of it burned by executors waiting on a driver that was already dead. The “optimization” had become the single point of failure, and it was the one component in the whole cluster nobody had ever profiled.</p><p>So: what does a broadcast actually cost the driver, where does the data physically go, and how do you size it so the free lunch stops killing the host?</p><h3>The Mechanic Underneath: Broadcast Is a Collect, Not a Shuffle</h3><p>Here is the mental model most engineers get wrong: a broadcast join does not avoid moving data. It avoids moving the <em>big</em> table. The price is that the <em>entire build side is materialized inside the driver’s heap first</em> — collect()-style, all of it, on one JVM — and only then fanned out to every executor.</p><p>Every broadcast is three phases, and phase one runs on your least-observed node:</p><ol><li><strong>Collect.</strong> The driver executes the build side of the plan and pulls every row into its own heap, then builds an in-memory hash table (HashedRelation / LongHashedRelation) for the join keys.</li><li><strong>Serialize and torrent.</strong> The hash table is serialized and split into 4MB blocks (spark.broadcast.blockSize, default 4m), then distributed via <strong>TorrentBroadcast</strong> — a BitTorrent-style peer-to-peer protocol where executors fetch blocks from the driver <em>and from each other</em>.</li><li><strong>Materialize everywhere.</strong> Each executor stores the full broadcast in its <strong>BlockManager</strong> and probes it locally. No shuffle of the big table happens at all.</li></ol><p>The trade is real and usually worth it — but it is a trade, not a freebie:</p><ul><li><strong>You trade cluster-wide shuffle I/O for driver-side memory pressure.</strong> The shuffle you avoided was spread across 40 executors. The collect you replaced it with lands on one heap.</li><li><strong>You trade it again for N copies.</strong> A 900MB broadcast on 40 executors is 36GB of cluster memory holding the same bytes. Small tables make this irrelevant; “small” tables that grew quietly make it a leak.</li><li><strong>You inherit the driver’s two hard budgets: heap and time.</strong> The collect must fit in driver memory <em>and</em> finish within spark.sql.broadcastTimeout (default 300 seconds), or the job dies before a single task launches.</li></ul><h3>Phase One Runs on the Driver, and the Driver Is Not a Data Node</h3><p>The driver is sized for orchestration: scheduling tasks, holding the DAG, serving the UI. It is typically the smallest node in the cluster — an m5.xlarge with spark.driver.memory=8g while executors get 64GB each. Broadcast is the one operation in all of Spark SQL that treats the driver like a data node, and it does so unconditionally once the decision is made.</p><p>What actually lives in the driver’s heap during a broadcast is worse than the table’s on-disk size. The rows arrive deserialized as JVM objects (no Tungsten off-heap encoding here), then get reorganized into a hash table with per-row overhead. A 14.6GB Parquet dimension easily becomes 20GB+ of heap. The driver doesn’t spill broadcast builds gracefully — it just dies.</p><p>This is why the failure signature is so distinctive: the job fails <em>fast-ish</em> (during planning/exchange, before real work), the stack trace names BroadcastExchangeExec, and the executors look idle in the UI because they never got anything to do.</p><h3>TorrentBroadcast: BitTorrent for DataFrames, in 4MB Chunks</h3><p>Once the driver survives the collect, distribution is genuinely clever. TorrentBroadcast chops the serialized relation into blocks (default 4MB via spark.broadcast.blockSize) and serves them peer-to-peer: the first executors pull from the driver, later ones pull from each other. The driver is not a bottleneck for distribution — only for the collect.</p><p>Each block lands in the executor’s BlockManager, counted against regular storage/execution memory. Two consequences worth knowing:</p><ul><li><strong>Broadcasts compete with caching and shuffle on executors.</strong> Five 2GB broadcasts on a busy executor is 10GB of BlockManager that isn’t available for anything else. Nobody sums these up because each broadcast is configured — and justified — in isolation.</li><li><strong>Broadcasts are per-job, not per-cluster.</strong> The same dimension broadcast in twelve concurrent jobs is twelve copies in flight. At 2 AM with one job running, fine. At 9 AM with the whole org’s dbt models firing, the math changes.</li></ul><h3>The 10MB Threshold Is a Gate, Not a Guarantee</h3><p>spark.sql.adaptive.enabled aside, the static planner broadcasts a relation when its estimated size is under spark.sql.autoBroadcastJoinThreshold — default <strong>10485760 bytes (10MB)</strong>. &quot;Estimated&quot; is doing heavy lifting: without table statistics, the estimate can be wildly wrong, and with stale statistics it&#39;s wrong in the dangerous direction.</p><p>Worse, the SQL hint /*+ BROADCAST(t) */ bypasses the gate entirely. It is an instruction, not a suggestion. It broadcasts a 14.6GB table with the same confidence as a 14KB one, and it will keep doing so every night until someone reads the query. Hints are the number one source of broadcast incidents I&#39;ve seen, because they encode a size assumption from the day they were written and never expire.</p><p>Setting the threshold to -1 disables auto-broadcast completely — a legitimate choice for pipelines where every join is large and you&#39;d rather pay for shuffles than debug 3 AM driver OOMs.</p><h3>The Three Failure Shapes of Broadcast</h3><p>In production, broadcast fails in exactly three ways, and each has a different smell:</p><ul><li><strong>Driver OOM during collect.</strong> Stack trace in BroadcastExchangeExec, fast failure, idle executors. Cause: build side bigger than driver heap. Fix: shrink the build side or grow the driver.</li><li><strong>Broadcast timeout.</strong> Could not execute broadcast in 300 secs. The collect technically fits but is slow — usually a build side that itself requires a heavy scan or shuffle. Fix: spark.sql.broadcastTimeout, or better, make the build side cheap (pre-aggregate, filter early).</li><li><strong>Executor memory pressure.</strong> No single failure; everything is just slower, with excessive GC and spilled blocks. Cause: too many or too-large broadcasts sharing BlockManager with real work. Fix: audit what you’re broadcasting — the sum, not each one.</li></ul><p>Note what is <em>not</em> on this list: skew. Broadcast is actually the correct answer to a skewed join when the skewed side is the <em>small</em> side — the whole point is that every executor gets the full build side, so a hot key on the probe side just probes a local hash table. Part VII’s skew playbook covers the salt-and-split approach for when both sides are large; broadcast covers the asymmetric case. The failure mode is never “broadcast can’t handle skew” — it’s “the small side wasn’t small.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DqXSLbzt9XlW71VLpcVw5g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ASBKW4oHck2ZyrQcexoWw.png" /></figure><h3>The Fix, and What It Cost</h3><p>The naive state of our reconciliation job: a BROADCAST hint written eighteen months ago, default driver memory, and a dimension table that had quietly 4.5x&#39;d.</p><pre># Naive: a hint that encodes a size assumption from 18 months ago,<br># broadcasting a 14.6 GB dimension into an 8g driver heap.<br>revenue = (<br>    payments<br>    .hint(&quot;broadcast&quot;, &quot;merchants&quot;)  # BROADCAST hint: bypasses the 10MB gate entirely<br>    .join(merchants, &quot;merchant_id&quot;)<br>)<br># Result: driver OOM in BroadcastExchangeExec, 3 attempts, 6h12m, $412, zero rows.</pre><p>The engine-aware fix had three parts, in order of leverage:</p><pre># Optimized: shrink the build side, let the planner decide, size the driver honestly.<br>spark.conf.set(&quot;spark.sql.autoBroadcastJoinThreshold&quot;, 100 * 1024 * 1024)  # 100MB: explicit, reviewed<br>spark.conf.set(&quot;spark.sql.broadcastTimeout&quot;, 600)  # 600s: the build side needs a scan first<br># spark.driver.memory=16g  # set at submit: broadcast collect must fit with headroom</pre><pre>merchants_small = (<br>    merchants<br>    .filter(F.col(&quot;is_active&quot;))          # 14.6 GB -&gt; 900 MB: only active merchants join<br>    .select(&quot;merchant_id&quot;, &quot;fee_tier&quot;)   # drop the 40 unused columns before the collect<br>)</pre><pre>revenue = payments.join(F.broadcast(merchants_small), &quot;merchant_id&quot;)<br># F.broadcast() still hints, but the build side is now 900MB against a 16g driver.</pre><p>The filter + select before the join is the highest-leverage line in the whole fix: the dimension was 14.6GB on disk mostly because of dead merchants and forty columns the join never touched. The broadcast decision was never the real problem — the unexamined build side was.</p><p><strong>Before/after:</strong> driver OOM loop, 6h12m wall clock, ~$412 EMR spend, zero rows → stable nightly run, <strong>38 minutes</strong>, ~$31. Driver heap peak went from dead-at-8g to a comfortable 2.3GB on a 16g driver. And the hint is gone — the 100MB threshold is a reviewed config, not a fossilized guess.</p><h3>The Catch: Broadcast Still Copies, Still Centralizes, Still Lies by Omission</h3><p>Three honest counterweights before you go broadcast everything under 100MB.</p><p>First, <strong>broadcast replicates to every executor</strong>, so its true cluster cost is size × executor count. A 900MB broadcast across 200 executors is 180GB of duplicated memory. On large clusters, the &quot;small&quot; table stops being small in aggregate, and a sort-merge join with a well-partitioned build side can be cheaper in total bytes moved. Do the multiplication; nobody does the multiplication.</p><p>Second, <strong>the hint problem doesn’t go away, it just moves.</strong> F.broadcast() in PySpark is the same fossilized assumption as the SQL hint. Prefer the threshold config where the build side is genuinely stable, and re-measure quarterly. Dimension tables grow. That is their nature.</p><p>Third, <strong>broadcast doesn’t fix a slow build side.</strong> If computing the small table requires its own two-hour shuffle, you’ve just moved the bottleneck into the driver’s critical path, serialized. Pre-aggregate it, persist it, or materialize it as a table — don’t make the driver compute it fresh every night under a 300-second gun.</p><h3>Takeaways</h3><ul><li><strong>A broadcast join is a driver-side collect first and a join second</strong> — size it against the driver’s heap, not the cluster’s.</li><li><strong>Hints are size assumptions with no expiry date</strong> — audit every BROADCAST hint against the table&#39;s current size, or delete it and use the threshold.</li><li><strong>Shrink the build side before tuning anything else</strong> — filters and column pruning beat every config knob, because the collect is the whole cost.</li><li><strong>spark.sql.broadcastTimeout (300s default) is the second budget</strong> — a build side that needs its own heavy scan will time out on a driver that could have held it.</li><li><strong>True broadcast cost is size × executor count</strong> — on big clusters, duplication can exceed the shuffle you avoided.</li><li><strong>Broadcast is the right answer to one-sided skew</strong> — a hot key on the large side is harmless when every executor holds the full small side locally.</li></ul><p>A shuffle spreads the pain across the cluster; a broadcast concentrates it on the one node you never look at.</p><p>If this saved you a 3 AM pager incident, <a href="https://medium.com/@chakshu-salgotra">follow me on Medium</a> for Twice a week for the next 3 Months of Advanced Data Engineering &amp; AI. Let’s connect on <a href="https://www.linkedin.com/in/chakshu-salgotra/">LinkedIn</a> and <a href="https://x.com/chakshusalgotra">X</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87189bb691ed" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/part-xiv-broadcast-variables-the-driver-bottleneck-nobody-profiles-87189bb691ed">Part XIV — Broadcast Variables: The Driver Bottleneck Nobody Profiles</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part XIII — Internal vs External Tables: The DROP TABLE That Deleted 40TB]]></title>
            <link>https://pub.towardsai.net/part-xiii-internal-vs-external-tables-the-drop-table-that-deleted-40tb-4fffae7f740c?source=rss-7afde1b096a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/4fffae7f740c</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[delta-lake]]></category>
            <dc:creator><![CDATA[chakshu_salgotra]]></dc:creator>
            <pubDate>Mon, 12 Oct 2026 20:01:02 GMT</pubDate>
            <atom:updated>2026-10-12T20:01:02.393Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Managed tables own your data; external tables borrow it. Confusing the two is how a schema cleanup script becomes a data-lake extinction event — here’s the engine-level difference and the routing rules that prevent it</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FEFtl84UdVYLfEaC4Jjr2A.png" /></figure><p><a href="https://chakshu-salgotra.medium.com/how-spark-reads-parquet-2333c7fa728c">Part I</a> | <a href="https://chakshu-salgotra.medium.com/spark-shuffle-partitioning-under-the-hood-of-a-wide-transformation-521907cd5f1c">Part II</a><strong> | </strong><a href="https://medium.com/@chakshu-salgotra/spark-catalyst-optimizer-internals-2b762ce0b2e2">Part III</a> <strong>| </strong><a href="https://pub.towardsai.net/jobs-stages-tasks-how-spark-actually-decomposes-your-query-e50bc8ed768e">Part IV</a> <strong>| </strong><a href="https://pub.towardsai.net/how-spark-manages-memory-the-unified-memory-model-spills-and-aqe-f37f21799fc6?sk=c91e30bf8b55b332b5611b7585b0b883">Part V</a> <strong>| </strong><a href="https://pub.towardsai.net/part-vi-the-spark-join-strategies-broadcast-vs-52a2048203e6?sk=aec7100814c2dbac55341898fb4f0686">Part VI<strong> </strong></a><strong>| </strong><a href="https://pub.towardsai.net/part-vii-data-skew-in-spark-why-199-tasks-finish-in-40-seconds-and-one-runs-for-3-hours-7daef3b1de44?sk=2262448f82bf04acda6dea9bfdeca857">Part VII</a> | <a href="https://pub.towardsai.net/part-viii-batch-vs-streaming-in-spark-one-sql-engine-two-execution-contracts-a4e888cb4748?sk=e8f00a5249b4a1ce3aa992f614e76705">Part VIII </a>| <a href="https://pub.towardsai.net/part-ix-the-anatomy-of-a-spark-application-what-the-driver-executors-and-cluster-manager-ffa2bba60dce?sk=d66767c2ca186aaab54860931975741e">Part IX</a> | <a href="https://pub.towardsai.net/part-x-z-ordering-and-data-clustering-explained-why-your-partitioned-table-still-scans-4b109e016c80?sk=dc7cfcb72d4bd523b6b5ca35f79f2b16">Part X </a>| <a href="https://pub.towardsai.net/part-xi-rdd-vs-dataframe-vs-dataset-why-your-type-safe-refactor-made-spark-3x-slower-99bdba6f7eaa?source=friends_link&amp;sk=3cef58b143ef9d20917ac93e62731ecc">Part XI</a> | <a href="https://pub.towardsai.net/da093c04b481?source=friends_link&amp;sk=1f9b68eee7c3c07c233d280b625b5fab">Part XII</a> | <strong>Part XIII</strong></p><h3>The Cleanup Script That Emptied the Lake</h3><p>A platform team runs a quarterly hygiene job: drop every table in the staging schema older than 90 days. It has run safely eight times. The ninth run pages the on-call at 2:47 AM — downstream Trino queries are failing with FileNotFoundException, and the S3 bucket that backs the curated events dataset is missing <strong>40TB of Parquet files</strong>.</p><p>The post-mortem is one line long. Six months earlier, an engineer migrating a pipeline had created what looked like a staging copy:</p><pre>CREATE TABLE staging.events_curated AS<br>SELECT * FROM lake.events_curated;</pre><p>Except they’d been debugging in a session where the schema’s default location pointed <em>at the lake path</em>, and the table was created <strong>managed</strong>. The cleanup script issued DROP TABLE staging.events_curated. For a managed table, the engine doesn&#39;t just delete the catalog entry — <strong>it deletes the data files it believes it owns.</strong> Forty terabytes, purged by a metadata operation that everyone assumed was metadata-only.</p><p>Recovery took 31 hours from object-store versioning. The real failure wasn’t the script — it was that the organization had <strong>no selection framework</strong> for when a table should be internal (managed) versus external. So: what does table ownership actually mean at the engine level, and which workloads belong on each side of the line?</p><h3>What “Managed” Actually Means: Lifecycle Coupling</h3><p>Every catalog-backed table — Hive Metastore, AWS Glue, Unity Catalog, an Iceberg REST catalog — is two things: a <strong>metadata entry</strong> (schema, location, properties, statistics) and a set of <strong>data files</strong> at some storage path. The internal/external distinction is exactly one bit: <strong>does the catalog own the lifecycle of the data files, or only the pointer to them?</strong></p><ul><li><strong>Internal (managed) table:</strong> the engine controls the storage path (typically under the warehouse root, e.g. spark.sql.warehouse.dir or the catalog&#39;s managed location). DROP TABLE deletes metadata <strong>and data</strong>. TRUNCATE, schema evolution, compaction, vacuum, and retention policies are the engine&#39;s prerogative — it assumes no one else touches those files.</li><li><strong>External table:</strong> the catalog holds a <strong>pointer</strong> to a path it does not own. DROP TABLE removes the pointer; <strong>the files survive</strong>. The engine treats the location as shared territory: other engines, other catalogs, and other tables may reference the same files.</li></ul><p>This single bit cascades into everything that matters in production:</p><ul><li><strong>Blast radius of DDL.</strong> Managed: a DROP is a data-destruction operation. External: a DROP is reversible in seconds by re-registering the location.</li><li><strong>Multi-engine access.</strong> Managed tables couple data to <em>one catalog’s</em> lifecycle semantics. External tables (and open table formats registered externally) let Spark write, Trino query, and a BI extract read — three engines, one copy of the data, zero ETL between them. (Part I of this series covered how Spark reads Parquet in depth — that’s the file layout these engines share.)</li><li><strong>Optimization entitlement.</strong> Because a managed table’s files are engine-private, the engine can safely rewrite them: auto-compaction of small files, clustering, OPTIMIZE/VACUUM, background garbage collection. On an external path, aggressive file rewriting risks breaking a concurrent reader that holds its own manifest of those files.</li><li><strong>Governance surface.</strong> Managed tables inherit catalog-level policies (retention, encryption, lineage) automatically. External tables require you to govern the <em>storage path</em> separately from the <em>catalog entry</em> — two ACL systems that can drift.</li></ul><p>The corollary most teams miss: <strong>open table formats didn’t erase this distinction — they moved it.</strong> An Iceberg or Delta table registered as managed in Unity Catalog or a REST catalog still has purge-on-drop semantics; the same table registered externally in three catalogs is a shared asset that no single engine may unilaterally vacuum. The bit still exists; it just lives in the catalog now, not the file layout.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oPdtBuaz2vnIqfJauWF3zw.png" /></figure><h3>The Selection Framework</h3><p>Route by <strong>who owns the data’s lifecycle</strong>, not by convenience or the CTAS default.</p><h3>When to Use Internal (Managed)</h3><p><strong>1. Medallion Bronze staging — transient by design.</strong></p><p>Bronze-layer landing tables in a <strong>medallion architecture</strong> are the canonical managed workload: raw ingest that exists to be transformed into Silver and then discarded or aged out. You <em>want</em> lifecycle coupling here:</p><ul><li>DROP/TRUNCATE genuinely should destroy the data — that&#39;s the retention policy working, not an accident.</li><li>The engine’s <strong>auto-compaction</strong> can freely rewrite the small-file mess that streaming ingest produces, because no external reader holds references to specific files.</li><li>Catalog-enforced retention (e.g. drop partitions older than N days) executes as a metadata-plus-data operation in one atomic step.</li></ul><p><strong>2. Sandbox and test environments.</strong></p><p>Developer schemas, CI-created tables, notebook experiments. The whole point is that tearing down the schema reclaims the storage. A sandbox built on external tables leaks orphaned files forever — every DROP leaves Parquet corpses that no catalog references and no cost report attributes.</p><p><strong>3. Performance-isolated BI data marts.</strong></p><p>A mart serving a dashboard SLA benefits from the engine owning the layout: clustering keys, Z-ordering/liquid clustering (Part X of this series covered Z-ordering and data clustering in depth), statistics collection, and file-size tuning all require the engine’s unilateral right to rewrite files. A managed gold-layer mart lets the engine hold <strong>p95 latency</strong> by continuously re-optimizing — something it cannot safely do on a path other engines read by direct file listing.</p><h3>When to Use External</h3><p><strong>1. The centralized corporate data lake.</strong></p><p>The system of record must survive any single catalog, engine, or workspace. External registration means a catalog migration (Hive → Glue → Unity, or between Iceberg catalogs) is a re-pointing exercise, not a data migration. It also means the 2:47 AM scenario above is structurally impossible: no DROP anywhere deletes the lake.</p><p><strong>2. Multi-engine architectures.</strong></p><p>The dominant modern pattern — <strong>Spark processes, Trino queries, Tableau visualizes</strong> — requires the data to belong to the <em>storage layer</em>, with each engine holding only a pointer:</p><ul><li>Spark runs the heavy ETL and writes Iceberg/Delta commits.</li><li>Trino serves interactive SQL against the same table, reading the same manifests.</li><li>The BI tool extracts or live-queries through Trino.</li></ul><p>One copy of the data, no synchronization pipeline, no divergence. This only works if no engine believes it owns the files.</p><p><strong>3. Immutable historical and audit logs.</strong></p><p>Compliance archives, CDC history, event logs under legal hold. Immutability is the requirement, and external registration enforces it <em>architecturally</em>: pair the external table with object-store controls (S3 Object Lock, versioning, deny-delete bucket policies) and the data becomes un-deletable even by a privileged catalog admin. A managed table can never make this guarantee — its own engine always retains the right to purge.</p><h3>The Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zvbZQTyK2HmjyC85mFmyKA.png" /></figure><p>Managed at the edges (transient ingest, SLA-bound serving), external in the middle (the shared, durable core). Ownership follows lifecycle: the engine owns what it alone consumes; the storage layer owns what everyone consumes.</p><h3>Making Ownership Explicit in DDL</h3><p>The 40TB incident happened because a CTAS <strong>defaulted</strong> into managed semantics. Never let the default decide ownership — declare it.</p><pre>-- Naive: ownership decided by session defaults; DROP may purge shared data<br>CREATE TABLE staging.events_curated AS<br>SELECT * FROM lake.events_curated;</pre><pre>-- Optimized: external registration — DROP removes only the pointer<br>CREATE TABLE lake.events_curated (<br>    event_id   BIGINT,<br>    user_id    BIGINT,<br>    event_ts   TIMESTAMP,<br>    payload    STRING<br>)<br>USING iceberg<br>LOCATION &#39;s3://corp-lake/curated/events/&#39;;</pre><p>For the managed Bronze layer, the same explicitness — plus the properties that only make sense when the engine owns the files:</p><pre>-- Managed on purpose: engine may compact, cluster, and expire freely<br>CREATE TABLE bronze.events_raw (<br>    raw_payload STRING,<br>    ingest_ts   TIMESTAMP<br>)<br>USING delta<br>TBLPROPERTIES (<br>    &#39;delta.autoOptimize.optimizeWrite&#39; = &#39;true&#39;,<br>    &#39;delta.autoOptimize.autoCompact&#39;   = &#39;true&#39;,<br>    &#39;delta.logRetentionDuration&#39;       = &#39;interval 7 days&#39;<br>);</pre><p>And the audit-time check that would have caught the incident before the ninth run — verify what the catalog <em>believes</em> it owns before any destructive DDL:</p><pre># Naive: trust the schema name (&quot;staging&quot;) to imply table type<br># Optimized: assert ownership semantics before destructive DDL<br>from pyspark.sql.functions import col</pre><pre>detail = spark.sql(&quot;DESCRIBE DETAIL staging.events_curated&quot;)<br>tbl = detail.select(&quot;name&quot;, &quot;location&quot;, &quot;tableFeatures&quot;).first()<br>info = spark.catalog.getTable(&quot;staging.events_curated&quot;)</pre><pre>assert info.tableType == &quot;EXTERNAL&quot; or tbl.location.startswith(<br>    &quot;s3://warehouse-managed/&quot;<br>), f&quot;Refusing DROP: managed table at non-warehouse path {tbl.location}&quot;</pre><p>Two lines of assertion versus 31 hours of object-store restore. The cleanup job at the top of this article now runs with exactly this guard — and a policy that the staging schema only admits managed tables under the warehouse root, enforced at CREATE time, not audit time.</p><h3>Decision Matrix</h3><p>WorkloadTable typeWhy (lifecycle logic)Bronze / landing zone<strong>Managed</strong>Transient by contract; engine compacts and expires atomicallySandboxes, CI, notebooks<strong>Managed</strong>Teardown must reclaim storage; external leaks orphan filesSLA-bound BI marts<strong>Managed</strong>Engine needs unilateral rewrite rights for clustering/compactionCorporate data lake<strong>External</strong>Must outlive any catalog; DROP is never a data operationMulti-engine (Spark → Trino → BI)<strong>External</strong>One copy, N pointers; no engine may claim ownershipAudit / compliance logs<strong>External</strong> + object lockImmutability enforced below the catalog, not by it</p><p>Three rules that survive contact with production:</p><ul><li><strong>Ownership follows the narrowest consumer.</strong> If exactly one engine reads and writes it, manage it and let that engine optimize. The moment a second engine reads the files, ownership must move down to the storage layer.</li><li><strong>Never let CTAS defaults assign ownership.</strong> Every production CREATE TABLE declares LOCATION (external) or explicitly targets the managed warehouse. Table type is an architectural decision, not a session-config side effect.</li><li><strong>Destructive DDL treats table type as untrusted input.</strong> Any automated DROP/TRUNCATE verifies tableType and location prefix first. Schema names lie; the catalog&#39;s ownership bit doesn&#39;t.</li></ul><h3>Takeaways</h3><ul><li><strong>One ownership bit:</strong> internal vs external is whether the catalog owns the files or just points at them — but it determines DROP blast radius, multi-engine access, optimization rights, and governance surface.</li><li><strong>Managed</strong> wins where lifecycle coupling is the feature: transient Bronze staging, disposable sandboxes, and BI marts where the engine’s rewrite rights hold the latency SLA.</li><li><strong>External</strong> wins where the data must outlive any single engine: the corporate lake, Spark-writes/Trino-reads/Tableau-visualizes architectures, and legally immutable history.</li><li><strong>The bit moved, it wasn’t deleted:</strong> open table formats moved ownership into the catalog — an Iceberg table registered as managed can still be purged on DROP.</li><li><strong>Implicit ownership is an incident waiting for a cleanup job:</strong> the 40TB deletion wasn’t a script bug — it was an ownership decision made by a session default, and every implicit ownership decision in your catalog is the same incident waiting for its ninth run.</li></ul><p>If this saved you a 3 AM pager incident, <a href="https://medium.com/@chakshu-salgotra">follow me on Medium</a> for Twice a week for the next 3 Months of Advanced Data Engineering &amp; AI. Let’s connect on <a href="https://www.linkedin.com/in/chakshu-salgotra/">LinkedIn</a> and <a href="https://x.com/chakshusalgotra">X</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4fffae7f740c" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/part-xiii-internal-vs-external-tables-the-drop-table-that-deleted-40tb-4fffae7f740c">Part XIII — Internal vs External Tables: The DROP TABLE That Deleted 40TB</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>