<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yeedu on Medium]]></title>
        <description><![CDATA[Stories by Yeedu on Medium]]></description>
        <link>https://medium.com/@yeedu?source=rss-a9bebe2a1730------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*CJFuNyKTYDHt2bjRA7hgpA.png</url>
            <title>Stories by Yeedu on Medium</title>
            <link>https://medium.com/@yeedu?source=rss-a9bebe2a1730------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 17:04:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yeedu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Trino federates your queries, not your source’s connection limits]]></title>
            <link>https://medium.com/towards-data-engineering/trino-federates-your-queries-not-your-sources-connection-limits-9b318a2e6602?source=rss-a9bebe2a1730------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b318a2e6602</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Yeedu]]></dc:creator>
            <pubDate>Sat, 14 Nov 2026 10:54:58 GMT</pubDate>
            <atom:updated>2026-11-14T10:54:58.663Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Trino federates your queries, not your source’s connection limits" src="https://cdn-images-1.medium.com/max/1024/1*6wFMaYKt5FflDM1u0zXxpQ.png" /></figure><p>A handful of ad-hoc Trino queries against a Postgres or MySQL OLTP catalog can exhaust that database’s connection limit or force a full-table scan, because Trino’s resource groups throttle query concurrency across the whole cluster and have no mechanism to cap concurrent load on a single connector or catalog.</p><p>We’ve watched this happen in production more than once: someone wires up a Postgres catalog for “just a quick join,” three analysts run overlapping queries against the same orders table, and the source starts rejecting connections while the Trino cluster itself looks perfectly healthy. That asymmetry, cluster fine, source on fire, is the whole story.</p><h3><strong>Why resource groups can’t protect a single catalog</strong></h3><p>Trino’s admission control lives in resource groups. You set soft Concurrency Limit and hard Concurrency Limit, and the scheduler enforces them against queries running in that group:</p><pre>{ <br> &quot;name&quot;: &quot;adhoc&quot;, <br> &quot;softMemoryLimit&quot;: &quot;10%&quot;, <br> &quot;softConcurrencyLimit&quot;: 10, <br> &quot;hardConcurrencyLimit&quot;: 50, <br> &quot;maxQueued&quot;: 1, <br> &quot;schedulingWeight&quot;: 10 <br>}</pre><p>That config bounds how many queries an adhoc group can run at once, cluster-wide. It says nothing about which catalog those queries touch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lG3bx9S-R4P0WiCDy50h6w.png" /></figure><p>A resource group with hard Concurrency Limit: 5 can still let five queries each spin up 200+ drivers, saturating task slots without ever tripping a per-source guard, a gap <a href="https://trino.io/docs/current/admin/resource-groups.html">Trino’s own resource-groups documentation</a> confirms has no per-catalog dimension. A <a href="https://github.com/trinodb/trino/issues/28373">separate open issue proposing per-query limits, driver-level admission control, and a planning concurrency limit</a> targets exactly this kind of driver saturation at the execution level, with runaway concurrency during planning flagged as a secondary, more niche concern. Neither that issue nor the resource-groups documentation describes any per-catalog or per-connector dimension, because as of Trino 483 none exists.</p><h3><strong>What happens when 80 concurrent queries hit one JDBC source?</strong></h3><p>A maintainer answering a 2020 feature request described the failure directly: a legacy SQL Server backend overwhelmed by “80 concurrent queries” or “100 users all doing presto-mysql queries at once,” with no lever inside Trino to stop it. The response was blunt: resource groups control concurrent queries cluster-wide, but there’s <a href="https://github.com/prestosql/presto/issues/3103">no way to control the limit for specific connectors</a>. That request for per-catalog concurrency caps predates Trino’s fork from PrestoSQL and remains open in 2026.</p><h3><strong>Where does connection pressure come from during planning?</strong></h3><p>It isn’t only execution. Before Trino runs a single split, JdbcMetadata#applyAggregation used to call JdbcClient#prepareQuery, which opened a live database connection during query planning, one that in most cases was never used. That planning-time overhead was a separate, additive source of connection pressure on top of execution, fixed only by introducing a lazy connection object that defers opening until actually needed, per the <a href="https://github.com/trinodb/trino/issues/6719">PR against issue #6719</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*frUnRw3g1H8EDK5N97k-CA.png" /></figure><p>If your source is already near its max_connections ceiling, planning storms alone can tip it over before any data moves.</p><h3><strong>How a handful of federated queries become a full-table scan</strong></h3><p>Unpartitioned JDBC sources get a single split for the whole table. There’s no parallel readahead the way there is against object storage with hundreds of files; Trino streams the result over one connection, one thread, at whatever pace the source’s JDBC driver hands rows back. The <a href="https://www.starburst.io/blog/jdbc-trino-starburst/">University of Maryland benchmark</a> cited by Starburst measured a 32GB PostgreSQL table extracted in 855 seconds, roughly 37MB/s, entirely because JDBC results stream single-threaded regardless of how large the Trino cluster is.</p><p>The VLDB 2017 numbers make the driver overhead explicit: pulling a 7GB table took 10.25s over raw netcat versus 101.22s through MySQL, 169.48s through DB2, and 201.89s through PostgreSQL, every JDBC path running more than 10x slower than the socket baseline. Database systems, as that analysis puts it, <a href="https://www.starburst.io/blog/jdbc-trino-starburst/">limit the number of input queries they are willing to process in parallel</a>, particularly from one requesting source.</p><p>That’s the mechanical root of the DDoS risk: Trino can issue concurrency the source was never built to absorb, and each of those queries is individually slow enough to hold a connection open far longer than an OLTP workload expects. Pushdown settings shape how much of that scan gets filtered before it leaves the source, but they cap predicate complexity, not concurrent connections:</p><pre>connection-url=jdbc:postgresql://host:5432/db <br>connection-user=trino <br>connection-password=*** <br>domain-compaction-threshold=256 <br>join-pushdown.enabled=true <br>join-pushdown.strategy=AUTOMATIC <br>dynamic-filtering.enabled=true</pre><p>domain-compaction-threshold defaults to 256: once a dynamic filter accumulates more than 256 predicate values, Trino collapses them into a coarser range predicate before pushing it down, per the <a href="https://trino.io/docs/current/connector/postgresql.html">PostgreSQL connector docs</a> (Trino 483). Past that threshold you lose selectivity, and a query that should have hit an index instead pushes something closer to a range scan.</p><p>join-pushdown.enabled defaults to true, but it’s cost-based: Trino only pushes a join to Postgres when table statistics suggest it will help, so a catalog with stale or missing stats can silently fall back to pulling both sides into Trino and joining there, doubling the read volume against the source.</p><h3><strong>The partial fixes, and where each stops short</strong></h3><h4><strong>Connection pooling ships for Oracle, not Postgres</strong></h4><p>This is the asymmetry that catches teams off guard. Trino’s Oracle connector supports native connection pooling. The PostgreSQL connector doesn’t, by default each query opens fresh connections rather than reusing a pool, per the <a href="https://trino.io/docs/current/connector/postgresql.html">connector docs</a> and the still-open <a href="https://github.com/trinodb/trino/issues/15888">feature request to add it</a>, which notes Oracle’s pooling “significantly reduces query times.” A team that migrates a catalog from Oracle to Postgres inherits a materially different connection-pressure profile without any config warning them.</p><h4><strong>Materialized views buffer repeats, not concurrency</strong></h4><p>Trino’s materialized views cache a federated result so repeat queries stop re-hitting the source:</p><pre>CREATE MATERIALIZED VIEW orders_summary <br>GRACE PERIOD INTERVAL &#39;1&#39; HOUR <br>WHEN STALE FAIL <br>AS <br> SELECT orderdate, sum(totalprice) AS price <br> FROM orders <br> GROUP BY orderdate;</pre><p>That helps the pattern where twenty dashboards run the same query every five minutes. It does nothing for the pattern that actually causes the incidents we’re describing: distinct ad-hoc queries, each touching the source differently, arriving concurrently. WHEN STALE FAIL is documented to mean that once the grace period lapses, queries accessing it fail with an error rather than falling back to a live source, so the mitigation itself needs monitoring or it becomes an outage of its own.</p><h4><strong>Comparing the partial fixes</strong></h4><p>None of these mechanisms address the same failure mode, which is why stacking one on top of another still leaves gaps:</p><figure><img alt="Comparing the partial fixes" src="https://cdn-images-1.medium.com/max/877/1*_E_wvkABdRCOdcU474gnnQ.png" /></figure><h3><strong>What actually protects the source</strong></h3><h4><strong>What does Denodo do that Trino doesn’t?</strong></h4><p>Denodo’s Virtual DataPort ships an explicit, server-level setting to <a href="https://community.denodo.com/docs/html/browse/8.0/en/vdp/administration/server_configuration/limiting_the_number_of_concurrent_requests/limiting_the_number_of_concurrent_requests">limit the number of concurrent requests</a> the Virtual DataPort Server will accept, documented as a first-class admin feature alongside cache and thread-pool configuration. Trino has no equivalent at any layer, not even a cluster-wide request cap tied to backend protection.</p><p>The popular advice to “just federate instead of building a pipeline” glosses over that gap: teams federating against production OLTP systems are relying on safeguards Trino itself doesn’t provide, because the request for per-catalog concurrency limits has been open since 2020 and remains unresolved through the 2026 tracking issue.</p><h4><strong>What actually stops the DDoS before it starts?</strong></h4><p>Source-side, not Trino-side. <a href="https://www.datamanagementblog.com/data-virtualization-performance-source-system-impact-part-2/">Practitioner analysis of data virtualization incidents</a> frames it plainly: the real failure mode is ad-hoc queries the source was never optimized to handle, and the fix that holds up is constraining query shape at the boundary, for example allowing only queries that specify a customer_id, so no federated query can trigger an unbounded scan regardless of how many run concurrently.</p><p>Layer that with client-side connection pooling in front of whatever issues the queries, and server-level queuing to absorb bursts before they reach the source. None of these live inside Trino’s configuration surface. They’re operational controls you build around it, because the engine, as of Trino 483, still treats every catalog as infinitely resilient to concurrent load, right up until it isn’t.</p><p><strong>In practice, teams that get this right build the controls in this order:</strong></p><ol><li>Constrain query shape at the boundary, for example rejecting any query against a sensitive OLTP catalog that doesn’t specify a customer_id or other bounding predicate, so no federated query can trigger an unbounded scan.</li><li>Add client-side connection pooling in front of whatever issues the queries, so bursts of ad-hoc requests don’t each open a fresh connection to the source.</li><li>Add server-side request queuing at the source, so concurrency spikes are absorbed and smoothed rather than hitting the database’s connection ceiling directly.</li><li>Monitor the mitigation itself, materialized view staleness, pool exhaustion, queue depth, since each of these controls can fail silently and become an outage of its own.</li></ol><p>Visit our website at: <a href="https://yeedu.com/">Yeedu.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b318a2e6602" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/trino-federates-your-queries-not-your-sources-connection-limits-9b318a2e6602">Trino federates your queries, not your source’s connection limits</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Spark Shuffle Breaks Down at Petabyte Scale]]></title>
            <link>https://medium.com/towards-data-engineering/why-spark-shuffle-breaks-down-at-petabyte-scale-b89646e379c9?source=rss-a9bebe2a1730------2</link>
            <guid isPermaLink="false">https://medium.com/p/b89646e379c9</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-spark-shuffle]]></category>
            <category><![CDATA[spark-shuffle]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Yeedu]]></dc:creator>
            <pubDate>Sat, 07 Nov 2026 08:03:33 GMT</pubDate>
            <atom:updated>2026-11-07T08:03:33.615Z</atom:updated>
            <content:encoded><![CDATA[<p>Past a few hundred terabytes of daily shuffle, tuning spark.sql.shuffle.partitions or flipping on adaptive query execution stops moving the needle, because the failure has shifted from job configuration to the shuffle subsystem’s local-disk architecture itself: NVMe drives wearing out months ahead of schedule, fetch failures cascading across stages, and driver metadata growing faster than any heap setting can absorb.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TyekhhlH5khLqHaP7-u5Kw.png" /></figure><h4><strong>Local disk-based shuffle collapses under petabyte-scale write volume</strong></h4><p>Spark’s shuffle was built around a simple assumption: each executor writes its map output to local disk, and reducers fetch directly from that disk over the network. That design held up fine for the 2014-era Daytona GraySort benchmark, where <a href="https://www.databricks.com/blog/2014/10/10/spark-petabyte-sort.html">sort-based shuffle and an external shuffle service let Spark sort a petabyte</a> across 190 i2.8xlarge nodes in 234 minutes, beating Hadoop’s prior petabyte-sort record of 3,800 nodes with roughly 20x fewer machines. The assumption starts failing once an org’s aggregate daily shuffle crosses into the hundreds of terabytes, because now hundreds of mappers are writing to, and thousands of reducers are fetching from, the same local disks on the same compute nodes that are also running the next stage’s tasks.</p><p>This plays out as a hardware procurement problem before it ever looks like a software bug. Per Uber’s account of operating its shuffle infrastructure at scale, sustained shuffle write and read I/O wore out commodity 1TB SSDs, rated for one drive write per day (DWPD) over a three-year design life, in about six months. That’s not a tuning fix. No partition count changes how many bytes a spinning or NVMe disk physically absorbs.</p><h4><strong>Fetch failures stop being exceptions and become a tolerance budget</strong></h4><p>At this volume, FetchFailedException is an expected background rate you plan retries around rather than a sign that something is broken. One widely cited production job read 60TB of compressed input and shuffled 90TB across 250,000 tasks in a single stage, the largest real-world Spark shuffle reported at the time. That job hit fetch failures routinely enough that the operators raised the max allowed fetch failures per stage from 4 to 20 and fixed an O(N²) bottleneck in driver task bookkeeping that made the driver unresponsive under that task count. Teams running Spark at this scale don’t chase fetch failures to zero; they size the retry budget to the failure rate they actually observe.</p><h4><strong>The structural ceilings tuning cannot move past</strong></h4><p>Below petabyte scale, spark.sql.shuffle.partitions and AQE genuinely are the right levers. Spark 3.0 made partition count mostly self-correcting: AQE coalesces small post-shuffle partitions toward spark.sql.adaptive.advisoryPartitionSizeInBytes and auto-splits skewed ones at runtime. But a handful of limits in Spark’s shuffle internals are architectural rather than configurable, and no amount of AQE tuning reaches them.</p><h4><strong>What actually happens when a shuffle block crosses 2GB?</strong></h4><p>Spark’s shuffle and cache block transfer path is backed by a ByteBuffer sized with a 32-bit int, which hard-caps any single block at 2GB regardless of executor memory or cluster size. Cross that line and you’ll see a failure like this:</p><p>org.apache.spark.shuffle.FetchFailedException: Too large frame: &lt;n&gt; <br>caused by: java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE</p><p>SPARK-6235 closed most of this for cached and shuffle blocks in Spark 2.4.0 by replacing raw buffers with ChunkedByteBuffer and streaming abstractions, but the per-block 2GB ceiling itself persists in later versions. The only remedy is forcing more, smaller partitions so no single block’s data crosses 2GB, which is exactly why practitioners default spark.sql.shuffle.partitions into the thousands rather than computing one “correct” value per job, since the right number changes every run anyway.</p><h4><strong>Why does raising the broadcast threshold crash the driver instead of helping?</strong></h4><p>Raising spark.sql.autoBroadcastJoinThreshold past its 10MB default to dodge a shuffle join feels like a win until the driver OOMs, because Spark enforces a hard 8GB and 512-million-row ceiling on any broadcast relation independent of whatever threshold you configured, and the relation is materialized on the driver before it’s ever sent to executors. The error reads plainly: Cannot broadcast the table that is larger than 8GB: &lt;n&gt; GB. Databricks runtimes add a second, separate AQE threshold (spark.databricks.adaptive.autoBroadcastJoinThreshold, default 30MB) that operates at execution time rather than planning time, and conflating the two static and runtime knobs is a common misconfiguration that makes the behavior look inconsistent across jobs.</p><h4><strong>Does AQE’s skew-join handling save you at this scale?</strong></h4><p>Partially, and only for sort-merge joins. AQE flags a partition as skewed when its size exceeds skewedPartitionFactor (default 5.0) times the median partition size and also exceeds skewedPartitionThresholdInBytes (default 256MB), per the <a href="https://spark.apache.org/docs/latest/sql-performance-tuning.html">Spark SQL performance tuning docs</a>. spark.sql.adaptive.forceOptimizeSkewedJoin defaults to false, so this protection simply doesn’t engage for other join strategies. It splits skewed partitions; it does nothing for disk wear, fetch-failure rate, or driver metadata growth.</p><h4><strong>How many fetch failures should you expect to tolerate?</strong></h4><p>There’s no universal number, but the pattern from orgs operating at scale is to raise the per-stage retry ceiling well past Spark’s conservative default of 4 and watch the trend rather than the absolute count. LinkedIn’s clusters, running tens of petabytes a day across more than 10,000 nodes and over 70% of total compute, saw fetch failures grow into the hundreds to low thousands per day before the team concluded the root cause was HDD throughput on small random reads rather than a config value they’d missed.</p><h4><strong>What Meta, Uber and LinkedIn built instead of tuning harder</strong></h4><p>Each of these orgs reached the same conclusion from a different angle: shuffle had to leave the compute node’s local disk entirely.</p><p>Uber runs Spark across more than 10,000 nodes processing hundreds of petabytes a day. Their Remote Shuffle Service moves <a href="https://www.uber.com/blog/ubers-highly-scalable-and-distributed-shuffle-as-a-service/">roughly 8–10PB of daily shuffle</a> off compute nodes onto a 400-node cluster of dedicated servers, each with 80 vCores, 384GB RAM and four 4TB NVMe drives. RSS inverts the usual map-reduce placement: every mapper for a given partition writes to one remote server, so a reducer fetches from a single place instead of fanning out to every mapper’s node. That single change cut shuffle-related container failures by close to 95% and pushed disk wear-out life from roughly six months back out to around three years.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*Ziy6SbkY_YfK_5qI6vufew.png" /></figure><p>LinkedIn took a different first step. Rather than disaggregating shuffle storage immediately, the Magnet design kept shuffle local but converted the access pattern from small random reads into large sequential ones by merging mapper output server-side before reducers fetch it, a fix aimed squarely at HDD throughput rather than CPU. On one roughly 2TB-shuffle ML feature job, <a href="https://www.linkedin.com/blog/engineering/open-source/introducing-magnet">shuffle fetch wait time fell 98%</a>, from 20,636 minutes to 445, and end-to-end runtime improved 26%. Magnet’s push-based shuffle shipped upstream in Spark 3.2 as spark.shuffle.push.enabled.</p><blockquote>spark.shuffle.service.enabled=true <br>spark.shuffle.push.enabled=true <br>spark.shuffle.push.maxBlockSizeToPush=1m <br>spark.shuffle.push.mergersMinThresholdRatio=0.05 <br>spark.network.timeout=800s <br>spark.network.maxRemoteBlockSizeFetchToMem=200m</blockquote><p>LinkedIn later moved past Magnet to a fully disaggregated platform, Apache Celeborn, with NVMe-backed shuffle workers on 100Gbps networking serving both Spark and Flink. Against the Magnet baseline, Celeborn delivered <a href="https://daily.dev/posts/scaling-spark-and-flink-shuffle-at-linkedin-with-apache-celeborn-ekiypbecm">20x fewer task failures during maintenance windows</a>, with daily fetch failures dropping from around 750 to around 20 and overall compute cost falling roughly 10%.</p><h4><strong>Deciding whether your org actually needs this</strong></h4><p>Most teams reading this don’t shuffle hundreds of terabytes a day, and copying Uber’s RSS or standing up Celeborn before you hit that wall adds operational surface area you don’t need yet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*f-cqud57qnLyvU__s3m5bA.png" /></figure><h4><strong>When does it make sense to build or adopt a remote shuffle service yourself?</strong></h4><p>Build or adopt one once local disk wear, rather than CPU or memory, is your binding constraint, and once fetch failures are a cluster-wide pattern rather than a symptom you can isolate to a handful of skewed jobs. Below that point, the baseline AQE configuration below, paired with a high shuffle.partitions ceiling, handles the vast majority of production workloads without taking on a new distributed system to operate.</p><blockquote>spark.sql.shuffle.partitions=4000 <br>spark.sql.adaptive.enabled=true <br>spark.sql.adaptive.coalescePartitions.enabled=true <br>spark.sql.adaptive.advisoryPartitionSizeInBytes=128m <br>spark.sql.adaptive.skewJoin.enabled=true <br>spark.sql.adaptive.skewJoin.skewedPartitionFactor=5 <br>spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes=256m</blockquote><p>We’d treat this as the floor rather than the ceiling. The moment your platform team is replacing NVMe drives on a quarterly cadence instead of a multi-year one, no value you plug into advisoryPartitionSizeInBytes is going to fix that. At that point the conversation shifts from Spark SQL configuration to where shuffle bytes physically live.</p><p>Check our Website <a href="https://yeedu.com/">Yeedu.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b89646e379c9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/why-spark-shuffle-breaks-down-at-petabyte-scale-b89646e379c9">Why Spark Shuffle Breaks Down at Petabyte Scale</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Spark Small-File Problem: Why Compaction Is a Recurring Tax]]></title>
            <link>https://medium.com/towards-data-engineering/the-spark-small-file-problem-why-compaction-is-a-recurring-tax-e652023c6dd4?source=rss-a9bebe2a1730------2</link>
            <guid isPermaLink="false">https://medium.com/p/e652023c6dd4</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Yeedu]]></dc:creator>
            <pubDate>Mon, 26 Oct 2026 14:25:07 GMT</pubDate>
            <atom:updated>2026-10-26T14:25:07.875Z</atom:updated>
            <content:encoded><![CDATA[<p>Spark chokes on small files because every file carries a fixed metadata cost that doesn’t shrink with the data inside it, and running OPTIMIZE or rewrite_data_files on a schedule defers and repackages that cost into a recurring compaction bill paid in GET, PUT and DELETE requests every cycle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YZFQe429Ap8pQL1Bro02ew.png" /></figure><h4><strong>Why small files cripple Spark before a single row loads</strong></h4><p>The damage starts on the driver, long before any executor does work. Spark’s listLeafFiles path issues recursive calls against S3’s list() API to enumerate every object under a prefix, and each of those calls carries real latency that accumulates fast across thousands of objects. Because this listing and partition-size estimation happens entirely driver-side, a job can sit for minutes with zero tasks running and nothing to show for it in the UI.</p><h4><strong>Why does the job look idle while nothing executes?</strong></h4><p>The “tasks running: 0” view in the Spark UI is misleading, because the driver is still walking the directory tree and has not yet handed work to any executor. Teams hitting this usually cut the number of objects the driver has to enumerate, or avoid the S3 list API altogether with a secondary index or a queue of new-file notifications, per <a href="https://spoddutur.github.io/spark-notes/s3-filesystem-as-datasource-in-spark.html">this practitioner writeup</a>.</p><h4><strong>What does a small file actually cost per read?</strong></h4><p>Each file forces Spark to open a connection, read the Parquet footer, parse the schema and extract column statistics before it touches a single row, which analysis of small-file overhead on object storage puts at roughly 200ms of pure metadata overhead per file on S3. Multiply that across a partition with tens of thousands of files and the overhead dwarfs the actual scan. At object-storage request pricing of $0.005 per 1,000 LIST calls, reading a million small files runs roughly 1,000 times the API-call cost of reading the same data as a thousand right-sized files.</p><h4><strong>How many files does it take to break the driver or the NameNode?</strong></h4><p>On HDFS the NameNode holds about 150 bytes of heap per file, so 10 million files consume roughly 3 GB of heap, and production clusters have reported NameNode OutOfMemoryError failures under 30 million files. On the Spark side, schema inference across many small JSON files can exhaust the driver before any executor even starts, surfacing as The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached. The file count is what breaks it, regardless of the data volume. A 50MB file with deeply nested fields can still blow the driver’s inference budget.</p><p>Supplying an explicit StructType instead of letting Spark infer one sidesteps it entirely, per the <a href="https://kb.databricks.com/en_US/python/apache-spark-driver-stops-and-restarts-while-reading-json-file-data-in-an-s3-bucket-">Databricks KB on this exact failure</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*StTXz46I9FgwqLVnbyT7yw.png" /><figcaption><em>Diagram 1</em></figcaption></figure><h4><strong>Compaction works, but it isn’t free</strong></h4><p>Running rewrite_data_files or OPTIMIZE does fix the symptom. AWS’s own guidance traces the mechanism directly: a stage reading 430 files totalling 47.4 GiB to produce 160,796,570 rows shows the file count sitting right in the Spark UI’s SQL/DataFrame tab, and consolidating those files toward the 128 MB target AWS recommends for input and output removes the per-file overhead without touching the query itself, per the <a href="https://docs.aws.amazon.com/prescriptive-guidance/latest/tuning-aws-glue-for-apache-spark/reduce-data-scan.html">AWS Prescriptive Guidance on reducing data scan</a>.</p><p>That’s the part everyone quotes. What gets left out is what the compaction job itself costs to run.</p><h4><strong>Why does a compaction cycle cost more than it looks like?</strong></h4><p>Every rewrite operation reads the old files and writes new ones, which means the storage layer sees the same PUT and DELETE volume on every single cycle, not just once as a cleanup pass. Until the old files are expired, the table also temporarily holds roughly double its steady-state storage footprint, per Dremio’s breakdown of Iceberg storage costs. Schedule OPTIMIZE nightly against a table that keeps generating small files at the same rate, and you’ve built a standing write-amplification cost on top of the read-side tax you were trying to remove.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*tPkhCVEJ-TAMW6-lagz3og.png" /><figcaption><em>Diagram 2</em></figcaption></figure><h4><strong>When is a compaction job wasted compute?</strong></h4><p>Setting a compaction trigger threshold too aggressively low means the job spends cycles rewriting partitions that are already close to target size, for no measurable read-side benefit. Iceberg guards against this directly: files below min-file-size-bytes, which defaults to 75% of the target size, are the only ones treated as rewrite candidates, so a partition sitting at 220MB against a 256MB target gets left alone rather than rewritten for nothing.</p><h4><strong>What OPTIMIZE and rewrite_data_files actually do</strong></h4><p>Both tools do the same fundamental thing: bin-pack existing files up to a target size, optionally scoped to a predicate so already-compacted partitions aren’t touched again.</p><pre>CALL catalog.system.rewrite_data_files( <br> table =&gt; &#39;db.orders&#39;, <br> strategy =&gt; &#39;binpack&#39;, <br> options =&gt; map( <br> &#39;target-file-size-bytes&#39;, &#39;268435456&#39;, <br> &#39;min-input-files&#39;, &#39;5&#39; <br> ) <br>)</pre><p>This targets 256MB files using Iceberg’s binpack strategy, the fastest option because it concatenates files to size without a shuffle or sort, per the <a href="https://iceberg.apache.org/docs/latest/spark-procedures/">Iceberg Spark procedures docs</a>. min-input-files keeps it from firing a rewrite group over a handful of files that don’t justify the job overhead.</p><h4><strong>Controlling file size at write time instead of after</strong></h4><p>If compaction is a recurring bill, the only way to actually reduce it is to shrink what you owe at write time. That means tuning the write path so files land near target size on the first commit, rather than relying on a nightly job to clean up afterward.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HYTpQFo2ZS0DRbKIuP1AWg.png" /></figure><h4><strong>Delta’s optimizeWrite and Auto Compaction thresholds</strong></h4><p>optimizeWrite adds a shuffle stage before the commit so output lands in roughly 512MiB bins rather than one file per task, with optimizeWrite.numShuffleBlocks capped at 50,000,000 and maxShufflePartitions at 2,000, per the Delta Lake optimizations docs. Auto Compaction sits downstream of that as a backstop: once a partition’s file count crosses minNumFiles (50) while files sit under maxFileSize (128MB), Delta fires a synchronous OPTIMIZE right after the commit, rather than waiting for a scheduled job. We’d rather pay that cost inline, on the write that caused it, than let it accumulate into a bigger batch job later.</p><h4><strong>Iceberg, AQE, and Glue’s read-time grouping</strong></h4><p>Iceberg exposes target sizing per-table rather than per-session, and min-file-size-bytes at 75% of target keeps the rewrite scope honest. On the engine side, AQE’s coalescePartitions folds together shuffle partitions below minPartitionSize before they ever become separate output files, which is Spark’s own write-time defense against fragmentation, available from Spark 3.2.0+. On the read side, when upstream files are already small and out of your control, Glue’s DynamicFrame reader can group many tiny inputs into larger logical splits before task creation:</p><pre>dyf = glueContext.create_dynamic_frame_from_options( <br> &quot;s3&quot;, <br> {&#39;paths&#39;: [&quot;s3://input-s3-path/&quot;], <br> &#39;recurse&#39;: True, <br> &#39;groupFiles&#39;: &#39;inPartition&#39;, <br> &#39;groupSize&#39;: &#39;1048576&#39;}, <br> format=&quot;json&quot;)</pre><p>Setting groupSize to 1048576 bytes bundles roughly 1MB of small files per logical read task, cutting per-file task overhead and driver OOM risk, per AWS’s Glue tuning guide. It’s a read-side workaround, and it’s the right tool when you don’t own the producer.</p><h4><strong>CDC and streaming tables outrun any compaction schedule</strong></h4><p>Upsert-heavy pipelines break the compaction math outright. CDC writes are high-frequency and low-volume per event, so a copy-on-write compaction after every commit would mean rewriting full data files over and over through the day, which is write amplification by design.</p><h4><strong>Merge-on-read defers the bill</strong></h4><p>Iceberg’s merge-on-read approach with delete files sidesteps that by deferring the rewrite: updates land as small delete markers against existing data files, and a batched compaction job reconciles them later instead of on every commit. That trades write-side amplification for read-side delete-file accumulation, which is a better trade for most CDC workloads, but it’s still a trade. Scheduled OPTIMIZE on a table taking continuous small upserts frequently can’t keep pace, so the table never actually settles at its target file size between runs.</p><p>Format v3’s deletion vectors change the arithmetic somewhat. Binary, position-based vectors supersede the older positional delete files specifically, cutting read and write amplification versus that approach, though equality deletes remain a supported option alongside them. They narrow the gap. They don’t remove the underlying tension between how fast CDC generates files and how fast any batch job can clear them, which is exactly why write-time control is the lever worth pulling first.</p><p>Check out our website <a href="http://yeedu.com/">yeedu.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e652023c6dd4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/the-spark-small-file-problem-why-compaction-is-a-recurring-tax-e652023c6dd4">The Spark Small-File Problem: Why Compaction Is a Recurring Tax</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Kafka-to-Flink pipelines still deliver duplicates even with exactly-once enabled]]></title>
            <link>https://yeedu.medium.com/why-kafka-to-flink-pipelines-still-deliver-duplicates-even-with-exactly-once-enabled-088d6099acac?source=rss-a9bebe2a1730------2</link>
            <guid isPermaLink="false">https://medium.com/p/088d6099acac</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[flink]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Yeedu]]></dc:creator>
            <pubDate>Wed, 07 Oct 2026 08:30:27 GMT</pubDate>
            <atom:updated>2026-10-07T08:30:27.221Z</atom:updated>
            <content:encoded><![CDATA[<p>Exactly-once checkpointing in Flink guarantees that Flink’s own state and any sink that fully implements two-phase commit agree with the last completed checkpoint, not that every record a downstream system sees was written exactly once. The gap between “committed to the sink” and “checkpoint marked complete” is where duplicates live, and production bug reports show it is a real gap, not a theoretical one. We walk through two of them below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*94_gWmW4JKiZ5R-kiUHLog.png" /></figure><h3><strong>What exactly-once actually covers</strong></h3><h4><strong>Two-phase-commit sink requirements</strong></h4><p>Flink’s <a href="https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/fault-tolerance/checkpointing/">checkpointing documentation</a> is fairly blunt about the scope of EXACTLY_ONCE mode. It covers Flink’s internal state and any sink that implements the two-phase-commit protocol, full stop. A two-phase-commit sink has to hold a pointer to its last open transaction across a checkpoint boundary, and the task blocks on the final checkpoint succeeding before the operator is allowed to close. That is a lot of coordination machinery for a fairly narrow promise, as we’ll see below.</p><p>It gets narrower. Flink offers no processing guarantees at all for iterative jobs, since records in flight on loop edges are simply dropped on failure. And any state checkpointed after close() or finish() is discarded on restore, with the sole exception of the transaction pointer the exactly-once sink needs to recover cleanly. None of this is a criticism of Flink’s design, it’s a precise contract. The trouble starts when we read “exactly-once checkpointing” and assume it means “exactly-once delivery,” full stop, everywhere downstream.</p><p>On the Kafka side, the underlying primitives are equally specific. <a href="https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/">Confluent’s writeup</a> on idempotent producers describes how each producer gets a Producer ID and a per-partition sequence number, persisted to the replicated log so a leader failover does not reset the dedup state. That’s solid, and pairing it with a stable transactional.id is what gives a producer continuity of transactional state across application restarts. The guarantee still only covers what happens inside Kafka’s own transaction log, which is a narrower boundary than most postmortems assume, in our experience.</p><p>Here’s what that boundary actually looks like in practice, component by component:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*ITuhqto9ssoEJjpMz4OrmQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_DpY5k-sOhNF_By4US5sgg.png" /></figure><h4><strong>What does isolation.level=read_committed actually protect against?</strong></h4><p>It stops a downstream consumer from reading records that belong to a transaction that later aborted. Kafka’s default is read_uncommitted, which means a consumer sees every write the moment it lands, including ones the producer subsequently rolled back. Set isolation.level=read_committed and the consumer waits for the transaction marker before exposing those records at all.</p><p>We’ve seen teams forget to apply this to verification tooling constantly. <a href="https://www.conduktor.io/kafka-streams/exactly-once-duplicates">Conduktor’s analysis</a> of Kafka Streams duplicate reports lists this as the single most common cause: someone points a plain console consumer or a monitoring job at the output topic, it defaults to read_uncommitted, and it reports phantom duplicates from aborted transactions that a properly configured downstream reader would never see.</p><h3><strong>Where the checkpoint-commit gap actually bites</strong></h3><p>This is the part we find the marketing slide skips. Two documented incidents show duplicates surviving textbook-correct configuration.</p><p><a href="https://www.mail-archive.com/dev@flink.apache.org/msg44101.html">FLINK-20753</a>, filed against Flink 1.12.0, is about as clean a reproduction as we’ve found. The job had a 10-second checkpoint interval, EXACTLY_ONCE mode, RocksDB state backend, idempotence enabled on the producer, acks=all, max.in.flight.requests=1, and a 15-minute transaction timeout. Every box checked. After a SIGKILL and restart, records t1-k-48 and t1-k-49 showed up twice in partition 0 of the sink topic. The root cause: the producer failed to roll back a partially-completed transaction before replaying data on recovery, so the old write and the replayed write both landed.</p><p>Then there is <a href="https://github.com/apache/iceberg/issues/10765">the Iceberg issue</a>, reported against Iceberg 1.4.3 running under Flink. Checkpoint 19516 completed cleanly at 04:57:33. Checkpoint 19517 was triggered a minute later and timed out at 04:58:42 with a RemoteRpcInvocation error. On restart, Flink rolled back to checkpoint 19516, and the committer re-ran the commit that had, in fact, already succeeded. The table ended up with two snapshots, both referencing the exact same 17,554-row parquet file. Table maintenance couldn’t fix it either: the report notes both rewrite_data_files and expire_snapshots failed to remove the duplicated rows. We’d call that the worst kind of duplicate: one you can’t clean up after the fact.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*p1TKvH3KH8pB8ScSyth_fg.png" /></figure><p>Two different bugs, two different sinks, same shape of failure. Data gets physically committed to the sink before Flink’s checkpoint is marked complete, and that ordering isn’t atomic even with a two-phase-commit sink involved, in our reading of both reports.</p><h3><strong>Why does exactly-once configuration still leave gaps outside Kafka?</strong></h3><p>Because Kafka’s transactional guarantees only extend as far as Kafka’s own log. Anything a consumer does outside that transaction, a database write, an HTTP call, a webhook, a cache update, happens with no atomicity tie to the Kafka commit at all, and reprocessing after a retry repeats it.</p><p>Conduktor’s breakdown names this directly: side effects outside the transaction boundary just replay on retry, full stop. A <a href="https://medium.com/@sudeepwrites/stop-lying-to-yourself-your-kafka-flink-pipeline-is-not-idempotent-heres-the-ugly-truth-9b8cbe402515">practitioner writeup on the same theme</a> puts it more sharply, arguing that most pipelines calling themselves exactly-once are really running at-least-once with a prayer attached. We’d agree with that framing. Non-transactional JDBC sinks, arbitrary REST calls, external caches: none of it is covered by Kafka’s or Flink’s transaction machinery no matter how carefully the Kafka side is tuned.</p><h3><strong>What causes non-deterministic replay to silently diverge?</strong></h3><p>There’s a third, subtler failure mode Conduktor’s breakdown calls out separately: non-determinism. If a retried operator reads wall-clock time, generates a UUID, or queries mutable external state during reprocessing, the replay produces a different output than the original attempt did. Exactly-once semantics assumes replaying identical inputs yields identical outputs. Break that assumption and you get silent divergence, not a clean duplicate you could at least filter on. We’ve watched this bite teams harder than a plain duplicate would.</p><h3><strong>Producer fencing and the zombie-writer problem</strong></h3><p><a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging">KIP-98</a>, the original design doc for Kafka’s transactional messaging, defines producer fencing: a new producer instance sharing a transactional.id bumps an epoch and fences out the previous one, which then gets a ProducerFencedException on its next write. That is meant to stop stale (“zombie”) producers from corrupting state after a failover.</p><p>It’s a clean mechanism on paper: the old producer is expected to receive ProducerFencedException the next time it tries to write, and shut down. KIP-98 itself flags a subtler race: a consumer-group rebalance can leave a stalled consumer instance still holding a reference to a producer that has already been fenced, and that instance can attempt a write after fencing has occurred. The design assumes the application shuts down cleanly on ProducerFencedException rather than retrying, which is a coordination requirement the application has to honor, not something the broker can enforce on its own.</p><p><strong>Practical fixes: idempotent sinks and upsert patterns</strong></p><p>Idempotent sinks. Design the write so applying it once or a hundred times produces the same end state, then duplicates stop mattering regardless of what caused them.</p><p>The pattern practitioners converge on is upsert-on-primary-key at the destination:</p><blockquote><strong>INSERT</strong> <strong>INTO</strong> orders (<strong>id</strong>, total) <strong>VALUES</strong> (?, ?) <br><strong>ON</strong> CONFLICT (<strong>id</strong>) DO <strong>UPDATE</strong> <strong>SET</strong> total = EXCLUDED.total;</blockquote><p>Putting this into practice follows a consistent sequence once you’ve identified an at-risk sink:</p><ol><li>Identify every sink that receives writes outside Kafka’s or Flink’s transaction boundary — databases, HTTP calls, caches, webhooks.</li><li>Attach a natural or synthetic primary key to each output record so a retried write can be matched against a prior one.</li><li>Replace plain INSERT statements with upsert semantics (ON CONFLICT … DO UPDATE, or MERGE) keyed on that identifier.</li><li>For streaming SQL paths, deduplicate before the sink using a ROW_NUMBER() window so only the latest record per key reaches the write.</li><li>Set isolation.level=read_committed on every downstream consumer, including verification and monitoring tooling.</li><li>Configure transaction.state.log.replication.factor=3 and transaction.state.log.min.isr=2 on the broker so initTransactions() doesn’t hang.</li><li>Load test at your real commit interval to confirm the throughput cost is acceptable before shipping.</li></ol><p>For streaming SQL, the equivalent is a deduplication query using ROW_NUMBER() OVER (PARTITION BY key ORDER BY event_time) to keep one record per key within a window before it ever reaches the sink. Neither of these is exotic. They’re standard reconciliation techniques that happen to be the only thing actually protecting the destination table when a checkpoint boundary and a sink commit fall out of sync, which the FLINK-20753 and Iceberg incidents both show happens in real deployments, not just in edge-case thought experiments.</p><p>There’s a cost to running exactly-once mode even when it works as designed. Confluent’s own benchmarks describe idempotence alone as having negligible throughput impact, but full transactional writes at a 100ms commit interval for 1KB messages cost roughly 3% versus at-least-once, and Kafka Streams-style short commit intervals can cost 15 to 30% depending on message size.</p><p>That’s a real number to weigh against a guarantee that, per the incidents above, doesn’t actually eliminate the need for idempotent sinks anyway. Broker-side, transaction.state.log.replication.factor=3 and transaction.state.log.min.isr=2 need to be set correctly too, or initTransactions() simply hangs for the default 60-second timeout and the whole application stalls in a rebalancing state before it processes a single record.</p><p>None of this argues against enabling EXACTLY_ONCE checkpointing. It argues against treating it as the finish line. Kafka’s transactions and Flink’s checkpoint barriers buy real protection for the parts of the pipeline they actually cover, and idempotent, upsert-based sinks catch what falls outside that boundary, which in practice is most of what a duplicate-sensitive downstream system actually cares about.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=088d6099acac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ClickHouse’s ‘Too Many Parts’ Wall: Why Streaming Small Inserts Breaks a Batch-Oriented Merge…]]></title>
            <link>https://medium.com/towards-data-engineering/clickhouses-too-many-parts-wall-why-streaming-small-inserts-breaks-a-batch-oriented-merge-acd972fcc21e?source=rss-a9bebe2a1730------2</link>
            <guid isPermaLink="false">https://medium.com/p/acd972fcc21e</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[clickhouse]]></category>
            <category><![CDATA[databases-big-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Yeedu]]></dc:creator>
            <pubDate>Fri, 18 Sep 2026 13:00:47 GMT</pubDate>
            <atom:updated>2026-09-18T13:00:47.675Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>ClickHouse’s ‘Too Many Parts’ Wall: Why Streaming Small Inserts Breaks a Batch-Oriented Merge Scheduler</strong></h3><p>ClickHouse throws DB::Exception: Too many parts (600). Merges are processing significantly slower than inserts. when the background merge scheduler falls behind the rate at which INSERTs create new parts, and the usual fixes, async_insert and Buffer tables, only delay the wall rather than remove it because MergeTree was built for batch writes, not row-at-a-time streaming.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3asjzfzfdYT8Q4WqT44hsg.png" /></figure><h3><strong>Why part accumulation happens in the first place</strong></h3><p>Every synchronous INSERT into a MergeTree table creates at least one part on disk. A batch of 10,000 rows in one statement produces one part. The same 10,000 rows sent as 10,000 individual single-row inserts produce 10,000 parts, identical data, but radically more work for the background merge pool to fold back together. Background merges exist to fix this, folding small parts into progressively larger ones. But merging is bounded by CPU threads and disk throughput, while part creation is bounded by nothing except how fast your client can open a connection. When creation outruns consumption, active parts pile up per partition, and ClickHouse has two guardrails against that: slow the inserts down, then refuse them outright.</p><h3><strong>Delay-and-throw thresholds and their version history</strong></h3><p>Two MergeTree settings gate this behaviour: parts_to_delay_insert, which artificially slows INSERTs once a partition crosses it, and parts_to_throw_insert, the hard ceiling past which ClickHouse rejects writes outright. With the pre-23.6 default of 300, that shows up as DB::Exception: Too many parts (301). On versions before 23.6 those defaults were 150 and 300. From <a href="https://clickhouse.com/docs/operations/settings/merge-tree-settings">23.6 onward</a> they’re 1000 and 3000, alongside a newer max_avg_part_size_for_too_many_parts (default 1 GiB) that only trips the check when parts are both numerous <em>and</em> small. Dashboards and blog posts written against pre-23.6 clusters will cite numbers that no longer apply to your cluster, so check your server version before trusting any threshold advice, including this article’s.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*youpH37lbaAJ8yc5oiZZMw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nqg07dBvZm413ybzS_OitA.png" /></figure><p>That size-aware refinement wasn’t an accident. In 2019, ClickHouse creator Alexey Milovidov opened <a href="https://github.com/ClickHouse/ClickHouse/issues/6551">issue #6551</a> pointing out that a 150 TB table with 1,000 parts of 150 GB each is perfectly healthy, so a flat part-count cap unfairly penalizes large tables. His proposed fix, delay or throw only when part count is high and average part size is below a fraction of the max, is what eventually shipped as max_avg_part_size_for_too_many_parts. The threshold has been under active redesign for years. It is not a knob someone forgot to document; it is a genuinely hard scheduling problem. The <a href="https://github.com/ClickHouse/ClickHouse/issues/3174">original bug report</a> for this error, filed against server 18.6.0, shows the failure mode hasn’t changed shape since: a user hit the wall inserting 177 GB across 2,050 files even after manually raising parts_to_delay_insert to 300 and parts_to_throw_insert to 600. The exception traces back to DB::MergeTreeData::delayInsertOrThrowIfNeeded(), the merge queue falling behind, not a disk-space or file-descriptor limit. Raising the ceiling doesn’t fix that; it just moves the point of failure further out.</p><p><strong>Diagnosing which partition is actually the problem</strong></p><p>Before touching any setting, find where parts are accumulating. ClickHouse only merges parts within the same partition, so a high-cardinality partition key (daily buckets on a years-old table, or partitioning by user ID) multiplies the number of independent merge queues competing for the same background pool. <a href="https://bigdataboutique.com/blog/clickhouse-too-many-parts">BigDataBoutique’s writeup</a> treats 100 to 300 active parts per partition as the signal that merges aren’t keeping up, and above 300 as requiring action now, not next sprint.</p><p>The diagnostic workflow, in order:</p><ol><li>Identify the partition key that’s multiplying merge queues, since ClickHouse only merges parts within a single partition, never across partitions.</li><li>Run the query below against system.parts for the table showing symptoms.</li><li>Compare the returned active_parts and average row count against the severity table below.</li><li>Treat a climbing part count paired with shrinking average part size as an active problem, not a metric to revisit later.</li></ol><p><strong>SELECT</strong> <br> <strong>database</strong>, <strong>table</strong>, partition_id, <br> count() <strong>AS</strong> active_parts, <br> sum(<strong>rows</strong>) <strong>AS</strong> <strong>rows</strong>, <br> formatReadableSize(sum(bytes_on_disk)) <strong>AS</strong> size_on_disk <br><strong>FROM</strong> <strong>system</strong>.parts <br><strong>WHERE</strong> active <strong>AND</strong> <strong>database</strong> = ‘my_db’ <strong>AND</strong> <strong>table</strong> = ‘my_table’ <br><strong>GROUP</strong> <strong>BY</strong> <strong>database</strong>, <strong>table</strong>, partition_id <br><strong>ORDER</strong> <strong>BY</strong> active_parts <strong>DESC</strong>;</p><p>ClickHouse’s own <a href="https://clickhouse.com/docs/tips-and-tricks/too-many-parts">tips-and-tricks page</a> grades severity on this query’s output: below 100 parts is fine, above 100 is caution, above 500 is a warning, above 1000 is critical. It grades average part size too: under 1,000 rows per part is poor, over 100,000 is excellent. Run this on a schedule, not just when the exception fires.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/1*JOQaZR_1I4gac0XozvOtSQ.png" /></figure><h3><strong>The workarounds, and where each one still fails</strong></h3><p>Two mechanisms exist specifically to stop small, frequent inserts from becoming small, frequent parts. Neither is a substitute for controlling write shape at the source.</p><h4><strong>Should you just enable async_insert and move on?</strong></h4><p>Enabling it shifts batching responsibility to the server, but it still writes durably before acknowledging, so it doesn’t remove the underlying part-creation pressure if your insertion rate genuinely exceeds merge capacity. It buffers rows in memory and flushes as a single part once a size or time threshold hits.</p><p><strong>INSERT</strong> <strong>INTO</strong> my_table <br>SETTINGS <br> async_insert = 1, <br> wait_for_async_insert = 1 <br><strong>VALUES</strong> (…);</p><p>The <a href="https://clickhouse.com/docs/optimize/asynchronous-inserts">defaults</a> matter here: async_insert_max_data_size flushes at 10 MiB (100 MiB on ClickHouse Cloud), async_insert_busy_timeout_ms flushes at 200ms (1000ms on ClickHouse Cloud), whichever comes first. Since 24.2, async_insert_use_adaptive_busy_timeout adjusts that window dynamically between 50ms and 200ms instead of using a fixed value. Setting wait_for_async_insert = 0 turns this into fire-and-forget: rows sitting in the server-side buffer are lost outright if the process crashes before flush. That’s a durability tradeoff, not a free lunch, and it’s easy to miss because the INSERT still returns success.</p><h3><strong>Why do Buffer tables still lose data teams thought was safe?</strong></h3><p>Buffer engine tables hold rows in RAM and periodically flush them to a MergeTree destination, which was the original answer to many-small-inserts before async_insert existed. The catch is durability: data in the buffer is volatile, so a crash before flush loses whatever hadn’t been written yet, exactly the failure mode wait_for_async_insert = 0 reintroduces. <a href="https://blog.hellmar-becker.de/2025/10/07/pitfalls-using-buffer-tables-in-clickhouse-cloud/">ClickHouse Cloud’s</a> own pitfalls documentation catalogs the ways Buffer tables misbehave there. Part of the mismatch is architectural: Cloud’s default <a href="https://clickhouse.com/docs/cloud/reference/shared-merge-tree">SharedMergeTree</a> engine keeps all durable data in shared object storage and Keeper, with local disk serving only as ephemeral cache, so a RAM-resident Buffer table sits awkwardly in front of it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tyOP-vx7Q-04SBvRKQFScA.png" /></figure><h3><strong>When the fix has to be architectural, not a setting</strong></h3><p>If merges are structurally behind, not just momentarily spiky, more threads buy real headroom. <a href="https://kb.altinity.com/altinity-kb-setup-and-maintenance/altinity-kb-aggressive_merges/">Altinity’s aggressive-merges guidance</a> recommends dropping background_merges_mutations_concurrency_ratio from its default of 2 down to 1 while clearing an active backlog, since a lower ratio means merges get scheduled more eagerly relative to the pool size rather than being spread thin across mutations too.</p><p>&lt;<strong>clickhouse</strong>&gt; <br> &lt;<strong>background_pool_size</strong>&gt;36&lt;/<strong>background_pool_size</strong>&gt; <br> &lt;<strong>background_merges_mutations_concurrency_ratio</strong>&gt;1&lt;/<strong>background_merges_mutations_concurrency_ratio</strong>&gt; <br> &lt;<strong>merge_tree</strong>&gt; <br> &lt;<strong>number_of_free_entries_in_pool_to_lower_max_size_of_merge</strong>&gt;32&lt;/<strong>number_of_free_entries_in_pool_to_lower_max_size_of_merge</strong>&gt; <br> &lt;/<strong>merge_tree</strong>&gt; <br>&lt;/<strong>clickhouse</strong>&gt;</p><p>That last setting should sit at 50 to 90 percent of background_pool_size, here 32 on a 36-core host, near the top of that band so large merges start shrinking only once free pool slots are genuinely scarce, not before. Sometimes the honest fix is admitting the defaults were fine and the workload was wrong. Once you’ve confirmed merges genuinely can’t keep pace even with a healthy pool, you can move the ceiling deliberately rather than reactively:</p><p><strong>ALTER</strong> <strong>TABLE</strong> my_table <strong>MODIFY</strong> SETTING <br> parts_to_delay_insert = 1000, <br> parts_to_throw_insert = 3000, <br> max_avg_part_size_for_too_many_parts = 1073741824;</p><p>This is worth doing consciously on 23.6+, where those are already the shipped defaults, not as an emergency override on an older cluster still running 150/300.</p><h3><strong>This keeps happening at companies that know what they’re doing</strong></h3><p><a href="https://www.posthogstatus.com/">PostHog runs a public status page</a> for its ClickHouse-backed infrastructure, and ingestion-lag incidents on its EU cluster show up in that history. HN threads about running ClickHouse in production echo similar lessons from the operator side. One self-hosted user reports seven years of high-volume, 24/7 operation without real issues. Others counter that you need real knowledge of the internals to use it effectively, particularly around table partitioning and ordering-key design, before that kind of stability holds. A separate commenter warns that INSERT … SELECT needs to be chunked manually, because the generated partitions coexist in memory before flushing, so an unbounded SELECT can generate an unbounded number of parts in one shot.</p><p>None of this is really about a missing config value. The Kafka-engine-plus-materialized-view pattern many teams build expecting low-latency, row-at-a-time delivery is working against that assumption, not with it. Async_insert and Buffer tables move the batching decision server-side; they don’t remove MergeTree’s underlying identity as a batch-oriented compactor with hard per-partition ceilings. Raise parts_to_throw_insert all you like. If your write pattern is fundamentally one row per network round trip, you’ve bought time, not headroom, and production incident histories like PostHog’s suggest even mature, well-tuned clusters eventually cash that check.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=acd972fcc21e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/clickhouses-too-many-parts-wall-why-streaming-small-inserts-breaks-a-batch-oriented-merge-acd972fcc21e">ClickHouse’s ‘Too Many Parts’ Wall: Why Streaming Small Inserts Breaks a Batch-Oriented Merge…</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Idempotent backfills: why the fast partition-overwrite patterns in Spark, Delta and Iceberg are the…]]></title>
            <link>https://yeedu.medium.com/idempotent-backfills-why-the-fast-partition-overwrite-patterns-in-spark-delta-and-iceberg-are-the-2bc15a9407c6?source=rss-a9bebe2a1730------2</link>
            <guid isPermaLink="false">https://medium.com/p/2bc15a9407c6</guid>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Yeedu]]></dc:creator>
            <pubDate>Mon, 14 Sep 2026 12:25:48 GMT</pubDate>
            <atom:updated>2026-09-14T12:25:48.255Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Idempotent backfills: why the fast partition-overwrite patterns in Spark, Delta and Iceberg are the ones most likely to corrupt data</strong></h3><p>Every backfill path we reach for because it’s fast, Spark’s dynamic partition overwrite, Delta’s replaceWhere, Iceberg’s INSERT OVERWRITE, has a documented case where it deletes, duplicates, or silently admits data it shouldn’t. MERGE INTO, the option most teams skip because it’s slower, is the one that stays correct when partition schemes change or a connector misbehaves underneath you.</p><p>We keep reaching for partition overwrite because it’s cheap and legible: replace the day, move on. But “replace the day” is an assumption about what the engine actually does under the hood, and that assumption breaks in different ways on every platform we’ve tested it against.</p><h3><strong>Why the fast backfill primitives are the risky ones</strong></h3><p>Dynamic partition overwrite in Spark, enabled since 2.3, only touches partitions that appear in the batch you’re writing. It does not pre-delete anything.</p><p><em># Spark dynamic partition overwrite for a daily backfill</em> <br>spark.conf.set(“spark.sql.sources.partitionOverwriteMode”, “dynamic”) <br> <br>(df.write <br> .mode(“overwrite”) <br> .partitionBy(“event_date”) <br> .saveAsTable(“analytics.events”)) <br><em># Only partitions present in df’s event_date values are replaced;</em> <br><em># any event_date NOT in this batch is left untouched, not cleared.</em></p><p>That’s the whole appeal and the whole hazard in one behavior. If a batch is missing rows for a partition key it’s supposed to cover, dynamic mode leaves the stale data sitting right next to whatever did land. Paul Staab’s writeup is explicit that idempotency here depends entirely on every delivery containing the <em>complete</em> set of rows for any partition it touches. Miss a row, and the old wrong value survives quietly. There’s no error, no log line, nothing to grep for. That’s what makes it dangerous: a broken upstream extract that drops 5% of rows for one partition still lets the job succeed, quietly leaving 5% of last month’s numbers sitting in this month’s table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8AJxByMukZhiAUaMr1Uu_w.png" /></figure><p>Now layer a non-native sink on top. On Databricks Runtime 15.4 LTS running Spark 3.5.0, teams writing through the spark-bigquery-connector set partitionOverwriteMode to DYNAMIC at the cluster level, at the writer-option level, and via the DataFrame writer, and the connector <a href="https://github.com/GoogleCloudDataproc/spark-bigquery-connector/issues/1325">deleted the entire target table anyway</a>, well beyond the single partition being backfilled. The issue was closed with no fix landed, still tagged “waiting for information.” Dynamic partition overwrite is a Spark-side contract; it does not automatically hold once the write path crosses into a connector that translates the write into its own semantics.</p><h3><strong>How does Delta’s replaceWhere fail differently?</strong></h3><p>replaceWhere forces you to write an explicit predicate instead of trusting inferred partition membership, and by default it validates every written row against that predicate before committing.</p><p>(replace_data.write <br> .mode(“overwrite”) <br> .option(“replaceWhere”, “start_date &gt;= ‘2017–01–01’ AND end_date &lt;= ‘2017–01–31’”) <br> .saveAsTable(“events”)) <br> <br><em># Disabling this makes rows outside the predicate get silently inserted:</em> <br>spark.conf.set(“spark.databricks.delta.replaceWhere.constraintCheck.enabled”, False)</p><p>That constraint check is the safety net, and <a href="https://docs.databricks.com/aws/en/delta/selective-overwrite">Databricks’ own docs</a> note it can be turned off, at which point rows outside the boundary get written with no warning at all. SQL REPLACE WHERE needs Databricks Runtime 12.2 LTS or later; the Python and Scala option has worked since DBR 9.1 LTS. Even with the check enabled, it isn’t perfectly deterministic. One Delta user running Spark 3.2.1 against a table partitioned on both dt and processed_dt backfilled ten identical daily batches with the same code and got:</p><p>AnalysisException: Data written out does not match replaceWhere ‘processed_dt == ‘2022–05–18’’</p><p>on three of the ten, and success on the other seven, <a href="https://github.com/delta-io/delta/issues/1147">with no logic change between runs</a>. If you’ve seen that exact string in your job logs, this is why: the predicate-versus-partition interaction here is not fully stable across otherwise identical writes on multi-column partitioned tables.</p><h3><strong>What actually goes wrong when a partition scheme changes mid-flight?</strong></h3><p>Repartitioning a table, say from daily to hourly grain, breaks the assumption INSERT OVERWRITE depends on: that a partition boundary uniquely identifies a slice of data.</p><p>Iceberg’s own docs describe this directly: after a repartitioning event, INSERT OVERWRITE clears only the partitions that match the <em>current</em> spec, leaving rows written under the old spec in place. Expedia’s engineering team walked through the concrete case: a table moves from daily to hourly partitioning, and a subsequent overwrite for “today” only touches the hourly buckets, leaving the old daily partition’s rows for that same day sitting untouched. Query across both and you double-count the day. Their conclusion, echoed in the <a href="https://iceberg.apache.org/docs/1.5.0/spark-writes/">Iceberg Spark writes docs</a> themselves: MERGE INTO is recommended over INSERT OVERWRITE precisely because it operates at the file level and doesn’t care what the partition spec looked like when a given file was written.</p><h3><strong>The case for defaulting to MERGE INTO</strong></h3><p>The objection to MERGE INTO is always cost: it’s a join, it rewrites more, it’s slower. That’s true in the worst case, and false often enough that we shouldn’t assume it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*wv1UzgiCEBYrllMuvxnqwg.png" /></figure><p><strong>MERGE INTO’s cost and correctness trade-off</strong></p><p><strong>MERGE</strong> <strong>INTO</strong> analytics.<strong>events</strong> t <br><strong>USING</strong> backfill_batch s <br><strong>ON</strong> t.event_id = s.event_id <strong>AND</strong> t.event_date = s.event_date <br><strong>WHEN</strong> MATCHED <strong>THEN</strong> <strong>UPDATE</strong> <strong>SET</strong> * <br><strong>WHEN</strong> <strong>NOT</strong> MATCHED <strong>THEN</strong> <strong>INSERT</strong> *</p><p>Expedia measured their own daily backfill under both strategies on the same Iceberg table: MERGE INTO with merge-on-read ran about an hour a day at $0.66, INSERT OVERWRITE ran about three hours a day at $2, a 66% cost reduction running in the opposite direction from the assumption everyone starts with. MERGE also sidesteps the partition-evolution failure above entirely, because it never relies on a partition boundary to decide what to replace, only on the join key.</p><p><strong>Where dbt’s merge and insert_overwrite diverge</strong></p><p>None of this is unique to Iceberg. dbt’s merge incremental strategy needs a unique_key config to do row-level dedup at all; leave it off and merge <a href="https://docs.getdbt.com/docs/build/incremental-strategy">silently degrades to append-only behavior</a>, no error, no dedup, just accumulating duplicates every time the model reruns. Compare that against insert_overwrite, which is documented as an engine-native partition swap for Spark and BigQuery-family adapters and doesn’t need a unique key because it isn’t doing row-level matching in the first place. Neither strategy protects you from the other’s failure mode. Pick based on which invariant your backfill actually needs to hold.</p><h3><strong>Making backfill windows themselves idempotent</strong></h3><p>Partition-level correctness is only half the problem. The orchestration layer that decides which dates to rerun has its own failure modes, and they compound with whatever write strategy sits underneath.</p><p><strong>Per-batch atomicity in dbt’s microbatch strategy</strong></p><p>dbt’s microbatch strategy, available from dbt-core 1.9, was built around per-batch atomicity: each day’s slice is processed and committed independently, so rerunning the same batch twice is safe by construction.</p><p><strong>{</strong>{ config( <br> materialized = ‘incremental’<strong>,</strong> <br> incremental_strategy = ‘microbatch’<strong>,</strong> <br> event_time = ‘session_start’<strong>,</strong> <br> batch_size = ‘day’<strong>,</strong> <br> begin = ‘2024–01–01’<strong>,</strong> <br> partition_by = [‘event_date’]<strong>,</strong> <br> lookback = 1 <br>) <strong>}</strong>} <br> <br><em># Backfill invocation:</em> <br><em># dbt run — event-time-start “2024–09–01” — event-time-end “2024–09–04”</em></p><p>The — event-time-start/ — event-time-end pair is mutually required, one without the other errors out before anything runs. dbt-spark and dbt-bigquery adapters additionally require an explicit partition_by. Skip configuring event_time on an upstream model that should be filtered, and dbt can’t push the batch window down to it, so every single batch triggers a full scan of that upstream table. That’s a performance failure, and on a wide DAG it’s the difference between a backfill finishing overnight and still running at standup.</p><h3><strong>Why does an Airflow backfill sometimes just skip dates?</strong></h3><p>Airflow 3.x ships a structured airflow backfill create command, and its — reprocess-behavior flag controls whether an existing run for a date gets recreated: none never recreates it regardless of state, failed only recreates failed runs, completed recreates both failed and successful ones.</p><p>airflow backfill create — dag-id tutorial \ <br> — from-date 2015–06–01 \ <br> — to-date 2015–06–07 \ <br> — reprocess-behavior failed \ <br> — max-active-runs 3</p><p>That — max-active-runs is scoped to the backfill and is independent of the DAG’s own max_active_runs setting, worth checking explicitly or a backfill can blow past whatever concurrency limit you set for the live schedule. There’s a sharper gotcha underneath the reprocess-behavior logic: if the most recent run for a logical date is still running or queued, Airflow refuses to create a new one no matter which reprocess-behavior you chose. Practitioners running fast backfill loops report DagRuns left in success state from a prior partial attempt causing the scheduler to skip tasks on the next pass rather than erroring, which is why guidance on backfilling without duplicates treats clearing state as mandatory. The standard mitigation is — reset-dagruns or manually clearing DagRuns before the backfill starts, plus pausing the live DAG or pinning max_active_runs=1 so the production schedule and the backfill aren’t racing to write the same partition concurrently.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2bc15a9407c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Airflow’s Scheduler Buckles When Dynamic Task Mapping Fans Out Wide]]></title>
            <link>https://yeedu.medium.com/why-airflows-scheduler-buckles-when-dynamic-task-mapping-fans-out-wide-f40c6ac8f840?source=rss-a9bebe2a1730------2</link>
            <guid isPermaLink="false">https://medium.com/p/f40c6ac8f840</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-airflow]]></category>
            <category><![CDATA[airflow]]></category>
            <dc:creator><![CDATA[Yeedu]]></dc:creator>
            <pubDate>Thu, 10 Sep 2026 17:54:31 GMT</pubDate>
            <atom:updated>2026-09-11T06:44:42.722Z</atom:updated>
            <content:encoded><![CDATA[<p>Dynamic task mapping lets a DAG decide how many tasks it needs at runtime instead of at parse time, but the scheduler underneath still evaluates dependencies and writes state one task instance at a time, so throughput collapses well before you hit any documented limit and the failure shows up as stalled loops and locked rows rather than a clean error. We’ve watched teams adopt dynamic task mapping expecting it to behave like a parallel execution engine: hand it a list, get elastic fan-out. It mostly works, right up until it doesn’t, and the point where it stops working is lower than most people assume.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jaOp8R51wiQf9MacyeXIhw.png" /></figure><h3><strong>AIP-42 gave DAGs runtime elasticity the scheduler wasn’t built for</strong></h3><p>Before <a href="https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-42+Dynamic+Task+Mapping">AIP-42</a> landed in Airflow 2.3.0, the number of tasks in a DAG was fixed the moment the parser read the file. A Python for-loop over a static list was the only way to fan out, and the DAG author had to know the cardinality in advance. AIP-42 rejected that constraint deliberately: the design doc frames map and reduce as “cornerstones to any distributed or undistributed list comprehension paradigm,” and argues that a plain for-loop would mislead users into expecting standard Python semantics Airflow can’t deliver at parse time. To make that work, the schema changed underneath the scheduler. That migration was benchmarked at roughly 38 minutes against a production-scale metadata database, which tells you how invasive it was even before anyone expanded a task.</p><h3><strong>A new primary key and a lazy proxy for XCom values</strong></h3><p>Mapped values arrive downstream as a “lazy proxy” object, an iterator-like wrapper that defers pulling the full XCom payload until something actually needs it. The <a href="https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/dynamic-task-mapping.html">official docs</a> warn explicitly that coercing one of these into a concrete list triggers a logged degradation notice: Coercing mapped lazy proxy return value … which may degrade performance. It’s there because converting eagerly defeats the entire point of deferring memory allocation for a reduce step over thousands of mapped outputs.</p><h3><strong>Binding constants against runtime values with partial() and expand()</strong></h3><p>The API split is simple on the surface. partial() binds arguments that stay constant across every mapped instance; expand() supplies the values that vary:</p><p>@task <br><strong>def</strong> add(x: int, y: int): <br> <strong>return</strong> x + y <br> <br>added_values = add.partial(y=10).expand(x=[1, 2, 3])</p><p>Three tasks get created here, each running add with a different x and the same y. That’s clean at three. It’s still clean at three hundred. The trouble starts once the count climbs into the low thousands, because every one of those task instances is a row the scheduler has to lock, evaluate and update individually.</p><p><strong>The fan-out cliff arrives well before max_map_length is reached</strong></p><p>max_map_length defaults to 1024. Most teams assume anything under that ceiling is safe. It isn’t. The scheduler’s own dependency evaluation degrades faster than the task count grows, and the degradation is nonlinear.</p><p><strong>What happens between 100 and 500 mapped tasks?</strong></p><p>One team benchmarking Airflow 2.10.4 on a 20-core Xeon W-1290, running the stock Docker Compose deployment, measured this directly and <a href="https://github.com/apache/airflow/issues/45991">filed the numbers as issue #45991</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*8GQNWwaETxhBNtfDulVDtQ.png" /></figure><p>A 5x increase in task count produced an 8x increase in runtime, and effective parallelism nearly halved. That’s not saturation, it’s collapse. Five hundred tasks is still well under the 1024 default.</p><p><strong>Why does the mini-scheduler give up under lock contention?</strong></p><p>The bottleneck isn’t disk or network. A <a href="https://github.com/apache/airflow/discussions/46044">related discussion thread</a> traces it to TaskInstance.are_dependencies_met, which is CPU-bound, and to TriggerRuleDep running in quadratic time: it loops over every finished task instance for every schedulable task instance in the run. That quadratic cost compounds fast as the mapped count climbs into the hundreds and thousands. The scheduler loop itself was logged holding steady at 8.49 seconds per cycle, scheduling only the default batch of 16 task instances each time, and the mini-scheduler, the fast path meant to pick up newly-eligible tasks immediately, gets skipped because the relevant rows are locked. This isn’t new territory either: a 2019 patch, <a href="https://github.com/apache/airflow/pull/4751">PR #4751</a>, collapsed the trigger-rule check from one query per task instance to one per DAG run and cut inter-task scheduler delay by about 20% on large DAGs. Dynamic task mapping just reintroduced the same class of problem at a scale nobody was testing against in 2019.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*k__FXY12wcDI7Cjv5PVq2A.png" /></figure><p><strong>Why does raising max_map_length just move the failure downstream?</strong></p><p>Teams that hit the 1024 ceiling often reach for the obvious lever and raise it. That’s how you find the next wall. A <a href="https://github.com/apache/airflow/discussions/28478">discussion thread</a> from a team running PostgreSQL 11.12 on a Bitnami Helm deployment describes pushing map size toward 5,000 and hitting this on the very next DagRun scheduling pass:</p><p>sqlalchemy.exc.OperationalError: (psycopg2.errors.StatementTooComplex) stack depth limit exceeded</p><p>The scheduler was building a single bulk UPDATE statement to set every new task instance to SCHEDULED in one shot, and Postgres refused to parse a statement that complex. Raising Postgres’s own max_stack_depth gets the query through, but then the scheduler goes unresponsive anyway, because the underlying cost was never the query parser, it was evaluating dependencies for thousands of rows serially. Airflow closed this specific gap in <a href="https://github.com/apache/airflow/issues/29959">issue #29959</a> by batching the expansion instead of issuing one bulk statement, but the batching only moves the ceiling, it doesn’t remove it. A separate report on Airflow 2.7.3 with roughly 100,000 mapped tasks found the pain even earlier in the pipeline: <a href="https://github.com/apache/airflow/discussions/36454">DAG file parsing itself</a>, which normally finishes in one to three seconds, started timing out at ten minutes because an internal _get_length() call and a slow join between task_instance and dag_run were both getting invoked far more often than the task count alone would suggest.</p><h3><strong>Diagnosing and containing fan-out before it takes the scheduler down</strong></h3><p>Once you know the shape of the collapse, the response is mostly about limiting exposure rather than tuning your way out of it.</p><h3><strong>What does the failure actually look like in the logs?</strong></h3><p>There’s rarely one smoking gun. Check these three signals in order:</p><ol><li>Scheduler heartbeat gaps that track the fan-out width, growing wider as the mapped task count grows.</li><li>A task marked failed with a message pointing at exceeding max_map_length, which is the clean failure mode and the easy one to catch.</li><li>A stall with no corresponding CPU spike on the database host, which usually means you’re locked rather than slow, and points at the TriggerRuleDep and dependency-evaluation cost rather than I/O.</li></ol><p>If you’re on Postgres and see the stack-depth error above, stop raising limits and start batching.</p><h3><strong>Which two config keys actually matter here?</strong></h3><p>Two settings do most of the real work, and both need explicit values rather than defaults left alone. max_map_length is a genuine airflow.cfg setting:</p><p><strong>[core]</strong> <br>max_map_length = 1024</p><p>max_active_tis_per_dag, by contrast, isn’t a config-file key at all, it’s a parameter you set per task in code:</p><p><em># per-task override to cap concurrent mapped instances</em> <br>@task(max_active_tis_per_dag=32) <br><strong>def</strong> process(item): <br> … ``` <br> <br>`max_map_length` <strong>is</strong> your hard ceiling on how many instances a single `expand()` can create; leave it at 1024 unless you’ve load-tested higher. `max_active_tis_per_dag` throttles how many of those instances run concurrently across all active DagRuns for that task, which matters because letting one mapped task claim every worker slot starves the rest of the DAG. We generally set it low, in the 20 to 40 range, and let the DAG take longer rather than let one task monopolize the pool.</p><blockquote>## Architectural workarounds that hold up in production <br> <br>Config tuning buys you headroom, not elasticity. The pattern that actually holds at scale reshapes the fan-out instead of asking the scheduler to absorb it whole. <br> <br>### Batching the fan-out instead of expanding it in one shot <br> <br>The workaround practitioners converged on in the 5,000-task Postgres thread is “expand of expands”: nest a `TriggerDagRunOperator` so each individual `expand()` call stays comfortably under `max_map_length`, and serialize the batches so the scheduler never has more than one wide fan-out active at once. <br> <br>```python <br>from airflow.operators.trigger_dagrun import TriggerDagRunOperator <br> <br># ‘expand of expands’: cap each batch near max_map_length (1024) <br># and serialize batches instead of one 5000-wide expand() <br>batch_trigger = TriggerDagRunOperator.partial( <br> task_id=”trigger_batch”, <br> trigger_dag_id=”child_dag”, <br> max_active_runs=1, <br>).expand(conf=batches_of_up_to_1024)</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*phjFF7bp3rRcMqut5zDtWA.png" /></figure><p>Each child DagRun stays small enough that dependency evaluation and the bulk scheduling UPDATE complete quickly, and max_active_runs=1 stops the batches themselves from creating a second layer of contention. It’s more DAG to maintain, and it trades some parallelism for predictability. Teams staring down millions of files via mapped tasks raise a related worry in a <a href="https://github.com/apache/airflow/discussions/26172">separate discussion</a>: spawning that many mapped task instances means spawning that many Python interpreters, and the compute overhead of the fan-out itself becomes the constraint before the scheduler’s bookkeeping does.</p><p>Treat expected fan-out as a sizing input the same way you’d size a Spark shuffle, before you write the DAG rather than after the scheduler starts falling behind its own queue:</p><ol><li>Estimate the realistic maximum fan-out width the source task could return, not the typical case.</li><li>If that width exceeds a few hundred, plan a batching layer (TriggerDagRunOperator fan-out or an equivalent split) instead of a single expand() call.</li><li>Set max_active_tis_per_dag deliberately for the mapped task rather than leaving it uncapped.</li><li>Load-test at the expected maximum width, not at 100 tasks, since the collapse only shows up past that point.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f40c6ac8f840" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is TPC Benchmarks? Are They Still Relevant in 2026?]]></title>
            <link>https://medium.com/towards-data-engineering/what-is-tpc-benchmarks-are-they-still-relevant-in-2026-2db23bf9762b?source=rss-a9bebe2a1730------2</link>
            <guid isPermaLink="false">https://medium.com/p/2db23bf9762b</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[tpc-ds]]></category>
            <dc:creator><![CDATA[Yeedu]]></dc:creator>
            <pubDate>Fri, 04 Sep 2026 15:11:30 GMT</pubDate>
            <atom:updated>2026-09-04T15:11:30.759Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zlc2BuCJuIOUZO4d-fCGoQ.png" /></figure><p><em>Why the data industry’s oldest performance standard still matters when every vendor’s whitepaper tells a different story, and where it falls short</em></p><p><strong>Quick answer:</strong> Yes, TPC benchmarks (especially TPC-DS) are still relevant in 2026, but only as the first stage of a four-stage evaluation. They give you a neutral, apples-to-apples comparison across Spark platforms, cloud-native warehouses, and lakehouse engines that vendor whitepapers can’t provide, since every vendor picks the workload that flatters its own product. <br> <br><strong>What TPC-DS can’t tell you: </strong>how a platform handles skewed production data, streaming latency, concurrent query load, or day-to-day operational overhead. Use it to build a shortlist, not to make a final decision.</p><h3>The problem every platform evaluation runs into</h3><p>Say you’re evaluating five Spark platforms for a migration. Each vendor hands you a whitepaper. Each whitepaper tells a different story: different workloads, different cluster configurations, different metrics. Some report query latency. Some report throughput. Some report cost. None of the numbers are comparable.</p><p>You have a budget decision to make, a migration to plan, and no reliable way to tell which platform will actually perform better for your team.</p><p>That problem hasn’t gotten easier in 2026. Five years ago you were comparing a handful of mature platforms. Today the market has fragmented into cloud-native warehouses, lakehouse engines, serverless compute, managed Spark, and proprietary execution layers. Every vendor has a performance story. Almost none of those stories are told using the same measuring stick.</p><p>In 2016 the question that buyers were asking was “which platform is the fastest?”, but in 2026 it’s “which platform gives me the best outcome per dollar spent?” That’s a harder question to answer from a whitepaper, and a much easier one to obscure with a carefully chosen benchmark.</p><p>This is why standardized <a href="https://yeedu.com/benchmarks/your-big-data-infrastructure-costs-should-not-be-a-line-item-you-cant-explain">TPC benchmarks</a> still matter in 2026. They’re not perfect. But without them, the only evidence you have is whatever each vendor decided to put in a marketing deck.</p><h3>Without a standard, every performance claim is marketing</h3><p>For Example you are comparing two platform, A and B. Platform A shows it completes an ETL pipeline in 4 minutes. Platform B processes 10 TB in under an hour. Platform C claims a 40% cost saving over competitors. Three different measurements, three different workloads, three different hardware setups. Comparing them is like comparing race times from three different tracks.</p><p>It gets worse in practice. Every vendor picks the workload that makes its platform look best:</p><ul><li>A columnar engine highlights a query-heavy analytical benchmark</li><li>A streaming platform highlights a latency test</li><li>A cost-competitive platform highlights a large batch job where fixed overhead amortizes well</li></ul><p>Each individual benchmark result is real and accurate. The comparison across them is still useless.</p><p>A standard benchmark only becomes meaningful when everyone follows the same rules: the same queries, run against the same data, using the same measurement methodology. Without that consistency, you’re comparing sales pitches, not platforms.</p><p>The <strong>Transaction Processing Performance Council (TPC)</strong> is an independent, non-profit body that publishes detailed benchmark specifications, including the exact dataset, schema, queries, measurement methodology, and reporting requirements. When two platforms both publish TPC results under the same specification, you can actually compare them. That’s rarer than it sounds.</p><h3>Why TPC-DS is harder than it looks</h3><p>For analytical workloads, meaning Spark platforms, cloud data warehouses, and lakehouse engines, <strong>TPC-DS is the benchmark that matters most</strong>. It models a multi-channel retail enterprise across 24 tables, 99 queries, and scale factors from 1 GB to 100 TB. That description sounds manageable, but in reality it isn’t.</p><p>TPC-DS was specifically designed to expose weaknesses in query planners, execution engines, and storage layers. Engineers who haven’t worked with it often assume it’s easy to tune for, until they actually run it.</p><p><strong>The queries are genuinely complex</strong></p><p>TPC-DS is not a set of straightforward GROUP BY queries. It includes:</p><ul><li><strong>Correlated subqueries</strong>, where a nested query’s predicates reference values from the outer query, forcing the engine to re-execute the inner query for every outer row</li><li><strong>Window functions</strong> like RANK, DENSE_RANK, and ROW_NUMBER, which run in multiple passes across large, partitioned datasets that must be fully sorted before windowing begins</li><li><strong>Multi-dimensional rollups</strong> that aggregate across several grouping sets at once, which many engines materialize inefficiently</li><li><strong>Deep multi-table joins</strong>, sometimes 8 to 10 tables in a single statement, where a suboptimal join order doesn’t just slow the query down, it can kill it entirely</li><li><strong>NULLs in every non-primary-key column</strong>, unlike the simpler TPC-H spec, which surfaces edge cases in aggregation and join logic that clean synthetic data would otherwise hide</li></ul><p>Any one of these is manageable on its own. All of them together, across 99 queries, at multi-terabyte scale, with no cherry-picking which queries to run, is a genuinely hard test.</p><h4>Completeness is a tell</h4><p>Many platforms publish TPC-DS results against a subset of queries, sometimes 80, sometimes fewer, without saying so explicitly. The dropped queries are almost always the correlated subqueries and complex window functions: the ones that break things.</p><p>If a platform completes <strong>all 99 queries</strong>, at scale, without modifications, that’s a meaningful signal. It means the query planner handles the full range of analytical SQL, not just the tractable half. That matters in production, where your real workloads don’t come pre-screened for what the engine can handle.</p><h4>Scale factor separates the architectures</h4><p>Run TPC-DS at 1 TB, then again at 10 TB. The ratio between those two results tells you more about the engine than either number alone. An engine with poor intermediate result materialization shows superlinear cost growth as data scales. A well-architected engine doesn’t. Most vendors publish results for a single scale factor. Ask for all three: 1 TB, 10 TB, and 100 TB.</p><h3>What TPC-DS results actually tell you</h3><h4>A real apples-to-apples comparison</h4><p>When two platforms both publish complete TPC-DS results, meaning the same scale factor, same query set, same reporting methodology, you’re comparing actual performance on the same problem, not marketing claims or a demo built specifically for your evaluation. In enterprise data infrastructure, that kind of comparison almost never happens any other way.</p><h4>Cost per workload, not just speed</h4><p>TPC-DS has always reported price-performance as a co-primary metric, but it was originally defined in terms of the 3-year total cost of ownership of the priced hardware configuration. In a consumption-priced cloud world, that ownership model no longer matches how most teams actually pay.</p><p>A platform that completes TPC-DS at 1 TB in 20 minutes on a $4/hour instance costs $1.33 for that run. One that finishes in 45 minutes on a $6/hour instance costs $4.50. Same benchmark, same result set, 3.4x the cost. That difference compounds at the scale most enterprises actually operate at. Publishing cost-per-run alongside execution time, not execution time alone, is the number that should drive procurement decisions. It’s still surprisingly rare to see vendors report it.</p><h3>Where TPC benchmarks fall short</h3><p>None of the above means you should evaluate a platform on TPC-DS alone. There are real gaps.</p><p><strong>Production data won’t look like benchmark data.</strong> TPC-DS generates data with a controlled distribution. Your production environment has skewed join keys, inconsistently formatted strings, schema drift between pipeline versions, and columns that are 30% NULL because of an upstream bug nobody’s fixed yet. A platform can ace TPC-DS and still fall apart on your actual data.</p><p>That’s a reason to keep evaluating further, not a reason to ignore TPC results.</p><p><strong>It’s a batch benchmark.</strong> TPC-DS has nothing to say about:</p><ul><li><strong>Streaming latency</strong>: how quickly events move from ingestion to query</li><li><strong>Concurrent query throughput</strong>: what happens when 50 analysts hit the platform at once</li><li><strong>Vector search performance</strong>: increasingly central to AI-integrated data pipelines</li><li><strong>Incremental processing</strong>: how the engine handles micro-batch updates to large tables</li></ul><h3>The TPC benchmark family: a quick reference</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*Bcu-GqGrERSCyM7bQy6LWg.png" /></figure><p>For most data engineering teams evaluating Spark platforms or lakehouse engines, <strong>TPC-DS is the benchmark that matters</strong>. TPC-H is simpler and less representative of modern analytical workloads, useful as a sanity check, not as a primary evaluation criterion.</p><h3>How to use TPC benchmarks in a real platform evaluation</h3><p>Most teams treat TPC benchmarks as either the whole answer or as completely irrelevant. Both are wrong. The right approach treats them as one layer in a staged evaluation.</p><h4>Stage 1: Use TPC-DS as a capability filter</h4><p>Before investing real time in any platform, verify it can complete the full TPC-DS suite at a scale factor representative of your data volumes, without query modifications. A platform that can’t clear this bar isn’t ready for your production workloads. This stage is fast and narrows your shortlist quickly.</p><h4>Stage 2: Compare cost-per-workload</h4><p>For platforms that clear the filter, run TPC-DS at your target scale on each one. Record total execution time and total infrastructure cost across compute, storage, and network. Use equivalent instance configurations across platforms. Now you have a normalized cost number you can actually bring into a budget conversation.</p><h4>Stage 3: Run your own workloads</h4><p>TPC-DS shows a platform is capable. Only your own workloads show whether it’s right for you. Pull a representative sample of production queries, your most expensive jobs and your most latency-sensitive pipelines, and run them on your shortlist. This is the only evaluation step that actually predicts production performance.</p><h4>Stage 4: Evaluate what you can’t benchmark</h4><p>Performance and cost are measurable. These aren’t:</p><ul><li>How much engineering effort does cluster management actually require day-to-day?</li><li>How does the platform handle job failures? Is retry automatic or manual?</li><li>What does observability look like? Can you find the problem when something breaks at 2 a.m.?</li><li>How does pricing scale with data volume: usage-based, fixed, or tiered?</li><li>What are the compliance implications of the deployment model?</li></ul><p>A platform that scores slightly lower on TPC-DS but cuts operational overhead significantly can easily be the better choice, especially for teams without dedicated infrastructure engineers.</p><h3>Frequently asked questions</h3><p><strong>Is TPC-DS still relevant for evaluating Spark platforms in 2026?</strong></p><p>Yes. TPC-DS remains the most complete, standardized analytical benchmark available, and it’s the only benchmark most vendors publish under a shared, auditable specification. Its relevance comes from comparability, not novelty: the workload doesn’t need to be new to be useful, it needs to be identical across every platform being compared.</p><p><strong>What is the difference between TPC-H and TPC-DS?</strong></p><p>TPC-H uses 22 simpler, ad-hoc decision-support queries over a supply chain schema. TPC-DS uses 99 queries that include correlated subqueries, multi-pass window functions, multi-dimensional rollups, and NULL-heavy columns, making it a far closer proxy for modern analytical and BI workloads.</p><p><strong>Can TPC-DS results predict how a platform will perform on my production data?</strong></p><p>Not directly. TPC-DS uses synthetic data with controlled distributions, while production data has skew, dirty formatting, and schema drift. Treat a strong TPC-DS result as a capability filter, then validate with your own workloads before making a final decision.</p><p><strong>Why do some vendors only publish TPC-DS results for a subset of queries?</strong></p><p>Usually because the excluded queries, most often correlated subqueries and complex window functions, expose weaknesses in the query planner. A vendor that publishes results for all 99 queries at scale, without modification, is giving you a more complete signal.</p><p><strong>What should I ask a vendor before trusting their published TPC-DS results?</strong></p><p>Ask whether all 99 queries were run or only a subset, what the exact hardware configuration was and whether it matches what customers actually deploy, whether caching was pre-warmed before the timed run, what software version and tuning parameters were used, what the total infrastructure cost was (not just execution time), and whether you can reproduce the results in your own environment on equivalent hardware.</p><p>The bottom line</p><p>TPC benchmarks are over thirty years old. The workloads they model predate cloud computing. Critics who call them outdated aren’t wrong about their age.</p><p>But the alternative isn’t better information. It’s five vendor whitepapers, each optimized to make that vendor look best, with no common basis for comparison. TPC benchmarks exist to create one: a floor of comparability in a market where every other signal is strategically crafted.</p><p>Use them for what they are: a starting filter and the only neutral baseline the industry has. Then go deeper with your own workloads and your own cost model. The platforms that hold up through all of that are the ones worth running in production.</p><p>The original blog was published in: <a href="https://yeedu.com/blog/are-tpc-benchmarks-still-relevant">https://yeedu.com/blog/are-tpc-benchmarks-still-relevant</a></p><p><em>Yeedu helps data engineering teams run and optimize Spark workloads on their own cloud infrastructure, with full visibility into cost and performance at every scale factor. Learn more at </em><a href="https://yeedu.com/"><em>yeedu.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2db23bf9762b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/what-is-tpc-benchmarks-are-they-still-relevant-in-2026-2db23bf9762b">What is TPC Benchmarks? Are They Still Relevant in 2026?</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spark Performance Optimization: 10 Techniques That Actually Move the Needle]]></title>
            <link>https://yeedu.medium.com/spark-performance-optimization-10-techniques-that-actually-move-the-needle-2093b9c86e61?source=rss-a9bebe2a1730------2</link>
            <guid isPermaLink="false">https://medium.com/p/2093b9c86e61</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[cloud-cost-optimization]]></category>
            <category><![CDATA[yeedu]]></category>
            <category><![CDATA[spark-cost-optimization]]></category>
            <dc:creator><![CDATA[Yeedu]]></dc:creator>
            <pubDate>Thu, 14 May 2026 14:01:02 GMT</pubDate>
            <atom:updated>2026-05-14T14:01:02.057Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OAziP2qQBezlCaAT2L6_oA.png" /></figure><p>Most Spark performance advice sounds right but rarely helps in production. This blog is written for engineers who already run Spark at scale and want to focus on optimizations that consistently reduce runtime and cost, not just tweak configurations.</p><h4>Introduction</h4><p>If you work with Spark in production, you already know the pattern. A job slows down, someone increases the cluster size, a few configs are tuned, and things improve slightly. Then the data grows, costs rise, and the same problem returns.</p><p>The issue is not lack of tuning knowledge. It is misplaced effort. Spark performance problems are rarely solved by a single parameter or a bigger cluster. They are solved by understanding how Spark executes work and where time is actually lost.</p><p>This post focuses on ten techniques that repeatedly make a real difference in production systems. No beginner explanations, no exhaustive tuning lists. Just the levers that matter.</p><h4>Start With Execution, Not Settings</h4><p>Spark jobs are slow for predictable reasons. Data moves too much, work is unevenly distributed, or memory is under pressure. These problems show up clearly in execution plans and stage metrics.</p><p>Yet many teams start by tuning configurations before understanding what the job is doing. That approach rarely works.</p><p>Before changing any setting, look at the physical plan, stage boundaries, and task metrics. Performance optimization without diagnosis is guesswork.</p><p><strong>1. Reduce Shuffles by Fixing Query Shape</strong></p><p>Shuffles dominate Spark runtime. They involve disk, network, and coordination across executors. Many shuffles exist simply because of how queries are written.</p><p>Reordering joins, pushing filters earlier, and collapsing unnecessary transformations can reduce shuffle volume dramatically. This usually delivers larger gains than any configuration tweak.</p><p><strong>2. Use Broadcast Joins Selectively</strong></p><p>Broadcast joins remove shuffles, which makes them tempting. Used blindly, they introduce memory pressure and instability.</p><p>Broadcast works best when the dataset is small, predictable, and reused across stages. When size fluctuates, it often becomes a hidden failure mode.</p><p>Broadcast by design, not by habit. Validate memory impact under realistic data sizes.</p><p><strong>3. Partition for Access Patterns, Not Defaults</strong></p><p>Default partitioning works until it does not. If downstream joins and aggregations do not align with partitioning strategy, Spark spends its time reshuffling data.</p><p>Partitioning should reflect how data is consumed, not just how it is written. Over time, access patterns change and partitioning must evolve with them.</p><p><strong>4. Detect Skew Using Metrics, Not Intuition</strong></p><p>Skew creates long tail tasks that stretch job completion time. It is one of the most common Spark performance killers.</p><p>Skew is visible. Uneven task durations, large variance in input sizes, and a handful of tasks dominating stage runtime are clear indicators.</p><p>Trust stage metrics. They tell you where Spark is struggling.</p><p><strong>5. Fix Skew Where It Exists</strong></p><p>Once skewed keys are identified, targeted fixes like key salting or splitting heavy keys can restore parallelism.</p><p>Applying skew handling globally often adds complexity without benefit. Most pipelines suffer from a small number of problematic keys.</p><p><strong>6. Treat Partition Counts as Dynamic</strong></p><p>A single global shuffle partition setting rarely works well for all workloads. Small jobs pay overhead costs, while large jobs still struggle.</p><p>Adjust partition counts at key pipeline boundaries based on data volume and cluster size.</p><p><strong>7. Cache Only With Proven Reuse</strong></p><p>Caching feels like an easy win, but it often creates memory pressure without reducing runtime. Caching helps only when recomputation is expensive and reuse is guaranteed.</p><p>If a dataset is consumed once, caching usually hurts more than it helps.</p><p>8<strong>. Use Spill and GC Metrics as Signals</strong></p><p>Spills and garbage collection are symptoms. Increasing memory may hide the issue temporarily, but the root cause usually lies in execution design.</p><p>When spill or GC dominates runtime, revisit query structure and data layout.</p><p><strong>9. Fix Data Layout Upstream</strong></p><p>File sizes, formats, and layout decisions shape performance across every downstream job. Too many small files increase scheduling overhead. Very large files reduce parallelism.</p><p>Getting data layout right early reduces the need for repeated tuning later.</p><p><strong>10. Simplify Pipelines With Fewer Wide Transformations</strong></p><p>Repeated joins and aggregations compound performance problems. Pipelines that look flexible often pay for that flexibility at runtime.</p><p>Consolidating wide transformations or materializing results strategically can reduce both runtime and cost.</p><h4>Conclusion</h4><p>Spark performance optimization is not about knowing every configuration. It is about knowing where effort pays off. The techniques above focus on execution behavior, data movement, and structural decisions that consistently matter at scale.</p><p>At Yeedu, we focus on helping engineers build this kind of judgment. Not shortcuts, but understanding that lasts as systems and data continue to grow.</p><p><em>Interested in related topics? Click here: </em><a href="https://yeedu.com/resources/blogs"><em>https://yeedu.com/resources/blogs</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2093b9c86e61" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Modern Query Engines Avoid the JVM — And What That Means for Spark]]></title>
            <link>https://yeedu.medium.com/why-modern-query-engines-avoid-the-jvm-and-what-that-means-for-spark-55f063c4cae7?source=rss-a9bebe2a1730------2</link>
            <guid isPermaLink="false">https://medium.com/p/55f063c4cae7</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[jvm]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[yeedu]]></category>
            <dc:creator><![CDATA[Yeedu]]></dc:creator>
            <pubDate>Tue, 05 May 2026 14:01:01 GMT</pubDate>
            <atom:updated>2026-05-05T14:01:01.974Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eN6lIOuLvApY-UD0T8W_dw.png" /></figure><p>A few years ago, most data teams didn’t question the execution layer of their analytics stack. If you were using Apache Spark, you were already operating at scale. The assumption was simple: distributed systems solve performance.</p><p>That assumption is now starting to break.</p><p>Not because Spark is flawed architecturally, but because the industry has quietly shifted its definition of performance. It is no longer just about scaling out. It is about how efficiently each core executes work. And on that axis, a new class of engines has changed expectations entirely.</p><p>Tools like DuckDB, Polars, and DataFusion are not just faster in benchmarks. They feel different. Queries return in seconds instead of minutes. Local workflows rival cluster-scale systems. And the most interesting part is not the algorithms they use, but where they choose to run.</p><p>They avoid the JVM.</p><p>That decision turns out to be more consequential than it first appears.</p><h4><strong>The shift from scale to efficiency</strong></h4><p>For a long time, performance conversations in data engineering revolved around distribution. More nodes meant more throughput. If a job was slow, you scaled the cluster.</p><p>But scaling hides inefficiency. It does not remove it.</p><p>Modern query engines have taken a different approach. Instead of asking how to distribute work better, they ask how to execute the same work more efficiently on a single core. That shift leads directly into hardware-aware design: SIMD instructions, cache locality, and predictable memory access patterns.</p><p>This is where the JVM starts to show its limits.</p><p>The JVM was designed for general-purpose application development. It prioritizes safety, portability, and abstraction. Those are valuable properties, but they come with trade-offs that become visible under analytical workloads.</p><p>Garbage collection introduces pauses. Object-heavy memory layouts reduce cache efficiency. And most importantly, the execution model struggles to fully exploit modern CPU capabilities like vectorized instructions.</p><p>Individually, each limitation looks manageable. Together, they form a structural ceiling.</p><h4><strong>Why native execution changes the equation</strong></h4><p>Engines written in C++ or Rust operate much closer to the hardware. They control memory layout explicitly. They process data in tight, vectorized loops. They treat the CPU not as a black box, but as something to be optimized directly.</p><p>This shows up most clearly in how they process data.</p><p>Instead of row-by-row execution, modern engines operate on columnar batches. Instead of scalar operations, they use SIMD to process multiple values in a single instruction. Instead of relying on garbage collection, they use memory pools and allocators designed for predictable behavior.</p><p>The result is not a marginal improvement. It is often a step change.</p><p>The same query plan, executed with the same logical optimizations, can run several times faster simply because the execution layer is different.</p><p>This is why comparisons like DuckDB vs Spark often feel surprising at first glance. It is not that one system is more sophisticated than the other. It is that one is fundamentally closer to the hardware.</p><h4><strong>Where Spark fits in this new world</strong></h4><p>Spark still solves a problem that these newer engines do not.</p><p>It handles distributed computation at scale. It manages failures, coordinates tasks across clusters, and provides a mature ecosystem for large data pipelines. None of that goes away.</p><p>But the execution layer inside Spark remains tied to the JVM. And that creates a mismatch.</p><p>You end up with a system that is excellent at distributing work, but less efficient at executing it once it reaches a node.</p><p>That mismatch shows up in ways most teams recognize. Jobs that scale linearly in data size but not in cost. Clusters that grow faster than workloads. Performance tuning that feels like diminishing returns.</p><p>At some point, adding more nodes stops being the right answer.</p><h4><strong>The emerging pattern: decoupling execution from orchestration</strong></h4><p>What we are starting to see across the ecosystem is a separation of concerns.</p><p>Distributed systems like Spark continue to handle orchestration: scheduling, fault tolerance, data movement.</p><p>But execution is increasingly being delegated to native engines that can run closer to the hardware.</p><p>Projects like Velox and Apache Arrow reflect this shift. They are not full platforms. They are execution layers designed to be embedded into larger systems.</p><p>This is an important change in architecture. It suggests that the future of data processing is not about replacing Spark, but about rethinking what runs inside it.</p><h4><strong>What this means in practice</strong></h4><p>For most teams, this shift will not look like a migration. It will look like an upgrade in execution efficiency.</p><p>The APIs remain the same. The pipelines remain the same. But the runtime becomes more efficient.</p><p>That efficiency compounds quickly.</p><p>Faster execution reduces resource usage. Lower resource usage reduces cost. And lower cost makes it possible to run more workloads, more frequently, with tighter feedback loops.</p><p>In other words, performance improvements translate directly into operational flexibility.</p><h4><strong>The broader takeaway</strong></h4><p>The move away from the JVM in modern query engines is not ideological. It is pragmatic.</p><p>As hardware evolves, the gap between abstracted execution environments and hardware-aware systems becomes more visible. Analytical workloads, with their heavy reliance on scanning, filtering, and aggregation, sit right at the center of that gap.</p><p>The engines that perform best today are the ones that acknowledge this reality and design accordingly.</p><p>Spark does not become irrelevant in that world. But its role becomes clearer.</p><p>It is no longer just about scaling data. It is about pairing scale with efficient execution.</p><p>And increasingly, that execution does not live on the JVM anymore.</p><p><em>Interested in similar topics? Click here: </em><a href="https://yeedu.com/resources/blogs"><em>https://yeedu.com/resources/blogs</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55f063c4cae7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>