<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aanchal Agrawal on Medium]]></title>
        <description><![CDATA[Stories by Aanchal Agrawal on Medium]]></description>
        <link>https://medium.com/@aanchal.mietcollege?source=rss-ba4ae1911ad0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*p7DnkwFIuv1mozx2</url>
            <title>Stories by Aanchal Agrawal on Medium</title>
            <link>https://medium.com/@aanchal.mietcollege?source=rss-ba4ae1911ad0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 08:04:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aanchal.mietcollege/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why Your Spark Jobs Are Still Slow — And How Storage Partition Join Changes the Economics]]></title>
            <link>https://medium.com/towards-data-engineering/why-your-spark-jobs-are-still-slow-and-how-storage-partition-join-changes-the-economics-efe2f5f0fe88?source=rss-ba4ae1911ad0------2</link>
            <guid isPermaLink="false">https://medium.com/p/efe2f5f0fe88</guid>
            <dc:creator><![CDATA[Aanchal Agrawal]]></dc:creator>
            <pubDate>Sun, 15 Mar 2026 08:39:31 GMT</pubDate>
            <atom:updated>2026-03-15T08:39:31.164Z</atom:updated>
            <content:encoded><![CDATA[<p>Most Spark performance conversations follow a familiar script. Someone notices a slow job, someone else suggests increasing spark.sql.shuffle.partitions, a third person recommends adding more executors, and the meeting ends with marginal gains and a higher cloud bill.</p><p>I want to offer a different conversation — one that starts not with tuning parameters, but with asking whether the shuffle needs to exist at all.</p><p>Storage Partition Join (SPJ), available from Spark 3.3 and significantly matured in Spark 3.4, answers that question directly. When both sides of a join are physically co-located on disk using the same bucketing scheme, Spark can prove at plan time that no data movement is required. The Exchange node — the stage boundary that serialises your data, moves it across the network, and writes it to local disk — simply disappears from the physical plan.</p><p>This is not a configuration trick. It is an architectural approach, and it requires you to think about your storage layout the same way a database engineer thinks about index design: as a first-class decision that shapes every query running against that data.</p><h3>The Real Cost of a Shuffle</h3><p>To appreciate what SPJ eliminates, it helps to be precise about what a shuffle actually costs.</p><p>When Spark executes a Sort Merge Join — its default strategy for large-to-large joins — the execution path involves two full network round-trips before a single row is joined:</p><pre>== Physical Plan (Sort Merge Join) ==<br>* SortMergeJoin<br>   :- * Sort<br>   :  +- Exchange hashpartitioning(id, 200)    ← shuffle #1: table A<br>   +- * Sort<br>      +- Exchange hashpartitioning(id, 200)    ← shuffle #2: table B</pre><p>Each Exchange node is a hard stage boundary. Every row is serialised, hashed onto a target partition, transferred over the network, and written to the receiving executor&#39;s local disk before the sort and join can begin. If any executor fails after a shuffle write, the entire upstream stage re-executes from scratch.</p><p>There is also a second shuffle most teams overlook: when writing results back to a partitioned Iceberg table, a further Exchange re-shuffles the output to match the target&#39;s partition layout. A typical large join in production therefore involves three full network round-trips — two on the read side, one on the write side — before a single row reaches its destination file.</p><p>Shuffle cost is not constant. It scales linearly with data volume. More machines reduce per-node load but do not eliminate the transfer itself. This is the ceiling you hit when you tune your way into a corner.</p><h3>The Join Strategy Ladder</h3><p>Understanding where SPJ sits requires a clear picture of the full progression of join strategies. Each step removes one layer of overhead.</p><p><strong>Sort Merge Join</strong> is the default. Two Exchanges, two Sorts. It handles arbitrary data sizes and is resilient under memory pressure, which is why Spark defaults to it. But it pays the full shuffle tax on every execution.</p><p><strong>Shuffle Hash Join</strong> removes the Sort steps by hashing rather than sorting the join keys. It is faster when data fits comfortably in executor memory, but the two Exchange nodes remain. You have removed the CPU-intensive sorting, not the network transfer. Enable it with:</p><pre>spark.conf.set(&quot;spark.sql.join.preferSortMergeJoin&quot;, &quot;false&quot;)</pre><p><strong>Broadcast Join</strong> eliminates shuffles entirely by replicating the smaller table to every executor. It is the fastest possible join strategy — but only when one side is small enough to fit in memory. The default broadcast threshold is 10MB (tunable via spark.sql.autoBroadcastJoinThreshold). For large-to-large joins, it is simply not an option.</p><p><strong>Storage Partition Join</strong> is what happens when you remove both Exchange nodes for large-to-large joins by encoding the join access pattern into your physical storage layout. It is the only strategy that eliminates shuffle overhead at scale without requiring one side to fit in memory.</p><h3>The Core Insight: Trust the Storage Layout</h3><p>The mechanism behind SPJ is straightforward once you see it.</p><p>When you write PARTITIONED BY (bucket(N, id)) in Iceberg, a deterministic hash is applied to every id value and the result taken modulo N. The row is physically stored in the directory corresponding to that bucket number. The hash is stable — the same id value always maps to the same bucket, regardless of when or how the data was written.</p><p>If table_a and table_b are both partitioned by bucket(256, id), then every row with id = 12345 from table_a lives in the same bucket as every row with id = 12345 from table_b. Spark can assign bucket N from both tables to the same executor and perform the join entirely in local memory — zero network transfer.</p><p>The mechanism that enables this is Iceberg’s implementation of Spark’s SupportsReportPartitioning V2 interface. Iceberg advertises its physical layout to Spark&#39;s query planner, which trusts this guarantee and removes the Exchange nodes:</p><pre>== Physical Plan (with SPJ active) ==<br>* SortMergeJoin<br>   :- BatchScan table_a    ← no Exchange, no Sort<br>   +- BatchScan table_b    ← no Exchange, no Sort</pre><p>The plan reads cleanly. Data flows from storage directly into the join operator.</p><h3>Designing for SPJ: Schema as Architecture</h3><p>SPJ is evaluated at plan time against your physical storage schema. The requirements are not negotiable:</p><ul><li>Both tables must be on Iceberg 1.2.0+ and registered in a V2 catalog</li><li>Both sides of the join must use the <strong>identical</strong> bucket function and bucket count</li><li>The join condition must include the partition columns (Spark 3.x), or you must enable subset key support (Spark 4.0+)</li></ul><p>A practical schema pattern for production pipelines:</p><pre>-- Orders table<br>CREATE TABLE prod.db.orders (<br>    order_id    BIGINT,<br>    customer_id BIGINT,<br>    amount      DECIMAL(10,2),<br>    region      STRING,<br>    event_date  DATE<br>)<br>USING iceberg<br>PARTITIONED BY (event_date, bucket(256, customer_id));</pre><pre>-- Customers table — bucket layout must mirror orders exactly<br>CREATE TABLE prod.db.customers (<br>    customer_id BIGINT,<br>    name        STRING,<br>    tier        STRING,<br>    region      STRING<br>)<br>USING iceberg<br>PARTITIONED BY (event_date, bucket(256, customer_id));</pre><p>The composite partitioning pattern serves two purposes. The identity partition (event_date) enables partition pruning — Spark reads only the relevant date slices. The bucket partition (bucket(256, customer_id)) enables SPJ — shuffle elimination within each date slice. These two optimisations are independent and compose cleanly.</p><p><strong>Bucket count selection</strong> matters and has no universal answer. Too few buckets produces large, poorly parallelised files. Too many creates a small-files problem that penalises your object store and metadata layer. A reasonable starting point for most production tables is between 128 and 512, calibrated against your typical partition sizes and executor count.</p><p>The non-negotiable constraint: bucket counts across co-joined tables must match. Treat this as a versioned architectural decision. A mismatch silently falls back to full shuffle — no error, no warning, just the shuffle tax restored.</p><h3>Configuration: What Each Setting Actually Does</h3><p>The full production configuration, with the reasoning behind each flag:</p><pre># Tell Spark to use V2 source partitioning information<br>spark.conf.set(&quot;spark.sql.sources.v2.bucketing.enabled&quot;, &quot;true&quot;)<br># Tell Iceberg to report its physical layout to Spark&#39;s planner<br>spark.conf.set(&quot;spark.sql.iceberg.planning.preserve-data-grouping&quot;, &quot;true&quot;)<br># Handle cases where partition value sets differ between tables<br># e.g. table_a has a partition for region=EU but table_b does not<br>spark.conf.set(&quot;spark.sql.sources.v2.bucketing.pushPartValues.enabled&quot;, &quot;true&quot;)<br># Allow SPJ on composite partition keys where the join key<br># does not cover every partition column<br>spark.conf.set(&quot;spark.sql.requireAllClusterKeysForCoPartition&quot;, &quot;false&quot;)<br># Skew handling: replicate smaller-side partitions to match<br># large skewed partitions on the other side<br>spark.conf.set(&quot;spark.sql.sources.v2.bucketing.partiallyClusteredDistribution.enabled&quot;, &quot;true&quot;)<br># Eliminate the write-side Exchange when writing to a bucketed target<br>spark.conf.set(&quot;spark.sql.iceberg.distribution-mode&quot;, &quot;none&quot;)<br># Also remove Sort nodes from the plan<br>spark.conf.set(&quot;spark.sql.join.preferSortMergeJoin&quot;, &quot;false&quot;)</pre><p>Two settings deserve particular attention.</p><p>partiallyClusteredDistribution addresses data skew on the join key — a common production reality. Without it, a single oversized bucket becomes a straggler executor that dictates your overall job duration. With it, Spark identifies the larger table via statistics and replicates the smaller side&#39;s partitions to match. This should always be enabled in production.</p><p>distribution-mode = none eliminates the write-side shuffle. A frequent mistake I see in production: teams enable the read-side SPJ configs, verify the Exchange nodes are gone before the join, and declare success. Their job still has a shuffle — it is on the write side. Without explicitly setting distribution-mode = none, Iceberg adds an Exchange after the join to correctly bucket the output for the target table. You need to audit your full physical plan, not just the pre-join section.</p><h3>MERGE: Where SPJ Delivers Maximum Value</h3><p>Benchmark results on SELECT queries establish the theoretical case. The highest practical value in production comes from a different operation: MERGE.</p><p>A MERGE statement joins a target table against a staging table to identify matching rows, then applies updates and inserts. The join inside a MERGE is structurally identical to any other join — and without SPJ, it shuffles both sides. For daily incremental pipelines, the target table is typically your largest, most expensive table. Shuffling it every day is the single biggest avoidable cost in many data platforms.</p><p>Industry benchmarks have shown that SPJ-enabled joins on large Iceberg tables can reduce both job duration and compute cost by 45–70% compared to standard Sort Merge Join. Expedia Group’s engineering team, for instance, reported a roughly 45% cost reduction on a join between a 2TB and 50GB table — down from 11 minutes at $12.20 to 6 minutes at $6.70 — with SPJ enabled on AWS EMR with Spark 3.4 and Iceberg 1.5.</p><p>For MERGE specifically, the structural requirement is that the staging table mirrors the target’s bucket layout exactly, and that all partition columns appear in the ON clause:</p><pre>CREATE TABLE prod.db.events (<br>    event_id   BIGINT,<br>    user_id    BIGINT,<br>    payload    STRING,<br>    event_date DATE<br>)<br>USING iceberg<br>PARTITIONED BY (event_date, bucket(512, event_id));</pre><pre>-- Staging table: bucket layout must be identical<br>CREATE TABLE prod.db.events_stage (<br>    event_id   BIGINT,<br>    user_id    BIGINT,<br>    payload    STRING,<br>    event_date DATE<br>)<br>USING iceberg<br>PARTITIONED BY (event_date, bucket(512, event_id));</pre><pre>spark.sql(&quot;&quot;&quot;<br>    MERGE INTO prod.db.events AS tgt<br>    USING prod.db.events_stage AS stg<br>    ON  tgt.event_id   = stg.event_id<br>    AND tgt.event_date = stg.event_date   -- include identity partition columns<br>    WHEN MATCHED THEN<br>        UPDATE SET tgt.payload = stg.payload<br>    WHEN NOT MATCHED THEN<br>        INSERT *<br>&quot;&quot;&quot;)</pre><p>Including event_date in the ON clause enables partition pruning — Spark reads only the relevant date slices from the target rather than the entire table. Without it, you get the SPJ shuffle elimination but lose the partition filter, and the scan cost climbs back.</p><p>Without SPJ, every daily MERGE shuffles your entire target table. With SPJ and a properly designed staging table, it shuffles nothing.</p><h3>Layering in Write Fanout</h3><p>Once you have eliminated the Exchange nodes from both the read and write sides, there is one further optimisation available on the write path.</p><p>By default, Iceberg sets write.spark.fanout.enabled = false, which forces a local sort of output data before writing. This ensures rows destined for the same bucket are written sequentially, keeping output files well-clustered. The sort is a sensible default when output order is unpredictable.</p><p>When SPJ is active and distribution-mode = none is set, the data arriving at the write stage is already ordered correctly by bucket assignment. The local sort is redundant. You can eliminate it with:</p><pre>ALTER TABLE prod.db.events<br>SET TBLPROPERTIES (&#39;write.spark.fanout.enabled&#39; = &#39;true&#39;);</pre><p>The fanout writer holds multiple output files open simultaneously, which increases executor memory usage. The trade-off is worthwhile in most SPJ pipelines — you eliminate the last remaining Sort from the physical plan, and the memory overhead is modest compared to the shuffle cost you have already removed.</p><h3>When SPJ Applies — and When It Doesn’t</h3><p><strong>SPJ delivers maximum value when:</strong></p><ul><li>Both sides of the join are large enough that broadcast is not viable</li><li>The join runs repeatedly on a stable key — the schema investment compounds over time</li><li>You are running MERGE or upsert operations against large target tables</li><li>You are on cloud infrastructure where network transfer has a direct and visible cost</li></ul><p><strong>SPJ is not applicable when:</strong></p><ul><li>One side is small enough for AQE to broadcast at runtime — broadcast is still faster and requires no schema changes</li><li>The join is a non-equi-join (range conditions, BETWEEN, &gt;) — SPJ only works on equality conditions</li><li>It is a one-time or ad-hoc query — the schema overhead is not justified</li></ul><p><strong>SPJ can actively hurt if:</strong></p><ul><li>You have significant bucket skew and partiallyClusteredDistribution is not enabled</li><li>Bucket counts differ between tables — SPJ silently falls back to full shuffle, no warning</li><li>You re-bucket one table for performance without updating the join partner</li></ul><p>On AQE compatibility: SPJ and Adaptive Query Execution are complementary, not competing. If AQE determines at runtime that one side is small enough to broadcast, it will do so regardless of SPJ configuration — and that is the correct behaviour. Keep AQE enabled in production. Use spark.sql.autoBroadcastJoinThreshold = -1 only when verifying SPJ activation during plan testing.</p><h3>What Goes Wrong in Production</h3><p><strong>Silent shuffle fallback after schema changes.</strong> SPJ is evaluated at plan time against the current physical schema. Any change to bucket count or partition function — even on one table — can silently revert to a full Sort Merge Join. Wire plan verification into your CI/CD pipeline:</p><pre>def assert_no_pre_join_shuffle(df):<br>    &quot;&quot;&quot;Fail the pipeline if Exchange nodes appear before the join.&quot;&quot;&quot;<br>    plan = df._jdf.queryExecution().explainString(&quot;formatted&quot;)<br>    exchange_idx = plan.find(&quot;Exchange&quot;)<br>    join_idx = plan.find(&quot;SortMergeJoin&quot;)<br>    assert exchange_idx == -1 or exchange_idx &gt; join_idx, \<br>        &quot;Pre-join shuffle detected — SPJ not active. Check partition schemas.&quot;</pre><p><strong>Bucket count drift.</strong> As tables grow, teams often re-bucket for performance. If one table’s bucket count changes without the join partner being updated, SPJ stops working. Treat bucket count as a versioned schema decision, not an operational tuning parameter.</p><p><strong>Testing against small datasets with AQE enabled.</strong> AQE will automatically broadcast a small test table, and SPJ will never activate. Validate against representative data volumes, or temporarily disable broadcast during plan verification with spark.sql.autoBroadcastJoinThreshold = -1.</p><p><strong>Forgetting the write-side Exchange.</strong> The most common mistake I see. Engineers verify the pre-join plan looks clean and stop there. Run explain(&quot;FORMATTED&quot;) on the full query including the write target and check for Exchange nodes after the join operator as well as before.</p><h3>Verifying SPJ Is Active</h3><p>Before shipping any SPJ-optimised pipeline to production, verify the physical plan explicitly:</p><pre>df.explain(&quot;FORMATTED&quot;)</pre><p>What you want to see:</p><pre>✅  BatchScan → SortMergeJoin          (no Exchange before the join)<br>✅  BatchScan → HashJoin               (no Exchange, no Sort — full optimisation)</pre><p>What signals a problem:</p><pre>❌  Exchange → Sort → SortMergeJoin   (SPJ has fallen back to shuffle)<br>❌  SortMergeJoin → Exchange          (write-side shuffle still present)</pre><p>Do not rely on job duration alone as a proxy for SPJ activation. A job that runs faster than before may still be shuffling — it might simply have better parallelism. The physical plan is the ground truth.</p><h3>Spark 4.0 and What Changes</h3><p>In Spark 3.x, SPJ requires the join condition to cover every partition column — a constraint that is often too restrictive for real-world composite partition schemes. Spark 4.0 relaxes this:</p><pre>spark.conf.set(<br>    &quot;spark.sql.sources.v2.bucketing.allowJoinKeysSubsetOfPartitionKeys.enabled&quot;,<br>    &quot;true&quot;<br>)</pre><p>This makes SPJ viable for a broader range of queries without requiring full partition column coverage in the join condition. If you are evaluating an upgrade path, this is one of the more practically significant changes for data platform teams.</p><h3>The Architectural Framing</h3><p>SPJ is not a tuning knob. It is a structural decision that encodes your join access patterns directly into your storage layout.</p><p>The upfront cost is real: you must design your schemas around co-location, coordinate bucket counts across related tables, maintain that coordination as schemas evolve, and wire plan validation into your deployment pipeline. That is genuine engineering investment.</p><p>The return is also real. For teams running large-scale recurring batch workloads — particularly daily MERGE operations against large Iceberg tables — eliminating the shuffle restructures the economics of the entire pipeline, not just individual queries.</p><p>The broader principle is worth holding onto. The progression from Sort Merge Join to Shuffle Hash Join to SPJ is a useful model for performance optimisation generally: at each step, ask what expensive operation you can prove is unnecessary given the actual properties of your data. Sorting? Removable, if you can guarantee order upstream. Shuffling? Removable, if you can guarantee co-location at write time.</p><p>The most powerful optimisation is always the one that proves work does not need to be done at all.</p><h3>Quick Reference</h3><p><strong>Requirements checklist:</strong></p><ul><li>Iceberg 1.2.0+ on both tables, registered in a V2 catalog</li><li>Identical bucket(N, key) on both sides of every join</li><li>Join condition covers all partition columns (Spark 3.x) or enable allowJoinKeysSubsetOfPartitionKeys (Spark 4.0+)</li></ul><p><strong>Full production config:</strong></p><pre>spark.conf.set(&quot;spark.sql.sources.v2.bucketing.enabled&quot;, &quot;true&quot;)<br>spark.conf.set(&quot;spark.sql.iceberg.planning.preserve-data-grouping&quot;, &quot;true&quot;)<br>spark.conf.set(&quot;spark.sql.sources.v2.bucketing.pushPartValues.enabled&quot;, &quot;true&quot;)<br>spark.conf.set(&quot;spark.sql.requireAllClusterKeysForCoPartition&quot;, &quot;false&quot;)<br>spark.conf.set(&quot;spark.sql.sources.v2.bucketing.partiallyClusteredDistribution.enabled&quot;, &quot;true&quot;)<br>spark.conf.set(&quot;spark.sql.iceberg.distribution-mode&quot;, &quot;none&quot;)<br>spark.conf.set(&quot;spark.sql.join.preferSortMergeJoin&quot;, &quot;false&quot;)</pre><p><strong>Plan verification:</strong></p><pre>df.explain(&quot;FORMATTED&quot;)<br># ✅ Good: BatchScan → (Sort →) SortMergeJoin, no Exchange before the join<br># ❌ Bad:  Exchange → Sort → SortMergeJoin — SPJ has fallen back to shuffle<br># ❌ Bad:  SortMergeJoin → Exchange — write-side shuffle still present</pre><p><strong>Red flags in production:</strong></p><ul><li>Exchange nodes before the join → SPJ not active, check partition schemas</li><li>Bucket counts differ between tables → SPJ impossible, will silently shuffle</li><li>Exchange nodes after the join → set distribution-mode = none</li><li>Job skew on bucket key → enable partiallyClusteredDistribution</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=efe2f5f0fe88" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/why-your-spark-jobs-are-still-slow-and-how-storage-partition-join-changes-the-economics-efe2f5f0fe88">Why Your Spark Jobs Are Still Slow — And How Storage Partition Join Changes the Economics</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Definitive Guide to Apache Spark Performance Tuning: From Slow to Blazing Fast]]></title>
            <link>https://medium.com/towards-data-engineering/the-definitive-guide-to-apache-spark-performance-tuning-from-slow-to-blazing-fast-3f690f6983ac?source=rss-ba4ae1911ad0------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f690f6983ac</guid>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[spark-performance]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Aanchal Agrawal]]></dc:creator>
            <pubDate>Sat, 28 Feb 2026 15:51:51 GMT</pubDate>
            <atom:updated>2026-03-01T04:14:52.589Z</atom:updated>
            <content:encoded><![CDATA[<p>Apache Spark is the backbone of large-scale data processing. But here’s what nobody tells you when you first spin up a cluster: <strong>getting Spark to run is easy. Getting it to run well is an art form.</strong></p><p>I’ve seen petabyte-scale pipelines brought to their knees by a single misconfigured shuffle partition count. I’ve watched teams throw money at bigger clusters when the real problem was a 200-line PySpark script that triggered an unnecessary full table scan. This guide distills everything you need to know about Spark performance tuning.</p><h3>Table of Contents</h3><ol><li>Understanding the Spark Execution Model (Before You Tune Anything)</li><li>Reading Query Plans — Your X-Ray Vision</li><li>Mastering DAGs — Seeing the Full Picture</li><li>Memory Management — The Most Misunderstood Topic in Spark</li><li>Executor Tuning — Right-Sizing Your Cluster</li><li>The Shuffle Problem — Spark’s Achilles Heel</li><li>Data Partitioning — The Foundation of Parallelism</li><li>Bucketing — Eliminating Shuffles at the Source</li><li>Caching — When to Store, When to Recompute</li><li>Data Skew — The Silent Performance Killer</li><li>Salting — The Surgical Fix for Skew</li><li>Adaptive Query Execution (AQE) — Let Spark Optimize Itself</li><li>Dynamic Partition Pruning — Reading Only What You Need</li><li>The Senior Architect’s Mental Model — Putting It All Together</li></ol><h3>1. Understanding the Spark Execution Model</h3><p>Before you tune a single configuration, you must deeply internalize how Spark actually executes your code. This is the single most important prerequisite to everything else in this guide.</p><p>When you write a DataFrame transformation in Spark, <strong>nothing happens immediately</strong>. Spark builds a logical plan — a tree of operations — and waits. This is <em>lazy evaluation</em>, and it’s both a blessing and a source of endless confusion for beginners.</p><p>The moment you call an <em>action</em> (.count(), .show(), .write()), Spark kicks off the Catalyst optimizer, which takes your logical plan through several stages:</p><p><strong>Logical Plan → Analyzed Logical Plan → Optimized Logical Plan → Physical Plan → Executed Plan</strong></p><p>The Catalyst optimizer applies hundreds of rules to rewrite your query. It pushes filters as early as possible, eliminates redundant projections, and chooses join strategies. Understanding this pipeline is why reading query plans (our next section) is so critical — it shows you what Spark <em>actually</em> decided to do with your code, not what you <em>think</em> you told it to do.</p><p>At runtime, Spark breaks your job into <strong>Jobs → Stages → Tasks</strong>:</p><ul><li>A <strong>Job</strong> is triggered by a single action.</li><li><strong>Stages</strong> are groups of tasks that can be executed in parallel without shuffling data across the network. A stage boundary is always a shuffle.</li><li><strong>Tasks</strong> are the atomic units of work, one per partition. Each task processes a single partition of data on a single executor core.</li></ul><p>This model has a profound implication: <strong>if you reduce the number of stages (by reducing shuffles), you dramatically reduce your job’s latency.</strong> This is the mental model you should carry into every optimization decision.</p><h3>2. Reading Query Plans — Your X-Ray Vision</h3><p>The explain() method is your single most powerful diagnostic tool. Most engineers never use it. That&#39;s like being a surgeon who refuses to look at X-rays.</p><pre>df.explain(True)  # True gives you ALL four plan levels</pre><p>The output shows you four plans:</p><ul><li><strong>Parsed Logical Plan</strong> — your code, translated to Spark’s internal representation</li><li><strong>Analyzed Logical Plan</strong> — after resolving column names and types</li><li><strong>Optimized Logical Plan</strong> — after Catalyst applies optimization rules</li><li><strong>Physical Plan</strong> — the actual execution strategy</li></ul><p><strong>What to look for in the Physical Plan:</strong></p><p><strong>Exchange</strong> nodes are your enemy. Every Exchange is a shuffle — data moving across the network. Each one is expensive. Count them and try to reduce them.</p><pre>== Physical Plan ==<br>AdaptiveSparkPlan (1)<br>+- == Final Plan ==<br>   Sort (2)<br>   +- Exchange (3)          ← SHUFFLE: network I/O happening here<br>      +- HashAggregate (4)<br>         +- Exchange (5)    ← SHUFFLE: another one!<br>            +- Scan parquet (6)</pre><p><strong>BroadcastHashJoin</strong> is your friend. It means Spark has decided to broadcast a small table to all executors and avoid a shuffle join entirely. If you see SortMergeJoin instead, that&#39;s a shuffle join — expensive.</p><p><strong>Filter</strong> appearing close to <strong>Scan</strong> is good — it means predicate pushdown is working. Filter appearing late in the plan means Spark is reading more data than necessary.</p><p>The golden rule: <strong>the best query plan is the shortest one with the fewest Exchange nodes.</strong></p><h3>3. Mastering DAGs — Seeing the Full Picture</h3><p>The Spark UI’s DAG (Directed Acyclic Graph) visualization gives you the operational view of your job. Where the query plan tells you <em>what</em> will happen, the DAG shows you <em>what is happening and what already happened</em>.</p><p>Navigate to the Spark UI → Jobs → click any job → you’ll see a stage-by-stage DAG.</p><p><strong>Reading the DAG like a pro:</strong></p><p>Each node in the DAG represents an RDD or DataFrame transformation. The edges show data flow. The <strong>stage boundaries</strong> (thick lines) represent shuffles — these are the most costly operations in your job.</p><p>Key things to notice:</p><ul><li><strong>How many stages?</strong> More stages = more shuffles = more latency.</li><li><strong>Are stages running sequentially when they could be parallel?</strong> Look for dependencies.</li><li><strong>Stage duration distribution</strong> — is one stage taking 95% of the time? That’s your bottleneck.</li><li><strong>Task duration distribution within a stage</strong> — if one task takes 10x longer than others, you have data skew (covered later).</li></ul><p>The DAG is especially revealing when combined with the Stage detail view, where you can see the distribution of task durations as a histogram. A good job shows a tight distribution. A skewed job shows a long tail — a few tasks taking 10x, 100x, or 1000x longer than the rest.</p><h3>4. Memory Management — The Most Misunderstood Topic in Spark</h3><p>Spark’s memory model looks simple on the surface. Underneath, it’s a carefully designed system that most engineers misunderstand, leading to either OOM errors or dramatically under-utilized memory.</p><h3>The Unified Memory Model</h3><p>Within each executor’s JVM heap, memory is divided into three regions:</p><p><strong>Reserved Memory (~300MB):</strong> A small buffer Spark keeps off-limits to prevent catastrophic failure. Don’t touch it. Don’t think about it.</p><p><strong>Spark Memory (controlled by </strong><strong>spark.memory.fraction, default 0.6):</strong> This is the shared pool for both execution and storage. The key insight is that these two pools are <em>unified</em> and <em>dynamic</em> — they borrow from each other as needed.</p><ul><li><strong>Execution Memory</strong> is used for shuffles, joins, sorts, and aggregations. It can evict cached data from storage memory if it needs more space.</li><li><strong>Storage Memory</strong> (controlled by spark.memory.storageFraction, default 0.5 of Spark Memory) is used for caching DataFrames and broadcast variables. It <em>cannot</em> evict execution memory.</li></ul><p><strong>User Memory (remaining 40%):</strong> Used for your custom data structures — think large Python dictionaries, custom accumulators, UDF state. If you have complex UDFs or use .foreachPartition() with large in-memory state, this matters.</p><h3>The Practical Breakdown (Example: 4GB Executor)</h3><pre>4096 MB total executor heap<br>├── 300 MB  Reserved memory (hands off)<br>├── 2277 MB Spark Memory (60% of remaining 3796 MB)<br>│   ├── 1138 MB Storage Memory (50% of Spark Memory, for cache)<br>│   └── 1138 MB Execution Memory (50% of Spark Memory, for shuffles/joins)<br>└── 1519 MB User Memory (40% of remaining 3796 MB)</pre><h3>What This Means for You</h3><p>If your job is doing heavy shuffles and joins, your execution memory is under pressure. Cached DataFrames will be evicted to make room. This is <em>by design</em> — Spark prioritizes getting your current computation done over keeping cache warm. The implication: <strong>don’t assume your cache persists across complex operations</strong>.</p><p>If your job is I/O-bound and needs to cache large datasets, you can increase spark.memory.storageFraction. But be careful — this shrinks execution memory, which can cause shuffle spills to disk.</p><p><strong>GC Pressure:</strong> High garbage collection is one of the most common Spark performance problems. Watch the GC time in your Stage metrics. If GC is consuming more than 10% of task time, you have a memory problem. Switch to Kryo serialization, reduce object creation in UDFs, and consider using primitive arrays instead of boxed types.</p><pre>spark.conf.set(&quot;spark.serializer&quot;, &quot;org.apache.spark.serializer.KryoSerializer&quot;)</pre><h3>5. Executor Tuning — Right-Sizing Your Cluster</h3><p>There is no universally correct executor configuration. There is only the right configuration for your workload and your cluster. Here’s how to think about it.</p><h3>The Three Levers</h3><p>You have three configuration knobs: number of executors, cores per executor, and memory per executor. They interact in non-obvious ways.</p><p><strong>Cores per executor</strong> controls task parallelism within a single JVM. Higher cores = more tasks running concurrently = more memory pressure and more GC competition. The sweet spot is typically <strong>4–5 cores per executor</strong>. Beyond 5, HDFS client throughput degrades (HDFS limits concurrent threads per client to ~5 by default), and GC pressure becomes punishing.</p><p><strong>Memory per executor</strong> should be sized to give each task ~2–4 GB of working space. With 5 cores, you want roughly 10–20 GB per executor, plus overhead.</p><p><strong>Number of executors</strong> = (Total cluster cores) / (cores per executor). Leave some cores for the OS and YARN/Kubernetes overhead (typically 5–10% of total).</p><h3>Memory Overhead — The Oft-Forgotten Piece</h3><p>Beyond heap memory, each executor also needs off-heap memory for JVM overhead, network buffers, and Python worker processes (for PySpark). This is configured separately:</p><pre>spark.conf.set(&quot;spark.executor.memoryOverhead&quot;, &quot;2g&quot;)  # 10-40% of executor memory</pre><p>For PySpark jobs with heavy UDF usage, this needs to be higher — Python workers run in separate processes and their memory doesn’t count against the JVM heap but does count against your total allocation.</p><h3>The Practical Formula</h3><pre>Total executor memory requested = spark.executor.memory + spark.executor.memoryOverhead</pre><pre>Example for a 32-core, 128GB node:<br>- Reserve 10% for OS: ~13GB, ~3 cores<br>- Usable: ~115GB, ~29 cores  <br>- Executors: 29 / 5 cores = 5 executors per node<br>- Memory per executor: 115GB / 5 = ~23GB<br>- Set spark.executor.memory = 18g, spark.executor.memoryOverhead = 4g</pre><h3>6. The Shuffle Problem — Spark’s Achilles Heel</h3><p>Shuffles are the most expensive operation in Spark. When a shuffle occurs, Spark must:</p><ol><li>Write shuffle data to local disk (the shuffle write phase)</li><li>Transfer data across the network to the target executors</li><li>Read and sort the received data (the shuffle read phase)</li></ol><p>This involves disk I/O, network I/O, and CPU — all three bottlenecks simultaneously.</p><h3>What Triggers a Shuffle?</h3><p>Operations that require grouping or joining data by key trigger shuffles: groupBy, join, distinct, orderBy, repartition. Understanding this is key to minimizing shuffles through intelligent query design.</p><h3>Shuffle Partitions — The Critical Configuration</h3><p>The most commonly misconfigured Spark parameter is spark.sql.shuffle.partitions, which defaults to <strong>200</strong>. This single number determines how many parallel tasks will process your shuffled data.</p><p>With 200 partitions and a small dataset, you’re creating 200 tiny tasks — massive overhead. With 200 partitions and a 10TB dataset, you’re creating tasks so large they spill to disk.</p><p><strong>How to set it correctly:</strong></p><p>The general rule is to target partition sizes between 100MB and 200MB.</p><pre># Calculate: total shuffled data size / target partition size<br># Example: 1TB of shuffled data / 128MB target = ~8000 partitions<br>spark.conf.set(&quot;spark.sql.shuffle.partitions&quot;, &quot;8000&quot;)</pre><p>But this is exactly the problem that AQE (covered in Section 12) was designed to solve — you often don’t know the shuffled data size in advance.</p><h3>Minimizing Shuffles by Design</h3><p>The best shuffle is the one you never trigger. Several design patterns help:</p><p><strong>Filter early.</strong> Push your .filter() operations as close to the source as possible. Every row you eliminate before a shuffle is a row that doesn&#39;t cross the network.</p><p><strong>Project early.</strong> Select only the columns you need before joining or grouping. Narrower rows = less shuffle data.</p><p><strong>Use map-side aggregations.</strong> Where possible, pre-aggregate data before the shuffle using reduceByKey instead of groupByKey. Spark&#39;s Tungsten engine handles this automatically for SQL-style aggregations, but it&#39;s worth knowing the principle.</p><h3>7. Data Partitioning — The Foundation of Parallelism</h3><p>Partitioning is how Spark divides data across the cluster. Get it wrong and you’ll either have too many tiny tasks (overhead dominates) or too few massive tasks (insufficient parallelism and potential OOM).</p><h3>Input Partitioning</h3><p>When Spark reads from HDFS or object storage, it typically creates one partition per input file block (default 128MB). Heavily fragmented data (thousands of tiny files) creates thousands of tiny tasks, each with significant scheduling overhead.</p><p><strong>The small files problem</strong> is endemic in streaming architectures. Fix it at the source by compacting files during write, or fix it at read time with coalesce:</p><pre># Coalesce: combine partitions WITHOUT shuffle (only works to reduce)<br>df = spark.read.parquet(&quot;path/&quot;).coalesce(100)<br># Repartition: redistribute WITH shuffle (can increase or decrease)<br>df = spark.read.parquet(&quot;path/&quot;).repartition(200)</pre><p><strong>Critical distinction:</strong> coalesce() avoids a shuffle but can produce uneven partitions. repartition() does a full shuffle but produces perfectly even partitions. Use coalesce when reducing partition count is acceptable with minor imbalance. Use repartition when you need balanced partitions for a subsequent expensive operation.</p><h3>Partitioning During Write</h3><p>How you write data determines how efficiently future jobs read it. Partitioning by a commonly filtered column allows Spark to skip irrelevant files entirely.</p><pre>df.write \<br>  .partitionBy(&quot;year&quot;, &quot;month&quot;, &quot;country&quot;) \<br>  .parquet(&quot;output/path/&quot;)</pre><p>This creates a directory structure like year=2024/month=01/country=US/. When a downstream query filters WHERE year = 2024 AND country = US, Spark only reads those directories — <strong>partition pruning</strong> in action.</p><p><strong>Partition column cardinality matters.</strong> Partitioning by a column with 100,000 unique values creates 100,000 directories, each potentially containing tiny files. Choose columns with manageable cardinality (dates, regions, product categories) not high-cardinality IDs.</p><h3>Repartitioning by Column Key</h3><p>Before a join, repartitioning both DataFrames by the join key eliminates the shuffle in the join itself — if both DataFrames are already colocated by the join key across partitions:</p><pre>df1 = df1.repartition(200, &quot;customer_id&quot;)<br>df2 = df2.repartition(200, &quot;customer_id&quot;)<br>joined = df1.join(df2, &quot;customer_id&quot;)  # No shuffle needed!</pre><p>This is a powerful optimization for repeated joins on the same key.</p><h3>8. Bucketing — Eliminating Shuffles at the Source</h3><p>Bucketing is partitioning’s more powerful sibling. While partitioning organizes data at the directory level, bucketing organizes data at the file level, enabling Spark to completely skip shuffles for certain join and aggregation patterns.</p><p>When you bucket a table by a column, Spark uses a hash function to assign rows to buckets and sorts within each bucket. The result: rows with the same key are always in the same bucket, across all files.</p><pre>df.write \<br>  .bucketBy(64, &quot;customer_id&quot;) \<br>  .sortBy(&quot;customer_id&quot;) \<br>  .saveAsTable(&quot;customers_bucketed&quot;)</pre><h3>Why Bucketing Eliminates Shuffles</h3><p>When you join two bucketed tables on their bucket column with the same number of buckets, Spark knows that matching rows are already colocated. No shuffle is needed — it can go directly to the join phase.</p><pre># Both tables bucketed on customer_id with 64 buckets<br># This join produces NO Exchange node in the query plan<br>result = customers_bucketed.join(orders_bucketed, &quot;customer_id&quot;)</pre><p>The shuffle elimination dramatically reduces latency — what might be a 10-minute sort-merge join becomes a 1-minute colocated join.</p><h3>Choosing the Right Number of Buckets</h3><p>Bucket count is a permanent decision embedded in your table metadata — you cannot change it without rewriting the table. Choose carefully.</p><p><strong>Rule of thumb:</strong> number of buckets should be a multiple of your typical executor count, and each bucket file should be roughly 128MB–256MB. Too few buckets and large files slow down individual tasks. Too many and you’re back to the small files problem.</p><h3>Bucket Pruning</h3><p>Like partition pruning, Spark can perform <strong>bucket pruning</strong> — when you filter on the bucket column, Spark reads only the specific buckets that could contain matching rows. This can reduce your read volume by an order of magnitude for point lookups.</p><h3>When NOT to Bucket</h3><p>Bucketing requires writing managed Hive-style tables (registered in the metastore), which isn’t always feasible. It also only helps if both sides of a join use compatible bucketing (same column, same count). If your joins vary in their key columns, bucketing’s benefits are limited.</p><h3>9. Caching — When to Store, When to Recompute</h3><p>Spark’s caching API is one of the most misused features in the entire framework. Engineers cache everything by reflex, consuming memory that would be better used for execution. A cache hit is fast, but an evicted cache entry that Spark must recompute is <em>slower</em> than if you had never cached at all.</p><h3>Storage Levels</h3><pre>from pyspark import StorageLevel</pre><pre>df.cache()                    # MEMORY_AND_DISK — the default<br>df.persist(StorageLevel.MEMORY_ONLY)          # Fastest, but OOM risk<br>df.persist(StorageLevel.MEMORY_AND_DISK)      # Spills to disk if needed<br>df.persist(StorageLevel.MEMORY_ONLY_SER)      # Serialized in memory, slower access but smaller<br>df.persist(StorageLevel.DISK_ONLY)            # Use sparingly; slower than recompute for simple ops</pre><h3>The Right Time to Cache</h3><p>Cache a DataFrame if:</p><ol><li>It will be <strong>used more than once</strong> in your pipeline (repeated actions, iterative algorithms)</li><li>The <strong>recomputation cost is high</strong> (complex joins, large aggregations)</li><li>You have <strong>sufficient memory</strong> and the cached data won’t evict execution memory needed for downstream operations</li></ol><p><strong>Do not cache</strong> if:</p><ul><li>The DataFrame is used only once</li><li>The transformation is fast (a simple filter or select)</li><li>The DataFrame is too large to fit comfortably in memory</li><li>You’re about to do a large shuffle — the cache may be evicted anyway</li></ul><h3>Unpersisting — The Forgotten Discipline</h3><p>Always unpersist when you’re done with a cached DataFrame. Spark <em>eventually</em> evicts unused cached data through LRU, but “eventually” isn’t good enough in long-running pipelines. Forgotten caches cause subtle memory pressure that shows up as mysterious slowdowns hours into a job.</p><pre>df.unpersist()  # Free up storage memory explicitly</pre><h3>10. Data Skew — The Silent Performance Killer</h3><p>Data skew is the single most common cause of “my Spark job is slow and I don’t know why.” You’ve tuned your partitions, optimized your joins, and your job is still stuck at 99% complete for 45 minutes. One task is running. Everything else finished long ago. That’s skew.</p><h3>What Is Skew?</h3><p>Skew occurs when data is unevenly distributed across partitions. One partition has 10 million rows, others have 1,000. Since Spark must wait for all tasks in a stage to complete before starting the next, one oversized partition becomes the bottleneck for the entire stage.</p><p>The most common cause: joining or grouping on a key that has extreme cardinality imbalance. User ID distributions in social networks are textbook examples — a handful of “power users” account for the majority of data.</p><h3>Detecting Skew</h3><p>The Spark UI is your friend here. In the Stage detail view:</p><ul><li>Look at the <strong>task duration</strong> distribution. A healthy job shows a bell curve. A skewed job shows a heavy tail.</li><li>Look at the <strong>data size per task</strong> distribution. Outlier tasks with 100x the data of average tasks are skewed.</li><li>Look for <strong>one task holding up the entire stage</strong> — the stage progress bar stalls at 99.x% while one task keeps running.</li></ul><h3>Skew vs. Spill</h3><p>Distinguish between skew and spill. <strong>Skew</strong> is an uneven key distribution problem. <strong>Spill</strong> is an insufficient memory problem — Spark doesn’t have enough execution memory and writes intermediate shuffle data to disk. A skewed job often causes spill (the overloaded partition runs out of memory), but a well-distributed job can also spill if executor memory is simply too small.</p><p>Look for Spill (Memory) and Spill (Disk) columns in your Stage metrics. Significant spill indicates memory pressure, regardless of skew.</p><h3>11. Salting — The Surgical Fix for Skew</h3><p>Salting is the primary technique for addressing join skew. It works by artificially inflating key cardinality to distribute the hot key across multiple partitions.</p><h3>How Salting Works</h3><p>The concept: instead of joining on customer_id = &#39;big_spender&#39; (which routes all matching rows to one partition), you join on customer_id + random_salt. But to make the join correct, you need to expand the <em>other</em> side to include all possible salt values.</p><pre>import pyspark.sql.functions as F<br># Step 1: Add a random salt (0 to N-1) to the skewed (large) table<br>N = 10  # Number of salt values<br>skewed_df = large_df.withColumn(&quot;salt&quot;, (F.rand() * N).cast(&quot;int&quot;))<br># Step 2: Explode all salt values into the small/lookup table<br>salt_range = spark.range(N).toDF(&quot;salt&quot;)<br>broadcast_df = small_df.crossJoin(F.broadcast(salt_range))<br># Step 3: Join on the composite key<br>result = skewed_df.join(broadcast_df, <br>                        on=[&quot;customer_id&quot;, &quot;salt&quot;], <br>                        how=&quot;inner&quot;) \<br>                  .drop(&quot;salt&quot;)</pre><h3>The Trade-off</h3><p>Salting increases the size of the broadcast table by N times. For N=10, your lookup table is 10x larger in memory. This is usually acceptable — lookup tables are small by definition. But don&#39;t blindly use N=100 just because you&#39;re nervous about skew.</p><h3>Targeted Salting for Extreme Skew</h3><p>When only a few keys are skewed (a common scenario — often 1% of keys hold 50% of the data), full salting is overkill. A more surgical approach:</p><ol><li>Identify the hot keys</li><li>Process hot-key rows separately with salting</li><li>Process normal rows with a regular join</li><li>Union the results</li></ol><p>This is more code, but it avoids unnecessarily expanding your broadcast table for the majority of clean keys.</p><h3>12. Adaptive Query Execution (AQE) — Let Spark Optimize Itself</h3><p>Introduced in Spark 3.0, AQE is one of the most significant performance improvements in Spark’s history. It enables Spark to make optimization decisions <em>at runtime</em>, based on actual data statistics collected during job execution — rather than relying solely on pre-execution estimates that are often wrong.</p><pre>spark.conf.set(&quot;spark.sql.adaptive.enabled&quot;, &quot;true&quot;)  # Default in Spark 3.2+</pre><h3>What AQE Does</h3><p><strong>1. Dynamic Coalescing of Shuffle Partitions</strong></p><p>Remember our discussion about shuffle partitions being notoriously hard to set correctly? AQE solves this. After a shuffle, AQE looks at the actual partition sizes and merges small partitions together, eliminating the 10,000 empty partitions that arise from setting spark.sql.shuffle.partitions too high.</p><pre>spark.conf.set(&quot;spark.sql.adaptive.coalescePartitions.enabled&quot;, &quot;true&quot;)<br>spark.conf.set(&quot;spark.sql.adaptive.advisoryPartitionSizeInBytes&quot;, &quot;128mb&quot;)</pre><p><strong>2. Dynamic Join Strategy Switching</strong></p><p>You set a spark.sql.autoBroadcastJoinThreshold, but Spark&#39;s pre-execution table size estimates are often wrong (especially after filters). AQE can switch a planned SortMergeJoin to a BroadcastHashJoin at runtime if it discovers that one side is actually small enough to broadcast.</p><pre>spark.conf.set(&quot;spark.sql.adaptive.autoBroadcastJoinThreshold&quot;, &quot;30mb&quot;)</pre><p><strong>3. Dynamic Skew Join Handling</strong></p><p>This is AQE’s most powerful feature. It detects skewed partitions <em>during execution</em> and automatically splits them into smaller sub-partitions, distributing the hot key’s data across multiple tasks.</p><pre>spark.conf.set(&quot;spark.sql.adaptive.skewJoin.enabled&quot;, &quot;true&quot;)<br>spark.conf.set(&quot;spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes&quot;, &quot;256mb&quot;)</pre><h3>AQE in Practice</h3><p>AQE is not a silver bullet. It works best for SQL workloads and DataFrame operations. It has less visibility into raw RDD operations. And it introduces some overhead from the additional planning passes at stage boundaries.</p><p>The right mental model: <strong>AQE is your safety net. Proper partitioning, good schema design, and thoughtful query structure are still your first line of defense. AQE handles the cases where you can’t know the data distribution upfront.</strong></p><p>Enable AQE by default for all production jobs running on Spark 3.x. The overhead is negligible and the benefits in unpredictable workloads are substantial.</p><h3>13. Dynamic Partition Pruning — Reading Only What You Need</h3><p>Dynamic Partition Pruning (DPP) is Spark’s mechanism for pushing filter conditions from a join’s inner table to the outer table’s partition scan — even when those filter conditions aren’t known until query planning time.</p><p>Consider this query:</p><pre>SELECT o.*, c.coutry <br>FROM orders o<br>JOIN customers c ON o.customer_id = c.id<br>WHERE c.country = &#39;Germany&#39;</pre><p>Without DPP, Spark scans all partitions of orders then joins, then filters. With DPP, Spark first determines which customer_ids belong to Germany, then <strong>uses that set to prune the orders table scan</strong> — only reading order partitions that contain German customers.</p><pre>spark.conf.set(&quot;spark.sql.optimizer.dynamicPartitionPruning.enabled&quot;, &quot;true&quot;)</pre><h3>Requirements for DPP to Trigger</h3><p>DPP only activates under specific conditions:</p><ol><li>The large table must be <strong>partitioned on the join key</strong></li><li>The filter must be on the <strong>small (dimension) table</strong></li><li>The small table side must be <strong>broadcastable</strong> (or DPP creates a subquery that’s executed first)</li></ol><p>This is why the star schema pattern (fact table + dimension tables) maps so naturally to DPP — your large fact table is partitioned on foreign keys, and your small dimension tables carry the filter conditions.</p><h3>Verifying DPP in the Query Plan</h3><p>When DPP is active, you’ll see a DynamicPruningExpression in your Physical Plan output. This is your confirmation that Spark is skipping irrelevant partitions.</p><h3>14. The Senior Architect’s Mental Model — Putting It All Together</h3><p>After covering all these techniques individually, let me give you the diagnostic framework I apply when I encounter a slow Spark job in the wild.</p><h3>The Performance Hierarchy</h3><p>Think of Spark optimization in layers, from most to least impactful:</p><p><strong>Layer 1 — Data Architecture (biggest impact)</strong></p><ul><li>Is your data in a columnar format (Parquet)? If not, fix this first.</li><li>Is your data partitioned on commonly-filtered columns?</li><li>Are your partition sizes reasonable (64MB–256MB)?</li><li>Do you have a small files problem?</li></ul><p><strong>Layer 2 — Query Design</strong></p><ul><li>Are you scanning more columns or rows than necessary? Push filters and projections early.</li><li>Are you doing wide joins (many-to-many)? They are quadratically expensive.</li><li>Are you doing unnecessary sorts or distinct operations?</li><li>Are you calling .collect() on large DataFrames? (Please don&#39;t.)</li></ul><p><strong>Layer 3 — Shuffle Optimization</strong></p><ul><li>Can you eliminate a shuffle by reordering operations?</li><li>Can bucketing eliminate a recurring join shuffle?</li><li>Is AQE enabled to handle unpredictable partition sizes?</li><li>Is your shuffle partition count appropriate?</li></ul><p><strong>Layer 4 — Skew and Resource Tuning</strong></p><ul><li>Is one task holding up your stage? Check for skew.</li><li>Are tasks spilling to disk? Check memory configuration.</li><li>Are your executors under-utilized? Check core/memory ratios.</li><li>Is GC time &gt; 10% of task time? Check serialization and object creation.</li></ul><p><strong>Layer 5 — Caching and Reuse</strong></p><ul><li>Are you recomputing the same DataFrame multiple times?</li><li>Are you caching DataFrames you only use once?</li><li>Are you using checkpointing for iterative algorithms?</li></ul><h3>The Diagnostic Workflow</h3><p>When you encounter a slow job:</p><ol><li><strong>Open the Spark UI.</strong> Go to the Stages tab. Find the longest-running stage.</li><li><strong>Check task duration distribution.</strong> Tight bell curve = even distribution, good. Long tail = skew, investigate.</li><li><strong>Check the query plan</strong> (explain(True)). Count Exchange nodes. Find Filter positions.</li><li><strong>Check shuffle read/write sizes.</strong> Unusually large shuffle = too little pre-aggregation or pre-filtering.</li><li><strong>Check spill metrics.</strong> Memory spill = insufficient executor memory or skewed data. Disk spill = serious memory pressure.</li><li><strong>Check GC time.</strong> High GC = object creation pressure. Consider Kryo serialization, reducing UDF complexity.</li><li><strong>Apply the fix.</strong> Address Layer 1 before Layer 4. Don’t tune executor memory when the real problem is a full table scan.</li></ol><h3>The Configuration Cheat Sheet</h3><pre># Adaptive Query Execution<br>spark.conf.set(&quot;spark.sql.adaptive.enabled&quot;, &quot;true&quot;)<br>spark.conf.set(&quot;spark.sql.adaptive.coalescePartitions.enabled&quot;, &quot;true&quot;)<br>spark.conf.set(&quot;spark.sql.adaptive.skewJoin.enabled&quot;, &quot;true&quot;)</pre><pre># Serialization<br>spark.conf.set(&quot;spark.serializer&quot;, &quot;org.apache.spark.serializer.KryoSerializer&quot;)</pre><pre># Broadcast threshold (adjust based on your data)<br>spark.conf.set(&quot;spark.sql.autoBroadcastJoinThreshold&quot;, &quot;50mb&quot;)</pre><pre># Dynamic Partition Pruning<br>spark.conf.set(&quot;spark.sql.optimizer.dynamicPartitionPruning.enabled&quot;, &quot;true&quot;)</pre><pre># Shuffle partitions (starting point, let AQE adjust)<br>spark.conf.set(&quot;spark.sql.shuffle.partitions&quot;, &quot;auto&quot;)  # Spark 3.2+</pre><pre># Memory management<br>spark.conf.set(&quot;spark.memory.fraction&quot;, &quot;0.7&quot;)<br>spark.conf.set(&quot;spark.memory.storageFraction&quot;, &quot;0.4&quot;)</pre><h3>Closing Thoughts</h3><p>Spark performance tuning is not about memorizing configuration parameters. It’s about developing an intuition for distributed systems — understanding where data is, how it moves, and what the cost of that movement is.</p><p>The engineers who tune Spark most effectively are the ones who have read query plans until they can look at an Exchange node and immediately ask &quot;could I eliminate this?&quot; They&#39;re the ones who open the Spark UI as a reflex, not an afterthought.</p><p>Every technique in this guide — from bucketing to salting to AQE — is a tool with a specific use case. Your job as a senior engineer is not to apply them all blindly, but to diagnose accurately and apply surgically.</p><p>Start with the data. Fix the architecture. Then tune the configuration. In that order, always.</p><p><strong>Tags:</strong> Apache Spark Data Engineering Performance Tuning Big Data PySpark Distributed Systems Databricks</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f690f6983ac" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/the-definitive-guide-to-apache-spark-performance-tuning-from-slow-to-blazing-fast-3f690f6983ac">The Definitive Guide to Apache Spark Performance Tuning: From Slow to Blazing Fast</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>