<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ram Avasarala on Medium]]></title>
        <description><![CDATA[Stories by Ram Avasarala on Medium]]></description>
        <link>https://medium.com/@sairam94.a?source=rss-9767b311a94c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*vTRHzbxTjyq8UhPq</url>
            <title>Stories by Ram Avasarala on Medium</title>
            <link>https://medium.com/@sairam94.a?source=rss-9767b311a94c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 18:14:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sairam94.a/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How a Hidden Cosmos DB Setting Crashed My Python Azure Function]]></title>
            <link>https://medium.com/@sairam94.a/the-8190-byte-error-how-a-hidden-cosmos-db-setting-crashed-my-python-azure-function-f4fd9d722f2e?source=rss-9767b311a94c------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4fd9d722f2e</guid>
            <category><![CDATA[http-trigger]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[cosmos-sdk]]></category>
            <category><![CDATA[azure-functions]]></category>
            <category><![CDATA[azure-cosmos-db]]></category>
            <dc:creator><![CDATA[Ram Avasarala]]></dc:creator>
            <pubDate>Sat, 15 Nov 2025 00:16:36 GMT</pubDate>
            <atom:updated>2025-11-16T03:01:54.434Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qs3Xi-P7-rZoKPM3ast3wg.png" /></figure><p>It’s the kind of bug that gives you trust issues.</p><p>You have an Azure Function that’s been running perfectly in production. It’s simple, clean, and does one job well. Then, one day, an internal customer reports that their data is missing. You check the logs, and you’re greeted by a cryptic error message that seems to come from deep within the Azure platform itself:</p><p><strong>“got more than 8190 bytes when reading header value is too long”</strong></p><p>My first reaction was… what? This isn’t my code. This isn’t a Python KeyError or a NullReferenceException. This is an error from the web host itself, appearing in the <strong>Azure Function logs under the Exceptions table</strong>.</p><p>This article is the story of that bug. It’s a debugging journey that starts with a high-volume customer, winds through the internals of the Cosmos DB Python SDK, and ends with a single, crucial-but-obscure setting that saved the day. If you use Python, Azure Functions, and Cosmos DB, this is the story of a bug that might be waiting for you, too.</p><h3>TL;DR — The Fix</h3><p>If you’re getting the “8190 bytes” error with Cosmos DB queries in Python:</p><pre>container.query_items(<br>    query=query,<br>    parameters=parameters,<br>    partition_key=customerId,<br>    continuation_token_limit=8  # Add this parameter!<br>)<br>    # your logic here</pre><p>Set continuation_token_limit=8 (KB) on your query_items() call to keep the continuation token under the Azure Function App&#39;s 8,190-byte header limit.</p><h3>The Architecture: A Simple, Scalable Setup</h3><p>To understand the bug, you first have to understand the setup. It was a standard, modern serverless architecture running on <strong>Azure Functions Linux Flex Consumption Plan</strong>:</p><p><strong>The Data:</strong> A Cosmos DB (NoSQL) container with <strong>autoscaling to max 10,000 RU/s</strong>, partitioned by customerId. This is a best-practice, highly efficient design.</p><p><strong>The Logic:</strong> A Python Azure Function using async and the azure-cosmos[aio]&gt;=4.6.0 library.</p><p><strong>The Job:</strong> The function’s purpose was to get a complete history for a single customer, which was spread across multiple documents. It would query all documents for a given customerId, loop through them, and build an in-memory dictionary where the key was a BusinessUnit and the value was a list of that unit&#39;s documents. Finally, it would return this complete JSON dictionary.</p><p>The code looked something like this:</p><pre>async def main(req: func.HttpRequest) -&gt; func.HttpResponse:<br>    <br>    customer_id = req.params.get(&#39;customerId&#39;)<br>    try:<br>      database = cosmos_client.get_database_client(&quot;myDatabase&quot;)<br>      container = database.get_container_client(&quot;myContainer&quot;)<br>        # The goal: Get ALL documents for the customer<br>       query = &quot;SELECT * FROM c WHERE c.customerId = @customerId&quot;<br>       parameters = [{&quot;name&quot;: &quot;@customerId&quot;, &quot;value&quot;: customer_id}]<br>       item_iterator=container.query_items(<br>            query=query,<br>            parameters=parameters,<br>            partition_key=customerId<br>        )<br>        <br>        results_by_bu = defaultdict(list)<br>        # This loop is where the magic... and the error... happens<br>        async for doc in item_iterator:<br>          bu = doc.get(&quot;businessUnit&quot;, &quot;unknown&quot;)<br>          results_by_bu[bu].append(doc)<br>        # Return the complete dictionary<br>        return func.HttpResponse(<br>            body=json.dumps(dict(results_by_bu)),<br>            mimetype=&quot;application/json&quot;,<br>            status_code=200<br>        )<br>    except Exception as e:<br>      return func.httpResponse(&quot;Error Processing your request&quot;)</pre><p>This code worked flawlessly for 99% of our customers. But for one, it was failing every single time with that “8190 bytes” error.</p><h3>The Debugging Descent: Finding the Real Culprit</h3><p>My first step was to see what was different about this one customer. I ran a simple SELECT COUNT(1) FROM c WHERE c.customerId = &#39;...&#39; against the database.</p><p><strong>The result: ~2,000 documents.</strong></p><p>That was the “Aha!” moment. This wasn’t a code logic error; it was a <strong>scale problem</strong>. My initial thought was that my in-memory dictionary was getting too large or the function was timing out. I even considered manually batching the query, running it in loops of 500 documents at a time.</p><p>But that felt like a messy workaround, not a real fix. Why was it failing with a <strong>header</strong> error?</p><p>The answer, as it so often is, was found in a <a href="https://github.com/Azure/azure-sdk-for-python/issues/27625">deep, dark GitHub issue thread</a>.</p><h3>The Real Problem: The Continuation Token</h3><p>The problem wasn’t my code. The problem was the <strong>continuation token</strong>.</p><p>When the azure-cosmos SDK iterates over a large result set using async for doc in ..., it&#39;s not actually getting all 2,000 documents in one go. It&#39;s intelligently <strong>paging</strong> the results. It queries for a batch, gets a continuation_token in the response header, and then uses that token in the next request&#39;s header to ask for the next page.</p><p>With 2,000 documents, the query was being broken into so many pages that the continuation token — which contains the state of all the pages already read — was <strong>growing larger and larger</strong>.</p><p>Eventually, it grew larger than <strong>8,190 bytes</strong>.</p><p>The <strong>Azure Function App host</strong> (specifically on Linux Flex Consumption Plan) has a hard limit on HTTP header size. My function wasn’t breaking; the SDK was trying to send a request with a header that the platform itself was rejecting.</p><h3>The Solution: Taming the Token</h3><p>The GitHub issue I found suggested a fix! It mentioned a property that you could set to limit the token size.</p><p>Here was the painful lesson: <strong>I was initially looking at solutions for the .NET SDK.</strong></p><p>This is a critical trap for polyglot developers. The concept was correct, but the implementation was completely different for Python.</p><p>After diving into the azure-cosmos Python SDK documentation, I found the real parameter: <strong>continuation_token_limit</strong></p><p>This parameter does exactly what it sounds like: it tells the SDK to keep the continuation token under a specific size in <strong>kilobytes</strong>. But here’s the key — it’s not set on the client constructor. <strong>It’s set on each </strong><strong>query_items() call.</strong></p><h3>The Fixed Code</h3><pre>container.query_items(<br>    query=query,<br>    parameters=parameters,<br>    partition_key=customerId,<br>    continuation_token_limit=8  # Add this parameter!<br>)<br>    # your logic here</pre><h3>Finding the Sweet Spot</h3><p>I had to find the right value through empirical testing:</p><ul><li><strong>2 KB</strong>: Worked</li><li><strong>8 KB</strong>: Worked</li><li><strong>9 KB</strong>: <strong>Failed </strong>with the 8,190-byte error</li></ul><p>The sweet spot was <strong>8 KB</strong>. Here’s why:</p><ul><li>8 KB = 8,192 bytes</li><li>The platform limit is 8,190 bytes</li><li>With a few bytes of overhead for other HTTP headers, 8 KB keeps you just under the limit</li></ul><p>Setting continuation_token_limit=8 ensures the token plus the other HTTP headers stays just under the platform&#39;s 8,190-byte threshold.</p><h3>Performance Impact: Negligible</h3><p>The best part? The “trade-off” was <strong>non-existent</strong> for my use case.</p><p>Forcing a smaller token means the SDK might have to make a few more, smaller queries to get the same data. With 2,000 documents, the SDK breaks the query into more page requests, but:</p><ul><li>Response time remained well within acceptable limits</li><li>RU consumption increased slightly but wasn’t significant in the grand scheme of things with our 10,000 RU/s autoscaling budget</li><li>The function completed successfully every time</li></ul><p>The fix solved the problem without any meaningful performance degradation.</p><h3>Conclusion: Key Takeaways for Azure Functions Developers</h3><p>This bug was a fantastic reminder of several key lessons:</p><h3>1. Your Error Might Not Be Your Error</h3><p>That “8,190 bytes” error looked like a platform issue because it <strong>was</strong>. It was a conflict between the SDK’s behavior (a giant header) and the Azure Function host’s rules (an 8KB header limit on Linux Flex Consumption Plan).</p><h3>2. SDKs Are Not Monolithic</h3><p>The solution you find on GitHub or Stack Overflow might be for a different language or SDK version. Always verify concepts against the <strong>specific SDK version and language</strong> you are using.</p><p>The .NET ResponseContinuationTokenLimitInKB (set on the client) is <strong>NOT</strong> the same as Python&#39;s continuation_token_limit (set on each query).</p><h3>3. Understand “Hidden” Mechanisms</h3><p>I never explicitly handled a continuation token. The async for loop in Python made it all invisible... until it broke.</p><p><strong>Knowing how your tools work under the hood</strong> (like automatic pagination) is the key to debugging them when they fail at scale.</p><h3>4. Scale Testing Matters</h3><p>This code worked perfectly for 99% of customers. The bug only appeared when a single customer had ~2,000 documents. Always test your code with realistic high-volume scenarios.</p><h3>5. Documentation Is Your Friend</h3><p>When cross-platform examples fail, go straight to the <strong>official Microsoft documentation</strong> for your specific SDK. In this case, the Python SDK docs clearly showed continuation_token_limit as a parameter on query_items().</p><h3>6. Monitoring and Alerting Save Time</h3><p>This error appeared in the Azure Function logs’ Exceptions table, but we only discovered it when a customer reported missing data. If we had been streaming logs to a centralized system like Databricks with proper alerting, we would have caught this issue immediately instead of troubleshooting reactively. Proactive monitoring is worth the investment.</p><h3>Additional Considerations</h3><h3>Partition Key Design</h3><p>My container is partitioned by customerId, which means all 2,000 documents for this customer are in a <strong>single logical partition</strong>. This is efficient for queries (no cross-partition queries needed), but be aware of Cosmos DB&#39;s 20GB logical partition limit if your customers have massive datasets.</p><h3>Query Optimization</h3><p>Consider if you really need SELECT *. Using projections can reduce both response size and continuation token overhead:</p><pre>query = &quot;SELECT c.id, c.businessUnit, c.relevantField FROM c WHERE c.customerId = @customerId&quot;</pre><h3>Hosting Plan Considerations</h3><p>This 8,190-byte header limit appears to be specific to <strong>Azure Functions on Linux Flex Consumption Plan</strong>. If you’re on a different hosting plan (Windows, Premium, etc.), you may or may not encounter this issue — but it’s worth being aware of.</p><h3>References</h3><ul><li><a href="https://learn.microsoft.com/en-us/python/api/azure-cosmos/azure.cosmos.containerproxy?view=azure-python#azure-cosmos-containerproxy-query-items">Azure Cosmos DB Python SDK Documentation — query_items</a></li><li><a href="https://pypi.org/project/azure-cosmos/">Azure Cosmos DB Python SDK on PyPI</a></li></ul><p><em>Have you encountered similar SDK pagination issues? Drop a comment below with your debugging war stories!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4fd9d722f2e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Schema Evolution in Spark Streaming: The Painful Truth About mergeSchema=true]]></title>
            <link>https://medium.com/@sairam94.a/schema-evolution-in-spark-streaming-the-painful-truth-about-mergeschema-true-716268e32b6f?source=rss-9767b311a94c------2</link>
            <guid isPermaLink="false">https://medium.com/p/716268e32b6f</guid>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[spark-streaming]]></category>
            <category><![CDATA[delta-lake]]></category>
            <dc:creator><![CDATA[Ram Avasarala]]></dc:creator>
            <pubDate>Thu, 06 Nov 2025 03:39:29 GMT</pubDate>
            <atom:updated>2025-11-06T03:39:29.471Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/312/1*Mku0zwkPCcmFMaK4QwQ-BQ.png" /></figure><p>If you’ve ever set up a Spark Structured Streaming job, you’ve probably seen the mergeSchema=true option when writing to a Delta table. It sounds like a magic bullet, doesn&#39;t it?</p><p>When I first implemented my pipeline — reading from Kafka with Schema Registry and writing to Delta — I thought I had schema evolution covered. Just add that one option, and Spark would magically handle any future schema changes. After all, we’re using Schema Registry! What could possibly go wrong?</p><p>As it turns out, almost everything.</p><h3>The “Future-Proof” Setup</h3><p>Here’s a simplified version of my initial pipeline. It’s a classic setup: read from Kafka, deserialize Avro using Schema Registry, and write to a Delta table.</p><pre># Reading from Kafka with Schema Registry<br>df = spark \<br>    .readStream \<br>    .format(&quot;kafka&quot;) \<br>    .option(&quot;kafka.bootstrap.servers&quot;, &quot;localhost:9092&quot;) \<br>    .option(&quot;subscribe&quot;, &quot;my-topic&quot;) \<br>    .option(&quot;startingOffsets&quot;, &quot;earliest&quot;) \<br>    .load()<br><br># Deserialize using Schema Registry<br>from pyspark.sql.avro.functions import from_avro<br>import requests<br><br>schema_registry_url = &quot;http://localhost:8081&quot;<br>response = requests.get(f&quot;{schema_registry_url}/subjects/my-topic-value/versions/latest&quot;)<br>avro_schema = response.json()[&#39;schema&#39;]<br><br>parsed_df = df.select(<br>    from_avro(col(&quot;value&quot;), avro_schema).alias(&quot;data&quot;)<br>).select(&quot;data.*&quot;)<br><br># Write to Delta table with the &quot;magic&quot; mergeSchema<br>query = parsed_df \<br>    .writeStream \<br>    .format(&quot;delta&quot;) \<br>    .option(&quot;checkpointLocation&quot;, &quot;/tmp/checkpoint&quot;)<br>    .option(&quot;mergeSchema&quot;, &quot;true&quot;) <br>    .outputMode(&quot;append&quot;) \<br>    .table(&quot;my_table&quot;)</pre><p>Simple, elegant, and, I assumed, future-proof. (Spoiler: it wasn’t.)</p><h3>Discovery #1: mergeSchema Really Means addNewColumns</h3><p>My first schema change was simple: I added a new, optional field customer_segment to my Kafka messages. The Schema Registry accepted the change (it was backward compatible), and I sat back, expecting Spark to handle it.</p><p><strong>Result:</strong> It worked! The new column appeared in my Delta table. I felt like a genius.</p><p>Emboldened, I made a <em>real</em> change. I wanted to refactor an existing address field from a simple string to a struct&lt;city: string, country: string&gt; to properly capture structured data.</p><p><strong>Result:</strong> <strong>And everything broke.</strong> Well, not “broke” as in a crash, which would have been nice. Instead, something far worse happened: my table started showing <strong>NULL</strong> values for the address column on all new records, even though I could see the rich JSON data in my Kafka topic.</p><h4>What I Learned</h4><p>This is when I learned the painful truth: mergeSchema=true has a very specific, and limited, definition in Spark Streaming:</p><ul><li><strong>Adding new top-level columns:</strong> Supported</li><li><strong>Removing columns:</strong> Ignored (old columns remain NULL for new data)</li><li><strong>Changing column data types:</strong> <strong>Not supported</strong></li><li><strong>Modifying nested struct schemas:</strong> <strong>Definitely not supported</strong></li></ul><p>The documentation says “merge schema,” but it really means “add new columns to the schema.” <strong>It’s additive only.</strong> It will never change an existing column’s type or structure.</p><h3>Discovery #2: The Checkpoint Remembers Everything</h3><p>My next thought was the classic “turn it off and on again.” I’ll just clear the checkpoint directory and restart the stream. The new schema will be picked up, and everything will be fine, right?</p><p>rm -rf /tmp/checkpoint</p><p><strong>This made things so much worse.</strong></p><p>Instead of silent NULL values, the stream now refused to start at all, throwing a massive error:</p><p>com.databricks.sql.transaction.tahoe.DeltaAnalysisException: [DELTA_FAILED_TO_MERGE_FIELDS] Failed to merge fields ‘address’ and ‘address’</p><p>Wait, what? I deleted the checkpoint!</p><h4>The Hidden Truth About Checkpoints</h4><p>Here’s the trap I fell into. When you clear the checkpoint <em>but keep the Delta table</em>, Spark wakes up confused. It tries to reconcile three different schemas:</p><ol><li><strong>The New Incoming Schema:</strong> From Kafka (e.g., address is a struct).</li><li><strong>The Existing Table Schema:</strong> In the Delta table (e.g., address is a string).</li><li><strong>The (Now Missing) Checkpoint:</strong> Spark can’t find it, so it looks at the <em>Delta table’s</em> schema as the “source of truth” to restore from, and it conflicts with the new data.</li></ol><p>This mismatch between the data Spark <em>wants</em> to write and the data the <em>table</em> already has is a hard stop. Deleting the checkpoint just removed Spark’s memory of <em>what</em> it had already processed, forcing this conflict to the surface immediately.</p><h3>Discovery #3: There Is No Magic Config Flag</h3><p>Surely, I was just missing a config. I spent an embarrassing amount of time digging through documentation and trying every flag I could find:</p><pre>spark.conf.set(&quot;spark.sql.streaming.schemaInference&quot;, &quot;true&quot;)<br>spark.conf.set(&quot;spark.databricks.delta.schema.autoMerge.enabled&quot;, &quot;true&quot;)</pre><p><strong>Result:</strong> Still failed. These configurations don’t change the fundamental limitations:</p><ul><li>schemaInference is for <em>batch</em> sources, not for an already-defined stream schema.</li><li>delta.schema.autoMerge.enabled (which is what option(&quot;mergeSchema&quot;, &quot;true&quot;) sets) is still additive-only.</li></ul><p>They simply don’t fix the core problem: a streaming query’s schema is locked in by its checkpoint, and mergeSchema can&#39;t alter existing columns.</p><h3>The Real Solutions (That Actually Work)</h3><p>After <em>finally</em> accepting the reality of the situation, I found there are only a few patterns to <em>actually</em> handle breaking schema changes.</p><h4>Solution 1: The “Rename and Recreate” Pattern</h4><p>When you need to change a column’s type or structure, you treat the new version as a completely new column.</p><p><strong>Step 1: Rename the old column.</strong> Before making any code changes, rename the problematic column in your target Delta table.</p><pre>ALTER TABLE my_table <br>RENAME COLUMN address TO address_legacy</pre><p><strong>Step 2: Clear the checkpoint.</strong> Now that the conflicting column name is gone from the table, it’s safe to clear the checkpoint. rm -rf /tmp/checkpoint</p><p><strong>Step 3: Restart the stream.</strong> Your new stream code is producing the address struct. Spark will start, see no address column in the target table, and mergeSchema will (finally!) do its job and add the new address struct column.</p><ul><li><strong>Pros:</strong> No data loss. You have a clear audit trail (address_legacy) and can even write a batch job later to backfill the new struct from the old data.</li><li><strong>Cons:</strong> Requires manual intervention and a (brief) stream restart. Downstream queries must be updated to use the new column.</li></ul><h4>Solution 2: The “Nuke and Pave” (Full Reprocessing)</h4><p>If your data retention policy allows and the dataset isn’t massive, you can just start over.</p><ol><li>Drop the Delta table entirely. spark.sql(&quot;DROP TABLE IF EXISTS my_table&quot;)</li><li>Clear the checkpoint. rm -rf /tmp/checkpoint</li><li>Restart the stream, making sure to set startingOffsets to &quot;earliest&quot; to reprocess all data from Kafka.</li><li>When you reprocess from earliest, you&#39;re reading messages that were written with the <strong>old schema</strong>, and <strong>new schema</strong>. You should add transformation logic in your code that handles all three scenarios</li></ol><ul><li><strong>Pros:</strong> You get a clean, consistent schema from start to finish.</li><li><strong>Cons:</strong> Involves downtime and can be extremely time-consuming and costly. This is not feasible for large or business-critical datasets.</li></ul><h4>Solution 3: The Production-Grade Versioning Strategy</h4><p>For production systems, this is the only way: plan for breaking changes from the start.</p><ol><li>When a breaking schema change is needed (like address string -&gt; struct), <strong>create a new Kafka topic</strong> (my-topic-v2).</li><li><strong>Create a new Delta table</strong> (my_table_v2).</li><li><strong>Deploy a new streaming application</strong> that reads from v2 and writes to v2.</li><li>Run both v1 and v2 streams in parallel during a migration period.</li><li>Update downstream consumers to read from my_table_v2.</li><li>Once migration is complete, decommission the v1 stream.</li></ol><ul><li><strong>Pros:</strong> Zero downtime. Explicit versioning makes rollbacks and management easy.</li><li><strong>Cons:</strong> Requires more infrastructure planning and setup.</li></ul><h3>Key Takeaways</h3><p>My painful journey, summarized for you:</p><ol><li><strong>mergeSchema=true just means </strong><strong>addNewColumns.</strong> It&#39;s additive only.</li><li><strong>A streaming checkpoint locks in the schema.</strong> Once started, it’s immutable for existing columns.</li><li><strong>Changing structs or column types</strong> is a <em>breaking change</em> for a live stream, period.</li><li><strong>Schema Registry compatibility ≠ Spark streaming compatibility.</strong> They solve two different problems.</li><li><strong>Plan for versioning from day one.</strong> If a pipeline is critical, use Solution 3.</li><li><strong>Test your <em>evolution plan</em>,</strong> not just your code. Before going to prod, prove you can add a column, <em>and</em> prove you can change a column’s type.</li></ol><h3>Conclusion</h3><p>Schema evolution in Spark Structured Streaming is far more nuanced than the documentation suggests. The combination of Spark’s immutable checkpoints, Delta’s schema enforcement, and the additive-only nature of mergeSchema creates a perfect storm of confusion.</p><p>The good news? Once you understand these limitations, you can design your pipelines to handle schema changes gracefully. The key is planning ahead and choosing the right strategy for your use case.</p><p>Have you encountered schema evolution headaches in your streaming pipelines? I’d love to hear your war stories and solutions in the comments below</p><p><em>If you found this helpful, follow me for more real-world data engineering insights.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=716268e32b6f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What I Learned About Spark Shuffles After 8 Years of Writing Production Jobs]]></title>
            <link>https://medium.com/@sairam94.a/what-i-learned-about-spark-shuffles-after-8-years-of-writing-production-jobs-33d454c92150?source=rss-9767b311a94c------2</link>
            <guid isPermaLink="false">https://medium.com/p/33d454c92150</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Ram Avasarala]]></dc:creator>
            <pubDate>Wed, 22 Oct 2025 00:50:28 GMT</pubDate>
            <atom:updated>2025-11-18T03:18:04.006Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*gs2WaRhLlUs9niaHuzL51A.png" /></figure><h3>Eight years of experience, and one interview question exposed a critical gap in my knowledge.</h3><p>I was interviewing at <strong>a FAANG company</strong> for a senior data engineer position. Eight years of experience, hundreds of Spark jobs in production — I knew how to build efficient pipelines. I could optimize queries, tune configurations, minimize shuffles, and debug performance issues.</p><p>Then the interviewer asked: “When a shuffle happens in Spark, where exactly is the data stored?”</p><p>“It’s written to disk on the executors,” I answered.</p><p>“Where on disk? What’s the directory structure?”</p><p>I paused. I knew shuffles were expensive. I knew how to avoid unnecessary ones. I knew spark.sql.shuffle.partitions affected performance. But I&#39;d never actually looked under the hood—where the files lived, how they were named, how long they persisted.</p><p>That question revealed a gap: I’d been working at the Spark API level for years without understanding the execution layer underneath. I was productive and effective, but I was missing the foundational knowledge that separates good engineers from great ones.</p><p>That interview sent me on a deep dive. Here’s what I learned.</p><p><strong>UPDATE (October 2025):</strong>After publishing this post, I discovered I’d described the old hash-based shuffle implementation (pre-Spark 1.4). Modern Spark uses sort-based shuffle, which is far more efficient — each map task creates just 2 files instead of hundreds. I’ve updated the technical sections below to reflect the current implementation. This was a great learning moment: even when you think you understand something, diving deeper often reveals you’re working with outdated information!</p><h3>The Basics: How Spark Decides Partition Count</h3><p>Before understanding shuffles, I needed to revisit partitions. When you load a file, how does Spark decide how to split it?</p><pre>sales = spark.read.csv(&quot;sales.csv&quot;)  # 300MB file</pre><p><strong>The configuration:</strong> spark.sql.files.maxPartitionBytes (default: 128MB)</p><p>Spark divides the total file size by this value:</p><pre>300MB / 128MB ≈ 3 partitions</pre><p>I knew this at a high level, but I’d never thought about how it affects downstream operations. More on that below.</p><h3>The Join: Where Shuffles Get Real</h3><p>Here’s where my knowledge gap became obvious. Consider this join:</p><pre>sales = spark.read.csv(&quot;sales.csv&quot;)          # 300MB, 3 partitions<br>customers = spark.read.csv(&quot;customers.csv&quot;)  # 50MB, 1 partition</pre><pre>result = (<br>    sales<br>    .filter(col(&quot;date&quot;) &gt;= &quot;2024-01-01&quot;)<br>    .withColumn(&quot;amount_with_tax&quot;, col(&quot;amount&quot;) * 1.1)<br>    .join(customers, &quot;customer_id&quot;)<br>)</pre><p><strong>I knew this would trigger a shuffle.</strong> What I <em>didn’t</em> know was exactly what happened at the file system level.</p><h3>Stage 1: Map Side (Shuffle Write)</h3><p>Here’s what actually happens in the first stage:</p><ul><li><strong>Read the sales data</strong> (3 partitions = 3 tasks)</li><li><strong>Apply transformations</strong> (filter and withColumn flow through without shuffling)</li><li><strong>Hash the join key</strong> to determine target partitions: partition_id = hash(customer_id) % spark.sql.shuffle.partitions</li><li><strong>Sort and write</strong> shuffle data to local disk on each executor</li></ul><p><strong>The part I never knew:</strong> The exact file structure.</p><p><strong>Files created:</strong></p><pre>spark.local.dir/blockmgr-{uuid}/<br>  shuffle_0_0.data    ← All partitions from map task 0, concatenated<br>  shuffle_0_0.index   ← Byte offsets for quick lookup<br>  shuffle_0_1.data    ← All partitions from map task 1<br>  shuffle_0_1.index<br>  shuffle_0_2.data    ← All partitions from map task 2<br>  shuffle_0_2.index</pre><p>File naming pattern: shuffle_{shuffleId}_{mapId}.data/index</p><p>shuffleId: Unique ID for this shuffle operation (0, 1, 2…)</p><p>mapId: Which input partition/task created this file (0, 1, 2…)</p><p>If spark.sql.shuffle.partitions = 16 and you have 3 input partitions:</p><ul><li>Each of the 3 tasks writes 2 <strong>files</strong> (index and data files)</li><li>Total: 6 <strong>shuffle files</strong> from the sales side alone</li><li>The same process happens for customers (1 partition × 2= 2 <strong>more files</strong>).</li><li>Each .data file contains output for all 16 target partitions, concatenated together. The .index file stores byte offsets so reduce tasks can quickly find their data</li></ul><p>This shuffle write marks the end of Stage 1.</p><h3>Stage 2: Reduce Side (Shuffle Read)</h3><p>Now Stage 2 begins:</p><ul><li><strong>Shuffle read:</strong> Spark creates 16 tasks (one per shuffle partition)</li><li>Each task reads its assigned partition from ALL previous tasks</li><li>If files are on remote executors, data transfers over the network</li><li>Files are read from spark.local.dir into memory</li><li><strong>Sort-merge join: </strong>Both datasets are sorted by customer_id</li><li>Matching keys are merged</li><li>All rows with the same customer_id are guaranteed to be in the same partition</li><li>Next operations flow through until another shuffle or action</li></ul><h3>What Finally Clicked</h3><p>After digging into this, several things became clear:</p><h4>1. Shuffle Files Live on Disk, Not Memory</h4><blockquote><strong><em>What I thought:</em></strong><em> “Shuffle data goes into memory for processing”</em></blockquote><blockquote><strong><em>Reality:</em></strong><em> Shuffle data is </em><strong><em>written to disk first</em></strong><em> (</em><em>spark.local.dir), then read back into memory. This provides:</em></blockquote><ul><li><strong>Fault tolerance</strong> (tasks can retry without recomputing)</li><li><strong>Memory relief</strong> (data doesn’t all fit in RAM)</li><li><strong>Persistence</strong> across stage boundaries</li></ul><h4>2. The Map/Reduce Split is Literal</h4><blockquote><strong><em>What I thought:</em></strong><em> “Stages are just logical groupings”</em></blockquote><blockquote><strong><em>Reality:</em></strong><em> The shuffle files on disk are the </em><strong><em>physical boundary</em></strong><em> between stages.</em></blockquote><ul><li><strong>Stage 1 (map):</strong> Process data → write shuffle files → stage ends</li><li><strong>Stage 2 (reduce):</strong> Read shuffle files → continue processing</li></ul><blockquote><em>The files persist on disk until the application completes.</em></blockquote><h4>3. Each Shuffle Gets a Unique ID</h4><p>If your job has multiple shuffles:</p><pre>result = (<br>    sales<br>    .join(customers, &quot;customer_id&quot;)      # shuffle_0<br>    .groupBy(&quot;country&quot;).agg(sum(&quot;amount&quot;)) # shuffle_1<br>    .orderBy(&quot;total&quot;)                      # shuffle_2<br>)</pre><p>You’ll see:</p><pre>spark.local.dir/blockmgr-abc123/<br>  ├── shuffle_0_0.data    ← All in same directory<br>  ├── shuffle_0_0.index<br>  ├── shuffle_0_1.data<br>  ├── shuffle_0_1.index<br>  ├── shuffle_1_0.data<br>  ├── shuffle_1_0.index<br>  ├── shuffle_1_1.data<br>  ├── shuffle_1_1.index<br>  ├── shuffle_2_0.data<br>  ├── shuffle_2_0.index<br>  └── ...</pre><p>Each shuffle operation creates its own set of files.</p><h4>4. Number of Tasks = Number of Partitions</h4><blockquote><strong><em>In shuffle write stage:</em></strong><em> Tasks = input partitions (3 in our example)</em></blockquote><blockquote><strong><em>In shuffle read stage:</em></strong><em> Tasks = </em><em>spark.sql.shuffle.partitions (default 200)</em></blockquote><p>I knew to tune this config, but now I understand <em>why</em>:</p><ul><li><strong>Too few</strong> (e.g., 4 on an 8-core cluster) = cores sit idle</li><li><strong>Too many</strong> (e.g., 1000 on an 8-core cluster) = task overhead dominates</li></ul><p>Rule of thumb: 2–4× your number of cores</p><pre># For 8 cores:<br>spark.conf.set(&quot;spark.sql.shuffle.partitions&quot;, &quot;32&quot;)</pre><h3>The Practical Implications</h3><p>Understanding these internals changed how I think about optimization:</p><ul><li><strong>Shuffle Disk Space Matters</strong> Large jobs can generate gigabytes of shuffle files. If spark.local.dir fills up, executors crash.</li><li><strong>What I do now:</strong> Monitor disk usage on executors, configure adequate disk space, and set appropriate shuffle cleanup timeouts.</li><li><strong>Network Becomes the Bottleneck</strong> When shuffle read happens, data moves across the network if a task needs data from a shuffle file on a different executor.</li><li><strong>What I do now:</strong> Minimize shuffle data size (filter early, project only needed columns), consider broadcast joins for small tables, and watch Spark UI metrics: “Shuffle Read” vs “Shuffle Read (local)”.</li><li><strong>What matters more:</strong> Minimizing shuffle size and count.</li></ul><h3>What This Taught Me</h3><p>I was already effective with Spark. I could write efficient code, optimize performance, and deliver production pipelines.</p><p>But understanding the internals made me <em>better</em>. Now when I tune configurations or debug slow jobs, I’m not just following best practices — I understand WHY they work.</p><p><strong>The lesson: There’s a difference between using a tool effectively and understanding how it works. Both matter.</strong></p><p>What about you? What tools do you use daily without fully understanding their internals? Sometimes it’s worth taking the dive.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=33d454c92150" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>