<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sidharth  Kumar on Medium]]></title>
        <description><![CDATA[Stories by Sidharth  Kumar on Medium]]></description>
        <link>https://medium.com/@k.sidharth?source=rss-63f18bcc00c9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*pqhihRGAj2WsRnz0MSMr6w.jpeg</url>
            <title>Stories by Sidharth  Kumar on Medium</title>
            <link>https://medium.com/@k.sidharth?source=rss-63f18bcc00c9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 18:06:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@k.sidharth/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a Workload-Aware AWS DataSync Agent Manager: How the Scheduler Actually Works (Part 2)]]></title>
            <link>https://medium.com/@k.sidharth/building-a-workload-aware-aws-datasync-agent-manager-how-the-scheduler-actually-works-part-2-7f1ff8d28183?source=rss-63f18bcc00c9------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f1ff8d28183</guid>
            <category><![CDATA[cloud-migration]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[hybrid-cloud]]></category>
            <category><![CDATA[aws-datasync]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Sidharth  Kumar]]></dc:creator>
            <pubDate>Thu, 13 Aug 2026 10:42:39 GMT</pubDate>
            <atom:updated>2026-08-13T11:12:00.716Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A deep dive into the DataSync Agent Manager’s</em> <em>three critical component that deliver load-aware, subnet-scoped agent selection, crash-safe provisioning, and lifecycle observability.</em></p><blockquote>This is Part 2 of a two-part series (<a href="https://medium.com/@k.sidharth/choosing-the-right-aws-datasync-agent-for-every-task-at-petabyte-scale-part-1-fd9cd1b2f7bf?sharedUserId=k.sidharth"><strong>Part-1 </strong>covers the design challenge and the core architectural decision</a>). The pattern described here is implemented in a petabyte-scale enterprise data migration platform, where the <strong>AWS DataSync Agent Manager</strong> is the scheduling and observability layer for all DataSync-backed transfers, integrated into the platform’s multi-stage migration orchestration pipeline.</blockquote><h3>Picking Up from Part 1</h3><p>Part 1 established two constraints that make agent selection hard at scale: AWS DataSync binds the agent ARN at task creation, making the choice permanent; and this platform migrates table by table, with each table spawning its own DataSync task created at runtime. Together they mean agent selection is not a one-time configuration decision but a per-task choice made thousands of times over. The architectural response was a state-driven model built on a continuously polled agent state table, used as the sole input to every selection — so each choice is load-aware and subnet-scoped. This post covers how the three components work, how selection fits into the orchestration pipeline, and the reliability and observability patterns underneath.</p><h3>Component Design and Rationale</h3><h4>Authorization Reconciler as a Startup-Only, Single-Owner Component</h4><p>Runs once at service startup. Reads the operator-configured list of approved AWS DataSync agents and marks each as eligible in the database. This is the only component with write access to the eligibility flag. The background poller never modifies it.</p><p>This separation is enforced by design. If polling logic could revoke agent eligibility, a transient AWS API failure during any maintenance window would silently block all new migration jobs, a failure mode that would be difficult to detect and alarming to discover in production. Confining authorization to a startup-only process driven solely by operator configuration eliminates that risk structurally.</p><h4>Background Poller with Explicit Fetch and Write Separation</h4><p>Runs on a dedicated thread pool, isolated from other platform schedulers carrying time-sensitive tasks at sub-second cadences. Each cycle is split into two explicitly separated phases.</p><p><strong>Fetch phase with no database connection held.</strong> The poller calls AWS DataSync APIs to enumerate agent inventory, scans active task executions scoped to platform-owned tasks, and computes per-agent load counts. Location type determines agent involvement. NFS, SMB, and HDFS-backed transfers route through agent VMs while cloud-native storage transfers do not. Results are assembled in memory, with no database connection held open during network I/O. This prevents connection pool exhaustion when AWS DataSync API responses degrade.</p><p><strong>Write phase using two short database transactions.</strong> The poller persists only what changed since the previous cycle, then executes a bulk reset of task counts for any eligible agent not seen in the current scan. Agents that go offline do not retain phantom load indefinitely.</p><h4>Selection Service with Subnet Scoping, Two-Tier Priority and Safe Degradation</h4><p>Read-only and state-driven. Every selection request passes through a fixed sequence before an agent ARN is returned for AWS DataSync task creation.</p><p><strong>1st Step: Narrows the candidate pool to the same subnet.</strong> AWS DataSync requires that the agent and the source endpoint share network reachability. This filter runs first, eliminating structurally ineligible agents before any load-based comparison begins.</p><p><strong>2nd Step: applies the eligibility check.</strong> Only approved, online agents proceed to tier evaluation.</p><ul><li><strong>Tier 1 covers idle agents</strong> ordered by time since last assignment. The longest-waiting agent in the subnet receives the next job, distributing load evenly across the eligible fleet.</li><li><strong>Tier 2 covers agents in verification only.</strong> Checksum verification is I/O-light relative to active transfer. An agent in this phase can accept a new task without meaningful overload risk.</li></ul><p><strong>When no agent is available</strong>, the caller receives an explicit unavailability signal and backs off rather than forcing a saturated assignment.</p><p>All multi-row database writes in both the poller and selection service are ordered by agent identity, eliminating the possibility of row-lock cycles under concurrent access without requiring pessimistic locking.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*fFCBb0MnI_wSGq6Q1Zxh2g.png" /><figcaption><em>Figure: Selection service decision flow showing subnet filter, eligibility check, two-tier priority, and safe fallback.</em></figcaption></figure><h3>AWS DataSync Agent Selection in the Orchestration Pipeline</h3><p>The Agent Manager is a gating component. Because every table migration triggers a new DataSync task at runtime, agent selection is exercised for every task creation event across potentially hundreds of concurrent migrations. The sequence is fixed: verify idempotency, select an agent, then provision infrastructure. If the subnet-scoped pool yields no eligible agent across either tier, provisioning is skipped entirely. No environment is created, no infrastructure is orphaned, and the failure surfaces as an explicit actionable signal. The selected agent ARN then flows into the infrastructure provisioning layer, where it is embedded into the AWS DataSync task being created, binding the source location to the chosen agent as the service requires.</p><h3>Reliability Under Failure: The Sentinel Pattern</h3><p>Infrastructure provisioning is a remote call that can fail, be retried, or be interrupted by a service restart. Without a guard, a retry after a crash creates a duplicate environment requiring manual cleanup.</p><p>The solution is the <strong>sentinel pattern</strong>: a placeholder record is written to the database before the external call is made. If the service restarts mid-operation, the placeholder signals provisioning is already in flight and the system resumes rather than restarts. On success, the placeholder is atomically replaced with the confirmed result. Once committed, that result is immutable and any attempt to overwrite it is rejected explicitly. The deeper architectural value is that every operation’s state is captured before the side effect is executed, making the system’s full state machine inspectable from the database at any point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*z0nRhZJfhSGlnaaBnog2qA.png" /><figcaption><em>Figure: Sentinel pattern sequence showing normal provisioning path and crash-recovery path, with the placeholder record acting as the in-flight signal.</em></figcaption></figure><h3>Observability Through Two Independent Lifecycle Loops</h3><p>Once an AWS DataSync transfer task is running, two separate polling services track progress on independent schedules. An <strong>infrastructure provisioning tracker</strong> monitors whether the provisioned environment came up successfully and extracts the DataSync task identifier on success. A <strong>transfer execution tracker</strong> monitors whether data is moving and tracks stage transitions through to completion.</p><p>These are kept deliberately independent so that an infrastructure API disruption cannot stall transfer monitoring for unrelated jobs.</p><h3>Trade-offs We Made</h3><p><strong>Polling staleness versus event-driven accuracy</strong>: scheduling decisions rely on data that can be up to five minutes old, which is acceptable for hours-to-days migrations but constraining for sub-minute workloads. An EventBridge-driven alternative was evaluated and rejected for this on-premises deployment; see Part 1 for the full comparison.</p><p><strong>Database as coordination layer.</strong> A standard relational database was chosen over a distributed cache. When the database is unavailable, the system returns a no-agent-available signal, which is safe and self-recovering, with no additional infrastructure required. A distributed cache would offer better fault tolerance but introduces overhead not justified at current scale.</p><p><strong>Eligibility changes require a service restart.</strong> Agent authorization is resolved at startup. Eligibility changes are rare and restarts are low-cost. An empty approved agent list produces a loud, safe failure rather than a silent one.</p><h3>What We Would Build Next</h3><p><strong>Event-driven state updates</strong>: if the platform moved to a cloud-hosted deployment, or if shorter migrations made five-minute staleness a limitation, an EventBridge-based model would be worth revisiting (see the Part 1 comparison). For the current on-premises architecture and workload profile, polling remains simpler and more cost-effective.</p><p><strong>Richer agent health signals.</strong> A deeper signal beyond online/offline, including last confirmed connection time and authoritative state transitions, would enable smarter selection under degraded fleet conditions.</p><p><strong>Size-proportional scoring:</strong> the current selection score is already aware of load and agent state, and adding estimated transfer size would enable size-based placement. This would steer petabyte-scale transfers to faster agents and route smaller tables to lightly loaded agents, increasing overall fleet throughput beyond what recency-based fairness alone can achieve.</p><h3>The Broader Design Principle</h3><p>The AWS DataSync Agent Manager is at its core a state-driven scheduling system built on a polling loop, a state table, and a selection query. Without it, agent availability would have existed only as a static ARN embedded in each task, with no cross-fleet record. Diagnosing a scheduling problem would require manually correlating task creation history, agent activity, and AWS API state. With it, agent fleet state is always readable from a single authoritative source, scheduling decisions are grounded in current reality, and failures surface as data patterns rather than ambiguous symptoms.</p><p>The design pattern generalizes. Wherever a platform makes decisions based on the state of an external system that does not push events, a purpose-built state cache collecting in the background and serving decisions from local state is the right abstraction. The consequential choices are in the boundaries: which component owns each field, how fetch and write are separated, and how the system behaves when it restarts mid-operation.</p><p><strong>Observability is not an operational convenience. It is a structural precondition for reliable decision-making at scale.</strong></p><p><strong>Key Takeaway:</strong> Three components with strict ownership boundaries: an authorization reconciler, a background poller with explicit fetch and write separation, and a subnet-scoped selection service. Together they form an AWS DataSync agent scheduling system that is observable, deadlock-safe, and resilient to partial failures.</p><h3>Further Reading</h3><ul><li><a href="https://medium.com/@k.sidharth/choosing-the-right-aws-datasync-agent-for-every-task-at-petabyte-scale-part-1-fd9cd1b2f7bf">Choosing the Right AWS DataSync Agent for Every Task at Petabyte Scale (Part-1)</a></li><li><a href="https://docs.aws.amazon.com/datasync/latest/userguide/configure-agent.html">Deploying your AWS DataSync agent</a></li><li><a href="https://iceberg.apache.org/docs/latest/">Introduction - Apache Iceberg™</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f1ff8d28183" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Choosing the Right AWS DataSync Agent for Every Task at Petabyte Scale (Part-1)]]></title>
            <link>https://medium.com/towards-data-engineering/choosing-the-right-aws-datasync-agent-for-every-task-at-petabyte-scale-part-1-fd9cd1b2f7bf?source=rss-63f18bcc00c9------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd9cd1b2f7bf</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[hybrid-cloud]]></category>
            <category><![CDATA[cloud-migration]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[aws-datasync]]></category>
            <dc:creator><![CDATA[Sidharth  Kumar]]></dc:creator>
            <pubDate>Thu, 13 Aug 2026 10:39:06 GMT</pubDate>
            <atom:updated>2026-08-14T17:42:11.652Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>With AWS DataSync, the agent you pick for a task is locked in for good. At petabyte scale that quietly becomes a scheduling problem, here’s how we fixed it by tracking agent state.</em></h4><h3>Platform and the Design Challenge</h3><p>We built a large-scale enterprise data migration platform to handle end-to-end cloud migration and lakehouse transformation at petabyte scale. The platform runs on-premises, ingesting data from on-premises sources including Hive/HDFS clusters and on-prem S3-compatible object storage, migrating it to Cloud, and transforming it into Iceberg-format lakehouse tables through automated, multi-stage pipeline orchestration.</p><p>The data copy stage runs through AWS DataSync, which by design requires the agent ARN to be specified at task creation time. The agent is bound statically to the task at creation, not at execution.</p><p>The initial recommendation was to use fixed source and destination staging locations with agents assigned in round-robin order, where data would be copied to staging first and then moved to the final target path. While this avoided dynamic DataSync resource creation, it undermined the platform’s goal of on-demand, table-specific transfers, added extra cloud-side copy cost, and limited scheduling flexibility.</p><p>With data sizes ranging from gigabytes to petabytes, a fixed set of pre-created tasks would overuse a few pre-bound agents while leaving much of the fleet idle, resulting in skewed utilization, inefficient throughput, and unpredictable migration performance.</p><p>Fixed DataSync tasks with round-robin agent selection work adequately when tasks are few, long-lived, and statically defined, such as a nightly database backup or a recurring NFS sync where each task is created once and reused. In those scenarios, agent assignment is a one-time setup decision rather than a real-time scheduling concern.</p><p>Our platform operates under a fundamentally different model. Migration runs table by table, and each table is an independent DataSync task with its own source and destination locations created dynamically at the time of execution. There is no pre-existing task to reuse. Every migration triggers on-demand creation of source location, destination location, and task, all bound together in real-time.</p><p>At the scale of thousands of databases and thousands of tables per database, this means hundreds of agent ARN selection decisions happening concurrently across the fleet. Each decision is irreversible once made. Selecting an ARN from a static list under these conditions meant making every one of those decisions without any knowledge of which agents were available, how loaded each was, or when one had last received work. Because the ARN cannot be changed once a task is submitted, a poor selection has no recovery path other than scrapping the task and starting over.</p><p>At petabyte scale, the cost compounds quickly. A misrouted task does not fail quietly. The data copy stage stalls, downstream lakehouse transformation jobs wait, data freshness SLAs erode, and operators get pulled into investigation that should never have been necessary. Solving this at design time, before the first task was ever created, was the only approach that made sense.</p><h3>Why Static Agent Configuration Would Not Scale</h3><p>To understand why round-robin fails here, consider the shape of the workload. A single agent might be assigned a 100GB table that completes in under an hour, or a 5PB table that runs for several days. Both are valid migration tasks. Both produce exactly one active AWS DataSync task execution on the agent.</p><p>Round-robin scheduling treats these identically. It has no awareness of how long an agent will be occupied, how much data it is currently transferring, or whether it is even online. Under a static round-robin model, a new migration could be dispatched to an agent that has just been assigned a multi-day, multi-petabyte transfer, leaving it effectively unavailable for days, while a fully idle agent sits waiting. There is no correction mechanism. The dispatcher has no signal to update from.</p><p>Across hundreds of concurrent migration jobs with table sizes ranging from 100GB to 5PB, this variance is not an edge case. It is the norm. An uninformed round-robin dispatch would produce consistently poor fleet utilisation and unpredictable queuing delays across the entire pipeline.</p><p>During the design of the platform’s other scheduling components, two background polling loops were already planned: an <strong>infrastructure provisioning tracker</strong> and a <strong>transfer execution tracker</strong>. Neither addressed agent-level visibility. Neither answered which agents were online, how loaded each was, or which had been idle longest. The gap was clear: the platform needed a dedicated component to continuously observe AWS DataSync agent state so that the right ARN could be selected at task creation time, every time.</p><p><strong>An agent ARN bound at task creation reflects a point-in-time choice. Reliable AWS DataSync scheduling requires continuously observed state.</strong></p><h3>The Core Architectural Decision: Making Agent State Observable from Day One</h3><p>The design decision was made upfront to replace the static model with a state-driven approach: a database table, one row per agent, maintained by a dedicated background poller running every five minutes, so that all scheduling decisions read from current observed state with no dispatch-time calls to AWS DataSync.</p><p><strong>Why polling over events?</strong> AWS DataSync does publish agent state change events via Amazon EventBridge, which provides a near real-time alternative to polling. However, the deployment model of this platform makes the EventBridge approach structurally impractical, and the workload profile makes it unnecessary.</p><p>A further constraint makes this comparison decisive: the data migration platform runs on-premises. EventBridge events are published within AWS cloud infrastructure and cannot be consumed directly by an on-prem service. Bridging the gap requires a cloud-side intermediary, either Lambda pushing events to an on-prem endpoint or SQS with the on-prem service polling it, effectively adding complexity and cost to achieve what the existing polling approach already does natively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*qIZ7bl5tOClAjDF_NpW1WA.png" /><figcaption>Background Polling vs. EventBridge + Lambda/SQS</figcaption></figure><p>The manager is workload-aware in the operational sense: every selection reads each agent’s current load and state — idle, verifying, or busy — <br> and routes to the least-loaded eligible agent. Size-proportional routing is a planned extension covered in <strong>Part 2</strong>.</p><p>The data migration platform runs on premises. EventBridge events cannot reach an on-prem service without a cloud-side intermediary, which transforms a near-real-time notification into a multi-component bridge that introduces the same polling pattern at higher cost. With table sizes ranging from 100GB to 1–5PB, the shortest migration runs for tens of minutes and the longest spans days, making five-minute accuracy sufficient across the entire workload range.</p><p>Background polling requires only outbound HTTPS calls to the same AWS endpoints already in use for DataSync operations, with no new infrastructure, no additional costs, and no dependency on cloud-side event bridging.</p><p><strong>Why a five-minute window is sufficient.</strong> Given the workload profile of this platform, where the shortest table migration runs for tens of minutes and petabyte-scale transfers span multiple days, a five-minute staleness window is negligible. An agent’s load at minute zero is a reliable predictor of its load at minute five, and the natural rate of change in the system is far slower than the polling cadence.</p><h3>System Architecture: Three Components, One Central State Table</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*YRsWrevBk8KL6ho-AnMYjw.png" /><figcaption><em>Figure: DataSync Agent Manager component architecture showing the Authorization Reconciler, Background Poller, Agent State Table, and Selection Service, integrated into the Migration Orchestration Pipeline.</em></figcaption></figure><p>The<strong> Authorization Reconciler</strong> controls which agents are eligible. It runs once at startup and is the only component that writes the eligibility flag, ensuring a runtime AWS API failure can never accidentally de-authorize agents. The <strong>Background Poller</strong> keeps the state table current by separating the fetch phase (AWS API calls, no database connection held) from the write phase (two short transactions).</p><p>The <strong>Selection Service</strong> reads only from the table, applies a subnet filter and eligibility check, then works through a two-tier priority model to return the best available agent ARN. All three components are covered in depth in Part 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*PdcUuDlEfgYQoNM6ur814A.png" /><figcaption><em>Figure: Before and after — static round-robin dispatch with no state awareness versus state-driven selection reading from a continuously observed agent state table.</em></figcaption></figure><h3>What the Design Prevented by Construction</h3><p><strong>Scheduling latency was controlled by design.</strong> The multi-minute queuing delays that static round-robin would have produced, with one agent occupied by a multi-day transfer while other agents sat idle, were eliminated by design before reaching production.</p><p>The multi-minute queuing delays that static round-robin would have produced — an agent occupied by a multi-day transfer while idle agents waited were eliminated by construction, before reaching production.</p><p>Agent ARN selection resolves as a single database read.</p><p><strong>Fleet observability was available from day one.</strong> Agent health, task counts, and assignment history were inspectable in a single view throughout development and testing.</p><blockquote><strong>Key Takeaway:</strong> AWS DataSync binds the agent ARN at task creation with no mechanism for runtime reassignment. Building an observable, continuously polled agent state layer before the first task was ever created eliminated an entire class of scheduling failures before they could reach production.</blockquote><h3>What Comes Next</h3><p><strong>Part 2</strong> covers the internal design of all three components, how agent selection integrates as a gating step in the migration orchestration pipeline, the sentinel pattern for safe infrastructure provisioning under failure, the two independent lifecycle loops that keep the full migration observable, and the trade-offs accepted along the way.</p><p><strong><em>Continue to Part 2</em></strong><em>: </em><a href="https://medium.com/@k.sidharth/building-a-workload-aware-aws-datasync-agent-manager-how-the-scheduler-actually-works-part-2-7f1ff8d28183"><em>Building a State-Driven AWS DataSync Agent Manager, covering Component Architecture, Polling Patterns and Observability</em></a></p><h3>References</h3><ul><li><a href="https://docs.aws.amazon.com/datasync/latest/userguide/configure-agent.html">Deploying your AWS DataSync agent</a></li><li><a href="https://iceberg.apache.org/docs/latest/">Introduction - Apache Iceberg™</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd9cd1b2f7bf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/choosing-the-right-aws-datasync-agent-for-every-task-at-petabyte-scale-part-1-fd9cd1b2f7bf">Choosing the Right AWS DataSync Agent for Every Task at Petabyte Scale (Part-1)</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Iceberg vs Hudi vs Delta: Why There’s No One-Size-Fits-All Table Format]]></title>
            <link>https://medium.com/@k.sidharth/iceberg-vs-hudi-vs-delta-why-theres-no-one-size-fits-all-table-format-03349782d82d?source=rss-63f18bcc00c9------2</link>
            <guid isPermaLink="false">https://medium.com/p/03349782d82d</guid>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[apache-hudi]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <dc:creator><![CDATA[Sidharth  Kumar]]></dc:creator>
            <pubDate>Sat, 21 Feb 2026 17:02:04 GMT</pubDate>
            <atom:updated>2026-02-24T03:27:28.709Z</atom:updated>
            <cc:license>https://creativecommons.org/publicdomain/mark/1.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*czBlrXMiGqiHArL8w4o_XQ.png" /></figure><p>When building petabyte-scale data platforms today, the operational challenges are well-known. Platforms usually start with a pristine, whiteboard-perfect architecture. Everything looks elegant in the design doc, but as data volume and velocity explode, the “every day it gets a little slower” problem inevitably emerges. Small-file proliferation chokes storage layers, metadata bottlenecks stall query planners, and simple backfills become weekend-ruining events.</p><p>To understand why the modern data stack is currently locked in a holy war over open table formats — and why standardizing on just <em>one</em> may not be the best idea.</p><h3>The Problem: From Hadoop Hangovers to the Lakehouse</h3><p>Rewinding a bit. Twenty years ago, the traditional Data Warehouse (think Teradata or Netezza) was king. It gave us highly structured, ACID-compliant tables, but it was incredibly rigid and egregiously expensive to scale.</p><p>Then came the Big Data boom. The data platform ecosystem entered a massive honeymoon phase with Apache Hadoop, happily dumping petabytes of raw, unstructured data into HDFS using cheap commodity hardware.</p><p>But over the decades, it outgrew it. Why? Because the core concept of <strong>data locality </strong>keeping compute right next to the physical disk which was Hadoop’s greatest strength, eventually became a liability as it tightly coupled compute and storage. With datacenter network speeds hitting 10–100Gbps, the need to keep processing close to the data evaporated. This rendered the coupled model unnecessary and drove the shift to decoupled storage and compute, enabling the flexible, elastic scaling we rely on today.</p><p>Without database-like ACID transactions or strict metadata governance, the pristine Data Lake quickly stagnated into an unmanageable data swamp.</p><p>Enter the <strong>Data Lakehouse</strong>: decoupled cloud storage paired with a smart metadata layer to bring transactional safety to raw Parquet files. But this sparked a new architectural dilemma: <em>which open table format should actually govern that metadata?</em></p><h3>Living with Hive: The Data Engineer’s nightmare</h3><p>Before evaluating the “Big Three”, it is important to acknowledge why the industry abandoning its go-to choice: Apache Hive. Apache Hive was the undisputed workhorse that brought SQL to the masses, but keeping it stable at production scale was/is an exercise in pure anxiety. If you’ve been there, these war stories will sound familiar:</p><p><strong>Scenario 1: The “3 AM Backfill” Nightmare</strong></p><ul><li><strong>The Old Way (Hive):</strong> An upstream bug corrupts yesterday’s revenue data. You kick off INSERT OVERWRITE to backfill the T-1 partition. At 3:15 AM, the leadership team’s automated dashboard queries that exact partition. Because Hive lacks transaction isolation, the dashboard reads the partition <em>while</em> it is being overwritten, returning half the data. Cue the pager alert.</li><li><strong>The New Way (Iceberg):</strong> Iceberg implements Serializable Isolation. You run the backfill, and Iceberg writes the new files in the background. The Leadership’s dashboard perfectly reads the <em>old</em> snapshot. Once the job finishes, Iceberg updates a single metadata pointer. The next query instantly sees the corrected data. Zero locking, zero 3 AM alerts.</li></ul><p><strong>Scenario 2: The “S3 Throttling” Wall</strong></p><ul><li><strong>The Old Way (Hive):</strong> When a query engine like Spark or Trino plans a query on a massive table, it has to execute s3:ListObjects to recursively find the physical files. As the table hits millions of partitions, AWS throttles requests with 503 Slow Down errors. Query <em>planning</em> alone takes 15 minutes before a single byte of data is read.</li><li><strong>The New Way (Iceberg):</strong> Iceberg abandons directory listing entirely. It uses a manifest file that explicitly tracks the exact S3 URI of every single data file. The engine reads one JSON file, bypasses S3 LIST operations entirely, and plans the query in milliseconds.</li></ul><p><strong>Scenario 3: The “Schema Drift” Panic</strong></p><ul><li><strong>The Old Way (Hive):</strong> A data steward decides user_id should actually be named account_id and pushes to production. Hive relies on strict file-level column names. Downstream pipelines crashes, historical data becomes unreadable, and you are forced into a massive table rewrite.</li><li><strong>The New Way (Iceberg):</strong> Iceberg tracks columns using unique, immutable IDs (e.g., ID: 4). Renaming a column simply updates the string alias mapped to ID 4. The underlying Parquet files are untouched, and old data gracefully reconciles with new data.</li></ul><h3>📌 The “Cheat Sheet”</h3><p>Evaluating these formats based on simple feature checklists is a trap, as all three now support core Lakehouse requirements. Based on their current releases as I am writing, here is how they compare:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WZSBPmWuAl6MFC3f4PZYWQ.png" /></figure><p>Note: For detailed feature comparison, please refer <a href="https://www.onehouse.ai/blog/apache-hudi-vs-delta-lake-vs-apache-iceberg-lakehouse-feature-comparison">Apache Iceberg vs Delta Lake vs Apache Hudi — Feature Comparison Deep Dive</a></p><h3>The Myth of the “Universal Standard” (Facts over Preferences)</h3><p>One frequent misstep is over‑standardizing on a single open table format at the enterprise level, without accounting for workload diversity</p><p>There is no universal standard because the physics of distributed systems demand different trade-offs and you must choose your trade-offs.</p><p><strong>1.Write-Heavy Streaming Problem (Enter Hudi/Delta): </strong>If a massive CDC pipeline is receiving thousands of updates per minute, a pure batch approach cannot be used. If a 100MB Parquet file is rewritten every time a single row changes (Write Amplification), Spark cluster will melt down and cloud bill will skyrocket.</p><blockquote><strong>Solution</strong>: Hudi and Delta solve this by <strong>only writing the changes</strong>. Using <strong>Merge-on-Read (MoR)</strong> or <strong>Deletion Vectors</strong>, they append tiny log files noting “Row X was updated/deleted.” The query engine reconciles the logs with the base file on the fly. The write is optimized at the slight expense of the read.</blockquote><p><strong>2. The Read-Heavy Batch Problem (Enter Iceberg): </strong>Conversely, when running massive end-of-day financial rollups or petabyte-scale ML training sets, the compute engine should not waste CPU cycles reconciling tiny log files on the fly. Blazing-fast, sequential read speeds are required.</p><blockquote><strong>Solution</strong>: Iceberg is inherently optimized for massive analytical scans. It pays the upfront compute cost of rewriting fully compacted, pristine Parquet files via <strong>Copy-on-Write (CoW)</strong> during ingestion. This guarantees that downstream batch jobs can scan billions of rows with zero overhead.</blockquote><p><strong>3. The Nightmare Scenario: Both Workloads, Same Table: </strong>Now here is the problem rarely discussed in architecture reviews; what happens when there are both high-frequency streaming writes <em>and</em> heavy batch reads hitting the <em>same</em> table simultaneously?</p><p>This is the classic <strong>read-write amplification trap</strong>, and it is where most “one format for everything” strategies completely fall apart.</p><p><strong><em>Picture this</em></strong>: Your real-time Flink CDC pipeline is continuously writing thousands of order update events per minute into a transactions table using Hudi&#39;s Merge-on-Read(<strong><em>MoR</em></strong>). Throughout the day, small Avro log files are stacking up on top of your base Parquet files like a growing pile of sticky notes. Each streaming write is fast and cheap, exactly as designed<em>.</em></p><p>Then at 11 PM, your end-of-month financial batch job kicks in and attempts to compute revenue rollups for the past 30 days across 500 million rows. Now your Spark executors are not just reading clean Parquet files, they are physically reconciling thousands of stacked log files on the fly for every single partition before they can even begin the aggregation. Your batch job that used to run in 40 minutes now takes 4 hours.</p><p><strong><em>Solution</em>: Async Compaction as a First-Class Citizen</strong></p><p>The solution is not to pick a different format — it is to treat <strong>compaction as an independent, always-running background service</strong> alongside your ingestion pipeline.</p><ul><li><strong>In Hudi:</strong> Run async compaction as a parallel Spark job that continuously merges log files into clean base Parquet files without blocking your streaming writers.​</li><li><strong>In Delta Lake:</strong> Schedule the OPTIMIZE command with ZORDER BY on your most-queried columns on a regular cadence. Delta 4.0&#39;s <strong>Predictive Optimization</strong> can now trigger this automatically based on table activity.​</li><li><strong>In Iceberg:</strong> Use the <strong>Rewrite Data Files</strong> action in Iceberg’s Table Maintenance API to compact small files in the background, leveraging its snapshot isolation to guarantee zero read downtime during the process.</li></ul><p>Think of it like garbage collection in a JVM, mandatory, continuous, and invisible when done right. You do not stop your application to run GC — it runs asynchronously, invisibly, in the background. Compaction on mixed-workload tables should be treated exactly the same way: a continuous, automated hygiene process, not a manual weekend job.​</p><blockquote><strong>The Golden Rule for Mixed Workloads</strong>: If the same table is being written by a streaming pipeline AND read by a batch job, your compaction frequency should directly track your streaming write volume. High write throughput → more aggressive compaction schedule → consistently fast batch reads. Treat it as a knob you tune, not a task you schedule once and forget.</blockquote><h3>When to Choose What?</h3><p>Because there is no silver bullet, you have to map the format to your platform’s operational center of gravity.</p><ul><li><strong>Choose Hudi if:</strong> You rely heavily on CDC pipelines, process a massive volume of out-of-order updates, and require near real-time data freshness. You need the engine to absorb write-heavy punishment.</li><li><strong>Choose Iceberg if:</strong> You prioritize absolute compute-engine neutrality (Spark + Snowflake + Trino) and manage massive, petabyte-scale analytical tables where fast batch reads and fool-proof schema evolution are mandatory.</li><li><strong>Choose Delta Lake if:</strong> Your center of gravity is Databricks or Microsoft Fabric, and you want raw read performance with out-of-the-box optimizations (like Liquid Clustering) without the burden of manual tuning.</li></ul><h3>The “Omni-Format” Future: Why the Table Format may not matter Soon</h3><p>For years, organizations lived in fear of vendor lock-in. Data written in Delta couldn’t easily be read by an Iceberg engine.</p><p>The reality today? <strong>The format wars are ending</strong> — not because one format won, but because interoperability did.</p><p>Projects like <strong>Apache XTable</strong> and <strong>Delta Lake UniForm</strong> are providing omnidirectional translation. They automatically convert metadata so data written in one format can be seamlessly read as another, entirely without duplicating the underlying Parquet files.</p><blockquote><strong>The AI Implication:</strong> Data engineers ingest clickstream data continuously using Delta Lake via Databricks. However, your ML engineers use a PyTorch training pipeline that relies on Iceberg readers for batch efficiency. Delta UniForm automatically generates Iceberg metadata alongside the Delta logs. The ML team queries the table natively as Iceberg. Zero friction, zero duplicated data.<em>​</em></blockquote><h3>Final thoughts:</h3><p>One common pitfall I have seen repeatedly when helping enterprises migrate off legacy Hadoop clusters is treating the table format decision like permanent infrastructure. It is not.</p><p>The framework I lean on is straightforward:</p><ul><li><strong>CDC-heavy streaming pipelines?</strong> → Hudi or Delta. Their MoR mechanics absorb write-heavy punishment without destroying storage layouts.</li><li><strong>Petabyte-scale analytical reads?</strong> → Iceberg. Clean CoW files, predictable query planning, and deep native cloud integrations make it the path of least resistance.</li><li><strong>Both on the same table?</strong> → Compaction is not optional. Treat it as a tier-1 background service or watch query SLAs silently collapse.</li></ul><p>Deploy XTable or UniForm early — even when it feels premature. The cost is near zero; the optionality is enormous.</p><p>Finally, with format interoperability becoming invisible, arguing Iceberg vs. Delta eventually will feel like debating TCP/IP. But, the real architectural decision is moving or has moved towards <strong>Catalog, </strong>whoever controls the metadata, RBAC, and lineage controls the platform. Keep that layer open.</p><h4>References:</h4><ul><li><a href="https://www.onehouse.ai/blog/apache-hudi-vs-delta-lake-vs-apache-iceberg-lakehouse-feature-comparison">Apache Iceberg™ vs Delta Lake vs Apache Hudi</a></li><li><a href="https://docs.delta.io/delta-uniform/">Universal Format (UniForm)</a></li><li><a href="https://www.onehouse.ai/blog/open-data-foundations-with-apache-xtable-hudi-delta-and-iceberg-interoperability">Open Data Foundations with Apache XTable</a></li><li><a href="https://aws.amazon.com/blogs/big-data/run-apache-xtable-in-aws-lambda-for-background-conversion-of-open-table-formats/">Run Apache XTable in AWS Lambda for background conversion</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=03349782d82d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spark Result Fragment Caching: Solving the “Every Day It Gets Slower” Problem on AWS EMR]]></title>
            <link>https://aws.plainenglish.io/spark-result-fragment-caching-solving-the-every-day-it-gets-slower-problem-on-aws-emr-947810650b47?source=rss-63f18bcc00c9------2</link>
            <guid isPermaLink="false">https://medium.com/p/947810650b47</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[amazon-emr]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Sidharth  Kumar]]></dc:creator>
            <pubDate>Wed, 07 Jan 2026 12:14:33 GMT</pubDate>
            <atom:updated>2026-01-07T13:30:40.407Z</atom:updated>
            <cc:license>https://creativecommons.org/publicdomain/mark/1.0/</cc:license>
            <content:encoded><![CDATA[<h3>Spark Result Fragment Caching: Solving the “Every Day It Gets Slower” Problem on Amazon EMR</h3><p><em>How RFC Transforms Cumulative Workloads, Accelerates Analytics, and Slashes Compute Costs</em></p><p>Have you ever shipped a year-to-date revenue report in Spark that starts out snappy, but by month six, it drags for several minutes — and by year-end, you’re staring at a 30-minute SLA risk just to keep your dashboard alive?</p><p>This “Every Day It Gets Slower” problem plagues analytics everywhere. The business logic stays the same, but every day your Spark jobs scan all historical partitions and recompute the same aggregates. The culprit? Data keeps growing, and most of it never changes , yet Spark treats each run as brand new, making you pay the full compute price over and over.</p><ul><li>YTD financial and sales reporting</li><li>Rolling 30-day fraud or anomaly detection</li><li>Compliance “ever seen” / “ever flagged” queries</li><li>Cumulative audit and risk dashboards</li></ul><p>Most of this data is immutable. Yesterday’s transactions rarely change, but Spark’s traditional caching falls short for these scheduled, cumulative workloads. Solutions like df.cache() are cluster-local and ephemeral, result set caching demands exact query matches, and materialized views add complexity. Enter <a href="https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-fragment-result-caching.html"><strong><em>Spark Result Fragment Caching</em></strong></a> (RFC) on AWS EMR — a purpose-built optimizer that caches expensive sub-results on S3 and reuses them across runs and clusters.</p><h3>What Is Spark Result Fragment Caching?</h3><p>Spark RFC is an EMR-specific optimization (available since Amazon EMR 6.6.0) that analyzes your Spark query plan, identifies subtrees (fragments) with high reduction ratios, and writes those results to S3. On future runs, Spark skips re-computation for those fragments and reads them from the S3 cache instead.</p><ul><li><strong>Fragment</strong>: A subtree of the query plan that can be safely cached and reused.</li><li><strong>Reduction ratio</strong>: How much a query step shrinks the data (default candidate: ratio ≥ 8:1).</li></ul><pre>Reduction Ratio=Output Size / Input Size</pre><ul><li><strong>S3 placement</strong>: Cached fragments are written to a user-specified S3 bucket and shared across clusters running the same EMR release.</li></ul><h4>How does RFC differ from classic caching?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gj_-Ptby-ikNQewEhP05iA.png" /></figure><h4>How Spark RFC Works</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fiII9D8zwgxXs9Sq4HxiLg.png" /></figure><p>1. <strong>Plan Analysis</strong>: EMR inspects the optimized Spark plan for subtrees reading Parquet data and applying filters/aggregations.</p><p>2. <strong>Fragment Detection</strong>: For each candidate subtree, RFC estimates input vs output size per RDD partition. If the reduction ratio meets the threshold (default 8.0), it’s cacheable.</p><p>3. <strong>S3 Write</strong>: Cacheable partitions are buffered (default 16MB, max 64MB) and written to S3.</p><p>4. <strong>Subsequent Runs</strong>: Cached fragments are validated (schema/version, freshness), and compute steps are replaced with S3 reads. Invalid/missing fragments trigger re-computation and optional cache refresh.</p><p>5. <strong>Safety</strong>: RFC never uses fragments from different EMR releases and disables itself for Spark Streaming, RecordServer, Ranger, or Lake Formation-governed tables.</p><p><em>Key configs (in </em><strong><em>spark-defaults.conf</em></strong><em>):</em></p><pre>spark.subResultCache.enabled=true #Enable/disable RFC<br>spark.subResultCache.fs.root.path=s3://my-rfc-cache-bucket/cache_dir/ #S3 root path for cached fragments<br>spark.sql.subResultCache.reductionRatioThreshold=8.0 #Reduction ratio threshold (default 8.0)<br>spark.sql.subResultCache.maxBufferSize=16777216 #Per-partition write buffer (default 16MB, max 64MB)</pre><blockquote><strong>Tip</strong>: Treat reductionRatioThreshold as your main “is this worth caching?” control.</blockquote><h4>Where RFC Delivers Maximum Value: 4 Key Use Cases</h4><h4>1. Rolling Window Analytics (e.g., 30-Day Fraud Detection)</h4><p>Daily jobs recompute a 30-day window, but with RFC, only new data gets processed each day — previous results are reused. Internal AWS benchmarks show up to 15.6× speedups for these workloads.</p><h4>2. Year-to-Date / Cumulative Reporting</h4><p>YTD queries normally rescan all historical data. RFC reuses cached daily/monthly aggregates, processing only new data. This is ideal for append-only, high-reduction, stable queries.</p><h4>3. ETL Jobs With High Reduction Ratios</h4><p>Transforming 500GB of raw logs into 10GB of metrics? RFC caches the downstream result, avoiding rescans for backfills or analytic queries. Best fit: log aggregation, clickstream, IoT event summarization.</p><h4>4. Compliance &amp; Audit Reporting</h4><p>Cumulative queries (“all customers ever flagged”) get stable SLAs as RFC maintains cached sets and merges incremental changes — critical for audit and regulatory workloads.</p><h4>How to Set Up RFC on EMR</h4><h4>Prerequisites</h4><ul><li>EMR version 6.6.0 or higher</li><li>Input data must be Parquet (not CSV/JSON/ORC)</li><li>File splits ≥ 128MB</li><li>Dedicated S3 bucket with lifecycle management</li></ul><h4>Step-by-Step Guide</h4><p>1. Create S3 Cache Bucket — enable versioning and configure lifecycle policy (e.g., 30-day expiration).</p><pre>ACCOUNT_ID=123456789012<br>REGION=us-east-1<br>BUCKET=&quot;emr-rfc-cache-${ACCOUNT_ID}-${REGION}&quot;<br>aws s3 mb s3://${BUCKET}<br><br># (Recommended) Enable versioning for audit and rollback<br>aws s3api put-bucket-versioning \<br> --bucket ${BUCKET} \<br> --versioning-configuration Status=Enabled</pre><p>2. Configure EMR Spark Defaults — add RFC settings to your EMR config (<strong>rfc-config.json</strong>).</p><pre># cat rfc-config.json:<br>{<br>  &quot;Classifications&quot;: [<br>    {<br>      &quot;Classification&quot;: &quot;spark-defaults&quot;,<br>      &quot;ConfigurationProperties&quot;: {<br>        &quot;spark.subResultCache.enabled&quot;: &quot;true&quot;,<br>        &quot;spark.subResultCache.fs.root.path&quot;: &quot;s3://emr-rfc-cache-123456789012-us-east-1/&quot;,<br>        &quot;spark.sql.subResultCache.reductionRatioThreshold&quot;: &quot;8.0&quot;,<br>        &quot;spark.sql.subResultCache.maxBufferSize&quot;: &quot;16777216&quot;<br>      }<br>    }<br>  ]<br>}<br># To Launch EMR Cluster<br>aws emr create-cluster \<br>  --name &quot;emr-rfc-demo&quot; \<br>  --release-label emr-6.10.0 \<br>  --applications Name=Spark Name=Hadoop \<br>  --instance-type m5.xlarge \<br>  --instance-count 10 \<br>  --configurations file://rfc-config.json</pre><p>3. Align Parquet &amp; Partitioning , use large Parquet files (128–512MB) and avoid partition skew.</p><blockquote><strong>Pro tip:</strong> Convert legacy CSV/JSON pipelines to Parquet for RFC and better baseline performance.</blockquote><h4>Cons and Mitigation Strategies</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ksU45RYY8KXK_yJbmwhIIA.png" /></figure><blockquote><strong>Mitigation tips:</strong> Use RFC for jobs that run at least 5+ times on the same window. Monitor S3 costs; standardize on Parquet; fix partition skew.</blockquote><h4>Performance Analysis &amp; Benchmarks</h4><p>From AWS’s own TPC-H–derived benchmarks, RFC has delivered speedups up to ~15× for rolling/incremental workloads.</p><ul><li>Day 1: 45 minutes (full cache warm-up)</li><li>Day 30: 2.5 minutes (only new data processed, rest reused from cache)</li><li>Break-even: Around day 5–7 for large clusters</li></ul><p>RFC is net positive for large clusters (≥ 10 nodes), heavy rolling/YTD jobs, and high reduction ratios (≥ 8:1).</p><h4>Critical Constraints &amp; Gotchas</h4><ul><li>Parquet only — RFC silently does nothing for non-Parquet data</li><li>File splits ≥ 128MB — smaller splits disable RFC</li><li>No Spark Streaming support</li><li>Disabled for Ranger, RecordServer, Lake Formation</li><li>Cache reuse is EMR-release bound</li></ul><blockquote><strong>Common issues: </strong>Zero speedup usually means non-Parquet input or reduction ratio below threshold. Cache growth without benefit? Likely low cache hit rate or unstable query shapes.</blockquote><h4>When Should You Use RFC?</h4><ul><li>Does the query run repeatedly? Yes → continue</li><li>Is the source Parquet with large splits? Yes → continue</li><li>High reduction ratios (≥8:1) or expensive compute? Yes → continue</li><li>Mostly static historical data reused? Yes → continue</li><li>Large cluster or expensive job? Yes → RFC is a good fit</li></ul><blockquote><strong>Use RFC for:</strong> Recurring queries, large data volumes, high reduction ratios, stable schemas, and non-governed clusters.</blockquote><blockquote><strong>Avoid RFC for:</strong> One-off analyses, streaming, small data, low reduction ratios, or governed paths where RFC is disabled.</blockquote><h4>Implementation Walkthrough: YTD Sales Dashboard</h4><ul><li>Audit Data and Estimate Reduction Ratio</li></ul><pre>from pyspark.sql import functions as F<br>df = spark.read.parquet(&quot;s3://data-lake/sales/&quot;)<br><br># Rough input size estimate (bytes) using catalog/metadata or custom stats table<br>raw_size_bytes = spark.table(&quot;sales_metadata&quot;) \<br>    .agg(F.sum(&quot;byte_size&quot;).alias(&quot;bytes&quot;)) \<br>    .collect()[0][&quot;bytes&quot;]<br><br># Example aggregation<br>agg_df = df.filter(F.year(&quot;transaction_date&quot;) == 2025) \<br>    .groupBy(&quot;region&quot;, &quot;product_id&quot;) \<br>    .agg(F.sum(&quot;amount&quot;).alias(&quot;total_revenue&quot;))<br><br># Estimate output cardinality and approximate size<br>approx_rows = agg_df.count()<br># Suppose we approximate 100 bytes/row for this schema:<br>result_size_bytes = approx_rows * 100<br><br>ratio = raw_size_bytes / max(result_size_bytes, 1)<br>print(f&quot;Approx reduction ratio: {ratio:.1f}:1&quot;)</pre><ul><li>Create dedicated bucket and apply a 30-day expiration policy.</li><li>Configure EMR Spark Defaults (rfc-config.json)</li></ul><pre>{<br>   &quot;Classifications&quot;: [<br>      {<br>        &quot;Classification&quot;: &quot;spark-defaults&quot;,<br>        &quot;ConfigurationProperties&quot;: {<br>        &quot;spark.subResultCache.enabled&quot;: &quot;true&quot;,<br>        &quot;spark.subResultCache.fs.root.path&quot;: &quot;s3://emr-rfc-cache-123456789012-us-east-1/&quot;,<br>        &quot;spark.sql.subResultCache.reductionRatioThreshold&quot;: &quot;8.0&quot;,<br>        &quot;spark.sql.subResultCache.maxBufferSize&quot;: &quot;16777216&quot;<br>        }<br>      }<br>   ]<br>}</pre><ul><li>Implement YTD Job</li></ul><pre>import pyspark.sql.functions as F<br><br>sales = spark.read.parquet(&quot;s3://data-lake/sales/&quot;)<br>ytd_2025 = (<br>    sales<br>    .filter(F.year(&quot;transaction_date&quot;) == 2025)<br>    .groupBy(&quot;region&quot;, &quot;product_id&quot;)<br>    .agg(F.sum(&quot;amount&quot;).alias(&quot;total_revenue&quot;))<br>)<br>ytd_2025.write.mode(&quot;overwrite&quot;) \<br>    .parquet(&quot;s3://dashboards/ytd-sales/2025/&quot;)</pre><ul><li>No code changes are required for RFC — just configuration, the runtime decides which fragments to cache and reuse.</li><li>Validate Improvements</li><li>Compare runtimes and costs with/without RFC</li><li>Monitor S3 cache size and compute savings</li></ul><h4>Comparing RFC With Alternatives</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eWismONWEP-lohdE8HE1sA.png" /></figure><p>RFC doesn’t replace these options — it uniquely accelerates recurring, cumulative Spark jobs on EMR.</p><h3>Conclusion &amp; Next Steps</h3><p>Spark Result Fragment Caching may look small on paper, but it can reshape your analytics cost curve for the right workloads. It excels at recurring, cumulative, and windowed queries over immutable Parquet data making query time behave like O(1) no matter how much history you have.</p><ul><li>Identify heavy recurring queries on Parquet with large partitions</li><li>Set up an RFC cache bucket with lifecycle management</li><li>Enable RFC on a non-production cluster and benchmark performance</li><li>If results are positive, automate RFC configs and cache management</li><li>Don’t “just turn it on everywhere” — profile and optimize for the right jobs</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=947810650b47" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/spark-result-fragment-caching-solving-the-every-day-it-gets-slower-problem-on-aws-emr-947810650b47">Spark Result Fragment Caching: Solving the “Every Day It Gets Slower” Problem on AWS EMR</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>