<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vincent DANIEL on Medium]]></title>
        <description><![CDATA[Stories by Vincent DANIEL on Medium]]></description>
        <link>https://medium.com/@vincent_daniel?source=rss-f736f50ce5d2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*w8DUSwr6ZboLdo7p.</url>
            <title>Stories by Vincent DANIEL on Medium</title>
            <link>https://medium.com/@vincent_daniel?source=rss-f736f50ce5d2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:45:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vincent_daniel/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How I Taught an AI to Read My Logs]]></title>
            <link>https://medium.com/expedia-group-tech/how-i-taught-an-ai-to-read-my-logs-9c04f1cfa750?source=rss-f736f50ce5d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c04f1cfa750</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Vincent DANIEL]]></dc:creator>
            <pubDate>Tue, 10 Nov 2026 12:01:02 GMT</pubDate>
            <atom:updated>2026-11-10T12:01:02.199Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/expedia-group-tech/subpage/data">Expedia Group Technology - Data</a></h4><h4><em>AI-assisted data engineering at Expedia Group — and less squinting at Spark execution plans</em></h4><figure><img alt="A bridge in Queensland, Australia at night" src="https://cdn-images-1.medium.com/max/1024/0*_5DV4iBoB3hVucxE" /><figcaption>Photo by Queensland, Australia on Unsplash</figcaption></figure><p>Anyone who has actually opened a Spark execution plan to figure out why a job is slow or expensive knows what the pain looks like. It’s not one screen — it’s the Spark UI’s stage DAG, then the SQL tab’s physical plan with nested Exchange, BroadcastHashJoin, and SortMergeJoin operators stacked five levels deep, then a separate executors tab to check whether any of that actually correlates with memory pressure, then cross-referencing timestamps against a job&#39;s config history to rule out &quot;did someone change something last week.&quot; Reading a plan correctly means holding all of that in your head at once, and most of it is stored in different tabs that don&#39;t talk to each other. Doing this well takes real practice, and doing it under time pressure — a pipeline failed overnight, someone needs an answer before standup — is exactly when people take the fastest wrong guess instead of the right slow one.</p><p>This is the part of the job that AI turned out to be genuinely good at, and it’s the part of the last two years worth focusing on.</p><h3>A brief origin point: Iceberg V3</h3><p>The first place this paid off concretely was migrating tables to Apache Iceberg’s V3 format after it shipped in mid-2025. The work itself was conversational — reading a table’s delete-file accumulation, reading the actual scan plan against that table, deciding whether a merge-on-read migration to deletion vectors was worth it, then checking the resulting plan again to confirm it worked. Daily compute cost on a couple of pipelines that got this treatment dropped by more than 90%. That’s a good result, but the more important thing it proved was narrower: that reading a real execution plan and matching it against real table metadata, rather than trusting what the DDL or the job config claimed, was something an AI could do reliably enough to act on. (The deletion-vector mechanics themselves are covered in more depth separately: <a href="https://medium.com/@vincent_daniel/deletion-vectors-and-puffin-files-merge-in-the-new-v3-iceberg-format-565188036d0c">Deletion Vectors and Puffin Files: MERGE in the New V3 Iceberg Format</a>.) That one-off, table-by-table exercise is what eventually got packaged into a repeatable skill — but the skill’s value was never really about Iceberg. It was about not having to read the plan by hand anymore.</p><h3>Where this actually shows up: reading the execution plan instead of guessing from configuration</h3><p>Two skills now do this reading for us — one after a job has finished, one while it’s still running — and both are built around the same premise: the plan and the execution trace tell you what happened; the config and the code only tell you what someone intended.</p><p><strong>The cost-analysis skill reads the plan after the fact, on every expensive job.</strong> For the two or three most costly jobs in a pipeline, it pulls the actual stage summary (shuffle and spill volumes, task skew), the actual executor summary (peak memory against what was allocated), and the literal SQL execution plan the engine produced — the same tabs a human would have to cross-reference by hand, done automatically and correlated in seconds instead of the better part of an hour. That turns into diagnoses that are hard to make correctly by eye:</p><ul><li><strong>Skipped stages vs. wasted stages.</strong> A stage the plan skips is a cache hit doing its job, not idle compute — easy to misread as “this stage never ran” if you’re scanning the DAG quickly, and the skill is built to recognize the difference rather than flag it as an anomaly. It’s the same judgment call discussed separately in <a href="https://medium.com/@vincent_daniel/optimizing-cte-on-apache-spark-when-to-cache-and-when-not-to-0fe764766a63">Optimizing CTEs on Apache Spark: When to Cache and When Not To</a>: a repeatedly-rescanned CTE is real cost worth fixing, but reflexively caching everything isn’t free either, and telling the two apart from a plan requires actually tracing which stage re-executes the same subtree.</li><li><strong>Near-capacity vs. already spilling.</strong> Peak executor memory at 99% of a container with zero spill and the same peak with active spill look nearly identical on a summary chart — one is a warning for next week, the other is money being spent right now. Distinguishing them means reading the actual spill metric per stage, not the aggregate memory graph.</li><li><strong>Full-scan </strong><strong>MERGE INTO detection.</strong> This is the plan-reading catch I&#39;d put ahead of the others: the skill checks whether a merge statement&#39;s target-side condition is filtered the same way its source is. A plan with an unfiltered target ON clause against a windowed source looks completely unremarkable at a glance — it&#39;s the kind of thing a human reviewer skims past — but it forces a full-table scan every run, and where a WHEN NOT MATCHED BY SOURCE clause is present, it can delete rows outside the intended window. This was caught as active, ongoing data loss on more than one pipeline during a routine cost sweep, because the skill compared the plan&#39;s actual scan boundaries against what the source windowing implied instead of trusting that the code did what its comments said. More on why this specific plan shape is so costly on Iceberg tables specifically: <a href="https://medium.com/@vincent_daniel/boost-iceberg-performance-and-cut-compute-costs-with-well-scoped-merge-statements-e0a8f702c321">Boost Iceberg Performance and Cut Compute Costs with Well-Scoped MERGE Statements</a>.</li><li><strong>Driver-side vs. executor-side memory failures.</strong> Recurring out-of-memory kills are the textbook trigger for “give it more executor memory” — a reasonable guess from the error message alone. Reading the actual heap trace changes the diagnosis: if executor heap usage at the moment of failure was near zero, the failure is on the driver side (in one case, file-transfer buffering), and more executor memory would have done nothing.</li><li><strong>JVM heap pressure vs. off-heap memory pressure.</strong> A separate pipeline kept failing with the same OOM error after someone manually doubled executor memory — because in both configurations, JVM heap usage topped out at only 11–23% of what was allocated. The memory being exhausted was Python process memory outside the JVM, invisible to anyone reading the Spark memory config and assuming that’s where the ceiling was.</li><li><strong>Stale tables, caught through metadata rather than assumption.</strong> Not every finding comes from a Spark plan — some come from asking the catalog itself a question a human would rarely think to ask on a routine pass: when did this table actually last get written to? Through a Trino MCP connection, the skill queries a table’s Iceberg snapshot history directly — no ad hoc SQL session, no remembering the right metadata table name — and checks the most recent commit timestamp against the analysis window. A table with no write in the last 30 days but ongoing shared maintenance cost is pure waste hiding in plain sight, invisible unless something actually checks the snapshot log instead of assuming a table is “probably fine” because nobody’s complained. The same Trino MCP path is used to reconcile a repo’s DDL against what genuinely exists in the live catalog — turning up tables that were dropped in one environment but never removed from source, or tables the catalog thinks are current but haven’t been touched in months. Once a table is confirmed genuinely stale or corrupted, the lowcode deployment skill takes over: registering it cleanly as a trash table with the right ownership tags before dropping it, or, for a table stuck in a corrupted metadata state, running the unregister-and-re-register recovery sequence — again pulling the current state through the same metadata queries rather than guessing at what’s safe to touch.</li></ul><p>Reading a plan well enough to catch any one of these by eye is a real, learned skill — the kind senior engineers build over years of exactly this kind of incident. Having it done automatically, correctly, and consistently on every top-cost job is what turned troubleshooting from “hope the person on call has seen this exact failure before” into a repeatable ten-minute pass. The limits get documented rather than papered over: a pre-signed dashboard URL that’s already been used needs a fresh request, and a job older than roughly 30 days has no server-side trace left to read at all — a hard platform limit, not something worth retrying around.</p><p><strong>The lowcode deployment skill reads the plan while the job is still running — and, increasingly, before it’s even committed.</strong> Rather than deploying straight to production and waiting for a downstream alert to say something went wrong, it can run the same job as a live dry run first: executing the actual statement against a non-production table or branch (the same Write-Audit-Publish pattern behind Iceberg’s branch-based validation — see <a href="https://medium.com/expedia-group-tech/chill-your-data-with-iceberg-write-audit-publish-746c9eb3db48">Chill Your Data With Iceberg: Write-Audit-Publish</a>), watching the real plan and stage progress it produces, and confirming the change behaves as expected before anything touches production data. For a risky operation — an Iceberg format upgrade, a DDL change with ownership implications — that dry run is what turns “we think this is safe” into “we watched it run safely.” Once a deployment does go to production, the skill keeps watching: it polls the running job’s stage progress every ten seconds or so for the duration of the deployment, the same live equivalent of the plan a human would otherwise only get to inspect after the fact. That’s the difference between discovering a failure an hour later and starting a fresh debugging session, versus watching it fail in the same session it was deployed in and fixing it before moving on. The same live view — from the dry run or the real deployment — is what lets it pick the right recovery procedure — a corrupted table metadata state, a write that committed conflicting deletion vectors, a stale table confirmed through the catalog checks above — from the actual job trace rather than guessing from an error message.</p><h3>What actually changed</h3><p>Not the model getting smarter — the willingness to go read the plan instead of guessing from the config, every single time, on every job that matters, instead of only when someone finally has an hour free to dig in properly. That’s the same instinct that started with one engineer squinting at a scan plan to decide whether an Iceberg table was worth migrating, now running automatically across a whole portfolio of pipelines — querying the catalog directly through Trino MCP, dry-running risky changes before they touch production, reading real execution traces, and fixing what it finds rather than just reporting it. The dollar savings and the incident catches are both downstream of that one habit, not the point of it.</p><p><em>Related reading:</em></p><ul><li><a href="https://medium.com/@vincent_daniel/deletion-vectors-and-puffin-files-merge-in-the-new-v3-iceberg-format-565188036d0c"><em>Deletion Vectors and Puffin Files: MERGE in the New V3 Iceberg Format</em></a></li><li><a href="https://medium.com/@vincent_daniel/optimizing-cte-on-apache-spark-when-to-cache-and-when-not-to-0fe764766a63"><em>Optimizing CTEs on Apache Spark: When to Cache and When Not To</em></a></li><li><a href="https://medium.com/@vincent_daniel/boost-iceberg-performance-and-cut-compute-costs-with-well-scoped-merge-statements-e0a8f702c321"><em>Boost Iceberg Performance and Cut Compute Costs with Well-Scoped MERGE Statements</em></a></li><li><a href="https://medium.com/expedia-group-tech/chill-your-data-with-iceberg-write-audit-publish-746c9eb3db48"><em>Chill Your Data With Iceberg: Write-Audit-Publish</em></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c04f1cfa750" width="1" height="1" alt=""><hr><p><a href="https://medium.com/expedia-group-tech/how-i-taught-an-ai-to-read-my-logs-9c04f1cfa750">How I Taught an AI to Read My Logs</a> was originally published in <a href="https://medium.com/expedia-group-tech">Expedia Group Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing CTE on Apache Spark: When to Cache and When Not to]]></title>
            <link>https://medium.com/@vincent_daniel/optimizing-cte-on-apache-spark-when-to-cache-and-when-not-to-0fe764766a63?source=rss-f736f50ce5d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/0fe764766a63</guid>
            <category><![CDATA[caching]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Vincent DANIEL]]></dc:creator>
            <pubDate>Tue, 21 Jul 2026 22:17:50 GMT</pubDate>
            <atom:updated>2026-08-25T22:35:12.487Z</atom:updated>
            <content:encoded><![CDATA[<h3>Smart caching with CTE on Spark</h3><p>Apache Spark is a powerful engine for distributed data processing, but achieving optimal performance requires careful management of intermediate results. One of the most important tools in Spark’s performance toolkit is caching.</p><p>Caching allows Spark to store the results of a computation in memory so that subsequent actions on the same dataset can be executed much faster. However, caching is a double-edged sword — when used incorrectly, it can lead to wasted memory, performance degradation, or even application crashes.</p><p>This is especially important when working with CTEs (Common Table Expressions), temporary views, or deeply nested queries, where intermediate results may be reused multiple times.</p><p>When (and When Not) to Cache</p><p>To use caching effectively, follow these best practices:</p><ul><li>Avoid caching tiny tables.<br>Small tables are processed quickly even without caching, so storing them in memory doesn’t yield meaningful performance gains.</li><li>Avoid caching very large tables.<br>If the dataset cannot fit in memory, caching it will only cause disk spills or eviction of other useful data, ultimately slowing down your job.</li><li>Cache medium-sized, frequently used tables.<br>The sweet spot lies in caching medium-sized datasets that comfortably fit in memory and are accessed repeatedly during a Spark job. This can significantly reduce computation time by preventing redundant transformations.</li><li>Cache only when reuse is guaranteed.<br>There’s no benefit in caching a dataset you won’t use again. Caching should be applied selectively to data that’s reused multiple times in the same job or across iterative operations.</li></ul><p>💡 Pro Tip: Monitor Your Cache</p><p>Use Spark’s UI (available at http://&lt;driver-node&gt;:4040/storage/) to monitor cached tables and memory usage. This helps you ensure that caching is actually improving performance instead of consuming unnecessary memory.</p><p>Caching can be a game-changer for Spark performance — if used wisely. Always evaluate your data size, memory capacity, and reuse frequency before deciding to cache. A thoughtful caching strategy can lead to substantial performance gains, while careless caching can just as easily slow you down.</p><h3>Basics of Caching in Spark</h3><p>Caching allows Spark to store the results of a DataFrame or table in memory (or on disk) to speed up subsequent computations. Spark provides multiple ways to cache:</p><ul><li><strong>Lazy caching</strong> (evaluated only when an action is triggered):</li></ul><pre>df.cache() spark.catalog.cacheTable(&#39;t0&#39;)</pre><ul><li><strong>Eager caching</strong> (evaluated immediately):</li></ul><pre>spark.sql(&#39;CACHE TABLE t0&#39;)</pre><ul><li><strong>Spark 3 lazy SQL caching</strong>:</li></ul><pre>CACHE LAZY TABLE t0</pre><h3>Uncaching</h3><p>To free memory, cached tables can be removed:</p><pre>df.unpersist()               # uncache a specific DataFrame<br>spark.catalog.clearCache()   # uncache all cached tables<br>spark.sql(&quot;UNCACHE TABLE t0&quot;)  # uncache a SQL table</pre><h3>Example: Caching in Action</h3><pre>l = [(&#39;Alice&#39;, 1)]<br>df = spark.createDataFrame(l)<br>df.cache()  # lazy caching</pre><pre>df1 = spark.createDataFrame(l)<br>df1.registerTempTable(&#39;df1&#39;)<br>spark.catalog.cacheTable(&#39;df1&#39;)  # lazy caching via catalog</pre><pre>df2 = spark.createDataFrame(l)<br>df2.registerTempTable(&#39;df2&#39;)<br>spark.sql(&#39;CACHE TABLE df2&#39;)  # eager caching</pre><p>You can verify all cached DataFrames using Spark’s persistent RDDs:</p><pre>for (id, rdd) in spark.sparkContext._jsc.getPersistentRDDs().items():<br>    print(id)</pre><p>And clear caches when done:</p><pre>spark.catalog.clearCache()</pre><h3>Caveat: CTEs</h3><p><strong>CTEs are not automatically cached</strong>. This means that even if a CTE is referenced multiple times, Spark recomputes it every time.</p><p>Example:</p><pre>WITH t0 AS (<br>    SELECT *<br>    FROM selected_companies<br>    WHERE region = &#39;TH&#39;<br>),<br>t1 AS (<br>    SELECT * FROM t0 WHERE business = 1000<br>),<br>t2 AS (<br>    SELECT * FROM t0 WHERE business = 2000<br>)<br>SELECT * FROM t1<br>UNION<br>SELECT * FROM t2</pre><p>In this query, t0 is evaluated <strong>twice</strong>, once for t1 and once for t2.</p><h3>Nested CTEs Compound the Problem</h3><p>The recomputation issue doesn’t stop at a single level. If a CTE (t1) is itself built on top of another CTE (t0), and t1 is referenced multiple times, Spark recomputes the <em>entire chain</em> — t0 and t1 — on every reference. The cost multiplies with nesting depth, not just reference count.</p><pre>WITH t0 AS (<br>  SELECT * FROM selected_companies WHERE region = &#39;TH&#39;<br>),<br>t1 AS (<br>  SELECT * FROM t0 WHERE revenue &gt; 1000000<br>)<br>SELECT * FROM t1 WHERE year = 2023<br>UNION ALL<br>SELECT * FROM t1 WHERE year = 2024</pre><p>Here, t1 is referenced twice — and each time, Spark re-evaluates t1&#39;s definition, which itself re-evaluates t0. So t0 actually runs <em>four</em> times (2 references to t1 × the underlying scan), not two. The deeper the CTE chain, the more this compounds.</p><p>The fix is the same principle as before, applied at the right layer: cache the CTE that sits at the branching point — the one that’s referenced more than once downstream — not necessarily the innermost one.</p><pre>t0 = spark.sql(&quot;SELECT * FROM selected_companies WHERE region = &#39;TH&#39;&quot;)<br>t1 = spark.sql(&quot;SELECT * FROM t0 WHERE revenue &gt; 1000000&quot;)<br>t1.cache()<br>t1.registerTempTable(&#39;t1&#39;)</pre><p>Caching t1 (not t0) is enough here, since t0 is only ever consumed through t1. Caching every level of a nested chain wastes memory for no benefit — cache at the point where the fan-out actually happens.</p><h3>🔹 Explicit Caching for Reuse</h3><p>You can explicitly cache the base table to avoid repeated computation:</p><pre>t0 = spark.sql(&quot;SELECT * FROM selected_companies WHERE region = &#39;TH&#39;&quot;)<br>t0.cache()  # lazy caching<br>t0.registerTempTable(&#39;t0&#39;)</pre><p>Then your CTE-based query can reuse the cached t0:</p><pre>WITH t1 AS (<br>    SELECT * FROM t0 WHERE business = 1000<br>),<br>t2 AS (<br>    SELECT * FROM t0 WHERE business = 2000<br>)<br>SELECT * FROM t1<br>UNION<br>SELECT * FROM t2</pre><p>Now t0 is read from memory (InMemoryTableScan), and the expensive filter is <strong>not re-evaluated</strong>.</p><p><strong>Views Don’t Support Caching</strong></p><p>Unlike temp tables registered via registerTempTable() / createOrReplaceTempView(), a Spark SQL <strong>view</strong> (CREATE VIEW) cannot be cached directly with CACHE TABLE. Running:</p><pre>CACHE TABLE my_view</pre><p>on a view doesn’t persist the underlying data the way it does for a table — the view is just a stored query definition, so referencing it still re-triggers the underlying computation. If you need caching, cache the resolved DataFrame instead:</p><pre>df = spark.sql(&quot;SELECT * FROM my_view&quot;)<br>df.cache()<br>df.createOrReplaceTempView(&quot;my_view_cached&quot;)</pre><p>Then reference my_view_cached downstream instead of the original view.</p><p><strong>Permanent Views Can’t Reference a Temp Cached Table</strong></p><p>Caching a DataFrame and registering it as a <strong>temp view</strong> (createOrReplaceTempView() / registerTempTable()) only makes that view visible within the current Spark session. If you try to define a <strong>permanent view</strong> — either a Spark catalog view (CREATE VIEW ... AS SELECT * FROM &lt;temp_view&gt;) or a Trino/Presto view — on top of that temp table, it fails.</p><pre>t0 = spark.sql(&quot;SELECT * FROM selected_companies WHERE region = &#39;TH&#39;&quot;)<br>t0.cache()<br>t0.createOrReplaceTempView(&quot;t0_cached&quot;)</pre><pre>-- This fails: t0_cached doesn&#39;t exist in the persistent catalog<br>CREATE VIEW my_permanent_view AS<br>SELECT * FROM t0_cached WHERE business = 1000</pre><p>The reason: a permanent view is just a stored query definition in the metastore/catalog, resolved at <strong>query time</strong>, potentially in a completely different session (or by Trino, which doesn’t share Spark’s session state at all). Since a temp cached table only exists in the session that created it, the reference can’t resolve — you’ll get a table/view-not-found error.</p><p><strong>Takeaway:</strong> caching is a session-scoped optimization for reuse <em>within one job/session</em>. It’s not a substitute for materializing a real table when the result needs to be queryable by permanent views, other sessions, or other engines (e.g., Trino). If you need that, write the cached result out to an actual (Iceberg) table instead.</p><h3>⚖️ To Cache or Not to Cache?</h3><p>Deciding whether to cache depends on the query complexity and data size:</p><p><strong>Do not cache</strong> if:</p><ul><li>The computation is simple.</li><li>The dataset is huge, making caching slower than recomputation.</li></ul><p><strong>Cache</strong> if:</p><ul><li>The query is complex.</li><li>The intermediate result is small but used multiple times.</li></ul><blockquote><em>In one real-world example, caching a simple but large DataFrame took </em><strong><em>~14 minutes</em></strong><em>, while recomputing it without caching took only </em><strong><em>~9 minutes</em></strong><em>.</em></blockquote><h3>🔹 Eager Caching for Better Monitoring</h3><p>Spark’s SQL page in the <strong>YARN UI</strong> shows query progress. With lazy evaluation, multiple queries can appear as a single execution plan. Using <strong>eager caching </strong>helps break the execution plan into manageable steps, making it easier to identify bottlenecks.</p><pre>CACHE TABLE intermediate_table</pre><h3>Preventing StackOverflow in Nested Queries</h3><p>Nested transformations can cause StackOverflowError due to deep query plans. Consider:</p><pre>df = df.withColumn(&#39;c1&#39;, complex_udf1(df[&#39;c0&#39;]))<br>df = df.withColumn(&#39;c2&#39;, complex_udf2(df[&#39;c1&#39;]))<br>df = df.withColumn(&#39;c3&#39;, complex_udf3(df[&#39;c2&#39;]))<br># ...<br>df.cache()  # store intermediate plan off-stack</pre><p>Caching at the right point in the transformation pipeline reduces stack depth and prevents errors.</p><h3>Key Takeaways</h3><ol><li><strong>CTEs are recomputed by default.</strong> Use caching if they are referenced multiple times.</li><li><strong>Nesting compounds recomputation — cache at the fan-out point, not every level</strong></li><li><strong>Consider dataset size and complexity</strong> when deciding to cache.</li><li><strong>Eager caching improves visibility</strong> in Spark UI.</li><li><strong>Cache strategically</strong> in long transformation chains to prevent stack overflows.</li></ol><p>By combining CTEs, views, and caching strategically, Spark pipelines can achieve both <strong>efficiency</strong> and <strong>readability</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0fe764766a63" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deletion vectors and Puffin files merge in the new v3 Iceberg format]]></title>
            <link>https://medium.com/@vincent_daniel/deletion-vectors-and-puffin-files-merge-in-the-new-v3-iceberg-format-565188036d0c?source=rss-f736f50ce5d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/565188036d0c</guid>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[optimization]]></category>
            <dc:creator><![CDATA[Vincent DANIEL]]></dc:creator>
            <pubDate>Wed, 15 Oct 2025 18:06:44 GMT</pubDate>
            <atom:updated>2026-08-17T21:21:16.094Z</atom:updated>
            <content:encoded><![CDATA[<h4>Travelling Penguins team up with efficiency on better Iceberg.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Zg3gkH_zwXzE74EC" /><figcaption>Photo by <a href="https://unsplash.com/@cooksphotography?utm_source=medium&amp;utm_medium=referral">Hal Cooks</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Data teams constantly face the challenge of keeping large tables synchronized with upstream sources. Change Data Capture (CDC) workflows — such as incremental MERGE operations—are critical, but performance often degrades at scale.</p><p>Enter <strong>Iceberg format v3 with deletion vectors</strong>, now available from Iceberg 1.8. This release promises substantial performance gains for CDC workloads, particularly for large-scale updates and deletes.</p><p>In this article, we’ll walk through a hands-on benchmark comparing Iceberg format versions 2 and 3 for CDC MERGE operations, highlighting how v3’s innovations deliver measurable performance and cost benefits.</p><h3>Why Iceberg format v3 Matters</h3><p>Iceberg introduces <strong>deletion vectors (DVs)</strong>, a mechanism for efficiently tracking row-level deletions without rewriting entire data files. This improves the efficiency of:</p><ul><li>Large-scale updates and deletes</li><li>Incremental merges in CDC pipelines</li><li>Filtered reads on high-churn tables</li></ul><p>Deletion vectors optimize merge-on-read operations, reduce I/O, and accelerate queries — a perfect opportunity to validate these claims in a controlled benchmark.</p><p><strong>How it fits with Iceberg v3 deletion vectors</strong><br>MERGE INTO with Merge-on-Read (MOR) on Iceberg v3 tables generate deletion vectors (DVs) when rows are removed, storing them efficiently in Puffin files. This allows CDC pipelines to:</p><ul><li>Apply incremental updates without rewriting large files.</li><li>Enable faster filtered reads by ignoring deleted rows.</li><li>Minimize storage and compute overhead, lowering overall costs.</li></ul><p>In short, <strong>MERGE INTO with MOR plus Iceberg v3 deletion vectors</strong> is like giving your penguins ice skates: incremental changes glide smoothly, merges happen faster, and your CDC workflow stays stable — even under heavy updates.</p><p>Iceberg provides several ways to encode row-level deletions, evolving across versions:</p><ul><li><strong>Position delete files</strong> (deprecated in v3) track deleted rows by file and row position. Existing files are valid in v3 but <strong>new deletes should use deletion vectors</strong>.</li><li><strong>Equality delete files</strong> track deleted rows by column values and remain supported across versions.</li><li><strong>Deletion vectors</strong>, introduced in v3, efficiently encode deleted positions <strong>within a single file</strong> using a bitmap. They replace position delete files.</li></ul><p>Deletion vectors are compact binary representations stored in <strong>Puffin files</strong>, optimized for execution and storage. They support 64-bit row positions internally but use 32-bit Roaring bitmaps for most cases. The DV metadata tracks the referenced data file, offset, and length of the bitmap within the Puffin file. Each data file can have at most one deletion vector per snapshot, replacing any existing DV. DVs are produced by default when a table is set to format version 3.</p><p><strong>Key benefits of deletion vectors:</strong></p><ul><li>Efficiently represent deletes without rewriting entire data files.</li><li>Reduce I/O and memory overhead during queries.</li><li>Integrate seamlessly with existing manifests for tracking deletes.</li><li>Enable advanced optimizations such as predicate pushdown and cost-based query planning.</li></ul><h3>Benchmark Overview</h3><p>My benchmark simulates a <strong>dimension synchronization pattern</strong>, executing two consecutive MERGE cycles against Iceberg tables using both format v2 and v3.</p><p><strong>Key steps:</strong></p><ol><li>Initialize the target table with the desired format version.</li><li><strong>First MERGE:</strong> baseline insert/update/delete using checksum detection (id + name).</li><li><strong>Second MERGE:</strong> simulate real-world changes — updates, deletes, inserts, and ID shifts — at <strong>100x scale</strong>, applying dramatic modifications to stress-test the system and highlight the benefits of v3.</li><li>Measure performance for both merges, as well as full vs. filtered table reads.</li></ol><p>The test ran on Amazon EMR 7.11 with Apache Spark 3.5.6, using Iceberg 1.9.1-amzn-1. This version provides key deletion vector improvements — v2-to-v3 rewrites and dangling deletion vector detection. No Iceberg maintenance like REWRITE_POSITION_DELETE_FILE has been operated since it might bias the results especially for v2 format by merging position delete files.</p><p>Here’s the full benchmark code you can use for comparing Iceberg CDC MERGE performance between <strong>format-version 2 and 3</strong>.</p><pre>&quot;&quot;&quot;<br>Iceberg CDC MERGE benchmark (format-version 2 vs 3).<br>Overview<br>========<br>This script measures relative performance of two consecutive MERGE (CDC) cycles<br>against Iceberg tables using format-version 2 and 3. The workload simulates a<br>dimension/fact style synchronization pattern:<br>1. Create target table for given format-version and perform initial MERGE<br>   using checksum change detection (hash of id + name).<br>2. Perform a second MERGE sourcing from `egdp_prod_supply.lodging_profile_eg`.<br>3. Measure timing of both merges plus a full table read and a filtered read.<br>4. Print comparison between v2 and v3.<br>&quot;&quot;&quot;<br>from __future__ import annotations<br>import time<br>from dataclasses import dataclass<br>from typing import Tuple<br>import log as l<br>import spark_session as s<br>from pyspark.sql import functions as F<br># ---------------------------------------------------------------------------<br># Configuration &amp; Constants<br># ---------------------------------------------------------------------------<br>logger = l.setup_logger(__name__)<br>l.set_level(logger, True)<br>FILTER = 100000<br>DB_NAME = &#39;stg&#39;<br># Scale factor for the second MERGE change set to amplify workload volume<br>VOLUME_MULTIPLIER = 100<br>STRIDE = FILTER * 20000<br>@dataclass<br>class TableResult:<br>    version: int<br>    merge_time_first: float<br>    merge_time_second: float | None<br>    full_read_time: float<br>    filtered_read_time: float<br># ---------------------------------------------------------------------------<br># Spark Session Builder<br># ---------------------------------------------------------------------------<br>def init_spark():<br>    &quot;&quot;&quot;Initialize and return a Spark session using standard project helper.&quot;&quot;&quot;<br>    return s.get_spark(&quot;iceberg-delete-benchmark&quot;)<br># ---------------------------------------------------------------------------<br># Performance Helpers<br># ---------------------------------------------------------------------------<br>def timed(fn, *args, **kwargs):<br>    start = time.time()<br>    result = fn(*args, **kwargs)<br>    return result, time.time() - start<br>def test_read_performance(spark, table_name: str) -&gt; Tuple[float, float, int, int]:<br>    q_base = f&quot;SELECT COUNT(1) FROM {DB_NAME}.{table_name}&quot;<br>    (_, full_time) = timed(lambda: spark.sql(q_base).collect())<br>    total_count = spark.sql(q_base).collect()[0][0]<br>    mid = FILTER<br>    q_filt = f&quot;SELECT COUNT(1) FROM {DB_NAME}.{table_name} WHERE property_id &gt; {mid}&quot;<br>    (_, filtered_time) = timed(lambda: spark.sql(q_filt).collect())<br>    filtered_count = spark.sql(q_filt).collect()[0][0]<br>    return full_time, filtered_time, total_count, filtered_count<br># ---------------------------------------------------------------------------<br># Core Test (MERGE benchmark)<br># ---------------------------------------------------------------------------<br>def test_iceberg_table(spark, version: int) -&gt; TableResult:<br>    &quot;&quot;&quot;Run two MERGE cycles for a given Iceberg table format version.<br>    Returns timing metrics for first and second MERGE plus read timings.<br>    &quot;&quot;&quot;<br>    table_name = f&quot;merge_benchmark_iceberg_v{version}&quot;<br>    fq_table = f&quot;{DB_NAME}.{table_name}&quot;<br>    logger.info(f&quot;\n=== Testing Iceberg format-version {version} with CDC MERGE ===&quot;)<br>    # Target table<br>    spark.sql(f&quot;&quot;&quot;<br>        CREATE TABLE IF NOT EXISTS {fq_table} (<br>            property_id BIGINT,<br>            property_name STRING,<br>            checksum STRING,<br>            update_datetime TIMESTAMP<br>        ) USING iceberg<br>        TBLPROPERTIES (<br>            &#39;format-version&#39; = &#39;{version}&#39;,<br>            &#39;write.metadata.delete-after-commit.enabled&#39; = &#39;true&#39;,<br>            &#39;write.metadata.previous-versions-max&#39; = &#39;1&#39;,<br>            &#39;write.delete.mode&#39;=&#39;merge-on-read&#39;,<br>            &#39;write.update.mode&#39;=&#39;merge-on-read&#39;,<br>            &#39;write.merge.mode&#39;=&#39;merge-on-read&#39;,<br>            &#39;write.merge.distribution-mode&#39;=&#39;hash&#39;,<br>            &#39;history.expire.min-snapshots-to-keep&#39; = &#39;1&#39;,<br>            &#39;history.expire.max-snapshot-age-ms&#39; = &#39;1&#39;,<br>            &#39;write.wap.enabled&#39;=&#39;true&#39;<br>        )<br>    &quot;&quot;&quot;)<br>    prod_source_db = &quot;prod&quot;<br>    prod_source_table = f&quot;{prod_source_db}.inventory&quot;<br>    # First MERGE (baseline)<br>    merge_sql = f&quot;&quot;&quot;<br>WITH<br>  snapshot AS (<br>        SELECT p.property_id, p.property_name FROM {prod_source_table} p<br>  ),<br>  changes AS (<br>    SELECT<br>            s.property_id,<br>            s.property_name,<br>            MD5(concat_ws(&#39;|&#39;, s.property_id, s.property_name)) AS checksum,<br>      CURRENT_TIMESTAMP AS update_datetime<br>    FROM snapshot s<br>  )<br>MERGE INTO {fq_table} t<br>USING changes c<br>ON t.property_id = c.property_id<br>WHEN MATCHED AND NOT(t.checksum &lt;=&gt; c.checksum) THEN UPDATE SET *<br>WHEN NOT MATCHED BY TARGET THEN INSERT *<br>WHEN NOT MATCHED BY SOURCE THEN DELETE<br>&quot;&quot;&quot;<br>    _, first_merge_time = timed(lambda: spark.sql(merge_sql))<br>    # Second MERGE: simulate environment drift &amp; large-scale changes<br>    merge_sql_2 = f&quot;&quot;&quot;<br>WITH base AS (<br>    SELECT p.property_id, p.property_name FROM {prod_source_table} p<br>),<br>dup AS (<br>    SELECT id AS dup_idx FROM range({VOLUME_MULTIPLIER})<br>),<br>snapshot AS (<br>    SELECT<br>        b.property_id + d.dup_idx * {STRIDE} AS property_id,<br>        CASE WHEN b.property_id % 2 = 0 THEN concat(b.property_name,&#39;_CHG_&#39;, d.dup_idx) ELSE b.property_name END AS property_name<br>    FROM base b<br>    CROSS JOIN dup d<br>    WHERE b.property_id % 7 != 0<br>    UNION ALL<br>    SELECT<br>        b.property_id + ({VOLUME_MULTIPLIER} + d.dup_idx) * {STRIDE} AS property_id,<br>        concat(b.property_name,&#39;_NEW_&#39;, d.dup_idx) AS property_name<br>    FROM base b<br>    CROSS JOIN dup d<br>    WHERE b.property_id &lt;= 10<br>),<br>changes AS (<br>    SELECT<br>        s.property_id,<br>        s.property_name,<br>        MD5(concat_ws(&#39;|&#39;, s.property_id, s.property_name)) AS checksum,<br>        CURRENT_TIMESTAMP AS update_datetime<br>    FROM snapshot s<br>)<br>MERGE INTO {fq_table} t<br>USING changes c<br>ON t.property_id = c.property_id<br>WHEN MATCHED AND NOT(t.checksum &lt;=&gt; c.checksum) THEN UPDATE SET *<br>WHEN NOT MATCHED BY TARGET THEN INSERT *<br>WHEN NOT MATCHED BY SOURCE THEN DELETE<br>&quot;&quot;&quot;<br>    _, second_merge_time = timed(lambda: spark.sql(merge_sql_2))<br>    full_read_time, filtered_read_time, total_count, filtered_count = test_read_performance(spark, table_name)<br>    logger.info(<br>        f&quot;First merge: {first_merge_time:.3f}s | Second merge: {second_merge_time:.3f}s | &quot;<br>        f&quot;Full read: {full_read_time:.3f}s | Filtered read: {filtered_read_time:.3f}s&quot;<br>    )<br>    logger.info(<br>        f&quot;Row counts after second merge -&gt; total: {total_count}, filtered(property_id&gt;{FILTER}): {filtered_count}&quot;<br>    )<br>    return TableResult(version, first_merge_time, second_merge_time, full_read_time, filtered_read_time)<br># ---------------------------------------------------------------------------<br># Comparison Reporter<br># ---------------------------------------------------------------------------<br>def print_comparison(v2: TableResult, v3: TableResult):<br>    logger.info(&quot;\n=== Comparison v2 vs v3 ===&quot;)<br>    def improvement(old, new):<br>        return ((old - new) / old * 100.0) if old &gt; 0 else 0.0<br>    logger.info(<br>        f&quot;First merge: v2={v2.merge_time_first:.3f}s v3={v3.merge_time_first:.3f}s &quot;<br>        f&quot;(Δ={improvement(v2.merge_time_first, v3.merge_time_first):.1f}%)&quot;<br>    )<br>    logger.info(<br>        f&quot;Second merge: v2={v2.merge_time_second:.3f}s v3={v3.merge_time_second:.3f}s &quot;<br>        f&quot;(Δ={improvement(v2.merge_time_second, v3.merge_time_second):.1f}%)&quot;<br>    )<br>    logger.info(f&quot;Full read:  v2={v2.full_read_time:.3f}s v3={v3.full_read_time:.3f}s (Δ={improvement(v2.full_read_time, v3.full_read_time):.1f}%)&quot;)<br>    logger.info(f&quot;Filtered read: v2={v2.filtered_read_time:.3f}s v3={v3.filtered_read_time:.3f}s (Δ={improvement(v2.filtered_read_time, v3.filtered_read_time):.1f}%)&quot;)<br># ---------------------------------------------------------------------------<br># Main Entrypoint<br># ---------------------------------------------------------------------------<br>def main():<br>    spark = init_spark()<br>    try:<br>        v2 = test_iceberg_table(spark, 2)<br>        v3 = test_iceberg_table(spark, 3)<br>        print_comparison(v2, v3)<br>    finally:<br>        s.stop_spark(spark)<br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><h3>Benchmark Results</h3><p>The performance improvements were significant:</p><ul><li><strong>Filtered reads:</strong> nearly <strong>2x faster</strong> in v3 thanks to deletion vectors, which avoid scanning unnecessary deleted rows.</li><li><strong>Merge performance:</strong> <strong>4x faster</strong> execution under high churn, with reduced I/O overhead.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0sZICkuMPJeC91YrfXQ-hQ.png" /></figure><blockquote>During the benchmark, I observed that v3 produced roughly twice as many Puffin files compared to v2 delete files. However, unlike v2 delete files — which accumulate with each update — the number of deletion vector (DV) files remains stable over time. Each data file can have at most one DV per data file, and any new DV supersedes previous ones for that file. As a result, the total number of DV files will eventually be lower than the number of v2 delete files, improving long-term storage efficiency.</blockquote><p>v3 doesn’t just improve speed — it reduces cost across storage, compute, and S3 access.</p><ul><li><strong>Storage Costs:</strong> DVs are stored as <strong>compact</strong> <strong>bitmaps in Puffin files</strong>, replacing large positional delete Parquet files used in v2.</li><li><strong>S3 Access Costs:</strong> Fewer and better-sized files reduce the number of S3 object operations.</li><li><strong>Compute Costs:</strong> Faster queries mean Spark jobs finish sooner, lowering EMR resource usage.</li></ul><h3>Migrating from Version 2 to Version 3</h3><p>To migrate a table from Iceberg V2 to V3 and take advantage of deletion vectors:</p><h4>Upgrade the table to V3</h4><p>Update the table’s format version by altering its properties:</p><pre>ALTER TABLE [table_name] SET TBLPROPERTIES(&#39;format-version&#39;=&#39;3&#39;);</pre><h4>Rewrite existing V2 position delete files to deletion vectors</h4><p>After upgrading, V2 position delete files should be converted to deletion vectors. Run the following procedure:</p><pre>CALL iceberg.system.rewrite_position_delete_files(table =&gt; &#39;[table_name]&#39;, options =&gt; map(&#39;rewrite-all&#39;, &#39;true&#39;));</pre><h4>Verify the migration</h4><p>After the rewrite, there should be no V2 position delete files remaining — only Puffin files storing deletion vectors.</p><h3>Do you need compaction with deletion vectors?</h3><p>One of the most common questions with Iceberg is whether you need to run REWRITE_DATA_FILES (compaction &amp; position delete files) or REWRITE_POSITION_DELETE_FILE (position delete files only). The answer depends heavily on the delete format:</p><ul><li><strong>In v2 with position deletes, e</strong>very delete creates new delete files that accumulate over time. Rewriting position delete files is almost <strong>mandatory</strong> at scale to avoid excessive small files and read amplification.</li><li><strong>In V3 with deletion vectors</strong>, deletes are stored as bitmaps within Puffin files. Each data file can have only one deletion vector per snapshot, and new deletion vectors replace the old ones. As a result, deletion vectors do not accumulate the way position delete files do, and they have minimal impact on performance. This makes rewriting position delete files optional and less frequent, although compaction may still be necessary. Note that rewrite_position_delete_files does <strong>not</strong> affect deletion vectors; use rewrite_data_files instead.</li></ul><p>That said, if you’re migrating from v2, rewriting position delete files consolidate old delete files into DVs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XbwiNndhDj0LseMkNAh4EQ.png" /></figure><h3>Engine Compatibility</h3><h4><strong>Apache Spark 3.5 + Iceberg 1.8+</strong></h4><p>Officially supports <strong>deletion vectors (DVs)</strong> and <strong>Puffin metadata</strong> introduced in Iceberg v3. Spark is currently the reference implementation for Iceberg v3 functionality.</p><h4>Starburst Enterprise 476-e</h4><p><strong>Supports Iceberg v3 </strong>with DV read/write capabilities.</p><h4>Trino OS 480</h4><p>S<strong>upport Iceberg v3</strong> (including DVs or Puffin files).</p><h4>Databricks Runtime 18.0</h4><p>S<strong>upport Iceberg v3. </strong>Doesn’t support Iceberg v2 position delete files.</p><h3>When the “One DV Per Data File” Invariant Breaks</h3><p>The article already notes that <em>“each data file can have at most one deletion vector per snapshot.”</em> That’s not just a spec detail — Iceberg enforces it as a hard invariant at <strong>read planning time</strong>, and violating it takes the table down completely, not just degrades performance.</p><p><strong>What happens:</strong> if two DV puffin files ever end up referencing the same data file, DeleteFileIndex$Builder throws:</p><pre>org.apache.iceberg.exceptions.ValidationException: Can&#39;t index multiple DVs for &lt;data-file&gt;</pre><p>This fires during query <em>planning</em> — before a single Spark task runs — so the table becomes fully unreadable, not just slow.</p><p><strong>Root cause:</strong> this is a real, documented Iceberg bug, not a theoretical edge case. With Spark’s Adaptive Query Execution (AQE) — on by default since Spark 3.2.0 — a single data file can get split across multiple tasks during a MERGE INTO / delete operation. Each task independently writes its own DV for the same underlying file. In Iceberg versions <strong>before 1.10.2 / 1.11.0</strong>, MergingSnapshotProducer didn&#39;t detect or merge these duplicate DVs before committing, so both puffin files land in the manifest — and the table locks up on the next read.</p><p><strong>Fix:</strong> upgraded in Apache Iceberg <a href="https://github.com/apache/iceberg/pull/15654">1.10.2</a> and <a href="https://github.com/apache/iceberg/pull/15006">1.11.0</a> — MergingSnapshotProducer now detects and merges duplicate DVs on the commit path. Most managed runtimes (e.g., EMR) lag upstream Iceberg releases, so check your engine&#39;s bundled Iceberg version before assuming you&#39;re safe.</p><p><strong>If you hit this before upgrading:</strong></p><ul><li><em>Workaround:</em> disable AQE for MERGE INTO jobs on V3 tables (spark.sql.adaptive.enabled=false), or temporarily fall back to V2 merge-on-read.</li><li><em>Recovery (data-safe):</em> since maintenance procedures like rewrite_position_delete_files require scanning the table — which is exactly what&#39;s broken — recovery means unregistering the table (metadata only), recreating it at the same location, and re-registering the existing Parquet files with add_files. No data is rewritten or lost.</li></ul><p>V3’s single-DV-per-file model is what makes it fast, but it also means duplicate-DV corruption is a full outage, not a slow query. Pin an Iceberg runtime ≥1.10.2/1.11.0 before relying on V3 DVs in production, especially with AQE enabled.</p><h3>Key Takeaways</h3><p><strong>Merge-on-read with deletion vectors</strong> is a game-changer for CDC workloads.</p><ul><li><strong>Filtered reads</strong> see the most dramatic speedups, especially under high churn.</li><li><strong>Scaling changes</strong> amplifies v3’s advantages, making it highly relevant for pipelines in the travel industry like in Expedia.</li><li><strong>Cost savings</strong> compound through reduced compute, storage and S3 access overhead.</li><li><strong>Rewriting position deletes files</strong> in v3 is not necessary, thanks to stable and bounded deletion vectors.</li></ul><h3>Conclusion</h3><p>Apache Iceberg goes beyond ACID transactions and data lakehouse semantics. With innovations like deletion vectors and Puffin files, it provides a metadata-rich foundation for query performance optimization — across any engine.</p><p>Because the Iceberg specification is open, these benefits aren’t locked into a single tool or vendor. Engines can leverage metadata for advanced optimizations like predicate pushdown, cost-based planning, and dynamic partition pruning.</p><p>If your ETL pipelines struggle with large merges or filtered reads on frequently updated tables, <strong>upgrading to the new Iceberg format v3 could unlock dramatic speedups and cost reductions</strong>.</p><p><strong>References</strong></p><ul><li><a href="https://iceberg.apache.org/">Apache Iceberg Documentation</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=565188036d0c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why You Should Prefer MERGE INTO Over INSERT OVERWRITE in Apache Iceberg]]></title>
            <link>https://medium.com/expedia-group-tech/why-you-should-prefer-merge-into-over-insert-overwrite-in-apache-iceberg-b6b130cc27d2?source=rss-f736f50ce5d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6b130cc27d2</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Vincent DANIEL]]></dc:creator>
            <pubDate>Tue, 30 Sep 2025 12:02:09 GMT</pubDate>
            <atom:updated>2025-10-17T17:38:09.027Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/expedia-group-tech/subpage/data">EXPEDIA GROUP TECHNOLOGY — DATA</a></h4><h4>Stop overwriting — start merging: a smarter approach to updating Iceberg tables</h4><figure><img alt="A person stands in a national park at sunset" src="https://cdn-images-1.medium.com/max/1024/0*hHlu4wpqXNZky6Sl" /><figcaption>Photo by NEOM on Unsplash</figcaption></figure><p>Apache Iceberg has emerged as a leading table format for data lakes, offering robust support for schema evolution, hidden partitioning, and time travel. When managing large-scale data, especially in partitioned tables, choosing the right data modification strategy is crucial. This article explores why MERGE INTO with the Merge-on-Read (MOR) strategy is often more advantageous than INSERT OVERWRITE, particularly as partitioning schemes evolve.​</p><h3>Understanding MERGE INTO vs. INSERT OVERWRITE</h3><p>Apache Iceberg supports two primary strategies for handling row-level updates:​</p><ul><li><strong>Copy-on-Write (COW):</strong> Updates are applied by rewriting the entire file, ensuring that all changes are reflected immediately. This approach provides strong consistency but can be resource-intensive.​</li><li><strong>Merge-on-Read (MOR):</strong> Updates are stored as separate delta logs, which are merged with the base data during query execution. This strategy optimizes write performance and is particularly beneficial for real-time analytics.​</li></ul><p>While both COW and MOR are designed for row-level updates, another common approach — INSERT OVERWRITE, rewrites entire partitions, which can be inefficient and lead to inconsistencies when partitioning schemes change.​</p><h3>INSERT OVERWRITE</h3><ul><li><strong>Purpose</strong>: Completely replaces data in a table or partition with the result of a query.</li><li><strong>Use Case</strong>: Ideal for batch processing scenarios where entire partitions need to be refreshed.</li><li><strong>Behavior</strong>: Removes existing data and writes new data.</li><li><strong>Schema Requirements</strong>: Strict — column names, data types, and order must match.</li><li><strong>Performance</strong>: Can be faster for large-scale replacements but may overwrite more data than necessary.</li></ul><h3>MERGE INTO</h3><ul><li><strong>Purpose</strong>: Conditionally updates, deletes, or inserts rows into a table based on a matching condition.</li><li><strong>Use Case</strong>: Suitable for incremental updates, change data capture (CDC), or handling slowly changing dimensions (SCD).</li><li><strong>Behavior</strong>: Only modifies affected rows, leaving the rest of the data intact.</li><li><strong>Schema Requirements</strong>: More flexible — does not require exact column order or names to match.</li><li><strong>Performance</strong>: Generally more efficient for targeted updates but can be complex to implement.</li></ul><h3>Performance benefits of MERGE INTO (MOR)</h3><p>Using MERGE INTO with the MOR strategy offers several performance advantages:​</p><ul><li><strong>Faster Writes:</strong> By appending changes as delta files rather than rewriting entire files, write operations are significantly faster.​</li><li><strong>Reduced I/O Operations:</strong> Since only the affected data is modified, the number of I/O operations is minimized, leading to cost savings.​</li><li><strong>Improved Query Performance:</strong> MOR enables efficient querying by combining delta files with base files during read operations, streamlining data retrieval. However, compaction becomes necessary as the number of delta files grows to maintain optimal performance.</li></ul><p>These benefits make MOR particularly suitable for analytics workloads.​ For practical guidance on implementing high-performance, cost-effective, and well-scoped merge statements — especially in cases where a natural key is absent — refer to <a href="https://medium.com/@vincent_daniel/boost-iceberg-performance-and-cut-compute-costs-with-well-scoped-merge-statements-e0a8f702c321">this article</a>.</p><h3>Cost savings with MERGE INTO (MOR)</h3><p>The MOR strategy can lead to significant cost savings:​</p><ul><li><strong>Lower Storage Costs:</strong> By avoiding the need to rewrite entire partitions, storage costs are reduced.​</li><li><strong>Efficient Resource Utilization:</strong> Fewer resources are required for write operations, leading to lower operational costs.​</li><li><strong>Scalability:</strong> MOR scales efficiently with growing data volumes, maintaining performance and cost-effectiveness.​</li></ul><p>These factors contribute to a more economical approach to data management in large-scale environments.​</p><h3>Real-world cost and performance example</h3><p>Let’s take a real-world example of a data pipeline using <strong>Amazon EMR 7.8</strong> and <strong>Apache Iceberg 1.7.1-amzn-1</strong>, where daily updates are required.</p><h4>Scenario details:</h4><ul><li><strong>Timeframe</strong>: Update data for the <strong>last 3 weeks</strong> (21 days).</li><li><strong>Update Frequency</strong>: 2GB of data needs to be updated daily.</li><li><strong>Week Size</strong>: 10 GB.</li><li><strong>Instance Type</strong>: Mix of <strong>r8g.2xlarge</strong>, <strong>r8g.4xlarge</strong>, <strong>r8g.8xlarge</strong> for the EMR fleet.</li><li><strong>Region</strong>: US East 1.</li><li><strong>Storage</strong>: S3.</li><li><strong>Purge</strong>: Disabled.</li></ul><h4>INSERT OVERWRITE costs:</h4><p>For <strong>INSERT OVERWRITE</strong>, the <strong>3 weeks of data</strong> (30GB) would be rewritten every day, leading to higher compute costs and storage overhead.</p><ul><li><strong>Compute Time per Day</strong>: Each overwrite operation (30GB) takes <strong>3 hours</strong>.</li><li><strong>Hourly Cost of r8g.4xlarge</strong>: $0.65982.</li><li><strong>Daily Compute Cost</strong>: <strong>$1.97946</strong> (for 3 hours of usage).</li><li><strong>Storage Costs</strong>: Every day, 30GB of data is written to S3, leading to an increase in <strong>S3 storage costs</strong>.</li><li><strong>Total Storage Cost per Day</strong>: $0.000767 per GB = <strong>$0.023</strong>.</li><li><strong>Total Cost (Compute + Storage)</strong>: <strong>$2 per day</strong>.</li></ul><h4>MERGE INTO (MOR) costs:</h4><p>With <strong>MERGE INTO (MOR)</strong>, only <strong>2GB of data</strong> is updated daily, which results in far lower compute costs and storage usage.</p><ul><li><strong>Compute Time per Day</strong>: Each update (2GB) takes <strong>1 hour</strong>.</li><li><strong>Hourly Cost of r8g.4xlarge</strong>: $0.65982.</li><li><strong>Daily Compute Cost</strong>: <strong>$0.65982 </strong>(for 1 hour of usage).</li><li><strong>Storage Costs</strong>: Each day, only <strong>2GB of data</strong> is written to S3, so the storage cost is lower.</li><li><strong>Total Storage Cost per Day</strong>: $0.000767 per GB = <strong>$0.001534</strong>.</li><li><strong>Total Cost (Compute + Storage)</strong>: <strong>$0.66 per day</strong>.</li></ul><h4>Compaction costs (twice a week for MERGE INTO):</h4><ul><li><strong>Compaction Time</strong>: 6 mins per compaction using <strong>r8g.4xlarge</strong>.</li><li><strong>Hourly Cost of r8g.4xlarge</strong>: $0.65982.</li><li><strong>Total Compaction Cost per Week</strong>: <strong>$0.13</strong> (for two compaction operations).</li></ul><figure><img alt="A table comparing the total 30-day costs of two Apache Iceberg update strategies: INSERT OVERWRITE and MERGE INTO (MOR) with compaction. The table breaks down compute, storage, and compaction costs. INSERT OVERWRITE incurs a compute cost of $59.38 and storage cost of $0.69, totaling $60.07. MERGE INTO (MOR) shows a compute cost of $19.79, storage cost of $0.046, and compaction cost of approximately $0.56, totaling $20.40" src="https://cdn-images-1.medium.com/max/1024/0*SaJQjNPvWHcWT65c.png" /><figcaption>Cost Comparison of INSERT OVERWRITE vs. MERGE INTO (MOR) + Compaction for 30 Days</figcaption></figure><blockquote><em>💡 Using </em><em>MERGE INTO instead of </em><em>INSERT OVERWRITE results in a </em><strong><em>reduction of up to 66% in costs</em></strong><em> in this scenario — and potentially even more for large tables, especially when data changes are minimal. This approach is particularly effective for incremental loads (e.g., new properties), Change Data Capture (CDC), and Slowly Changing Dimensions (SCD).</em></blockquote><h3>Handling partition evolution</h3><p>Apache Iceberg supports partition evolution. Using the ALTER TABLE ... ADD PARTITION FIELD command, you can add new partition fields to a table. This operation is metadata-only and does not affect existing data files.</p><p>One of the challenges with INSERT OVERWRITE arises when partitioning schemes evolve. Dynamic partition overwrite behavior changes when the partitioning scheme evolves. For example, if a table is initially partitioned by day and later partitioned by hour, an INSERT OVERWRITE operation will only affect hourly partitions, <strong>leaving daily ones intact and causing duplicate data </strong>when both are read for the same period.​ Therefore, explicit partition overwrites should be performed using the DataFrameWriterV2 API to ensure consistency.​</p><p>In contrast, MERGE INTO is more resilient to changes in partitioning schemes. Since it operates at the file level, it can adapt to new partitioning strategies without the risk of duplicating data.​</p><h3>Compaction and Maintenance: The Missing Piece</h3><p>While preferring MERGE INTO over INSERT OVERWRITE helps avoid correctness issues and snapshot instability, long-term performance still depends on good maintenance. As highlighted in <a href="https://medium.com/@vincent_daniel/automating-apache-iceberg-maintenance-with-spark-and-python-ee1a253de86c">this article</a>, <strong>compaction is essential for keeping Iceberg tables</strong> <strong>healthy, especially with merge-on-read</strong>, where frequent updates can quickly generate many small data and delete files. Without compaction, query performance and metadata overhead will degrade over time.</p><p>Some key takeaways worth integrating into any MERGE INTO-heavy workflow:</p><ul><li><strong>Table-size aware strategies: </strong>Use different thresholds for small, medium, and large tables (via min-input-files and min-file-size-bytes) so compaction is aggressive on smaller datasets but cost-efficient at scale.</li><li><strong>Solve the small files problem: </strong>Target a target-file-size-bytes (commonly ~256 MB) so that frequent merges don’t leave you with thousands of tiny files. As described in <a href="https://medium.com/@vincent_daniel/optimizing-spark-performance-with-aqe-mastering-shuffle-partition-coalescing-e9d3eef8579f">this article</a>, enabling AQE shuffle partition coalescing reduces the creation of small shuffle partitions, which directly lowers the risk of small output files.</li><li><strong>Smart delete handling: </strong>Apply a delete-file-threshold so rewrites only occur when enough deletes accumulate on a data file, avoiding wasteful rewrites.</li><li><strong>Dangling delete cleanup:</strong> On Iceberg 1.7+, enable remove-dangling-deletes to clear out delete files that reference data files which no longer exist.</li></ul><p>Taken together, these practices ensure that the extra snapshots, delete files, and fragmentation produced by MERGE INTO are continuously cleaned up. The end result: you keep the safety and expressiveness of MERGE INTO, while maintaining the query performance and storage efficiency that Iceberg is designed for.</p><h3>Best practices for data modification in Apache Iceberg</h3><p>To optimize data modification operations in Apache Iceberg, consider the following best practices:</p><ul><li>Use MERGE INTO with <strong>Merge-On-Read (MOR)</strong> for workloads that require high write throughput and involve minimal changes.</li><li>Opt for MERGE INTO with <strong>Copy-On-Write (COW)</strong> when updates impact a large portion of data and compaction becomes too resource-intensive.</li><li>Avoid using INSERT OVERWRITE in environments with evolving partitioning schemes, as it can introduce data duplication, result in costly backfilling, and increase maintenance overhead.</li></ul><h3>Deletion Vectors in Iceberg v3</h3><p>Iceberg v3 introduces <strong>deletion vectors (DVs)</strong>, compact bitmaps stored in <strong>Puffin files</strong> that track row-level position deletes <strong>without rewriting entire data files</strong>.</p><p><strong>Why it matters</strong></p><ul><li><strong>Replaces v2’s position-delete files</strong>, which accumulated endlessly.</li><li>Each data file has at most one DV per snapshot; new ones replace old, <strong>keeping file counts stable</strong>.</li><li>DVs are <strong>smaller</strong>, <strong>faster</strong> to apply, and reduce both compute and S3 access costs.</li></ul><p><strong>Benefits</strong></p><ul><li><strong>Faster MERGE &amp; reads</strong>: Queries skip deleted rows <strong>efficiently</strong>, especially in filtered reads.</li><li><strong>Lower cost</strong>: Compact DV Puffin files reduce storage, object operations, and cluster runtime.</li><li><strong>Simpler maintenance</strong>: <strong>Compaction is optional </strong>— needed only if a file has very dense deletes or when migrating from v2.</li></ul><p><strong>Takeaway</strong></p><p>With deletion vectors, Iceberg v3 makes CDC and high-churn MERGE workloads faster, cheaper, and easier to maintain.</p><p>👉 <strong>Want to dive deeper?</strong><br> Check out the full article on Medium:<br> <a href="https://medium.com/@vincent_daniel/deletion-vectors-and-puffin-files-merge-in-the-new-v3-iceberg-format-565188036d0c"><em>Deletion Vectors and Puffin Files: MERGE in the New v3 Iceberg Format</em></a></p><h3>Conclusion</h3><p>In the dynamic landscape of big data, where data volumes and access patterns evolve rapidly, choosing the right data update strategy is paramount. MERGE INTO with the MOR strategy offers a more efficient, cost-effective, and maintainable approach compared to INSERT OVERWRITE, especially in partitioned tables. By aligning your data operations with the strengths of Apache Iceberg, you can ensure optimal performance and scalability for your data lake infrastructure.​</p><p><strong>Further reading:</strong></p><ul><li><a href="https://iceberg.apache.org/docs/nightly/spark-ddl/">Apache Iceberg DDL Documentation</a></li><li><a href="https://medium.com/@geekfrosty/copy-on-write-or-merge-on-read-what-when-and-how-64c27061ad56">Copy-on-Write or Merge-on-Read? What, When, and How?</a></li><li><a href="https://www.linkedin.com/pulse/copy-on-write-cow-merge-on-read-mor-open-table-formats-ashok-k-sahoo-cstrc/">Copy-On-Write (COW) and Merge-On-Read (MOR) in Open Table Formats: Delta, Hudi, Iceberg</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6b130cc27d2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/expedia-group-tech/why-you-should-prefer-merge-into-over-insert-overwrite-in-apache-iceberg-b6b130cc27d2">Why You Should Prefer MERGE INTO Over INSERT OVERWRITE in Apache Iceberg</a> was originally published in <a href="https://medium.com/expedia-group-tech">Expedia Group Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Chill Your Data with Iceberg Write Audit Publish]]></title>
            <link>https://medium.com/expedia-group-tech/chill-your-data-with-iceberg-write-audit-publish-746c9eb3db48?source=rss-f736f50ce5d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/746c9eb3db48</guid>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[cost]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <dc:creator><![CDATA[Vincent DANIEL]]></dc:creator>
            <pubDate>Tue, 02 Sep 2025 11:01:42 GMT</pubDate>
            <atom:updated>2025-09-29T10:24:34.932Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/expedia-group-tech/subpage/data">EXPEDIA GROUP TECHNOLOGY — DATA</a></h4><h4><em>A strategic shift toward data flows that are version-controlled, testable, and auditable</em></h4><p>Ensuring data quality in production environments is critical — but challenging — especially when test environments lack access to live data. Traditional approaches often involve duplicating production tables for User Acceptance Testing (UAT) and validation, leading to resource overhead, governance challenges, and inconsistencies.</p><figure><img alt="People row boats in spring" src="https://cdn-images-1.medium.com/max/1024/0*iE2enMFwR2FLl72L" /><figcaption>Photo by Yu Kato on Unsplash</figcaption></figure><h3>What is Write‑Audit‑Publish?</h3><p><strong>Write‑Audit‑Publish (WAP)</strong> is a robust data validation workflow that improves data quality by staging data changes through three steps:</p><figure><img alt="Write New data is written to an isolated, non-production environment, preventing disruption to production. Audit The data is validated with quality checks such as, NULL detection, Duplicate checks or schema integrity. Publish After passing audits, data is atomically committed to the production environment, ensuring consistency and accuracy." src="https://cdn-images-1.medium.com/max/1024/0*PFgk_9Fta0bTwLyN.png" /><figcaption>Source: <a href="https://www.dremio.com/blog/streamlining-data-quality-in-apache-iceberg-with-write-audit-publish-branching/">Write-Audit-Publish Stages</a></figcaption></figure><ol><li><strong>Write</strong><br>New data is written to an <strong>isolated</strong>, non-production environment, preventing disruption to production.</li><li><strong>Audit</strong><br>The data is <strong>validated</strong> with quality checks such as, NULL detection, Duplicate checks or schema integrity.</li><li><strong>Publish</strong><br>After passing audits, data is <strong>atomically committed</strong> to the production environment, ensuring consistency and accuracy.</li></ol><p>This method reduces risk, enhances reviewability, and ensures only clean data enters production systems. Apache Iceberg offers a powerful alternative which allows teams to validate and test changes using isolated branches — <strong>without touching or duplicating production data</strong>.</p><h3>A different approach with Iceberg WAP</h3><p>Apache Iceberg introduces <strong>Write‑Audit‑Publish</strong>, enabling teams to write data into an <strong>isolated branch</strong>, perform <strong>audits and validations</strong>, and only <strong>publish to the main branch</strong> once the data passes quality checks. This ensures zero disruption to production and avoids unnecessary duplication.</p><pre>                     ┌───────────────────┐<br>                     │    Main Branch    │<br>                     └────────┬──────────┘<br>                              │<br>                     Create WAP Branch<br>                              │<br>                     ┌────────▼────────┐<br>                     │   UAT Branch    │<br>                     └────────┬────────┘<br>                              │<br>              Write Data → Perform Validations &amp; Audits<br>                              │<br>                     ┌────────▼────────┐<br>                     │ Validation Pass?│<br>                     └───────┬─────────┘<br>                     ┌───────┴───────────────┐<br>                     │                       │<br>                 Yes ▼                       ▼ No<br>              ┌──────────────┐        ┌────────────┐<br>              │ Fast-Forward │        │  Delete    │<br>              └──────┬───────┘        │  Branch    │<br>                     │                └────────────┘<br>                     ▼<br>             ┌───────────────────┐<br>             │   Main Branch     │<br>             │     Updated       │<br>             └───────────────────┘</pre><p>Here’s how WAP works with Iceberg:</p><ol><li><strong>Write (on a WAP branch)</strong><br>Data is written to a hidden branch (e.g.,uat_branch) that doesn’t impact production.</li><li><strong>Audit (from the WAP branch)</strong><br>Teams can validate data directly on this branch using SQL, Trino or Spark — without affecting users or the main table.</li><li><strong>Publish</strong><br>Once validated, publishing is done via <strong>fast-forwarding</strong> the metadata to production — efficiently and atomically.</li></ol><h3>WAP: Generic vs Iceberg comparison</h3><figure><img alt="WAP: Generic vs Iceberg Comparison" src="https://cdn-images-1.medium.com/max/1024/0*8sv2o8UnIazUv7lT.png" /><figcaption>WAP: Generic vs Iceberg comparison</figcaption></figure><h3>Why this matters</h3><p>In many enterprise data platforms, production environments are secured, making direct access to production data from test either restricted or impossible. This often leads to:</p><ul><li>Data duplication for testing purposes</li><li>Risk of stale or inconsistent data in UAT</li><li>Operational overhead in maintaining cloned environments</li></ul><h3>Key benefits:</h3><ul><li><strong>No code duplication</strong><br>Reuse the same Extract, Transform, Load (ETL) logic and pipelines across production and UAT — no need for separate workflows.</li><li><strong>Isolated testing environments</strong><br>Branches act as safe sandboxes for testing transformations or validations without impacting production.</li><li><strong>Minimal storage overhead</strong><br>No full table duplication — branches reuse underlying data files, reducing storage and compute costs.</li><li><strong>Significant cost savings</strong><br>By avoiding duplicated tables, redundant jobs, and unnecessary storage, this approach reduces infrastructure and operational expenses.</li><li><strong>Efficient promotion to production</strong><br>Publishing changes is a fast metadata operation — no file rewrites, table swaps, or downtime.</li><li><strong>Auditability and traceability</strong><br>Each branch and commit is versioned, making it easy to track, validate, and roll back changes.</li><li><strong>Built-in rollback support</strong><br>Easily discard or revert branches if validation fails — production remains untouched.</li><li><strong>Support for concurrent writers</strong><br>Enable multiple developers or pipelines to test and validate independently in isolated branches.</li></ul><h3>Step-by-step implementation</h3><h4><strong>Enable WAP on the table</strong></h4><p>First, ensure that the WAP pattern is enabled on your target table:</p><pre>ALTER TABLE prod.db.table SET TBLPROPERTIES ( &#39;write.wap.enabled&#39; = &#39;true&#39; );</pre><h4><strong>Create the UAT branch</strong></h4><p>Create a new branch starting from a specific snapshot (e.g., snapshot 3) and retain it for 7 days:</p><pre>ALTER TABLE prod.db.table CREATE BRANCH `uat_branch` AS OF VERSION 3 RETAIN 7 DAYS;</pre><h4><strong>Perform writes on the UAT branch</strong></h4><p>Set the Spark session to write to the uat_branch:</p><pre>SET spark.wap.branch = uat_branch;</pre><p>Then, perform your data insertion or updates:</p><pre>INSERT INTO prod.db.table VALUES (3, &#39;c&#39;);</pre><h4><strong>Audit the UAT branch</strong></h4><p>After the write operations, validate the data quality on the uat_branch. This can include checks for data integrity, completeness, and consistency.</p><h4><strong>Publish changes to the main branch</strong></h4><p>Once the data quality checks are successful, publish the changes from uat_branch to the main branch:</p><pre>CALL catalog_name.system.fast_forward(&#39;prod.db.table&#39;, &#39;main&#39;, &#39;uat_branch&#39;);</pre><p>Only an Iceberg branch (e.g., uat_branch) that is an ancestor of another branch (e.g., main) can be fast-forwarded to match the state of that other branch. If any ancestor snapshot has been deleted, you might encounter the following error message:</p><blockquote><em>“Cannot fast-forward: main is not an ancestor of branch_uat.”</em></blockquote><p>This occurs because Iceberg requires a direct lineage between branches for fast-forwarding to work.</p><p>You can use the set_current_snapshot procedure to set the current snapshot of the current table state. This procedure does not require the target snapshot to be an ancestor of the current table state. This makes it especially useful when fast-forwarding fails due to missing ancestor snapshots.</p><h4><strong>Expire the UAT branch</strong></h4><p>After the changes are published, the uat_branch will be automatically removed when its retention period expires (7 days in this case).</p><h3>Extending branch retention</h3><p>Sometimes you may want to keep a branch around longer than its initial retention period to allow extended testing or audits. Iceberg lets you update a branch’s retention period by recreating or replacing the branch starting from its current state with a new retention window.</p><p>For example, to extend the retention of the uat_branch from 7 days to 14 days, you can run:</p><pre>ALTER TABLE prod.db.table <br>CREATE OR REPLACE BRANCH uat_branch AS OF BRANCH uat_branch RETAIN 14 DAYS;</pre><p>This command resets the branch starting point to its current version while updating the retention period.</p><p><strong>Key points:</strong></p><ul><li>This operation does not affect the branch data but updates how long snapshots and metadata are kept before automatic cleanup.</li><li>Extending retention increases storage requirements, so plan accordingly.</li><li>Avoid concurrent writes while changing retention to prevent conflicts.</li></ul><p>By managing branch retention flexibly, teams gain more control over their data lifecycle and can accommodate longer validation windows without duplicating data or environments.</p><h3>Why drop a branch?</h3><p>When a branch is used for testing or validation and <strong>the data quality checks fail</strong>, it’s best to drop the branch immediately instead of waiting for its retention period to expire<strong> </strong>(if one exists). This approach helps because:</p><ul><li><strong>Avoids confusion:</strong> Keeping a failed test branch around can mislead teams into thinking it’s still active or valid.</li><li><strong>Frees resources early:</strong> Dropping the branch immediately releases metadata and any associated resources sooner, reducing storage overhead.</li><li><strong>Maintains clean environment:</strong> It keeps the branching structure tidy by removing invalid or obsolete branches quickly.</li><li><strong>Prevents accidental usage:</strong> Eliminates the risk that stale or failed test data might accidentally be queried or used.</li></ul><p>In essence, dropping branches right after test failures ensures your environment stays clean, efficient, and focused only on valid, tested data.</p><h3>Why disable WAP?</h3><p>Disabling WAP blocks writes on all branches. This can be helpful for <strong>temporarily</strong> making <strong>branches read-only</strong>, but should only be done with a clear understanding of the potential impact on other processes that rely on WAP.</p><h3>Lightweight auditing with historical tags</h3><p>Beyond branching for WAP, Iceberg also supports <strong>tags</strong>, which are immutable, named pointers to snapshots. These enable low-overhead saving of historical versions — ideal for audit, compliance, or long-term snapshots:</p><pre>-- Tag end-of-week snapshot, retain for 7 days<br>ALTER TABLE prod.db.table CREATE TAG `EOW-01`<br>  AS OF VERSION 7 RETAIN 7 DAYS;<br>-- Tag end-of-month snapshot, retain for 6 months<br>ALTER TABLE prod.db.table CREATE TAG `EOM-01`<br>  AS OF VERSION 30 RETAIN 180 DAYS;<br>-- Tag end-of-year snapshot indefinitely<br>ALTER TABLE prod.db.table CREATE TAG `EOY-2024`<br>  AS OF VERSION 365;</pre><p>These tags inform expire_snapshots not to <strong>clean up the referenced snapshots</strong>—even as other intermediate snapshots are discarded.</p><p>You can then query past states directly using:</p><pre>SELECT * FROM prod.db.table VERSION AS OF &#39;EOM-01&#39;;</pre><p>Tags serve as a <strong>fast</strong> and <strong>cost-effective alternative to backup tables</strong> with snapshots and selective retention baked in.</p><h3>Disadvantages</h3><p>While branching and tagging offer numerous advantages, there are some important downsides to consider:</p><h4>Storage overhead</h4><p>Each branch and tag introduces additional metadata — and in some implementations, data duplication — into the system. In large-scale environments with heavy branching and extensive tagging, this can result in substantial storage overhead. Teams must adopt effective data retention, pruning, and cleanup strategies to prevent unnecessary data bloat.</p><h4>Complexity</h4><p>Managing a growing number of branches and tags adds organizational complexity:</p><ul><li>Ensuring consistent naming conventions becomes crucial to maintain clarity.</li><li>Understanding the branching structure requires discipline and tooling.</li><li>Orphaned or stale branches and tags must be regularly pruned to avoid confusion and drift.</li></ul><p>Without strong governance, teams can quickly lose visibility into the role and status of each branch or tag.</p><h4>Schema evolution with branches</h4><p>The schema associated with a table is shared across all branches, meaning that when writing data to a branch, the table’s current schema is always used for validation. In contrast, querying a tag relies on the schema tied to the snapshot at the time the tag was created. This difference can introduce confusion when schema evolution occurs frequently, so teams must carefully manage compatibility between branches and snapshots.</p><h3>Cost Savings in Practice: Expedia Group</h3><p>One of the biggest advantages of Iceberg WAP is <strong>cost efficiency</strong>. Traditionally, each release required duplicating production tables and reprocessing the same ETL pipeline twice — once in test (UAT) and again in production. This doubled compute costs, inflated storage usage, and added operational overhead to keep UAT environments in sync.</p><p>With Iceberg branches, data is <strong>written once</strong> into a temporary branch, validated in place, and then promoted to production through a lightweight metadata fast-forward. This means the test run is the production run — <strong>eliminating duplicate work</strong>. At scale, this shift can reduce release costs by <strong>up to 99%</strong>.</p><p>At Expedia, this change was transformative. Previously, every new pipeline required several terabytes of duplicated UAT data and a full second run of the ETL process. After adopting WAP:</p><ul><li>UAT tables were replaced with <strong>temporary Iceberg branches</strong>, avoiding full data copies.</li><li>ETL pipelines were <strong>executed only once</strong>, reused seamlessly across test and production.</li><li>Publishing became nearly instantaneous, as it was just a metadata operation.</li></ul><p>The results were clear:</p><ul><li><strong>~30% lower storage consumption</strong> by eliminating UAT table duplication.</li><li><strong>Up to 99% reduction in release costs</strong>, since pipelines no longer reprocessed data twice.</li><li><strong>Faster, safer deployments</strong> with cleaner governance and traceable audit history.</li></ul><p>By removing the need to reprocess data twice, Iceberg WAP turns testing and release from a costly bottleneck into a <strong>lightweight, scalable, and cost-effective workflow</strong>.</p><h3>Final thoughts</h3><p>The Iceberg WAP model isn’t just a technical improvement — it’s a shift toward <strong>version-controlled, testable, and auditable data pipelines</strong>. By validating in isolated branches and publishing with lightweight metadata operations, teams reduce risk, cut costs, and speed up delivery.</p><p>The result: <strong>cleaner data, lower overhead, and greater trust</strong> in production.</p><h3>Further reading</h3><p>Explore how this pattern is implemented in real-world scenarios:</p><ul><li><a href="https://aws.amazon.com/blogs/big-data/build-write-audit-publish-pattern-with-apache-iceberg-branching-and-aws-glue-data-quality/">AWS Blog: Write-Audit-Publish with Iceberg and Glue Data Quality</a></li><li><a href="https://vutr.substack.com/p/how-does-netflix-ensure-the-data?r=2rj6sg&amp;utm_campaign=post&amp;utm_medium=web&amp;triedRedirect=true">Netflix’s Approach to Scaling Iceberg Data Quality</a></li><li><a href="https://iceberg.apache.org/docs/latest/branching/">Iceberg Docs: Branching &amp; Tagging</a></li></ul><blockquote><a href="https://careers.expediagroup.com/life/">Learn about life at Expedia Group</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=746c9eb3db48" width="1" height="1" alt=""><hr><p><a href="https://medium.com/expedia-group-tech/chill-your-data-with-iceberg-write-audit-publish-746c9eb3db48">Chill Your Data with Iceberg Write Audit Publish</a> was originally published in <a href="https://medium.com/expedia-group-tech">Expedia Group Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing Spark Performance with AQE: Mastering Shuffle Partition Coalescing]]></title>
            <link>https://medium.com/@vincent_daniel/optimizing-spark-performance-with-aqe-mastering-shuffle-partition-coalescing-e9d3eef8579f?source=rss-f736f50ce5d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/e9d3eef8579f</guid>
            <dc:creator><![CDATA[Vincent DANIEL]]></dc:creator>
            <pubDate>Fri, 29 Aug 2025 20:44:59 GMT</pubDate>
            <atom:updated>2025-09-11T16:22:37.537Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn how Adaptive Query Execution dynamically merges partitions, balances workloads, and reduces small files for faster, more efficient Spark jobs.</h4><p>Apache Spark 3.x introduced <strong>Adaptive Query Execution (AQE)</strong>, a powerful feature that dynamically optimizes query plans at runtime. By enabling AQE, Spark can adjust partitioning strategies, rebalance data, and optimize join strategies based on actual data statistics, leading to significant performance improvements.</p><h3>Dynamically Coalescing Shuffle Partitions</h3><p>One of the key features of AQE is the ability to dynamically coalesce shuffle partitions. Shuffle operations in Spark are resource-intensive, and finding the optimal number of shuffle partitions is often challenging. If there are too few partitions, tasks may become large and spill to disk, leading to performance degradation. Conversely, too many partitions can result in small tasks with high overhead, reducing efficiency.</p><p>AQE addresses this by adjusting the number of shuffle partitions based on runtime statistics. After the initial stages of a query are executed, Spark collects metrics such as the size and number of shuffle partitions. Using this information, AQE can merge adjacent small partitions into larger ones, optimizing task sizes and improving overall performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*4SW-6o5w2B-Hn4lY9Mw-lA.png" /><figcaption>Dynamically Coalescing Shuffle Partitions</figcaption></figure><h3>How AQE helps with the small files issue</h3><p>AQE’s shuffle partition coalescing merges small shuffle partitions, cutting down the number of tiny files / tasks. Configs like advisoryPartitionSizeInBytes and coalescePartitions.minPartitionSize let you control when partitions are considered “too small” and should be merged. This avoids many tiny files which hurt I/O, scheduling, overhead, etc.</p><p><strong>Without coalescing</strong>, one often ends up with many small or empty files under several circumstances:</p><ul><li>When spark.sql.shuffle.partitions is set too high relative to the total output size: many partitions are tiny or empty.</li><li>When operations like joins and unions reduce final partition sizes, so what looked fine upstream produces many small outputs downstream.</li><li>Selective filters producing temporary datasets.</li></ul><p>And they point out that <strong>reading small files</strong> causes bad thruput over I/O and higher overhead (task setup, scheduling) because many small partitions / tasks.</p><h3>Spark AQE Configuration</h3><p>To fully leverage AQE —Dynamically Coalescing Shuffle Partitions, it’s essential to fine-tune the following configs:</p><h3>spark.sql.adaptive.enabled</h3><ul><li><strong>Description</strong>: Enables or disables AQE.</li><li><strong>Default</strong>: true</li><li><strong>Recommendation</strong>: Set to true to activate AQE.</li></ul><h3>spark.sql.adaptive.coalescePartitions.parallelismFirst</h3><ul><li><strong>Description</strong>: Determines whether to prioritize parallelism or partition size when coalescing shuffle partitions.</li><li><strong>Default</strong>: true</li><li><strong>Recommendation</strong>: Set to false to respect the target partition size specified by spark.sql.adaptive.advisoryPartitionSizeInBytes.</li></ul><h3>spark.sql.adaptive.advisoryPartitionSizeInBytes</h3><ul><li><strong>Description</strong>: Sets the target size for shuffle partitions during adaptive optimization.</li><li><strong>Default</strong>: 64MB</li><li><strong>Recommendation</strong>: Increase to 128MB or higher, depending on your data distribution.</li></ul><h3>spark.sql.adaptive.coalescePartitions.minPartitionSize</h3><ul><li><strong>Description</strong>: Specifies the minimum size of shuffle partitions after coalescing.</li><li><strong>Default</strong>: 1MB</li><li><strong>Recommendation</strong>: Set to 32MB or higher to avoid creating too many small partitions.</li></ul><h3>spark.sql.adaptive.rebalancePartitionsSmallPartitionFactor</h3><ul><li><strong>Description</strong>: Defines the factor for merging small partitions during the rebalancing process.</li><li><strong>Default</strong>: 0.2</li><li><strong>Recommendation</strong>: Increase to 0.5 to merge smaller partitions more aggressively.</li></ul><h3>spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes</h3><ul><li><strong>Description</strong>: Defines when a partition is considered skewed. If a partition size is larger than this threshold <em>and</em> larger than a factor of the median partition size, AQE will split it into smaller tasks to avoid stragglers.</li><li><strong>Default</strong>: 256MB</li><li><strong>Recommendation</strong>: Set larger than advisoryPartitionSizeInBytes, e.g., 300MB–512MB, especially for datasets with skew.</li></ul><h3>Best Practices for Enabling AQE</h3><ul><li><strong>Monitor Performance</strong>: After enabling AQE, monitor your Spark application’s performance using the Spark UI to ensure that partitions are balanced and tasks are completing efficiently.</li><li><strong>Iterative Tuning</strong>: Adjust the AQE parameters iteratively. Start with the recommended values and fine-tune based on the specific characteristics of your data and workload.</li><li><strong>Avoid Over-Partitioning</strong>: Ensure that the number of shuffle partitions is appropriate for your data size. Too many partitions can lead to overhead, while too few can cause data skew.</li></ul><h3>Conclusion</h3><p>Enabling and tuning AQE in Apache Spark can lead to significant performance improvements by dynamically optimizing query execution plans. By understanding and configuring the key AQE parameters, you can ensure that your Spark applications run efficiently and scale effectively.</p><p>For more in-depth information on AQE and its internals, consider exploring the following resources:</p><ul><li><a href="https://spark.apache.org/docs/latest/sql-performance-tuning.html">Performance Tuning</a></li><li><a href="https://engineering.salesforce.com/how-to-optimize-your-apache-spark-application-with-partitions-257f2c1bb414/">How to Optimize Your Apache Spark Application with Partitions</a></li><li><a href="https://www.databricks.com/blog/2020/05/29/adaptive-query-execution-speeding-up-spark-sql-at-runtime.html">Adaptive Query Execution: Speeding Up Spark SQL at Runtime</a></li><li><a href="https://www.linkedin.com/pulse/spark-aqe-coalescing-partitions-valdas-gylys/">Spark AQE: Coalescing Partitions</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9d3eef8579f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Boost Iceberg Performance and Cut Compute Costs with Well-Scoped MERGE Statements]]></title>
            <link>https://medium.com/@vincent_daniel/boost-iceberg-performance-and-cut-compute-costs-with-well-scoped-merge-statements-e0a8f702c321?source=rss-f736f50ce5d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0a8f702c321</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Vincent DANIEL]]></dc:creator>
            <pubDate>Wed, 02 Jul 2025 13:40:09 GMT</pubDate>
            <atom:updated>2025-08-29T20:53:18.405Z</atom:updated>
            <content:encoded><![CDATA[<h4>Avoid full-table scans, slash costs, and simplify ETL pipelines with partition-targeted merges.</h4><p>Apache Iceberg has transformed data lake management by offering ACID transaction support and SQL-friendly operations like MERGE INTO. This has made upserts and slowly changing dimension (SCD) workloads more manageable at scale. However, there is a subtle yet significant performance pitfall when using the WHEN NOT MATCHED BY SOURCE clause in MERGE statements. While it might seem like a convenient way to delete rows that don&#39;t match the source dataset, this clause can trigger full table scans, leading to significant increases in both runtime and cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/0*M_BwwtQBz3EXTSUe.png" /></figure><p>In this article, we’ll explore why this happens, how a well-scoped MERGE operation can avoid this issue, and why this approach is often more efficient than splitting the logic into a MERGE followed by a separate DELETE.</p><h3>The Problem: Full Table Scans with “WHEN NOT MATCHED BY SOURCE”</h3><p>In Iceberg’s 1.4.0 release, the WHEN NOT MATCHED BY SOURCE clause was introduced to facilitate the deletion of rows in the target table that don’t have a corresponding entry in the source dataset. This feature can be particularly useful in data synchronization tasks.</p><p>Consider the following example:</p><pre>-- 🔄 MERGE statement illustrating common performance pitfall in Iceberg<br>MERGE INTO target_table AS target<br>USING source_table AS source<br>ON target.id = source.id<br>AND target.event_date = CURRENT_DATE<br>-- ⚠️ This predicate *looks* like it limits processing to today&#39;s partition...<br>-- ❌ BUT: Iceberg ignores this filter when using WHEN NOT MATCHED BY SOURCE clause,<br>--      causing a FULL TABLE SCAN of the entire target table regardless of the predicate<br>-- 🔥 This leads to:<br>--    - Excessive I/O and CPU usage<br>--    - Out-of-memory errors on large datasets<br>--    - Severe scalability and performance issues<br><br>WHEN MATCHED THEN<br>-- 🔁 Update existing rows when matching IDs exist in both source and target<br>  UPDATE SET target.value = source.value<br><br>WHEN NOT MATCHED BY TARGET THEN<br>-- ➕ Insert new rows from source that don&#39;t exist in the target<br>  INSERT *<br><br>WHEN NOT MATCHED BY SOURCE AND target.event_date = CURRENT_DATE THEN<br>-- ❌ Delete rows from the target that are not in the source (e.g., obsolete) <br>  DELETE</pre><h3>The Issue:</h3><p>When you use the WHEN NOT MATCHED BY SOURCE clause, Iceberg <strong>does not push down the predicates</strong> from the ON clause or the WHEN NOT MATCHED BY SOURCE condition itself. As a result, Iceberg ends up scanning the <strong>entire target table</strong>, even if only a small subset of data (e.g., a specific date range or partition) is relevant.</p><p>As data volume grows, the <strong>cost</strong> of scanning <strong>large tables</strong> <strong>increases</strong> significantly — not only in terms of <strong>execution time</strong> but also in <strong>resource consumption</strong>. Full table scans for wide-scope merges can lead to <strong>memory pressure</strong> on compute engines, potentially causing <strong>out-of-memory errors</strong>. This makes such approaches increasingly <strong>inefficient at scale</strong> and highlights the importance of <strong>minimizing the scope of operations</strong> whenever possible.</p><p>For more context or ongoing discussion around this topic, see the related <a href="https://github.com/apache/iceberg/issues/11248">GitHub issue #11248</a>.</p><h3>Cost of Full-Table-Scan by Table Size</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ImW4c8HOGhBJXi5D.png" /></figure><h3>In Contrast:</h3><p>If you remove the WHEN NOT MATCHED BY SOURCE clause and adjust the logic accordingly, Iceberg can apply filtering earlier in the process. This enables <strong>partition pruning</strong>, meaning only the relevant partitions are scanned. The result is a significant improvement in performance and a reduction in resource consumption.</p><h3>Why Not Just Use a Separate DELETE?</h3><p>A common workaround to avoid the full table scan caused by the WHEN NOT MATCHED BY SOURCE clause is to split the operation into two steps:</p><ol><li><strong>Use </strong><strong>MERGE</strong> for upserts (insert and update operations).</li><li><strong>Execute a separate </strong><strong>DELETE</strong> for obsolete rows, often with explicit filtering conditions.</li></ol><p>While this avoids the full table scan, it introduces additional challenges:</p><ul><li><strong>Increased Complexity</strong>: Splitting the logic into two operations complicates the pipeline. Both MERGE and DELETE operations must be coordinated, which increases the risk of errors.</li><li><strong>Consistency Issues</strong>: In streaming or concurrent environments, changes may occur between the two operations, leading to potential data inconsistencies.</li><li><strong>Atomicity</strong>: Managing two separate operations makes ensuring atomicity more difficult. If one operation fails, the pipeline may end up in an inconsistent state, requiring additional error handling and retries.</li></ul><h3>The Efficient Solution: Well-Scoped MERGE</h3><p>Rather than relying on the WHEN NOT MATCHED BY SOURCE clause or splitting the operations, the most efficient approach is to <strong>scope the </strong><strong>MERGE operation to the relevant partitions</strong>. This allows Iceberg to avoid a full table scan while still achieving the desired result—upserting and deleting rows in a single, atomic operation.</p><h3>What Makes This Efficient?</h3><ol><li><strong>Partition Pruning</strong>: By scoping the MERGE to a specific date range (e.g., event_date BETWEEN ...), Iceberg will only scan the relevant partitions of the target table, significantly reducing the data that needs to be processed.</li><li><strong>No “WHEN NOT MATCHED BY SOURCE” Clause</strong>: By avoiding this clause, Iceberg can apply filtering early and push down the filter, thus bypassing the need to scan irrelevant data.</li><li><strong>All-in-One Operation</strong>: Using a single MERGE operation handles both upserts and deletes atomically, simplifying the logic and reducing the complexity of your pipeline.</li></ol><h3>Example Code:</h3><pre>-- 1. Scope your target to just today&#39;s partition for efficient partition pruning<br>WITH target AS (<br>  SELECT<br>    t.*,<br>    MD5(concat_ws(&#39;|&#39;, t.id, t.column_a, t.event_date)) AS checksum -- Remove if present in the table<br>  FROM my_table t<br>  WHERE t.event_date = CURRENT_DATE  -- limits processed data to today’s partition<br>),<br>-- 2. Prepare the source dataset, computing a checksum for efficient change detection<br>source AS (<br>  SELECT <br>    s.*,<br>    MD5(concat_ws(&#39;|&#39;, s.id, s.column_a, s.event_date)) AS checksum<br>  FROM source_table s<br>  WHERE s.event_date = CURRENT_DATE  -- only relevant source rows<br>),<br>-- 3. Compute the change set using a full outer join to identify inserts, updates, and deletes<br>changes AS (<br>  SELECT<br>    COALESCE(s.event_date, t.event_date) AS event_date,<br>    COALESCE(s.id, t.id)               AS id,<br>    s.column_a,<br>    s.checksum,<br>    CASE<br>      WHEN s.checksum IS NULL THEN &#39;DELETE&#39;  -- present in target but not in source<br>      WHEN t.checksum IS NULL THEN &#39;INSERT&#39;  -- new row in source<br>      ELSE &#39;UPDATE&#39;                    -- existing row with changed data<br>    END AS cdc<br>  FROM target t<br>  FULL OUTER JOIN source s<br>    ON t.checksum = s.checksum<br>   AND t.event_date = s.event_date<br>  WHERE NOT (s.checksum &lt;=&gt; t.checksum)  -- filter out unchanged rows<br>)<br>-- 4. Perform a single MERGE: scoped to today&#39;s partition, without scanning full table<br>MERGE INTO my_table AS t<br>USING changes c<br>  ON t.checksum = c.checksum<br> AND t.event_date = CURRENT_DATE -- Filter pushdown on the partition — now working.<br>WHEN MATCHED AND c.cdc = &#39;DELETE&#39; THEN<br>  DELETE                                -- remove obsolete rows<br>WHEN MATCHED AND c.cdc = &#39;UPDATE&#39; THEN<br>  UPDATE SET *                          -- update changed rows<br>WHEN NOT MATCHED BY TARGET THEN<br>  INSERT *                              -- insert new rows<br>;</pre><p>In this example:</p><ul><li>The MERGE operation is scoped to a specific date, ensuring only relevant partitions are scanned.</li><li>By omitting WHEN NOT MATCHED BY SOURCE, Iceberg avoids the full table scan and processes only the relevant partitions.</li><li>Upserts and deletes are efficiently handled in a single operation.</li></ul><h3>The Role of Checksum</h3><p>The checksum in this example is used as a lightweight, efficient way to <strong>detect changes</strong> between the source and target records. Instead of performing a deep comparison of all columns in the record, the checksum allows for a <strong>fast comparison</strong> of the data content. If the checksum differs between the source and target, it signifies that at least one of the relevant fields has changed.</p><ul><li><strong>Efficient Change Detection</strong>: The checksum comparison (s.checksum &lt;=&gt; t.checksum) ensures that only records with actual content changes (in id, column_a, or event_date) are considered for upsert operations. This saves computation by skipping rows that haven&#39;t changed.</li><li><strong>Performance Improvement</strong>: By leveraging the checksum, the query avoids performing full row comparisons. Instead, it compares a compact hash value, which is computationally cheaper. This is especially beneficial when dealing with large datasets, where a direct column-by-column comparison would be inefficient.</li><li><strong>Integrity Check</strong>: The checksum also acts as a safeguard to detect any discrepancies or corruption in the data. If two records have the same checksum, it can be assumed that they are identical in terms of the selected columns (though note that there is a small risk of checksum collisions).</li></ul><h3>Real-World Impact</h3><p>By using a well-scoped MERGE like the one described, we’ve observed significant performance improvements in our production pipelines. Specifically:</p><ul><li><strong>Query Times Reduced</strong>: What used to take hours can now be executed in just minutes.</li><li><strong>Cost Savings</strong>: In most cases, we saw a <strong>10x reduction in compute costs</strong>, as Iceberg no longer needs to scan &amp; “merge” the entire target table.</li><li><strong>Improved Pipeline Simplicity</strong>: By keeping the logic in a single, efficient operation, we reduced the complexity of the pipeline and maintained consistent performance over time.</li></ul><h3>Conclusion</h3><p>While the WHEN NOT MATCHED BY SOURCE clause in Iceberg’s MERGE INTO is a convenient feature for handling deletions, its current implementation can lead to expensive full table scans. Instead, by scoping your MERGE to the relevant partitions and avoiding the WHEN NOT MATCHED BY SOURCE clause, you can significantly improve the efficiency of your operations.</p><p>This well-scoped MERGE approach helps keep your data lake fast, scalable, and cost-effective—without the added complexity and risks of managing separate delete operations. It’s a simple yet powerful pattern that minimizes resource usage while ensuring that your pipelines remain efficient and easy to maintain.</p><h3>Bug Reference &amp; Further Reading:</h3><ul><li><a href="https://github.com/apache/iceberg/issues/11248#issuecomment-2885986510">GitHub Issue — Iceberg Bug: Full Table Scans caused by </a><a href="https://github.com/apache/iceberg/issues/11248#issuecomment-2885986510">WHEN NOT MATCHED BY SOURCE</a></li><li><a href="https://github.com/apache/iceberg/issues/3607#issuecomment-979424495">GitHub Issue — Iceberg Performance Improvement Discussion</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0a8f702c321" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Right-Sizing Spark Executors on EMR Instances: A Practical Guide]]></title>
            <link>https://medium.com/@vincent_daniel/right-sizing-spark-executors-on-emr-spot-instances-a-practical-guide-8ce63c843792?source=rss-f736f50ce5d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ce63c843792</guid>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[scaling]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[emr]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Vincent DANIEL]]></dc:creator>
            <pubDate>Thu, 29 May 2025 09:35:35 GMT</pubDate>
            <atom:updated>2025-08-04T08:13:08.255Z</atom:updated>
            <content:encoded><![CDATA[<h3>🚀 Right-Sizing Spark Executors on EMR Spot Instances: A Practical Guide</h3><p>Running Apache Spark applications on Amazon EMR using EC2 Spot Instances offers <strong>significant cost savings</strong>, but it also introduces complexities in resource allocation. One of the most critical areas to get right is configuring <strong>Spark executors</strong> — the heart of your processing logic — to maximize performance while minimizing risk from Spot instance interruptions.</p><p>In this guide, we’ll walk through key principles and best practices to effectively right-size Spark executors for EMR Spot clusters.</p><h3>🧩 Understanding the Challenge</h3><p>Spark jobs rely on executors to perform distributed computing tasks. When you run these jobs on Spot Instances:</p><ul><li>Large executors can make your cluster inflexible and more prone to interruption.</li><li>Small executors may underutilize resources, leading to poor performance.</li></ul><p>So how do you strike the right balance?</p><h3>📌 Key Considerations for Right-Sizing Executors</h3><h3>🔄 1. Diversify Instance Types</h3><p>To improve cluster resilience and increase the chances of acquiring capacity:</p><ul><li>Use a mix of compatible EC2 instance generations (e.g., r8g.xlarge, r7g.xlarge, r6g.xlarge).</li><li>Select instance families with comparable vCPU-to-memory ratios to ensure consistent executor configurations (e.g., r8g.xlarge, r8gd.xlarge).</li></ul><blockquote>Tip: Use the EMR Instance Fleet with Spot diversification.</blockquote><h3>⚙️ 2. Tune Executor Memory and Core Settings</h3><p>Default settings often don’t align well with real-world instance sizes.</p><ul><li><strong>Use </strong><strong>--executor-memory and </strong><strong>--executor-cores</strong> to control how much memory and CPU each executor uses.</li><li>A balanced setup like --executor-memory 18G and --executor-cores 4 works well for R family instance types (1:8 vCPU:Mem ratio).</li><li>Avoid overcommitting memory — leave headroom for the OS and YARN overhead.</li></ul><blockquote>Rule of Thumb: Leave at least 10% of memory unallocated on each node.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qjeXdOL2RkOEEKs7y18G3g.png" /></figure><h3>🧠 3. Account for YARN Memory Overhead</h3><p>YARN places limits on how much memory an executor can use:</p><ul><li><strong>yarn.scheduler.maximum-allocation-mb</strong>: Max memory any container (like a Spark executor) can request.</li><li><strong>spark.yarn.executor.memoryOverhead</strong>: Extra memory allocated outside the JVM heap (e.g., for off-heap storage).</li></ul><p>Ensure these settings are aligned with your instance memory, or your executors might crash unexpectedly.</p><h3>📉 4. Monitor Spot Interruption Rates</h3><p>Not all Spot Instances are created equal. Use the <strong>Spot Instance Advisor</strong> to choose:</p><ul><li>Instance types with historically <strong>lower interruption rates</strong></li><li>Types that are <strong>less in demand</strong> but still meet your technical specs</li></ul><blockquote>Lower interruption = fewer job restarts = more stability</blockquote><h3>✅ Best Practices Summary</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yDzjnZPdI9Z0qKbsyEMolg.png" /></figure><h3>🎯 Final Thoughts</h3><p>Right-sizing your Spark executors is <strong>not a one-size-fits-all</strong> task — especially when using Spot Instances. But by:</p><ul><li><strong>Diversifying your cluster setup</strong></li><li><strong>Tuning executor configs carefully</strong></li><li><strong>Respecting YARN’s memory limits</strong></li><li><strong>Watching interruption trends</strong></li></ul><p>…you’ll be in a strong position to build a <strong>robust, scalable, and cost-efficient Spark infrastructure</strong> on EMR.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ce63c843792" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Mastering Amazon EMR Instance Fleets: Guidelines and a Real-World Configuration Example]]></title>
            <link>https://medium.com/@vincent_daniel/mastering-amazon-emr-instance-fleets-guidelines-and-a-real-world-configuration-example-75bbbff59e2d?source=rss-f736f50ce5d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/75bbbff59e2d</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[emr]]></category>
            <category><![CDATA[graviton]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Vincent DANIEL]]></dc:creator>
            <pubDate>Fri, 23 May 2025 16:31:40 GMT</pubDate>
            <atom:updated>2025-05-23T16:31:40.609Z</atom:updated>
            <content:encoded><![CDATA[<p>Amazon EMR (Elastic MapReduce) is a powerhouse for big data processing, offering flexible and scalable clusters to run Apache Spark, Hadoop, and other frameworks. One of the most powerful features that makes EMR flexible and cost-effective is the <strong>Instance Fleet</strong> — an advanced way to configure and manage the compute instances your cluster uses.</p><p>In this article, we’ll dive deep into <strong>Instance Fleet guidelines</strong> from the official AWS documentation, demystify their best practices, and walk through a concrete example configuration to help you optimize your EMR clusters.</p><h3>🔎 What Are Instance Fleets?</h3><p>Instance Fleets allow you to provision clusters using multiple instance types and pricing models (On-Demand and Spot Instances) within the same fleet. Unlike the older instance group model where you choose one instance type per node group, Instance Fleets enable:</p><ul><li>⚙️ Multiple instance types in one fleet</li><li>📊 Weighted capacities per instance type (to represent different compute capacities)</li><li>💰 Spot bid prices as a percentage of On-Demand prices</li><li>🎯 Priority settings to control which instances launch first</li><li>🔄 Mixed usage of On-Demand and Spot Instances within the same fleet</li></ul><p>This flexibility helps you <strong>maximize cluster availability, improve fault tolerance, and reduce costs</strong>, especially when leveraging Spot Instances.</p><h3>⚙️ How Instance Fleets Work</h3><p>Amazon EMR’s Instance Fleet model abstracts the complexity of capacity provisioning by automatically selecting instance types and quantities based on your specified preferences and available capacity. It manages:</p><ul><li>🚀 <strong>Provisioning</strong>: Requests instances across the selected instance types and Availability Zones.</li><li>🔄 <strong>Rebalancing</strong>: Adjusts cluster capacity dynamically to optimize for price or availability.</li><li>⚠️ <strong>Spot Interruptions Handling</strong>: Replaces interrupted Spot Instances with other specified types or pricing models.</li></ul><p>By contrast, the older <strong>Instance Group</strong> model fixes one instance type per group and doesn’t support Spot or On-Demand mixing within a group.</p><h3>📋 Key Guidelines from AWS Documentation</h3><p>Here are some core best practices from the <a href="https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-instances-guidelines.html">AWS EMR Instance Fleet guidelines</a>:</p><h3>1. 🤹‍♂️ Diversify Instance Types for Spot Fleets</h3><p>Spot Instances can be interrupted if AWS needs capacity elsewhere, so diversifying instance types reduces your risk by letting your cluster fall back to alternatives when some types are unavailable.</p><h3>2. ⚖️ Use Weighted Capacity to Balance Resources</h3><p>Since instance types have different CPU, memory, and network performance, assigning weighted capacity helps EMR understand how many of each type it needs to meet your cluster capacity goals.</p><h3>3. 💸 Set Bid Price Wisely</h3><p>When using Spot Instances, you define a bid price as a percentage of the On-Demand price. This bid sets the <strong>maximum you’re willing to pay per hour</strong>, and it’s a crucial lever for controlling cost vs. availability.</p><h3>4. 🎯 Prioritize Instance Types</h3><p>Priority controls which instance types get provisioned first. Assign higher priority to preferred instance types for optimal performance or availability.</p><h3>5. 🔀 Leverage Mixed On-Demand and Spot Fleets</h3><p>For critical workloads, configure your fleet to use both On-Demand and Spot Instances, ensuring stable capacity with cost-efficient overflow capacity.</p><h3>🛠️ Example Configuration: Worker Instance Fleet Setup</h3><p>To bring these guidelines to life, here’s a sample configuration for a <strong>worker instance fleet</strong> that balances cost, capacity, and availability by mixing various r8g instance types:</p><pre>&quot;worker_instance_type_config&quot;: [<br>  {<br>    &quot;instance_type&quot;: &quot;r8g.2xlarge&quot;,<br>    &quot;weighted_capacity&quot;: 1,<br>    &quot;bid_price_as_percentage_of_on_demand_price&quot;: 50,<br>    &quot;priority&quot;: 4<br>  },<br>  {<br>    &quot;instance_type&quot;: &quot;r8g.4xlarge&quot;,<br>    &quot;weighted_capacity&quot;: 2,<br>    &quot;bid_price_as_percentage_of_on_demand_price&quot;: 50,<br>    &quot;priority&quot;: 4<br>  },<br>  {<br>    &quot;instance_type&quot;: &quot;r8g.8xlarge&quot;,<br>    &quot;weighted_capacity&quot;: 4,<br>    &quot;bid_price_as_percentage_of_on_demand_price&quot;: 50,<br>    &quot;priority&quot;: 3<br>  },<br>  {<br>    &quot;instance_type&quot;: &quot;r8g.12xlarge&quot;,<br>    &quot;weighted_capacity&quot;: 6,<br>    &quot;bid_price_as_percentage_of_on_demand_price&quot;: 50,<br>    &quot;priority&quot;: 2<br>  },<br>  {<br>    &quot;instance_type&quot;: &quot;r8g.16xlarge&quot;,<br>    &quot;weighted_capacity&quot;: 8,<br>    &quot;bid_price_as_percentage_of_on_demand_price&quot;: 50,<br>    &quot;priority&quot;: 1<br>  }<br>]</pre><h3>🧐 Breaking It Down:</h3><ul><li>🧠 <strong>Instance Types (</strong><strong>r8g family):</strong> This family is optimized for memory-intensive workloads — perfect for Spark jobs with heavy data processing.</li><li>⚖️ <strong>Weighted Capacity:</strong> Each instance type is assigned a capacity reflecting its relative power — r8g.2xlarge counts as 1 unit, while r8g.16xlarge counts as 8.</li><li>💵 <strong>Bid Price:</strong> All Spot bids are set at 50% of On-Demand price, balancing cost savings with availability.</li><li>🎯 <strong>Priority:</strong> Lower number means higher priority. The cluster tries to provision larger instances (r8g.16xlarge) before smaller ones, but can fall back if necessary.</li></ul><h3>🧠 About the r8g Family: Powered by Graviton3</h3><p>All the instances in this configuration belong to the r8g family, which is powered by <strong>AWS Graviton3</strong>, the <strong>third generation</strong> of AWS’s custom ARM-based processors.</p><p>Why does this matter?</p><ul><li>🚀 <strong>Performance</strong>: Graviton3 delivers up to 25% better performance than Graviton2 and significantly more than equivalent x86 instances.</li><li>💰 <strong>Cost Efficiency</strong>: Typically offers 20–40% better price-performance for memory-bound workloads.</li><li>🌱 <strong>Sustainability</strong>: Graviton3 is up to 60% more energy efficient, helping reduce your carbon footprint.</li><li>🧩 <strong>Compatibility</strong>: Works seamlessly with most big data frameworks (Apache Spark, Hadoop, Presto), which are now ARM64-compatible on EMR.</li></ul><p>The r8g family is particularly well-suited for <strong>memory-intensive analytics</strong> and <strong>data lake processing</strong>, which are common in EMR workloads.</p><h3>📊 How EMR Handles Capacity with Weighted Capacity and Priority</h3><p>EMR calculates how many instances of each type it needs to meet the total target capacity by multiplying the number of instances by their weighted capacity. For example, if your target capacity is 16 units, this could be satisfied by:</p><ul><li>2 r8g.16xlarge instances (2 x 8 = 16 units), or</li><li>4 r8g.8xlarge instances (4 x 4 = 16 units), or</li><li>A combination of various types, depending on availability and priority</li></ul><p>The priority helps EMR decide the order in which instance types are provisioned. Higher priority (lower number) instance types are allocated first, but EMR can fall back to lower priority types if capacity is constrained.</p><h3>⚠️ Spot Interruption and Recovery</h3><p>When using Spot Instances, interruptions can happen unexpectedly. EMR Instance Fleets handle this gracefully by:</p><ul><li>🔄 Automatically replacing interrupted Spot Instances with others from your fleet’s configuration.</li><li>📈 Scaling the cluster dynamically to maintain the target capacity.</li><li>🧩 Allowing you to specify allocation strategies like capacity-optimized to reduce interruptions.</li></ul><p>This means your big data jobs can run more reliably even with volatile Spot capacity.</p><h3>💡 Additional Tips to Maximize Your Instance Fleets</h3><ul><li>📈 <strong>Monitor Spot Interruptions:</strong> Use Amazon CloudWatch to detect and react to Spot interruptions, enabling automated adjustments.</li><li>🔀 <strong>Mix On-Demand and Spot:</strong> For critical parts of your cluster (like primary and core nodes), use On-Demand; task nodes can use Spot for cost savings.</li><li>🌐 <strong>Choose Multiple Availability Zones:</strong> Specifying multiple AZs spreads your capacity requests, reducing capacity shortages.</li><li>🚦 <strong>Plan for vCPU Limits:</strong> AWS enforces limits on vCPU usage per region and account; ensure your fleet stays within limits or request increases.</li></ul><h3>🎉 Conclusion</h3><p>Instance Fleets represent a leap forward in EMR’s flexibility and efficiency, letting you:</p><ul><li>🔄 Mix instance types and pricing models seamlessly</li><li>⚖️ Control provisioning with weighted capacities and priorities</li><li>💰 Balance cost savings with availability using Spot and On-Demand combinations</li><li>⚠️ Automatically recover from Spot interruptions</li></ul><p>Armed with these guidelines and the example configuration, you can optimize your Amazon EMR clusters for both performance and cost-efficiency.</p><p>For the full deep dive, check out the official AWS docs on <a href="https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-instance-fleet.html">EMR Instance Fleets</a> and <a href="https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-instances-guidelines.html">Instance Fleet Planning Guidelines</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75bbbff59e2d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating Apache Iceberg Maintenance with Spark and Python]]></title>
            <link>https://medium.com/@vincent_daniel/automating-apache-iceberg-maintenance-with-spark-and-python-ee1a253de86c?source=rss-f736f50ce5d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee1a253de86c</guid>
            <category><![CDATA[maintenance]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[spark-sql]]></category>
            <category><![CDATA[icebergs]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Vincent DANIEL]]></dc:creator>
            <pubDate>Sun, 18 May 2025 21:43:37 GMT</pubDate>
            <atom:updated>2025-09-08T20:41:36.958Z</atom:updated>
            <content:encoded><![CDATA[<h4>Keeping your data lake fast, clean, and cost-efficient with a reusable script</h4><p>Apache Iceberg is an increasingly popular open table format for large-scale analytics. It provides schema evolution, hidden partitioning, time travel, and more. But with great power comes great responsibility: <strong>Iceberg tables require regular maintenance</strong> to stay performant and cost-efficient.</p><p>Without it, you’ll run into:</p><ul><li><strong>Tiny files everywhere</strong> → hurting query performance.</li><li><strong>Old snapshots piling up</strong> → inflating storage costs.</li><li><strong>Orphaned files</strong> → cluttering your data lake.</li></ul><p>This article shows how to <strong>automate Iceberg maintenance with Spark and Python</strong> — using a configurable script that handles multiple table categories (small, medium, large, xlarge).</p><h3>Why Automate Maintenance?</h3><p>Iceberg maintenance typically involves three key tasks:</p><ol><li><strong>Rewriting data files</strong> → merge small files into fewer, larger ones.</li><li><strong>Expiring old snapshots</strong> → remove history beyond a retention period.</li><li><strong>Cleaning orphan files</strong> → drop unreferenced files to free up space.</li></ol><p>Doing this manually is error-prone. With automation, you can:</p><ul><li>Apply <strong>different policies per table size</strong>.</li><li>Run <strong>maintenance as a scheduled job</strong>.</li><li>Ensure <strong>consistent cleanup across environments</strong>.</li></ul><h3>Table Categories and Strategy</h3><p>We classify tables into four categories, each with its own rewrite strategy:</p><ul><li><strong>Small</strong> → aggressive rewriting (min 1 file, 128MB target).</li><li><strong>Medium</strong> → balanced rewriting (min 5 files, 256MB target).</li><li><strong>Large</strong> → conservative rewriting (min 5 files, 512MB target).</li><li><strong>XLarge</strong> → very conservative (min 10 files, 1GB target).</li></ul><p>This ensures smaller tables stay optimized, while large tables avoid excessive churn.</p><h3>The Iceberg Maintenance Script</h3><p>Here’s the full Python script. It’s modular, configurable, and production-ready.</p><pre>&quot;&quot;&quot;<br>Iceberg Maintenance Script<br><br>This script performs maintenance operations on Iceberg tables.<br>It handles file rewriting, snapshot expiration, and orphan file cleanup based on table-specific configurations.<br><br>Table Categories:<br>- Small: Low-volume tables with aggressive rewriting (min 1 file)<br>- Medium: Standard tables with moderate rewriting (min 5 files) <br>- Large: High-volume tables with conservative rewriting (min 10 files)<br>- XLarge: Very high-volume tables with most conservative rewriting (min 10 files, 1GB target)<br><br>Operations:<br>- Data file rewriting: Consolidates small files for better query performance<br>- Snapshot expiration: Removes old snapshots to manage storage costs<br>- Orphan file cleanup: Removes unreferenced files to keep storage clean<br>&quot;&quot;&quot;<br><br>import log as l<br>import spark_session as s<br>from datetime import datetime, timedelta<br><br>class IcebergMaintenanceManager:<br>    &quot;&quot;&quot;Manages Iceberg maintenance operations for production tables.&quot;&quot;&quot;<br>    <br>    def __init__(self):<br>        &quot;&quot;&quot;Initialize the maintenance manager with table configurations.&quot;&quot;&quot;<br>        # --- Table configuration ---<br>        self.table_configs = {<br>            # Small tables<br>            &quot;small&quot;: [<br>                {&quot;name&quot;: &quot;stg.demo&quot;, &quot;where&quot;: &quot;property_id &gt;= 10000&quot;},<br>                {&quot;name&quot;: &quot;stg.demo&quot;, &quot;where&quot;: &quot;property_id &gt;= 100000&quot;}<br>            ],<br>            # Medium tables<br>            &quot;medium&quot;: [<br>            ],<br>            # Large tables<br>            &quot;large&quot;: [<br>            ],<br>            # Extra large tables<br>            &quot;xlarge&quot;: [<br>            ]<br>        }<br>        <br>        # Initialize memory size constants and configurations<br>        self._setup_configurations()<br>    <br>    def _setup_configurations(self):<br>        &quot;&quot;&quot;Setup memory constants and rewrite configurations.&quot;&quot;&quot;<br>        # Memory size constants for file size thresholds<br>        self._1024MB = self._mb_to_bytes(1024)<br>        self._512MB = self._mb_to_bytes(512)<br>        self._256MB = self._mb_to_bytes(256)<br>        self._128MB = self._mb_to_bytes(128)<br>        self._64MB = self._mb_to_bytes(64)<br>        self._1B = 1<br><br>        # Base configuration with common settings across all table sizes<br>        self.rewrite_default = {<br>            &quot;max-concurrent-file-group-rewrites&quot;: 100,  # Maximum number of file groups to be simultaneously rewritten<br>            &quot;delete-file-threshold&quot;: 1,  # Minimum deletes associated with a data file for it to be considered for rewriting<br>            # Dangling deletes: Remove delete files that reference data files no longer in the table<br>            # These can occur when concurrent operations create delete files for already-removed data files<br>            # Enabling this prevents accumulation of unnecessary delete files that slow down queries<br>            &quot;remove-dangling-deletes&quot;: &quot;true&quot;  # Clean up orphaned delete files that reference missing data files<br>        }<br><br>        self.rewrite_small = {<br>            **self.rewrite_default,  # Inherit common settings<br>            &quot;target-file-size-bytes&quot;: self._128MB,  # Recommended target size for small tables<br>            &quot;min-input-files&quot;: 1,  # Any file group exceeding this number will be rewritten regardless of other criteria<br>            &quot;min-file-size-bytes&quot;: self._1B,  # Files under this threshold considered for rewriting regardless of other criteria (any size)<br>        }<br>        <br>        self.rewrite_medium = {<br>            **self.rewrite_default,  # Inherit common settings<br>            &quot;target-file-size-bytes&quot;: self._256MB,  # Recommended target size for medium tables<br>            &quot;min-input-files&quot;: 5,  # Any file group exceeding this number will be rewritten regardless of other criteria<br>            &quot;min-file-size-bytes&quot;: self._64MB,  # Files under this threshold considered for rewriting regardless of other criteria<br>        }<br>        <br>        self.rewrite_large = {<br>            **self.rewrite_default,  # Inherit common settings<br>            &quot;target-file-size-bytes&quot;: self._512MB,  # Recommended target size for large tables<br>            &quot;min-input-files&quot;: 5,  # Any file group exceeding this number will be rewritten regardless of other criteria<br>            &quot;min-file-size-bytes&quot;: self._128MB,  # Files under this threshold considered for rewriting regardless of other criteria<br>        }<br>        <br>        self.rewrite_xlarge = {<br>            **self.rewrite_default,  # Inherit common settings<br>            &quot;target-file-size-bytes&quot;: self._1024MB,  # 1GB target size for extra-large tables<br>            &quot;min-input-files&quot;: 10,  # Any file group exceeding this number will be rewritten regardless of other criteria<br>            &quot;min-file-size-bytes&quot;: self._256MB,  # Files under this threshold considered for rewriting regardless of other criteria<br>        }<br><br>    @staticmethod<br>    def _mb_to_bytes(mb):<br>        &quot;&quot;&quot;Convert megabytes to bytes.&quot;&quot;&quot;<br>        return mb * 1024 * 1024<br><br>    def get_all_tables(self):<br>        &quot;&quot;&quot;Get a flat list of all table dicts from the configuration.&quot;&quot;&quot;<br>        all_tables = []<br>        for table_list in self.table_configs.values():<br>            all_tables.extend(table_list)<br>        return all_tables<br><br>    def get_table_size(self, table_name):<br>        &quot;&quot;&quot;Determine the configuration size for a given table name.&quot;&quot;&quot;<br>        for config_type, table_list in self.table_configs.items():<br>            for table in table_list:<br>                if table.get(&quot;name&quot;) == table_name:<br>                    return config_type<br>        raise ValueError(f&quot;Table &#39;{table_name}&#39; not found in any configuration category. Please add it to table_configs.&quot;)<br><br>    def get_formatted_date_offset(self, days):<br>        &quot;&quot;&quot;Get a formatted date string offset by the specified number of days from now.&quot;&quot;&quot;<br>        now = datetime.now()<br>        return (now - timedelta(days=days)).strftime(&quot;%Y-%m-%d %H:%M:%S&quot;)<br><br>    # --- Configuration methods ---<br>    <br>    # max_concurrent_deletes Parameter:<br>    # Controls parallelism for remove_orphan_files and expire_snapshots operations.<br>    # Without this, files are deleted serially which can be significantly slower.<br>    # Creates a thread pool for parallel deletes; 20 provides good performance balance.<br><br>    def get_expire_snapshots_config(self, table):<br>        &quot;&quot;&quot;Configure snapshot expiration based on table size.<br>        <br>        Production tables use appropriate retention periods based on business needs.<br>        &quot;&quot;&quot;<br>        return {<br>            &quot;max_concurrent_deletes&quot;: 20,  # Increase concurrency limit for snapshot cleanup<br>            &quot;older_than&quot;: self.get_formatted_date_offset(3)  # Expire snapshots older than 3 days<br>        }<br><br>    def get_orphan_config(self):<br>        &quot;&quot;&quot;Configure orphan file cleanup.<br>        <br>        Removes orphaned files that are no longer referenced by any snapshot.<br>        Uses 3-day retention for production safety.<br>        &quot;&quot;&quot;<br>        return {<br>            &quot;max_concurrent_deletes&quot;: 20,  # Same concurrency limit as snapshot cleanup<br>            &quot;older_than&quot;: self.get_formatted_date_offset(3)  # Remove orphans older than 3 days<br>        }<br><br>    def get_rewrite_config(self, table_size):<br>        &quot;&quot;&quot;Get the appropriate rewrite configuration for a table size.&quot;&quot;&quot;<br>        if table_size == &quot;small&quot;:<br>            return self.rewrite_small<br>        elif table_size == &quot;medium&quot;:<br>            return self.rewrite_medium<br>        elif table_size == &quot;large&quot;:<br>            return self.rewrite_large<br>        elif table_size == &quot;xlarge&quot;:<br>            return self.rewrite_xlarge<br>        else:<br>            raise ValueError(f&quot;Unknown table size &#39;{table_size}&#39;. Expected &#39;small&#39;, &#39;medium&#39;, &#39;large&#39;, or &#39;xlarge&#39;.&quot;)<br><br>    def build_maintenance_config(self):<br>        &quot;&quot;&quot;Build the complete maintenance configuration for all tables.&quot;&quot;&quot;<br>        all_tables = self.get_all_tables()<br>        im_table_list = {}<br><br>        for table in all_tables:<br>            table_name = table[&quot;name&quot;]<br>            table_size = self.get_table_size(table_name)<br>            rewrite_data_files = dict(self.get_rewrite_config(table_size))  # Copy to avoid mutating shared config<br>            # If a where predicate is specified, add it directly to the rewrite config<br>            if &quot;where&quot; in table:<br>                rewrite_data_files[&quot;where&quot;] = table[&quot;where&quot;]<br>            im_table_list[table_name] = {<br>                &quot;rewrite_data_files&quot;: rewrite_data_files,<br>                &quot;rewrite_manifests&quot;: None,<br>                &quot;expire_snapshots&quot;: self.get_expire_snapshots_config(table_name),<br>                &quot;remove_orphan_files&quot;: self.get_orphan_config()<br>            }<br>        return im_table_list<br><br>    def run_maintenance(self):<br>        &quot;&quot;&quot;Execute Iceberg maintenance operations for all configured tables.&quot;&quot;&quot;<br>        im_table_list = self.build_maintenance_config()<br>        <br>        from setd_iceberg_maintenance import IcebergMaintenance as im<br>        run_im = im(im_table_list)<br>        return run_im<br><br><br># --- Main Execution ---<br>if __name__ == &quot;__main__&quot;:<br>    # Setup logging<br>    logger = l.setup_logger(__name__)<br>    l.set_level(logger, True)<br>    <br>    logger.info(f&quot;Starting Iceberg maintenance&quot;)<br>    <br>    # Create maintenance manager and run maintenance operations<br>    maintenance_manager = IcebergMaintenanceManager()<br>    result = maintenance_manager.run_maintenance()<br>    <br>    logger.info(&quot;Iceberg maintenance completed successfully&quot;)</pre><h3>How the Script Works</h3><p><strong>Configuration Setup</strong></p><ul><li>Defines memory thresholds (64MB, 128MB, 256MB, 512MB, 1GB).</li><li>Builds rewrite configs per table category.</li></ul><p><strong>Table Management</strong></p><ul><li>Tables are grouped into categories (small, medium, large, xlarge).</li><li>Each table can have an optional WHERE clause to scope rewriting.</li></ul><p><strong>Maintenance Operations</strong></p><ul><li><strong>Rewrite data files</strong>: consolidates small files.</li><li><strong>Expire snapshots</strong>: keeps only last 3 days (configurable).</li><li><strong>Remove orphan files</strong>: deletes files older than 3 days not referenced by snapshots.</li></ul><p><strong>Execution Flow</strong></p><ul><li>Script parses arguments (env), sets up logging.</li><li>Runs Iceberg maintenance for each table using Spark.</li><li>Reports status in logs.</li></ul><h3>Best Practices</h3><ul><li><strong>Tune file size thresholds</strong> to match workload (e.g., 512MB vs 1GB).</li><li><strong>Adjust retention days</strong> depending on business needs (default: 3 days).</li><li><strong>Test on staging first</strong> before running in production.</li><li><strong>Schedule with Airflow, cron, or Databricks Jobs</strong> for automation.</li></ul><h3>Next Steps</h3><ul><li>Extend script to handle <strong>partition-level rewriting</strong>.</li><li>Add <strong>metrics collection</strong> (e.g., files compacted, snapshots deleted).</li><li>Hook into <strong>monitoring dashboards</strong> to visualize maintenance impact.</li></ul><p>✅ With this setup, your Iceberg tables will stay lean, fast, and cost-efficient — without constant manual cleanup.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee1a253de86c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>