<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tuan Minh Vu on Medium]]></title>
        <description><![CDATA[Stories by Tuan Minh Vu on Medium]]></description>
        <link>https://medium.com/@minhtun_46814?source=rss-43353cda9414------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*IjDYkGokq1GUuuHExonPpQ.jpeg</url>
            <title>Stories by Tuan Minh Vu on Medium</title>
            <link>https://medium.com/@minhtun_46814?source=rss-43353cda9414------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 07:25:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@minhtun_46814/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Streaming to Iceberg: Kafka Connect vs Flink]]></title>
            <link>https://blog.dataengineerthings.org/streaming-to-iceberg-kafka-connect-vs-flink-218fd71132ee?source=rss-43353cda9414------2</link>
            <guid isPermaLink="false">https://medium.com/p/218fd71132ee</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[kafka-connect]]></category>
            <category><![CDATA[apache-flink]]></category>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <dc:creator><![CDATA[Tuan Minh Vu]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 11:05:47 GMT</pubDate>
            <atom:updated>2026-11-17T16:22:21.311Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WvKdcdVBRv2PDzDe" /><figcaption>Photo by <a href="https://unsplash.com/@thanhharry?utm_source=medium&amp;utm_medium=referral">Harry Tran</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>All Data engineers have to do data ingestion, which is a process of collecting, importing, and moving raw data from diverse sources into a central storage or processing platform. Now, i’m talking about moving data from databases to Iceberg tables. Ideally, Spark would be used, since it not only supports most of the databases (all databases I have worked with 😄), but also provides strong Iceberg support. But ingestion is not just about collecting and importing data, it is also the collaboration across teams: if the database owner does not give us the permission to read the data because they think our ingestion processes might affect their database badly, we can do nothing. There is also another way to ingest data without reading the table directly, thus having very little impact on the source database, which is to use log-based CDC. The change events will be pushed to a message queue (e.g. Kafka), then a tool will be used to write data to Iceberg tables. The question here is, what tool to use?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/309/1*YDDySwWX4vbcT76BlrqQkA.png" /></figure><h3>Kafka Connect</h3><p>In case you have never heard of it,<strong> </strong>Kafka Connect is an open-source framework built into <a href="https://kafka.apache.org/documentation/">Apache Kafka</a> that standardizes moving data between Kafka and external systems. I will just barely describe the architecture of Kafka Connect, diving deeply in that would need another article.</p><ul><li><strong>Connectors:</strong> Reusable plugins that define the logic for copying data. There are 2 types of connectors: Source Connectors (importing data from external systems into Kafka topics) and Sink Connectors (writing data consumed from Kafka topics to external systems)</li><li><strong>Tasks:</strong> The actual data-processing actors split from a connector. 1 connector can be configured to have multiple tasks, thus allowing high throughput through parallel execution across partitions</li><li><strong>Workers:</strong> The background JVM processes that run connectors and tasks</li><li><strong>Converters:</strong> Components that translate data formats (e.g. JSON, Avro, etc.) as it moves in and out of Kafka</li><li><strong>Single Message Transformations (SMTs):</strong> Functions that modify individual messages on the fly</li></ul><p>Using Kafka Connect is very simple: you deploy a Kafka Connect cluster, create connectors with needed configurations then submit them to the cluster. After that, they run! This is a much-needed for a lazybone like me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/335/1*NHXAhvHDf0XOotEuuLj72Q.png" /></figure><p>Now to the main part: Kafka Connect writes data to Iceberg. It is a great tool, and I will tell you why. Kafka Connect is officially supported by Iceberg, and documented clearly at its <a href="https://iceberg.apache.org/docs/nightly/kafka-connect">official document</a>. The configurations needed to configure the Iceberg Sink Connector are also included in that document. I will give ya an example: You have a topic T that messages are avro-serialized, and want a connector that writes to table catalog_x.db_y.tbl_z, the Iceberg table is auto-created, partitioned by partition_col, commit interval of 60 seconds. Assume you use Apache Polaris as the catalog. Then you have these simples configurations:</p><pre>key.converter: &quot;io.confluent.connect.avro.AvroConverter&quot;<br>key.converter.schema.registry.url: &quot;http://schema-registry:8081&quot;<br>value.converter: &quot;io.confluent.connect.avro.AvroConverter&quot;<br>value.converter.schema.registry.url: &quot;http://schema-registry:8081&quot;<br>topics: &quot;T&quot;<br>iceberg.tables: &quot;db_y.tbl_z&quot;<br>iceberg.tables.auto-create-enabled: &quot;true&quot;<br>iceberg.tables.default-partition-by: &quot;partition_col&quot;<br>iceberg.control.commit.interval-ms: &quot;60000&quot;<br>iceberg.catalog: &quot;catalog_x&quot;<br>iceberg.catalog.type: &quot;rest&quot;<br>iceberg.catalog.uri: &quot;...&quot;<br>iceberg.catalog.warehouse: &quot;...&quot;<br>iceberg.catalog.credential: &quot;...&quot;<br>iceberg.catalog.scope: &quot;...&quot;<br>iceberg.catalog.oauth2-server-uri: &quot;...&quot;<br>iceberg.catalog.io-impl: &quot;...&quot;</pre><p>Also, there must be an additional configuration that must be defined if you want the connectors to run, which is iceberg.control.topic.</p><p>Well, how you think? That’s great, isnt it? Very little work to do to write data from a Kafka topic to an Iceberg table. However, Kafka Connect has some limitation that should be considered before using:</p><ul><li><strong>Not a tool for complex transformation logic.</strong> SMTs should just be used for simple transformation, like casting types, update timezone, etc. Putting complicated logic in SMTs is misused.</li><li><strong>Not support upsert mode. </strong>The <a href="https://github.com/databricks/iceberg-kafka-connect">Tabular implementation</a> had upsert mode, but after donating the module to Apache Iceberg, upsert mode was removed. For more detail, look at the class <a href="https://github.com/apache/iceberg/blob/main/kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkConfig.java">IcebergSinkConfig</a>, there is no config for upsert. This is because the design was changed, and upsert mode was considered unsuitable. There is currently an open issue for this that you can follow: <a href="https://github.com/apache/iceberg/issues/10842">https://github.com/apache/iceberg/issues/10842</a></li></ul><h3>Flink</h3><p>Flink is no stranger to us all. It is currently one of the most well-known distributed streaming framework, and it supports Iceberg well (read more at <a href="https://iceberg.apache.org/docs/latest/flink/">https://iceberg.apache.org/docs/latest/flink/</a>).</p><p>Compared to Kafka Connect, Flink is much more complicated. In order to use Flink, firstly, we have to deploy a Flink cluster. Not quite, it depends on the <a href="https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/overview/#deployment-modes">deployment mode</a>, but in modern stack, where all the things are deployed on Kubernetes, Application Mode is mostly preferred with the help of Flink Kubernetes Operator. Then, we have to write a never-ending Flink Application in Java/Python/whatever. Finally, submit the application to the Flink cluster. Not done yet, tuning the configuration of the Flink cluster, checkpoint, state backend, etc. is very complicated and time-consuming, which requires a good knowledge into Flink.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*W8eNAjU3a8El3owY_WCRfw.png" /></figure><p>However, there must be something that comes with this complexity that makes Flink worth using. I list here, 3 points:</p><ul><li><strong>Kinda beat all kinds of complex logic. </strong>Flink has been popular for its stateful streaming ability for years, along with multiple internal built-in features that support handling complex logic.</li><li><strong>Iceberg upsert mode available</strong>. In contrast to Kafka Connect, Iceberg upsert mode is supported in Flink. If you just simply crave the upsert mode, Flink is what you are looking for.</li><li><a href="https://iceberg.apache.org/docs/latest/flink-writes/#flink-dynamic-iceberg-sink"><strong>Flink Dynamic Iceberg Sink</strong></a>. This is a new feature of Flink Iceberg, which automatically evolves the schema of target Iceberg table based on the data. With this, schema change in the source data might no longer be a nightmare for ya.</li></ul><h3>Sum Up</h3><p>I have pointed out the difference between the 2 tools, along with pros and cons of each. Are you struggling choosing 1 of them? Hope that after this article, you will have some more useful perspectives to choose the approach wisely, pal</p><ul><li><strong>Use Kafka Connect,</strong> if you want something simple, fast, low-effort, and append-only is acceptable. For example, a Kafka topic containing logs, and these need to be sinked to Iceberg table for future analysis. They are append-only, and very little needs to be transformed (maybe just something like timezone), Kafka Connect is the right tool.</li><li><strong>Use Flink,</strong> if you need to write stateful, complex streaming application, or you just simply want the upsert mode/dynamic Iceberg sink feature. For example, your team needs to build some streaming ETL pipelines, with complicated logic, window aggregation, etc. then the data is updated to Iceberg with upsert mode. In this case, Flink is the one, no doubt.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=218fd71132ee" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/streaming-to-iceberg-kafka-connect-vs-flink-218fd71132ee">Streaming to Iceberg: Kafka Connect vs Flink</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Min/Max in Iceberg, why it’s ridiculously fast?]]></title>
            <link>https://blog.dataengineerthings.org/min-max-in-iceberg-why-its-ridiculously-fast-d5771707abce?source=rss-43353cda9414------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5771707abce</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Tuan Minh Vu]]></dc:creator>
            <pubDate>Sat, 07 Nov 2026 10:31:47 GMT</pubDate>
            <atom:updated>2026-11-09T15:01:10.494Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Cgw0i9u4AqF6ISc6" /><figcaption>Photo by <a href="https://unsplash.com/@hanhkv67?utm_source=medium&amp;utm_medium=referral">Hạnh Kiều</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I have been working on ingestion recently, from RDBMS to Iceberg. The jobs are operated on batches. Basically, everyone will immediately think of querying the table incrementally with the condition of the column representing updated time (e.g. updated_at, etc.) &gt; α, then inserting that batch to Iceberg table. So how to decide the value of α? Some might think that we should store that α somewhere to use for next runs because they think that getting the maximum value of that updated time column (not partitioning column) from a multi-million-row Iceberg table requires scanning the whole table, which not only takes time but also adds up storage operation cost. Well, they probably misunderstand how Iceberg works. In this article, we gonna find out how Iceberg handles that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*p5p0Sv0_pg_2kHan.png" /></figure><h3>Architecture</h3><p>As we already know, Iceberg architecture has 3 layers: Catalog layer, Metadata layer and Data layer, as shown in the figure below. The metadata of Iceberg table is stored in metadata layer’s files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/0*wFIPQJar3s7UfWsN.png" /><figcaption>Iceberg architecture</figcaption></figure><p>If you have not understood the Iceberg architecture, go read it on the internet before reading my next section, or you might not understand a thing 😵</p><h3>Iceberg mechanism</h3><p>The reader has to read the manifest files before fetching the data files, undoubtedly. In manifest file, each column lower bound and upper bound stats are stored</p><pre>{<br>  &quot;status&quot;: 1,<br>  &quot;snapshot_id&quot;: 1210521780013676019,<br>  &quot;data_file&quot;: {<br>    &quot;content&quot;: 0,<br>    &quot;file_path&quot;: &quot;00000-0-1595179f-...parquet&quot;,<br>    &quot;file_format&quot;: &quot;PARQUET&quot;,<br>    &quot;partition&quot;: {},<br>    &quot;record_count&quot;: 3,<br>    &quot;file_size_in_bytes&quot;: 1419,<br>    &quot;column_sizes&quot;:      [{1: 117}, {2: 114}, {3: 127}],<br>    &quot;value_counts&quot;:      [{1: 3},   {2: 3},   {3: 3}],<br>    &quot;null_value_counts&quot;: [{1: 0},   {2: 0},   {3: 0}],<br>    &quot;nan_value_counts&quot;:  [],<br>    &quot;lower_bounds&quot;: [{1: &quot;6400000000000000&quot;}, {2: &quot;43414e43&quot;}, {3: &quot;004924a0325f0600&quot;}],<br>    &quot;upper_bounds&quot;: [{1: &quot;6600000000000000&quot;}, {2: &quot;53484951&quot;}, {3: &quot;00df9e963a5f0600&quot;}],<br>    &quot;split_offsets&quot;: [4]<br>  }<br>}</pre><p>Reading it, you may probably tell how Iceberg works, ain’t you? Yes! min/max can definitely be delivered from manifest file column stats.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/326/0*Zc-o4iA9-ddfZ_Qq" /></figure><p>The reader would read current snapshot manifest files, then scan all the bounds to keep the lowest min/highest max in memory. Having that, the result is then emitted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Va-W1kh-8UYIlARlGx3EaA.png" /></figure><p>However, there is 1 thing to note, is that <strong>column type String or Binary don’t work in this situation</strong>. Take Iceberg 1.11.0 and Spark 4.1 as the sample, look at the <a href="https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java#L209">implementation</a>:</p><pre>...<br>  private boolean metricsModeSupportsAggregatePushDown(List&lt;BoundAggregate&lt;?, ?&gt;&gt; aggregates) {<br>    MetricsConfig config = MetricsConfig.forTable(table());<br>    for (BoundAggregate aggregate : aggregates) {<br>      String colName = aggregate.columnName();<br>      if (!colName.equals(&quot;*&quot;)) {<br>        MetricsModes.MetricsMode mode = config.columnMode(colName);<br>        if (mode instanceof MetricsModes.None) {<br>          LOG.info(&quot;Skipping aggregate pushdown: No metrics for column {}&quot;, colName);<br>          return false;<br>        } else if (mode instanceof MetricsModes.Counts) {<br>          if (aggregate.op() == Expression.Operation.MAX<br>              || aggregate.op() == Expression.Operation.MIN) {<br>            LOG.info(<br>                &quot;Skipping aggregate pushdown: Cannot produce min or max from count for column {}&quot;,<br>                colName);<br>            return false;<br>          }<br>        } else if (aggregate.type().typeId() == Type.TypeID.STRING<br>            || aggregate.type().typeId() == Type.TypeID.BINARY) {<br>          // lower_bounds and upper_bounds may have been truncated before, so disable push down<br>          // regardless of the current mode<br>          if (aggregate.op() == Expression.Operation.MAX<br>              || aggregate.op() == Expression.Operation.MIN) {<br>            LOG.info(<br>                &quot;Skipping aggregate pushdown: Cannot produce min or max from truncated values for column {}&quot;,<br>                colName);<br>            return false;<br>          }<br>        }<br>      }<br>    }<br><br>    return true;<br>  }<br>...</pre><p>The reason is the default metrics mode. The table property write.metadata.metrics.default defaults to truncate(16) (read more at <a href="https://iceberg.apache.org/docs/latest/configuration/#write-properties">latest configuration document</a>). For a String column the stored bounds are only a 16-character prefix, with the upper bound bumped so it still sorts above every real value. That is fine for filtering pruning, because a truncated bound is still a valid bound. In contrast, it’s not good for returning as the answer to min/max, because the truncated string is not an actual value in the table. Additionally, Iceberg does not trust even if the current mode is full. Files written earlier under a different mode might still carry truncated bounds, and the manifest does not record which mode produced them. So the code disables min/max pushdown for String and Binary type, unconditionally.</p><h3>Conclusion</h3><p>Well, you find it helpful? If yes, then next time, don’t store the timestamp/date checkpoint in another place, it’s really redundant and annoying because of extra implementation. Be straightforward, query the min/max value from the Iceberg table directly. String/Binary column don’t work, so find another way. If nothing works, check the config write.metadata.metrics.default, if it is none or counts then no bounds are stored. Hope this short article can help 😙</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5771707abce" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/min-max-in-iceberg-why-its-ridiculously-fast-d5771707abce">Min/Max in Iceberg, why it’s ridiculously fast?</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Do you really understand Apache Iceberg Delete file?]]></title>
            <link>https://blog.dataengineerthings.org/do-you-really-understand-apache-iceberg-delete-file-d21fc43db75b?source=rss-43353cda9414------2</link>
            <guid isPermaLink="false">https://medium.com/p/d21fc43db75b</guid>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Tuan Minh Vu]]></dc:creator>
            <pubDate>Tue, 06 Oct 2026 16:01:37 GMT</pubDate>
            <atom:updated>2026-10-07T15:10:50.468Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i275F9nTnntEQD6o" /><figcaption>Photo by <a href="https://unsplash.com/@duytoo?utm_source=medium&amp;utm_medium=referral">Di Nè!</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>A crucial concept in Apache Iceberg: <strong>merge-on-read (MOR)</strong> and <strong>copy-on-write (COW)</strong>. The default mode is COW, but COW does not fit every use cases, so, we have to set the mode to MOR sometimes. However, unlike COW, MOR would lead to the creation of some special files, called <strong>Delete file</strong> — you must have also read about this, too.</p><p>2 types of Delete file exists: <strong><em>position delete file</em></strong> &amp; <strong><em>equality delete file</em></strong>. You must be fed up with reading the difference between these 2 types of delete file, as it can be found anywhere on the internet. Therefore, in this article, I’m not gonna tell you about the difference between them again, but when to use which, and how to maintain these delete files properly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/0*23HB4ctOmWoVMBoS" /></figure><h3>When to use which</h3><p>You must have tried finding the configuration to set the delete file format at least once. Me too, and what I found was interesting: no configuration for it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/0*pYCwzI4h5QWEa_WH.gif" /></figure><p>After diving in Iceberg implementation, I realized the delete file format is decided by the implementation of the writer, not via a configuration.</p><p>Let’s take Iceberg 1.11.0 with v2 table and Spark 4.0 as the sample. In <a href="https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeltaWrite.java">SparkPositionDeltaWrite.java</a>, Spark batch writes only produce position delete files, never equality delete files. No setting makes Spark write equality deletes. There are only 2 types of delete writers: ClusteredPositionDeleteWriter and FanoutPositionOnlyDeleteWriter, both write position delete files.</p><p>Unlike Spark, Flink writes both equality and position delete files, but only when it is writing changelog or upsert data. As aforementioned, no setting to choose a type. Which type each delete gets depends on where the old row lives. Read the <a href="https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/format/spec.md?plain=1">spec.md</a> in Iceberg</p><pre>* A _position_ delete file must be applied to a data file when all of the following are true:<br>    - The data file&#39;s `file_path` is equal to the delete file&#39;s `referenced_data_file` if it is non-null<br>    - The data file&#39;s data sequence number is _less than or equal to_ the delete file&#39;s data sequence number<br>    - The data file&#39;s partition (both spec and partition values) is equal [4] to the delete file&#39;s partition<br>    - There is no deletion vector that must be applied to the data file (when added, such a vector must contain all deletes from existing position delete files)<br>* An _equality_ delete file must be applied to a data file when all of the following are true:<br>    - The data file&#39;s data sequence number is _strictly less than_ the delete&#39;s data sequence number<br>    - The data file&#39;s partition (both spec id and partition values) is equal [4] to the delete file&#39;s partition _or_ the delete file&#39;s partition spec is unpartitioned<br><br>In general, deletes are applied only to data files that are older and in the same partition, except for two special cases:<br><br>* Equality delete files stored with an unpartitioned spec are applied as global deletes. Otherwise, delete files do not apply to files in other partitions.<br>* Position deletes (vectors and files) must be applied to data files from the same commit, when the data and delete file data sequence numbers are equal. This allows deleting rows that were added in the same commit.</pre><p>In short, you just have to know that</p><ul><li>Position delete means row #N of file F. It is harmless to apply to the same commit when Flink performs Upsert<br>-&gt; Used when the row being removed was written earlier in the same checkpoint, by the same writer</li><li>Equality delete means any row where id=1. Applying it to the same commit would kill the new version. The consequence is, it can not touch same-commit rows, hence the map and the position delete fallback<br>-&gt; Used when the row being removed was written in an earlier checkpoint (any previous commit), or by another writer</li></ul><p>I know, you might be questioning: What about Kafka Connect Iceberg sink? Unfortunately, after the Tabular implementation was donated to Apache Iceberg, upsert mode has gone 🙏 It is append-only now, so no delete file is written.</p><h3>Maintenance</h3><p>Small file has always been one of the major pain points when using Iceberg. Not just data files, we could also face the small delete files problem. So how to carry out the maintenance for the delete files properly?</p><p>Similar to rewriting data files, Iceberg also provides a procedure to rewrite delete files, but only position delete files, via rewrite_position_delete_files. Well, then how about equality delete files? If there ain’t a way, the tables written by Flink with upsert mode is dead. Lucky me, there is a way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/0*RND35n895l1xUpfV.gif" /></figure><p>Unlike position delete files, there is nothing called rewrite_equality_delete_files. Why? Take a moment to think before reading on 😈.</p><p>A position delete is pinned to a file, but an equality delete is pinned to a point in time. You can repack the first without changing what it deletes, not to the second. An equality delete says id=1, as of commit N, and applies to every data file in the partition with sequence number &lt; N. For example:</p><pre>seq 5: E5  id=1  (commit with sequence number = 5, equality delete with id=1)<br>seq 7: D7  id=1  (commit with sequence number = 7, re-insert a row with id=1)<br>seq 9: E9  id=2  (commit with sequence number = 9, equality delete with id=1)</pre><p>If merging E5 and E9 into one file. What sequence number does it get?<br>- 9 -&gt; id=1 now applies to D7 (7 &lt; 9) and deletes the re-inserted row. Data loss.<br>- 5 -&gt; id=2 no longer applies to files at seq 5..8 that it used to apply to. Deleted rows come back.</p><p>In order to perform maintenance for equality delete files, there is a way, which is to use procedure rewrite_data_files with option delete-file-threshold or delete-ratio-threshold. This also works with position delete file 🆒.</p><ul><li>delete-file-threshold: Counts the number of delete files that scan planning attached to the data file. That is every equality and position delete file that applies to it under the sequence-number and partition rules.</li><li>delete-ratio-threshold: Estimates the fraction of rows deleted in the file.</li></ul><p>rewrite_data_files with at least 1 of these 2 options reads the old data files, drops deleted rows, writes the survivors into new files, and swaps the old files for the new ones in metadata. The delete files themselves are not rewritten, they are left in place and become dangling.</p><blockquote><em>💡 </em>Add option<em> </em>remove-dangling-deletes in order to remove the delete files which do not belong to any live data file.</blockquote><h3>Final Thought</h3><p>Well, now you know how the 2 types of delete file differ. If an Iceberg table written by Flink of yours has been slow lately, this might be due to small equality delete files. Go optimize them 😙</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d21fc43db75b" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/do-you-really-understand-apache-iceberg-delete-file-d21fc43db75b">Do you really understand Apache Iceberg Delete file?</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A myth of Spark repartition and coalesce]]></title>
            <link>https://medium.com/towards-data-engineering/a-myth-of-spark-repartition-and-coalesce-dee6b48e7d19?source=rss-43353cda9414------2</link>
            <guid isPermaLink="false">https://medium.com/p/dee6b48e7d19</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Tuan Minh Vu]]></dc:creator>
            <pubDate>Sat, 19 Sep 2026 08:48:32 GMT</pubDate>
            <atom:updated>2026-09-19T08:48:32.523Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jaDO7nCvbo3M6KbF" /><figcaption>Photo by <a href="https://unsplash.com/@nguyenletri88?utm_source=medium&amp;utm_medium=referral">Tri Nguyen</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Spark is no stranger to Data engineers. Our ETLs run smoothly and scalably thanks to Spark. Partition is one of the most basic concept in Spark: literally, Spark just executes tasks, and each task corresponds to one partition. And, to handle the number of partition, we use repartition or coalesce. Based on the theory, because repartition requires a full shuffle, we should use coalesce when dropping the number of partitions to avoid an expensive data shuffle. But is that always true?</p><h3>Basic Concept</h3><p>I will only briefly cover the concept of repartition and coalesce since they are already well-known concepts.</p><p>repartition is an expensive operation. It triggers a full shuffle, redistributing data based on the configuration — whether by the number of partitions or by key. This involves significant network communication, serialization, temporary files, and I/O overhead. repartition can be used to increase or decrease the number of partitions, because it does a full shuffle to redistribute the data evenly across partitions.</p><p>On the other hand, coalesce triggers no shuffle, and only can be used to decrease the number of partition. When scaling down the number of partitions, the executor only needs to move data off the extra partitions while keeping the rest where it is. So, it would do things like this</p><pre>Node 1 = 1,2<br>Node 2 = 3,4<br>Node 3 = 5,6<br>Node 4 = 7,8<br><br># coalesce(2)<br>Node 1 = 1,2 + (7,8)<br>Node 3 = 5,6 + (3,4)</pre><blockquote>Node 1 and Node 3 did not require its original data to move.</blockquote><h3>The Myth</h3><p>coalesce(1) is often used before writing to files if the developer wants to write the data to only 1 file. However, there is a case where repartition(1) is a better option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*soEbmsrM8HaePj8M.png" /></figure><p>Sounds weird, right? How could an expensive data shuffle possibly be a better choice?</p><p>For example, I have a transformation, which looks like</p><pre>df = spark.read.parquet(&quot;./data/&quot;, header=True)<br>df_join = df.alias(&quot;a&quot;).join(df.alias(&quot;b&quot;), [&quot;id&quot;, &quot;name&quot;]).select(&quot;a.*&quot;)py</pre><h4>With coalesce</h4><pre>df_join.coalesce(1).write.mode(&quot;overwrite&quot;).parquet(&quot;./out&quot;)</pre><p>When using coalesce(1), coalesce takes much longer, because there is no parallelism. As it can be seen, there is only 1 stage in job id 3, and there is just 1 task</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*edbL-9yilk4Nid4K.png" /></figure><p>The stage looks like</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*v4psvdekbkcgu14d.png" /></figure><h4>With repartition</h4><pre>df_join.repartition(1).write.mode(&quot;overwrite&quot;).parquet(&quot;./out&quot;)</pre><p>In contrast repartition(1) is lightning fast, because there are several tasks and they are executed in parallel. The number of task is equal the number of partition, which is 2001 (set by spark.sql.shuffle.partitions)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Fv6mKFi9zeNuZIYJ.png" /></figure><p>However, repartition introduces 1 additional Exchange, which is data shuffling</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TKBYoZFpVLevZ3fD.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FmLv5HqNLBv69Gge.png" /></figure><h4>Conclusion</h4><p>Well, in short, the neglected fact is for coalesce(1) to be performed and avoid shuffling, and it shrinks its parent RDD to explicitly use the same number of partitions. Contrarily, repartition(1) preserves the number of partitions of its parent. Choose your weapon wisely 😉</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dee6b48e7d19" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/a-myth-of-spark-repartition-and-coalesce-dee6b48e7d19">A myth of Spark repartition and coalesce</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Iceberg Isolation levels deep dive: Serializable vs Snapshot]]></title>
            <link>https://medium.com/towards-data-engineering/iceberg-isolation-levels-deep-dive-serializable-vs-snapshot-fa60234c6750?source=rss-43353cda9414------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa60234c6750</guid>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Tuan Minh Vu]]></dc:creator>
            <pubDate>Sun, 13 Sep 2026 11:20:52 GMT</pubDate>
            <atom:updated>2026-09-19T14:54:00.907Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rVlpc_6r8Us8VnlVI7OZlw.jpeg" /><figcaption>Photo by <a href="https://www.gettyimages.com/search/2/image?artistexact=MB%20Photography">MB Photography</a> on <a href="https://www.gettyimages.com/">Getty Images</a></figcaption></figure><p>I have been working with Apache Iceberg for a few years, searching and configuring table properties is part of my work. Iceberg allows user to configure the isolation level for Delete, Update, Merge; however, do you really understand the isolation level in Iceberg? It took me quite some time to understand this concept. So, in this article, let’s find out.</p><h3>Concurrent write conflicts management</h3><p>Well, before that, we should understand how Iceberg manage concurrent write conflicts, or else we would be like a first-grader trying to read calculus theory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/0*9WVc8wfe8suO4s3Q" /></figure><p>The diagram below illustrates an Iceberg transaction write flow</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZDBCvpaeVkGwdghQnUnpSQ.png" /></figure><ol><li>Read the current state. For operations that depend on existing data (OVERWRITE, MERGE, DELETE), the engine scans the current snapshot to identify the relevant files or rows. Operations like INSERT can skip this.</li><li>Work out what the transaction changes, and write the new data files.</li><li>Fetch the table’s latest metadata and pick the metadata version the update will be based on.</li><li>Verify the change prepared in Step 2 is still compatible with the table state read in Step 3. If it isn’t, the transaction aborts.</li><li>Write the new metadata files.</li><li>Commit those metadata files to the catalog. A failed commit retries from Step 3, up to the configured retry limit.</li></ol><p>There are 2 types of conflict that can occur: <strong>Catalog commit conflict</strong> and <strong>Data conflict</strong>.</p><h4>Catalog commit conflict</h4><p>This type of conflict happens when multiple writers try to update the table metadata at the same time. The conflict happens because each writer commits with the snapshot it read as its expected base, so whichever commits second finds that the catalog pointer has already moved and its base is stale.</p><p>Catalog commit conflicts are usually short-lived and can resolve on their own once the operation is retried.. When one hits a conflict, Iceberg retries the operation automatically according to the table’s properties (e.g. commit.retry.num-retries, commit.retry.total-timeout-ms, etc.). Only the metadata commit is repeated — the whole transaction isn’t redone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*VOxNTaHsjJ-cEC-g2dhaKA.png" /></figure><blockquote>The snapshot used in Step 1 and Step 3 might be 2 different snapshots, no problem unless the Data conflict check fails. That’s a really cool work of Iceberg.</blockquote><h4>Data conflict</h4><p>Data conflict is severe. Unlike Catalog commit conflict, if the Data conflict check fails, the transaction <strong>aborts immediately</strong> without retry.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*Egaxu3BMUSZ2FErN.png" /><figcaption>Technical foul</figcaption></figure><p>In the below diagram</p><ul><li>T1 prepares an update to a record on top of snapshot 0 and commits it successfully, so the table’s latest snapshot becomes 1.</li><li>At the same time, T2 is preparing an update to the same record, also based on snapshot 0.</li><li>The snapshot was still 0 when T2 first scanned the data, but T1 has since advanced it to 1, so T2 chooses snapshot 1 as the base.</li><li>When T2 runs its data conflict check, it finds that its change conflicts with snapshot 1, and the transaction fails. The row T2 targeted (id = 1) was rewritten into new data files by T1’s commit, so the files T2 planned to replace are no longer part of the current snapshot.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uv3JXUiC7EceazhBm9MRMA.png" /></figure><h3>Isolation level</h3><p>After a long lesson on how Iceberg handles concurrent write conflicts, here comes the Isolation level. There are 2 levels: serializable and snapshot, and the default is serializable. What are they? And how they differ from each other?</p><p>According to <a href="https://iceberg.apache.org/javadoc/1.11.0/org/apache/iceberg/IsolationLevel.html">Iceberg Javadoc</a>:</p><blockquote>The serializable isolation level guarantees that an ongoing UPDATE/DELETE/MERGE operation fails if a concurrent transaction commits a new file that might contain rows matching the condition used in UPDATE/DELETE/MERGE. For example, if there is an ongoing update on a subset of rows and a concurrent transaction adds a new file with records that potentially match the update condition, the update operation must fail under the serializable isolation but can still commit under the snapshot isolation.</blockquote><p>Well, sounds nice and useful, but I understand nothing 🤷‍♂️</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/230/0*B98Le0V-KftugabI" /></figure><p>Thus, why don’t we go explore what Iceberg really does with these 2 isolation levels?</p><p>I take <strong>Iceberg 1.11.0</strong> and <strong>Spark 4</strong> as the sample. Have a look at Iceberg class <a href="https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java">SparkWrite</a>, under the method commit of the private class OverwriteByFilter or DynamicOverwrite. The only difference is serializable has an additional validation validateNoConflictingData() compared to snapshot. Now we are going to examine what those 2 validations really do.</p><pre>abstract class SparkWrite implements Write, RequiresDistributionAndOrdering {<br>  <br>  ...<br><br>  private class DynamicOverwrite extends BaseBatchWrite {<br>    @Override<br>    public void commit(WriterCommitMessage[] messages) {<br>      DataFileSet files = files(messages);<br><br>      if (files.isEmpty()) {<br>        LOG.info(&quot;Dynamic overwrite is empty, skipping commit&quot;);<br>        return;<br>      }<br><br>      ReplacePartitions dynamicOverwrite = table.newReplacePartitions();<br>      IsolationLevel isolationLevel = writeConf.isolationLevel();<br>      Long validateFromSnapshotId = writeConf.validateFromSnapshotId();<br><br>      if (isolationLevel != null &amp;&amp; validateFromSnapshotId != null) {<br>        dynamicOverwrite.validateFromSnapshot(validateFromSnapshotId);<br>      }<br><br>      if (isolationLevel == SERIALIZABLE) {<br>        dynamicOverwrite.validateNoConflictingData();<br>        dynamicOverwrite.validateNoConflictingDeletes();<br>      } else if (isolationLevel == SNAPSHOT) {<br>        dynamicOverwrite.validateNoConflictingDeletes();<br>      }<br>    }<br>  }<br><br>  ...<br>}</pre><p>Spark (or writer) reads the table at some snapshot, computes new files, and only later commits “remove old files, add new files”. The gap between the read and the commit is where other writers can sneak in. The 2 methods provide checks that run inside that commit against whatever the table looks like at commit time, more specifically, they just set flags. The real work happens in method validate(TableMetadata currentMetadata, Snapshot parent), which is overridden by children of org.apache.iceberg.SnapshotProducer.</p><p>Think of the write as a promise: “I read the table at snapshot S (the newest by default), I computed new files, now replace the old rows with them”. Both validations answer one question at commit time: “Did anyone else change the rows I am touching between S and now?”.</p><p>At commit it walks the snapshot chain from the current table state back to S, and for each snapshot in that range it opens the manifests that snapshot wrote and looks at the file entries in them. Iceberg filters those entries against the region you are overwriting: the filter for overwrite by filter, or the set of partitions your new files land in for dynamic overwrite. If anything survives the filter, the commit fails.</p><ul><li><strong>Validating conflicting data</strong> answers the question: “Did someone add rows I would silently drop?”. It looks only at snapshots that add data (appends and overwrites). If a file that might hold rows in your region was added after S, you are about to replace it without ever having seen its content.<br>E.g. writer W1 reads day=2026–12–28 at S, writer W2 appends new day=2026–12–28 events, W1 commits “replace day=2026–12–28” -&gt; W2 rows vanish. This check stops that.</li><li><strong>Validating conflicting deletes</strong> answers the question: “Did someone remove rows I would update?”. It looks at snapshots that can remove data (overwrites, deletes, and compactions) and at two kinds of entries: new delete files whose scope covers your region, and data files in your region marked “deleted”. Either means rows you read at S have since been removed, but your newly written files still contain them.<br>E.g. writer W1 reads day=2026–12–28 at S, writer W2 deletes user 68 from that day, W1 commits the rewrite for that day -&gt; User 68 is back. This check stops that.</li></ul><p><strong>FAQ: Why compaction affects validating deletes?</strong> <br><strong>Copy-on-write:</strong> Writer read file F_old, wrote F_new as “F_old minus the changed rows”, and plans to commit “remove F_old, add F_new”. Compaction meanwhile rewrote F_old into F_compact. F_old is already gone, and F_compact still holds every row the job meant to remove or change. Committing would make F_new and F_compact both live: the changed rows come back, and every untouched row is now duplicated.</p><pre>S0: starting state, one file `F_old`<br><br>| id | name | balance |<br>|----|------|---------|<br>| 1  | Ann  | 100     |<br>| 2  | Bob  | 200     |<br>| 3  | Cy   | 300     |<br><br>W1 (not yet committed): `DELETE FROM accounts WHERE id = 2`. Reads `F_old` at S0, writes `F_new` = `F_old` without Bob. Planned commit: remove `F_old`, add `F_new`.<br><br>| F_new | id | name | balance |<br>|-------|----|------|---------|<br>|       | 1  | Ann  | 100     |<br>|       | 3  | Cy   | 300     |<br><br>S1: W2 compaction commits first (`replace`). Rewrites `F_old` into `F_compact`, same rows. Commit: remove `F_old`, add `F_compact`.<br><br>| F_c | id | name | balance |<br>|-----|----|------|---------|<br>|     | 1  | Ann  | 100     |<br>|     | 2  | Bob  | 200     |<br>|     | 3  | Cy   | 300     |<br><br>S2: W1 commits on top of S1 with no validation. &quot;Remove `F_old`&quot; finds nothing to remove. &quot;Add `F_new`&quot; succeeds.<br><br>| file  | id | name | balance |<br>|-------|----|------|---------|<br>| F_c   | 1  | Ann  | 100     |<br>| F_c   | 2  | Bob  | 200     |<br>| F_c   | 3  | Cy   | 300     |<br>| F_new | 1  | Ann  | 100     |<br>| F_new | 3  | Cy   | 300     |</pre><p><strong>Merge-on-read: </strong>Writer writes position deletes that say “file F_old, row 17”. After compaction, row 17 of F_old no longer exists, and F_compact has different positions. The delete becomes a no-op and the row is resurrected.</p><pre>S0: starting state, one file `F_old` (merge-on-read table)<br><br>| pos | id | name | balance |<br>|-----|----|------|---------|<br>| 0   | 1  | Ann  | 100     |<br>| 1   | 2  | Bob  | 200     |<br>| 2   | 3  | Cy   | 300     |<br><br>W1 (not yet committed): `DELETE FROM accounts WHERE id = 2`. Reads `F_old` at S0 and finds Bob at position 1. <br>Does not rewrite any data. Writes a position delete file `D` and plans to commit: add `D`.<br><br>| D: file_path | pos |<br>|--------------|-----|<br>| F_old        | 1   |<br><br>S1: W2 compaction commits first (`replace`). Rewrites `F_old` into `F_compact`, same rows. Commit: remove `F_old`, add `F_compact`.<br><br>| F_c pos | id | name | balance |<br>|---------|----|------|---------|<br>| 0       | 1  | Ann  | 100     |<br>| 1       | 2  | Bob  | 200     |<br>| 2       | 3  | Cy   | 300     |<br><br>S2: W1 commits on top of S1 with no validation. &quot;Add `D`&quot; succeeds. Live files are `F_compact` and `D`.<br><br>| file | id | name | balance | applied delete            |<br>|------|----|------|---------|---------------------------|<br>| F_c  | 1  | Ann  | 100     |                           |<br>| F_c  | 2  | Bob  | 200     | none, `D` targets `F_old` |<br>| F_c  | 3  | Cy   | 300     |                           |</pre><h3>Conclusion</h3><p>Well, that was a long article to read. In conclusion, the snapshot level allows looser commit rules, while serializable is stricter. Which level you use depends on the use case you are working on, but most of the time tables should use the serializable level, the default behavior, to ensure data correctness.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa60234c6750" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/iceberg-isolation-levels-deep-dive-serializable-vs-snapshot-fa60234c6750">Iceberg Isolation levels deep dive: Serializable vs Snapshot</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>