<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Serefcanhel on Medium]]></title>
        <description><![CDATA[Stories by Serefcanhel on Medium]]></description>
        <link>https://medium.com/@serefcanhel?source=rss-c75dd195f433------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*mVrNgz0jghC0rI21</url>
            <title>Stories by Serefcanhel on Medium</title>
            <link>https://medium.com/@serefcanhel?source=rss-c75dd195f433------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 08:43:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@serefcanhel/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How Apache Iceberg Handles CRUD Operations Across Metadata and Data Layers (4/4)]]></title>
            <link>https://medium.com/@serefcanhel/how-apache-iceberg-handles-crud-operations-across-metadata-and-data-layers-4-4-43f1c444a3c4?source=rss-c75dd195f433------2</link>
            <guid isPermaLink="false">https://medium.com/p/43f1c444a3c4</guid>
            <dc:creator><![CDATA[Serefcanhel]]></dc:creator>
            <pubDate>Thu, 05 Mar 2026 07:06:58 GMT</pubDate>
            <atom:updated>2026-03-05T07:06:58.660Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><a href="https://medium.com/@bora.polat/how-apache-iceberg-handles-crud-operations-across-metadata-and-data-layers-3-4-c6ccf6c4cb03">Previous: UPDATE &amp; DELETE Commits</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A6_-AMo81bXTQA66eJpQSQ.png" /></figure><h3>How Query Engine Resolves Snapshots and Manifests</h3><blockquote><em>Please note that CRUD operations in this series were executed using Copy-on-Write (COW) mode in Apache Iceberg.</em></blockquote><p>So far, how Iceberg evolves metadata to maintain snapshot-based table state has been examined. The final step is to understand how query engine interprets this metadata to produce a consistent read.</p><p>When extracting data from a table, the query engine reads the current snapshot, which can be found in the latest version of the metadata file. The engine looks at the <strong>current-snapshot-id</strong> field at the top level of the JSON and directly selects the corresponding object from the “snapshots” array. From this object, the engine takes two critical pieces of metadata: <strong>snapshot-id (2245795311685794621)</strong> and <strong>manifest-list (s3a://…manifest_list_delete.avro)</strong>.</p><pre>...<br>&quot;current-snapshot-id&quot;: 2245795311685794621, <br>&quot;refs&quot;: {&quot;main&quot;: {&quot;snapshot-id&quot;: 2245795311685794621,&quot;type&quot;: &quot;branch&quot;}}, <br>&quot;snapshots&quot;: [ <br>    { <br>    &quot;sequence-number&quot;: 1, <br>    &quot;snapshot-id&quot;: 4736832364012705278, <br>    &quot;timestamp-ms&quot;: 1767620736289, <br>    &quot;summary&quot;: {&quot;operation&quot;: &quot;append&quot;,&quot;spark.app.id&quot;: &quot;spark-eb4ccda57c0e4aa2a609860027e4dbb6&quot;,&quot;added-data-files&quot;: &quot;8&quot;,&quot;added-records&quot;: &quot;14&quot;,&quot;added-files-size&quot;: &quot;5966&quot;,&quot;changed-partition-count&quot;: &quot;1&quot;,&quot;total-records&quot;: &quot;14&quot;,&quot;total-files-size&quot;: &quot;5966&quot;,&quot;total-data-files&quot;: &quot;8&quot;,&quot;total-delete-files&quot;: &quot;0&quot;,&quot;total-position-deletes&quot;: &quot;0&quot;,&quot;total-equality-deletes&quot;: &quot;0&quot;,&quot;engine-version&quot;: &quot;3.5.7&quot;,&quot;app-id&quot;: &quot;spark-eb4ccda57c0e4aa2a609860027e4dbb6&quot;,&quot;engine-name&quot;: &quot;spark&quot;,&quot;iceberg-version&quot;: &quot;Apache Iceberg 1.9.2 (commit 071d5606bc6199a0be9b3f274ec7fbf111d88821)&quot;}, <br>    &quot;manifest-list&quot;: &quot;s3a://../manifest_list_insert.avro&quot;, <br>    &quot;schema-id&quot;: 0 <br>    }, <br>    { <br>    &quot;sequence-number&quot;: 2, <br>    &quot;snapshot-id&quot;: 5806419972017972782, <br>    &quot;parent-snapshot-id&quot;: 4736832364012705278, <br>    &quot;timestamp-ms&quot;: 1767686685665, <br>    &quot;summary&quot;: {&quot;operation&quot;: &quot;overwrite&quot;,&quot;spark.app.id&quot;: &quot;spark-91bb060812d044c0ac156a0bd24bf48e&quot;,&quot;added-data-files&quot;: &quot;1&quot;,&quot;deleted-data-files&quot;: &quot;1&quot;,&quot;added-records&quot;: &quot;1&quot;,&quot;deleted-records&quot;: &quot;1&quot;,&quot;added-files-size&quot;: &quot;830&quot;,&quot;removed-files-size&quot;: &quot;754&quot;,&quot;changed-partition-count&quot;: &quot;1&quot;,&quot;total-records&quot;: &quot;14&quot;,&quot;total-files-size&quot;: &quot;6042&quot;,&quot;total-data-files&quot;: &quot;8&quot;,&quot;total-delete-files&quot;: &quot;0&quot;,&quot;total-position-deletes&quot;: &quot;0&quot;,&quot;total-equality-deletes&quot;: &quot;0&quot;,&quot;engine-version&quot;: &quot;3.5.7&quot;,&quot;app-id&quot;: &quot;spark-91bb060812d044c0ac156a0bd24bf48e&quot;,&quot;engine-name&quot;: &quot;spark&quot;,&quot;iceberg-version&quot;: &quot;Apache Iceberg 1.9.2 (commit 071d5606bc6199a0be9b3f274ec7fbf111d88821)&quot;}, <br>    &quot;manifest-list&quot;: &quot;s3a://../manifest_list_update.avro&quot;, <br>    &quot;schema-id&quot;: 0 <br>    }, <br>    { <br>    &quot;sequence-number&quot;: 3, <br>    &quot;snapshot-id&quot;: 2245795311685794621, <br>    &quot;parent-snapshot-id&quot;: 5806419972017972782, <br>    &quot;timestamp-ms&quot;: 1767687723029, <br>    &quot;summary&quot;: {&quot;operation&quot;: &quot;overwrite&quot;,&quot;spark.app.id&quot;: &quot;spark-91bb060812d044c0ac156a0bd24bf48e&quot;,&quot;added-data-files&quot;: &quot;1&quot;,&quot;deleted-data-files&quot;: &quot;1&quot;,&quot;added-records&quot;: &quot;1&quot;,&quot;deleted-records&quot;: &quot;2&quot;,&quot;added-files-size&quot;: &quot;774&quot;,&quot;removed-files-size&quot;: &quot;741&quot;,&quot;changed-partition-count&quot;: &quot;1&quot;,&quot;total-records&quot;: &quot;13&quot;,&quot;total-files-size&quot;: &quot;6075&quot;,&quot;total-data-files&quot;: &quot;8&quot;,&quot;total-delete-files&quot;: &quot;0&quot;,&quot;total-position-deletes&quot;: &quot;0&quot;,&quot;total-equality-deletes&quot;: &quot;0&quot;,&quot;engine-version&quot;: &quot;3.5.7&quot;,&quot;app-id&quot;: &quot;spark-91bb060812d044c0ac156a0bd24bf48e&quot;,&quot;engine-name&quot;: &quot;spark&quot;,&quot;iceberg-version&quot;: &quot;Apache Iceberg 1.9.2 (commit 071d5606bc6199a0be9b3f274ec7fbf111d88821)&quot;}, <br>    &quot;manifest-list&quot;: &quot;s3a://../manifest_list_delete.avro&quot;, <br>    &quot;schema-id&quot;: 0 <br>    } <br>],<br>...</pre><p>The engine directly goes to <em>manifest_list_delete.avro</em> and takes all entries into consideration ( <em>manifest_file_delete_m1.avro, manifest_file_update_m1.avro and manifest_file_delete_m0.avro)</em></p><blockquote><em>Please note that in more advanced scenarios, partition-level summaries stored in the manifest list may allow the engine to skip entire manifest files before reading them.</em></blockquote><p>Let’s say the engine iterates <em>manifest_file_delete_m0.avro</em>. All entries within the <em>manifest_file_delete_m0.avro</em> file whose <strong>status = 0 </strong>are eligible for reading and are included in the scan plan. However, the entry with <strong>status = 2</strong> is considered logically deleted; therefore, the underlying data file is not included in the scan plan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PcalArYi6kTtMJTx.png" /><figcaption>manifest_file_delete_m0.avro</figcaption></figure><p>While<em> manifest_file_delete_m1.avro</em> is iterated, the engine come across with an entry <strong>status = 1</strong>. Because it is also eligible for reading, it goes to the data file …/<em>00000–8.parquet</em> and brings the stored data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3TY2536UqONHr3J2.png" /><figcaption>manifest_file_delete_m1.avro</figcaption></figure><p>Last, <em>manifest_file_update_m1.avro </em>is iterated and data file …/<em>00000–4.parquet</em> is read and previously updated row, i.e <em>updated_customer_1 </em>is included.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sEHOBw46O_jNFB24.png" /><figcaption>manifest_file_update_m1.avro</figcaption></figure><p>Finally, the following file paths are read and the data inside becomes a part of the query result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/0*XK2rV9mhRzmqEsKI.png" /><figcaption>query result and associated file paths</figcaption></figure><p><strong>Final Thoughts</strong></p><p>This four-part series walked through how Apache Iceberg behaves across both the metadata and data layers during CRUD operations. It demonstrated exactly what happens under the internals by breaking down the changes in the manifest list, manifest file, and metadata file after each specific operation. In short, Apache Iceberg transforms a traditional data lake into a transactional, metadata-driven system, making it a core foundation of modern lakehouse architectures.</p><blockquote><em>This article series was authored by Sarp Bora Polat and Şerefcan Helvacıoğlu.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43f1c444a3c4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Apache Iceberg Handles CRUD Operations Across Metadata and Data Layers (2/4)]]></title>
            <link>https://medium.com/@serefcanhel/how-apache-iceberg-handles-crud-operations-across-metadata-and-data-layers-2-4-72484d5dd015?source=rss-c75dd195f433------2</link>
            <guid isPermaLink="false">https://medium.com/p/72484d5dd015</guid>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[data-engineer]]></category>
            <dc:creator><![CDATA[Serefcanhel]]></dc:creator>
            <pubDate>Thu, 26 Feb 2026 07:01:02 GMT</pubDate>
            <atom:updated>2026-03-03T07:02:49.071Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><a href="https://medium.com/@bora.polat/how-apache-iceberg-handles-crud-operations-across-metadata-and-data-layers-1-4-e0f2c6b9a8b2"><em>Previous: Architecture &amp; Metadata Foundations</em></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*3Eb1iqkMeqE-yqNFjkXwyQ.png" /></figure><h4>CREATE &amp; INSERT Commit</h4><p>In Part 1, it is explored Apache Iceberg’s architectural foundations. Now it’s time to observe them in action.</p><blockquote><em>Please note that CRUD operations in this series were executed using Copy-on-Write (COW) mode in Apache Iceberg.</em></blockquote><p>Second article walks through the very first commit: <strong>CREATE TABLE</strong> followed by <strong>INSERT</strong>.</p><p>As a first step `customer` table is created under the default schema with the following DDL.</p><pre>CREATE TABLE iceberg.default.customer (  <br>customer_id INT,  <br>customer_name STRING <br>);</pre><p>Then, following rows are inserted into the `customer` table with the following DML.</p><pre>INSERT INTO iceberg.default.customer ( <br>  customer_id, <br>  customer_name<br>)<br>VALUES<br> (1,  &#39;customer_1&#39;),<br> (2,  &#39;customer_2&#39;),<br> (3,  &#39;customer_3&#39;),<br> (4,  &#39;customer_4&#39;),<br> (5,  &#39;customer_5&#39;),<br> (6,  &#39;customer_6&#39;),<br> (7,  &#39;customer_7&#39;),<br> (8,  &#39;customer_8&#39;),<br> (9,  &#39;customer_9&#39;),<br> (10, &#39;customer_10&#39;),<br> (11, &#39;customer_11&#39;),<br> (12, &#39;customer_12&#39;),<br> (13, &#39;customer_13&#39;),<br> (14, &#39;customer_14&#39;)<br>;</pre><p><strong>Manifest File After CREATE &amp; INSERT</strong></p><p>At this stage, the table has no prior snapshots, so all files introduced by the INSERT operation form the initial snapshot.</p><p>After the INSERT commit, a total of 14 rows of data are written into 8 data files. A total of 12 rows are evenly distributed across 6 different folders, and each of which contains two distinct Parquet files. For other 2 rows, Iceberg created 2 different folders and placed each row into a separate Parquet file. Data inside data files can be found below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iyHuTEe4q6lcVNO6.png" /><figcaption>manifest_file_create_insert.avro</figcaption></figure><p><strong>Manifest List After CREATE &amp; INSERT</strong></p><p>The first manifest list is created with a single entry, which contains metadata about the snapshot with <strong>added_snapshot_id = 473683236412705278</strong>. As shown in the <strong>added_files_count</strong> field, eight data files were added, and these files contain total of 14 new rows as shown in <strong>added_rows_count </strong>field. As stated below, the list of the paths of the individual parquet files is located in <em>manifest_file_create_insert.avro</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ywizH6mSLlxmaRVQ.png" /><figcaption>manifest_list_create_insert.avro</figcaption></figure><p><strong>Metadata File After CREATE &amp; INSERT</strong></p><p>This metadata file represents the state of the Apache Iceberg table after the INSERT commit is applied. A <strong>last_sequence_number</strong> value of 1 indicates that only a single commit has occurred so far. The current snapshot contains 14 records distributed across 8 data files.</p><pre>{ <br> &quot;format-version&quot;: 2, <br> &quot;table-uuid&quot;: &quot;2acb7098-1481-4977-9c5d-667fd6988576&quot;, <br> &quot;location&quot;: &quot;s3a://lakestore/iceberg_warehouse/default/customer&quot;, <br> &quot;last-sequence-number&quot;: 1, <br> &quot;last-updated-ms&quot;: 1767620736289, <br> &quot;last-column-id&quot;: 2, <br> &quot;current-schema-id&quot;: 0, <br> &quot;schemas&quot;: [{&quot;type&quot;: &quot;struct&quot;,&quot;schema-id&quot;: 0,&quot;fields&quot;: [{&quot;id&quot;: 1,&quot;name&quot;: &quot;customer_id&quot;,&quot;required&quot;: false,&quot;type&quot;: &quot;int&quot;},{&quot;id&quot;: 2,&quot;name&quot;: &quot;customer_name&quot;,&quot;required&quot;: false,&quot;type&quot;: &quot;string&quot;}]}], <br> &quot;default-spec-id&quot;: 0, <br> &quot;partition-specs&quot;: [{&quot;spec-id&quot;: 0,&quot;fields&quot;: []}], <br> &quot;last-partition-id&quot;: 999, <br> &quot;default-sort-order-id&quot;: 0, <br> &quot;sort-orders&quot;: [{&quot;order-id&quot;: 0,&quot;fields&quot;: []}], <br> &quot;properties&quot;: { &quot;owner&quot;: &quot;sarp&quot;, &quot;history.expire.max-snapshot-age-ms&quot;: &quot;1000&quot;, &quot;write.metadata.previous-versions-max&quot;: &quot;10&quot;, &quot;write.parquet.compression-codec&quot;: &quot;zstd&quot;, &quot;write.metadata.delete-after-commit.enabled&quot;: &quot;true&quot;, &quot;write.object-storage.enabled&quot;: &quot;true&quot;, &quot;history.expire.min-snapshots-to-keep&quot;: &quot;1&quot;}, <br> &quot;current-snapshot-id&quot;: 4736832364012705278, <br> &quot;refs&quot;: {&quot;main&quot;: {&quot;snapshot-id&quot;: 4736832364012705278,&quot;type&quot;: &quot;branch&quot;}}, <br> &quot;snapshots&quot;: [<br>       {&quot;sequence-number&quot;: 1,&quot;snapshot-id&quot;: 4736832364012705278,&quot;timestamp-ms&quot;: 1767620736289,&quot;summary&quot;: { &quot;operation&quot;: &quot;append&quot;, &quot;spark.app.id&quot;: &quot;spark-eb4ccda57c0e4aa2a609860027e4dbb6&quot;, &quot;added-data-files&quot;: &quot;8&quot;, &quot;added-records&quot;: &quot;14&quot;, &quot;added-files-size&quot;: &quot;5966&quot;, &quot;changed-partition-count&quot;: &quot;1&quot;, &quot;total-records&quot;: &quot;14&quot;, &quot;total-files-size&quot;: &quot;5966&quot;, &quot;total-data-files&quot;: &quot;8&quot;, &quot;total-delete-files&quot;: &quot;0&quot;, &quot;total-position-deletes&quot;: &quot;0&quot;, &quot;total-equality-deletes&quot;: &quot;0&quot;, &quot;engine-version&quot;: &quot;3.5.7&quot;, &quot;app-id&quot;: &quot;spark-eb4ccda57c0e4aa2a609860027e4dbb6&quot;, &quot;engine-name&quot;: &quot;spark&quot;, &quot;iceberg-version&quot;: &quot;Apache Iceberg 1.9.2 (commit 071d5606bc6199a0be9b3f274ec7fbf111d88821)&quot;},&quot;manifest-list&quot;: &quot;s3a://…/manifest_list_create_insert.avro&quot;,&quot;schema-id&quot;: 0}<br> ], <br> &quot;statistics&quot;: [], <br> &quot;partition-statistics&quot;: [], <br> &quot;snapshot-log&quot;: [{&quot;timestamp-ms&quot;: 1767620736289,&quot;snapshot-id&quot;: 4736832364012705278}], <br> &quot;metadata-log&quot;: [{&quot;timestamp-ms&quot;: 1767620709356,&quot;metadata-file&quot;: &quot;s3a://lakestore/iceberg_warehouse/default/customer/metadata/00000-7fa4f983-194d-4407-9b30-b4747d157ac1.metadata.json&quot;}] <br>}</pre><p><strong>Data Files After CREATE &amp; INSERT</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/0*0OwF8Ftz4nSjw-FQ.png" /></figure><p>At this point, the Iceberg table has:</p><ul><li>One snapshot</li><li>Eight immutable data files</li><li>A metadata file that fully describes the table’s state</li></ul><h4>What’s Next?</h4><p>In <strong>Part 3</strong>, history will be introduced:</p><ul><li>UPDATE commit</li><li>DELETE commit</li><li>Manifest file status transitions</li><li>Snapshot parent relationships</li></ul><p><strong>👉 </strong>Continue to Part 3<strong>— </strong><a href="https://medium.com/@bora.polat/how-apache-iceberg-handles-crud-operations-across-metadata-and-data-layers-3-4-c6ccf6c4cb03"><strong>UPDATE &amp; DELETE Commit</strong></a></p><blockquote><em>This article series was authored by Sarp Bora Polat and Şerefcan Helvacıoğlu.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72484d5dd015" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>