<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vladislav Sidorovich on Medium]]></title>
        <description><![CDATA[Stories by Vladislav Sidorovich on Medium]]></description>
        <link>https://medium.com/@fenix900?source=rss-da7bf6ffa401------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*2iKisBQn2wSK9MmqAsXkJw.jpeg</url>
            <title>Stories by Vladislav Sidorovich on Medium</title>
            <link>https://medium.com/@fenix900?source=rss-da7bf6ffa401------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 03:26:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@fenix900/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Leveraging the Freedom of Open-Table Formats: A Guide to Cross-Cloud Migration]]></title>
            <link>https://medium.com/google-cloud/leveraging-the-freedom-of-open-table-formats-a-guide-to-cross-cloud-migration-7ac8a7a11fbd?source=rss-da7bf6ffa401------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ac8a7a11fbd</guid>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <dc:creator><![CDATA[Vladislav Sidorovich]]></dc:creator>
            <pubDate>Tue, 18 Aug 2026 13:10:32 GMT</pubDate>
            <atom:updated>2026-08-21T01:10:12.108Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iEG6m2Y6tyM1pdc3KW4m_g.jpeg" /></figure><h3>Introduction</h3><p>In 2026, open-table formats such as Apache Iceberg and Delta Lake have surged in popularity, rapidly becoming the de-facto industry standard for modern data analytics.</p><p>The most critical value proposition of these formats lies right in the name: “<strong>open</strong>.” This term represents a massive promise to businesses — the ultimate freedom from vendor lock-in. By decoupling storage from compute, open formats guarantee that organizations can choose any cloud provider, query engine, or data visualization platform to work with their datasets.</p><p>If a specific tool or vendor no longer meets your business needs, the market offers many many alternatives. You can seamlessly swap out your compute layer or BI tools while your business-critical data remains entirely intact and accessible. This flexibility is a game-changer for long-term business strategies. Furthermore, it perfectly aligns with the goals of technical stakeholders; data architects, data engineers, and business analysts can continuously evolve their data stack without the friction of fundamentally restructuring the underlying data.</p><p>But what happens when the day comes to actually act on this promise? What does it truly take to pick up your data and move it to a new environment?</p><p>In this article, we will explore the technical realities and strategic approaches of cross-cloud migration. We will look past the surface-level assumptions and dive deep into The Complexities of Migrating Open-Table Formats: Beyond File Copying.</p><blockquote><strong>Note</strong>: For the scope of this article, we will also include Hive parquet tables in our analysis, specifically focusing on them as a migration source.</blockquote><h3>Breaking Down the Migration: A High-Level Overview</h3><p>To migrate open-table formats between clouds regardless of the specific tools you use, every cross-cloud migration boils down to four mandatory, high-level steps.</p><h4>1. Discover</h4><p>Before you can move anything, you must know exactly what you have. This step involves taking a comprehensive inventory of your source data. How many namespaces and tables are in your catalog? What are the exact physical storage locations of these tables? Crucially, you also need to identify the current snapshot or version of each table to establish a reliable baseline for the migration.</p><h4>2. Move Bytes</h4><p>It doesn’t matter how many abstraction layers sit in front of your data — whether it’s S3/GCS buckets, Parquet files, Iceberg views, Spark clusters, or analytic UIs. At the very bottom of the stack, your data is bytes stored on physical disks in a cloud data center. The core of migration is moving petabytes of these bytes from disks in one cloud to disks in another. Depending on your strategy, this massive data transfer happens either as a direct physical copy or alongside a logical transformation.</p><h4>3. Conversion (Optional but Common)</h4><p>Often, a cloud migration presents the perfect opportunity to modernize your data stack. This step is required if you are simultaneously migrating from one table format to another — for example, upgrading legacy Hive Parquet tables to Apache Iceberg or Delta Lake while moving to the new cloud environment.</p><h4>4. Registration and Continuous Sync</h4><p>Finally, the migrated tables must be registered in the target catalog so your query engines and business users can actually start working with them. However, it is rarely a simple, one-time registration.</p><h3>Complexity</h3><p>While each individual step from our high-level overview might seem simple and straightforward in isolation, the reality of executing them together at a petabyte scale introduces complexity.</p><p>In the real world, no one puts a business on pause just because the data engineering team is migrating tables. While you are moving petabytes of historical data, new data is constantly being written to the source tables. Therefore, “registration” actually means establishing a continuous data refresh loop. Your pipeline must constantly sync the new data diffs from the source to the target — a process that can run for weeks or even months until the final cutover is complete.</p><p>Because of this, you aren’t just executing these migration steps once; you are forced to build a robust, fault-tolerant pipeline to continuously repeat them.</p><p>Furthermore, the exact way a particular step is implemented strictly limits the options available for downstream steps. For instance, consistently copying millions of files and petabytes of data across networks is a known technical problem that can be solved with existing infrastructure tools. However, this copying must be done incrementally, and the target table state must remain perfectly consistent after every sync. This forces data engineering teams to deeply intertwine raw bytes copying mechanisms with format-specific logic.</p><p>When you add table format conversion into the mix (e.g., migrating from Delta to Iceberg alongside the cloud move), you face an open, industry-wide challenge. The combination of satisfying incremental sync requirements while preserving the original table history across all types of SQL operations (INSERTs, UPDATEs, and DELETEs) makes format conversion exceptionally difficult.</p><p>To solve this overarching problem, technical strategies generally fall into one of two core architectural patterns:</p><ol><li>[<strong>CTAS</strong>] <em>Logical Tier Conversion</em>: Operations on the table data are performed based on its logical representation. This typically involves using a compute engine to read the source data as rows and rewrite it into the target format.</li><li>[<strong>In-place</strong>] <em>Physical Tier Conversion</em>: Operations are handled at the files layer. This involves directly manipulating and converting the physical data files (like Parquet) and metadata representations directly on the disk, without passing the data through a heavy SQL compute layer.</li></ol><h3>Technical strategies</h3><h4>Discovery</h4><p>Before you move a single byte of data, you must build an inventory of your existing data landscape (tables/locations/versions). A common practice is moving tables sequentially in logical groups. Defining these groups is specific to each migration, which is exactly why you must collect metadata about your existing data environment before taking any action.</p><p>The discovery process begins by connecting to your source data catalog (like Unity Catalog, HMS, Apache Polaris and so on) and running a traversal algorithm (like DFS or BFS). Because formats like Apache Iceberg utilize nested namespaces, your catalog is essentially structured as a tree, with your actual tables sitting as the leaves of the tree.</p><p>As you traverse this tree, you extract the vital table metadata required for the subsequent migration steps. You will need to store this information somewhere for quick access. According to a recent <a href="https://dl.acm.org/doi/abs/10.1145/3722212.3724459">Databricks white paper</a>, 100% of analyzed customer metadata sets are less than <strong>100 MB</strong>, with 90% being under ~<strong>10 MB</strong>. Given these small footprints, simply running a PySpark notebook and saving the metadata inventory to a single file (like a CSV or JSON) is acceptable. If you prefer traditional databases for this metadata it’s also ok.</p><blockquote>There is a critical catch: this step cannot be a one-time operation. It must be executed continuously.</blockquote><p>Remember that active tables are constantly being updated. Imagine you discover a table named Transactions at version `<strong>123</strong>` and successfully migrate it to your target cloud. Meanwhile, production systems continue writing to the source, advancing it to version `<strong>456</strong>`. All of that new data remains completely invisible to your target cloud until you re-run the discovery process, fetch the new metadata, and trigger the migration flow for the updated table version.</p><p>For every iteration of the discovery process, your goal is to capture four core pieces of data for each table:</p><ol><li>Fully Qualified Table Name: The complete path identifying the table (e.g., namespace1/namespace2/my_table).</li><li>Table Type: The specific open format being used (e.g., Apache Iceberg, Delta Lake, or Hive)</li><li>Table Location: The absolute physical storage path on your cloud provider (e.g., the specific S3 URI or GCS bucket path).</li><li>Table Version and Properties: The path or identifier for the latest table snapshot, along with any relevant key-value string pairs that define the table’s configuration.</li></ol><h4>Logical data transfer (CTAS)</h4><p>This strategy is deeply rooted in the concept of <em>CTAS</em> (Create Table As Select) — a common SQL command used to create a new table and populate it with data derived from the results of a <em>SELECT</em> statement.</p><p>In this strategy, the migration happens in two distinct phases:</p><ol><li><strong>Read</strong>: The query engine’s source client reads the table data (e.g., a Delta Lake table located in AWS S3 or Azure ADLS) into memory as a logical representation, such as a Spark DataFrame.</li><li><strong>Write</strong>: The engine’s target client writes this DataFrame to a new physical location (e.g., Google Cloud Storage) in the designated output format.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RzG7TM8Uogvr3ls2WVRnwA.png" /></figure><p>This process effectively rewrites all the underlying data files from scratch and simultaneously generates the corresponding metadata (like Apache Iceberg manifests), creating a brand-new, independent table. Because the data is loaded into memory, you also have full control on the logical tier — you can easily inject additional data processing logic, change partition schemes, or update compression codecs without changing the high-level architecture.</p><p>The conversion in this strategy means that a query engine will read all source data into a logical table and write this logical table in a target format.</p><pre>spark.read()<br>  .format(&quot;delta&quot;)<br>  .option(&quot;versionAsOf&quot;, deltaSnapshotVersion)<br>  .load(deltaSource)<br>  // ... any additional data processing ...<br>  .writeTo(icebergTableName)<br>  .using(&quot;iceberg&quot;);</pre><h4>The Incremental Challenge and DML Operations with CTAS</h4><p>While this architecture is incredibly versatile, the complexity spikes when you need to handle continuous, incremental syncs.</p><p>If the source table only receives new records, <em>INSERT</em> operations can be safely and easily appended to the target table. However, if the source data includes <em>UPDATE</em> or <em>DELETE</em> operations, simply appending data will result in duplicates and inconsistencies. To process these changes safely on the logical tier, you must rely on a <em>MERGE INTO</em> statement:</p><pre>MERGE INTO target t USING source s<br>  ON t.key = s.key  -- a condition<br>  WHEN MATCHED AND s._change_type = &#39;delete&#39; THEN DELETE<br>  WHEN MATCHED THEN UPDATE SET *<br>  WHEN NOT MATCHED AND s._change_type = &#39;insert&#39; THEN INSERT *</pre><p>This introduces significant technical hurdles for a generalized migration tool. The tricky part is preparing a generic match condition (<strong>t.key = s.key</strong>) that works reliably across all tables and edge cases. Furthermore, handling schema evolution during long-running incremental executions requires manual, custom logic to ensure the pipeline doesn’t break when a column is added or dropped at the source.</p><h4><strong>The Cost of CTAS flexibility</strong></h4><p>The logical data transfer strategy is completely agnostic to table format versions, features, and cloud storage providers. It allows you to reuse native format clients, guaranteeing high compatibility.</p><p>This freedom comes with a price:</p><ol><li>Complex Orchestration: Incremental updates require custom merging logic per table, manual checkpoint handling, and additional data validation steps to ensure integrity.</li><li>Expensive Retries: If a petabyte-scale job fails near the end, a full retry is highly expensive in terms of both compute resources and outbound network egress traffic (between clouds).</li><li>Schema evolution: handling of schema evolution can be tricky and error prone.</li></ol><h4><strong>The “All-in-One CTAS” Temptation</strong></h4><p>Because the CTAS approach relies so heavily on query engines like Spark, it makes it entirely possible to combine all the migration steps — from discovery to registration — into a single script or pipeline phase.</p><p>While this sounds great on paper, I strongly recommend resisting the temptation. For very small, ad-hoc migrations, this monolithic approach might be the way to go. But at an enterprise scale, combining these steps together severely limits your technical abilities and tuning options for each distinct phase. When migrating petabytes of data, you need the architectural freedom to independently scale, monitor, and retry the discovery, data movement, and registration components.</p><h4>Physical data transfer</h4><p>Physical data transfer is a well-known problem that is completely agnostic to open-table formats. While many different tools and services can be used to solve this, at the petabyte scale, simple scripts and CLI tools will struggle with edge cases and ultimately prove insufficient. Fortunately, major cloud providers offer dedicated, scalable data transfer services (often at no cost) specifically designed to move massive volumes of plain data.</p><p>However, because these services only transfer raw files, post-processing is absolutely required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-HmPbdh9y1qyh8rUpNoQdw.png" /></figure><p>The complexity of this post-processing depends heavily on the format: Delta Lake and Hive tables can generally be moved to a new location or cloud and stay perfectly consistent. In contrast, Apache Iceberg metadata utilizes absolute URIs, meaning a path rewrite procedure is a mandatory step after the physical transfer. Furthermore, simply landing the bytes in target cloud storage does not make them accessible; no target system will know this data even exists. Therefore, your post-processing must include registering the tables in a target catalog. Cloud transfer services typically operate by listing and copying full directories based on storage prefixes. This behavior differs fundamentally from how Delta Lake or Iceberg logically handle their table structures. The physical copy may runs concurrently with active write transactions or VACUUM operations on the source, the transfer service might pick up uncommitted files (from pending or failed transactions) or copy orphan files. To prevent your target table from becoming corrupted, you must either handle these scenarios carefully in your post-processing step, or you must collect an exact list of precise file URIs during the discovery step to avoid bulk directory copying altogether.</p><p>In contrast to CTAS, physical file transfer handles incremental updates beautifully. Because open-table formats like Delta, Iceberg, and Hive are built on top of immutable files, every <em>INSERT</em>/<em>UPDATE</em>/<em>DELETE</em> operation and the table schema changes generates new <strong>immutable</strong> files. During the next scheduled migration job, the transfer service simply identifies and copies these newly generated immutable files to the target directory, making continuous synchronization highly efficient.</p><h4>Conversion</h4><p>Sometimes during a cloud migration, you will want to do more than just move bytes — you will want to take the opportunity to modernize your stack by moving from one table format to another.</p><p>If you rely on the logical CTAS approach, you get these conversion capabilities out of the box; you simply specify the new target format in your query engine’s write statement. However, if you choose the physical data transfer strategy, you will need to explicitly build format conversion logic into your post-processing step.</p><p>Today, Hive tables are essentially never the target format for a migration, but they are very frequently the source. Because Hive tables have no rich metadata tier, converting from it is relatively straightforward. For Apache Iceberg targets, you can utilize the <a href="https://iceberg.apache.org/docs/latest/spark-procedures/#add_files">add_files</a> <strong>Spark procedure</strong>, or you can run the command below if you are migrating to Delta Lake.</p><pre>CONVERT TO DELTA parquet.`s3://my-bucket/parquet-data`;<br><br>CONVERT TO DELTA iceberg.`gcs://my-bucket/my-namespaces/iceberg-data`;</pre><p>With Hive tables you just need to be careful with partitioning. You must collect the Hive table’s partitions during your initial discovery phase and explicitly specify them during the conversion process.</p><blockquote><strong>Note</strong>: Size of Hive partitions metadata can be in GBs. It’s often bigger then Iceberg or Delta tables metadata in catalogs.</blockquote><p>Converting between Delta Lake and Apache Iceberg is a much trickier scenario. Delta Lake exclusively supports Parquet for its data files. This means that if your Iceberg tables use a different data file format (which is uncommon, but possible), your only conversion option is the compute-heavy CTAS approach.</p><p>However, if both your source and target formats utilize Parquet data files, it opens up the opportunity for “in-place” metadata conversion. This approach means you do not touch or rewrite the heavy data files at all; instead, you read the source table’s metadata snapshot by snapshot and generate new metadata files in the target format. While this physical-tier metadata conversion is highly efficient and often the preferable route, it relies deeply on low-level protocol implementation details. Navigating the nuances of this “in-place” conversion is a complex topic that requires its own dedicated article.</p><h4>Registration in a target catalog</h4><p>Registration might seem like the simplest step in the migration flow, but it still carries critical nuances. Modern catalogs for Delta Lake and Iceberg provide a straightforward registration <a href="https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L971-L992">endpoint</a> to add new tables, which is exactly the functionality you need.</p><p>However, it is rarely a simple, one-time registration. Because active pipelines must constantly sync new data diffs from the source to the target, this incremental migration process can run for weeks or even months until the final cutover is complete. This means you must continually re-register the table in the target catalog — perhaps daily or hourly — to make each new version visible. Simply landing the new bytes in a target cloud storage is not enough.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JPsWFdzbXzPjSnvZkKmmnA.png" /></figure><p>You must handle this continuous registration phase very carefully. Once a migrated table is registered in the target catalog, it becomes visible to query engines and business users so they can actually start working with it. But crucially, it also becomes writable.If any user or application writes data to this table through the new catalog while your migration pipeline is still syncing updates, you will create a write-write conflict. This conflict can easily lead to data corruption and the loss of either the migrated data or the newly written records. To mitigate this risk, it is a highly recommended best practice to keep tables strictly read-only in the destination catalog during the active migration phase.</p><p>Finally, beware of automated catalog features. Some target catalogs automatically perform background optimizations, such as data compaction, which inherently create new table versions. These automated background jobs will also trigger write-write conflicts against your ongoing migration pipeline. Therefore, it is a very good idea to explicitly disable background optimization jobs in your target catalog until the migration for that specific table is fully completed.</p><h3>Real-word experience</h3><p>From my personal experience, the “<strong>Discovery</strong>” phase is frequently underestimated. Source catalogs are often locked behind complex authorizations like Kerberos or restricted networks. Instead of fighting firewall policies, it is usually simpler to extract your table metadata into a custom JSON or CSV file, upload it directly to cloud storage, and trigger the migration from there.</p><p>For moving the actual bytes, physical data transfer is generally preferable to CTAS. Even if CTAS is required for format conversion, the most robust pattern is a two-phase approach: first, physically copy the raw files to the target cloud, then run the CTAS conversion entirely within that new environment.</p><p>If you are migrating to Google Cloud, the <a href="https://docs.cloud.google.com/storage-transfer/docs/overview">Storage Transfer Service</a> (STS) is an excellent, highly scalable tool for this physical copy phase.</p><h4>Delta to Iceberg</h4><p>When migrating from Delta Lake to Apache Iceberg, Delta’s Uniform feature (<a href="https://docs.databricks.com/aws/en/iceberg/iceberg-v3">enableIcebergCompatV3</a>) is a helpful starting point. However, it is not a complete solution. Because the data physically moves to a new bucket, you must still perform an explicit path rewrite after the cross-cloud transfer to the new URIs.</p><p>If you are exploring “in-place” metadata conversion, <a href="https://github.com/apache/incubator-xtable">Apache XTable</a> might initially look exactly like what you need. In practice, however, it currently lacks support for recent Delta versions and Iceberg v3 — specifically struggling with deletion vectors. It can still be a suitable option for older (iceberg v2) tables.</p><p>It is worth keeping an eye on a standalone tool currently in progress directly within the Apache Iceberg repository (PR #<a href="https://github.com/apache/iceberg/pull/15407">15407</a>), which aims to natively generate Iceberg metadata for Delta tables and is definitely worth considering.</p><h4><strong>Write-write conflicts</strong></h4><p>Write-write conflicts occasionally happen during long-running incremental syncs, avoid cleaning up your source data too aggressively — you may need those original files for recovery.</p><h3>Summary</h3><p>Migrating open-table formats across clouds is a powerful way to achieve true freedom from vendor lock-in. However, as we have explored, realizing this promise requires looking far beyond a simple file copy. A successful migration demands a continuous loop of discovering metadata, executing massive data transfers, handling format conversions, and registering tables in a target catalog while the business continues to operate.</p><p>I hope this article clarifies the full migration process for you and highlights the critical edge cases that you need to take into consideration. I truly hope that clarifying these technical realities helps you to properly plan, architect, and estimate your end-to-end migration journey.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ac8a7a11fbd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/leveraging-the-freedom-of-open-table-formats-a-guide-to-cross-cloud-migration-7ac8a7a11fbd">Leveraging the Freedom of Open-Table Formats: A Guide to Cross-Cloud Migration</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrate Apache Iceberg tables from Hive Metastore to BigLake Iceberg REST catalog]]></title>
            <link>https://medium.com/google-cloud/migrate-apache-iceberg-tables-from-hive-metastore-to-biglake-iceberg-rest-catalog-adb27edcccac?source=rss-da7bf6ffa401------2</link>
            <guid isPermaLink="false">https://medium.com/p/adb27edcccac</guid>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[apache-hive]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <dc:creator><![CDATA[Vladislav Sidorovich]]></dc:creator>
            <pubDate>Fri, 09 Jan 2026 10:12:55 GMT</pubDate>
            <atom:updated>2026-01-12T21:09:54.395Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_-YZQzPR56srbmesshmuCw.png" /></figure><p>This blog post is intended for Data Engineers currently managing Apache Iceberg tables within a Hive Metastore (HMS) on Google Cloud who have already decided to migrate to Google’s BigLake Metastore.</p><h3>Problem definition</h3><p>The objective is straightforward: move the table management from a legacy Dataproc Metastore (or self-managed HMS) to the BigLake Iceberg REST catalog.</p><p>Crucially, this migration must be strictly a <strong>metadata operation</strong>. We want to register existing data files in the new catalog without rewriting the underlying data or metadata files, ensuring zero data duplication and minimal migration time.</p><p>In this tutorial, we will execute the migration using serverless Spark within BigQuery Notebooks.</p><p>Architecture Components:</p><ol><li><strong>Hive Metastore v3 (HMS): </strong>A legacy metastore running on GCP within a Virtual Private Cloud (VPC), currently serving as the catalog for the source Iceberg tables.</li><li><strong>Google Cloud Storage (GCS):</strong> The object storage where the actual Iceberg tables are stored.</li><li><strong>BigLake Iceberg REST catalog (BigLake):</strong> The target Google-managed Iceberg REST catalog.</li><li><strong>BigQuery Notebooks:</strong> The serverless Spark environment where we will execute the migration logic.</li></ol><h3>Connect to HMS and BigLake from BigQuery Notebooks</h3><p>To perform this migration, we need a compute environment that can “speak” to both the legacy world (HMS) and the modern world (BigLake) simultaneously. We will use BigQuery Notebooks with Spark, which provides a serverless PySpark environment directly within the BigQuery console.</p><p>To get started, navigate to the BigQuery section in the Google Cloud Console and select “Notebooks with Spark”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_UmeWyKezeZRcgZ3ok1Wmg.png" /></figure><p>The core of this migration script relies on configuring two distinct Iceberg catalogs within a single Spark session. This allows us to query metadata from the source and register it in the destination without restarting the context.</p><pre>import google.auth<br>import pyspark<br>from google.cloud.dataproc_spark_connect import DataprocSparkSession<br>from google.cloud.dataproc_v1 import Session<br><br>gcp_project_id = google.auth.default()[1] # Get current GCP project id<br>biglake_uri = &quot;https://biglake.googleapis.com/iceberg/v1/restcatalog&quot;<br>hms_uri = &quot;thrift://10.128.0.109:9083&quot;<br>gcs_bucket = &quot;gs://my_gcs_bucket&quot;<br><br>catalog_hms = &quot;iceberg_hms&quot;<br>catalog_google = &quot;iceberg_google&quot;<br><br>session = Session()<br>session.runtime_config.properties = {<br>  &quot;spark.sql.extensions&quot;:&quot;org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions&quot;,<br><br>  f&quot;spark.sql.catalog.{catalog_hms}&quot;:&quot;org.apache.iceberg.spark.SparkCatalog&quot;,<br>  f&quot;spark.sql.catalog.{catalog_hms}.type&quot;:&quot;hive&quot;,<br>  f&quot;spark.sql.catalog.{catalog_hms}.uri&quot;:hms_uri,<br><br>  f&quot;spark.sql.catalog.{catalog_google}&quot;:&quot;org.apache.iceberg.spark.SparkCatalog&quot;,<br>  f&quot;spark.sql.catalog.{catalog_google}.type&quot;:&quot;rest&quot;,<br>  f&quot;spark.sql.catalog.{catalog_google}.rest.auth.type&quot;:&quot;org.apache.iceberg.gcp.auth.GoogleAuthManager&quot;,<br>  f&quot;spark.sql.catalog.{catalog_google}.uri&quot;:biglake_uri,<br>  f&quot;spark.sql.catalog.{catalog_google}.warehouse&quot;:gcs_bucket,<br>  f&quot;spark.sql.catalog.{catalog_google}.header.x-goog-user-project&quot;:gcp_project_id,<br>  f&quot;spark.sql.catalog.{catalog_google}.io-impl&quot;:&quot;org.apache.iceberg.gcp.gcs.GCSFileIO&quot;,<br>}<br><br>spark = (<br>   DataprocSparkSession.builder<br>     .appName(&quot;HMS and BigLake connection&quot;)<br>     .dataprocSessionConfig(session)<br>     .getOrCreate()<br>)<br><br>spark.sql(f&quot;show namespaces in {catalog_hms}&quot;).show(truncate=False)<br>spark.sql(f&quot;show namespaces in {catalog_google}&quot;).show(truncate=False)</pre><p>Since your Hive Metastore likely resides within a private VPC you will need to add the following config with `subnetwork_name`, for more details please refer to the documentation at <a href="https://cloud.google.com/bigquery/docs/use-spark">https://cloud.google.com/bigquery/docs/use-spark</a>.</p><pre>session.environment_config.execution_config.subnetwork_uri = &quot;&lt;subnetwork_name&gt;&quot;</pre><h3>Create the target Namespace in BigLake</h3><p>Now that our Spark session is connected to both catalogs, we can begin setting up the migration structure. Since our goal is a “metadata-only” migration — leaving the heavy data files untouched — we must ensure that the <strong>namespace</strong> for these tables exists in BigLake and points to the correct Google Cloud Storage location.</p><p>First, we inspect the source namespace in Hive Metastore to retrieve its properties, specifically the <em>location</em>.</p><pre>namespace_name = &quot;notebook_namespace_1&quot;<br><br>spark.sql(f&quot;describe namespace {catalog_hms}.{namespace_name}&quot;).show(truncate=False)<br><br>+--------------+---------------------------------------------------------------------+<br>|info_name     |info_value                                                           |<br>+--------------+---------------------------------------------------------------------+<br>|Catalog Name  |iceberg_hms                                                          |<br>|Namespace Name|notebook_namespace_1                                                 |<br>|Location      |gs://my_gcs_bucket/iceberg/hms-namespaces/notebook_namespace_1       |<br>|Owner         |root                                                                 |<br>+--------------+---------------------------------------------------------------------+</pre><p>Next, we create the corresponding namespace in BigLake.</p><p>Important! The <em>LOCATION</em> property in the new BigLake namespace must match the location where your data currently resides. While you <em>can</em> give the namespace a different name in BigLake if needed (e.g., if <em>finance</em> already exists, you could call it <em>finance_migrated</em>), the underlying storage path must align. Iceberg enforces strict checks between the URIs defined in metadata files and the actual file paths; mismatches here can cause read errors later.</p><p>In the code below, we manually take the location retrieved from HMS and create the namespace in BigLake.</p><pre>spark.sql(f&quot;&quot;&quot;<br>    CREATE NAMESPACE IF NOT EXISTS {catalog_google}.{namespace_name}<br>    COMMENT &#39;Namespace for HMS tables&#39;<br>    LOCATION &#39;gs://my_gcs_bucket/iceberg/hms-namespaces/{namespace_name}&#39;<br>&quot;&quot;&quot;)</pre><p>For a production migration involving dozens of namespaces, you would likely automate this by looping through all HMS namespaces and programmatically copying over all properties (key-value pairs) to BigLake. For simplicity in this guide, we will focus on migrating a single namespace.</p><h3>Find Iceberg tables in the source Namespace</h3><p>A Hive Metastore namespace is often a mixed environment. It may contain legacy Hive tables (Parquet, ORC, CSV) alongside Apache Iceberg tables. Since our migration method relies on Iceberg-specific procedures, we must explicitly filter for Iceberg tables to avoid errors.</p><p>There are multiple ways to verify if a table is an Iceberg table:</p><ol><li><em>SHOW TBLPROPERTIES</em>: Inspect `format` property, can be iceberg/parquet, iceberg/avro, and so on</li><li><em>DESCRIBE EXTENDED</em>: Check `Provider` field, it must be `iceberg`.</li></ol><pre>def is_iceberg_table(spark: DataprocSparkSession, full_table_name: str) -&gt; bool:<br>    try:<br>        props_rows = spark.sql(f&quot;SHOW TBLPROPERTIES {full_table_name} (&#39;format&#39;)&quot;).collect()<br>        <br>        for row in props_rows:<br>            col = row[0]<br>            val = row[1]<br>            val = val.lower() if val is not None else &quot;&quot;<br>            <br>            if &quot;format&quot; == col.lower() and val.startswith(&quot;iceberg&quot;):<br>                return True<br>                <br>        return False<br><br>    except Exception as e:<br>        print(f&quot;   [!] ERROR accessing {full_table_name}: {e}&quot;)<br>        return False</pre><h3>Select current metadata location for Iceberg table</h3><p>To register a table in the new catalog, the full path to the specific <strong><em>metadata.json</em></strong> file is required. While Iceberg supports <em>time-travel</em> (allowing you to register a specific historical version), our goal here is to migrate the table in its <strong>current</strong> state.</p><p>You might be tempted to simply query the <em>metadata_log_entries</em> system table and grab the record with the latest timestamp. However, this approach is flawed. If a <em>rollback_to_snapshot</em> operation was ever performed on the source table, the “latest” metadata file by timestamp might actually point to a discarded timeline rather than the current active state.</p><p>A more robust way to identify the correct metadata file is a two-step process:</p><ol><li><strong>Identify the Snapshot:</strong> Retrieve the current-snapshot-id from the table’s properties.</li><li><strong>Map to Metadata:</strong> Query the metadata_log_entries table to find the specific file (URI) associated with that snapshot id.</li></ol><p><strong>Note</strong>: It is important to handle edge cases where a table exists but contains no data. In this scenario, the current-snapshot-id will return none (or null).</p><pre>from typing import Optional<br><br>def select_current_metadata_location(spark: DataprocSparkSession, full_table_name: str) -&gt; Optional[str]:<br>    try:<br>        props_rows = spark.sql(f&quot;SHOW TBLPROPERTIES {full_table_name} (&#39;current-snapshot-id&#39;)&quot;).collect()<br>        <br>        for row in props_rows:<br>            col = row[0]<br>            current_snapshot = row[1]<br>            <br>            if &quot;current-snapshot-id&quot; == col.lower():<br>                if current_snapshot is None or &quot;none&quot; == current_snapshot.lower():<br>                    print(&quot; (table is empty)&quot;)<br>                    <br>                    log_rows = spark.sql(f&quot;SELECT file FROM {full_table_name}.metadata_log_entries&quot;).collect()<br>                    return log_rows[0][0] if log_rows else None<br>                <br>                snapshot_rows = spark.sql(f&quot;&quot;&quot;<br>                    SELECT file <br>                    FROM {full_table_name}.metadata_log_entries <br>                    WHERE latest_snapshot_id = {current_snapshot}<br>                &quot;&quot;&quot;).collect()<br>                <br>                return snapshot_rows[0][0] if snapshot_rows else None<br>                <br>        return None<br><br>    except Exception as e:<br>        print(f&quot;   [!] ERROR accessing {full_table_name}: {e}&quot;)<br>        return None</pre><h3>Register tables in BigLake</h3><p>We now have all the necessary components: a connection to both catalogs, a target namespace in BigLake, and a reliable method to identify the correct metadata file for any given table.</p><p>In this step, we bring it all together in a loop. The script iterates through every table in the source HMS namespace, applies our Iceberg filter, and executes the migration.</p><p>To register tables we will use <a href="https://iceberg.apache.org/docs/latest/spark-procedures/#register_table">register_table</a>, the procedure creates only a pointer to the table and doesn’t touch any data or other table files.</p><p>In this example, we assume a clean migration where the target namespace is empty. If a table with the same name already exists in the target, the registration will fail. While you <em>could</em> rename the table during registration (e.g., <em>register_table(‘target_db.new_table_name’, …)</em>), we will keep the names identical to ensure transparent application switching.</p><pre>print(f&quot;HMS [NAMESPACE]: {namespace_name}&quot;)<br><br># Schema of SHOW TABLES: | namespace | tableName | isTemporary |<br>tables = spark.sql(f&quot;SHOW TABLES IN {catalog_hms}.{namespace_name}&quot;).collect()<br><br>if not tables:<br>    print(&quot;   (No tables found)&quot;)<br><br>for table_row in tables:<br>    table_name = table_row[1]<br>    is_temporary = table_row[2]<br><br>    if is_temporary:<br>        continue<br><br>    full_table_name = f&quot;{catalog_hms}.{namespace_name}.{table_name}&quot;<br><br>    if is_iceberg_table(spark, full_table_name):<br>        metadata_location = select_current_metadata_location(spark, full_table_name)<br>        <br>        if metadata_location:<br>            print(f&quot;Table {full_table_name} has metadata location: {metadata_location}&quot;)<br><br>            target_table_full_name = f&quot;{catalog_google}.{namespace_name}.{table_name}&quot;<br>            <br>            register_sql = f&quot;&quot;&quot;<br>                CALL {catalog_google}.system.register_table(<br>                    table =&gt; &#39;{target_table_full_name}&#39;, <br>                    metadata_file =&gt; &#39;{metadata_location}&#39;<br>                )<br>            &quot;&quot;&quot;<br>            <br>            spark.sql(register_sql).show(truncate=False)</pre><p>Now we can execute the entire BigQuery Notebook. Once it succeeded, verify that all the tables are available in the BigLake Iceberg REST catalog.</p><pre>tables_count = spark.sql(f&quot;show tables in {catalog_google}.{namespace_name}&quot;).count()<br>print(f&quot;Migrated tables: {tables_count}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uAzk8HYxof_hmOol9X5jDA.png" /></figure><p>Moreover, all the migrated tables are now accessible not only from Spark but directly from BigQuery Studio (see the<a href="https://www.google.com/search?q=https://cloud.google.com/biglake/docs/iceberg-tables"> official documentation</a> for details).</p><h3>Handling long-running migration</h3><p>While the PySpark script above might register 1,000 tables in a matter of minutes, a real-world migration involves much more than just catalog pointers. You also need to migrate upstream ingestion pipelines and downstream BI queries. Consequently, the overall migration project can span days, weeks, or even months.</p><p>During this migration period, your tables will exist in both the HMS and BigLake Iceberg REST catalog. This introduces a critical operational risk.</p><p>If you write to a table via HMS, the HMS snapshot advances. If you write to the same table via BigLake, the BigLake snapshot advances. These changes will <strong>not</strong> automatically sync. The result is a “split brain” scenario where the catalogs point to diverging versions of the data.</p><p>Best Practices for Coexistence:</p><ul><li><strong>Single writer:</strong> Ensure that at any given time, only <em>one</em> catalog acts as the writer for a specific table.</li><li><strong>Minimize transition window:</strong> Plan your pipeline cutovers to minimize the time a table is active in both systems.</li></ul><p>If you must continue writing to HMS during the migration but need those changes reflected in BigLake for testing or read-only consumers, you cannot simply re-run the registration script. As of January 2026, the <a href="https://iceberg.apache.org/docs/latest/spark-procedures/#register_table">register_table</a> procedure does not support an <a href="https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L3637">overwrite</a> parameter. This parameter is in the Iceberg REST catalog specification, so it should be added eventually to the procedure.</p><p>For now, it will be required to drop the table and register it again with a new <strong><em>metadata.json</em></strong>.</p><h3>Conclusion</h3><p>In this article was demonstrated a <strong>metadata-only migration</strong> from Hive Metastore to BigLake using serverless Spark. This approach allows you to “lift and shift” your catalog without rewriting a single data file.</p><p>With your tables now in the BigLake REST catalog, you gain native integration with BigQuery and a fully managed environment. Just ensure you strictly manage the “split-brain” risk during the transition.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=adb27edcccac" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/migrate-apache-iceberg-tables-from-hive-metastore-to-biglake-iceberg-rest-catalog-adb27edcccac">Migrate Apache Iceberg tables from Hive Metastore to BigLake Iceberg REST catalog</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrating Apache Iceberg Tables to GCS and BigLake Iceberg REST Catalog]]></title>
            <link>https://medium.com/google-cloud/migrating-apache-iceberg-tables-to-gcs-and-biglake-iceberg-rest-catalog-c68399d7a03d?source=rss-da7bf6ffa401------2</link>
            <guid isPermaLink="false">https://medium.com/p/c68399d7a03d</guid>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[google-big-lake-metastore]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Vladislav Sidorovich]]></dc:creator>
            <pubDate>Wed, 05 Nov 2025 02:54:44 GMT</pubDate>
            <atom:updated>2025-11-05T02:54:44.502Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EHwc-Lwnbynjr9ozx9SMZQ.png" /></figure><p>This blog post describes the process and steps required to migrate <strong>Apache Iceberg</strong> tables to Google Cloud in order to leverage the unified data management capabilities provided by BigLake Metastore.</p><h3>Problem definition</h3><p>We live in a data-driven world, where organizations already manage Petabytes and Exabytes of data, relying on it to make critical business decisions. The Apache Iceberg table format has become widely adopted, to handle this scale and complexity as an open-source standard for data storage. Iceberg has broad support across popular query engines like Spark, Trino, Flink, and Presto. These Iceberg tables are typically registered and governed using metastores such as the Hive Metastore (HMS), Polaris and others.</p><p>BigQuery and Dataproc have been continually increasing their support for Apache Iceberg. This growing integration is a major driver for choosing to migrate existing Iceberg tables to Google Cloud.</p><p><strong>BigLake Metastore</strong> now offers GA support for the Apache Iceberg REST Catalog. This capability significantly simplifies the process of integrating Iceberg tables directly into the Google Cloud ecosystem, enabling seamless interoperability with GCP services like BigQuery for example.</p><p>Let’s migrate one Apache Iceberg table to see how the process might look and what are required steps and services.</p><h4>Understanding the components</h4><p>Before diving into the migration process, it’s essential to understand the key components involved:</p><ul><li><strong>Apache Iceberg</strong>: An open table format for huge analytic datasets. Our table’s format.</li><li><strong>Local file system</strong>: Used as source data storage for simplicity. <strong>S3 / ADLS / HDFS </strong>can be used without code changes.</li><li><strong>Hive Metastore (HMS)</strong>: source Apache Iceberg catalog.</li><li><strong>Google Cloud Storage (GCS)</strong>:<strong> </strong>target data storage.</li><li><strong>BigLake Iceberg REST catalog</strong>: target Iceberg catalog.</li><li><strong>Apache Spark</strong>: local Apache Spark cluster to execute Spark scripts.</li></ul><h3>Local Iceberg table</h3><p>Let’s prepare a local Apache Iceberg table to work with. I will use HMS and connect a <em>spark-sql</em> shell to it.</p><p>Run HMS:</p><pre>docker run -p 9083:9083 --env SERVICE_NAME=metastore --name metastore-standalone apache/hive:3.1.3</pre><p>Run Spark:</p><pre>spark-sql \<br>--packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.10.0 \<br>--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \<br>--conf spark.sql.catalog.iceberg_catalog=org.apache.iceberg.spark.SparkCatalog \<br>--conf spark.sql.catalog.iceberg_catalog.type=hive \<br>--conf spark.sql.catalog.iceberg_catalog.uri=thrift://localhost:9083 \<br>--conf spark.sql.catalog.iceberg_catalog.warehouse=/tmp/iceberg-warehouse \<br>--conf spark.sql.defaultCatalog=iceberg_catalog</pre><p>Create Iceberg table:</p><pre>CREATE NAMESPACE mydb;<br>USE mydb;<br>CREATE TABLE demo_table(num BIGINT, text STRING) USING iceberg partitioned by (num);<br><br>INSERT INTO demo_table values (1, &quot;text one&quot;);<br>INSERT INTO demo_table values (2, &quot;text two&quot;);<br><br>ALTER TABLE demo_table SET TBLPROPERTIES (&#39;my.custom.property&#39; = &#39;a value&#39;);</pre><p>At this stage, we have an Apache Iceberg table `<em>demo_table</em>` managed by a local Hive Metastore. Before migrating this table to the GCS and BigLake Iceberg REST catalog, let’s check the table details.</p><pre>SHOW TBLPROPERTIES demo_table;<br><br>DESCRIBE TABLE EXTENDED demo_table;</pre><p>In the output we are looking for the following information:</p><pre>...<br>Location             file:/tmp/iceberg-warehouse/mydb.db/demo_table                     <br>Provider             iceberg <br>...</pre><p>This information confirms that our table is Apache Iceberg table and shows where the table is located. Now we are ready to migrate.</p><h3>Prepare GCP</h3><p>Before migrating any data, let’s ensure the destination environment on Google Cloud is properly provisioned and configured.</p><ol><li><strong>GCS bucket</strong>: If it doesn’t already exist, let’s create a GCS bucket for our Iceberg tables. I will use `gs://my-gcp-bucket`.</li><li>Folder for <strong>Iceberg namespace</strong>: If it doesn’t already exist, let’s create a folder in the bucket for the future Iceberg namespace. I will use `/my/existing/gcs/path/`.</li><li>Enable <strong>BigLake Iceberg REST catalog</strong>: <a href="https://docs.cloud.google.com/biglake/docs/blms-rest-catalog">instruction</a>.</li></ol><h3>Rewrite Iceberg metadata</h3><p>While the most intuitive first step might seem to be simply copying all the table’s files (data and metadata) to a new GCS bucket, this will not work. Iceberg metadata files contain <a href="https://iceberg.apache.org/spec/#data-file-fields">absolute file paths</a>, so this should be fixed first.</p><blockquote><strong>Note</strong>: There is a <a href="https://s.apache.org/iceberg-spec-relative-path">proposal</a> for relative paths in Apache Iceberg <a href="https://github.com/apache/iceberg/milestone/58">v4 specification</a>. If implemented, this feature would likely make the current path-rewriting step obsolete in future Iceberg versions.</blockquote><p>Luckily, there is a Spark `<em>rewrite_table_path</em>` <a href="https://iceberg.apache.org/docs/nightly/spark-procedures/#table-replication">procedure</a> that helps us to create new metadata files with a new location. The procedure must be executed in an original Spark cluster.</p><pre>CALL iceberg_catalog.system.rewrite_table_path(<br>  table =&gt; &#39;iceberg_catalog.mydb.demo_table&#39;, <br>  source_prefix =&gt; &#39;file:/tmp/iceberg-warehouse/mydb.db/demo_table&#39;,<br>  target_prefix =&gt; &#39;gs://my-gcp-bucket/my/existing/gcs/path/demo_table&#39; <br>);<br><br>....<br>file:/tmp/iceberg-warehouse/mydb.db/demo_table/metadata/copy-table-staging-4d2f3649a7d6/file-list<br>Time taken: 0.262 seconds, Fetched 1 row(s)</pre><blockquote>`copy-table-staging-4d2f3649a7d6/metadata`</blockquote><p>The output of this procedure is a <strong>new</strong> metadata folder inside of the existing `metadata` folder of the table. This new folder contains the updated metadata files that are ready for the target GCS location.</p><p>Optionally, `<em>start_version</em>` and `<em>end_version</em>` can be provided to limit the scope to an incremental rewrite. An incremental rewrite will only rewrite metadata files added between start_version and end_version.</p><p>Get current snapshot in HMS:</p><pre>SHOW TBLPROPERTIES mydb.demo_table (&#39;current-snapshot-id&#39;);<br><br>...<br>current-snapshot-id 3963040927721816275<br>Time taken: 0.208 seconds, Fetched 1 row(s)</pre><p>Find metadata file name for the snapshot in HMS:</p><pre>spark-sql (mydb)&gt; SELECT file FROM mydb.demo_table.metadata_log_entries <br>WHERE latest_snapshot_id = 3963040927721816275;<br><br>....<br>file:/tmp/iceberg-warehouse/mydb.db/demo_table/metadata/00003-xyz.metadata.json<br>Time taken: 1.064 seconds, Fetched 1 row(s)</pre><p>The current metadata in our example is `<em>00003-xyz.metadata.json</em>`. Now we have our table prepared for moving and registering in GCP.</p><h3>Copy table to GCS</h3><p>The next step is to execute the files transfer to GCS. It is important to emphasize the use of the recursive copy command, as Iceberg tables very often partitioned and the table’s folders structure is not flat.</p><p>First copy data files:</p><pre>gsutil -m cp -r \<br> /tmp/iceberg-warehouse/mydb.db/demo_table/data \<br> gs://my-gcp-project/my/existing/gcs/path/demo_table/</pre><p>Copy rewritten metadata. Please pay attention that we copy to metadata the folder `<em>copy-table-staging-4d2f3649a7d6/metadata</em>` created in the step above:</p><pre>gsutil -m cp -r \<br>/tmp/iceberg-warehouse/mydb.db/demo_table/metadata/copy-table-staging-4d2f3649a7d6/metadata \<br>gs://my-gcp-project/my/existing/gcs/path/demo_table/</pre><p>For scenarios involving <strong>incremental migration</strong> or periodic synchronization after the initial load, the <em>gsutil rsync</em> command is often preferred.</p><p>For huge data transfers originating from another cloud provider or for handling petabyte-scale migrations, consider using the <a href="https://cloud.google.com/storage-transfer-service">Storage Transfer Service</a>.</p><h3>Register table</h3><p>At this stage, the Iceberg table’s data and path-corrected metadata are successfully residing in GCS, but the table remains invisible to query engines and GCP services for example BigQuery. The table needs to be managed by a catalog that these services can access. This is where the BigLake Iceberg REST Catalog, comes into play.</p><blockquote>Apache Iceberg `1.10` or higher is required for the Spark commands below.</blockquote><p>First, let’s connect to BigLake Iceberg REST catalog from local spark:</p><pre>spark-sql  --packages \<br>org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.10.0 \<br> --jars https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-gcp-bundle/1.10.0/iceberg-gcp-bundle-1.10.0.jar,https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-gcp/1.10.0/iceberg-gcp-1.10.0.jar \<br> --conf &quot;spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions&quot; \<br> --conf &quot;spark.sql.defaultCatalog=iceberg_on_gcs&quot; \<br> --conf &quot;spark.sql.catalog.iceberg_on_gcs=org.apache.iceberg.spark.SparkCatalog&quot; \<br> --conf &quot;spark.sql.catalog.iceberg_on_gcs.warehouse=gs://my-gcp-bucket&quot; \<br> --conf &quot;spark.sql.catalog.iceberg_on_gcs.type=rest&quot; \<br> --conf &quot;spark.sql.catalog.iceberg_on_gcs.uri=https://biglake.googleapis.com/iceberg/v1/restcatalog&quot; \<br> --conf &quot;spark.sql.catalog.iceberg_on_gcs.rest.auth.type=google&quot; \<br> --conf &quot;spark.sql.catalog.iceberg_on_gcs.header.x-goog-user-project=my-gcp-project&quot; \<br> --conf &quot;spark.sql.catalog.iceberg_on_gcs.io-impl=org.apache.iceberg.gcp.gcs.GCSFileIO&quot;</pre><p>Create a namespace for the GCS folder created early:</p><pre>CREATE NAMESPACE IF NOT EXISTS my_gcp_namespace <br>LOCATION &#39;gs://my-gcp-project/my/existing/gcs/path&#39;;<br><br>DESCRIBE NAMESPACE my_gcp_namespace;</pre><p>To register the table we will use the existing Spark <a href="https://iceberg.apache.org/docs/nightly/spark-procedures/#register_table">procedure</a> `<em>register_table</em>`. To avoid data discrepancy it’s required to provide the same `<em>XXX.metadata.json</em>` as registered in the original metastore.</p><pre>CALL iceberg_on_gcs.system.register_table(<br>  table =&gt; &#39;iceberg_on_gcs.my_gcp_namespace.demo_table&#39;, <br>  metadata_file =&gt; &#39;gs://my-gcp-project/my/existing/gcs/path/demo_table/metadata/00003-xyz.metadata.json&#39; <br>)</pre><p>Now this Iceberg table is officially part of the Google Cloud ecosystem.</p><h3>Testing</h3><p>The final step is to validate that the table is accessible through the BigLake Iceberg REST catalog. The following Spark SQL scripts will work in our local Spark, or Dataproc, or any other environment configured to use the BigLake Iceberg REST Catalog.</p><p>Make sure that the following command return the same result for the original and migrate tables:</p><pre>DESCRIBE EXTENDED iceberg_on_gcs.my_gcp_namespace.demo_table;<br>SHOW TBLPROPERTIES iceberg_on_gcs.my_gcp_namespace.demo_table;<br>SELECT * FROM iceberg_on_gcs.my_gcp_namespace.demo_table.history;</pre><p>All Iceberg snapshots are equal to the original table, so we can run some time travel queries:</p><pre>SELECT * FROM iceberg_on_gcs.my_gcp_namespace.demo_table VERSION AS OF 7088481241341197370;</pre><h3>Dual writes</h3><p>Migrating a production table is not a single, instantaneous event; it’s a process. To maintain data consistency for applications writing to the table, a dual-write strategy must be implemented.</p><p>All write operations will generate new data and metadata files for the table.</p><p>Please keep in mind that copying these data and metadata files to a new GCS location is not enough, it’s also required to update the BigLake Iceberg REST catalog to point to the latest valid metadata version.</p><p>Iceberg REST catalog API <a href="https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L3542">supports</a> the `<em>overwrite</em>` parameter, but this parameter is not supported by Apache Spark `register_table`. The solution can be a direct REST call to BigLake Iceberg REST catalog or `DROP TABLE` + `register_table` by Apache Spark.</p><h3>Best Practices</h3><ul><li><strong>Incremental Migration:</strong> For very large tables, consider an incremental migration strategy.</li><li><strong>Schema Evolution:</strong> Iceberg’s schema evolution capabilities will be preserved during migration, but it’s good to re-verify after the move.</li><li><strong>Partitioning:</strong> Ensure your partitioning strategy remains optimal for GCS access patterns.</li><li><strong>Cost Management:</strong> Be mindful of egress costs when transferring data out of S3 or ADLS.</li><li><strong>Egress Cost Reduction</strong>: For large and old tables combining <a href="https://iceberg.apache.org/docs/latest/spark-procedures/#rewrite_data_files">rewrite_data_files</a>, <a href="https://iceberg.apache.org/docs/latest/spark-procedures/#expire_snapshots">expire_snapshots</a>, and <a href="https://iceberg.apache.org/docs/latest/spark-procedures/#remove_orphan_files">remove_orphan_files</a> Iceberg procedures on the source cluster, can significantly reduce the data volume and number of files that must be transferred to GCS.</li><li><strong>Monitoring and Alerting:</strong> Set up monitoring for your GCS buckets and BigLake catalog to track usage, errors, and performance.</li><li><strong>Rollback Plan:</strong> Always have a rollback plan in case issues arise during the migration.</li><li><strong>Documentation:</strong> Document all steps taken, configurations, and validation results for future reference and troubleshooting.</li></ul><h3>Conclusion</h3><p>The central challenge we set out to address was how to migrate an Apache Iceberg table from its legacy catalog and storage to Google Cloud. The goal was to unlock next-generation analytics and unified governance provided by GCP and services like BigLake Metastore.</p><p>This article demonstrated that this complex migration is not only possible but provides a clear path and the best practices to take into account. By moving data to GCP it’s possible to delegate to Google Cloud to play a role of a single source of truth that is scalable, open, and ready for high-performance analytics.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c68399d7a03d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/migrating-apache-iceberg-tables-to-gcs-and-biglake-iceberg-rest-catalog-c68399d7a03d">Migrating Apache Iceberg Tables to GCS and BigLake Iceberg REST Catalog</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>