<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Samuel Valente on Medium]]></title>
        <description><![CDATA[Stories by Samuel Valente on Medium]]></description>
        <link>https://medium.com/@spcvalente?source=rss-5eb07f09f9af------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*_1-GykYz4xHnFBQMCaqjKg.jpeg</url>
            <title>Stories by Samuel Valente on Medium</title>
            <link>https://medium.com/@spcvalente?source=rss-5eb07f09f9af------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 23:08:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@spcvalente/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Snowflake with Iceberg: Lakekeeper, dbt, and some Sparks Flying]]></title>
            <link>https://medium.com/fresha-data-engineering/snowflake-with-iceberg-lakekeeper-dbt-and-some-sparks-flying-a6231fcb35a7?source=rss-5eb07f09f9af------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6231fcb35a7</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Samuel Valente]]></dc:creator>
            <pubDate>Thu, 14 May 2026 13:20:10 GMT</pubDate>
            <atom:updated>2026-05-14T13:20:10.848Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="A futuristic ice bridge leads into a tunnel carved through a massive iceberg, with warning signs, traffic cones, and faint digital lights suggesting a fragile data pipeline through an icy landscape." src="https://cdn-images-1.medium.com/max/1024/1*AjYwo6X77h7uOipTHHG_Mg.png" /><figcaption>Snowflake, Iceberg, and the long road through the caveats.</figcaption></figure><h3>TL;DR</h3><p>It <strong>kinda</strong> works, but it’s far from perfect. It’s like asking your dishwasher to also do the laundry — with enough duct tape you can make it happen, but then you start questioning your life choices.</p><h3>Background</h3><p>Here at Fresha, we don’t pretend to <a href="https://medium.com/fresha-data-engineering/how-we-accidentally-became-one-of-uks-first-starrocks-production-pioneers-7db249f10010">only use the latest and greatest tools</a>, but we’re also keen on using the ones that are already proven and well-established. We’ve been using Snowflake for a long time, and we’re happy with it. We ingest data from all our sources in Snowflake and then we transform with dbt.</p><p>Honestly, it’s a great platform for data warehousing and analytics. We have no plans to abandon it, we intend to keep using it where it shines, and to use other tools to complement it where it doesn’t.</p><p>If you’re interested in how we ingest data into Snowflake, check out <a href="https://medium.com/fresha-data-engineering/from-json-to-avro-in-the-cdc-pipeline-ff24ac9c9abc">the post</a> where we peek into the architecture in more detail.</p><h3>The Use Case</h3><p>We need to leverage Snowflake data <a href="https://medium.com/fresha-data-engineering/the-real-time-data-journey-connecting-flink-airflow-and-starrocks-7c6481660e45">in other platforms, such as StarRocks</a>, to avoid duplicating data ingestion and processing efforts, in a way that is efficient, scalable, and reliable. Portability and interoperability come into the consideration when we work across different systems and tools.</p><p>Naturally, Iceberg is a great fit for this use case, it’s open source, scalable, reliable, and well-supported.</p><p>To achieve our goal, we set up an Airflow DAG that waits for a dbt Snowflake job to finish in dbt Cloud, spins up a Spark job to ingest data from Snowflake into Iceberg, and then performs maintenance operations on the Iceberg table.</p><figure><img alt="Diagram of the current data export pipeline: Airflow waits for dbt Cloud, triggers Spark jobs to move Snowflake tables into Iceberg on S3, runs Iceberg maintenance, registers metadata in Lakekeeper, and makes the data available to StarRocks." src="https://cdn-images-1.medium.com/max/1024/1*mL-FAJx39pffsaSDQbWSQw.png" /><figcaption>The current pipeline, meh.</figcaption></figure><p>While this works, it’s not ideal, and it’s not something we’re proud of. There are quite a few steps, moving parts, and boilerplate code that we had to write to make it work.</p><h3>The Promise of Iceberg on Snowflake</h3><p>But wait… Snowflake <strong>does</strong> support Iceberg tables! And they even support them through an external REST catalog! This means that we can use the tools we want and when we want to. Power to the Data Engineers! The prophecy is fulfilled!</p><p>Relevant docs: <a href="https://docs.snowflake.com/en/user-guide/tables-iceberg">Apache Iceberg™ tables</a> · <a href="https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration-rest">REST catalog integration</a> · <a href="https://docs.snowflake.com/en/user-guide/opencatalog/overview">Snowflake Open Catalog (Polaris)</a></p><h3>The Raised Eyebrows</h3><p>While this is a great promise, we’re cautious and skeptical. When looking at the Snowflake docs, they mostly focus on their own catalog, a managed version of Polaris, and while they mention the REST catalog, it doesn’t seem to be as polished as their proprietary one. Being familiar with Snowflake, there’s always a catch to be expected, normally in the form of some weird limitation. Besides, as a SaaS platform, would you expect anything else? At the end of the day, they don’t want you to move your data to another platform.</p><blockquote><em>Yes, we support Open Table Formats!</em>*</blockquote><blockquote>*But you better use our catalog.</blockquote><p>Anyway, let’s give it a try for the sake of science.</p><h3>The Ideal Pipeline</h3><p>Before we dive into the implementation, let’s define the ideal pipeline.</p><p>The dbt-snowflake adapter supports Iceberg, so instead of hacking that complicated pipeline we have now, we can simply use the materialisation and add an additional node to the dbt run to run the Iceberg models, porting the Snowflake base tables to Iceberg.</p><figure><img alt="Ideal pipeline diagram showing dbt Cloud writing Snowflake data directly to Iceberg tables on S3, with Lakekeeper as the REST catalog and StarRocks querying the Iceberg data." src="https://cdn-images-1.medium.com/max/1024/1*mZD-0VsdCp1lJzna0yQnUw.png" /><figcaption>Neat!</figcaption></figure><p>No Spark jobs, no waiting for this and that, no custom code, just a simple dbt run (which already runs in Snowflake, so no need to change that) and we’re done!</p><h3>Infrastructure Setup</h3><p>Before we get to the fun part, let’s talk about how we actually wired this up in Terraform.</p><p>Here’s the catch: the Snowflake Terraform provider, at the time of writing, does <strong>not</strong> support external volumes, catalog integrations, or linked databases. These are relatively new Snowflake features and the provider simply hasn’t caught up yet.</p><p>So what do you do? You use snowflake_execute.</p><p>snowflake_execute is Terraform&#39;s polite way of saying &quot;just run this SQL and trust the process&quot;. You hand it a CREATE statement, a DROP for the revert, and a SHOW query so it can check if the resource already exists. It works. It&#39;s not pretty, but it works.</p><blockquote><em>It’s the Terraform equivalent of </em><em>// TODO: fix this properly — except you ship it, and then it runs in production, and then you stop thinking about it.</em></blockquote><p><strong>How It All Fits Together</strong></p><p>Three Snowflake-side resources need to exist before dbt can write a single file: the <strong>external volume</strong> tells Snowflake which S3 path and IAM role to use for storage; the <strong>catalog integration</strong> gives Snowflake a connection to Lakekeeper (the REST catalog) and the OAuth credentials to authenticate; the <strong>linked database</strong> is a Snowflake database that is wired to that integration, so every namespace and table registered in Lakekeeper automatically appears as schemas and tables inside Snowflake.</p><p>On the dbt side, three more pieces line up with those: the feature flag in dbt_project.yml unlocks the Iceberg materialisation; catalogs.yml maps the external volume and linked database into a named catalog dbt can reference; and the per-model config selects that catalog, sets the namespace, and defines partitioning.</p><p>At runtime, dbt writes Parquet files to S3 via the external volume, registers the table metadata in Lakekeeper via the catalog integration, and the linked database keeps Snowflake’s view of the catalog in sync automatically.</p><figure><img alt="Diagram of the Snowflake infrastructure setup: an external volume connects Snowflake to S3, a catalog integration connects to Lakekeeper through REST and Keycloak authentication, and a linked database auto-refreshes metadata from the catalog." src="https://cdn-images-1.medium.com/max/1024/1*RP_CXtqx7QqdvqDcwRbj8g.png" /><figcaption>The Snowflake components in action</figcaption></figure><p>Now, onto the details. If you’re not too keen on Terraform and dbt, I would suggest jumping into the next section, <strong>Setup Limitations</strong>.</p><h4>External Volume</h4><p>The external volume tells Snowflake where the Iceberg data lives — an S3 path, an IAM role, and a flag to allow writes.</p><pre>resource &quot;snowflake_execute&quot; &quot;lakekeeper_external_volume&quot; {<br>  execute = &lt;&lt;-SQL<br>    CREATE OR REPLACE EXTERNAL VOLUME DATA_LAKEHOUSE<br>    STORAGE_LOCATIONS = ((<br>      NAME = &#39;lakekeeper&#39;<br>      STORAGE_PROVIDER = &#39;S3&#39;<br>      STORAGE_BASE_URL = &#39;s3://&lt;bucket&gt;/&lt;lakekeeper-prefix&gt;/&#39;<br>      STORAGE_AWS_ROLE_ARN = &#39;&lt;aws-role-arn&gt;&#39;<br>      ALLOW_WRITES = TRUE<br>    ))<br>  SQL<br>  query  = &quot;SHOW EXTERNAL VOLUMES LIKE &#39;DATA_LAKEHOUSE&#39;&quot;<br>  revert = &quot;DROP EXTERNAL VOLUME IF EXISTS DATA_LAKEHOUSE&quot;<br>}</pre><p>snowflake_execute wrapping raw SQL. Perfectly functional. Completely unhinged.</p><h4>Catalog Integration</h4><p>This is where Snowflake actually learns that Lakekeeper exists. The catalog integration tells Snowflake where the REST catalog endpoint is and how to authenticate with it.</p><p>We use OAuth, with Keycloak as the identity provider.</p><pre>resource &quot;snowflake_execute&quot; &quot;lakekeeper_catalog_integration&quot; {<br>  execute = &lt;&lt;-SQL<br>    CREATE OR REPLACE CATALOG INTEGRATION LAKEKEEPER<br>      CATALOG_SOURCE = ICEBERG_REST<br>      TABLE_FORMAT = ICEBERG<br>      CATALOG_NAMESPACE = &#39;&lt;catalog-namespace&gt;&#39;<br>      REST_CONFIG = (<br>        CATALOG_URI = &#39;&lt;lakekeeper-catalog-uri&gt;&#39;<br>        CATALOG_NAME = &#39;&lt;warehouse-name&gt;&#39;<br>      )<br>      REST_AUTHENTICATION = (<br>        TYPE = OAUTH<br>        OAUTH_TOKEN_URI = &#39;&lt;keycloak-token-uri&gt;&#39;<br>        OAUTH_CLIENT_ID = &#39;&lt;oauth-client-id&gt;&#39;<br>        OAUTH_CLIENT_SECRET = &#39;&lt;oauth-client-secret&gt;&#39;<br>        OAUTH_ALLOWED_SCOPES = (&#39;&lt;oauth-scope&gt;&#39;)<br>      )<br>      ENABLED = TRUE<br>      REFRESH_INTERVAL_SECONDS = 900<br>  SQL<br>  query  = &quot;SHOW CATALOG INTEGRATIONS LIKE &#39;LAKEKEEPER&#39;&quot;<br>  revert = &quot;DROP CATALOG INTEGRATION IF EXISTS LAKEKEEPER&quot;<br>}</pre><p>snowflake_execute. Again. You&#39;re seeing a pattern.</p><h4>Linked Database</h4><p>Here’s the part that actually feels like it was designed to work. Once the catalog integration is in place, you create a database that is <strong>linked</strong> to it:</p><pre>resource &quot;snowflake_execute&quot; &quot;lakekeeper_database&quot; {<br>  execute = &lt;&lt;-SQL<br>    CREATE OR REPLACE DATABASE LAKEKEEPER<br>      LINKED_CATALOG = (CATALOG = &#39;LAKEKEEPER&#39;)<br>      EXTERNAL_VOLUME = &#39;DATA_LAKEHOUSE&#39;<br>  SQL<br>  query  = &quot;SHOW DATABASES LIKE &#39;LAKEKEEPER&#39;&quot;<br>  revert = &quot;DROP DATABASE IF EXISTS LAKEKEEPER&quot;<br>}</pre><p>That’s it. Snowflake talks to Lakekeeper, discovers all the namespaces and tables, and they show up as schemas and Iceberg tables inside the database. No manual table registration, no DDL per table. You can query your Iceberg data from Snowflake as if it had always lived there.</p><h3>dbt Setup</h3><p>Now that the Snowflake side is wired up, we need to tell dbt about it. Three places to touch: dbt_project.yml, catalogs.yml, and the model config.</p><p>Relevant docs: <a href="https://docs.getdbt.com/docs/mesh/iceberg/snowflake-iceberg-support?version=1.10">Snowflake and Apache Iceberg | dbt Developer Hub</a></p><h4>dbt_project.yml</h4><p>First, enable the Iceberg materialisations feature flag:</p><pre>flags:<br>  enable_iceberg_materializations: True</pre><p>Without this, dbt will not recognise the Iceberg-specific config options.</p><h4>catalogs.yml</h4><p>At your project root, create a catalogs.yml file. This tells dbt which external catalog to write to and which external volume to use for storage.</p><pre>catalogs:<br>  - name: lakekeeper<br>    active_write_integration: &lt;write-integration-name&gt;<br>    write_integrations:<br>      - name: &lt;write-integration-name&gt;<br>        catalog_type: iceberg_rest<br>        table_format: iceberg<br>        external_volume: &lt;external-volume-name&gt;<br>        adapter_properties:<br>          catalog_name: &lt;catalog-name&gt;<br>          catalog_linked_database: &lt;linked-database-name&gt;<br>          auto_refresh: true</pre><p>Two things to highlight:</p><ul><li>catalog_linked_database is the Snowflake database we created in the previous step — the one linked to Lakekeeper. Without it, dbt can write files to S3 but can&#39;t fully manage the table lifecycle.</li><li>auto_refresh: true tells Snowflake to automatically sync metadata from Lakekeeper when the catalog integration refreshes, rather than requiring manual intervention.</li></ul><h4>Model config</h4><p>Each Iceberg model needs to know which catalog to target, which schema namespace to write into, and how to partition the data:</p><pre>{{<br>  config(<br>    materialized=&#39;table&#39;,<br>    catalog_name=&#39;lakekeeper&#39;,<br>    schema=&#39;&quot;&lt;catalog-namespace&gt;&quot;&#39;,<br>    partition_by=&#39;bucket(128, id)&#39;,<br>    unique_key=&#39;id&#39;<br>  )<br>}}</pre><pre>select * from {{ ref(&#39;my_source_model&#39;) }}</pre><p>The schema value must match the CATALOG_NAMESPACE set in the Snowflake catalog integration — both use &lt;catalog-namespace&gt;. The partition_by uses Iceberg partition spec syntax — bucket(N, col) for hash bucketing, month(col) / day(col) for time-based partitioning.</p><p>dbt handles the rest — creating the Iceberg table in Lakekeeper, writing the files to S3 via the external volume, and keeping the metadata in sync. The resulting tables are proper Iceberg tables, registered in Lakekeeper, queryable from Snowflake via the linked database, and accessible by any other tool that speaks the Iceberg REST spec.</p><h3>Setup Limitations</h3><p>Remember when we said there’s always a catch? Here it is.</p><h4>No CREATE OR REPLACE ICEBERG TABLE</h4><p>In a catalog-linked database, CREATE OR REPLACE is not supported. Every full refresh must be an explicit DROP TABLE followed by CREATE TABLE. This matters because dbt&#39;s standard table materialisation uses CREATE OR REPLACE — so it breaks out of the box and requires a custom materialisation.</p><p>It also means the naive full-refresh approach creates a downtime window: the table is unavailable between the drop and the end of the insert. For large tables, that window can be minutes.</p><h4>No CTAS</h4><p>You cannot use CREATE TABLE AS SELECT for Iceberg tables in a catalog-linked database. Lakekeeper rejects it at the catalog commit stage with a table-not-found error. Every materialisation must do a CREATE TABLE (columns) followed by a separate INSERT INTO.</p><h4>Standard dbt adapter is blind to catalog-linked tables</h4><p>dbt’s standard adapter cannot detect existing Iceberg tables in catalog-linked databases — adapter.get_relation() silently returns none. This breaks is_incremental(), which uses that call internally. If you don&#39;t replace it with a SHOW ICEBERG TABLES check, the incremental WHERE clause is always skipped and every run inserts the full dataset.</p><h4>No ALTER ICEBERG TABLE ... RENAME TO</h4><p>Snowflake does not support renaming Iceberg tables in catalog-linked databases via SQL. This means you can’t do an atomic in-place swap of a staged table into the live slot through Snowflake. The rename has to go through the Lakekeeper REST API directly, bypassing Snowflake entirely.</p><h4>No DROP TABLE ... CASCADE</h4><p>Also blocked in catalog-linked databases. Clean teardown requires dropping dependent objects manually before the table.</p><p>Each of these is a papercut on its own. Together, they mean that writing to Iceberg through Snowflake requires custom materialisations, workaround patterns, and the occasional direct REST call to Lakekeeper — none of which you’d expect when the promise was “just use dbt and it works”.</p><h3>Workarounds</h3><p>To get around the limitations above, we built two custom dbt materialisations in a shared internal package: one for full reloads (iceberg_full_refresh) and one for incremental updates (iceberg_incremental). Both use CREATE TABLE + INSERT INTO instead of CTAS, and SHOW ICEBERG TABLES instead of adapter.get_relation() for existence checks.</p><p>The full reload materialisation never touches the live table during a dbt run. It writes to {table}__new instead, keeping the live table available throughout. Once the dbt job completes, an Airflow task promotes the new table into the live slot via two metadata-only renames in Lakekeeper:</p><pre>{table}      → {table}__old<br>{table}__new → {table}</pre><p>No data is moved. The live table is only unavailable for milliseconds.</p><figure><img alt="Sequence diagram showing the full-refresh workaround: dbt Cloud writes data to a temporary table__new, Airflow waits for the dbt job to complete, then calls Lakekeeper to rename the live table to table__old and promote table__new to the live table." src="https://cdn-images-1.medium.com/max/1024/1*dSt8LESvOfRSjckGdoC51A.png" /><figcaption>The great hack.</figcaption></figure><p>Because Snowflake can’t rename Iceberg tables in catalog-linked databases, the swap bypasses Snowflake entirely and calls the Lakekeeper REST API directly.</p><p>We also ship iceberg_is_incremental() as a drop-in replacement for dbt&#39;s is_incremental(), which uses SHOW ICEBERG TABLES instead of adapter.get_relation() — otherwise the incremental WHERE clause is always skipped and every run inserts the full dataset.</p><h3>Querying the Data</h3><p>Now that the data is in Iceberg, we can query it from StarRocks and assess performance!</p><p>We set up a benchmark script that runs a series of queries against the Iceberg tables — both Snowflake-written and Spark-written — and measures the time it takes to execute. The queries are based on real-world customer-facing queries that we run in our production environment, mostly filtering by provider and date range.</p><p>The first results were not promising. The Snowflake-written tables were significantly slower than the Spark-written tables.</p><p>What’s significantly slower? 50–100% slower. That’s a lot! And with 10x scanned data, that’s a lot of wasted time and resources.</p><p>Why, though? Because partition pruning is not working as expected…</p><h3>More Workarounds</h3><p>When analysing the tables, we noticed that the partition keys had the wrong data type. Snowflake writes them as decimal(38,0), while Spark writes them as long. This needed to be fixed.</p><p>To fix this, we changed our materialisation to use CREATE TABLE + INSERT INTO instead of CREATE TABLE AS SELECT, which required passing the column types to the materialisation. Again, not ideal, but it worked, the tables were fixed, and the performance improved.</p><h3>Querying the Data Again</h3><p>At this point, we should be able to get a fair comparison between the two pipelines, right? Right?</p><p>Running the same benchmark script again, we were pleasantly surprised to see that the scanned data was now comparable between the two pipelines; however, the query times were still significantly higher than the Spark-written, approx. 50% higher…</p><p>The benchmark query — per-provider monthly revenue aggregate, last 365 days:</p><pre>SELECT provider_id, DATE_TRUNC(&#39;month&#39;, invoice_date) AS month, item_type_name,<br>       COUNT(*) AS item_count, SUM(total_sales) AS revenue<br>FROM &lt;table&gt;<br>WHERE provider_id = {p}<br>  AND invoice_date &gt;= DATE_SUB(CURRENT_DATE(), INTERVAL 365 DAY)<br>GROUP BY 1, 2, 3<br>ORDER BY 1, 2, 3</pre><pre>|----------|---------|---------|---------|---------|---------|<br>| Provider | SF time | SP time | Diff    | Diff %  | Rows    |<br>|----------|---------|---------|---------|---------|---------|<br>| A        | 1.994s  | 0.783s  | +1.211s | +154.7% | 511,415 |<br>| B        | 1.152s  | 0.761s  | +0.391s | +51.4%  | 506,861 |<br>| C        | 1.446s  | 0.787s  | +0.659s | +83.7%  | 33      |<br>| D        | 1.083s  | 0.766s  | +0.317s | +41.4%  | 575,936 |<br>| E        | 0.922s  | 0.775s  | +0.147s | +19.0%  | 366,889 |<br>| F        | 1.052s  | 0.745s  | +0.307s | +41.2%  | 257,560 |<br>| G        | 1.081s  | 0.748s  | +0.333s | +44.5%  | 406,496 |<br>| H        | 0.989s  | 0.752s  | +0.237s | +31.5%  | 308,798 |<br>| I        | 1.010s  | 0.798s  | +0.212s | +26.6%  | 452,747 |<br>| J        | 1.040s  | 0.863s  | +0.177s | +20.5%  | 245,363 |<br>|----------|---------|---------|---------|---------|---------|<br>| TOTAL    | 11.769s | 7.778s  | +3.991s | +51.3%  | —       |<br>|----------|---------|---------|---------|---------|---------|</pre><p>Provider C is particularly telling: only 33 rows returned, yet Snowflake takes 1.446s — query time has almost no correlation with result size, which points to file layout rather than data volume.</p><p>Why, though?</p><p>The Snowflake-written tables had significantly more files than the Spark-written ones, 2–3x more! While you can tweak the target_file_size in the catalog, you can&#39;t control the actual file size, which is ultimately determined by Snowflake internals.</p><h3>More Pathologies</h3><p>After further analysis, we found several additional root causes — all of which compound the small-files problem or make it impossible to fix. The summary table at the end of this section covers all pathologies, including the decimal type issue described above.</p><p><strong>DELTA_BINARY_PACKED encoding breaks compaction.</strong> Snowflake writes integer columns with this Parquet encoding. Spark&#39;s vectorised reader can&#39;t handle it and throws an UnsupportedOperationException. With partial-progress.enabled=true, the compaction job absorbs the errors, reports success, and rewrites nothing. The table stays fragmented indefinitely. Fix: spark.sql.iceberg.vectorization.enabled=false on any Spark job reading Snowflake-written files.</p><p><strong>No sort order or distribution mode.</strong> Snowflake writes everything unsorted (sort_order_id=0). Spark uses sort_order_id=1 with write.distribution-mode=range, so each file covers a tight, contiguous slice of provider_id with narrow column stats. On the Snowflake side, files within each bucket contain interleaved values and can&#39;t be skipped. This alone accounts for roughly 2–3x of the query time gap, even after partition pruning is fixed.</p><p><strong>No write properties.</strong> Snowflake sets none of the standard Iceberg write properties — no target file size, no compression codec (zstd), no sort order, no merge/delete/update modes (merge-on-read), no metadata retention limits. It treats Iceberg as a dump format, not a managed table.</p><p><strong>No Puffin statistics.</strong> Without them, the query planner has no NDV or theta-sketch estimates, which affects join ordering and filter selectivity. Lower priority than the above, but worth running compute_table_stats after any schema fix and compaction.</p><h3>Even More Workarounds</h3><p>Once the schema is fixed, the table still needs to be rehabilitated via Spark-based maintenance — because Snowflake will never do it.</p><p>First, set the missing table properties (file size, compression, sort order, merge modes, metadata retention). Then run compaction — but only after setting spark.sql.iceberg.vectorization.enabled=false, otherwise the job silently no-ops on Snowflake&#39;s DELTA_BINARY_PACKED-encoded files. Once compaction rewrites all the small files into properly-sized, sorted chunks, run compute_table_stats to generate Puffin statistics.</p><p>None of this should be necessary. But here we are.</p><p>But wait… At this point, does it even make sense to proceed with this? Are we better off with the Spark pipeline?</p><p>Yes, we are.</p><h3>Conclusion</h3><p>It technically works. Incremental runs are fine, the catalog integration wires up cleanly, and the linked database is genuinely elegant. If all you need is Snowflake reading its own Iceberg tables, you’re done in an afternoon.</p><p>The problem is everything else. Custom materialisations, atomic swaps via direct REST calls, compaction jobs that silently no-op unless you know the right incantation, a 50% query time penalty after all the fixes — the overhead comfortably exceeds the cost of the Spark pipeline this was meant to replace.</p><p>Looking at the limitations, none of them feel accidental. Snowflake’s Iceberg story is built around their managed version of Polaris. The external REST catalog path exists, but it’s not where the polish went. At the end of the day, they don’t want you to move your data out — and the product reflects that.</p><p>To be fair, this is a moving target. Snowflake is actively investing in Iceberg support and some of these gaps will close. It’s worth revisiting in 6–12 months.</p><p>For now, though, the dishwasher is going back to washing dishes. The laundry is going in the washing machine.</p><p>Some of our other posts go deeper into the surrounding pieces: <a href="https://medium.com/fresha-data-engineering/automq-shared-storage-architecture-deep-dive-043c5226847e">diskless Kafka</a>, <a href="https://medium.com/fresha-data-engineering/optimising-starrocks-queries-in-practice-scans-joins-and-what-to-look-for-3f5a537b85f1">StarRocks query performance</a>, <a href="https://medium.com/fresha-data-engineering/hot-cold-data-separation-with-starrocks-and-iceberg-part-1-f2b3b820b4f1">hot/cold data separation</a> with Iceberg, <a href="https://medium.com/fresha-data-engineering/bottling-the-river-apache-fluss-on-eks-6aa63c00d9e9">newest streaming storage engines</a>, and other practical trade-offs of building modern data infrastructure on open formats.</p><p>If that sounds interesting, follow the Fresha Data Engineering blog. We will keep writing about the parts that work, the parts that do not, and the awkward bits in between.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6231fcb35a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/fresha-data-engineering/snowflake-with-iceberg-lakekeeper-dbt-and-some-sparks-flying-a6231fcb35a7">Snowflake with Iceberg: Lakekeeper, dbt, and some Sparks Flying</a> was originally published in <a href="https://medium.com/fresha-data-engineering">fresha-data-engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>