<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Naveen Alan on Medium]]></title>
        <description><![CDATA[Stories by Naveen Alan on Medium]]></description>
        <link>https://medium.com/@alandeepak?source=rss-5ed895c8dd32------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*zm1tCdhaSj9cmyKS9cblnQ.jpeg</url>
            <title>Stories by Naveen Alan on Medium</title>
            <link>https://medium.com/@alandeepak?source=rss-5ed895c8dd32------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 13:26:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alandeepak/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[What 3.7 Stars Won’t Tell You]]></title>
            <link>https://medium.com/snowflake/what-3-7-stars-wont-tell-you-fcaa1c7058b2?source=rss-5ed895c8dd32------2</link>
            <guid isPermaLink="false">https://medium.com/p/fcaa1c7058b2</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[cortex-ai-function]]></category>
            <category><![CDATA[snowflake-data-cloud]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[hex]]></category>
            <dc:creator><![CDATA[Naveen Alan]]></dc:creator>
            <pubDate>Wed, 09 Sep 2026 14:01:04 GMT</pubDate>
            <atom:updated>2026-09-09T14:01:04.803Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How we turned 98 product reviews into a published safety-intelligence app — in four prompts, one notebook session, using Snowflake Cortex AI Functions and Hex.</em></p><p><em>By Naveen Thomas (Principal Partner Data Cloud Architect, Snowflake) &amp; Rachel Herrera (Product Evangelist, Hex)</em></p><h3>The Problem: A 3.7-Star Product Hiding a Safety Crisis</h3><p>The GE Profile Opal 2.0 is a $500 countertop nugget ice maker. It has a 3.7-star rating on Amazon. On paper, that looks mediocre — maybe worth improving, not worth panicking over.</p><p>But buried in those reviews is something a star rating will never tell you:</p><ul><li>Customers are reporting <strong>black mold growth</strong> inside the machine</li><li>Multiple reviewers mention <strong>filing BBB complaints</strong> after receiving defective refurbished replacements</li><li>One reviewer titled their post: <strong>“Serious health concern — do NOT buy”</strong></li><li>Another: <strong>“DO NOT BUY. Go get the Opal 1 version, this one is just plain terrible.”</strong></li></ul><p>At the same time, other customers write things like “my family is obsessed with this ice” and “best purchase we’ve made.” The 5-star reviews are genuinely enthusiastic.</p><p>This isn’t a bad product. It’s a polarized product — simultaneously beloved and generating safety escalations. A star average tells you nothing useful here. You need to know <em>what specifically is failing</em>, <em>how often</em>, whether it&#39;s<em> getting worse</em>, and <em>which reviews need immediate attention</em>.</p><p>There was nowhere we could go for this information, so we built it. Four prompts, one Hex Agent session, under 20 minutes from blank page to published app.</p><p><strong>This walkthrough uses real Amazon review data</strong> from the <a href="https://app.snowflake.com/marketplace/listing/GZTYZ36VTKF/datahive-ai-amazon-best-sellers-ratings-and-reviews?search=Amazon%20Best%20Sellers%20Ratings%20and%20Reviews">Amazon Best Sellers dataset</a> on Snowflake Marketplace (free, 256K reviews across 2,700 products). Nothing is mocked up. Every query runs against real reviews. Every result is real.</p><h3>Four Prompts to a Published App</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q2aMzkNYeuRZJQwCss7T3g.jpeg" /></figure><h4>Prompt 1: “Show me what’s in this data”</h4><p>We opened <a href="https://hex.tech/capability/ai/">Hex</a>, connected to Snowflake, pointed at the GE Opal reviews, and typed:</p><blockquote>“Show me the GE Profile Opal ice maker reviews. I want to see the star distribution and a sample across all ratings. The product ASIN is B0964BF4N7.”</blockquote><p>The agent found the product — 3.7 stars across 4,375 total ratings, 98 reviews in the dataset — and immediately rendered the star distribution. When we looked at it closer — the polarization was stark: roughly equal clusters at 1-star and 5-star with a bulge at 3-star. But the distribution chart didn’t tell us <em>why</em>. So we prompted further.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p_KSFHQu0jw5jg3jS0R1iQ.png" /></figure><h4>Prompt 2: “What’s actually wrong?”</h4><blockquote>“Analyze these Opal reviews: score sentiment (more nuanced than stars), classify each by complaint topic (quality_defect, mold_hygiene, customer_support, noise_level, setup_maintenance, value_for_money), flag reviews where the customer is returning the product or describing a health/safety concern, and prioritize them: safety &gt; critical &gt; high &gt; medium &gt; low.”</blockquote><p>The Hex agent used Snowflake Cortex AI Functions to enrich all 98 reviews in a single pass — scoring sentiment, classifying topics, detecting escalation risk, extracting the specific defect, and computing a priority tier. It validated the schema on 2 rows first (AI functions bill per row), then scaled to all 98.</p><p>The results revealed what was hidden by the star rating:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KsTDzoniQcSvlvBQTMYUsg.png" /></figure><p><strong>With this analysis, we found that the 3-star bucket hid a lot.</strong> A 3-star rating implies “<strong>mediocre</strong>.” But 21 of the 23 reviews were actually high-severity — dead pumps, snapped cylinders, bacteria in the tubing. Your negative rate isn’t 29% (1–2 stars), it’s <strong>~55%</strong> once 3-star reviews are read properly.</p><p>And the safety flags? 3 of 6 are rated 3 stars or higher. A star-based filter would never surface them.</p><h4>Prompt 3: “Give me the executive summary”</h4><blockquote>“Summarize the top 3 recurring quality issues across all negative reviews in one executive paragraph. Be specific about what’s failing, how quickly, and recommend engineering prioritization.”</blockquote><p>The Hex Agent extracted failure components and time-to-failure from all 52 negative reviews, then synthesized:</p><p><strong>The Opal 2.0’s 3.7-star rating is driven by three recurring hardware failures, not by taste or expectation mismatch.</strong> First, the ice auger drivetrain — 23 of 44 identified failures (52%) — described progressive squealing, screeching, or grinding that began at a median of 6 months. Additionally, 14 of those 23 units were unusable, and 8 reviewers reported a replacement unit failing the same way. Second, the water-delivery subsystem contributed 13 failures: 7 false “Add Water” sensor faults, 3 pump failures, and 3 leaks. Third, biofilm and mold contamination in the internal tubing appeared in 5 reviews flagged as health/safety — with 3 of those 5 coming from customers who still rated the product 3 stars or higher, making this risk invisible to star-based monitoring.</p><p><strong>Engineering prioritization:</strong> P0 — auger drivetrain redesign (highest volume, earliest onset, confirmed to recur in replacements). P0 — mold mitigation in water path (low volume but health/safety liability). P1 — water-level sensing and pump reliability.</p><p>What came from the Hex agent wasn’t a “summary of reviews.” It was a product engineering brief — with component names, failure timelines, recurrence rates, and prioritized next steps — generated from one prompt.</p><h4>Prompt 4: “Build the app and publish it”</h4><p>Finally, we prompted the agent:</p><blockquote>“Build me an interactive app with: summary cards (total reviews, safety alerts, critical count, top defect), filters for priority and complaint topic, a bar chart of topics, and a sortable review table with priority coloring. Title it ‘What 3.7 Stars Won’t Tell You’. Then publish it.”</blockquote><p>Hex built a <strong>Generative Data App</strong> — a full interactive application with layout, components, and live data. Inside it we found:</p><ul><li><strong>Summary cards:</strong> 98 total reviews, 6 safety alerts (red), 19 critical priority (amber), top defect: compressor/motor noise (23 occurrences, median 6 months to onset)</li><li><strong>Filters:</strong> priority and complaint topic multiselects with live result count</li><li><strong>Bar chart:</strong> reviews by complaint topic — Quality defect dominates at 58, followed by Setup &amp; maintenance (16), Noise level (9)</li><li><strong>Review table:</strong> priority-sorted with colored badges (Safety red → Critical amber → High → Medium → Low), sentiment scores in red for negative, click-to-expand for full review body and model rationale</li></ul><p>One click to publish. The result is a live URL anyone on the team can open.</p><p><a href="https://app.hex.tech/snowflake/app/033m48khnOyIxAaMb5Sy77/latest"><strong>Open “What 3.7 Stars Won’t Tell You” →</strong></a></p><p>Explore 98 GE Opal reviews enriched with Cortex AI Functions. See the safety alerts, the topic breakdown, the defect extraction. Fork the notebook and swap in your own product’s reviews.</p><p><strong>From a free Marketplace dataset to a published, AI-powered product intelligence app. One person. One notebook session. Four prompts.</strong></p><p>The traditional version of this — data engineering exports the reviews, an NLP team builds a classification model, a BI team designs the dashboard, a product manager writes the spec for priority tiers, an analyst maintains the refresh schedule — crosses four teams and takes weeks. This took one session.</p><h3>This Isn’t Just for Product Reviews</h3><p>The GE Opal story is one instance of a general pattern. Swap the data source and the classification labels — the architecture is identical:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XDiQ32m03xcEsSdMSTNL-Q.png" /></figure><p>The pattern is always: unstructured text → AI functions for classification, extraction, and filtering → app for stakeholders. Only the data and labels change.</p><h3>How It Works Under the Hood</h3><p>Every AI capability in this walkthrough executes inside Snowflake as SQL-native functions. You use them the same way you’d use COUNT() or SUM().</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*lIws8ci2JJZKBa8zGtIYfw.png" /></figure><p>Snowflake offers additional functions — AI_TRANSLATE (multilingual reviews before analysis), AI_REDACT (strip PII before sharing), AI_PARSE_DOCUMENT (PDFs and scanned forms), AI_TRANSCRIBE (call recordings) — that compose with the above for different data types.</p><h4><strong>What this architecture means in practice:</strong></h4><ul><li><strong>No data leaves your account.</strong> The review text is processed in-platform. For teams handling customer data under GDPR, HIPAA, or internal DLP policies, this is often the difference between “approved” and “blocked by legal.”</li><li><strong>Governance applies automatically.</strong> If a user can’t SELECT a column, they can’t feed it to an AI function. Your existing masking policies, row-level security, and audit logging work without any additional configuration.</li><li><strong>No infrastructure to manage.</strong> No API keys, no model hosting, no GPU clusters, no token limits to monitor. It’s a SQL function.</li></ul><p><strong>On production pipelines:</strong> for this demo, the data is static (a Marketplace dataset). In production with your own live data, you’d wrap the enrichment in a <strong>dynamic table</strong> with a target lag — new reviews flow through AI enrichment automatically on each refresh cycle, and the app always shows current state. Cortex AI Functions support incremental refresh, so you only pay for new rows, not re-processing the entire table.</p><h3>Who Opens This App — and What Changes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y8kmx1HIJQYTB4oLCJMKgQ.png" /></figure><h3>Start Building</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HJZkeTrdd03wVHJAL7UhuA.png" /></figure><p>The distance between “I have unstructured data” and “I have a published AI-powered app my VP bookmarks” used to require four teams and weeks of coordination. Today it requires one person, one notebook, and four prompts.</p><p><a href="https://app.hex.tech/snowflake/app/033m48khnOyIxAaMb5Sy77/latest"><strong>Open the published app →</strong></a> | <a href="https://app.snowflake.com/marketplace/listing/GZTYZ36VTKF/datahive-ai-amazon-best-sellers-ratings-and-reviews?search=Amazon+Best+Sellers+Ratings+and+Reviews"><strong>Get Amazon Reviews (free) →</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fcaa1c7058b2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/what-3-7-stars-wont-tell-you-fcaa1c7058b2">What 3.7 Stars Won’t Tell You</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Battle for Compute: Unlocked by Iceberg V3 and Snowflake Interoperability]]></title>
            <link>https://medium.com/snowflake/the-battle-for-compute-unlocked-by-iceberg-v3-and-snowflake-interoperability-603c59773c6b?source=rss-5ed895c8dd32------2</link>
            <guid isPermaLink="false">https://medium.com/p/603c59773c6b</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[modern-data-architecture]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[snowflake-data-cloud]]></category>
            <dc:creator><![CDATA[Naveen Alan]]></dc:creator>
            <pubDate>Mon, 30 Mar 2026 19:01:02 GMT</pubDate>
            <atom:updated>2026-03-30T19:01:02.098Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why Interoperability Actually Matters to Us</h4><p><strong>Interoperability.</strong> If you&#39;re an architect like me, you probably hear this word in your sleep. But it&#39;s more than just a buzzword; it&#39;s about freedom. Apache Iceberg won because it let us stop worrying about where to store our data and focus on what we can actually do with it.</p><p>Think about it: Snowflake was built on the idea of separating storage from compute. Now, with Iceberg, Snowflake is really leaning into that heritage, meeting our data exactly where it lives instead of forcing us into a proprietary corner.</p><p>Let&#39;s be honest—storage is basically a commodity now. The real battle we&#39;re all fighting is over <strong>compute</strong>. Whether it&#39;s AI Agents or new MCP endpoints, everyone wants the heavy lifting to happen on <em>their</em> engine. But as users, we just want it to work wherever it makes the most sense.</p><p>We are seeing a historic shift toward partnership to ensure customer success. The recent <a href="https://blog.fabric.microsoft.com/en-us/blog/microsoft-onelake-and-snowflake-interoperability-is-now-generally-available?ft=All"><strong>Microsoft and Snowflake collaboration</strong></a> is a prime example — <strong>OneLake</strong> interoperability with Snowflake has reached General Availability, enabling natively storing Snowflake-managed Iceberg tables in OneLake or &quot;shortcutting&quot; data into Microsoft Fabric without moving a single byte. It&#39;s a clear signal that platform vendors have realized success depends on a &quot;both-and&quot; scenario, not a proprietary silo.</p><p>I&#39;ve been watching Snowflake lately, and their release pace is honestly impressive. They seem determined to make sure we don&#39;t fail. Today, we’re diving into a critical ‘unlock’: <a href="https://docs.snowflake.com/en/user-guide/tables-iceberg-v3-specification-support"><strong>Iceberg V3</strong></a> support and Snowflake’s <strong>two complementary interoperability paths</strong> — <a href="https://docs.snowflake.com/en/user-guide/tables-iceberg-catalog-linked-database"><strong>Catalog-Linked Databases</strong></a> (CLD) and <a href="https://www.snowflake.com/en/engineering-blog/bidirectional-interoperability-iceberg-snowflake-horizon-catalog/"><strong>Horizon IRC</strong></a> (Iceberg REST Catalog). Together, they deliver truly bidirectional access. They&#39;re the &quot;secret sauce&quot; for keeping data accessible <strong>and writable</strong> across engines while automating your pipelines.</p><p><strong>Note:</strong> <a href="https://docs.snowflake.com/en/user-guide/tables-iceberg-v3-specification-support">Iceberg V3</a> support in Snowflake is currently in <strong>Public Preview</strong> (available to all accounts). Features discussed here are production-ready for testing but may evolve before GA.</p><p>Full replication scripts for everything in this post are available on <a href="https://github.com/sfc-gh-naveenalanthomas/iceberg-v3-snowflake-interoperability">GitHub</a>.</p><p>Below is the simplified architecture diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rP5qTvjFP2hOFCf5IjAoLQ.png" /></figure><h3>The Missing Link: Row Lineage &amp; Deletion Vectors</h3><p>For a long time, we had a technical &quot;missing link&quot;: <strong>Identity</strong>. In the old data lake world, if you updated a record, the row&#39;s identity just vanished into a new Parquet file. It made CDC a nightmare, often forcing us into limited INSERT_ONLY streams.</p><p><strong>Iceberg V3 changes the game with Row Lineage.</strong> It’s what transforms a “Data Lake” into a “Live Database” that Snowflake can actually follow. It&#39;s like giving every row its own permanent serial number (the METADATA$ROW_ID) the moment it&#39;s created.</p><ul><li><strong>The Old Way:</strong> Snowflake would see a file changed, but it couldn&#39;t tell if you added a new customer or just updated an old one.</li><li><strong>The V3 Way:</strong> Because of that ID, Snowflake knows exactly what happened. If one ID leaves and a new one arrives with the same key, it says, <em>&quot;Aha, that&#39;s an update.&quot;</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*fyyx74KvLxexy4Il_XMjsw.png" /><figcaption>Identity Model</figcaption></figure><blockquote>My favorite part? <strong>Deletion Vectors</strong>. In our 10 million row scale test, the difference was striking: a single UPDATE affecting 1,000 rows generated a <strong>493-byte</strong> Puffin marker instead of rewriting a <strong>~2MB</strong> Parquet file. That results in a <strong>4,358 times reduction</strong> in write overhead.</blockquote><p>You can inspect this directly:</p><pre>SELECT<br>    CASE<br>        WHEN FILE_NAME LIKE &#39;%-deletes.puffin&#39; THEN &#39;DELETION_VECTOR&#39;<br>        WHEN FILE_NAME LIKE &#39;%.parquet&#39; THEN &#39;DATA_FILE&#39;<br>    END AS file_type,<br>    COUNT(*) AS file_count,<br>    SUM(FILE_SIZE) AS total_bytes,<br>    AVG(FILE_SIZE)::INT AS avg_bytes<br>FROM TABLE(<br>    INFORMATION_SCHEMA.ICEBERG_TABLE_FILES(<br>        &#39;glue_iceberg_db.&quot;your_database&quot;.&quot;orders_scale_10m&quot;&#39;<br>    )<br>)<br>GROUP BY file_type;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*dXXG5wxrsE5TNlAGJgBa8g.png" /><figcaption>Deletion Vector puffin marker</figcaption></figure><p>At production scale (billion-row tables with GB-sized files), this is the difference between minutes and hours of CDC lag.</p><h3>A Quick Heads-Up on Engines</h3><p>Just a warning from my own testing: not every engine is ready for prime time with row lineage yet. Here&#39;s what I found:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*phmZyW_Ol-t6Rw7pHOMD5Q.png" /><figcaption>Critical Engine Caveat</figcaption></figure><p><strong>Glue version matters.</strong> Glue 5.0 ships Iceberg 1.7.1, which predates V3 format support — the next-row-id field in metadata is null, and Snowflake cannot create standard streams. <strong>Glue 5.1</strong> (Nov 2025) upgraded to Iceberg 1.10.0 and added full V3 row lineage support. If you&#39;re on Glue, ensure you&#39;re running <strong>5.1+</strong>. For EMR, <strong>7.12+</strong> works out of the box.</p><p>You can verify row lineage in your metadata:</p><pre>aws s3 cp s3://your-bucket/.../metadata/latest.metadata.json - | grep next-row-id<br># Must return a number (e.g., &quot;next-row-id&quot;: 6), NOT null</pre><h3>Keeping Control: Governance Without the Headache</h3><p>I often get asked: <em>&quot;If I put my data in an open bucket, don&#39;t I lose control?&quot;</em> Honestly, it&#39;s the opposite.</p><h4>1. Unified RBAC &amp; Horizon</h4><p>Snowflake&#39;s RBAC works on Iceberg just like it does for native tables. You get all the power of <a href="https://www.snowflake.com/en/product/features/horizon/"><strong>Snowflake Horizon</strong></a>—search, lineage, and monitoring—without feeling like your data is locked in a basement.</p><p>Here&#39;s the proof — masking policies, object tags, and row access policies all work on externally managed Iceberg tables:</p><pre>CREATE OR REPLACE MASKING POLICY mask_financial_amount<br>AS (val NUMBER(10,2))<br>RETURNS NUMBER(10,2) -&gt;<br>CASE<br>WHEN CURRENT_ROLE() IN (&#39;FINANCE_ADMIN&#39;, &#39;ACCOUNTADMIN&#39;) THEN val<br>ELSE -1.00<br>END;<br><br>ALTER ICEBERG TABLE glue_iceberg_db.&quot;your_database&quot;.&quot;customer_orders_v3&quot;<br>MODIFY COLUMN &quot;amount&quot; SET MASKING POLICY mask_financial_amount;<br><br>ALTER ICEBERG TABLE glue_iceberg_db.&quot;your_database&quot;.&quot;customer_orders_v3&quot;<br>SET TAG SNOWFLAKE.CORE.SEMANTIC_CATEGORY = &#39;FINANCIAL&#39;;</pre><p>These policies <strong>survive schema evolution</strong>. If your external engine adds or renames columns, masking and row access policies on existing columns stay enforced.</p><h4>2. Catalog-Linked Databases (CLD): The Bidirectional Bridge</h4><p>A <a href="https://docs.snowflake.com/en/user-guide/tables-iceberg-catalog-linked-database">Catalog-Linked Database</a> is a Snowflake database that “links” directly to an external REST catalog (like AWS Glue or Unity Catalog). <strong>Critically, CLD is now bidirectional by default.</strong> Snowflake doesn’t just consume from your external catalog; it can write back to it. You can run INSERT, UPDATE, DELETE, and even CREATE TABLE from Snowflake, and the changes are immediately reflected in your external catalog and storage. You apply your masking policies once in Snowflake, and they stay enforced even as the external schema evolves. [Additional read — <a href="https://www.snowflake.com/en/engineering-blog/catalog-linked-databases-cortex-code-cli-iceberg/">Jeemin Sim’s Blog</a>]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dwfhm-d2ttLTBKAziUHm2w.png" /><figcaption>The Bridge Model</figcaption></figure><pre>-- Creating a CLD pointing to the Glue catalog<br><br>CREATE DATABASE glue_iceberg_db<br>LINKED_CATALOG = (<br>CATALOG = &#39;glue_rest_catalog&#39;<br>EXTERNAL_VOLUME = &#39;s3_iceberg_volume&#39;<br>AUTO_REFRESH = TRUE<br>);<br><br>-- Here&#39;s the write-back in action:<br>-- Creating and populating a new table in Glue directly from Snowflake<br>USE DATABASE glue_iceberg_db;<br><br>CREATE ICEBERG TABLE &quot;your_database&quot;.&quot;orders_summary&quot; (<br>    &quot;region&quot; VARCHAR,<br>    &quot;total_orders&quot; INT,<br>    &quot;total_revenue&quot; NUMBER(12,2)<br>)<br>ICEBERG_VERSION = 3;<br><br>INSERT INTO &quot;your_database&quot;.&quot;orders_summary&quot;<br>SELECT &quot;region&quot;, COUNT(*), SUM(&quot;amount&quot;)<br>FROM &quot;your_database&quot;.&quot;customer_orders_v3&quot;<br>GROUP BY &quot;region&quot;;</pre><blockquote><strong><em>Why this matters:</em></strong><em> This writes Parquet files to your S3 bucket and registers the table in the Glue catalog simultaneously. Your EMR or Spark clusters can immediately query </em><em>orders_summary from the Glue catalog. This is true bidirectional interoperability.</em></blockquote><p>One nuance: CLD detects schema changes only when a <strong>new snapshot arrives</strong> (i.e., a data commit). Schema-only changes like ALTER TABLE ADD COLUMN won&#39;t sync until the next DML creates a new snapshot.</p><h4>3. Two Paths, One Governance Layer: CLD + Horizon IRC</h4><p>Snowflake offers two directions for Iceberg interoperability, and the choice depends on <strong>who owns the catalog</strong>:</p><ul><li><strong>Path 1 — CLD: </strong>Your external catalog (Glue, Unity, Polaris) is the source of truth. Snowflake links to it, auto-syncs schemas and snapshots, and can read <em>and write</em> back. This is ideal when you have an existing lakehouse and want to add Snowflake’s governance and compute without migrating.</li><li><strong>Path 2 — Horizon IRC:</strong> Snowflake’s Horizon Catalog is the source of truth. External engines like Spark, Trino, and Flink connect via the open Iceberg REST Catalog protocol (powered by Apache Polaris) and can read <em>and write</em> directly to Snowflake-managed Iceberg tables. External reads are GA; external writes are in Public Preview as of March 2026.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XY3X30KX4WJwLAmTyTXhVA.png" /></figure><blockquote><strong>The constant?</strong> <strong>Both paths share the same Horizon governance layer</strong>. Whether data enters via CLD or IRC, masking policies, object tags, row access policies, and data lineage all apply uniformly.</blockquote><p>For the full Horizon IRC walkthrough with Spark configuration and code, see <a href="https://www.snowflake.com/en/engineering-blog/bidirectional-interoperability-iceberg-snowflake-horizon-catalog/">Ashwin Kamath’s deep dive</a>.</p><h3>Let&#39;s Build It: CDC and Schema Sync</h3><p>I spent some time testing this with <strong>EMR 7.12</strong>. Here&#39;s how I set it up so you can try it yourself. All the scripts are over on <a href="https://github.com/sfc-gh-naveenalanthomas/iceberg-v3-snowflake-interoperability">my GitHub repo</a> if you want to fork them.</p><h4>1. EMR Spark Setup</h4><p>First, we&#39;ll create the table in EMR. We need to make sure we&#39;re explicitly using V3 and enabling merge-on-read to get those sweet deletion vectors.</p><p><a href="https://github.com/sfc-gh-naveenalanthomas/iceberg-v3-snowflake-interoperability/blob/main/scripts/create_v3_table.py">scripts/create_v3_table.py</a></p><pre>from pyspark.sql import SparkSession<br>spark = SparkSession.builder \<br>.appName(&quot;CreateV3TableWithRowLineage&quot;) \<br>.config(&quot;spark.sql.catalog.glue_catalog&quot;, &quot;org.apache.iceberg.spark.SparkCatalog&quot;) \<br>.config(&quot;spark.sql.catalog.glue_catalog.catalog-impl&quot;, &quot;org.apache.iceberg.aws.glue.GlueCatalog&quot;) \<br>.config(&quot;spark.sql.catalog.glue_catalog.warehouse&quot;, &quot;s3://your-iceberg-data/glue_tables&quot;) \<br>.getOrCreate()<br>spark.sql(&quot;&quot;&quot;<br>CREATE TABLE glue_catalog.your_database.customer_orders_v3 (<br>order_id INT,<br>customer_id INT,<br>order_date DATE,<br>product_name STRING,<br>amount DECIMAL(10,2),<br>region STRING<br>)<br>USING iceberg<br>TBLPROPERTIES (<br>&#39;format-version&#39; = &#39;3&#39;,<br>&#39;write.delete.mode&#39; = &#39;merge-on-read&#39;,<br>&#39;write.update.mode&#39; = &#39;merge-on-read&#39;,<br>&#39;write.merge.mode&#39; = &#39;merge-on-read&#39;<br>)<br>&quot;&quot;&quot;)<br>spark.sql(&quot;INSERT INTO glue_catalog.your_database.customer_orders_v3 VALUES ...&quot;)<br>spark.sql(&quot;UPDATE glue_catalog.your_database.customer_orders_v3 SET amount = 299.99 WHERE order_id = 2&quot;)<br>spark.sql(&quot;ALTER TABLE glue_catalog.your_database.customer_orders_v3 ADD COLUMN region STRING&quot;)</pre><h4>2. Snowflake Catalog-Linked Database &amp; Stream</h4><p>Once the catalog is linked, we just flip the switch to V3 and create a standard stream. <strong>Standard Streams</strong> (Full Change Tracking) are now possible on externally managed tables for the first time. This is the specific feature that enables <strong>Dynamic Iceberg Tables (DIT)</strong> to be efficient — without this, DITs would have to do full table scans.</p><p><a href="https://github.com/sfc-gh-naveenalanthomas/iceberg-v3-snowflake-interoperability/blob/main/sql/02_streams_and_cdc.sql">sql/02_streams_and_cdc.sql</a></p><pre>ALTER DATABASE my_db SET ICEBERG_VERSION_DEFAULT = 3;<br><br>SHOW PARAMETERS LIKE &#39;ICEBERG_VERSION&#39;<br>  IN TABLE glue_iceberg_db.&quot;your_database&quot;.&quot;customer_orders_v3&quot;;<br>-- Confirm: ICEBERG_VERSION = 3<br><br>CREATE OR REPLACE STREAM cdc_stream<br>    ON TABLE glue_iceberg_db.&quot;your_database&quot;.&quot;customer_orders_v3&quot;;<br><br>SHOW STREAMS LIKE &#39;CDC_STREAM&#39;;<br>-- Confirm: mode = DEFAULT (standard), type = DELTA<br>-- This is the breakthrough — it&#39;s a real delta stream, not just insert-only!<br><br>SELECT<br>    &quot;order_id&quot;,<br>    &quot;product_name&quot;,<br>    METADATA$ACTION,<br>    METADATA$ISUPDATE,<br>    METADATA$ROW_ID<br>FROM cdc_stream<br>ORDER BY &quot;order_id&quot;, METADATA$ACTION DESC;</pre><p><strong>CDC Results</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RhHa81pEpYkIXy-HuPwuqA.png" /></figure><p><strong>How to make sense of this:</strong></p><ul><li><em>INSERT + ISUPDATE = false</em> → new row</li><li><em>DELETE + INSERT + ISUPDATE = true</em> → an UPDATE (old value deleted, new value inserted, same logical key)</li><li><em>DELETE + ISUPDATE = false</em> → a true delete</li><li><em>METADATA$ROW_ID</em> → the V3 row lineage identifier</li></ul><p><strong>Note:</strong> Column names from Glue/Spark are case-sensitive in Snowflake. Use double quotes (e.g., “<em>order_id</em>”, not <em>ORDER_ID</em>).</p><h3>The Final Piece: Dynamic Iceberg Tables</h3><p>Now that CDC is working, you can finally use <a href="https://docs.snowflake.com/en/user-guide/dynamic-tables-create-iceberg"><strong>Dynamic Iceberg Tables</strong></a>. The cool thing here is that the <em>entire</em> pipeline stays in the open format. Snowflake reads the V3 changes and writes them back to your bucket. You get the automation without the lock-in.</p><p><a href="https://github.com/sfc-gh-naveenalanthomas/iceberg-v3-snowflake-interoperability/blob/main/sql/05_dynamic_iceberg_table.sql">sql/05_dynamic_iceberg_table.sql</a></p><pre>CREATE DYNAMIC ICEBERG TABLE customer_orders_enriched (<br>order_id NUMBER(10,0),<br>product_name VARCHAR,<br>amount NUMBER(10,2),<br>region VARCHAR,<br>order_month VARCHAR<br>)<br>TARGET_LAG = &#39;20 minutes&#39;<br>WAREHOUSE = compute_wh<br>EXTERNAL_VOLUME = &#39;s3_iceberg_volume&#39;<br>CATALOG = &#39;SNOWFLAKE&#39;<br>BASE_LOCATION = &#39;enriched/customer_orders&#39;<br>ICEBERG_VERSION = 3<br>AS<br>SELECT<br>&quot;order_id&quot;,<br>&quot;product_name&quot;,<br>&quot;amount&quot;,<br>&quot;region&quot;,<br>TO_CHAR(&quot;order_date&quot;, &#39;YYYY-MM&#39;) AS order_month<br>FROM glue_iceberg_db.&quot;your_database&quot;.&quot;customer_orders_v3&quot;;</pre><p><strong>Note:</strong> DIT output must be Snowflake-managed (CATALOG = ‘SNOWFLAKE’). The <em>source</em> can be externally managed via CLD.</p><p>It really completes the circle: your raw data comes in, gets enriched, and stays business-ready — all while staying open and automated.</p><h3>CLD CDC: Spark create_changelog_view</h3><p>If your pipeline stays in Spark, you can consume V3 changes natively using the Iceberg system procedure:</p><pre>CALL glue_catalog.system.create_changelog_view(<br>    table =&gt; &#39;your_database.customer_orders_v3&#39;,<br>    options =&gt; map(&#39;compute_updates&#39;, &#39;true&#39;)<br>);<br>SELECT * FROM customer_orders_v3_changes;<br>-- Returns: INSERT, DELETE, UPDATE_BEFORE, UPDATE_AFTER</pre><p>This gives Spark the same row-level change semantics that Snowflake streams provide — useful when your CDC consumer is another Spark job rather than a Snowflake pipeline.</p><h3>Horizon IRC CDC: Spark .changes API</h3><p>The section above covers CDC for <strong>externally managed tables</strong> (CLD pattern) — where the table lives in Glue or Unity, and Spark queries its changelog. But what about the inverse? If Snowflake is the catalog owner (Horizon IRC), external engines can still consume row-level CDC using the same Iceberg .changes suffix.</p><p>First, connect Spark to Horizon Catalog via the Iceberg REST API:</p><pre>from pyspark.sql import SparkSession<br><br>spark = SparkSession.builder \<br>    .appName(&quot;Horizon IRC - CDC Consumer&quot;) \<br>    .config(&quot;spark.sql.extensions&quot;, &quot;org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions&quot;) \<br>    .config(&quot;spark.sql.defaultCatalog&quot;, &quot;horizon&quot;) \<br>    .config(&quot;spark.sql.catalog.horizon&quot;, &quot;org.apache.spark.sql.snowflake.catalog.SnowflakeFallbackCatalog&quot;) \<br>    .config(&quot;spark.sql.catalog.horizon.catalog-impl&quot;, &quot;org.apache.iceberg.spark.SparkCatalog&quot;) \<br>    .config(&quot;spark.sql.catalog.horizon.type&quot;, &quot;rest&quot;) \<br>    .config(&quot;spark.sql.catalog.horizon.uri&quot;, f&quot;https://{SNOWFLAKE_ACCOUNT_URL}.snowflakecomputing.com/polaris/api/catalog&quot;) \<br>    .config(&quot;spark.sql.catalog.horizon.warehouse&quot;, SNOWFLAKE_DATABASE) \<br>    .config(&quot;spark.sql.catalog.horizon.scope&quot;, f&quot;session:role:{SNOWFLAKE_ROLE}&quot;) \<br>    .config(&quot;spark.sql.catalog.horizon.credential&quot;, SNOWFLAKE_PAT) \<br>    .config(&quot;spark.sql.catalog.horizon.io-impl&quot;, &quot;org.apache.iceberg.aws.s3.S3FileIO&quot;) \<br>    .config(&quot;spark.sql.catalog.horizon.header.X-Iceberg-Access-Delegation&quot;, &quot;vended-credentials&quot;) \<br>    .getOrCreate()</pre><p>Key details: Horizon vends temporary, scoped storage credentials to Spark at runtime — no long-lived S3 keys needed. The vended-credentials header triggers this behavior.</p><p>Once connected, query the changelog by appending .changes to any Iceberg V3 table:</p><pre>SELECT * FROM horizon.PUBLIC.DEMO_CDC.changes<br>ORDER BY _change_ordinal, ID;</pre><p>For incremental processing, filter by snapshot or timestamp range. Note the explicit IcebergSource format — required on Databricks to avoid defaulting to the shaded Databricks library:</p><pre>changes_between_snapshots_df = (<br>    spark.read.format(&quot;org.apache.iceberg.spark.source.IcebergSource&quot;)<br>        .option(&quot;start-snapshot-id&quot;, start_snapshot)<br>        .option(&quot;end-snapshot-id&quot;, end_snapshot)<br>        .load(&quot;horizon.PUBLIC.DEMO_CDC.changes&quot;)<br>)</pre><pre>changes_between_timestamps_df = (<br>    spark.read.format(&quot;org.apache.iceberg.spark.source.IcebergSource&quot;)<br>        .option(&quot;start-timestamp&quot;, START_TS)<br>        .option(&quot;end-timestamp&quot;, END_TS)<br>        .load(&quot;horizon.PUBLIC.DEMO_CDC.changes&quot;)<br>)</pre><p>This completes the CDC picture: Snowflake Streams consume changes from <strong>external catalogs</strong> (CLD), while Spark’s .changes API consumes changes from <strong>Snowflake-managed tables</strong> (Horizon IRC). Both directions are powered by Iceberg V3 row lineage.</p><blockquote><em>Spark IRC connection code and CDC examples adapted from </em><a href="https://medium.com/@scottteal/integrating-snowflake-and-databricks-with-iceberg-v3-f162be001d0b"><em>Scott Teal’s integration guide</em></a><em>.</em></blockquote><h3>Conclusion: Don’t Build Bridges, Build Foundations</h3><p>We’re seeing the “walled garden” era end right before our eyes. Open standards like <strong>Iceberg V3</strong> are proving that the future belongs to compute engines that actually talk to each other. By moving away from proprietary silos, we’re finally moving toward a world where data is a shared foundation, not a locked asset.</p><p>Leveraging features like <strong>Catalog-Linked Databases</strong> allows you to stop paying the “metadata tax” of maintaining dual systems. Your data stays in an open format, your external engines stay fast, and Snowflake handles the heavy lifting of syncing and governance — all while giving you the flexibility to write back to the lake whenever your business logic demands it.</p><p>For architects, the question is no longer <em>“Should I use Snowflake or my lakehouse?”</em> It has evolved into something much more strategic: <strong>“Which catalog should own this specific table?”</strong> Whether the answer is Glue, Unity, or Snowflake Horizon, you now have full bidirectional access and unified governance across the board. It’s a win-win for the ecosystem, and more importantly, a win for the engineers building it.</p><p>The full code for everything in this post — infrastructure setup, EMR scripts, CDC streams, deletion vector inspection, Horizon IRC integration, governance policies, Dynamic Iceberg Tables, and a MERGE consumption pattern — is available on <a href="https://github.com/sfc-gh-naveenalanthomas/iceberg-v3-snowflake-interoperability">GitHub</a>.</p><p><em>Which path are you exploring — CLD to bring your lakehouse into Snowflake, or Horizon IRC to open Snowflake-managed tables to your Spark and Trino clusters? Or both? I’d love to hear about the architectural trade-offs you’re weighing in the comments.</em></p><p><strong>Disclaimer:</strong></p><p><em>The opinions expressed in this post are solely my own and do not represent the views or opinions of my employer.</em></p><ul><li>*As of:** March 25, 2026 (validated in the environment described in the repo). **Iceberg v3 in Snowflake is Public Preview** — behavior and GA timing can change; treat this post as a walkthrough, not a contract.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=603c59773c6b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/the-battle-for-compute-unlocked-by-iceberg-v3-and-snowflake-interoperability-603c59773c6b">The Battle for Compute: Unlocked by Iceberg V3 and Snowflake Interoperability</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NativeApp and Snowflake Series — How to build an XGboost Forecasting App]]></title>
            <link>https://medium.com/snowflake/nativeapp-and-snowflake-series-how-to-build-an-xgboost-forecasting-app-86a6860b8318?source=rss-5ed895c8dd32------2</link>
            <guid isPermaLink="false">https://medium.com/p/86a6860b8318</guid>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[snowpark-python]]></category>
            <category><![CDATA[manufacturing]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Naveen Alan]]></dc:creator>
            <pubDate>Tue, 12 Mar 2024 12:01:40 GMT</pubDate>
            <atom:updated>2024-03-12T12:01:40.684Z</atom:updated>
            <content:encoded><![CDATA[<h3>Snowflake Native App and Snowflake Series — How to build an XGboost Forecasting App</h3><blockquote>If you love to build a highly secured applications in production, then our Native Application is the way forward. Please refer to the Snowflake <a href="https://docs.snowflake.com/en/developer-guide/native-apps/native-apps-about">doc</a> or my previous <a href="https://medium.com/@alandeepak/data-engineering-and-snowflake-series-part-2-app-development-nativeapp-d2dde21b1a9">blog</a> for info on Native application.</blockquote><p>With the advent of competition, hourly electricity prices are being determined by a variety of market mechanisms, rather than cost-based engineering calculations. As a result, electric utilities, generators, and traders face a new set of short-term forecasting problems. These problems are unlike those in other industries since electricity must be produced at the same time that it is consumed. As a result, prices are determined hourly, 24 hours a day, 7 days a week.</p><p>Historically, price forecasting has been performed with least-cost optimization models. These models compute marginal cost based on assumptions about system loads, power plant availability, and fuel prices. These models do not explain price variations related to market strategy and to buyer and seller behavior in a market system. Statistical models, which reflect actual market outcomes, are better suited to short-term forecasting in this dynamic environment.</p><p>With this context, let us build a time-series model forecasting which is widely used in this space. This app can perform Feature selection (from the data from <a href="https://www.snowflake.com/en/data-cloud/marketplace/">Marketplace</a>), Hyper-parameter tuning, Model training and building, Backtesting, Monitor performance, and Deploying the model for Energy price forecasting use-case. This demo was built using the support of @JenPrusa who built the DS components and <a href="https://medium.com/u/3358ce85c540">Titiaan Palazzi</a> who is the brain behind this.</p><h3>Energy Price Forecasting Application</h3><h4>Components of Native Application</h4><p>To begin with, we should understand what are the components of Native Application. There are 3 major components of the Native Application object. They are as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R6wDFCOwWpkY6yHdRCTw1Q.png" /><figcaption>Components of Native Application</figcaption></figure><h4>Prerequisites:</h4><ol><li>Snowflake trial account. Signup <a href="https://signup.snowflake.com/">here</a>. Role with access to install NativeApps from a marketplace like ACCOUNTADMIN</li><li>Agree to Anaconda Python Packages using ORGADMIN role under the “Billing &amp; Terms” section (if not enabled by default).</li><li>You must have conda/miniconda installed. If not, please install following the <a href="https://docs.conda.io/projects/miniconda/en/latest/">doc</a>.</li><li>Any environment to run notebooks like <strong>Jupyterlab</strong>, <strong>Jupyter Notebook,</strong> etc.,</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5uUu6CblFW-y-Kc9IgHFSA.png" /></figure><h3>Step by Step Instructions to build the App:</h3><h4>1. Bringing in Data:</h4><p>The data we use is available in the <a href="https://www.snowflake.com/en/data-cloud/marketplace/">Marketplace</a> for free. Thanks to <a href="https://medium.com/u/46cee027465a">Yes Energy</a> and <a href="https://medium.com/u/268c4312e18e">Samuel Lockshin</a> for making the sample data available for building the demo.</p><ol><li>Please navigate to the Marketplace tab under Data Products</li><li>Ensure you are in the “ACCOUNTADMIN” role</li><li>Type in “Yes Energy” in the search box</li><li>Select the provider “Yes Energy”</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c7msUl0sCIq6ysKDDpWBMA.png" /></figure><ol><li>Select the “Yes Energy — Sample Data”</li><li>Click “Get”</li><li>Click “Get” again on the next screen</li><li>Click “Done” on the final screen.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mGH2u6akuz3u5P_aQrg5TQ.png" /></figure><p>That’s it, within a few clicks the whole 3 years’ worth of data is available for you to play around. Isn’t it amazing!!! Thanks for the <a href="https://docs.snowflake.com/en/guides-overview-sharing">data-sharing</a> feature of Snowflake.</p><h4>2. Replicating the Algorithm in your local:</h4><p>All our code is in Python, hence we use <a href="https://docs.snowflake.com/en/developer-guide/snowpark/index">Snowpark</a> to run them within Snowflake.</p><p>The link to the gist is available <a href="https://gist.github.com/naveenalanthomas/d443b72da6540cf7d126abb50b2710cf">here</a>. Please download the gist to your local and follow the folder structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/354/1*-TlqGZmSo-R0TWbUhOk9NQ.png" /><figcaption>Local folder structure</figcaption></figure><h4>NativeApp Folders:</h4><p>Now, please create a folder named “energy_price_forecasting_nativeapp” and create subfolders “python”, “scripts” and “streamlit” as per the below screenshot. The final structure should be as below now. Copy the respective files from the gist <a href="https://gist.github.com/naveenalanthomas/68a6ebb7578cec025c8aaca87b76b046">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*P3fAPqrSdLpuxV3h8TfMrg.png" /><figcaption>Native App folder structure</figcaption></figure><h4>3. Building Native Application</h4><p>To build the application, just run all the cells in the <a href="https://gist.github.com/naveenalanthomas/d443b72da6540cf7d126abb50b2710cf#file-setup_energy_price_forecasting-ipynb">Setup_Energy_Price_Forecasting.ipynb</a> and the app gets deployed in your Snowflake account.</p><p>Currently, <a href="https://streamlit.io/">Streamlit</a> is the only supported UI in the Native Application framework, with <a href="https://docs.snowflake.com/en/developer-guide/snowpark-container-services/overview">SPCS</a> in the future this will change. Hence our app also uses Streamit to build the front-end UI to our Energy Price Forecasting Native application.</p><h3>Forecast Price and deploy via the Native App</h3><p>Follow the below steps to forecast and deploy the model in Snowflake.</p><h4>1. Feature Selection:</h4><p>Please select the right view, target column, and all the features in the dropdowns and click “SELECT FEATURES” as shown in the picture below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MLgIb3zStD3MpwpHuh2I9w.png" /><figcaption>Feature Selection</figcaption></figure><h4>2. Exploration:</h4><p>Based on the features selected, the exploration part changes dynamically. With all the features selected, it should be as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JpaJD9Rhz22JzQGVJ_zfZw.png" /><figcaption>Data Exploration and Correlation</figcaption></figure><h4>3. Hyperparameter optimization:</h4><p>Please click the “Run hyperparameter optimization” to run the proc <em>sproc_optuna_optimized_model. </em>The parameter results will vary for each person, so do not worry if your results are different.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sSuMKzk3HG5LTgoR6U-qAA.png" /></figure><h4>4. Build model and run backtest:</h4><p>Once the hyperparameter optimization is complete, please click “Run Backtest” which will build the forecasting model, upload the <em>.sav</em> file to the snowflake stage, and then run a backtest using the latest model file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lG6iDTK9rhvYyXX6LddU6Q.png" /><figcaption>Backtest</figcaption></figure><h4>4. View model performance:</h4><p>Now, we can see some performance metrics of the model that we built in the next block.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xuK-zY0WwP-Pe_fJXRveSw.png" /><figcaption>Performance metrics</figcaption></figure><h4>5. Deploy Model:</h4><p>Once you are happy with the model, you can deploy the model by clicking “Deploy Model” which will run inferencing using the selected model and display the predicted results alongside actual data.</p><blockquote>Note: <a href="https://docs.snowflake.com/en/user-guide/dynamic-tables-about">Dynamic table</a> is a new feature that will simplify deployment. This feature will aid in saving maintenance overhead. Another feature called <a href="https://docs.snowflake.com/en/developer-guide/snowpark-ml/snowpark-ml-mlops-model-registry">Model registry </a>is coming up as well which will avoid holding the model files in stages and gives you a registry to maintain model versions and its associated metadata.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NyPO3luCKJS4hRxysW_WyQ.png" /><figcaption>Model deployment</figcaption></figure><blockquote>You can repeat the same process for spike forecasting as well if you are interested.</blockquote><h3>Thank you!</h3><p>Glad you all reached here! Yes, now you have built your first Native Application in Snowflake for free which is highly secured, highly scalable, and sharable. Secure, because the data never leaves Snowflake, bringing the processing as close as possible to the data improves the data security and the performance. Scalable, the storage and computing are independent of each other, hence our application is highly scalable. This is sharable because you can share this application with another Snowflake account in the same region (this needs Enterprise+ Snowflake edition), so you can build an app and share it with your vendors or customers in the same region securely. If you or your organization already have a product you can very well build your application and try to monetize the app by listing them in our marketplace too.</p><h3>Summary:</h3><p>Hope you enjoyed building the app along with me! We will build a few more applications together in this series. So keep building!!!</p><blockquote>Make it work, make it secure, make it fast, make it grow!</blockquote><blockquote>Opinions expressed in this post are solely my own and do not represent the views or opinions of any of my employer.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86a6860b8318" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/nativeapp-and-snowflake-series-how-to-build-an-xgboost-forecasting-app-86a6860b8318">NativeApp and Snowflake Series — How to build an XGboost Forecasting App</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Engineering and Snowflake Series — Part 2 #App Development — NativeApp]]></title>
            <link>https://medium.com/@alandeepak/data-engineering-and-snowflake-series-part-2-app-development-nativeapp-d2dde21b1a9?source=rss-5ed895c8dd32------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2dde21b1a9</guid>
            <category><![CDATA[snowflake-data-cloud]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-warehousing]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[native-app]]></category>
            <dc:creator><![CDATA[Naveen Alan]]></dc:creator>
            <pubDate>Tue, 26 Jul 2022 21:54:38 GMT</pubDate>
            <atom:updated>2022-07-26T21:56:25.451Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Engineering and Snowflake Series — Part 2 #App Development — NativeApp</h3><blockquote>Thinking of storyline for this blog was really interesting, it made me think of my first mobile phone <a href="https://en.wikipedia.org/wiki/Nokia_1100">Nokia 1100</a> back in my college days. I primarily used it for texting without seeing those physical keypads (most of the times when my professors were teaching Engineering :)) and Trust me, with less or no mistakes. But thinking about now, I feel am using my mobile just like breathing. I wake up by its alarm and I go to bed when the mobile gets into sleep mode. Everything within Mobile is App (like blood, it keeps running…) and that is where our story begins.</blockquote><p>Long Long ago, the software industry was dominated by very few large companies without much competition. In July 2008, Apple introduced its App-store with 500 applications and in October 2008 Google introduced Play-Store with few dozen of apps. This is a break-through revolution as this paved the pathway for all Entrepreneurs/Developers to reach the whole world. With just a decade of this invention, as of 2021 Q1 <strong><em>Apple had 2.22 million apps and Google had 3.48 million apps. </em></strong>No wonder, 90% of our time spent on mobile phones are in some Apps :). According to <a href="https://www.statista.com/statistics/269025/worldwide-mobile-app-revenue-forecast/">Statista</a>, mobile applications are predicted to generate 935.2 billion US dollars by 2023 compared to 808.7 billion dollars in 2022.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*GcgpPvQ1iis6rcPD.png" /></figure><blockquote>All these millions of Mobile apps falls under just 3 types at high level.</blockquote><blockquote><strong>Native Mobile Apps</strong>: → are designed to be “native” to one platform, whether it’s Apple iOS, Google’s Android, or Windows Phone. The native platform can be advantageous because it tends to optimize the user experience. Because it was developed specifically for the platform, it can operate more quickly and intuitively.</blockquote><blockquote><strong>Web Apps</strong>: → are responsive versions of websites that can work on any mobile device or OS because they’re delivered using a mobile browser.</blockquote><blockquote><strong>Hybrid Mobile Apps: → </strong>are combinations of both native and web apps, but wrapped within a native app, giving it the ability to have its own icon or be downloaded from an app store.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/0*d3KNV5rwmc4KlW5-.png" /></figure><p>Now with all this context, Snowflake is going with the most successful idea that revolutionised mobile world to the data world with <a href="https://www.snowflake.com/native-apps/">Native Apps</a>. Snowflake is providing a platform for every developers, business, entrepreneurs to deploy data apps. It is a breakthrough idea (Don’t you think), as the data world are trying to move the application as close as possible to your data irrespective of regions and cloud provider the consumer is, the application can be deployed and run across cloud and across region. This provides better data governance for the customer and also reduces lot of I/Os spent on moving data to and fro across platforms.</p><blockquote>Some references on the above statement are “<em>Moving Computation is Cheaper than Moving Data</em>” (from the <a href="http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html#aMoving_Computation_is_Cheaper_than_Moving_Data">Apache Hadoop documentation</a>). A good article on healthcare “<a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7420687/">Bringing Code to Data: Do Not Forget Governance”</a>.</blockquote><p>Hope you all know that we currently support <a href="https://docs.snowflake.com/en/user-guide/data-sharing-intro.html">data sharing</a> where you can share data (basically monetising the data) to snowflake accounts and non-snowflake users. But can we do the same with functions or algorithms? We will find out more in this blog.</p><blockquote>This is an extension of my previous blog on <a href="https://medium.com/@alandeepak/data-engineering-and-snowflake-series-part-1-unstructured-data-ba93caa385da">Unstructured data</a>. Just to reiterate, we created a function to extract text from images in the <a href="https://medium.com/@alandeepak/data-engineering-and-snowflake-series-part-1-unstructured-data-ba93caa385da">part1</a> of the series. I would recommend everyone to read the part1 series as we will be re-using the functions created in my previous blog here.</blockquote><h3>What is Native App?</h3><p>Before getting to know Native App, based on how applications that exists today within Snowflake, we observe the below usage patterns Connected apps or Traditional managed apps</p><h4>Traditional Managed Apps:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*wVeRm5Kk8jyZ8vKwiX0UNg.png" /><figcaption>Traditional App</figcaption></figure><h4>Connected Apps:</h4><p>Last year Snowflake introduced <a href="https://www.snowflake.com/blog/powered-by-snowflake-building-a-connected-application-for-growth-and-scale/">connected apps</a> where the customer data is processed into the SAAS provider platform maintaining single source of truth of data (Unified Data Source).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W_eTRPSWMAC25MoYjG-3Fg.png" /><figcaption>Connected Apps</figcaption></figure><h4>Native Apps:</h4><p>The Native Application Framework takes connected applications to the next level by allowing providers to bring their application code to their customers’ data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*jjonNuEAYkm9UnJRjKONyg.png" /></figure><blockquote>The Native Application Framework makes it easy to build, sell, and deploy applications within the Data Cloud. You can build applications using Snowflake core functionalities such as UDFs and stored procedures, and even the Streamlit integration (currently in development), then sell them to customers across the Data Cloud via <a href="https://www.snowflake.com/marketplace/">Snowflake Marketplace</a>.</blockquote><p>The framework also provides telemetry tools to make it easy to monitor and support your applications. What’s really powerful is that native apps are deployed in the customer’s account in a way that gives the customer control over their data, while still protecting the provider’s intellectual property.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pSgpAfetyIFerI1wblo7-Q.png" /><figcaption>Deeper look into the Native App</figcaption></figure><p>It all started with Data Sharing via Data Marketplace and now Snowflake is going one level up to open up Sharing Functions/Procedures. There are lot of opportunities in this space, as most of the customers have built tools/algorithms for their business needs WHY NOT MONETISE THEM?? Capital one bank invested and built a tool for managing Snowflake platform, now they are trying to monetise it. Read <a href="https://www.capitalone.com/software/solutions/">here</a> for more details. Similarly, we are going to share our Lambda function (developed in our <a href="https://medium.com/@alandeepak/data-engineering-and-snowflake-series-part-1-unstructured-data-ba93caa385da">p</a>art1 of the series) to a consumer (another demo account) and see how the Consumer can be benefited. Another thing to note here is that the consumer will have no clue on how the app/tool behaves, what packages we use or how we are scaling (more like Microservices). Various methods of payments will be possible, but we wont be covering those details here in this blog. Please read <a href="https://www.snowflake.com/native-apps/">here</a> for more details.</p><h4>Common use-cases:</h4><p>The use-cases of native apps are evolving, below are few that we see customers are using this feature for.</p><ol><li>Any business or ML/AI algorithms</li><li>ETL connectors</li><li>Data cleanrooms for secure multi-party data analysis</li><li>Identity and transcoding solutions</li><li>Sentiment analysis using NLTK libraries</li><li>Standardised industry data models</li><li>Data enrichment apps from Factset, Equifax, Experian</li></ol><h3>Text extraction from Images Native App:</h3><p>For this part2 of the series, we will pick the first use-case of building Native application using our algorithm (Text Extract from Images). Below is the step by step process to create a function and share the same with another Snowflake Account. This is still in Private Preview (which means you can not try this yet, but can request for it for testing purposes in customer accounts) but will be in GA soon. For this demo, you need 2 snowflake accounts (one being a producer and another being a consumer).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bzhAGHsI7pSc7ka8i0zc3Q.png" /><figcaption><strong>Architecture of the implementation</strong></figcaption></figure><p>The below diagram shows how the data flows, between producer and consumer accounts when the consumer invokes the API. We will see them in action below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*fF7uCUhbas2hfpYRQRYn0A.png" /><figcaption><strong>Data flow diagram</strong></figcaption></figure><h4>Provider Account Steps:</h4><p>Run the below script in the Producer Account. It performs the below steps.</p><ol><li>Create a setup store proc that creates external function with API integration</li><li>Create the store proc to call the external function</li><li>Create an installer script</li><li>Create a role and provide necessary grants for the proc</li><li>Create a share object and provide the necessary grants</li></ol><pre>-- #### Creating Database and Schema for Storing the functions ####<br>create or replace database ocr_processing;<br>use database ocr_processing;<br>create or replace schema ocr_image_extract;<br>use schema ocr_image_extract;</pre><pre>-- ############# SETUP #######################<br>CREATE or replace PROCEDURE ocr_processing.ocr_image_extract.setup(integration_name string)<br>RETURNS STRING<br>LANGUAGE JAVASCRIPT<br>EXECUTE AS OWNER<br>AS $$<br>var endpoint = &quot;<a href="https://5399etw01k.execute-api.us-east-1.amazonaws.com/dev/ocr">https://XXXXX.execute-api.us-east-1.amazonaws.com/dev/ocr</a>&quot;;</pre><pre>var create_ext_func_cmd = &quot;create or replace external function ocr_text_extract.ext_ocr(imagelocation string)&quot;<br>        + &quot; returns variant&quot;<br>        + &quot; api_integration = &quot; + INTEGRATION_NAME<br>        + &quot; as &#39;&quot; + endpoint + &quot;&#39;&quot;;<br>    snowflake.execute({ sqlText: create_ext_func_cmd });</pre><pre>return &quot;SETUP COMPLETED&quot;;<br>$$;</pre><pre>-- #### PROCEDURE that calls external function ####<br>create or replace procedure ocr_processing.ocr_image_extract.extract_text_and_load(rel_path string)<br>  returns variant<br>  language javascript<br>  as     <br>  $$<br>    var call_ext_func_cmd = &quot;select ocr_text_extract.ext_ocr(select concat(&#39;{\&quot;image\&quot;: \&quot;&#39;,&#39;&quot;+REL_PATH+&quot;&#39;,&#39;\&quot;}&#39;))&quot;;<br>    var res = snowflake.execute({ sqlText: call_ext_func_cmd });<br>    res.next();</pre><pre>return res.getColumnValue(1);<br>    <br>  $$<br>  ;</pre><pre>-- ############# GRANTS #######################<br>CREATE or replace DATABASE ROLE shared_db_role;<br>GRANT USAGE ON DATABASE ocr_processing to DATABASE ROLE shared_db_role;<br>GRANT USAGE ON SCHEMA ocr_processing.ocr_image_extract TO DATABASE ROLE shared_db_role;<br>GRANT USAGE ON PROCEDURE ocr_processing.ocr_image_extract.setup(string) TO DATABASE ROLE shared_db_role;<br>GRANT USAGE ON PROCEDURE ocr_processing.ocr_image_extract.extract_text_and_load(string) TO DATABASE ROLE shared_db_role;</pre><pre>-- ############# SETUP #######################<br>CREATE or replace PROCEDURE ocr_processing.ocr_image_extract.installer()<br>RETURNS STRING<br>LANGUAGE SQL<br>EXECUTE AS OWNER<br>AS $$<br>  begin<br>    CREATE SCHEMA ocr_text_extract;<br>    GRANT DATABASE ROLE shared_db_role TO DATABASE ROLE APP_EXPORTER;<br>    return &#39;installer script Done&#39;;<br>  end;<br>$$;</pre><pre>-- ############# SHARE CREATION #######################<br>CREATE or replace SHARE ocr_share installer = ocr_processing.ocr_image_extract.installer();<br>GRANT USAGE ON DATABASE ocr_processing TO SHARE ocr_share;<br>GRANT USAGE ON SCHEMA ocr_processing.ocr_image_extract TO SHARE ocr_share;<br>GRANT DATABASE ROLE shared_db_role TO SHARE ocr_share;<br>GRANT USAGE ON PROCEDURE ocr_processing.ocr_image_extract.installer() TO SHARE ocr_share;</pre><pre>ALTER SHARE ocr_share ADD ACCOUNTS=XXXX; -- ** Consumer Account Name</pre><p>So with this you have created an Image extraction function and created a share object within Snowflake.</p><h4>Consumer Account steps:</h4><p>Next step is to use this share object in Consumer account and use it for processing. Below are steps performed.</p><ol><li>Create storage integration object for accessing S3 where images are stored at consumer end. (similar to our Storage steps above)</li><li>Create an external stage (similar to our Storage steps above)</li><li>Create an API Integration Object</li><li>Create a role and provide necessary grants</li><li>List share and create an database pointing to the share created in the provider account. This will install all the functions that was created in provider account in the consumer account.</li><li>Please note that, you can only see the parameters passed to the functions but not the body of it (this is called secure functions in Snowflake world ensuring the Consumers are not able to see the code/algorigthms of producer)</li><li>List your stage object and test the function by passing pre-signed URL from consumer stage. You will notice the lambda is invoked from Producer AWS account and the response is returned to the Consumer account. (Obviously, you can create a Stream, PROC and Task similar to PROCESSING step above to automate it in prod)</li></ol><pre>-- #### STORAGE INTEGRATION CREATION #### --<br>use database native_app_streamlit;</pre><pre>CREATE OR REPLACE STORAGE INTEGRATION storage_integ     -- ********* This is establishing the connection between AWS S3 with Snowflake user<br>  TYPE = EXTERNAL_STAGE<br>  STORAGE_PROVIDER = S3<br>  ENABLED = TRUE<br>  STORAGE_AWS_ROLE_ARN = &#39;arn:aws:iam::XXXXX:role/s3fullaccess_role&#39;<br>  STORAGE_ALLOWED_LOCATIONS = (&#39;s3://snowpark-demo-python-naveen/&#39;, &#39;s3://snowpark-demo-python-naveen/nativeapp&#39;);</pre><pre>DESC INTEGRATION storage_integ;</pre><pre>-- #### STAGE CREATION #### --<br>CREATE OR REPLACE STAGE s3_stage_native_app <br>        directory = (enable = true)  -- ********* Enabling directory creates directory table for unstructured data<br>        storage_integration = storage_integ<br>        url = &#39;s3://snowpark-demo-unstruc-naveen/nativeapp&#39;;<br>  <br>list <a href="http://twitter.com/s3_stage_native_app">@s3_stage_native_app</a>;</pre><pre>-- #### API INTEGRATION CREATION ############################## --<br>CREATE OR REPLACE api integration native_app_ocr_int<br>  api_provider = aws_api_gateway<br>  api_aws_role_arn = &#39;arn:aws:iam::XXXXXX:role/apifullaccess&#39;<br>  api_allowed_prefixes = (&#39;<a href="https://5399etw01k.execute-api.us-east-1.amazonaws.com/dev&#39;">https://XXXXX.execute-api.us-east-1.amazonaws.com/dev&#39;</a>)<br>  enabled = true;</pre><pre>DESCRIBE integration native_app_ocr_int;</pre><pre>-- #### NATIVE APP SCRIPT #### --</pre><pre>CREATE or replace ROLE ocr_app_role;<br>GRANT USAGE ON INTEGRATION native_app_ocr_int to ROLE ocr_app_role;<br>GRANT USAGE ON DATABASE native_app_streamlit to ROLE ocr_app_role;<br>GRANT USAGE ON SCHEMA native_app_streamlit.PUBLIC to ROLE ocr_app_role;<br>GRANT SELECT ON STREAM native_app_streamlit.PUBLIC.s3_stream to ROLE ocr_app_role;</pre><pre>-- #### Create Database from Shares (installing the functions locally)<br>SHOW SHARES;<br>CREATE OR REPLACE DATABASE CONSUMER_ocr_processing FROM share &quot;XXXX&quot;.&quot;OCR_SHARE&quot;;<br>GRANT ROLE ocr_app_role TO DATABASE CONSUMER_ocr_processing;<br>CALL CONSUMER_ocr_processing.ocr_image_extract.setup(&#39;native_app_ocr_int&#39;); -- Installing (Creating the External Function with API Integration)</pre><pre>-- ## Listing S3 Stage and call the image extraction function</pre><pre>list <a href="http://twitter.com/s3_stage_native_app">@s3_stage_native_app</a>;</pre><pre>SELECT GET_PRESIGNED_URL(<a href="http://twitter.com/s3_stage_native_app">@s3_stage_native_app</a>, &#39;snowflake.jpg&#39;, 3600);   -- Calling the External function at the Producer end to retrieve the data content</pre><pre>call consumer_ocr_processing.ocr_image_extract.extract_text_and_load(&#39;<a href="https://snowpark-demo-python-naveen.s3.eu-west-2.amazonaws.com/nativeapp/snowflake.jpg?X-Amz-Security-Token=FwoGZXIvYXdzEH0aDKcRz6Qixz70Ni%2F6NCKtAaHiOAQ%2B56shhzSaozKyLaQL93dUVuCzTJ2Lp7cQlObE7JIULX%2BWZ7KrVzXG0NUXpjGH3KElkxjv7LI6g249y%2B9QCcf4%2B%2B5eJBnReiW6kEBNBQekl8wGBWsGmhMl2cB3hKZ0VPu2avijL0md%2BIPNaygWZ2RP0LZMJlszASoFU%2BLV%2FqXjyEofpDjdkO63GHQkgN%2BUDolPCY%2F%2BDOHyuGB32hkop6ZkEzeK4cFH%2BisIKNLc85MGMi2Ws2z61SapGSaJXJDb1CxzpZNrG1biEccQa4UaeyDe7zByaZGnqfLQgM2vOGk%3D&amp;X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Date=20220512T112404Z&amp;X-Amz-SignedHeaders=host&amp;X-Amz-Expires=3599&amp;X-Amz-Credential=ASIAXBUYSSFILJMS7P5V%2F20220512%2Feu-west-2%2Fs3%2Faws4_request&amp;X-Amz-Signature=a77b293e60473736af46be913dccde55767c54975a3a5e43c434fcd976c2ecb5&#39;">pre_signed_url from previous statement&#39;</a>);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fF7uCUhbas2hfpYRQRYn0A.png" /><figcaption><strong>DataFlow when the call is made</strong></figcaption></figure><h4>Summary:</h4><p>Isn’t it great! you can create functions and share them with other snowflake accounts. You can list them via Snowflake marketplace and can generate new revenue stream for your Business. This is where the world is moving towards, data sharing and app sharing. This supports better data governance principle and simplified architecture. With Streamlit acquisition, Snowflake clearly is planning to have a deployment environment where you can host your data application in Snowflake and make it available in Snowflake app store. Feel free to read more on this topic <a href="https://www.snowflake.com/blog/introducing-snowflake-native-application-framework/">here</a>.</p><blockquote>Happy to be part of it! Stay tuned for more!</blockquote><blockquote>Opinions expressed in this post are solely my own and do not represent the views or opinions of any of my employer.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2dde21b1a9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Engineering and Snowflake Series — Part 1 #Unstructured Data]]></title>
            <link>https://medium.com/@alandeepak/data-engineering-and-snowflake-series-part-1-unstructured-data-ba93caa385da?source=rss-5ed895c8dd32------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba93caa385da</guid>
            <category><![CDATA[unstructured-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[snowflake-data-cloud]]></category>
            <dc:creator><![CDATA[Naveen Alan]]></dc:creator>
            <pubDate>Thu, 30 Jun 2022 10:18:57 GMT</pubDate>
            <atom:updated>2022-06-30T10:18:57.376Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Engineering and Snowflake Series — Part 1 #Unstructured Data</h3><blockquote><em>Being human, we all love stories. So do I. Since my childhood, the first sentence of any story/movies that excites me is “Once upon a time!” as that brings audience to an entrance of an imaginary world that the narrator has created it for them. So I would love to start my first blog on Medium with the same.</em></blockquote><p><strong>Once upon a time</strong>, we were using tally sticks to store and analyse data (especially to predict how long food supplies will last), then came punching machines, documents, Hollerith Tabulating Machine (1880’s US census), Relation Databases (1970’s SEQUEL), Excel (1985), Data warehouse (1980&#39;s), Data Lake/Lake House (2010)…</p><p>Even though we collected data ages ago, selling the data or using the data for generating revenue marks a crucial turning point for data, as this marks the day when we started monetising data. Dated around 1841, to help American Merchants on decision making a company started to sell data about other companies using a network of correspondents (The company is now known as <a href="https://www.dnb.co.uk/about-us/company/history.html">Dun&amp;Bradstreet</a>). In 1865, one of the banker started using data analysis to get competitive advantage (coined the term “<em>Business Intelligence</em>”). Around 1870’s direct marketing using direct mail and posters started. The direct marketing was performed by agents collecting sales/invoice data. Even computers were marketed using posters and direct mail in 1950’s. With computers, we were able to store huge data and with the birth of internet we were able to transport data across the world, but to my surprise, even now big enterprises are stuck with technology challenges of storing and analysing data at scale resulting in not able to monetise data as much as they would like to.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DetDm8uwI1EqrUR-Myzeow.jpeg" /><figcaption>Growth of data and its monetisation</figcaption></figure><blockquote>This series assumes you understand what is Snowflake and how to get started with it. If not, please register a 30 day free trial demo account and get started. Please feel free to read through Snowflake <a href="https://docs.snowflake.com/en/">doc</a> for more details.<br>The demo in this series needs <strong>Snowflake</strong> and an <strong>AWS</strong> account to proceed.</blockquote><h3>Intro:</h3><p><a href="https://docs.snowflake.com/en/user-guide/intro-key-concepts.html">Snowflake</a> is a revolutionary product, as it was built on cloud storage in 2014 and naturally supported Datalake/Lakehouse need of the hour. With the growth of this platform, Snowflake started to support various workloads Datalake, Data Engineering, Data Science, Cybersecurity, etc. Basically, everything an Enterprise needs for OLAP. In this post, we will be focussing on <strong>Data Engineering(DE)</strong> to be specific.</p><p>I personally love DE. Am basically from Production Engineering background, DE is similar to a production line in manufacturing sectors. With various sources of materials/parts, we need to plan and architect a clear step by step process and design the production line to get the desired product outcome obviously applying lean principles. Data Engineering in software is no different, we design pipelines for various product owners and create data as a product in general.</p><blockquote>Parts that i’m planning to cover in this series are as follow (not necessarily in the same order and the list will grow too). So watch this space for more learning!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w6qbbWm-Ev40kVpmGtjfgA.png" /><figcaption>Series planned for this year</figcaption></figure><h3>Part1 — Unstructured data loading and processing</h3><h4>Scope:</h4><ol><li>What is unstructured data</li><li>Common usecase with unstructured data</li><li>How Unstructured data is stored in Snowflake, how it can be accessed</li><li>How to develop pipeline for processing Unstructured data in Snowflake and bundle them with structured data in the final target table.</li><li>Touch upon further possibilities.</li></ol><h4>What is Unstructured Data:</h4><ul><li>Unstructured data is information that is not arranged according to a pre-set data model or schema, and therefore cannot be stored in a traditional relational database or RDBMS.</li><li>Text and multimedia are two common types of unstructured content. Many business documents are unstructured, as are email messages, videos, photos, webpages, and audio files.</li><li>Unstructured data stores contain a wealth of information that can be used to guide business decisions. However, unstructured data has historically been very difficult to analyse. With the help of AI and machine learning, new software tools are emerging that can search through vast quantities of it to uncover beneficial and actionable business intelligence.</li></ul><h4>Common use cases of Unstructured data:</h4><ul><li>Store PDF documents to extract key/value pairs for the purpose of analytics.</li><li>Store call centre recordings (audio files) to derive insights such as sentiment analysis.</li><li>Run machine learning on medical images (DICOM) to derive insights from them for research purposes.</li><li>Store screenshots of documents such as insurance cards or prescription pills and run an optical character recognition process on them to extract text for analytics.</li><li>Car Insurance claim fraud detection using image analysis</li></ul><h4>Usecase for Part 1 Demo: Text Extraction from jpg/jpeg and process them together with Structured data</h4><p>For this part1 series, we will pick the usecase of extracting text from images as I have seen this use case in my previous experience within a data company where we had to maintain/purchase a licensed image processing datastore. So literally we were not using the rich data content within those images but were just storing them and making them available for download. Obviously Snowflake did not exist at that time but now, thanks for snowflake’s unstructured data support and Snowpark capability any data company can just load and process unstructured data along-side of structured data and have them stored together in a single table. i.e., The invoice image (unstructured) can be stored along side of the invoice details (structured). Also for Banks where few business transactions needs to be approved via email both approval email and the transaction details can be in single table. (Isn’t it Great!).</p><p>Let’s see how to do this with Snowflake in this part 1 series. The following are the sections within this demo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v7eDOTiXQK_blNcEFkZ3rw.png" /><figcaption>Steps of the demo</figcaption></figure><h4>Storage:</h4><p>Snowflake uses blob storage for all its storage irrespective of structured, semi or unstructured data. Because of this nature handling unstructured data is fairly straight forward for Snowflake. But the only difference is that you have to create stages to access Unstructured data. It can be either Internal Stage within Snowflake (which internally maintains how the data is stored in S3) or External Stage (which you have full control over how the data is stored and accessed). In this demo, we will pick External Storage with S3.</p><h4>Step 1: Create Role and S3 buckets for Stage</h4><p>Create an AWS Role and attach the S3 access policy and Trust relationship to the role following the doc <a href="https://docs.snowflake.com/en/user-guide/data-load-s3-config-storage-integration.html">link</a>. The link also covers Step 2 and Step 3 of the blog.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*H8_bNoclBHBN3raV.png" /><figcaption>Recommended approach of connecting to S3</figcaption></figure><blockquote>S3 Bucket: snowpark-demo-unstruc-naveen (created for this demo)</blockquote><blockquote>JSON Policy (sample): You can restrict to prefix level if needed too.</blockquote><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;s3:PutObject&quot;,<br>                &quot;s3:GetObject&quot;,<br>                &quot;s3:GetObjectVersion&quot;,<br>                &quot;s3:DeleteObject&quot;,<br>                &quot;s3:DeleteObjectVersion&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;arn:aws:s3:::snowpark-demo-unstruc-naveen/*&quot;<br>        },<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;s3:ListBucket&quot;,<br>                &quot;s3:GetBucketLocation&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;arn:aws:s3:::snowpark-demo-unstruc-naveen&quot;,<br>            &quot;Condition&quot;: {<br>                &quot;StringLike&quot;: {<br>                    &quot;s3:prefix&quot;: [<br>                        &quot;*&quot;<br>                    ]<br>                }<br>            }<br>        }<br>    ]<br>}</pre><blockquote>IAM Role: snowpark_demo_role</blockquote><blockquote>Trust Relationship policy: (Initially just provide dummy AWS user and ExternalID and update the same after creating Storage Integration in Snowflake)</blockquote><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;<strong><em>AWS and EXTERNALID populated from DESC command of Snowflake</em></strong>&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;AWS&quot;: &quot;<strong><em>arn:aws:iam::XXXXXXXX:user/***-*****</em></strong>&quot;<br>            },<br>            &quot;Action&quot;: &quot;sts:AssumeRole&quot;,<br>            &quot;Condition&quot;: {<br>                &quot;StringEquals&quot;: {<br>                    &quot;sts:ExternalId&quot;: &quot;<strong><em>***********************</em></strong>&quot;<br>                }<br>            }<br>        }<br>    ]<br>}</pre><h4>Step 2: Create Storage Integration and connecting S3 and Snowflake</h4><p>Open the snowflake UI/Snowsight UI and create a worksheet for this demo.</p><blockquote>Create a Storage Integration to connect to S3</blockquote><pre>— #### DATABASE CREATION #### — <br>CREATE DATABASE unstrucdemo;<br>USE DATABASE unstrucdemo; —- by default the objects within are created in PUBLIC schema</pre><pre>— #### STAGE CREATION #### —<br>— ********* This is establishing the connection between AWS S3 with Snowflake user/account</pre><pre>CREATE OR REPLACE STORAGE INTEGRATION snowpark_demo_si <br> TYPE = EXTERNAL_STAGE<br> STORAGE_PROVIDER = S3<br> ENABLED = TRUE<br> STORAGE_AWS_ROLE_ARN = ‘arn:aws:iam::XXXXXXX:role/snowpark_demo_role’ -- Role created in Step 1<br> STORAGE_ALLOWED_LOCATIONS = (‘s3://snowpark-demo-unstruc-naveen/’); -- This can be a list of S3 locations that you wanted to access.</pre><pre>— #### DESC Storage Creation #### —</pre><pre>DESC INTEGRATION snowpark_demo_si; -- This will produce a result as the image below, pick the user and externalID from here to your Trust Policy</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pd4UEIq6EkF88CxzhWehVQ.png" /><figcaption>Results from Desc command</figcaption></figure><h4>Step 3: Creating External Stage with Directory tables</h4><pre>CREATE OR REPLACE STAGE my_s3_stage <br>        directory = (enable = true)  -- *** Enabling directory creates directory table for unstructured data<br>        storage_integration = snowpark_demo_si -- ** SI from Step 2<br>        url = &#39;s3://snowpark-demo-unstruc-naveen/&#39;;</pre><pre>LIST <a href="http://twitter.com/my_s3_stage">@my_s3_stage</a>; -- ** this should produce no results if you don&#39;t have anything. But if it fails, then recheck the steps above to make sure everything is intact.</pre><blockquote>Directory Table content:</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*TD7_XjP4nlW0hj-Dr4BeWw.png" /><figcaption>Details stored as an table for querying and processing unstructured data</figcaption></figure><blockquote>SQL Functions that are inbuilt for file access in Stage: Please read the <a href="https://docs.snowflake.com/en/user-guide/unstructured-intro.html#sql-functions">link</a> for more details.</blockquote><pre>-- #### URLS FOR FILE ACCESS #### --</pre><pre>SELECT GET_PRESIGNED_URL(<a href="http://twitter.com/my_s3_stage">@my_s3_stage</a>, &#39;images/test.jpeg&#39;, 3600); -- Presigned URL for external function access</pre><pre>SELECT BUILD_SCOPED_FILE_URL(<a href="http://twitter.com/my_s3_stage">@my_s3_stage</a>, &#39;images/test.jpeg&#39;); -- This is scoped URL which has limited/restricted access</pre><pre>SELECT GET_RELATIVE_PATH(<a href="http://twitter.com/my_s3_stage">@my_s3_stage</a>, &#39;s3://snowpark-demo-unstruc-naveen/images/test.jpeg&#39;);</pre><pre>SELECT BUILD_STAGE_FILE_URL(<a href="http://twitter.com/my_s3_stage">@my_s3_stage</a>, &#39;images/test.jpeg&#39;); -- Stage URL</pre><h4>Processing:</h4><p>With Snowflake’s latest Snowpark feature you can process the images within the snowflake using <strong>Java</strong> and <strong>Scala</strong> functions or libraries. With Python in the roadmap towards end of this year, this can be very useful too. But the Snowpark has few limitations too, like you can not use “tesseract” package within it, as it involves installing C/C+ binaries so the workaround is to use External functions to perform this operation. In this demo, we will focus on External function using Lambda (just to make it interesting!). Please refer to <a href="https://quickstarts.snowflake.com/guide/extract_attributes_dicom_files_java_udf/index.html?index=..%2F..index#0">Snowpark quick starts lab</a> which demo’s DICOM image processing UDF within Snowflake using Snowpark Java.</p><h4>Step 4: Setup streams on Stage</h4><p>The streams for external stage is a great feature, as we can understand what all has updated/inserted into this stage since the previous run of the task that consumes this stream data. This is the CDC support from Snowflake and it is widely used by all our customers. It is a simple SQL statement that creates the stream object.</p><pre>create or replace stream s3_stream on stage my_s3_stage;</pre><h4>Step 5: Load images to S3 and see data in Stream</h4><p>In this step, we load the images that needed to be processed into S3. This can be an IOT device or Mobile device or any application that is streaming/loading in batch to S3 buckets. In our demo, am going to load them manually into S3, in real world this could be any services.</p><pre>alter stage my_s3_stage refresh; -- ** This will refresh stage (This is needed for External Stage to get the changes reflected in directory tables currently, but Snowflake is changing this to happen automatically soon)</pre><pre>select * from s3_stream; -- ** The image below is a sample. Using Metadata$action you can perform actions on stream data.</pre><pre>select relative_path from s3_stream where relative_path like &#39;%.jpg&#39; or relative_path like &#39;%.jpeg&#39;; -- *** Selecting JPG files for loading</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qh1liIp-yu4KM0S5E0OvwQ.png" /></figure><h4>Step 6: Create External function to process Images</h4><p>Creating a Lambda and API gateway is out of scope of this blog. But below are few details.</p><ol><li>Lambda function was created in Python.</li><li>Created lambda layer with Tesseract, Pillow and dependant packages</li><li>A simple Python code as below in Lambda handler. Please note that when we call External function from Snowflake the Input from Snowflake is an array of records with Key value pairs and the response has to be an array of objects with KV pairs, as usually it will be bundle of records that are processed together in chunks.</li><li>API Gateway to invoke this Lambda function.</li></ol><pre>def ocr_extract(event, context):<br>    retVal= {}<br>    retVal[&quot;data&quot;] = []<br>    event_body = event[&quot;body&quot;]<br>    payload = json.loads(event_body)<br>    <br>    for row in payload[&quot;data&quot;]:<br>        sfref= row[0]<br>        sfdata = row[1]<br>        request_body = json.loads(sfdata)<br>        url = request_body[&#39;image&#39;]<br>        response = requests.get(url)<br>        img = Image.open(BytesIO(response.content))<br>        buffered = BytesIO()<br>        img.save(buffered, format=&quot;JPEG&quot;)<br>        img_str = base64.b64encode(buffered.getvalue()).decode() <br>        image = io.BytesIO(base64.b64decode(img_str))<br>        text = pytesseract.image_to_string(Image.open(image))<br>        body = {<br>            &quot;text&quot;: text<br>        }<br>        retVal[&quot;data&quot;].append([sfref,text])<br>    response = {<br>        &quot;statusCode&quot;: 200,<br>        &quot;body&quot;: json.dumps(retVal)<br>    }<br>    return response</pre><h4>Step 7: Snowflake calling the Lambda using API integration</h4><p>Similar to S3 integration object, we need to setup API integration object to call external API’s. The below image describes how <a href="https://docs.snowflake.com/en/sql-reference/external-functions-introduction.html">External function</a> works within Snowflake.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wW6a_5gZgGycbD8p.png" /><figcaption>How External Function call works</figcaption></figure><p>Create and IAM role with the below Trust Relationship policy, similar to the Step 2 update the policy after creating the API Integration object.</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>             &quot;Sid&quot;: &quot;<strong><em>AWS role and EXTERNALID populated from DESC command of Snowflake</em></strong>&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;AWS&quot;: &quot;<strong>arn:aws:iam::XXXXXX:user/***********</strong>&quot;<br>            },<br>            &quot;Action&quot;: &quot;sts:AssumeRole&quot;,<br>            &quot;Condition&quot;: {<br>                &quot;StringEquals&quot;: {<br>                    &quot;sts:ExternalId&quot;: &quot;<strong>*************</strong>&quot;<br>                }<br>            }<br>        }<br>    ]<br>}</pre><p>In Snowflake worksheet, input the below commands. So below are the steps that are happening within them.</p><ol><li>Snowflake is creating a pre-signed URL (in the demo i have set it up to 1 hour for the URL TTL, in real world it can be in minutes)</li><li>Calling the lambda function with the pre-signed URL</li><li>Lambda uses the URL and downloads the image</li><li>Lambda uses Tesseract module to extract the text content and return the same in response</li></ol><pre>-- #### CREATE OUTPUT TABLE #### --<br>create or replace table image_with_text(<br>    image_id int identity(1,1),<br>    relative_path nvarchar(200) not null,<br>    pre_signed_url text not null,<br>    text_data variant not null,<br>    row_mod_timestamp timestamp default to_timestamp_ntz(current_timestamp)</pre><pre>);<br>-- #### CREATE API Integration Object #### --</pre><pre>CREATE OR REPLACE api integration unstrucprocessing<br>  api_provider = aws_api_gateway<br>  api_aws_role_arn = &#39;arn:aws:iam::XXXXXXX:role/UnstrucProcessingRole&#39;<br>  api_allowed_prefixes = (&#39;<a href="https://5399etw01k.execute-api.us-east-1.amazonaws.com/dev/ocr&#39;">https://XXXXXX.execute-api.us-east-1.amazonaws.com/dev/ocr&#39;</a>) -- ** this can be a list of API&#39;s<br>  enabled = true;</pre><pre>DESCRIBE integration unstrucprocessing;</pre><pre>CREATE OR REPLACE external function ExtractTextFromImage(imagelocation string)<br>    returns variant<br>    api_integration = unstrucprocessing                                     <br>    as &#39;<a href="https://5399etw01k.execute-api.us-east-1.amazonaws.com/dev/ocr&#39;">https://5399etw01k.execute-api.us-east-1.amazonaws.com/dev/ocr&#39;</a>;    -- ********* API to invoke</pre><pre>-- ** Testing the function<br>SELECT ExtractTextFromImage(select concat(&#39;{&quot;image&quot;: &quot;&#39;,GET_PRESIGNED_URL(<a href="http://twitter.com/my_s3_stage">@my_s3_stage</a>, &#39;images/snowflake.jpg&#39;, 3600),&#39;&quot;}&#39;)) as image_content;</pre><h4>Step 8: Create a Proc and Task to call External Function and load the response to target tables</h4><p>Below are the steps that will be carried out in this step.</p><ol><li>A Store Proc is created to process all the images in stage</li><li>The PROC will iterate on Stage files and invoke the External function to process the images using Pre-signed URL</li><li>The returned response is inserted into VARIANT column in the target output table. In real world scenario, it can be a json object returned with list if elements that needs to be extracted from images etc.</li><li>A task (which invokes the store proc) listening to the stream (created in Step 4) on stage runs every 5 minutes to process the bunch of images that are available in stream for that interval. The Interval and scheduling can be fine tuned based on the needs of the Customer/use cases. The Store PROC can be orchestrated via airflow or any tools like Fivetran, Talend etc.,</li></ol><pre>-- ** Store PROC to iterate over the Stage to process image<br>create or replace procedure extract_text_and_load()<br>  returns string<br>  language javascript<br>  as     <br>  $$  <br>    <br>    try {<br>        var sql_command = <br>            &quot;select relative_path, size from s3_stream where relative_path like &#39;%.jpg&#39; or relative_path like &#39;%.jpeg&#39;&quot;;<br>        var resultset1 = snowflake.execute (<br>            {sqlText: sql_command}<br>            );<br>        while (resultset1.next())<br>        {<br>            var rel_path = resultset1.getColumnValue(1);<br>            var sql_insert = &quot;insert into image_with_text (select 1, &#39;&quot; + rel_path + &quot;&#39;, GET_PRESIGNED_URL(<a href="http://twitter.com/my_s3_stage">@my_s3_stage</a>, &#39;&quot; + rel_path + &quot;&#39;, 3600) as pre_signed_url, ExtractTextFromImage(select concat(&#39;{\&quot;image\&quot;: \&quot;&#39;,GET_PRESIGNED_URL(<a href="http://twitter.com/my_s3_stage">@my_s3_stage</a>, &#39;&quot;+rel_path+&quot;&#39;, 3600),&#39;\&quot;}&#39;)) as text_data, to_timestamp_ntz(current_timestamp));&quot;;                 <br>            <br>            <br>            try{<br>                snowflake.execute (<br>                    {sqlText: sql_insert}<br>                    );<br>                <br>            }<br>            catch (err)  {<br>                return &quot;Failed1: &quot; + err;   // Return a success/error indicator.<br>            }<br>        }<br>        return &quot;Succeeded.&quot;;   // Return a success/error indicator.<br>        }<br>    catch (err)  {<br>        return &quot;Failed: &quot; + err;   // Return a success/error indicator.<br>        }<br>  $$<br>  ;</pre><pre>-- ** Task to call the store PROC ONLY WHEN STREAM HAS DATA</pre><pre>CREATE OR REPLACE TASK load_unstruct_data<br>  WAREHOUSE = LOAD_WH<br>  SCHEDULE = &#39;5 minute&#39;<br>WHEN<br>  SYSTEM$STREAM_HAS_DATA(&#39;s3_stream&#39;)<br>AS<br>  call extract_text_and_load();</pre><pre>-- ** Final target table<br>Select pre_signed_url as URL, text_data as TEXT_CONTENT from image_with_text;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aoaUiQ023BFglXR0rsaAkg.png" /><figcaption>Sample column and data content</figcaption></figure><h3>Further Possibilities:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gB5XN3Snr0zwy3JwrAjlTw.png" /></figure><p>Possibilities are limitless. With all these features and especially with <a href="https://www.snowflake.com/snowpark/">Snowpark</a>, <a href="https://www.snowflake.com/native-apps/">NativeApps</a> and <a href="https://docs.snowflake.com/en/sql-reference/external-functions-introduction.html">external function</a> options. You can certainly perform any data engineering or data science workloads on Snowflake and in this context on Unstructured data. Snowflake recently acquired <a href="https://streamlit.io/">Streamlit</a> and a simple demo of an Streamlit app will follow in other parts. Snowflake is working on having an UX framework, so basically you can host an Webapp in Streamlit or UX Framework in Snowflake with all the backend processing happening in Snowflake from Data Ingestion, Data Transformation, Data Extraction and Viz with frontend on Streamlit. Snowflake is trying to give users an experience of a Mobile App store within Snowflake UI. We know what Mobile Apps have done to this world, cant wait to see what Snowflake Apps are going to do!</p><p>Happy to be part of it! Stay tuned for more!</p><blockquote><em>Opinions expressed in this post are solely my own and do not represent the views or opinions of any of my employer.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba93caa385da" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>