<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lakhan Dhanashree on Medium]]></title>
        <description><![CDATA[Stories by Lakhan Dhanashree on Medium]]></description>
        <link>https://medium.com/@lakhansd777?source=rss-4c54a1289c4f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*wg42fn3PC573F8VROsKFgw.jpeg</url>
            <title>Stories by Lakhan Dhanashree on Medium</title>
            <link>https://medium.com/@lakhansd777?source=rss-4c54a1289c4f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 15:49:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lakhansd777/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Openflow: Snowflake’s Quiet Attack on Traditional ETL Tools]]></title>
            <link>https://medium.com/@lakhansd777/openflow-snowflakes-quiet-attack-on-traditional-etl-tools-16ae3d843665?source=rss-4c54a1289c4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/16ae3d843665</guid>
            <category><![CDATA[snowflake-openflow]]></category>
            <dc:creator><![CDATA[Lakhan Dhanashree]]></dc:creator>
            <pubDate>Tue, 10 Nov 2026 19:05:01 GMT</pubDate>
            <atom:updated>2026-11-10T19:05:01.074Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>It didn’t come with a keynote explosion. But it might be the most disruptive thing Snowflake has shipped in years.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xw9Fd9ZGveB_wbcJj2tUMg.png" /></figure><p>Some product launches come with fireworks. Others come with a shrug and then, eighteen months later, you look up and realize half your data stack is quietly obsolete.</p><p><strong>Openflow</strong> is the second kind.</p><p>While everyone was busy arguing about which LLM is smarter this week, Snowflake did something far less flashy and, frankly, far more consequential for anyone running a data platform: it built ingestion, transformation, and orchestration <em>directly into the warehouse itself</em>. No more standing up a separate ETL tool, licensing another vendor, or babysitting a fleet of connectors. Data movement just… became a native Snowflake capability.</p><p>If you’re a Fivetran, Informatica, or Airbyte customer and haven’t heard about this yet, you should probably keep reading.</p><p><strong>Wait, Didn’t Snowflake Already Have Snowpipe?</strong></p><p>Fair question. Snowpipe was always great at one thing: getting files into Snowflake fast. But it was never a full ETL/ELT platform no visual pipeline builder, no rich transformation logic, no native handling of messy unstructured data like PDFs, audio, or sensor streams.</p><p>Openflow is a fully managed service built on Apache NiFi, pairing a Snowflake-managed control plane with a “Bring Your Own Cloud” (BYOC) data plane that runs inside your own VPC. The product traces back to Snowflake’s acquisition of Datavolo, later extended for the demands of modern AI systems and if you know NiFi, you know it wasn’t built by some random startup. It’s a battle-tested, open-source dataflow automation engine originally developed by the NSA before being donated to the Apache Software Foundation.</p><p>Translation: Snowflake didn’t build a toy. It absorbed a mature, production-grade orchestration engine and wrapped it in enterprise governance.</p><p><strong>The Buzzword Stack (That Actually Means Something This Time)</strong></p><p>Let’s get the jargon out of the way, because this launch is drowning in it but unlike most product marketing, most of it is doing real work here:</p><ul><li><strong>ETL for AI</strong> — pipelines built not just to load a table, but to feed vector stores, embeddings, and LLM context windows</li><li><strong>Multimodal ingestion</strong> — structured and unstructured data, batch and streaming, all moving through one platform, including text, images, audio, video, and sensor data</li><li><strong>CDC replication</strong> — live change-data-capture from systems like Oracle, SQL Server, MySQL, and PostgreSQL straight into Snowflake, with full lineage tracking</li><li><strong>BYOC (Bring Your Own Cloud)</strong> — the control plane is managed, but the data plane runs in <em>your</em> VPC, so sensitive data never has to leave your perimeter</li><li><strong>Zero-copy governance</strong> — access control, encryption, and lineage that travel with the pipeline, not bolted on afterward</li><li><strong>AI-native preprocessing</strong> — unstructured data extracted, preprocessed using Cortex LLM functions, loaded, and activated directly within the ETL pipeline</li></ul><p>That last point is the real story. This isn’t just “ETL, but inside Snowflake.” It’s ETL that’s <em>AI-aware by design</em> — pipelines that don’t just move data but prep it for retrieval, embeddings, and agent consumption on the way in.</p><p><strong>Why This Is a Genuine Threat to the Traditional ETL Market</strong></p><p>Here’s the uncomfortable math for the incumbent ETL vendors: before Openflow, getting data into Snowflake typically meant reaching for a third-party tool Fivetran, Informatica, custom scripts or settling for Snowflake’s more limited native ingestion options. That meant extra licensing, extra integration surface area, extra credentials to manage, extra latency between “data exists” and “data is usable.”</p><p>Openflow collapses that entire layer. Extraction and loading now happen directly as part of the Snowflake platform, with no external ETL tool required which is a very polite way of saying: the toll booth a lot of vendors have been sitting at for a decade just got bypassed.</p><p>This doesn’t mean Fivetran and friends disappear overnight. Best-of-breed connectors, mature SaaS ecosystems, and multi-warehouse support still matter. But if your entire stack already lives in Snowflake, the calculus just changed hard.</p><p><strong>Real Business Use Cases (Not Just “Faster ETL”)</strong></p><p><strong>1.</strong> <strong>Replacing Legacy CDC Tools for Core Systems</strong> Instead of licensing a separate CDC product to stream changes from an on-prem Oracle or SQL Server system, teams are wiring that replication directly into Snowflake cutting a vendor, a contract, and a point of failure out of the architecture.</p><p><strong>2. Feeding RAG Pipelines Without a Side Project</strong> Enterprise knowledge assistants need documents chunked, embedded, and indexed continuously. Instead of building a bespoke ingestion service for SharePoint or Google Drive content, that pipeline now runs natively — landing preprocessed, AI-ready data straight into Snowflake for Cortex-powered search and agents.</p><p><strong>3. Unifying Structured + Unstructured Data for Compliance </strong>Financial services and healthcare teams dealing with a mix of transaction tables and scanned documents, call recordings, or sensor logs no longer need three separate ingestion tools with three separate governance models. One platform, one lineage trail, one audit story.</p><p><strong>4. Real-Time Operational Dashboards</strong> Manufacturing and logistics teams streaming sensor and IoT data can land it near-real-time instead of batch-loading overnight turning “yesterday’s report” into “this morning’s decision.”</p><p><strong>5. Consolidating a Fragmented Data Stack</strong> Plenty of data teams are running four or five ingestion tools because each one was the right answer for a specific source at the time. Openflow gives platform teams a genuine opportunity to consolidate that sprawl into one governed, observable layer fewer vendors to manage, fewer blind spots in lineage.</p><p><strong>The Part That Should Make You Pause</strong></p><p>Bringing ingestion, transformation, AI preprocessing, and governance into one platform is powerful. It’s also a deeper form of vendor lock-in than most teams are used to reckoning with. When your extraction, transformation, <em>and</em> warehouse all live in the same platform, switching costs stop being theoretical.</p><p>That’s not a reason to avoid it Snowflake customers already made that bet years ago. But it’s worth naming plainly instead of pretending “unified platform” is a free lunch.</p><p><strong>So, Is This the End of Traditional ETL?</strong></p><p>Not the end. But definitely the beginning of the end for “ETL as a separate line item.” The center of gravity is shifting from a fragmented stack of point tools stitched together with duct tape and Airflow DAGs, to a single governed platform where ingestion, transformation, and AI consumption all happen in the same place.</p><p>Snowflake didn’t announce this with fireworks. It didn’t need to. The vendors who should be paying attention already are.</p><p><em>Are you seeing Openflow replace parts of your ETL stack yet, or is it still early days in your organization? Genuinely curious what’s actually moving to production versus what’s still in evaluation.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16ae3d843665" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Snowflake Advanced Data Engineer Certification]]></title>
            <link>https://medium.com/@lakhansd777/snowflake-advanced-data-engineer-certification-5f8dd2dc3f20?source=rss-4c54a1289c4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f8dd2dc3f20</guid>
            <dc:creator><![CDATA[Lakhan Dhanashree]]></dc:creator>
            <pubDate>Tue, 10 Nov 2026 19:04:01 GMT</pubDate>
            <atom:updated>2026-11-10T19:04:01.591Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Snowflake Advanced Data Engineer Certification</strong></p><p>30-Question Practice Test</p><p>Focus on advanced, scenario-based questions similar to SnowPro Advanced Data Engineer</p><p>1. Change Data Capture</p><p>A data engineering team must process only newly inserted and updated records from a source table. Which Snowflake feature is specifically designed for this purpose?</p><p>A. Materialized Views<br>B. Streams<br>C. Search Optimization Service<br>D. Clustering Keys</p><p>Answer: <strong>B</strong></p><p>2. Stream Behavior</p><p>A stream is created on a table. No consumer process reads the stream for several days. What happens?</p><p>A. Data is automatically deleted from the table<br>B. Change metadata accumulates until consumed or retention expires<br>C. Stream becomes invalid immediately<br>D. Warehouse automatically scales</p><p>Answer: <strong>B</strong></p><p>3. Snowpipe Streaming</p><p>What is the primary advantage of Snowpipe Streaming over traditional Snowpipe?</p><p>A. Lower storage costs<br>B. Eliminates the need for stages and achieves lower latency<br>C. Increases Time Travel retention<br>D. Eliminates warehouses</p><p>Answer: <strong>B</strong></p><p>4. Query Optimization</p><p>A query frequently searches for a single customer ID within billions of rows. Which feature should be evaluated first?</p><p>A. Search Optimization Service<br>B. Data Sharing<br>C. Dynamic Tables<br>D. Tasks</p><p>Answer: <strong>A</strong></p><p>5. Dynamic Tables</p><p>Which statement best describes Dynamic Tables?</p><p>A. They only refresh manually<br>B. They automate incremental transformation pipelines<br>C. They replace warehouses<br>D. They are external storage layers</p><p>Answer: <strong>B</strong></p><p>6. File Loading</p><p>Which command loads data from a stage into a table?</p><p>A. PUT<br>B. GET<br>C. COPY INTO<br>D. MERGE</p><p>Answer: <strong>C</strong></p><p>7. Semi-Structured Data</p><p>Which Snowflake data type stores JSON data?</p><p>A. OBJECT<br>B. ARRAY<br>C. VARIANT<br>D. STRING</p><p>Answer: <strong>C</strong></p><p>8. FLATTEN Function</p><p>What is the primary purpose of FLATTEN?</p><p>A. Compress data<br>B. Normalize warehouse usage<br>C. Explode semi-structured arrays into rows<br>D. Create materialized views</p><p>Answer: <strong>C</strong></p><p>9. Streams and Tasks</p><p>Which architecture is most commonly used for incremental CDC pipelines?</p><p>A. Search Optimization + Clustering<br>B. Streams + Tasks<br>C. Replication + Failover<br>D. Data Sharing + Marketplace</p><p>Answer: <strong>B</strong></p><p>10. MERGE Statement</p><p>When is MERGE typically used?</p><p>A. Managing upserts into target tables<br>B. Creating warehouses<br>C. Securing stages<br>D. Creating shares</p><p>Answer: <strong>A</strong></p><p>11. Clustering</p><p>The primary purpose of clustering keys is:</p><p>A. Encryption<br>B. Physical organization for pruning efficiency<br>C. Backup recovery<br>D. Replication</p><p>Answer: <strong>B</strong></p><p>12. Query Performance</p><p>Which Snowflake object helps diagnose slow SQL execution?</p><p>A. Query Profile<br>B. Stage Browser<br>C. Stream Metadata<br>D. Resource Monitor</p><p>Answer: <strong>A</strong></p><p>13. Data Sharing</p><p>What storage overhead occurs when sharing data through Secure Data Sharing?</p><p>A. Full copy of data required<br>B. Data duplicated in consumer account<br>C. No data copy required<br>D. Additional external storage required</p><p>Answer: <strong>C</strong></p><p>14. Warehouse Caching</p><p>If a warehouse is suspended and restarted, which cache is lost?</p><p>A. Result Cache<br>B. Metadata Cache<br>C. Local Disk Cache<br>D. Cloud Services Cache</p><p>Answer: <strong>C</strong></p><p>15. Time Travel</p><p>Time Travel enables users to:</p><p>A. Replicate databases globally<br>B. Recover historical versions of data<br>C. Encrypt tables<br>D. Share data externally</p><p>Answer: <strong>B</strong></p><p>16. Dynamic Table Refresh</p><p>What determines when a Dynamic Table refreshes?</p><p>A. Target Lag configuration<br>B. Warehouse size only<br>C. Table size only<br>D. Role hierarchy</p><p>Answer: <strong>A</strong></p><p>17. Search Optimization</p><p>Which workload benefits most from Search Optimization Service?</p><p>A. Large table scans<br>B. Highly selective lookups<br>C. Full aggregation scans<br>D. File staging</p><p>Answer: <strong>B</strong></p><p>18. External Tables</p><p>External Tables are commonly used for:</p><p>A. Viewing data stored outside Snowflake without loading it<br>B. Replacing warehouses<br>C. Clustering internal tables<br>D. Dynamic Data Masking</p><p>Answer: <strong>A</strong></p><p>19. Task Dependencies</p><p>Why use task graphs?</p><p>A. To orchestrate dependent workloads<br>B. To secure databases<br>C. To replicate data<br>D. To optimize warehouses</p><p>Answer: <strong>A</strong></p><p>20. Multi-Cluster Warehouses</p><p>Multi-cluster warehouses primarily address:</p><p>A. High concurrency workloads<br>B. Data governance<br>C. Security compliance<br>D. Retention management</p><p>Answer: <strong>A</strong></p><p>21. Replication</p><p>Database replication supports:</p><p>A. Business continuity and disaster recovery<br>B. Query acceleration only<br>C. Result caching<br>D. Search optimization</p><p>Answer: <strong>A</strong></p><p>22. Materialized Views</p><p>Materialized Views are most useful when:</p><p>A. Frequently executing expensive repeated queries<br>B. Loading flat files<br>C. Managing user roles<br>D. Creating streams</p><p>Answer: <strong>A</strong></p><p>23. Data Governance</p><p>Which feature masks sensitive data dynamically?</p><p>A. Secure Views<br>B. Dynamic Data Masking Policies<br>C. Tasks<br>D. Streams</p><p>Answer: <strong>B</strong></p><p>24. CDC Pipeline Scenario</p><p>A source database sends inserts, updates, and deletes. What Snowflake combination best captures these changes?</p><p>A. Stage + PUT<br>B. Streams + MERGE<br>C. Search Optimization + Views<br>D. Clustering + Materialized Views</p><p>Answer: <strong>B</strong></p><p>25. Warehouse Sizing</p><p>A query is CPU-bound and consistently slow. What should be evaluated first?</p><p>A. Larger warehouse size<br>B. Additional database roles<br>C. New stages<br>D. Failover groups</p><p>Answer: <strong>A</strong></p><p>26. Semi-Structured Querying</p><p>Which function retrieves JSON fields from VARIANT data?</p><p>A. FLATTEN only<br>B. Colon notation (:) C. COPY INTO<br>D. STREAM</p><p>Answer: <strong>B</strong></p><p>Example:</p><p>SELECT payload:customer.name FROM orders;</p><p>27. Snowflake Failover Group</p><p>Failover Groups are used to:</p><p>A. Manage data encryption keys<br>B. Replicate critical account objects across regions/accounts<br>C. Replace warehouses<br>D. Create streams automatically</p><p>Answer: <strong>B</strong></p><p>28. Dynamic Tables vs Streams</p><p>Which is generally preferred when the goal is minimizing pipeline maintenance?</p><p>A. Dynamic Tables<br>B. Streams Only<br>C. Tasks Only<br>D. External Tables</p><p>Answer: <strong>A</strong></p><p>29. Cost Optimization</p><p>A warehouse experiences long periods of inactivity. Best practice?</p><p>A. Increase cluster count<br>B. Enable Auto-Suspend and Auto-Resume<br>C. Use larger warehouse sizes permanently<br>D. Disable caching</p><p>Answer: <strong>B</strong></p><p>30. Architecture Scenario</p><p>A business requires:</p><p>Data freshness under 5 minutes</p><p>CDC ingestion</p><p>Automated transformation</p><p>Low operational overhead</p><p>Best solution?</p><p>A. Manual COPY jobs every hour<br>B. Streams + Dynamic Tables + Snowpipe Streaming<br>C. Data Sharing only<br>D. Materialized Views only</p><p>Answer: <strong>B</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f8dd2dc3f20" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hybrid Tables just crossed another cloud boundary. ❄️☁️]]></title>
            <link>https://medium.com/@lakhansd777/hybrid-tables-just-crossed-another-cloud-boundary-%EF%B8%8F-%EF%B8%8F-b6b72312267b?source=rss-4c54a1289c4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6b72312267b</guid>
            <category><![CDATA[hybrid]]></category>
            <dc:creator><![CDATA[Lakhan Dhanashree]]></dc:creator>
            <pubDate>Mon, 02 Nov 2026 04:50:23 GMT</pubDate>
            <atom:updated>2026-11-02T04:50:23.046Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KgR2HxQ8wHsjCylub1am5A.png" /></figure><p>🚀 Snowflake Hybrid Tables just expanded to Google Cloud.</p><p>As of October 30, 2026, Hybrid Tables are available in Preview across Google Cloud commercial regions.</p><p>But the interesting part isn’t just another cloud becoming available.</p><p>It’s what this enables for real application architectures.</p><p>❄️ What makes Hybrid Tables different?</p><p>Hybrid Tables bring transactional and analytical patterns closer together inside Snowflake.</p><p>Instead of automatically designing:</p><p>Application → Operational DB → CDC/ETL → Snowflake → Analytics</p><p>For the right workload, we can start evaluating:</p><p>Application/API → Snowflake Hybrid Tables → Analytics &amp; AI</p><p>That can mean fewer moving parts and less synchronization complexity.</p><p>💡 Where would I practically evaluate Hybrid Tables?</p><p>🔹 Workflow &amp; pipeline state</p><p>Store ingestion status, processing checkpoints, job metadata or application state where frequent row-level reads and updates are required.</p><p>🔹 Low-latency data serving</p><p>Imagine an application needs a precomputed customer metric, account summary or operational KPI. Hybrid Tables can be evaluated as the serving layer rather than immediately introducing another operational database.</p><p>🔹 Lightweight transactional applications</p><p>For Snowflake-centric applications requiring relational data, primary/foreign keys and transactional behavior, Hybrid Tables provide an interesting architecture option.</p><p>🔹 Operational + analytical data together</p><p>Imagine combining:</p><p>→ Current customer/application state → Historical transactions → Analytical metrics → AI-generated insights</p><p>Rather than continuously moving data between operational and analytical platforms, selected workloads could stay closer to the Snowflake ecosystem.</p><p>🔹 AI application state</p><p>This is another area I would experiment with.</p><p>AI applications and agents often need operational state, metadata, enterprise context and analytical results. Hybrid Tables could be worth evaluating for lightweight state-management scenarios alongside Snowflake’s broader AI and data platform.</p><p>⚠️ But this is not “replace every OLTP database with Snowflake.”</p><p>Architecture still matters.</p><p>Before production adoption, I would validate:</p><p>✅ Read/write patterns ✅ Concurrency and latency requirements ✅ Transaction boundaries ✅ Indexing strategy ✅ Storage and throughput limits ✅ Required Snowflake features ✅ Governance requirements ✅ Preview limitations on Google Cloud</p><p>My approach would be simple:</p><p>Start with ONE focused operational workload.</p><p>Benchmark it.</p><p>Compare it against the existing operational DB + integration architecture.</p><p>Then decide whether Hybrid Tables actually simplify the platform.</p><p>💡 My key takeaway:</p><p>Hybrid Tables aren’t interesting simply because Snowflake can handle more transactional patterns.</p><p>They’re interesting because they challenge a long-standing architectural assumption:</p><p>“Operational data and analytical data must always live on completely separate platforms.”</p><p>Sometimes they absolutely should.</p><p>But increasingly, the better question may be:</p><p>“Do we really need another database and synchronization pipeline for this workload?”</p><p>That’s where Hybrid Tables become interesting.</p><p>💬 If you’re running Snowflake on Google Cloud, which use case would you test first: workflow state, application serving, lightweight transactions, or AI application state?</p><p>🚀 Think workload-first, not technology-first.</p><p>#Snowflake #GoogleCloud #HybridTables #Unistore #DataArchitecture #DataEngineering #CloudArchitecture #EnterpriseAI <a href="https://www.linkedin.com/company/snowflake-computing/">Snowflake</a> <a href="https://www.linkedin.com/company/snowflake-educational-services/">Snowflake Educational Services</a> <a href="https://www.linkedin.com/showcase/snowflake-partner-network/">Snowflake Partner Network</a> <a href="https://www.linkedin.com/showcase/snowflake-developers/">Snowflake Developers</a> <a href="https://www.linkedin.com/showcase/snowflake-challenge/">Snowflake Challenge</a> <a href="https://www.linkedin.com/company/google/">Google</a> <a href="https://www.linkedin.com/showcase/google-cloud/">Google Cloud</a> <a href="https://www.linkedin.com/company/accentureindia/">Accenture in India</a> <a href="https://www.linkedin.com/company/accenture/">Accenture</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6b72312267b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Snowflake CoCo: From Coding Assistant to AI Teammate]]></title>
            <link>https://medium.com/@lakhansd777/snowflake-coco-from-coding-assistant-to-ai-teammate-00703bc0c64e?source=rss-4c54a1289c4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/00703bc0c64e</guid>
            <category><![CDATA[coco]]></category>
            <dc:creator><![CDATA[Lakhan Dhanashree]]></dc:creator>
            <pubDate>Mon, 02 Nov 2026 04:48:18 GMT</pubDate>
            <atom:updated>2026-11-02T04:48:18.264Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qjEnFfCDWg86FskR8R1NBg.png" /></figure><p>❄️ Snowflake CoCo Desktop is becoming less like an AI chat window and more like a serious agentic development environment for data teams.</p><p>The October 29, 2026 v1.23.1 release may look incremental, but several changes directly improve the everyday developer workflow: fewer interruptions, broader file context, stronger SQL control, and cleaner agent configuration. <a href="https://docs.snowflake.com/en/release-notes/2026/other/2026-10-29-coco-desktop-v1-23-1">[docs.snowflake.com]</a></p><p>So, what actually changed?</p><p>🔹 Fewer interruptions during agentic coding</p><p>With Default Approvals, ordinary file edits no longer require approval, including edits outside the current workspace.</p><p>Importantly, Snowflake retains approval prompts for sensitive areas such as shell startup files, SSH configuration, Git configuration/hooks, and agent/MCP settings.</p><p>That balance matters.</p><p>The goal of an AI coding agent shouldn’t be “remove every human checkpoint.”</p><p>It should be “remove unnecessary friction while retaining controls where risk is higher.”</p><p>🔹 Any file type can now become part of the conversation</p><p>The Upload file option now accepts any file type, including examples such as .html and .mp4. The existing 30 MB per-file limit remains unchanged.</p><p>Practical use case:</p><p>Imagine investigating a data application issue where context isn’t limited to SQL or Python.</p><p>You could bring different supporting artifacts into the CoCo conversation rather than treating the agent as purely a code-file assistant.</p><p>For enterprise teams, richer context can mean more useful agent-assisted investigation.</p><p>🔹 SQL Playground gets a very practical improvement</p><p>SQL Playground now runs queries asynchronously, and clicking Stop cancels the running query in Snowflake.</p><p>Simple change?</p><p>Yes.</p><p>Useful for someone iterating on expensive or unexpectedly long-running SQL?</p><p>Absolutely.</p><p>For me, practical controls like this matter just as much as headline AI capabilities.</p><p>🔹 Better context hygiene for AI conversations</p><p>When developers retry a prompt or revert to an earlier message and send again, CoCo no longer passes the replaced or reverted prompts and responses to the model.</p><p>That is an underrated improvement.</p><p>Agentic development isn’t only about giving an AI more context.</p><p>Sometimes it is equally important to ensure the model sees the right context.</p><p>🔹 Large SQL results become more useful to the agent</p><p>When an individual SQL result cell exceeds 4 KB, CoCo now saves the full value to a file the agent can read.</p><p>Previously, the agent received the truncated value and could issue additional queries trying to recover the missing information.</p><p>Think about troubleshooting commands such as DESCRIBE AGENT that can return large values.</p><p>This is the kind of improvement that can make agent-assisted debugging more efficient.</p><p>🔹 Agent extensibility becomes easier to manage</p><p>MCP servers, skills, agents, hooks, rules and plugins now have a dedicated Customize page within Agent view.</p><p>This becomes increasingly relevant as enterprise AI agents stop being standalone assistants and start connecting to broader development ecosystems.</p><p>Snowflake also states that CoCo Desktop supports MCP servers for connecting with tools such as Google Workspace, Jira and GitHub. <a href="https://www.snowflake.com/en/product/snowflake-coco/downloads/">[snowflake.com]</a></p><p>🏗️ The Architect’s Perspective</p><p>The real opportunity isn’t another set of AI coding features.</p><p>It’s reducing the friction between data, code, tools, context and developer actions.</p><p>Snowflake positions CoCo as a data-native AI coding agent that can work across data engineering, analytics and AI workflows, with Desktop, CLI, Snowsight and extensions offering different ways to work with it. <a href="https://www.snowflake.com/en/product/snowflake-coco/">[snowflake.com]</a></p><p>That creates some interesting enterprise scenarios:</p><p>🔹 Data engineering</p><p>Use conversational development around pipelines and data workflows while keeping the developer close to Snowflake and local development tools.</p><p>🔹 SQL troubleshooting</p><p>Run and investigate SQL interactively, stop problematic queries, and give the agent access to complete large-cell results.</p><p>🔹 Agent development</p><p>Bring together agents, MCP servers, skills, hooks, rules and plugins through the dedicated Customize experience.</p><p>🔹 Development with richer artifacts</p><p>Use different file types as contextual inputs when the problem extends beyond traditional source-code files.</p><p>⚠️ Production Reality Check</p><p>More agent autonomy also makes governance more important, not less.</p><p>For enterprise adoption, I would pay particular attention to:</p><p>✅ Which file operations can happen without approval</p><p>✅ Which sensitive operations still require explicit approval</p><p>✅ What MCP servers and tools agents can access</p><p>✅ What organizational context is exposed to the agent</p><p>✅ How agent-generated changes fit existing Git, testing and review practices</p><p>✅ Cost and operational controls around AI-assisted development</p><p>Snowflake specifically retaining approval requirements for sensitive files is therefore an important detail in this release.</p><p>💡 My takeaway:</p><p>The future of data engineering may not be developers writing every line manually or AI autonomously doing everything.</p><p>The more practical destination is somewhere in between:</p><p>Developer intent → AI agent → Enterprise context → Tools &amp; data → Human-controlled execution → Production outcome</p><p>CoCo Desktop v1.23.1 is another small but meaningful step in that direction.</p><p>💬 As AI coding agents become part of the data engineering lifecycle, where would you keep mandatory human approval: code changes, production SQL, security configuration, deployment, or all of them?</p><p>🚀 Worth exploring not just as an AI coding assistant, but as part of the evolving agentic data engineering workflow.</p><p>#Snowflake #SnowflakeCoCo #AgenticAI #DataEngineering #EnterpriseAI</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=00703bc0c64e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cortex Agent SDK (Public Preview): From Agent Demo to Production App on Snowflake]]></title>
            <link>https://medium.com/@lakhansd777/cortex-agent-sdk-public-preview-from-agent-demo-to-production-app-on-snowflake-c55cb4af0390?source=rss-4c54a1289c4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c55cb4af0390</guid>
            <dc:creator><![CDATA[Lakhan Dhanashree]]></dc:creator>
            <pubDate>Wed, 14 Oct 2026 05:22:34 GMT</pubDate>
            <atom:updated>2026-10-14T05:22:34.533Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZIWCPYAyUiE5xkffkJD_Zw.png" /></figure><p><em>By a Senior Data &amp; AI Architect | Audience: Data Engineers, Architects, Snowflake professionals and learners</em></p><p>Most agent demos die at the same moment: someone asks a real question, the answer takes forty seconds, the browser tab closes, and the work is gone. A demo can ignore that. A production app cannot.</p><p><em>On October 13, 2026</em>, Snowflake put a Public Preview SDK in front of exactly that problem. The Cortex Agent SDK, for Python and TypeScript, is aimed at applications that run Cortex Agents and Cortex Code, keep conversations and threads, observe durable turns, and reconnect to long-running work. If you have been wiring raw REST calls and hand-rolled streaming code around agents, this is the release to read carefully.</p><p><strong>Before you build:</strong> this is a preview. I worked from Snowflake’s release notes and the public package listing, so confirm method names, parameters and limits in the official SDK reference before copying any code below. Treat the Python in this post as a design sketch, not a tested snippet.</p><h3>Why this matters now</h3><p>Agents on Snowflake have matured fast. Recent releases added a code execution tool (generally available October 5, 2026) and a compact endpoint, agent:compact, that summarizes a conversation so later agent:run calls spend fewer tokens (preview since September 21). Those are platform capabilities. What was missing for application teams was a supported client layer that handles the awkward parts: streaming, threads, long-running runs and reconnection. That is the gap an SDK closes.</p><p>For architects, the shift is from ‘can the agent answer?’ to ‘can the agent be operated?’ Operability is where projects succeed or stall.</p><h3>The simple explanation</h3><p>Think of four building blocks:</p><p>• <strong>Agent:</strong> the configured Cortex Agent object (instructions, tools, models) living in a Snowflake schema.</p><p>• <strong>Thread:</strong> the conversation container, so context survives across messages.</p><p>• <strong>Turn / run:</strong> one request-and-response cycle. A run can be streamed, non-streamed or run in the background.</p><p>• <strong>Reconnect:</strong> if the client drops, you pick the run back up instead of restarting it.</p><p>The public package listing for snowflake-cortex-agent-sdk (version 0.0.1 at the time I checked) describes resources for running agents, coding-agent runs, threads (create, list, describe, update, delete, search), feedback, and a runs resource for reconnecting, cancelling and waiting on background work. It also mentions a higher-level Conversation abstraction and typed response views.</p><p>One distinction to keep straight: there is also a separate <strong>Cortex Code Agent SDK</strong> (cortex-code-agent-sdk) that drives the Cortex Code CLI. Different package, different prerequisites. This post focuses on the new Cortex Agent SDK.</p><h3>Production-grade reference architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*MIlQ84O_BY1Kdr1oBeNWuw.png" /></figure><p>Key design choice: the SDK lives in your <strong>application tier</strong>, not in the browser. The service holds credentials, enforces identity mapping and decides what a user may ask.</p><h3>Three realistic enterprise use cases</h3><p><strong>1. Finance analytics copilot.</strong> Analysts ask questions over curated finance marts through a semantic view, and long reconciliation-style questions run as background runs. Threads keep follow-ups (‘now exclude intercompany’) coherent, and reconnection means a closed laptop does not kill the analysis.</p><p><strong>2. Operations and support knowledge agent.</strong> An agent combines Cortex Search over runbooks and tickets with SQL tools over incident data. Feedback captured per turn feeds a weekly review of weak answers. Compacting long threads keeps token spend predictable.</p><p><strong>3. Data platform engineering assistant.</strong> Using the Cortex Code side of the SDK, a platform team builds an internal tool that reviews pipeline code and proposes fixes, with cancellation and turn limits as guardrails. This one needs the tightest permissions because it can touch code and SQL.</p><h3>Implementation steps</h3><h3>Step 1: Lock down who can use agents</h3><p>By default the SNOWFLAKE.CORTEX_USER database role is granted to PUBLIC. Snowflake’s agent documentation says you can revoke that and grant SNOWFLAKE.CORTEX_AGENT_USER to specific roles for agent-only access. Database roles cannot be granted straight to users, so go through a custom role.</p><p><em>— Run once as ACCOUNTADMIN<br> REVOKE DATABASE ROLE SNOWFLAKE.CORTEX_USER FROM ROLE PUBLIC;<br> <br> CREATE ROLE IF NOT EXISTS AGENT_APP_ROLE;<br> GRANT DATABASE ROLE SNOWFLAKE.CORTEX_AGENT_USER TO ROLE AGENT_APP_ROLE;<br> <br> — Object access (SYSADMIN / agent owner)<br> GRANT USAGE ON DATABASE AI_PROD TO ROLE AGENT_APP_ROLE;<br> GRANT USAGE ON SCHEMA AI_PROD.AGENTS TO ROLE AGENT_APP_ROLE;<br> GRANT USAGE ON AGENT AI_PROD.AGENTS.FINANCE_COPILOT TO ROLE AGENT_APP_ROLE;<br> <br> — Observability for the ops role only<br> GRANT MONITOR ON AGENT AI_PROD.AGENTS.FINANCE_COPILOT TO ROLE AI_OPS_ROLE;</em></p><p>If a role also holds CORTEX_USER, it keeps access to all covered AI features, so revoke it where you want agent-only access.</p><h3>Step 2: Isolate cost</h3><p>CREATE WAREHOUSE AGENT_WH WAREHOUSE_SIZE = ‘XSMALL’<br> AUTO_SUSPEND = 60 AUTO_RESUME = TRUE;<br> <br> CREATE RESOURCE MONITOR AGENT_RM WITH CREDIT_QUOTA = 200<br> TRIGGERS ON 80 PERCENT DO NOTIFY<br> ON 100 PERCENT DO SUSPEND;<br> ALTER WAREHOUSE AGENT_WH SET RESOURCE_MONITOR = AGENT_RM;</p><h3>Step 3: Create a service identity</h3><p>Use a service user with key-pair or another non-password method, a network policy, and a role limited to AGENT_APP_ROLE. Do not run app traffic as a human user.</p><h3>Step 4: Call the agent from your service (sketch)</h3><p>The resource layout below follows the package listing (agent, threads, runs). Method names and arguments are <strong>illustrative</strong>, so check the SDK reference.</p><p><em># Design sketch — verify names against the SDK reference (package is 0.0.1)<br> import os</em><br> <br> <em># 1. Build the client with key-pair auth; constructor details are in the SDK docs<br> client = make_client(<br> account=os.environ[‘SNOWFLAKE_ACCOUNT’],<br> user=os.environ[‘SNOWFLAKE_SVC_USER’],<br> private_key_path=os.environ[‘SNOWFLAKE_KEY_PATH’],<br> )</em><br> <br> <em>def ask(user_id: str, question: str, thread_id: str | None = None):<br> # 2. One thread per user conversation; store the id in your app database<br> thread_id = thread_id or client.threads.create(metadata={‘user’: user_id}).id<br> <br> # 3. Long questions: start a background run so it survives a client drop<br> run = client.agent.run(<br> agent=’AI_PROD.AGENTS.FINANCE_COPILOT’,<br> thread=thread_id,<br> message=question,<br> background=True,<br> )<br> save_run_id(user_id, run.id) # persist for reconnect<br> return thread_id, run.id<br> <br> def resume(run_id: str):<br> # 4. After a reconnect, resume the same run instead of re-asking<br> return client.runs.resume(run_id)<br> <br> def cancel(run_id: str):<br> return client.runs.cancel(run_id) # user hit stop, or a timeout fired</em></p><p>The TypeScript SDK follows the same ideas for Node-based backends and web services.</p><h3>Step 5: Add feedback and traces</h3><p>Send a thumbs-up or thumbs-down per turn through the feedback resource, and tie the turn to your own request ID so support teams can trace a complaint to a trace.</p><h3>POC to production</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/1*xauF2rYLCFLobkCCUmkfLQ.png" /></figure><p>My rule: do not promote until you can answer three questions. Who asked what, which data did the agent touch, and what did it cost?</p><h3>Architecture decisions and trade-offs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*n25WUhKgdFJxtpPGyKrrXw.png" /></figure><p>Caller’s rights when row-level governance matters. The preview notes for the code execution tool say owner’s rights are not supported with it, so check the current GA behaviour</p><h3>Security, RBAC and governance</h3><p>• Use least privilege: CORTEX_AGENT_USER for app roles, not CORTEX_USER, and no PUBLIC grants.</p><p>• Treat user input as untrusted. Put guardrails in agent instructions and validate in your API layer too.</p><p>• Decide per tool whether caller’s or owner’s rights apply, and test that masking and row access policies behave as you expect for each tool type.</p><p>• Keep sensitive columns masked or redacted before they reach any index or semantic model.</p><p>• For Cortex Code use cases, restrict permissions and set turn limits; the SDK exposes options such as a maximum number of turns and an abort mechanism.</p><h3>Observability and reliability</h3><p>Snowflake documents automatic logging for Agent API requests, with one trace per turn, and you need MONITOR on the agent to inspect it. Build on that:</p><p>• Alert on error rate, latency percentiles and cancelled runs.</p><p>• Log run IDs and thread IDs in your own application logs for correlation.</p><p>• Add timeouts and retries with backoff on the client, and make reconnection idempotent so a retry never creates a duplicate run.</p><p>• Cap concurrency per user and per tenant.</p><h3>CI/CD</h3><p>1. Keep agent definitions, semantic views and grants in Git.</p><p>2. Deploy with a pipeline using a service role, never by hand.</p><p>3. Run an evaluation set (golden questions with expected behaviour) in the test environment as a gate.</p><p>4. Promote a versioned agent to prod, and keep the previous version for rollback.</p><p>5. Pin the SDK version, since 0.0.x packages can change quickly.</p><h3>Cost optimization</h3><p>• Dedicated warehouse, 60-second auto-suspend, resource monitor and budget alerts.</p><p>• Compact long conversations so you stop re-sending history.</p><p>• Pick the smallest model that passes your evaluation set.</p><p>• Cap turns and tool calls per request.</p><p>• Cache stable answers (for example definition lookups) in your app tier.</p><p>• Review spend by agent and by tenant weekly during the first month.</p><h3>Common mistakes and fixes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eISQMPCQnV9RLml72t7lFg.png" /></figure><h3>Save-worthy best-practice checklist</h3><p>☐ CORTEX_USER revoked from PUBLIC; agent roles use CORTEX_AGENT_USER</p><p>☐ Service user with key-pair auth and a network policy</p><p>☐ Least-privilege grants on database, schema and agent; MONITOR only for ops</p><p>☐ Dedicated warehouse with resource monitor and budget alerts</p><p>☐ Masking, row access and tagging verified for every agent tool</p><p>☐ Threads mapped to users; run IDs persisted for reconnect</p><p>☐ Timeouts, cancel support and idempotent retries</p><p>☐ Per-turn traces reviewed; dashboards and alerts live</p><p>☐ Evaluation set gating every release</p><p>☐ Agents versioned in Git; rollback tested</p><p>☐ SDK version pinned and wrapped behind your own interface</p><p>☐ Compaction or thread limits to control tokens</p><p>☐ Preview terms and data-use implications reviewed with security</p><h3>Five key takeaways</h3><p>1. The Cortex Agent SDK targets the operational gaps of agent apps: threads, durable turns and reconnection.</p><p>2. Design for long-running work from day one, because real questions are slow.</p><p>3. Governance does not come with the SDK. You still design roles, rights models and data policies.</p><p>4. Observability and evaluation gates are what separate a demo from a service.</p><p>5. It is a preview. Adopt it, but isolate it behind your own interface and pin versions.</p><h3>Your turn</h3><p>How are you handling agent state today: raw REST, your own conversation store, or something else? And what would make you trust a preview SDK enough to put it in front of business users? Share your approach in the comments, and I will compare notes in a follow-up post on evaluation gates for Cortex Agents.</p><h3>SEO package</h3><p><strong>SEO title:</strong> Snowflake Cortex Agent SDK Preview: Production Architecture Guide</p><p><strong>Meta description:</strong> Learn how to take Snowflake Cortex Agents to production with the new Agent SDK: architecture, RBAC, observability, CI/CD, cost control and a checklist.</p><p><strong>URL slug:</strong> /snowflake-cortex-agent-sdk-production-guide</p><p><strong>Keywords:</strong> Snowflake Cortex Agent SDK, Cortex Agents, Snowflake AI agents, Cortex Agent Python SDK, Cortex Agent TypeScript SDK, agent:run, Snowflake agent RBAC, CORTEX_AGENT_USER, Snowflake AI observability, Snowflake agent architecture, GenAI on Snowflake, Snowflake data engineering</p><h3>FAQs</h3><p><strong>What is the Cortex Agent SDK?</strong> A Public Preview SDK for Python and TypeScript that helps applications run Cortex Agents and Cortex Code, maintain conversations and threads, observe durable turns and reconnect to long-running work.</p><p><strong>Is it production-ready?</strong> It is a preview, and the Python package I checked was at version 0.0.1. Use it for pilots and wrap it behind your own interface until it reaches general availability.</p><p><strong>Which role do I need to run agents?</strong> A role with SNOWFLAKE.CORTEX_USER or SNOWFLAKE.CORTEX_AGENT_USER, plus USAGE on the database, schema and agent as documented. For agent-only access, revoke CORTEX_USER from PUBLIC and use CORTEX_AGENT_USER.</p><p><strong>What is the difference between the Cortex Agent SDK and the Cortex Code Agent SDK?</strong> They are separate packages. The Cortex Code Agent SDK drives the Cortex Code CLI, which must be installed separately. The Cortex Agent SDK targets Cortex Agents and related run, thread and feedback features.</p><p><strong>How do I monitor agent usage?</strong> Snowflake logs Agent API requests automatically with one trace per turn, and you need MONITOR on the agent to inspect it. Pair that with your own application logs and cost views.</p><p><strong>How do I control cost?</strong> Use a dedicated warehouse with a resource monitor, cap turns, compact long threads, choose the smallest model that passes your evaluations, and review spend by agent.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c55cb4af0390" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SELECT * FROM the Sky: Turning Drone, Satellite, and LiDAR Data into Business Intelligence]]></title>
            <link>https://medium.com/@lakhansd777/select-from-the-sky-turning-drone-satellite-and-lidar-data-into-business-intelligence-57de59c5195a?source=rss-4c54a1289c4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/57de59c5195a</guid>
            <dc:creator><![CDATA[Lakhan Dhanashree]]></dc:creator>
            <pubDate>Wed, 30 Sep 2026 07:11:23 GMT</pubDate>
            <atom:updated>2026-09-30T07:11:23.845Z</atom:updated>
            <content:encoded><![CDATA[<p>Your Drone Just Captured 40,000 Images. Now What? Turning LiDAR and Aerial Imagery into Decisions with Snowflake</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iKRuH9HPs9skFfZCuqjfjw.png" /></figure><p><em>How to stop treating pixels and point clouds like “special data” and start querying them like any other table.</em></p><p>Last year I watched a team spend three weeks manually reviewing drone footage of transmission lines. Three weeks, for one inspection cycle. The images sat in an S3 bucket, the LiDAR scans sat on an engineer’s external drive, and the “analytics” was a shared spreadsheet with a column called <em>“looks bad??”</em></p><p>Sound familiar?</p><p>Most companies aren’t short on imagery. Drones, satellites, CCTV, dashcams, and handheld LiDAR scanners generate an absurd amount of it. What they’re short on is a production-grade pipeline that turns raw pixels and point clouds into something a business can act on.</p><p>In this post I’ll walk through how to do exactly that on Snowflake: an architecture that works, the real business use cases, and the parts nobody puts in the demo.</p><p>Why Snowflake for Geospatial and Vision Workloads?</p><p>A fair question. Snowflake isn’t the first name that comes to mind for computer vision. But the shift to a unified AI Data Cloud changes the equation:</p><ul><li>Unstructured data is now first-class. Images, PDFs, and point cloud files live in stages with directory tables, governed by the same RBAC, masking, and lineage as your structured data.</li><li>Native geospatial. The GEOGRAPHY type, spatial joins, and H3 indexing functions are built in. You don’t need a separate GIS server for most analytics.</li><li>Cortex AI / AISQL. Multimodal LLM functions can reason over images directly from SQL, with no model hosting and no GPU babysitting for the common cases.</li><li>Snowpark Container Services (SPCS). When you need heavy lifting (PDAL for LiDAR, custom YOLO or SAM models, GPU inference), you run containers <em>next to your data</em> instead of shipping terabytes around.</li><li>One governance story. Your imagery insights join to asset registries, work orders, weather, and finance data without a single ETL hop to another platform.</li></ul><p>The real win isn’t any single feature. It’s that imagery stops being a silo.</p><p>The Architecture (Medallion, But for Pixels)</p><p>Here’s the pattern I keep coming back to. Think of it as a medallion architecture adapted for unstructured data.</p><p>Bronze: Raw capture Drone JPEGs, GeoTIFFs, satellite tiles, and LAS/LAZ point clouds land in cloud storage (S3, ADLS, or GCS) and are exposed to Snowflake through an external stage with a directory table. Nothing is transformed. You just get a queryable catalog of files, with metadata like path, size, and last modified.</p><p>Silver: Extracted and enriched This is where the work happens:</p><ul><li>EXIF/GPS metadata is parsed and geo-tagged</li><li>Point clouds are tiled, classified, and summarized (ground, vegetation, structures, wires)</li><li>Images are run through vision models for detection, classification, and captioning</li><li>Embeddings are generated for similarity search</li><li>Everything is stamped with H3 cell IDs for spatial joining</li></ul><p>Gold: Business-ready Feature tables, change-detection results, risk scores, and dashboards. This is what Streamlit apps, BI tools, and downstream agents consume.</p><p>Orchestration ties it together: Streams on the directory table detect new files, Tasks or Dynamic Tables trigger processing, and everything is idempotent so a re-run doesn’t double-count.</p><p>Step 1: Make Your Files Queryable</p><p>Start with the boring but critical foundation.</p><p><em>CREATE OR REPLACE STAGE raw_imagery</em></p><p><em>URL = ‘s3://your-bucket/drone-captures/’</em></p><p><em>STORAGE_INTEGRATION = s3_int</em></p><p><em>DIRECTORY = (ENABLE = TRUE AUTO_REFRESH = TRUE)</em></p><p><em>ENCRYPTION = (TYPE = ‘SNOWFLAKE_SSE’);</em></p><p>— Your images are now a table</p><p><em>SELECT relative_path, size, last_modified</em></p><p><em>FROM DIRECTORY(@raw_imagery)</em></p><p><em>WHERE relative_path ILIKE ‘%.jpg’;</em></p><p>Add a stream on the directory table and you’ve got event-driven ingestion for free.</p><p>Step 2: Let Cortex Read the Image (No GPU Required)</p><p>For a huge class of problems (damage assessment, roof condition, vegetation encroachment, PPE compliance), a multimodal model with a good prompt gets you surprisingly far, and you can do it from SQL.</p><p><em>SELECT</em></p><p><em>relative_path,</em></p><p><em>AI_COMPLETE(</em></p><p><em>‘claude-3–5-sonnet’, — use whichever multimodal model is available in your region</em></p><p><em>‘You are a utility inspection assistant. Look at this image of a power pole.</em></p><p><em>Return JSON with: defect_type, severity (1–5), vegetation_encroachment (true/false),</em></p><p><em>and a one-sentence explanation.’,</em></p><p><em>TO_FILE(‘@raw_imagery’, relative_path)</em></p><p><em>) AS inspection_json</em></p><p><em>FROM DIRECTORY(@raw_imagery)</em></p><p><em>WHERE relative_path ILIKE ‘%pole%’;</em></p><p>Pair it with AI_CLASSIFY for quick labeling, AI_FILTER to find “images that contain X,” and AI_EMBED to build a visual similarity search (“show me every image that looks like this corroded joint”).</p><p><em>Heads up: model names and function availability vary by region and change quickly. Check the current Cortex documentation before you commit to a design.</em></p><p>Step 3: LiDAR, Where It Gets Real</p><p>LiDAR is a different beast. A single survey can be tens of gigabytes of LAS/LAZ, and no LLM is going to “look at” a billion points. This is where Snowpark Container Services earns its keep.</p><p>The pattern:</p><ol><li>Tile the point cloud (COPC or spatial tiles) so processing is parallelizable.</li><li>Run PDAL / laspy / Open3D in an SPCS job to classify points, compute canopy height models, extract building footprints, or detect wires.</li><li>Write compact derived features back to Snowflake as tables or Iceberg/Parquet, not the raw points.</li></ol><p><em># Simplified: runs inside an SPCS job</em></p><p><em>import pdal, json</em></p><p><em>pipeline = pdal.Pipeline(json.dumps([</em></p><p><em>{“type”: “readers.copc”, “filename”: “tile_0421.copc.laz”},</em></p><p><em>{“type”: “filters.smrf”}, # ground classification</em></p><p><em>{“type”: “filters.hag_nn”}, # height above ground</em></p><p><em>{“type”: “filters.range”, “limits”: “HeightAboveGround[2:60]”},</em></p><p><em>]))</em></p><p><em>pipeline.execute()</em></p><p><em>arr = pipeline.arrays[0]</em></p><p><em># Summarize, don’t store a billion rows</em></p><p><em>features = {</em></p><p><em>“tile_id”: “0421”,</em></p><p><em>“max_canopy_height_m”: float(arr[“HeightAboveGround”].max()),</em></p><p><em>“point_count”: int(len(arr)),</em></p><p><em>}</em></p><p># -&gt; write to Snowflake via Snowpark session</p><p>The rule I live by: keep raw points in object storage, keep insights in Snowflake. Storing every point as a row is technically possible and financially embarrassing.</p><p>Then join your derived features spatially:</p><p><em>SELECT a.asset_id, a.asset_type, t.max_canopy_height_m</em></p><p><em>FROM assets a</em></p><p><em>JOIN lidar_tile_features t</em></p><p><em>ON H3_LATLNG_TO_CELL(</em><a href="http://a.lat/"><em>a.lat</em></a><em>, a.lon, 9) = t.h3_cell</em></p><p><em>WHERE t.max_canopy_height_m &gt; 8;</em> — trees tall enough to threaten the line</p><p>That single query is the payoff of the whole architecture.</p><p>Step 4: Change Detection, the Money Maker</p><p>The most valuable question isn’t <em>“what’s in this image?”</em> It’s <em>“what changed since last time?”</em></p><p>Store each capture with a timestamp and an H3 cell. Then compare periods:</p><p><em>SELECT</em></p><p><em>cur.h3_cell,</em></p><p><em>cur.vegetation_index — prev.vegetation_index AS ndvi_delta</em></p><p><em>FROM satellite_features cur</em></p><p><em>JOIN satellite_features prev</em></p><p><em>ON cur.h3_cell = prev.h3_cell</em></p><p><em>WHERE cur.capture_date = ‘2026–09–01’</em></p><p><em>AND prev.capture_date = ‘2026–03–01’</em></p><p><em>AND cur.vegetation_index — prev.vegetation_index &lt; -0.15;</em></p><p>Now you have an alerting layer: deforestation, encroachment, construction, flood extent, crop stress. It runs as a Task, and a Snowflake Alert or Streamlit app pushes it to the humans who care.</p><p>Real-World Use Cases</p><p>These are established application patterns across industries. Specific ROI figures vary widely, so treat any number you see in vendor decks as a hypothesis to test on your own data.</p><p>1. Utilities: vegetation management and pole inspection Utilities fly drones and LiDAR along right-of-ways to find trees threatening lines and to spot damaged hardware. Combining canopy height from LiDAR with asset locations lets crews prioritize trimming by actual risk instead of a fixed cycle. This is one of the most mature LiDAR use cases in the industry.</p><p>2. Insurance: catastrophe response and underwriting After a hurricane or hailstorm, aerial imagery classified for roof damage lets carriers triage claims in days rather than weeks. On the underwriting side, roof condition, pool presence, and defensible space around a property feed risk models.</p><p>3. Agriculture: crop health and yield forecasting Satellite and drone multispectral imagery yields vegetation indices (NDVI and friends) per field. Joined with weather and soil data, this drives irrigation, input application, and yield forecasts.</p><p>4. Construction and infrastructure: progress tracking Regular drone scans compared against BIM/design models show what’s built versus planned, calculate stockpile volumes, and flag deviations early, when they’re cheap to fix.</p><p>5. Retail and CPG: shelf and store analytics Camera images of shelves classified for out-of-stocks, planogram compliance, and pricing errors. Join to sales data and you can quantify the revenue lost to an empty shelf.</p><p>6. Telecom: tower and site audits Drone imagery of towers replaces climbs. Vision models flag missing equipment, corrosion, and antenna misalignment, and results join to site inventory automatically.</p><p>7. Real estate, urban planning, and smart cities Building footprints, heights, green cover, and heat-island analysis from LiDAR and satellite, joined with parcel and permit data.</p><p>“Production-Ready” Is Mostly Unglamorous</p><p>Demos are easy. Here’s what actually separates a POC from something you can sleep next to:</p><p>Cost control</p><ul><li>Set resource monitors on warehouses and GPU compute pools. Vision workloads can burn credits quietly.</li><li>Run the cheap filter first (metadata, geofence, a small classifier), and only send survivors to the expensive multimodal call.</li><li>Auto-suspend SPCS services and size pools appropriately.</li></ul><p>Idempotency and reliability</p><ul><li>Key every result on (file path + file version/ETag + model version). Re-runs shouldn’t duplicate.</li><li>Log failures to a dead-letter table and retry. Corrupt files <em>will</em> show up.</li></ul><p>Governance and privacy</p><ul><li>Imagery often contains faces, license plates, and private property. Decide upfront on blurring or redaction, retention windows, and access policies.</li><li>Use masking and row access policies on location fields. Precise coordinates can be sensitive.</li><li>Check drone and data regulations in your jurisdiction.</li></ul><p>Model quality and MLOps</p><ul><li>Register custom models in the Snowflake Model Registry and version them.</li><li>Keep a human-reviewed golden set and track precision/recall over time. Foundation models drift when providers update them.</li><li>For anything safety-critical, keep a human in the loop. AI triages, humans decide.</li></ul><p>Data engineering hygiene</p><ul><li>Standardize coordinate reference systems (CRS) early. Mixed CRS is the silent killer of geospatial projects.</li><li>Use cloud-optimized formats (COG for rasters, COPC for point clouds) so you read only what you need.</li><li>Version your tiling scheme. Changing it later hurts.</li></ul><p>Observability</p><ul><li>Track files ingested, files processed, latency, error rate, and cost per 1,000 images. If you can’t tell me the last one, you don’t yet control the pipeline.</li></ul><p>What About the “Agentic” Angle?</p><p>Once your imagery insights are structured and governed, layering a Cortex Agent or semantic view on top is natural. An operations manager can ask, in plain English, <em>“Which substations had severe vegetation encroachment in the last 30 days, and which crews are nearest?”</em> and get an answer that blends LiDAR-derived features, asset data, and crew schedules.</p><p>That’s the real promise: not “AI looks at pictures,” but imagery becoming a queryable, joinable, explainable business signal.</p><p>The Takeaway</p><p>If you remember one thing: don’t build an imagery platform. Extend your data platform to understand imagery.</p><p>Land files in a governed stage. Extract features with the lightest tool that works (Cortex for images, containers for LiDAR). Store insights, not raw bulk. Index everything spatially. Compare over time. Put humans where the stakes are high.</p><p>Your drones are already flying. The only question is whether their data ends up in a spreadsheet column called <em>“looks bad??”</em> or in a pipeline that tells you what to fix first.</p><p><em>If you found this useful, follow for more on AI + data architecture, and drop a comment with the messiest imagery pipeline you’ve inherited. I’d love to hear it.</em></p><p><em>#Snowflake #LiDAR #ComputerVision #GeospatialAI #DataEngineering #GenAI #DroneAnalytics #RemoteSensing</em> <a href="https://www.linkedin.com/company/accenture/">Accenture</a> <a href="https://www.linkedin.com/company/snowflake-computing/">Snowflake</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57de59c5195a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CoCo in Snowflake: From Healthcare Data Ingestion to Secure Power BI Reporting]]></title>
            <link>https://medium.com/@lakhansd777/coco-in-snowflake-from-healthcare-data-ingestion-to-secure-power-bi-reporting-efffcaba8d81?source=rss-4c54a1289c4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/efffcaba8d81</guid>
            <dc:creator><![CDATA[Lakhan Dhanashree]]></dc:creator>
            <pubDate>Sun, 27 Sep 2026 05:07:20 GMT</pubDate>
            <atom:updated>2026-09-27T05:07:20.081Z</atom:updated>
            <content:encoded><![CDATA[<p>Most discussions about Snowflake CoCo focus on generating SQL, explaining queries, or fixing data engineering code. But the more interesting enterprise use case is this:</p><p><strong>Can a data engineer use CoCo to build an end-to-end healthcare data platform — from multi-source ingestion to a governed Gold layer and Power BI reporting?</strong></p><p>Yes. CoCo can help with much more than individual SQL statements. It can discover Snowflake objects, execute SQL, work with dbt projects, troubleshoot pipelines, and help create data applications. The important distinction is that CoCo is an AI development assistant, not a replacement for your production orchestration, governance, or deployment processes. (<a href="https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code?utm_source=chatgpt.com">Snowflake Documentation</a>)</p><p>Let’s understand this through a realistic conversation between a healthcare data engineer and a Snowflake SME.</p><h3>The scenario: A healthcare analytics platform</h3><p>Imagine a healthcare organization called <strong>MediCare360</strong>.</p><p>The organization has:</p><ul><li>Patient registration data from an operational database.</li><li>Claims data arriving as CSV files in Azure Blob Storage.</li><li>Laboratory results from an API.</li><li>Hospital encounter data from an electronic health record system.</li><li>Power BI dashboards used by authorized healthcare analysts.</li></ul><p>The data engineering team needs to build a Snowflake medallion architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eNzGB0qXKG9vsZR7" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d-AO1QT-AFDCMijR" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VGNxRE4xwMjm6UX8" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/0*XCBjXtwkVDFcIBod" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*gHoqmZm9VT2fHyAf" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/0*_IdgugeqpukZSUYo" /></figure><p>The target architecture is:</p><pre>Operational Database<br>Azure Blob Storage<br>Laboratory API<br>EHR System<br>       |<br>       v<br>+----------------------+<br>| Ingestion Layer      |<br>| ADF / Snowpipe / API |<br>+----------------------+<br>       |<br>       v<br>+----------------------+<br>| BRONZE / RAW         |<br>| Original source data |<br>+----------------------+<br>       |<br>       v<br>+----------------------+<br>| SILVER               |<br>| Cleaned and joined   |<br>| healthcare data      |<br>+----------------------+<br>       |<br>       v<br>+----------------------+<br>| GOLD                 |<br>| Secure KPI views     |<br>| Business metrics     |<br>+----------------------+<br>       |<br>       v<br>+----------------------+<br>| Power BI             |<br>| Governed reporting   |<br>+----------------------+</pre><p>Now let’s see where CoCo enters the story.</p><h3>Conversation 1: “I need to build the entire platform”</h3><p><strong>Rahul-Data Engineer:</strong><br>“I have been asked to build a healthcare analytics platform in Snowflake. We have patient data in SQL Server, claims files in Azure Blob, and laboratory results from an API.</p><p>I need to ingest everything, create Bronze, Silver, and Gold layers, implement data quality, and expose secure KPIs to Power BI.</p><p>I have been working on this for weeks. Every time I solve one issue, another comes up.”</p><p><strong>Anita -Snowflake SME:</strong><br>“Have you tried CoCo?”</p><p><strong>Rahul:</strong><br>“I use AI assistants for SQL generation. But how will CoCo help me build a real production data engineering project?”</p><p><strong>Anita:</strong><br>“Think of CoCo as an AI pair engineer who understands your Snowflake environment and can work with your local project files.</p><p>Instead of asking it isolated questions, give it the entire engineering objective. It can help you discover objects, write SQL and Python, work with dbt, investigate failures, and execute approved Snowflake operations through your authorized connection.”</p><p><a href="https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code">https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code</a></p><p><strong>Rahul:</strong><br>“So I can ask it to build the architecture?”</p><p><strong>Anita:</strong><br>“Yes. But start with a plan, not immediate execution.”</p><h3>Rahul’s first CoCo prompt</h3><pre>I am building a healthcare analytics platform in Snowflake.</pre><pre>Sources:<br>1. Patient registration database<br>2. Claims CSV files in Azure Blob Storage<br>3. Laboratory results API<br>4. Hospital encounter data</pre><pre>Target:<br>- Bronze, Silver, Gold medallion architecture<br>- Metadata-driven ingestion<br>- Data quality checks<br>- Secure healthcare KPIs<br>- Power BI consumption</pre><pre>First:<br>1. Inspect the existing Snowflake databases and schemas.<br>2. Identify relevant source and target objects.<br>3. Propose a target architecture.<br>4. Recommend an ingestion pattern for each source.<br>5. Identify security and data quality requirements.<br>6. Do not execute DDL or DML until I approve the plan.</pre><p><strong>Anita:</strong><br>“CoCo can inspect the environment and propose an implementation. Review its plan before allowing changes. For complex tasks, CoCo supports a plan-and-approval workflow, and its Snowflake tools can perform object discovery and SQL execution.”</p><h3>Conversation 2: “How do I ingest multiple healthcare sources?”</h3><p><strong>Rahul:</strong><br>“The architecture looks good. But now I need to implement ingestion. SQL Server, Blob Storage, and an API all have different patterns. How do I use CoCo here?”</p><p><strong>Anita:</strong><br>“This is where you stop asking for generic SQL and give CoCo the actual project context.</p><p>For example, ask it to design a metadata-driven ingestion framework.”</p><h3>Prompt Rahul gives CoCo</h3><pre>Design a metadata-driven ingestion framework for our healthcare<br>analytics platform.</pre><pre>Sources:<br>- SQL Server patient registration<br>- Azure Blob claims CSV<br>- Laboratory API</pre><pre>Requirements:<br>- Snowflake Bronze layer<br>- Incremental ingestion where supported<br>- File-level audit logging<br>- Retry handling<br>- Error quarantine<br>- Source-to-target mapping<br>- Configurable ingestion frequency<br>- No hardcoded patient identifiers in logs</pre><pre>Create:<br>1. Metadata tables<br>2. Bronze table definitions<br>3. Ingestion stored procedures where appropriate<br>4. Data quality framework<br>5. Audit logging design<br>6. Sample configuration for each source</pre><pre>First show the design and SQL. Do not execute changes.</pre><p><strong>Anita:</strong><br>“CoCo can generate the metadata tables, SQL, and implementation files. For a real project, you still choose the appropriate ingestion technology and validate the generated code.”</p><p>For example:</p><p>SourcePossible ingestion patternSQL ServerCDC or incremental extraction through an orchestration toolAzure Blob claimsSnowpipe or Snowpipe Streaming, depending on the file and ingestion requirementsLaboratory APIAPI extraction service writing to a Snowflake-supported landing locationHospital encountersIncremental extraction or CDC, depending on source capabilities</p><p>CoCo can help implement and troubleshoot these patterns, but it does not automatically make every source production-ready.</p><p><strong>Rahul:</strong><br>“Can it generate the Bronze layer?”</p><p><strong>Anita:</strong><br>“Yes. Ask it to generate Bronze DDL using the actual source schemas. For example:”</p><pre>Create Bronze tables for patient, claims, laboratory results,<br>and hospital encounters.</pre><pre>Requirements:<br>- Preserve source data types where practical.<br>- Include ingestion timestamp.<br>- Include source system.<br>- Include batch ID.<br>- Include source file name where applicable.<br>- Include record hash for deduplication.<br>- Do not expose patient identifiers in operational logs.<br>- Generate DDL only.</pre><p>CoCo’s SQL tools can execute DDL against Snowflake, subject to the user’s permissions and approval controls.</p><h3>Conversation 3: “My Bronze layer is ready. How do I build Silver?”</h3><p><strong>Rahul:</strong><br>“Now I have the Bronze tables. But healthcare data is messy.</p><p>Patient names are inconsistent. Dates have different formats. Claims contain duplicates. Lab results have missing values.</p><p>I need to create a clean Silver layer.”</p><p><strong>Anita:</strong><br>“Ask CoCo to inspect the actual Bronze data before generating transformations.”</p><h3>Prompt</h3><pre>Inspect the Bronze healthcare tables.</pre><pre>Identify:<br>1. Duplicate patient records<br>2. Invalid dates<br>3. Missing patient keys<br>4. Invalid claim amounts<br>5. Laboratory results with missing test codes<br>6. Inconsistent source-system values</pre><pre>Generate:<br>- Data quality profiling SQL<br>- Silver transformation SQL<br>- Rejected-record handling<br>- Data quality audit tables<br>- Recommended tests</pre><pre>Do not modify Bronze data.</pre><p><strong>Rahul:</strong><br>“Will it understand healthcare business rules?”</p><p><strong>Anita:</strong><br>“Only if you provide them. For example, CoCo cannot reliably infer your organization’s definition of an active patient or a valid claim. You must provide those business rules.”</p><h3>Example Silver transformation</h3><pre>CREATE OR REPLACE VIEW SILVER.VALID_CLAIMS AS<br>SELECT<br>    CLAIM_ID,<br>    PATIENT_ID,<br>    CLAIM_DATE,<br>    CLAIM_AMOUNT,<br>    CLAIM_STATUS,<br>    SOURCE_SYSTEM,<br>    INGESTION_TIMESTAMP<br>FROM BRONZE.CLAIMS<br>WHERE CLAIM_ID IS NOT NULL<br>  AND PATIENT_ID IS NOT NULL<br>  AND CLAIM_AMOUNT &gt;= 0;</pre><p>This is a simplified illustration, not a complete healthcare claims validation framework.</p><p>CoCo can help generate more comprehensive transformations, tests, and dbt models based on your actual schema and requirements. Its code mode is specifically intended for software engineering tasks such as writing and debugging dbt projects. (<a href="https://docs.snowflake.com/en/user-guide/cortex-code/code-mode?utm_source=chatgpt.com">Snowflake Documentation</a>)</p><h3>Conversation 4: “How do I create secure healthcare KPIs?”</h3><p><strong>Rahul:</strong><br>“Now comes the most important part.</p><p>The business wants:</p><ul><li>Total patients.</li><li>Monthly claims.</li><li>Average claim amount.</li><li>Readmission rate.</li><li>Laboratory test volume.</li></ul><p>But patient information is sensitive. I cannot expose everything to every Power BI user.”</p><p><strong>Anita:</strong><br>“Exactly. This is where a Snowflake Architect must separate <strong>data engineering, semantic modeling, and security</strong>.</p><p>CoCo can help you create the SQL and security objects, but you must define who is allowed to see which data.”</p><h3>Prompt</h3><pre>Design a secure Gold layer for healthcare analytics.</pre><pre>Business users need:<br>- Total active patients<br>- Monthly claim count<br>- Total claim amount<br>- Average claim amount<br>- Readmission rate<br>- Laboratory test volume</pre><pre>Security requirements:<br>- Analysts should not see raw patient identifiers.<br>- Regional users should see only their assigned region.<br>- Finance users should see financial KPIs.<br>- Clinical users should see approved clinical metrics.<br>- Power BI should consume governed views.</pre><pre>Create:<br>1. Gold KPI views<br>2. Role and privilege design<br>3. Row access policy where appropriate<br>4. Masking policy where appropriate<br>5. Recommended secure reporting model<br>6. Validation queries</pre><p><strong>Rahul:</strong><br>“Can CoCo create the masking policies and row access policies?”</p><p><strong>Anita:</strong><br>“It can help generate and validate them. But do not assume that generated security code is correct simply because it compiles.</p><p>For example, the business may decide that a regional analyst can see only data for Odisha. You need a mapping table and a policy that enforces that rule.”</p><h3>Simplified row access example</h3><pre>CREATE OR REPLACE ROW ACCESS POLICY<br>GOLD.REGION_ACCESS_POLICY<br>AS (REGION_CODE VARCHAR)<br>RETURNS BOOLEAN -&gt;<br>    EXISTS (<br>        SELECT 1<br>        FROM SECURITY.USER_REGION_ACCESS<br>        WHERE USER_NAME = CURRENT_USER()<br>          AND REGION_CODE = REGION_CODE<br>    );</pre><p>This is illustrative only. The policy must be tested carefully, and production implementations should avoid ambiguous column references and use a properly designed authorization model.</p><p><strong>Anita:</strong><br>“Also remember: Gold is not automatically secure because it is Gold. Security is enforced through Snowflake privileges, masking policies, row access policies, secure views, and the reporting connection’s permissions.”</p><h3>Conversation 5: “How do I expose Gold to Power BI?”</h3><p><strong>Rahul:</strong><br>“Now I have secure Gold views. How do I make them available to Power BI?”</p><p><strong>Anita:</strong><br>“CoCo can help you create the reporting model, but Power BI connectivity and semantic modeling still need to be configured.”</p><p>The target is:</p><pre>Snowflake Gold Views<br>        |<br>        v<br>Power BI Semantic Model<br>        |<br>        v<br>Power BI Reports</pre><h3>Prompt to CoCo</h3><pre>Our Snowflake Gold layer contains secure healthcare KPI views.</pre><pre>Help me prepare the Power BI reporting model.</pre><pre>Requirements:<br>- Use only approved Gold views.<br>- Do not expose raw patient identifiers.<br>- Define relationships between KPI views and dimensions.<br>- Recommend import versus DirectQuery based on the workload.<br>- Provide sample SQL validation queries.<br>- Document the Snowflake roles required by the reporting connection.<br>- Generate a data dictionary for the reporting team.</pre><p><strong>Anita:</strong><br>“CoCo can help you build the SQL, documentation, and even data applications. But the Power BI semantic model, refresh configuration, and final report publication are still part of your reporting team’s workflow.”</p><p>Snowflake CoCo supports data engineering and data application workflows, while Power BI remains the reporting platform in this example.</p><h3>Conversation 6: “My pipeline failed at 2 AM”</h3><p><strong>Rahul:</strong><br>“Everything worked yesterday. But this morning, the claims ingestion failed. The Silver model also failed. The Gold dashboard is showing yesterday’s data.”</p><p><strong>Anita:</strong><br>“This is where CoCo becomes particularly useful. Instead of searching through multiple SQL files and logs manually, ask it to investigate the failure.”</p><h3>Prompt</h3><pre>Investigate why the healthcare claims pipeline failed.</pre><pre>Context:<br>- Bronze claims ingestion completed yesterday.<br>- Silver claims transformation failed this morning.<br>- Gold KPI refresh did not complete.</pre><pre>Please:<br>1. Inspect recent query history and task history.<br>2. Identify the first failure.<br>3. Check whether the source schema changed.<br>4. Review the relevant SQL and dbt model.<br>5. Explain the root cause.<br>6. Propose a fix.<br>7. Do not modify production objects until I approve.</pre><p><strong>Rahul:</strong><br>“Will it actually inspect Snowflake query history?”</p><p><strong>Anita:</strong><br>“CoCo has Snowflake tools for querying data and investigating database objects. With the appropriate permissions, it can help inspect query history, task execution, and relevant objects. It can also work with local project files and dbt code.” (<a href="https://docs.snowflake.com/en/user-guide/cortex-code/tools?utm_source=chatgpt.com">Snowflake Documentation</a>)</p><p>The important distinction is that CoCo helps investigate and fix the issue. It does not replace your production monitoring, alerting, or incident-management process.</p><h3>The complete CoCo workflow</h3><p>Here is how I would structure the project as a Snowflake Architect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*A9kfwbNVvsZzaxap" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DDU-tA9rkJbv9Ycu" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hTj17mOZgQTLq7jM" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W-NkBODCqB__ORPd" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8NuhPhpiKH_Lz91F" /></figure><p>PhaseCoCo’s roleEngineer’s responsibilityDiscoveryFind objects, inspect schemas, understand existing environmentConfirm requirements and accessArchitecturePropose ingestion, Bronze, Silver, Gold designApprove target architectureDevelopmentGenerate SQL, Python, dbt, tests, and documentationReview code and business logicIngestionHelp build and troubleshoot ingestion objectsConfigure production orchestrationData qualityGenerate profiling and validation queriesDefine acceptable quality thresholdsSecurityDraft roles, masking policies, row access policiesApprove and validate securityGoldCreate KPI views and semantic model definitionsValidate KPI definitionsReportingPrepare reporting SQL and documentationConfigure Power BIOperationsInvestigate failures and optimize queriesOwn monitoring, release, and incident response</p><h3>What makes this different from a normal SQL chatbot?</h3><p><strong>Rahul:</strong><br>“So the real benefit is not just SQL generation?”</p><p><strong>Anita:</strong><br>“Correct. The difference is the workflow.”</p><p>A basic SQL chatbot might answer:</p><blockquote><em>“Write a SQL query to calculate monthly claims.”</em></blockquote><p>CoCo can work with a broader engineering objective:</p><blockquote><em>“Inspect the healthcare data platform, understand the existing claims tables, identify the relevant KPI definition, generate the Gold view, validate it against the business rule, and prepare the changes for review.”</em></blockquote><p>The latter requires understanding the project context, database objects, code, and execution workflow. CoCo’s Snowflake integration and local project capabilities are designed to support these kinds of multi-step tasks.</p><h3>A practical starting point for your next project</h3><p>If you want to demonstrate this in a real POC, I would start with a small healthcare dataset:</p><p><strong>Sources:</strong> Patient CSV + Claims CSV + Laboratory CSV<br><strong>Target:</strong> Snowflake Bronze → Silver → Gold<br><strong>Reporting:</strong> Power BI<br><strong>AI assistant:</strong> CoCo CLI or CoCo Desktop</p><p>Start with this prompt:</p><pre>Act as a Senior Snowflake Data Engineering Architect.</pre><pre>Build a healthcare medallion architecture POC.</pre><pre>Sources:<br>- Patient CSV<br>- Claims CSV<br>- Laboratory CSV</pre><pre>Target:<br>- Bronze raw tables<br>- Silver cleaned tables<br>- Gold KPI views<br>- Secure reporting access</pre><pre>Requirements:<br>- Metadata-driven ingestion<br>- Data quality checks<br>- Audit logging<br>- RBAC and masking design<br>- Power BI-ready Gold views</pre><pre>First create an implementation plan.<br>Then generate the SQL and project files.<br>Do not execute production changes without approval.</pre><p>CoCo is available through Snowsight, Desktop, and CLI experiences. For a hands-on engineering workflow with local SQL or dbt files, the CLI or Desktop is particularly relevant.</p><p><strong>The key message for a Data Architect:</strong> CoCo helps you move from <em>“I need to write this SQL”</em> to <em>“I need to deliver this governed data engineering outcome.”</em> The architecture, business rules, security design, and production accountability still belong to the engineering team.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=efffcaba8d81" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Snowflake AIM: The Fastest Path from Legacy Data Chaos to an AI-Ready Enterprise]]></title>
            <link>https://medium.com/@lakhansd777/snowflake-aim-the-fastest-path-from-legacy-data-chaos-to-an-ai-ready-enterprise-4c590bb82d1c?source=rss-4c54a1289c4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c590bb82d1c</guid>
            <dc:creator><![CDATA[Lakhan Dhanashree]]></dc:creator>
            <pubDate>Wed, 09 Sep 2026 02:58:27 GMT</pubDate>
            <atom:updated>2026-09-09T02:58:27.504Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fUvC3Y_NrXViMpeLuW4i_g.png" /></figure><h3>It’s 2:30 AM. The Database Renewal Notice Just Landed in Your Inbox.</h3><p>Imagine you’re the CIO of a large global enterprise.</p><p>Your organization relies on a 15-year-old data warehouse platform. Thousands of reports, dashboards, ETL jobs, and business-critical applications depend on it every single day.</p><p>Then reality hits.</p><ul><li>The platform renewal cost has increased by 35%.</li><li>Business teams are asking for GenAI copilots.</li><li>Data scientists want real-time access to enterprise data.</li><li>Executives demand AI-powered decision intelligence.</li><li>Regulators expect stronger governance and lineage.</li><li>Competitors are already leveraging Agentic AI and autonomous analytics.</li></ul><p>Meanwhile, your data estate looks like a maze:</p><ul><li>Teradata running critical reports</li><li>SQL Server supporting operational analytics</li><li>Redshift powering cloud workloads</li><li>Spark jobs scattered across environments</li><li>Hundreds of undocumented ETL pipelines</li><li>Data duplicated across multiple platforms</li></ul><p>Everyone agrees modernization is necessary.</p><p>Nobody wants to be responsible for breaking the business while doing it.</p><p>And that’s where most organizations get stuck.</p><p>Not because they lack vision.</p><p>Not because they lack budget.</p><p>But because the fear of migration risk is greater than the excitement of innovation.</p><p>For years, enterprise modernization projects have followed the same painful pattern:</p><p><em>Months of assessment. Years of migration. Endless testing. Massive consulting efforts. Millions of dollars.</em></p><p>By the time organizations finish migrating, the technology landscape has already evolved.</p><p>Today, in the age of Generative AI, Agentic AI, Data Products, and Intelligent Applications, enterprises need a radically different approach.</p><p>They don’t just need migration.</p><p>They need AI-driven modernization.</p><p>This is exactly why Snowflake introduced Snowflake AIM (AI-Driven Modernization and Virtualization), an intelligent platform designed to help organizations accelerate migrations, modernize legacy ecosystems, virtualize workloads, and become AI-ready significantly faster. <a href="https://docs.snowflake.com/en/migrations/snowflake-aim">[</a><a href="http://docs.snowflake.com/">docs.snowflake.com</a><a href="https://docs.snowflake.com/en/migrations/snowflake-aim">]</a></p><h3>Why This Matters Right Now</h3><p>The AI revolution has created an uncomfortable truth:</p><blockquote>Most organizations are trying to build next-generation AI solutions on top of yesterday’s data architecture.</blockquote><p>The success of AI initiatives is no longer determined by the sophistication of the AI model.</p><p>It’s determined by the quality, accessibility, governance, and modernization of the data supporting it.</p><p>Before organizations can unlock GenAI, AI Agents, Enterprise Search, Intelligent Data Products, or Autonomous Analytics, they must first solve the modernization challenge.</p><p>And that’s precisely where Snowflake AIM changes the game.</p><p><em>Why are so many organizations still spending millions maintaining legacy data platforms while trying to keep up with the AI revolution?</em></p><p>Let’s be honest.</p><p>Most enterprises today are stuck in a difficult position.</p><p>They want to build GenAI applications, deploy AI agents, accelerate analytics, and unlock business value from data. Yet their data ecosystem often looks like a patchwork of legacy technologies:</p><ul><li>Teradata warehouses approaching expensive renewals</li><li>SQL Server environments growing beyond capacity</li><li>Redshift, BigQuery, Databricks, and Synapse estates spread across clouds</li><li>Thousands of ETL jobs nobody fully understands</li><li>Complex Spark code bases maintained by a handful of specialists</li><li>Data silos creating governance nightmares</li><li>Growing pressure to become AI-ready</li></ul><p>The result?</p><p>Organizations spend more time maintaining infrastructure than creating innovation.</p><p>The biggest challenge isn’t building AI.</p><p>It’s modernizing the data ecosystem that AI depends on.</p><p>This is exactly where Snowflake AIM (AI-Driven Modernization and Virtualization) enters the picture.</p><h3>The Real Enterprise Problem</h3><p>Over the last few years, I have seen many organizations facing the same questions:</p><blockquote>“How do we migrate from Teradata without rewriting everything?”</blockquote><blockquote>“Can we modernize SQL Server and Redshift workloads without disrupting business operations?”</blockquote><blockquote>“How do we move thousands of ETL workflows into a modern Data Cloud?”</blockquote><blockquote>“Can we reduce migration timelines from years to months?”</blockquote><blockquote>“How do we become AI-ready without another massive transformation program?”</blockquote><p>Traditionally, migration projects are painful.</p><p>They involve:</p><ul><li>Months of assessment</li><li>Manual code conversion</li><li>Multiple testing cycles</li><li>Repeated business validation</li><li>Downtime risks</li><li>High consulting costs</li><li>Significant project uncertainty</li></ul><p>Many modernization initiatives fail not because of technology limitations, but because the risk outweighs the expected reward.</p><p>Organizations become trapped in what many call technical debt paralysis.</p><h3>What is Snowflake AIM?</h3><p>Snowflake AIM (AI-Driven Modernization and Virtualization) is an AI-powered modernization platform designed to help organizations migrate, modernize, and virtualize their data ecosystem on Snowflake. It automates major portions of migration projects, including code analysis, dependency discovery, conversion recommendations, workload modernization, and virtualization.</p><p><a href="https://docs.snowflake.com/en/migrations/snowflake-aim">https://docs.snowflake.com/en/migrations/snowflake-aim</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*U9DFLyHVsJIOCNzJfNiE3g.png" /></figure><p>In simple terms:</p><p>Instead of spending years manually rebuilding legacy data platforms, Snowflake AIM uses AI-assisted migration intelligence to accelerate transformation while reducing risk.</p><p>Think of it as an intelligent modernization agent for your entire data estate.</p><h3>Why Snowflake AIM is Creating Buzz</h3><p>The data industry is entering a new era dominated by:</p><ul><li>AI-Native Architectures</li><li>Agentic AI</li><li>Data Products</li><li>Open Data Lakehouse</li><li>Data Mesh</li><li>Semantic Layers</li><li>Real-Time Analytics</li><li>Multi-Cloud Strategies</li><li>Iceberg Ecosystems</li><li>Enterprise AI</li></ul><p>The problem?</p><p>Most legacy platforms were never designed for these capabilities.</p><p>Snowflake AIM bridges that gap.</p><p>It combines:</p><p>✅ AI-Assisted Migration</p><p>✅ Intelligent Code Conversion</p><p>✅ Workload Modernization</p><p>✅ Data Virtualization</p><p>✅ Governance and Lineage</p><p>✅ Snowpark Transformation</p><p>✅ Cloud-Native Architecture</p><p>✅ AI-Ready Data Foundations</p><h3>How Snowflake AIM Works</h3><h3>1. Discover &amp; Assess</h3><p>AI analyzes:</p><ul><li>Database objects</li><li>SQL code</li><li>Stored procedures</li><li>ETL pipelines</li><li>Spark workloads</li><li>Dependencies</li><li>Usage patterns</li></ul><p>Many organizations have no complete inventory of their environments.</p><p>AIM immediately provides visibility.</p><h3>2. Intelligent Modernization</h3><p>Snowflake AIM can help modernize:</p><ul><li>SQL Server</li><li>Teradata</li><li>Redshift</li><li>BigQuery</li><li>Synapse</li><li>Oracle</li><li>PostgreSQL</li><li>Databricks SQL</li></ul><p>Workloads can be translated into Snowflake-native patterns with AI assistance. <a href="https://docs.snowflake.com/en/migrations/snowflake-aim">[</a><a href="http://docs.snowflake.com/">docs.snowflake.com</a><a href="https://docs.snowflake.com/en/migrations/snowflake-aim">]</a></p><h3>3. Spark to Snowpark Transformation</h3><p>One of the most exciting capabilities is Spark modernization.</p><p>Organizations running thousands of lines of:</p><ul><li>Python Spark</li><li>Scala Spark</li><li>Java Spark</li></ul><p>can analyze compatibility and generate Snowpark-oriented modernization paths.</p><p>This dramatically reduces manual effort.</p><h3>4. Virtualization Without Rewriting</h3><p>Perhaps the most disruptive capability.</p><p>Instead of immediately rewriting applications, organizations can virtualize workloads and continue operations while modernizing over time.</p><p>Benefits include:</p><ul><li>Zero downtime approaches</li><li>Reduced business risk</li><li>Faster cloud adoption</li><li>Incremental modernization</li></ul><p>For many enterprises, this is the difference between approving or rejecting a migration initiative.</p><h3>Real Business Use Cases</h3><h3>1. Banking &amp; Financial Services</h3><h3>Challenge</h3><p>Large banking platforms often run legacy Teradata environments supporting:</p><ul><li>Risk analytics</li><li>Fraud detection</li><li>Customer insights</li><li>Regulatory reporting</li></ul><p>Migration timelines can stretch beyond two years.</p><h3>AIM Advantage</h3><ul><li>Faster assessment</li><li>Automated code analysis</li><li>Reduced migration risk</li><li>AI-assisted modernization</li></ul><h3>Business Outcome</h3><p>Quicker transition to AI-powered fraud detection and customer intelligence.</p><h3>2. Retail &amp; E-Commerce</h3><h3>Challenge</h3><p>Data spread across:</p><ul><li>POS systems</li><li>Supply chain platforms</li><li>CRM solutions</li><li>Loyalty applications</li></ul><p>Different teams create duplicated pipelines and inconsistent reporting.</p><h3>AIM Advantage</h3><ul><li>Centralized modernization</li><li>Reduced data duplication</li><li>Intelligent data discovery</li><li>AI-ready architecture</li></ul><h3>Business Outcome</h3><p>Personalized recommendations, dynamic pricing, and improved customer experience.</p><h3>3. Telecommunications</h3><h3>Challenge</h3><p>Petabytes of network, usage, and operational data distributed across legacy environments.</p><h3>AIM Advantage</h3><ul><li>Large-scale migration automation</li><li>Virtualized access patterns</li><li>Accelerated modernization</li></ul><h3>Business Outcome</h3><p>Predictive maintenance, network optimization, and AI-driven operations.</p><h3>4. Manufacturing</h3><h3>Challenge</h3><p>Factories generate data from:</p><ul><li>IoT devices</li><li>Sensors</li><li>ERP systems</li><li>MES applications</li></ul><p>Typically stored across disconnected platforms.</p><h3>AIM Advantage</h3><ul><li>Unified data ecosystem</li><li>Faster ingestion modernization</li><li>AI-driven analytics foundation</li></ul><h3>Business Outcome</h3><p>Predictive maintenance, quality improvement, and operational efficiency.</p><h3>5. Healthcare</h3><h3>Challenge</h3><p>Highly regulated environments managing:</p><ul><li>Clinical systems</li><li>Claims platforms</li><li>Electronic health records</li></ul><p>while maintaining governance and compliance.</p><h3>AIM Advantage</h3><ul><li>Secure modernization</li><li>Governance consistency</li><li>Controlled migration paths</li></ul><h3>Business Outcome</h3><p>Better patient insights and faster clinical analytics.</p><h3>Why Enterprises Care</h3><p>Snowflake AIM is not simply a migration tool.</p><p>It is an acceleration platform.</p><p>Organizations are looking for:</p><ul><li>Faster Time-to-Value</li><li>Lower Total Cost of Ownership</li><li>AI Readiness</li><li>Reduced Risk</li><li>Business Continuity</li><li>Improved Governance</li><li>Modern Data Architecture</li></ul><p>AIM addresses all of these simultaneously.</p><h3>The AI Multiplier Effect</h3><p>Here’s where things get really interesting.</p><p>Modernization used to be the final destination.</p><p>Today, modernization is merely the starting point.</p><p>Once data lands on a modern Snowflake architecture, organizations can rapidly leverage:</p><ul><li>Snowflake Cortex AI</li><li>Enterprise Search</li><li>AI Agents</li><li>Semantic Models</li><li>GenAI Applications</li><li>Natural Language Analytics</li><li>Autonomous Data Products</li><li>Agentic Workflows</li></ul><p>In other words:</p><p>Snowflake AIM helps organizations move from data migration to AI transformation.</p><h3>Final Thoughts</h3><p>The enterprise data landscape is changing at incredible speed.</p><p>The winners of the next decade will not necessarily be the organizations with the most data.</p><p>They will be the organizations that can modernize, govern, and activate their data fastest.</p><p>Snowflake AIM represents a major shift in how modernization projects are delivered.</p><p>Instead of years of manual effort, organizations can increasingly rely on AI-assisted discovery, migration intelligence, workload modernization, and virtualization to accelerate transformation while controlling risk.</p><p>The future is no longer about moving data from Platform A to Platform B.</p><p>The future is about creating an AI-ready, governed, scalable data ecosystem capable of powering GenAI, Agentic AI, advanced analytics, and intelligent decision-making.</p><p>And that is exactly where Snowflake AIM shines.</p><h3>Key Takeaway</h3><p>Snowflake AIM is not just helping enterprises migrate faster. It is helping them modernize smarter, reduce risk, lower costs, and build the foundation for the next generation of AI-powered business innovation.</p><p>#Snowflake #SnowflakeAIM #DataCloud #AI #GenAI #AgenticAI #DataEngineering #CloudMigration #DataModernization #Snowpark #DataTransformation #EnterpriseAI #DigitalTransformation #Analytics #Innovation #FutureOfData #DataArchitecture #SnowflakeSummit #AIDrivenTransformation #ModernDataPlatform</p><p><a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Snowflake</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Accenture</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Veenu Yadav</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Prasad Revalkar</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Somen Swain</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Teresa Tung</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Elsa Mayer</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Vivek Luthra</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Amilee Alesna</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Shubhangi Singh</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Ryan Nunez</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Jeff Wojnar</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Emma MacGregor</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Phillip Lim</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Ethan Lee</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Tamanna Godara</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Aba Micah</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Dash DesAI</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Marcella Larsen</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Modem</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">David Simmons</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Priti Sawant</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Leo Mathew</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Richard James</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Pallab Deb</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Avishek Ghoshal</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Jordan Greig</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Fred Rushmer</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">James Butler</a> <a href="https://www.linkedin.com/feed/update/urn:li:share:7502947942631178240/">Elli Pennanen</a> <a href="https://medium.com/u/1429b5c40410">Riya Khandelwal</a> <a href="https://medium.com/u/e56d5e16d8b9">Somen Swain</a> <a href="https://medium.com/u/b4ce2903617a">Rajiv Gupta</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c590bb82d1c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Future of Data Platforms: 7 Architecture Patterns Every Cloud Data Leader Should Master]]></title>
            <link>https://medium.com/@lakhansd777/the-future-of-data-platforms-7-architecture-patterns-every-cloud-data-leader-should-master-769d798d9a29?source=rss-4c54a1289c4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/769d798d9a29</guid>
            <category><![CDATA[architectpattern]]></category>
            <dc:creator><![CDATA[Lakhan Dhanashree]]></dc:creator>
            <pubDate>Thu, 03 Sep 2026 10:42:17 GMT</pubDate>
            <atom:updated>2026-09-03T10:42:17.933Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Why some data platforms power AI innovation while others become expensive technical debt.</em></p><p><strong>The Harsh Truth About Data Architecture in 2026</strong></p><p>A few years ago, building a data platform was relatively straightforward.</p><p>Create a data warehouse.</p><p>Run nightly ETL jobs.</p><p>Build dashboards.</p><p>Go home.</p><p>Fast forward to 2026, and the game has completely changed.</p><p>Today, the same platform must support:</p><p>✅ Real-time analytics<br> ✅ GenAI applications<br> ✅ RAG architectures<br> ✅ AI agents<br> ✅ Streaming workloads<br> ✅ Self-service analytics<br> ✅ Regulatory compliance<br> ✅ Cost governance<br> ✅ Multi-cloud strategies</p><p>And if your architecture cannot support AI workloads, it is already becoming obsolete.</p><p>Organizations are no longer asking:</p><p><em>“Can we use AI?”</em></p><p>They’re asking:</p><p><em>“Why isn’t AI delivering business value yet?”</em></p><p>The answer is almost always the same:</p><p><strong>Bad data architecture.</strong></p><p>Modern AI is only as smart as the architecture feeding it.</p><p>According to multiple 2026 industry reports, Lakehouse, Data Mesh, Data Contracts, Event-Driven Platforms, and AI-Native Architectures have become dominant patterns for enterprise-scale platforms. Meanwhile, data products, semantic layers, and metadata-driven engineering are emerging as critical enablers for AI consumption.</p><p>Let’s explore the architecture patterns every architect and data engineer should master before they become tomorrow’s technical debt.</p><p><strong>1. Lakehouse Architecture</strong></p><p><strong>The New Foundation of Enterprise Data Platforms</strong></p><p>For more than a decade, companies maintained separate systems:</p><ul><li>Data Lake for storage</li><li>Data Warehouse for analytics</li></ul><p>This created:</p><ul><li>Duplicate data</li><li>Higher costs</li><li>Complex pipelines</li><li>Governance nightmares</li></ul><p>The Lakehouse changed everything.</p><p>A Lakehouse combines the best of both worlds:</p><ul><li>Data lakes for scalability</li><li>Warehouses for performance and governance</li></ul><p>One platform.</p><p>One copy of data.</p><p>One governance model.</p><p><strong>Why It Matters in 2026</strong></p><p>AI, Machine Learning, BI, analytics, and data science all need access to the same datasets.</p><p>A Lakehouse allows them to coexist efficiently.</p><p><strong>Real Business Use Case</strong></p><p><strong>Healthcare Provider</strong></p><p>A healthcare organization stores:</p><ul><li>Patient records</li><li>Medical images</li><li>Clinical notes</li><li>IoT monitoring data</li></ul><p>Rather than maintaining separate systems for analytics and AI models, everything resides in a Lakehouse.</p><p>Result:</p><ul><li>Faster diagnostics</li><li>Unified governance</li><li>Reduced infrastructure costs</li></ul><p><strong>2. Medallion Architecture</strong></p><p><strong>The Secret Behind Clean Data at Scale</strong></p><p>If Lakehouse is the foundation, Medallion Architecture is the discipline.</p><p>Data flows through multiple layers:</p><p>🥉 Bronze = Raw Data</p><p>🥈 Silver = Cleaned &amp; Validated Data</p><p>🥇 Gold = Business-Ready Data</p><p>This pattern is becoming the default building block of enterprise platforms because it makes data quality traceable and scalable. Enterprise data programs increasingly use Bronze, Silver, and Gold layers as the foundation for production-grade platforms and AI-ready data products.</p><p><strong>Real Business Use Case</strong></p><p><strong>Retail Giant</strong></p><p>Daily ingestion:</p><ul><li>POS transactions</li><li>E-commerce orders</li><li>Customer interactions</li></ul><p>Raw data enters Bronze.</p><p>Validated records flow to Silver.</p><p>Business KPIs are published in Gold.</p><p>Outcome:</p><ul><li>Consistent reporting</li><li>Trusted AI recommendations</li><li>Improved governance</li></ul><p><strong>3. Event-Driven Architecture</strong></p><p><strong>Because Business Happens in Real Time</strong></p><p>Customers do not wait for nightly batch jobs.</p><p>Neither should your architecture.</p><p>Event-driven systems process information the moment it happens.</p><p>Powered by:</p><ul><li>Kafka</li><li>Event Streams</li><li>Change Data Capture (CDC)</li><li>Streaming platforms</li></ul><p>This architecture is critical whenever latency directly impacts revenue. Event-driven platforms are widely used for fraud detection, personalization, and other latency-sensitive business processes.</p><p><strong>Real Business Use Case</strong></p><p><strong>Banking Fraud Detection</strong></p><p>A credit card transaction arrives.</p><p>Within milliseconds:</p><ul><li>Risk scoring executes</li><li>Fraud patterns are checked</li><li>Alerts trigger automatically</li></ul><p>Without event-driven architecture?</p><p>Fraud gets detected hours later.</p><p>Too late.</p><p><strong>4. Data Mesh</strong></p><p><strong>When Centralized Teams Become the Bottleneck</strong></p><p>Many enterprises struggle because one central data team owns everything.</p><p>Requests pile up.</p><p>Business teams wait.</p><p>Innovation slows.</p><p>Data Mesh introduces a different philosophy:</p><p><strong>The domain owns its data.</strong></p><p>Marketing owns marketing data.</p><p>Finance owns finance data.</p><p>Supply Chain owns supply chain data.</p><p>The platform team provides:</p><ul><li>Governance</li><li>Infrastructure</li><li>Security</li><li>Standards</li></ul><p>While domains own the actual data products. Hybrid Data Mesh models are increasingly adopted where central platforms provide governance and self-service capabilities while domains own data outcomes.</p><p><strong>Real Business Use Case</strong></p><p><strong>Global Manufacturing Enterprise</strong></p><p>Over 40 business units worldwide.</p><p>Instead of routing every request through a central team:</p><ul><li>Plants own operational datasets</li><li>Finance owns revenue products</li><li>Procurement owns supplier products</li></ul><p>Result:</p><ul><li>Faster innovation</li><li>Better accountability</li><li>Reduced engineering bottlenecks</li></ul><p><strong>5. Metadata-Driven Architecture</strong></p><p><strong>Stop Building 100 Pipelines Manually</strong></p><p>One of the biggest mistakes data teams make:</p><p>Creating custom pipelines for every source.</p><p>This does not scale.</p><p>Metadata-driven architectures separate logic from execution.</p><p>Instead of coding:</p><p>Customer Pipeline</p><p>Orders Pipeline</p><p>Products Pipeline</p><p>Inventory Pipeline</p><p>Define transformations in metadata.</p><p>Generate pipelines automatically.</p><p>The pattern becomes highly valuable as organizations scale beyond dozens of ingestion pipelines.</p><p><strong>Real Business Use Case</strong></p><p><strong>Insurance Company</strong></p><p>Managing 500+ source systems.</p><p>Rather than maintaining thousands of lines of ETL code:</p><ul><li>Metadata defines mappings</li><li>Framework generates pipelines</li><li>Governance becomes automated</li></ul><p>Result:</p><ul><li>70% faster onboarding</li><li>Reduced maintenance effort</li><li>Consistent standards</li></ul><p><strong>6. Data Contracts</strong></p><p><strong>The Missing Piece Most Teams Ignore</strong></p><p>Imagine APIs without contracts.</p><p>Chaos.</p><p>The same applies to data.</p><p>Data Contracts create formal agreements between:</p><p>Data Producers ↔ Data Consumers</p><p>Defining:</p><ul><li>Schema</li><li>Quality expectations</li><li>Ownership</li><li>SLAs</li><li>Versioning</li></ul><p>In 2026, data contracts are increasingly viewed as a hallmark of mature decentralized data platforms.</p><p><strong>Real Business Use Case</strong></p><p><strong>E-Commerce Platform</strong></p><p>Customer Service depends on Order Data.</p><p>When Order teams modify schemas unexpectedly:</p><p>Everything breaks.</p><p>With Data Contracts:</p><ul><li>Changes are validated</li><li>Breaking modifications are prevented</li><li>Downstream teams remain unaffected</li></ul><p><strong>7. AI-Native Data Architecture</strong></p><p><strong>The Pattern That Will Define The Next Decade</strong></p><p>This is where the industry is heading.</p><p>Traditional architectures were built for dashboards.</p><p>Modern architectures must support:</p><ul><li>LLMs</li><li>AI Agents</li><li>RAG systems</li><li>Semantic Search</li><li>Vector Databases</li></ul><p>AI workloads consume data differently than BI systems. AI-focused architectures increasingly require semantic clarity, governed data products, and support for retrieval and agent-based consumption patterns.</p><p><strong>Real Business Use Case</strong></p><p><strong>Utilities Company</strong></p><p>An operations engineer asks:</p><p><em>“Why did transformer failures increase last month?”</em></p><p>Instead of searching dashboards:</p><p>An AI Agent:</p><ul><li>Retrieves operational logs</li><li>Reads maintenance reports</li><li>Queries structured datasets</li><li>Generates insights instantly</li></ul><p>Minutes become seconds.</p><p><strong>The Architecture Stack Winning in 2026</strong></p><p>The most successful organizations are not choosing one pattern.</p><p>They’re combining several.</p><p><strong>Typical Modern Stack</strong></p><p>✅ Lakehouse Foundation</p><p>✅ Medallion Layers</p><p>✅ Event-Driven Processing</p><p>✅ Data Contracts</p><p>✅ Metadata-Driven Pipelines</p><p>✅ Data Mesh Governance</p><p>✅ AI-Native Consumption Layer</p><p>This combination creates a platform capable of serving analytics, machine learning, and AI applications from a unified, governed foundation.</p><p><strong>Final Thoughts</strong></p><p>Being a Data Architect in 2026 is no longer about designing databases.</p><p>It’s about designing platforms that can power:</p><p>🚀 Agentic AI<br> 🚀 Real-Time Intelligence<br> 🚀 Enterprise Analytics<br> 🚀 Autonomous Decisioning<br> 🚀 Data Products<br> 🚀 Multi-Modal AI Workloads</p><p>The architects who understand only ETL will struggle.</p><p>The architects who master <strong>Lakehouse, Medallion, Data Mesh, Event-Driven Systems, Data Contracts, Metadata-Driven Engineering, and AI-Native Architecture</strong> will define the next generation of enterprise platforms.</p><p><strong>Remember:</strong></p><p>Data is the new oil.</p><p>Data products are the new currency.</p><p>But architecture is the refinery that creates business value.</p><p><strong>#DataArchitecture #DataEngineering #Snowflake #Databricks #DataMesh #Lakehouse #AIArchitecture #GenAI #AgenticAI #DataProducts #RAG #StreamingData #ModernDataStack #CloudArchitecture #DataGovernance #DataContracts #MetadataDriven #DataArchitect #AnalyticsEngineering #FutureOfData</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=769d798d9a29" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cortex Agents: The Future Beyond Chatbots]]></title>
            <link>https://medium.com/@lakhansd777/cortex-agents-the-future-beyond-chatbots-2e951f95e8b2?source=rss-4c54a1289c4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e951f95e8b2</guid>
            <category><![CDATA[snowflake-cortex]]></category>
            <dc:creator><![CDATA[Lakhan Dhanashree]]></dc:creator>
            <pubDate>Wed, 02 Sep 2026 12:18:58 GMT</pubDate>
            <atom:updated>2026-09-02T12:18:58.728Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why the next wave of enterprise AI isn’t about talking to a bot — it’s about getting things done</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B3g9cyExw84xkSAerm4AMw.png" /></figure><p>Let’s be honest: we’re all a little tired of chatbots.</p><p>Not the <em>idea</em> of them the idea was always great. Ask a question, get an answer, skip the ticket queue. But most enterprise chatbots today are glorified FAQ machines wearing a GenAI costume. You ask something slightly off-script, and you’re back to “I’m sorry, I didn’t understand that” the 2015 experience, just with a fancier font.</p><p>Meanwhile, something much more interesting has been quietly happening under the hood of platforms like Snowflake: <strong>Cortex Agents</strong>. And if you’re still thinking of this as “chatbot 2.0,” you’re underselling it by a mile.</p><p>This isn’t about answering questions anymore. It’s about <em>reasoning</em>, <em>retrieving</em>, <em>deciding</em>, and <em>acting</em> — autonomously, across structured and unstructured data, with guardrails, observability, and governance baked in. Welcome to the agentic AI era.</p><h3>From “Ask a Question” to “Get It Done”</h3><p>Here’s the mental shift that matters: a chatbot is a <strong>conversation interface</strong>. An agent is a <strong>reasoning system with tools</strong>.</p><p>A traditional chatbot retrieves an answer and hands it to you. A Cortex Agent orchestrates a <em>chain of decisions</em> — it figures out what you actually need, decides which tool or data source can answer it (a semantic model? a document store? a live SQL query?), executes that step, evaluates whether the result is good enough, and only then responds — or takes the next action.</p><p>That’s the difference between a search bar and a junior analyst who never sleeps.</p><p>Under the hood, this is powered by a stack of buzzwords that are, for once, actually earning their hype:</p><ul><li><strong>Retrieval-Augmented Generation (RAG)</strong> — grounding responses in real enterprise data instead of the model’s imagination</li><li><strong>Tool-calling / function orchestration</strong> — the agent decides <em>which</em> tool to use, not a hardcoded flowchart</li><li><strong>Multi-agent orchestration</strong> — specialized sub-agents (a SQL agent, a document agent, a search agent) collaborating on one task</li><li><strong>AI Observability</strong> — because “the model said so” is not an audit trail</li><li><strong>Semantic layers</strong> — turning “how did Q3 revenue trend in APAC” into governed, accurate SQL instead of a hallucinated guess</li><li><strong>Hallucination mitigation via grounding</strong> — the unglamorous, unbelievably important part nobody puts on a slide</li></ul><p>Put together, this is what people mean when they say <strong>agentic AI</strong>. Not a smarter chatbot. A system with judgment.</p><h3>Why This Matters More Than the Chatbot Hype Cycle</h3><p>Every enterprise spent 2023–2024 bolting a chatbot onto their knowledge base and calling it “AI transformation.” Most of those projects quietly died in pilot purgatory. Why?</p><p>Because a chatbot answers <em>what you ask</em>. An agent answers <em>what you actually need</em> — and increasingly, it just handles the task itself.</p><p>Cortex Agents specifically blend a few things that used to require five different vendors stitched together with duct tape:</p><ol><li><strong>Hybrid retrieval</strong> : semantic + keyword search across structured tables <em>and</em> unstructured documents (think: PDFs, contracts, filings) in one query</li><li><strong>Natural language to governed SQL</strong> no more “please write me a query,” the agent does it against your actual semantic model, respecting your row-level security</li><li><strong>Document intelligence</strong> parsing, classifying, and extracting structured data from messy real-world documents at scale</li><li><strong>Built-in governance </strong>masking, redaction, and access controls that travel <em>with</em> the data, not bolted on after</li><li><strong>Accuracy monitoring</strong> observability metrics that tell you when the agent is drifting, not just when it crashes</li></ol><p>This is the unglamorous plumbing that makes “AI you can actually trust in production” possible. Not another demo. Production.</p><h3>Real-World Use Cases (Not Hypotheticals)</h3><p>Here’s where it stops being theoretical. These are the kinds of use cases showing up right now in enterprises that have moved past the chatbot phase:</p><p><strong>1. Investment Research &amp; Portfolio Q&amp;A</strong> Instead of an analyst manually digging through prospectuses, filings, and market data, a RAG-grounded agent answers “what’s our exposure to mid-cap industrials in this fund” by pulling from structured portfolio data <em>and</em> unstructured filing documents with citations, not vibes.</p><p><strong>2. Enterprise Document Intelligence</strong> Contracts, invoices, compliance filings, statements automatically classified, parsed, and pushed into structured pipelines. What used to be a team of people doing manual data entry becomes a governed, auditable pipeline running in the background.</p><p><strong>3. Ask-Me-Anything Enterprise Knowledge Assistants</strong> Not “search our wiki” — actually reasoning across HR policy, engineering docs, and live operational data to answer a question end-to-end, escalating to a human only when confidence is genuinely low.</p><p><strong>4. Self-Service Analytics for Business Users</strong> “Show me churn by region for the last two quarters” turns into governed SQL against a semantic model-no data team bottleneck, no shadow spreadsheets, no waiting three days for a dashboard update.</p><p><strong>5. Compliance &amp; Risk Monitoring</strong> Agents scanning transactions or communications for policy violations, applying PII/PCI redaction automatically, and flagging anomalies for human review with a full audit trail, because “the AI did it” is not an answer regulators accept.</p><p><strong>6. Customer &amp; Internal Support That Actually Resolves Things</strong> Not “here’s an article that might help” an agent that checks your order status, your entitlement, your account state, and <em>takes the action</em>, only routing to a human when it genuinely needs judgment a machine shouldn’t make alone.</p><h3>The Part Nobody Wants to Talk About: Governance</h3><p>Here’s the uncomfortable truth about agentic AI: the more autonomy you give a system, the more it needs guardrails, not fewer.</p><p>The organizations getting real value out of Cortex Agents aren’t the ones that moved fastest they’re the ones that took <strong>responsible AI</strong> seriously from day one: role-based access control that the agent respects, data masking that travels with sensitive fields, observability that catches drift before it becomes a headline, and human-in-the-loop checkpoints for anything with real business consequences.</p><p>Agentic AI without governance isn’t innovation. It’s a liability with a nice UI.</p><h3>So, Is the Chatbot Dead?</h3><p>Not exactly. But it’s being demoted — from “the product” to “one interface into a much smarter system underneath.” The chat window is still how people interact. What’s changed is everything happening <em>behind</em> it: reasoning, retrieval, orchestration, and action.</p><p>The organizations that figure this out first won’t be the ones with the flashiest demo. They’ll be the ones who quietly automated the boring, expensive, error-prone work -document processing, compliance checks, analyst research -and freed their people to do the work that actually needs a human.</p><p>That’s the real story beyond the chatbot hype. Not a smarter way to talk to your data.</p><p>A smarter way to <em>act</em> on it.</p><p><em>What’s your experience been are you seeing agentic AI move past the pilot stage in your organization, or is it still stuck in demo purgatory? I’d love to hear what’s working (and what isn’t).</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e951f95e8b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>