<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Deepti Agrawal on Medium]]></title>
        <description><![CDATA[Stories by Deepti Agrawal on Medium]]></description>
        <link>https://medium.com/@deepti.agl2912?source=rss-abd297954d5f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3PoRNHZWx95Y7J6eiqzadA.jpeg</url>
            <title>Stories by Deepti Agrawal on Medium</title>
            <link>https://medium.com/@deepti.agl2912?source=rss-abd297954d5f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 04:56:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@deepti.agl2912/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Snowflake MCP: The Enterprise Gateway for AI Agents]]></title>
            <link>https://medium.com/@deepti.agl2912/snowflake-mcp-the-enterprise-gateway-for-ai-agents-d3b5e13335eb?source=rss-abd297954d5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/d3b5e13335eb</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[snowflake-cortex]]></category>
            <dc:creator><![CDATA[Deepti Agrawal]]></dc:creator>
            <pubDate>Tue, 01 Sep 2026 13:01:04 GMT</pubDate>
            <atom:updated>2026-09-01T13:01:04.282Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Af97_i0s19a9EVKgOGVXJw.png" /></figure><p>AI agents are only useful when they can securely access the systems where business data and actions actually live.</p><p>That is where <strong>Model Context Protocol, MCP</strong>, becomes important.</p><p>Snowflake now provides a <strong>Snowflake-managed MCP server</strong>, allowing MCP-compatible AI clients to securely discover and invoke Snowflake capabilities without deploying a separate MCP server infrastructure. The feature is generally available.</p><p>The interesting part is not just that Snowflake supports MCP.</p><p>It is <strong>what Snowflake can expose through MCP</strong>.</p><h3>What is Snowflake-managed MCP?</h3><p>MCP is an open standard that defines how AI applications interact with external tools and data systems.</p><p>Snowflake’s managed MCP implementation provides a standards-based endpoint that can expose Cortex capabilities and custom Snowflake logic while keeping authentication and authorization inside Snowflake governance.</p><p>At a high level:</p><p><strong>AI Client → Snowflake MCP Server → Snowflake Tools</strong></p><p>The AI client could be an MCP-compatible application such as ChatGPT, Claude, Cursor, or a custom application.</p><p>The MCP server acts as the interface.</p><p>Snowflake provides the underlying capabilities.</p><h3>MCP is a Snowflake object</h3><p>This is one of the most interesting parts of the implementation.</p><p>You create an MCP server directly inside a database and schema:</p><pre>CREATE OR REPLACE MCP SERVER MY_DB.AI.MY_MCP_SERVER<br>FROM SPECIFICATION $$<br>  tools:<br>    ...<br>$$;</pre><p>The specification defines which Snowflake capabilities the external MCP client is allowed to discover and invoke.</p><p>The server then becomes accessible through an endpoint with the pattern:</p><pre>https://&lt;account_url&gt;/api/v2/databases/&lt;database&gt;/schemas/&lt;schema&gt;/mcp-servers/&lt;server&gt;</pre><h3>What can you expose?</h3><p>Snowflake currently supports five MCP tool types.</p><h3>Cortex Agent</h3><pre>type: CORTEX_AGENT_RUN</pre><p>The MCP client sends a message to a Cortex Agent.</p><p>The Agent can then orchestrate other resources such as Cortex Analyst, Cortex Search, and custom tools.</p><h3>Cortex Analyst</h3><pre>type: CORTEX_ANALYST_MESSAGE</pre><p>Useful when the external AI client needs natural-language access to structured business data through a semantic model.</p><h3>Cortex Search</h3><pre>type: CORTEX_SEARCH_SERVICE_QUERY</pre><p>Useful for semantic retrieval across enterprise documents and other text-based datasets.</p><h3>SQL execution</h3><pre>type: SYSTEM_EXECUTE_SQL</pre><p>Allows controlled SQL execution through MCP.</p><h3>Custom tools</h3><pre>type: GENERIC</pre><p>Snowflake UDFs and stored procedures can be exposed as agent tools.</p><p>This means MCP is not limited to retrieval.</p><p>A custom procedure could potentially perform an enterprise action such as:</p><pre>- CREATE_TICKET<br>- RUN_DATA_QUALITY_CHECK<br>- GET_CUSTOMER_PROFILE<br>- TRIGGER_WORKFLOW</pre><h3>The architecture I would use</h3><p>Snowflake recommends an important design pattern for governed business-data applications:</p><p><strong>Expose a Cortex Agent as the primary client-facing MCP tool.</strong></p><p>Instead of exposing every Analyst, Search service, procedure, and SQL capability directly:</p><pre>MCP Client<br>   ↓<br>50 different tools</pre><p>use:</p><pre>MCP Client<br>   ↓<br>Cortex Agent<br>   ↓<br>Analyst + Search + Custom Tools</pre><p>The external AI client selects the Agent.</p><p>The Cortex Agent then selects and orchestrates the appropriate Snowflake resources.</p><p>This creates a clean separation:</p><p><strong>MCP handles external interoperability.</strong></p><p><strong>Cortex Agent handles internal orchestration.</strong></p><h3>Tool discovery</h3><p>MCP clients do not necessarily need hardcoded knowledge of every available capability.</p><p>They can discover tools using:</p><pre>tools/list</pre><p>and invoke them using:</p><pre>tools/call</pre><p>A tool call conceptually looks like:</p><pre>{<br>  &quot;jsonrpc&quot;: &quot;2.0&quot;,<br>  &quot;id&quot;: 1,<br>  &quot;method&quot;: &quot;tools/call&quot;,<br>  &quot;params&quot;: {<br>    &quot;name&quot;: &quot;incident_agent&quot;,<br>    &quot;arguments&quot;: {<br>      &quot;message&quot;: &quot;Which applications breached SLA this month?&quot;<br>    }<br>  }<br>}</pre><p>This is why <strong>tool names and descriptions matter</strong>.</p><p>The AI client may use those descriptions when deciding which capability to invoke. Snowflake specifically recommends precise, domain-specific names when exposing multiple agents.</p><h3>Security is layered</h3><p>Connecting to an MCP server does <strong>not</strong> automatically give the client access to every underlying resource.</p><p>Snowflake separates server access from tool access.</p><p>Depending on the configuration, roles may require privileges such as:</p><pre>USAGE on MCP SERVER<br>USAGE on CORTEX AGENT<br>USAGE on CORTEX SEARCH SERVICE<br>SELECT on SEMANTIC VIEW<br>USAGE on FUNCTION<br>USAGE on PROCEDURE</pre><p>This is important because the effective architecture becomes:</p><p><strong>Identity → Role → MCP Server → Tool → Underlying Data</strong></p><p>Snowflake recommends using least-privileged roles and avoiding broad MCP access grants.</p><h3>Authentication</h3><p>Snowflake-managed MCP uses <strong>Snowflake OAuth by default</strong>.</p><p>External OAuth can also be configured with enterprise identity providers such as Okta or Microsoft Entra ID.</p><p>A typical Snowflake OAuth integration looks like:</p><pre>CREATE SECURITY INTEGRATION MCP_OAUTH<br>  TYPE = OAUTH<br>  OAUTH_CLIENT = CUSTOM<br>  ENABLED = TRUE<br>  OAUTH_CLIENT_TYPE = &#39;CONFIDENTIAL&#39;<br>  OAUTH_REDIRECT_URI = &#39;&lt;redirect_uri&gt;&#39;<br>  OAUTH_USE_SECONDARY_ROLES = NONE<br>  ALLOWED_ROLES_LIST = (&#39;MCP_ACCESS_ROLE&#39;);</pre><p>Snowflake recommends leaving secondary roles disabled for MCP unless broader permissions are intentionally required.</p><p>Another practical detail:</p><p>The connecting user should have both a valid:</p><pre>DEFAULT_ROLE<br>DEFAULT_WAREHOUSE</pre><p>Otherwise authentication may succeed while the MCP session itself fails to initialize correctly.</p><h3>Network policies still apply</h3><p>If the Snowflake account uses network policies, remote MCP traffic originates from the <strong>MCP client’s infrastructure</strong>, not the user’s browser.</p><p>That means the provider’s outbound IP addresses may need to be explicitly allowed.</p><p>Otherwise even valid OAuth credentials can result in connection failures.</p><p>This is an easy detail to miss during implementation.</p><h3>A practical example</h3><p>Suppose we are building an Incident Management Agent.</p><p>Inside Snowflake we have:</p><p><strong>Cortex Analyst</strong></p><p>For:</p><pre>- Incident counts<br>- SLA breaches<br>- MTTR<br>- Application metrics</pre><p><strong>Cortex Search</strong></p><p>For:</p><pre>- RCA reports<br>- Postmortems<br>- Runbooks<br>- Knowledge articles</pre><p><strong>Stored procedures</strong></p><p>For actions such as:</p><pre>CREATE_ESCALATION<br>NOTIFY_OWNER</pre><p>We configure all of these behind a Cortex Agent.</p><p>Then expose only the Agent:</p><pre>CREATE OR REPLACE MCP SERVER INCIDENT_DB.AI.INCIDENT_MCP<br>FROM SPECIFICATION $$<br>tools:<br>  - title: &quot;Incident Management Agent&quot;<br>    name: &quot;incident_agent&quot;<br>    type: &quot;CORTEX_AGENT_RUN&quot;<br>    identifier: &quot;INCIDENT_DB.AI.INCIDENT_AGENT&quot;<br>    description: &quot;Analyzes incidents, searches RCA information, and supports incident investigation.&quot;<br>$$;</pre><p>Now an external AI client could ask:</p><blockquote><em>Which application has the most SLA breaches, what do previous RCA reports say, and what should we investigate first?</em></blockquote><p>One request can ultimately involve structured analytics, document retrieval, reasoning, and custom actions.</p><p>That is where MCP + Cortex Agents becomes powerful.</p><h3>Important limitations</h3><p>There are some constraints worth knowing.</p><p>A Snowflake MCP server currently supports a maximum of <strong>50 tools</strong>. Snowflake also warns that larger tool sets can hurt tool-selection accuracy.</p><p>Responses are currently <strong>non-streaming</strong>, and Snowflake’s implementation currently focuses on MCP tool capabilities rather than supporting every MCP construct.</p><p>Generic tools and SQL execution responses are also limited to <strong>250 KB</strong>, so tools should return targeted context rather than massive query results.</p><p>There is also a maximum recursion depth of <strong>10 invocations</strong>, designed to prevent uncontrolled agent-to-agent or MCP-to-MCP loops.</p><h3>The bigger picture</h3><p>The architecture is no longer simply:</p><p><strong>LLM → Database</strong></p><p>It is becoming:</p><p><strong>AI Client → MCP → Governed Agent → Enterprise Tools → Data</strong></p><blockquote>MCP solves the interoperability layer.</blockquote><blockquote>Snowflake RBAC and OAuth solve access control.</blockquote><blockquote>Cortex Agents handle orchestration.</blockquote><blockquote>Cortex Analyst handles structured data.</blockquote><blockquote>Cortex Search handles unstructured retrieval.</blockquote><blockquote>UDFs and stored procedures provide custom actions.</blockquote><p>That is what makes Snowflake-managed MCP interesting.</p><p>It turns Snowflake from something an AI application simply <strong>queries</strong> into a platform whose governed capabilities an AI agent can <strong>discover and invoke through a standard protocol</strong>.</p><p>And for enterprise agentic AI, that distinction matters.</p><h3>Want to explore more?</h3><p>I’ve built separate hands-on projects for MCP on my YouTube channel, Click the link below:</p><ul><li><a href="https://youtu.be/Jmps53113Ug">MCP Jira Root Cause Analysis in Snowflake</a></li></ul><p>For more Snowflake AI and Data Engineering content, follow me on <strong>LinkedIn: </strong><a href="http://www.linkedin.com/in/deeptiagrawal29">Deepti Agrawal</a><strong> </strong>and <strong>YouTube:</strong> <a href="https://www.youtube.com/@DeeptiBuilds">DeeptiBuilds</a></p><p>#Snowflake #SnowflakeMCP #ModelContextProtocol #MCP #CortexAI #CortexAgents #AgenticAI #GenerativeAI #DataEngineering #AIEngineering #EnterpriseAI #SnowflakeCortex</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3b5e13335eb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cortex Analyst, Search or Agent?]]></title>
            <link>https://medium.com/@deepti.agl2912/cortex-analyst-search-or-agent-e7702f6776a9?source=rss-abd297954d5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7702f6776a9</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[snowflake-cortex]]></category>
            <dc:creator><![CDATA[Deepti Agrawal]]></dc:creator>
            <pubDate>Sat, 29 Aug 2026 12:17:22 GMT</pubDate>
            <atom:updated>2026-08-29T12:17:22.399Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*1vTdPncFAI2Nn3rCKX0i5g.png" /></figure><p>If you have been exploring AI capabilities in Snowflake, you have probably come across these three names repeatedly:</p><blockquote><strong>Cortex Analyst. Cortex Search. Cortex Agents.</strong></blockquote><p>At first, they can sound like three different ways of doing the same thing.</p><p>They are not. They solve three different problems.</p><p>The simplest way I remember them is:</p><blockquote><strong><em>Cortex Analyst understands structured data.</em></strong><em><br></em><strong><em>Cortex Search retrieves unstructured data.</em></strong><em><br></em><strong><em>Cortex Agents orchestrate across tools to answer more complex questions.</em></strong></blockquote><p>Let’s break that down.</p><h3>1. Cortex Analyst: Ask Questions About Structured Data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fafjQENy42q2XP4DUpUYmw.png" /></figure><p>Imagine your Snowflake account contains tables like:</p><pre>- CUSTOMERS<br>- ORDERS<br>- PRODUCTS<br>- INCIDENTS<br>- APPLICATIONS</pre><p>Now a business user asks:</p><blockquote><em>Which product category generated the highest revenue last month?</em></blockquote><p>Normally, somebody would have to translate that question into SQL.</p><p>Cortex Analyst helps remove that step.</p><p>It allows users to ask business questions in natural language and generates SQL against structured Snowflake data using semantic views. Snowflake describes Cortex Analyst as a fully managed text-to-SQL capability for answering business questions over structured data.</p><p>For example: <strong>User asks:</strong></p><blockquote><em>What was the average order value by customer segment?</em></blockquote><p>Cortex Analyst understands the question, maps the business terminology to your semantic model, generates the SQL, executes it, and returns the result.</p><h3>Think of Cortex Analyst as:</h3><p><strong>Natural Language → SQL → Structured Data</strong></p><p>A typical use case could be:</p><pre>User Question<br>      ↓<br>Cortex Analyst<br>      ↓<br>Semantic View<br>      ↓<br>Snowflake Tables<br>      ↓<br>Answer</pre><h3>Best suited for questions like:</h3><ul><li>What was total revenue last month?</li><li>Which customers generated the highest sales?</li><li>How many orders were cancelled this quarter?</li></ul><p>These questions usually involve:</p><p><strong>aggregations, filters, joins, dimensions and metrics.</strong></p><p>That is Cortex Analyst territory.</p><h3>2. Cortex Search: Find Information Inside Unstructured Data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z4PN0ISCP7bUdVr6U3yefg.png" /></figure><p>Now imagine the information you need is not sitting neatly inside rows and columns.</p><p>Instead, you have:</p><pre>- PDF documents<br>- Policies<br>- Contracts<br>- Incident RCA documents<br>- Support tickets<br>- Knowledge-base articles</pre><p>A user asks:</p><blockquote><em>What does our refund policy say about damaged products?</em></blockquote><p>You probably do not need SQL.</p><p>You need to find the most relevant information inside documents.</p><p>That is where <strong>Cortex Search</strong> fits.</p><p>Cortex Search provides low-latency search over text data and uses hybrid retrieval combining vector and keyword search. It is commonly used as the retrieval layer for RAG and enterprise search applications.</p><p>The flow becomes:</p><pre>User Question<br>      ↓<br>Cortex Search<br>      ↓<br>Relevant Documents / Chunks<br>      ↓<br>Answer</pre><h3>Think of Cortex Search as:</h3><p><strong>Natural Language → Retrieval → Unstructured Data</strong></p><p>For example:</p><blockquote><em>Find documents explaining the root cause of payment gateway failures.</em></blockquote><p>Or:</p><blockquote><em>What does the employee travel policy say about international accommodation?</em></blockquote><p>Or:</p><blockquote><em>Find previous incidents related to API timeout errors.</em></blockquote><p>Cortex Search retrieves the most relevant passages from your data.</p><p>Snowflake Agents can also use Cortex Search as a retrieval layer across documents, policies, transcripts, contracts, and other text-based sources.</p><h3>Best suited for questions like:</h3><ul><li>What does this policy say?</li><li>Find documents related to this issue.</li><li>Find contracts mentioning a particular clause.</li></ul><p>These are primarily <strong>retrieval questions</strong>.</p><p>That is Cortex Search territory.</p><h3>So Analyst vs Search Is Actually Pretty Simple</h3><p>A useful way to think about it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*I-gWGrtnmSBnFg22t5uT5w.png" /></figure><p>If your answer requires <strong>calculating something from tables</strong>, think Cortex Analyst.</p><p>If your answer requires <strong>finding something inside text</strong>, think Cortex Search.</p><p>But then comes the interesting part.</p><p>What happens when the question requires <strong>both?</strong></p><h3>3. Cortex Agents: When One Tool Is Not Enough</h3><p>Consider this question:</p><blockquote><em>Which application had the highest number of critical incidents this month, and what were the common root causes from previous RCA reports?</em></blockquote><p>There are actually two different problems hidden inside that question.</p><h3>Problem 1</h3><blockquote><em>Which application had the highest number of critical incidents?</em></blockquote><p>That requires structured data.</p><p>We probably need something like:</p><pre>- INCIDENT_FACT<br>- APPLICATION_DIM<br>- PRIORITY_DIM<br>- DATE_DIM</pre><p>Cortex Analyst can handle that.</p><h3>Problem 2</h3><blockquote><em>What were the common root causes?</em></blockquote><p>Maybe those root causes exist inside incident reports, postmortems, or RCA documents.</p><p>Cortex Search can handle that.</p><p>But now something needs to decide:</p><ol><li>Which tool should run first?</li><li>What should we ask each tool?</li><li>How should the results be combined?</li><li>Is another tool call required?</li><li>What should the final answer look like?</li></ol><p>That is where <strong>Cortex Agents</strong> come in.</p><blockquote>Snowflake describes Cortex Agents as a managed agentic platform where an agent can reason over a request, plan its work, call tools, evaluate the results, and generate a response. Agents can use Cortex Analyst for structured data and Cortex Search for unstructured data within the same workflow.</blockquote><p>The architecture becomes:</p><pre>User<br>                   ↓<br>             Cortex Agent<br>              ↙         ↘<br>     Cortex Analyst    Cortex Search<br>           ↓                ↓<br>   Structured Data      Documents<br>              ↘         ↙<br>              Final Answer</pre><h3>Think of Cortex Agents as:</h3><p><strong>Understand → Plan → Select Tools → Execute → Combine → Respond</strong></p><h3>Analyst and Search Become Tools for the Agent</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*nAuEhAFU6zzCO6lPRM18Ug.png" /></figure><p>This is probably the most important concept.</p><p>Cortex Agents do not replace Cortex Analyst or Cortex Search.</p><p>They can <strong>use them as tools</strong>.</p><p>An agent can decide:</p><blockquote><em>I need structured metrics for this part of the question.</em></blockquote><p>Then invoke Cortex Analyst.</p><p>Next:</p><blockquote><em>I need supporting information from documents.</em></blockquote><p>Then invoke Cortex Search.</p><p>Then combine both results into one response.</p><p>Snowflake’s current Agents platform can also use other tools such as code execution, custom UDFs and stored procedures, data-to-chart capabilities, MCP connectors, and web search depending on the configuration.</p><p>So the architecture can grow into something like:</p><pre>Cortex Agent<br>                         |<br>          +--------------+--------------+<br>          |              |              |<br>       Analyst         Search       Custom Tool<br>          |              |              |<br>       Tables        Documents       Actions</pre><p>The Agent becomes the <strong>orchestration layer</strong>.</p><h3>Final Thoughts</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*McCvJqsyl1F_XPMazp9TNA.png" /></figure><p>When starting a Snowflake AI project, you do not always need an Agent.</p><p>Sometimes Cortex Analyst alone is enough. Sometimes all you need is Cortex Search.</p><p>Adding an orchestration layer when your application only performs one simple task can introduce unnecessary complexity.</p><p>But once your use case starts combining structured data, unstructured information, calculations, custom actions, or multiple reasoning steps, Cortex Agents become much more valuable.</p><p>So before choosing the technology, start with the question:</p><blockquote><strong><em>Where does the answer live, and what does the AI need to do to get it?</em></strong></blockquote><p>If the answer lives in <strong>tables</strong>, start with Cortex Analyst.</p><p>If it lives in <strong>documents</strong>, start with Cortex Search.</p><p>If getting the answer requires <strong>multiple tools and decisions</strong>, Cortex Agents are probably the right orchestration layer.</p><p>That is the simplest way I have found to separate the three.</p><h3>Want to explore more?</h3><p>I’ve built separate hands-on projects for all three on my YouTube channel, Click the link below:</p><ul><li>Cortex Analyst : <a href="https://youtu.be/uanaLwb5uwk">Conversational AI Cortex Analyst</a></li><li>Cortex Search : <a href="https://youtu.be/5QNJc-8P7YU">HR policies Cortex Search Teams Agent</a></li><li>Cortex Agent : <a href="https://youtu.be/Jmps53113Ug">JIRA Root Cause Analysis MCP Cortex Agent</a></li></ul><p>For more Snowflake AI and Data Engineering content, follow me on <strong>LinkedIn: </strong><a href="http://www.linkedin.com/in/deeptiagrawal29">Deepti Agrawal</a><strong> </strong>and <strong>YouTube:</strong> <a href="https://www.youtube.com/@DeeptiBuilds">DeeptiBuilds</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7702f6776a9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[13 Practical Steps to Build a Data Quality Framework in Snowflake❄️]]></title>
            <link>https://medium.com/@deepti.agl2912/13-practical-steps-to-build-a-data-quality-framework-in-snowflake-%EF%B8%8F-e6edbc272142?source=rss-abd297954d5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e6edbc272142</guid>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[snowflake-data-cloud]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Deepti Agrawal]]></dc:creator>
            <pubDate>Fri, 15 May 2026 15:05:04 GMT</pubDate>
            <atom:updated>2026-08-29T12:27:30.936Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hDjr0JU69S6SFI-i-qUSvQ.png" /></figure><p>Most people think Data Quality means checking for nulls and duplicates.</p><p>That is only the starting point.</p><p>In a real data engineering platform, a good Data Quality framework should cover the complete journey of data from raw ingestion to business reporting.</p><blockquote><strong>Let’s take a simple example:</strong></blockquote><blockquote>An order and sales pipeline.</blockquote><blockquote>A customer places an order<br>The order is confirmed<br>Payment is received<br>The item is shipped<br>Invoice is generated<br>Revenue appears in dashboard</blockquote><blockquote>Now imagine the dashboard shows ₹10 crore revenue.</blockquote><h3>Can the business trust it?</h3><p>Only if the data has passed proper quality checks at every stage.</p><p>Here is how I would build a simple Data Quality framework using 13 steps in Snowflake.</p><h3><strong>1. Start with profiling</strong></h3><p>Before writing rules, first understand the data.</p><p>For example:</p><blockquote>How many orders came today?<br>How many have missing customer IDs?<br>How many orders have negative amounts?<br>What are the top order statuses?<br>Are there any unexpected payment methods?</blockquote><p>In Snowflake, this can be done using simple profiling queries.</p><p>Example:</p><pre>SELECT<br>    COUNT(*) AS total_orders,<br>    COUNT_IF(order_id IS NULL) AS missing_order_ids,<br>    COUNT_IF(customer_id IS NULL) AS missing_customer_ids,<br>    COUNT(DISTINCT order_status) AS distinct_statuses,<br>    MIN(order_date) AS min_order_date,<br>    MAX(order_date) AS max_order_date<br>FROM silver.orders;</pre><p>Profiling helps us understand what is normal before we decide what is wrong.</p><h3><strong>2. Define Data Quality dimensions</strong></h3><p>A good DQ framework should not be random checks.</p><p>Each rule should belong to a clear quality dimension.</p><p>For example:</p><blockquote><strong>Completeness</strong><br>Required fields should not be missing.</blockquote><p>Example:<br>Order ID, customer ID, order date, and order amount should not be null.</p><blockquote><strong>Uniqueness</strong><br>Business keys should not be duplicated.</blockquote><p>Example:<br>One order ID should not appear multiple times in the final order table.</p><blockquote><strong>Validity</strong><br>Values should follow approved formats or allowed values.</blockquote><p>Example:<br>Order status should only be PLACED, CONFIRMED, CANCELLED, SHIPPED, or DELIVERED.</p><blockquote><strong>Consistency</strong><br>Data should make sense across tables.</blockquote><p>Example:<br>A shipment should not exist for an order that was cancelled.</p><blockquote><strong>Timeliness</strong><br>Data should arrive within the expected SLA.</blockquote><p>Example:<br>Daily sales data should be available before 8 AM.</p><blockquote><strong>Accuracy</strong><br>Data should match trusted reference data.</blockquote><p>Example:<br>Product ID in orders should exist in the product master.</p><blockquote><strong>Business logic</strong><br>The event sequence should make business sense.</blockquote><p>Example:<br>Payment date should not be before order date.<br>Shipment date should not be before payment confirmation.<br>Invoice amount should match order amount after discounts and tax.</p><h3><strong>3. Convert business understanding into rules</strong></h3><p>This is where data engineering and business knowledge come together.</p><p>Some simple rules for an order pipeline:</p><blockquote>Order ID must not be null<br>Order ID must be unique<br>Customer ID must exist in customer master<br>Product ID must exist in product master<br>Order amount must be greater than zero<br>Order date cannot be in the future<br>Cancelled orders should not have shipment records<br>Delivered orders must have delivery date<br>Payment amount should match final order amount<br>Invoice should not be generated before payment confirmation</blockquote><p>Example Snowflake SQL:</p><pre>SELECT *<br>FROM silver.orders<br>WHERE order_amount &lt;= 0;</pre><pre>SELECT order_id, COUNT(*) AS duplicate_count<br>FROM silver.orders<br>GROUP BY order_id<br>HAVING COUNT(*) &gt; 1;</pre><pre>SELECT s.*<br>FROM silver.shipments s<br>JOIN silver.orders o<br>    ON s.order_id = o.order_id<br>WHERE o.order_status = &#39;CANCELLED&#39;;</pre><p>These are simple checks, but they protect the business from wrong reporting.</p><h3><strong>4. Make the framework metadata-driven</strong></h3><p>In real projects, we should not write 200 separate scripts and manage them manually.</p><p>Create a DQ rule table in Snowflake.</p><blockquote>Example columns:</blockquote><blockquote>Rule ID<br>Rule name<br>Table name<br>Column name<br>DQ dimension<br>Severity<br>SQL logic<br>Threshold<br>Owner<br>Active flag</blockquote><p>Example:</p><pre>CREATE OR REPLACE TABLE dq.dq_rule_master (<br>    rule_id STRING,<br>    rule_name STRING,<br>    table_name STRING,<br>    dq_dimension STRING,<br>    severity STRING,<br>    rule_sql STRING,<br>    threshold_value NUMBER,<br>    owner STRING,<br>    is_active BOOLEAN<br>);</pre><p>This makes the framework scalable.</p><p>If a new rule is needed, we add a new row in the rule master instead of changing the whole pipeline.</p><h3><strong>5. Capture DQ results</strong></h3><p>Every rule execution should be logged.</p><p>Example result table:</p><pre>CREATE OR REPLACE TABLE dq.dq_rule_results (<br>    execution_id STRING,<br>    rule_id STRING,<br>    rule_name STRING,<br>    table_name STRING,<br>    dq_dimension STRING,<br>    error_count NUMBER,<br>    threshold_value NUMBER,<br>    status STRING,<br>    execution_time TIMESTAMP_NTZ<br>);</pre><p>This helps answer important questions:</p><blockquote>Which rules failed today?<br>Which table has the most issues?<br>Which source system is creating bad data?<br>Is data quality improving or getting worse?</blockquote><h3><strong>6. Capture failed records</strong></h3><p>Counting failures is useful, but we also need to know which records failed.</p><p>For example:</p><blockquote>Which order has missing customer ID?<br>Which shipment belongs to a cancelled order?<br>Which invoice has an amount mismatch?</blockquote><p>Create an exception table.</p><pre>CREATE OR REPLACE TABLE dq.dq_exception_records (<br>    exception_id STRING,<br>    rule_id STRING,<br>    table_name STRING,<br>    business_key STRING,<br>    error_reason STRING,<br>    record_payload VARIANT,<br>    created_at TIMESTAMP_NTZ<br>);</pre><p>This helps data stewards and source teams fix the actual issue.</p><h3><strong>7. Use cleansing where safe</strong></h3><p>Some issues can be corrected safely.</p><p>Examples:</p><blockquote>Trim extra spaces<br>Convert blank values to null<br>Standardize order status<br>Standardize country and state names<br>Convert dates into proper format<br>Remove special characters from phone numbers</blockquote><p>Example:</p><pre>SELECT<br>    TRIM(order_id) AS order_id,<br>    UPPER(TRIM(order_status)) AS order_status,<br>    TRY_TO_DATE(order_date) AS order_date,<br>    order_amount<br>FROM raw.orders;</pre><p>But not every issue should be auto-fixed.</p><p>If payment amount does not match invoice amount, that should be flagged, not silently corrected.</p><h3><strong>8. Enrich the data</strong></h3><p>Clean data is good.</p><p>Enriched data is useful.</p><p>For an order pipeline, we can add:</p><blockquote>Customer segment<br>Product category<br>Region<br>Order value bucket<br>First order flag<br>Repeat customer flag<br>Delivery time in days<br>Revenue after discount<br>Tax amount<br>Gross margin</blockquote><p>Example:</p><pre>SELECT<br>    o.order_id,<br>    o.customer_id,<br>    c.customer_segment,<br>    p.product_category,<br>    o.order_amount,<br>    DATEDIFF(&#39;day&#39;, o.order_date, s.delivery_date) AS delivery_days<br>FROM silver.orders o<br>LEFT JOIN silver.customers c<br>    ON o.customer_id = c.customer_id<br>LEFT JOIN silver.products p<br>    ON o.product_id = p.product_id<br>LEFT JOIN silver.shipments s<br>    ON o.order_id = s.order_id;</pre><p>DQ checks should also run after enrichment.</p><p>Example:</p><p>Delivery days should not be negative.<br>Revenue after discount should not be greater than original order amount.</p><h3><strong>9. Reconcile data across layers</strong></h3><p>A good Snowflake pipeline usually has layers like:</p><p>Raw<br>Silver<br>Gold<br>Reporting</p><p>We should reconcile data between layers.</p><p>Example:</p><blockquote>Raw orders count should match silver orders count after valid transformations.<br>Total order amount in silver should match total revenue in gold.<br>Cancelled orders should not contribute to net revenue.</blockquote><p>Example:</p><pre>SELECT<br>    raw_count.total_orders AS raw_orders,<br>    silver_count.total_orders AS silver_orders,<br>    raw_count.total_orders - silver_count.total_orders AS difference<br>FROM<br>    (SELECT COUNT(*) AS total_orders FROM raw.orders) raw_count,<br>    (SELECT COUNT(*) AS total_orders FROM silver.orders) silver_count;</pre><p>This helps ensure no data is lost during transformation.</p><h3><strong>10. Automate using Snowflake tasks</strong></h3><p>DQ should not depend on someone running queries manually.</p><p>In Snowflake, we can schedule DQ procedures using tasks.</p><p>Example:</p><pre>CREATE OR REPLACE TASK dq.run_order_dq_daily<br>WAREHOUSE = DQ_WH<br>SCHEDULE = &#39;USING CRON 0 7 * * * Asia/Kolkata&#39;<br>AS<br>CALL dq.run_dq_rules();</pre><p>This makes DQ part of the pipeline.</p><p>Not an afterthought.</p><h3><strong>11. Create a DQ scorecard</strong></h3><p>Business users do not want to read 100 SQL failures.</p><p>They need a simple scorecard.</p><p>Example:</p><blockquote>Orders completeness: 99.8 percent<br>Payments consistency: 98.5 percent<br>Shipment validity: 97.2 percent<br>Revenue reconciliation: Failed<br>Critical failures: 2<br>Open exceptions: 35</blockquote><p>This helps leadership understand whether the data can be trusted.</p><h3><strong>12. Add governance</strong></h3><p>Every DQ rule should have an owner.</p><p>For example:</p><blockquote>Order data owner<br>Payment data owner<br>Shipment data owner<br>Product master owner<br>Customer master owner</blockquote><p>Also define:</p><blockquote>Who fixes the issue?<br>What is the SLA?<br>Which rules are critical?<br>Which failures should stop the pipeline?<br>Which failures can pass with warning?</blockquote><p>Without governance, DQ becomes only a technical checklist.</p><p>With governance, it becomes an operating model.</p><h3><strong>13. Decide what should stop the pipeline</strong></h3><p>Not every issue should break the pipeline.</p><p>Example:</p><p><strong>Critical failures:</strong></p><blockquote>Order ID is null<br>Duplicate order ID<br>Revenue mismatch<br>Payment missing for completed order<br>Shipment exists for cancelled order</blockquote><p><strong>Warning failures:</strong></p><blockquote>Missing optional customer email<br>Unknown marketing channel<br>Late shipment update<br>Minor address formatting issue</blockquote><p>This severity model helps balance data quality with business continuity.</p><h3><strong>Why this matters</strong></h3><p>Bad data creates bad decisions.</p><blockquote>A wrong sales dashboard can impact revenue reporting.<br>A duplicate order can inflate performance numbers.<br>A missing payment can create finance issues.<br>A shipment mismatch can impact customer experience.<br>A late data load can make leadership lose trust in dashboards.</blockquote><p>Data Quality is not just about clean tables.</p><p>It is about trust.</p><p>In Snowflake, a practical DQ framework should include:</p><p>Profiling<br>DQ dimensions<br>Rule catalog<br>SQL validations<br>Exception capture<br>Cleansing<br>Enrichment<br>Reconciliation<br>Automation with tasks<br>Scorecards<br>Governance<br>Remediation workflow</p><p>The goal is simple:</p><p>Build pipelines that do not just move data.</p><p>Build pipelines that make data reliable, explainable, and trusted.</p><h3>Want to explore more?</h3><p>I’ve built hands-on projects for Data Quality Framework in Snowflake on my YouTube channel, Click the link below:</p><ul><li><a href="https://youtu.be/lh7IJwNew1M">No Code Data Quality Framework in Snowflake</a></li></ul><p>For more Snowflake AI and Data Engineering content, follow me on <strong>LinkedIn: </strong><a href="http://www.linkedin.com/in/deeptiagrawal29">Deepti Agrawal</a><strong> </strong>and <strong>YouTube:</strong> <a href="https://www.youtube.com/@DeeptiBuilds">DeeptiBuilds</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e6edbc272142" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ 5 Common Mistakes in Data Modeling for OLAP Systems — And How to Fix Them (The Simple Way)]]></title>
            <link>https://medium.com/@deepti.agl2912/5-common-mistakes-in-data-modeling-for-olap-systems-and-how-to-fix-them-the-simple-way-da136faabe58?source=rss-abd297954d5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/da136faabe58</guid>
            <category><![CDATA[data-ware-housing]]></category>
            <category><![CDATA[data-modeling]]></category>
            <category><![CDATA[olap]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Deepti Agrawal]]></dc:creator>
            <pubDate>Mon, 14 Apr 2025 13:51:24 GMT</pubDate>
            <atom:updated>2025-04-14T13:53:32.334Z</atom:updated>
            <content:encoded><![CDATA[<h3>🧱 5 Common Mistakes in Data Modeling for OLAP Systems — And How to Fix Them (The Simple Way)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/505/1*lWbmS5j86LweARqyH1IaNg.jpeg" /><figcaption>Data Modeling</figcaption></figure><p>As a data engineer, I’ve spent the last few years building reporting systems and dashboards that help businesses make better decisions. But I’ve learned this the hard way:</p><blockquote><em>A </em><strong><em>bad data model</em></strong><em> can ruin even the best reporting tools.<br> A </em><strong><em>good one</em></strong><em>? It makes insights fast, accurate, and scalable.</em></blockquote><p>So, in this article, I’m breaking down <strong>the most common mistakes</strong> I’ve seen in OLAP data modeling — and how you can avoid them.</p><h3>🤔 Wait…What’s the Difference Between OLTP and OLAP?</h3><p>This question is where many of us start, and it’s important to get it right.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*B_CAAb0reXq_cj0lrXTI3Q.png" /><figcaption>OLTP Vs OLAP</figcaption></figure><h3>🔁 So what’s different in data modeling?</h3><p><strong>OLTP</strong> modeling is about being clean and efficient with <strong>updates</strong>. You want zero duplication, lots of small tables, and strong integrity.</p><p><strong>OLAP</strong> modeling is about being smart with <strong>reads</strong>. You want queries to run fast, so you often <strong>combine related data</strong> (even if that means a bit of duplication).</p><p>In OLAP, we’re designing how data is <strong>structured, stored, and connected</strong> so it can be easily analyzed. It helps answer questions like:</p><ul><li>“What were our monthly sales?”</li><li>“How many customers churned in Q2?”</li><li>“Which products performed best in Europe?”</li></ul><p>If your data model is messy, your answers will be slow, wrong, or expensive.</p><p>Now that that’s clear, let’s look at the <strong>common mistakes</strong> we make when designing for OLAP data warehousing systems.</p><h3>⚠️ Mistake #1: Using Transactional Thinking for Analytical Work</h3><p><strong>What happens?</strong><br> You build tables that are highly normalized — like a web of small, tightly connected tables with IDs pointing everywhere.</p><p><strong>Why it’s a problem:</strong><br> For reporting, we want things fast and readable. Too many joins = slow dashboards and confused analysts.</p><p><strong>What to do instead:</strong><br> 📌 Use <strong>Star Schema</strong> — one big fact table in the middle (like sales) and smaller dimension tables (like customer, product, time) around it.<br> 📌 Group related info together. Denormalize where it makes sense.</p><p>🛠 Example:<br> Instead of this 👎</p><ul><li>customer_info → region → region_details → geo_data</li><li>dim_customer with customer name, region, and country <em>together</em></li></ul><p>Try this 👍</p><h3>⚠️ Mistake #2: Ignoring Changes Over Time (SCD)</h3><p><strong>What happens?</strong><br> Let’s say a customer moves from India to the US. If you only keep their latest country info, your past reports show them in the wrong place!</p><p><strong>Why it’s a problem:</strong><br> You lose historical truth — which leads to wrong trends and reports.</p><p><strong>Fix:</strong><br> 📌 Use <strong>Slowly Changing Dimensions (Type 2)</strong> to store historical changes. 📌 Add columns like start_date, end_date, and current_flag to track versions of the same record.</p><h3>⚠️ Mistake #3: Messy Table &amp; Column Names</h3><p><strong>What happens?</strong><br> You name things like tbl123, cust_info_new or final_final2, and no one knows what’s inside.</p><p><strong>Why it’s a problem:</strong><br> Others (and even <em>future you</em>) will get lost.</p><p><strong>Fix:</strong><br> 📌 Follow naming conventions like dim_customer, fact_sales.<br> 📌 Use tools like Snowflake&#39;s COMMENT feature or Unity Catalog to add descriptions.<br> 📌 Maintain a simple glossary — what each table and column means.</p><h3>⚠️ Mistake #4: No Aggregation Planning</h3><p><strong>What happens?</strong><br> Your dashboard does the heavy lifting — summing data, calculating averages — every single time.</p><p><strong>Why it’s a problem:</strong><br> This burns compute power and slows down reports.</p><p><strong>Fix:</strong><br> 📌 Create <strong>aggregated tables</strong> for frequent questions (monthly sales, region-wise churn).<br> 📌 Use <strong>materialized views</strong> or scheduled jobs in Snowflake or Databricks.<br> 📌 Organize data using clustering (Snowflake) or Z-ordering (Databricks) for faster filters.</p><h3>⚠️ Mistake #5: Skipping the Reporting Layer</h3><p><strong>What happens?</strong><br> You build great models, but analysts still need 5 joins and 7 filters to get anything useful.</p><p><strong>Why it’s a problem:</strong><br> It defeats the purpose of data modeling. BI tools like Tableau, Power BI or Looker suffer.</p><p><strong>Fix:</strong><br> 📌 Build a clean <strong>reporting layer</strong> or data marts — pre-joined, pre-cleaned, easy-to-use datasets.<br> 📌 Include predefined KPIs like revenue, active users, churn rate.<br> 📌 Collaborate with BI and analytics folks early — ask <em>what questions</em> they need answers to.</p><h3>💬 Final Thoughts (aka Real Talk)</h3><p>You don’t need to be a data modeling guru. But you do need to:</p><ul><li>Think about the people using the data.</li><li>Anticipate how it will grow.</li><li>Avoid making things harder than they need to be.</li></ul><blockquote>A good OLAP data model isn’t just “correct” — it’s <strong>usable</strong>, <strong>performant</strong>, and <strong>friendly to the people reading it</strong>.</blockquote><p>💡 Just ask: <em>“If I were answering business questions from this model, would I enjoy it… or dread it?”</em></p><p>I’ve made these mistakes. I’ve fixed them. And I’m still learning.</p><h3>🙌 Want to keep learning?</h3><p>I’m on a journey to share simple, relatable articles on data engineering and big data.</p><p>✅ Follow me if you’re a fellow data engineer or learner<br> ✅ Comment with your favorite modeling tips<br> ✅ Or share the biggest mistake <em>you’ve</em> made in OLAP systems — let’s learn from each other!</p><p>#DataEngineering #OLAP #Snowflake #Databricks #BigData #DataModeling #PySpark #SQL #Analytics</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da136faabe58" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Spark vs Hadoop: Why Spark Took the Lead]]></title>
            <link>https://medium.com/@deepti.agl2912/spark-vs-hadoop-why-spark-took-the-lead-11362a35ea1c?source=rss-abd297954d5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/11362a35ea1c</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[distributed-computing]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[hadoop]]></category>
            <dc:creator><![CDATA[Deepti Agrawal]]></dc:creator>
            <pubDate>Tue, 08 Apr 2025 13:30:33 GMT</pubDate>
            <atom:updated>2025-04-14T04:32:02.213Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WC07irwZNHKQS97zToGcAg.jpeg" /><figcaption>Apache Spark Vs Hadoop</figcaption></figure><blockquote>Before jumping into the <strong>difference between Apache Spark and Hadoop</strong>, let’s take a step back and <strong>answer some “What” and “Why” Questions First…</strong></blockquote><p>What exactly do these tools do?<br> Why were they even built?<br> why are they so important in the world of data?</p><p>To answer all that, we need to understand the problem they’re solving — <strong>Big Data</strong>.</p><p>These tools are designed to <strong>process Big Data</strong> — data that’s too large or complex for traditional systems to handle efficiently. They work on a <strong>distributed architecture</strong>, which allows them to handle massive volumes of data by breaking the workload across multiple machines.</p><h3>What is Big Data?</h3><p>Big Data isn’t just about having a lot of data — it’s about how complex, fast, and valuable that data is. It’s defined by the <strong>5 Vs</strong>:</p><ul><li><strong>Volume</strong> — Massive amounts of data generated every second, from apps, devices, social media, and more.</li><li><strong>Variety</strong> — Data comes in different formats: structured (tables), semi-structured (JSON, XML), and unstructured (images, videos, text).</li><li><strong>Velocity</strong> — The speed at which new data is created and needs to be processed, often in real-time.</li><li><strong>Veracity</strong> — The quality and trustworthiness of data. Not all data is clean or reliable.</li><li><strong>Value</strong> — The goal is to extract meaningful insights that can drive decisions or actions.</li></ul><p>Big Data systems need to handle all of this — and that’s where tools like Hadoop and Spark come in.</p><h3>What is Distributed Architecture?</h3><p>In simple terms, instead of relying on one powerful machine (like in monolithic architecture), <strong>distributed systems</strong> use a <strong>cluster of smaller machines (called nodes)</strong> working together.</p><p>Why is this better?</p><ul><li>If you <strong>double the resources</strong>, you can almost <strong>double the performance</strong>.</li><li>It’s <strong>scalable</strong>, <strong>efficient</strong>, and built for growth.</li><li>In contrast, a monolithic system often hits performance limits even if you add more power.</li></ul><h3>What all do we need to design a Big Data System?</h3><p>To build a robust big data solution, we typically need three key components:</p><ol><li><strong>Distributed Storage</strong> — To store large datasets across machines</li><li><strong>Distributed Processing</strong> — To process that data in parallel</li><li><strong>Scalability</strong> — To grow with increasing data without performance issues</li></ol><h3>Now, HERE COMES HADOOP…</h3><p>Hadoop was inspired by the growing need to process <strong>massive, unstructured data</strong> that traditional databases couldn’t handle efficiently. Its roots go back to the early 2000s, when Google published two influential papers:</p><ul><li><strong>Google File System (GFS)</strong> — A scalable distributed file system designed for large-scale data storage.</li><li><strong>MapReduce</strong> — A programming model for parallel processing of large data sets.</li></ul><p>Hadoop was built as an open-source implementation of these concepts. It made it possible to <strong>store and process petabytes of data across low-cost hardware</strong>, opening the door for scalable, distributed data processing long before the rise of cloud computing.</p><p>It has <strong>three core components</strong> that work together to store and process big data:</p><ul><li><strong>HDFS (Hadoop Distributed File System)</strong> — Handles <strong>distributed storage</strong> by breaking large files into blocks and storing them across multiple machines.</li><li><strong>MapReduce</strong> — The <strong>processing engine</strong> that performs data computation in parallel across the cluster.</li><li><strong>YARN (Yet Another Resource Negotiator)</strong> — Acts as the <strong>resource manager</strong>, coordinating and allocating system resources to run jobs efficiently.</li></ul><blockquote>While powerful, Hadoop comes with a steep learning curve.</blockquote><blockquote>Writing <strong>MapReduce programs</strong> can be challenging — they’re written in <strong>Java</strong>, and even simple operations require <strong>many lines of complex code</strong>. On top of that, Hadoop has a whole ecosystem of tools (like Hive, Pig, HBase, etc.) that are specifically built to work with it, but these tools can also be <strong>complex and not beginner friendly</strong>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*y1fCSaC2au47L35hdDPXJw.jpeg" /><figcaption>Hadoop ecosystem</figcaption></figure><h3>Why Hadoop Was Preferred Earlier</h3><ul><li>📦 <strong>Handled Large Data Volumes</strong> through distributed storage (HDFS)</li><li>⚙️ <strong>Parallel Data Processing</strong> with MapReduce across multiple machines</li><li>💰 <strong>Cost-Effective</strong> — Ran on low-cost, commodity hardware</li><li>🧑‍🤝‍🧑 <strong>Strong Open-Source Community</strong> and growing ecosystem</li><li>🛠️ <strong>Highly Customizable</strong> for on-premise infrastructure</li><li>📊 <strong>Support from Tools</strong> like Hive and Pig for easier data querying</li></ul><h3>Challenges with Hadoop</h3><p>While Hadoop was a game-changer in its time, it comes with several limitations that have made organizations look for better alternatives like Spark. Here are some of the common challenges:</p><ul><li>🧱 <strong>Complex Programming</strong>: Writing MapReduce jobs is not easy — it requires Java and involves a lot of boilerplate code even for simple tasks.</li><li>🐢 <strong>Slower Performance</strong>: MapReduce processes data in multiple stages and relies heavily on disk I/O, making it slower, especially for iterative or real-time tasks.</li><li>⚙️ <strong>Complex Ecosystem</strong>: Hadoop isn’t plug-and-play. Its ecosystem includes many tools like Hive, Pig, HBase, etc., each with its own setup, learning curve, and integration overhead.</li><li>📊 <strong>Not Designed for Real-Time</strong>: Hadoop is built for batch processing. It struggles with real-time data or low-latency requirements.</li><li>💻 <strong>Primarily On-Premise</strong>: Hadoop was built for <strong>on-premise infrastructure</strong>, which means it doesn’t scale as easily or cost-effectively as cloud-native solutions.</li><li>💰 <strong>Resource Heavy</strong>: The disk-based nature of MapReduce can be inefficient in terms of both storage and compute resources.</li></ul><h3>Then, CAME APACHE SPARK…</h3><p>Apache Spark is a <strong>general-purpose, in-memory compute engine</strong> designed for fast, large-scale data processing.</p><p>It was built as a fast, flexible, and general-purpose big data engine that could overcome Hadoop MapReduce’s limitations.</p><p>Unlike Hadoop, which includes storage (HDFS), processing (MapReduce), and resource management (YARN), <strong>Spark focuses only on the compute part</strong> — it’s essentially a more efficient alternative to <strong>MapReduce.</strong></p><p><strong>Then why is Spark often discussed alongside Hadoop?</strong><br>Because Spark can <strong>integrate seamlessly with HDFS and YARN</strong>, as well as other distributed storage and resource managers.</p><h3>What makes Spark powerful is its <strong>plug-and-play architecture</strong>:</h3><ul><li>🔌 <strong>Pluggable Storage</strong> — Works with HDFS, AWS S3, Azure Data Lake (Gen2), Google Cloud Storage, local file systems, and even cloud data warehouses like Snowflake using connectors.</li><li>🧠 <strong>Flexible Resource Management</strong> — Can run on YARN, Kubernetes, or Mesos, depending on your setup.</li><li>💻 <strong>Multi-language Support</strong> — Code in Scala, Python, Java, or R.</li><li>✨ <strong>Simplified Distributed Processing</strong> — Spark <strong>abstracts away cluster complexity</strong>. You can write simple code (like SQL or Python) without worrying about how it gets distributed across the nodes.</li></ul><p>In short, <strong>Spark offers speed, flexibility, and simplicity</strong>, making it a popular choice for modern big data processing.</p><h3>Conclusion: How Spark Took the Lead Over Hadoop</h3><p>While Hadoop laid the foundation for Big Data processing, <strong>Apache Spark emerged as the next step forward</strong> — offering a faster, more flexible, and developer-friendly solution.</p><p>Here’s how Spark pulled ahead:</p><ul><li>🚀 <strong>In-Memory Computing</strong><br> Spark processes data in memory, which is <strong>much faster</strong> than Hadoop’s disk-heavy MapReduce. This speed advantage becomes even more evident in iterative tasks like machine learning or interactive analytics.</li><li>🧠 <strong>Developer-Friendly APIs</strong><br> Writing MapReduce jobs in Hadoop requires complex Java code. Spark, on the other hand, supports <strong>concise, readable APIs</strong> in Python, Scala, Java, and R, drastically reducing development time and effort.</li><li>🔄 <strong>Supports Batch + Streaming</strong><br> While Hadoop is designed for batch processing only, Spark can handle <strong>both batch and real-time streaming data</strong>, making it suitable for modern use cases like fraud detection, live dashboards, and alerting systems.</li><li>🔌 <strong>Pluggable and Flexible</strong><br> Spark doesn’t tie you to the Hadoop ecosystem. You can <strong>connect it with any storage</strong> (HDFS, S3, ADLS, GCS, etc.) and run it on your resource manager of choice (YARN, Kubernetes, Mesos), or even as a standalone cluster.</li><li>☁️ <strong>Cloud-Ready</strong><br> Spark works well with cloud-native architectures and has native support/connectors for data warehouses like Snowflake, making it easy to scale and manage workloads in the cloud.</li><li>📈 <strong>Faster Innovation &amp; Ecosystem</strong><br> With libraries like Spark SQL, MLlib, GraphX, and Spark Streaming, Spark became a <strong>multi-purpose engine</strong> — handling everything from ETL to advanced analytics and machine learning in one unified framework.</li></ul><p>In summary, <strong>Spark didn’t just improve on Hadoop — it redefined what Big Data processing could look like</strong>. It kept the strengths of distributed computing, but removed the friction, making data processing <strong>faster, more intuitive, and future ready</strong>. That’s why today, when people talk about Big Data, Spark is often at the center of the conversation.</p><h3>Thanks for Reading!</h3><p>If you like my work and want to support me…</p><ol><li>The BEST way to support me is by following me on <a href="https://medium.com/@deepti.agl2912">Medium</a>.</li><li>For data engineering best practices and tips, follow me on <a href="http://www.linkedin.com/in/deeptiagrawal29">LinkedIn</a>.</li><li>Feel free to give claps so I know how helpful this post was for you.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11362a35ea1c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“Cracking SnowPro Advanced: My Journey Through Snowflake Architect and Data Engineer…]]></title>
            <link>https://medium.com/@deepti.agl2912/cracking-snowpro-advanced-my-journey-through-snowflake-architect-and-data-engineer-6268c1995771?source=rss-abd297954d5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/6268c1995771</guid>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[data-architect]]></category>
            <category><![CDATA[snowflake-certification]]></category>
            <dc:creator><![CDATA[Deepti Agrawal]]></dc:creator>
            <pubDate>Mon, 16 Dec 2024 17:30:55 GMT</pubDate>
            <atom:updated>2024-12-16T17:30:55.948Z</atom:updated>
            <content:encoded><![CDATA[<h3>“Cracking SnowPro Advanced: My Journey Through Snowflake Architect and Data Engineer Certifications”</h3><p>As a Snowflake professional, the most frequent question I receive is,</p><blockquote><em>“</em><strong><em>How do the SnowPro Advanced Architect and Engineer certifications differ, and how should I prepare for them?”</em></strong></blockquote><p>Having cleared the SnowPro Engineer certification in September 2024 and the Architect certification in October 2023, I’ve navigated both paths and gained valuable insights along the way.</p><p>In this article, I’m sharing my experiences, preparation strategies, and the key distinctions between these two certifications to help you ace your SnowPro journey with confidence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*ANgz_sPe716h573ylPbhuA.jpeg" /></figure><p>Firstly, here are the links to the study guides for these exams, which serves like a syllabus. These will be your main reference for preparation, so make sure to review them thoroughly. Since the content gets updated regularly, always refer to the latest version by checking on these links.</p><ul><li><a href="https://training.snowflake.com/lmt/clmsCatalogDetails.prMain?site=sf&amp;in_offeringId=98873813&amp;in_language_identifier=en-us&amp;in_region=us&amp;in_from_module=CLMSSHARE.PRMAIN">Snowflake Education</a></li><li><a href="https://training.snowflake.com/lmt/clmsCatalogDetails.prMain?site=sf&amp;in_offeringId=98873814&amp;in_language_identifier=en-us&amp;in_region=us&amp;in_from_module=CLMSSHARE.PRMAIN">Snowflake Education</a></li></ul><h4><strong>Architect Vs Engineer Exam</strong></h4><p><strong>Focus Areas:</strong></p><blockquote><strong>Architect:</strong> Focuses on designing end-to-end solutions, covering account management, security, Snowflake architecture, and performance optimization.</blockquote><blockquote><strong>Engineer:</strong> Concentrates on technical implementations like data movement, storage, security, and data transformation.</blockquote><p><strong>Exam Pattern:</strong></p><blockquote><strong>Architect:</strong> Includes many scenario-based questions, placing you in the role of an architect to design and suggest the best solutions by combining multiple Snowflake features. For example, understanding how materialized views, cloning, and data sharing work together.</blockquote><blockquote><strong>Engineer:</strong> Features technical data engineering questions along with scenario-based questions that test your ability to suggest use cases. For instance, addressing Snowpipe staleness or reloading data using the Copy command.</blockquote><p>In short, the <strong>Architect exam is more about the “why and how”</strong> of designing solutions, while the <strong>Engineer exam is about the “what and how” </strong>of building them.</p><p><em>Now, with this information, you’ll be able to decide which exam aligns with your goals. Once you’ve made your choice, you can follow the preparation guide I provide below.</em></p><p>But before that, I’m assuming that you have already cleared SnowPro Core Certification.</p><h4>Exam Preparation Guide</h4><ol><li><strong>Understand the Study Guide:</strong> Begin by thoroughly reviewing the study guide, understanding the topics, their weightage, and planning your strategy to divide your time effectively for each section.</li><li><strong>Review Snowflake Documentation:</strong> After getting familiar with the study guide, dive into the Snowflake documentation. Focus on the topics listed in the guide. I can’t emphasize enough how valuable the Snowflake documentation is — it’s incredibly well-organized and designed in an easy-to-navigate way, making it the perfect resource for your preparation.</li><li><strong>Revise Core Exam Topics:</strong> Revisit the topics you prepared for the SnowPro Core exam and practice related questions. This will help reinforce your foundation for the advanced exams, as approximately 50% of the questions in these Advanced Exams resemble the pattern of Core exam.</li><li><strong>Gain Practical Exposure:</strong> The exam guide recommends having 2+ years of experience as a Snowflake architect or engineer. However, even with that, you may not have worked on every feature mentioned in the study guide. To bridge this gap, utilize resources like Snowflake <a href="https://quickstarts.snowflake.com/">Quickstarts</a>, <a href="https://learn.snowflake.com/en/pages/hands-on-essentials-track/">hands-on essential badges</a>, <a href="https://docs.snowflake.com/en/tutorials">tutorials</a> to gain practical exposure and deepen your understanding.</li><li><strong>Understand Account Usage and Information Schema:</strong> Gain a thorough understanding of all the functions and views available in the ACCOUNT_USAGE and INFORMATION_SCHEMA. These are essential for monitoring, querying metadata, and understanding system-level details, which are frequently tested in the exams.</li><li><strong>Understand Feature Combinations:</strong> Learn how various Snowflake features work together in different scenarios. For example:</li></ol><ul><li>Cloning with Data Sharing</li><li>Materialized Views with Clustering</li><li>Time Travel with Streams</li><li>Tasks with Stored Procedures</li><li>Data Masking with Row Access Policies</li><li>External Tables with Stages</li></ul><p>These are just examples, but make sure you learn about all other possible combinations too.</p><p>7.<strong> Know Feature Availability by Edition:</strong> Understand which Snowflake features are available in different editions (Standard, Enterprise, Business Critical, etc.). This is crucial for designing solutions that align with the organization’s edition and for answering scenario-based questions in the exam.</p><p>8.<strong> Practice Exams:</strong> Take practice exams for both the Core and Advanced certifications. Review the topics where you make mistakes, revisit the relevant study material, and attempt the practice exams again. This iterative process will help you identify weak areas and improve steadily. I will link some of the practice sets I used in the references section below.</p><p>9. <strong>Set a Target Date:</strong> Decide on a target date for your exam and plan your preparation accordingly. For example, I gave myself two months to prepare for each exam, dedicating at least 2 hours daily and more on weekends. A clear timeline helps you stay focused and ensures consistent progress.</p><p>10. <strong>Have Faith in Yourself and Take the Exam:</strong> Believe in your preparation, trust your abilities, and schedule the exam. Stay confident and give it your best shot — you’re ready for this! All the best! 🌟</p><h4>References</h4><ol><li>Practice sets:</li></ol><p><strong>Core: </strong><a href="https://www.udemy.com/course/snowflake-snowpro-core-certification-exam-practice-sets/?couponCode=ST21MT121624">https://www.udemy.com/course/snowflake-snowpro-core-certification-exam-practice-sets/?couponCode=ST21MT121624</a></p><p><strong>Core:</strong> <a href="https://www.udemy.com/course/snowflake-snowpro-core-certification-practice-exams/?couponCode=ST21MT121624">https://www.udemy.com/course/snowflake-snowpro-core-certification-practice-exams/?couponCode=ST21MT121624</a></p><p><strong>Architect</strong>: <a href="https://www.udemy.com/course/snowflake-snowpro-advanced-architect-certification-practice-sets/?couponCode=ST21MT121624">https://www.udemy.com/course/snowflake-snowpro-advanced-architect-certification-practice-sets/?couponCode=ST21MT121624</a></p><p><strong>Engineer: </strong><a href="https://www.udemy.com/course/snowflake-snowpro-advanced-data-engineer-practice-sets/?couponCode=ST21MT121624">https://www.udemy.com/course/snowflake-snowpro-advanced-data-engineer-practice-sets/?couponCode=ST21MT121624</a></p><p>2. Snowflake Documentation: <a href="https://docs.snowflake.com/en/">https://docs.snowflake.com/en/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6268c1995771" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Everyday Data Engineering Use Cases in Snowflake]]></title>
            <link>https://medium.com/@deepti.agl2912/everyday-data-engineering-use-cases-in-snowflake-1c25ef7dcef9?source=rss-abd297954d5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c25ef7dcef9</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Deepti Agrawal]]></dc:creator>
            <pubDate>Thu, 03 Oct 2024 13:35:24 GMT</pubDate>
            <atom:updated>2024-10-03T13:35:24.436Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MtAskK20n0lBJJV04nwWfQ.jpeg" /><figcaption>The Data Family</figcaption></figure><blockquote>As a data engineer, I often witness various practices in our team that spark valuable discussions about efficiency and best practices. <strong>Recently, I noticed a teammate creating transient tables for backups of our datasets.</strong> While transient tables serve a purpose, it made me think about how much more effective our backup strategy could be if we utilized <strong>Snowflake’s zero-copy cloning</strong> feature instead. This not only streamlines the backup process but also reduces storage costs significantly.</blockquote><p>This experience prompted me to reflect on the common data engineering challenges we face daily and how Snowflake provides innovative solutions to address them.</p><p>In this article, we’ll explore several everyday use cases, complete with best practices for implementation in Snowflake.</p><h4>Use Case 1: Table Backup</h4><p>Imagine you’re about to make a big update on a table, but you want a quick backup just in case something goes wrong. Instead of creating a full copy, which takes time and space, Snowflake’s <strong>Zero-Copy Cloning</strong> allows you to clone the table instantly, without using additional storage.</p><blockquote>Cloning doesn’t physically duplicate the data. Both the original and the cloned table share the same underlying <strong>micro-partitions.</strong></blockquote><p><strong>Note:</strong> Updating the cloned copy will incur storage costs.</p><h4>Use Case 2: Data Recovery</h4><p>Ever deleted or messed up some rows by mistake? That heart-sinking moment! Snowflake’s <strong>Time Travel</strong> feature is your undo button. You can query data from a specific time before the error happened.</p><blockquote>Tip: You can combine <strong>Time Travel with Zero-Copy Cloning. </strong>This means you can clone a table as it was at a specific time in the past, creating a completely independent copy based on that historical data, which is great for testing or auditing purposes.</blockquote><p><strong>Note:</strong> Time Travel is limited by your account’s retention window, which can be up to 90 days (for Enterprise and above).</p><h4>Use Case 3: Optimizing Storage</h4><p>Got some data that you only need for a short time? Instead of keeping it around forever, you can use <strong>Transient Tables</strong> to save storage and cost.</p><p>These are commonly used in ETL processes where tables are frequently refreshed or discarded after processing.</p><p><strong>Note:</strong> Data is permanently deleted once purged — no recovery. Use transient tables for non-critical data, as they offer limited recovery options.</p><p>Another great example is<strong> </strong>using<strong> External tables </strong>which<strong> </strong>allows you to query data stored in external cloud storage (e.g., AWS S3, Azure Blob Storage) without importing it into Snowflake. This is useful for managing “cold” data that you don’t need to query frequently.</p><p><strong>Note:</strong> Querying external data can be slower than querying data stored directly in Snowflake. Use this for less frequent access or cold storage.</p><h4>Use Case 4: Frequent Table Updates Slowing Down Queries</h4><p>Lets say, You have a large table “<em>Customer_Transactions”</em> with millions of rows. Your analytics team frequently queries data by &quot;<em>Customer_id</em>” and “<em>Transaction_Date</em>”. Over time, these queries are getting slower as the table grows.</p><p>Define <strong>clustering keys</strong> on ”<em>Customer_id</em>” and “<em>Transaction_Date</em>”. This helps Snowflake organize the data so it can quickly find the relevant micro-partitions, speeding up your queries.</p><h4>Use Case 5: Discovering Data</h4><p>Ever spent precious minutes hunting for that one elusive table or function? We’ve all been there! <strong>Snowflake’s Universal Search</strong> is like a search engine for your data platform, instantly finding databases, tables, views, or even marketplace listings and documentation links. No more endless clicking — just type and locate exactly what you need!</p><p>I have also written an article for this feature, linking below:</p><p><a href="https://medium.com/@deepti.agl2912/snowflakes-universal-search-simplified-data-retrieval-c84a0b41878d">https://medium.com/@deepti.agl2912/snowflakes-universal-search-simplified-data-retrieval-c84a0b41878d</a></p><p><strong>Topics covered in this article</strong></p><ol><li>Zero Copy cloning: <a href="https://docs.snowflake.com/en/user-guide/object-clone">https://docs.snowflake.com/en/user-guide/object-clone</a></li><li>Time Travel: <a href="https://docs.snowflake.com/en/user-guide/data-time-travel">https://docs.snowflake.com/en/user-guide/data-time-travel</a></li><li>Transient Tables: <a href="https://docs.snowflake.com/en/user-guide/tables-temp-transient">https://docs.snowflake.com/en/user-guide/tables-temp-transient</a></li><li>External Tables: <a href="https://docs.snowflake.com/en/user-guide/tables-external-intro">https://docs.snowflake.com/en/user-guide/tables-external-intro</a></li><li>Clustering: <a href="https://docs.snowflake.com/en/user-guide/tables-clustering-keys">https://docs.snowflake.com/en/user-guide/tables-clustering-keys</a></li><li>Universal Search: <a href="https://docs.snowflake.com/en/user-guide/ui-snowsight-universal-search">https://docs.snowflake.com/en/user-guide/ui-snowsight-universal-search</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c25ef7dcef9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time Series Forecasting: End-to-End using Snowpark ML in Snowflake Notebooks]]></title>
            <link>https://medium.com/@deepti.agl2912/time-series-forecasting-end-to-end-using-snowpark-ml-in-snowflake-notebooks-9c07bc9b63ee?source=rss-abd297954d5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c07bc9b63ee</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[snowpark]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <dc:creator><![CDATA[Deepti Agrawal]]></dc:creator>
            <pubDate>Sat, 03 Aug 2024 16:47:06 GMT</pubDate>
            <atom:updated>2024-08-03T16:47:06.256Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*B3MnVgT83rX6jy7nB4NrOg.jpeg" /></figure><p>In this article, we will explore the process of building an end-to-end machine learning time series forecasting model using Snowpark Notebooks. Our focus will be on leveraging Snowflake’s powerful capabilities to create accurate forecasts, analyze long-term trends, and identify anomalies in time series data.</p><p>By walking through each step, from data preparation to model deployment, you’ll gain insights into how Snowpark simplifies complex forecasting tasks and makes advanced analytics accessible.</p><h3>Data Preparation</h3><blockquote>Our first step is to set up the necessary database objects for data loading. We’ll start by creating a stage location to securely store our data.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*urE5j8oABQeOCfwRFznrCw.png" /><figcaption>Objects Creation</figcaption></figure><blockquote>Next, we’ll upload our dataset, which contains shoe sales information over several years, to the SALES_DB.PUBLIC.DATA stage. Once the data is in place, we&#39;ll proceed to load it into the Snowflake tables.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xf6-N2GzOEyzdXdMNRNaIg.png" /><figcaption>Uploading Dataset on Stage</figcaption></figure><blockquote>Next, we’ll create a new Snowflake Notebook by navigating to the Projects &gt; Notebooks tab on the left-hand side of the interface.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_yJ7GjTMcR135Zc0m84rJw.png" /><figcaption>Creating a Snowflake notebook</figcaption></figure><p>Ensure that you create the notebook within the same database, schema, and warehouse that we established for this project.</p><h3>Data Ingestion</h3><blockquote>We’ll begin by importing all the required libraries, including Snowpark. Snowflake Notebooks use Python 3.9 by default and come pre-loaded with popular libraries for data science and machine learning, such as Altair, Pandas, NumPy, Snowflake-Snowpark-Python, and Streamlit.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q2DifbrCp8Tudod6rvUmvg.png" /><figcaption>importing libraries</figcaption></figure><p>We will also need the <a href="https://docs.snowflake.com/en/developer-guide/snowpark-ml/index">Snowpark ML (snowflake-ml-python)</a> package later when we build the model. To add any package to the notebook, simply search and select it from the Packages drop-down menu:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*nhomlSr1VvdMLllbZmktMQ.png" /><figcaption>installing packages</figcaption></figure><p>After selecting the packages you need, click “Start” to activate the virtual warehouse for your notebook, allowing you to begin executing your code.</p><blockquote>Next, we need to access an active Snowpark session, which is readily available and can be retrieved using the get_active_session() function.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LM8UgE5rcFRhWEv-RFxh2A.png" /><figcaption>getting Snowpark Session</figcaption></figure><blockquote>After this, we’ll load the data from the stage into a Snowflake table named Shoe_Sales within the same database.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KauDQjZTsbAX8cc3a4gzlw.png" /><figcaption>Laoding data into Table</figcaption></figure><blockquote>The next step is to create a Snowpark DataFrame from the table data using the snowpark.session.table function.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sc-5YmNZ2hlzlkRdI-qThA.png" /><figcaption>Creating Snowpark Dataframe</figcaption></figure><h3>Exploratory Data Analysis</h3><p>Time series data represents observations collected sequentially over time, allowing us to analyze trends, cycles, and seasonal patterns.</p><p>In addition to the preprocessing methods offered by Snowpark ML, Snowflake Notebooks also support popular visualization libraries like Matplotlib and Seaborn. These tools allow us to visually analyze our training data and perform feature selection.</p><blockquote>We will modify the dataset to include specific timestamps and then visualize it as a time series.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DffS1w1JVBsG0ClucILj-A.png" /><figcaption>Time Series Plot</figcaption></figure><blockquote>We will now decompose the time series to separate and examine its key components: <strong>trend, seasonality, and noise.</strong> This will help us better understand the underlying patterns and dynamics in the data.</blockquote><p>Key characteristics of time series data include <strong>trend</strong> (long-term movement), <strong>seasonality</strong> (regular, repeating patterns), and <strong>noise</strong> (random variations).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_1ezAa7Tp1MdOg0MYNwW3g.png" /><figcaption>Time Series Additive Decomposition</figcaption></figure><p>The above additive decomposition graph aims to separate the time series data into its individual components: trend, seasonality, and noise, for clearer analysis.</p><p>We can do all kinds of normal <a href="https://docs.snowflake.com/en/developer-guide/snowpark/python/working-with-dataframes">dataframe-style processing</a> using Python, visualize our output, and persist data to tables in our Snowflake account.</p><p>There are many other Exploratory Data Analysis (EDA) features to explore, such as checking the shape, information, missing values, and outliers. However, this article focuses specifically on the series decomposition as part of EDA.</p><h3>Model Building and Prediction</h3><blockquote>In this project, we are developing a linear regression model to forecast shoe sales.</blockquote><p>Linear regression is a straightforward yet powerful statistical method that predicts a dependent variable based on the relationship with one or more independent variables. By fitting a line to the historical sales data, the model will help us understand trends and make future predictions.</p><p>Using Snowpark ML, we’ll leverage Snowflake’s capabilities to train and evaluate this model efficiently within Snowflake Notebooks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2s9JD5CozSozmgW7CsvBIg.png" /><figcaption>Linear Regression SnowparkML</figcaption></figure><p>We can bundle our model building and pre-processing steps up together as a Pipeline object and save that as an artifact to use later during training and inference.</p><p>Here, we have developed a Linear Regression model using our training and test data with the LinearRegression class from snowflake.ml.modeling.linear_model.</p><blockquote>After building the Linear Regression model, we used the fit() function to train the model on our training data. This function adjusts the model parameters to best capture the relationship between the features and the target variable.</blockquote><blockquote>Following training, we applied the predict() function to generate forecasted sales based on the test data. This function uses the trained model to estimate future values, allowing us to evaluate the model&#39;s performance and make informed predictions.</blockquote><p>Now, we will plot the training data, testing data and Linear Regression test predicted data together to visualize the series.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b3g-vm-jLkBi6AaILNTCZg.png" /><figcaption>Model Prediction Plot</figcaption></figure><p>After generating predictions with our model and plotting the results, we observed that the accuracy was not satisfactory.</p><p>This indicates the need for further refinement. To improve our predictions, we should consider experimenting with various models, such as XGBoost, Naive Bayes, and smoothing techniques, before selecting the best-performing model for final predictions.</p><p>This iterative approach will help us find a more accurate and reliable model for forecasting.</p><h3>Conclusion</h3><p>Snowflake Notebooks provide a seamless environment for executing complete end-to-end machine learning workflows directly within the Snowsight UI. With Snowflake and Snowpark’s familiar tools and features integrated into this user-friendly notebook interface, you benefit from the same flexibility, cost efficiency, and ease of use you’re accustomed to, all without leaving the Snowflake platform.</p><p>If you’re interested in exploring similar use cases, a QuickStart guide for time series forecasting with Snowpark ML is available on the Snowflake website via the link below.</p><p><a href="https://quickstarts.snowflake.com/guide/hex/index.html?index=..%2F..index#0">Building and deploying a time series forecast with Hex + Snowflake</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c07bc9b63ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Beginner’s Guide to Snowflake UDFs (User Defined Functions)]]></title>
            <link>https://medium.com/@deepti.agl2912/a-beginners-guide-to-snowflake-udfs-user-defined-functions-702db33ad49c?source=rss-abd297954d5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/702db33ad49c</guid>
            <category><![CDATA[user-defined-functions]]></category>
            <category><![CDATA[snowflake-data-cloud]]></category>
            <dc:creator><![CDATA[Deepti Agrawal]]></dc:creator>
            <pubDate>Mon, 29 Jul 2024 06:46:53 GMT</pubDate>
            <atom:updated>2024-07-29T06:46:53.301Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine you’re a chef with a cookbook of standard recipes. Sometimes, you need a unique dish that isn’t in the book.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YMZ_L25nrt_Pizxb" /><figcaption>Photo by <a href="https://unsplash.com/@danielcgold?utm_source=medium&amp;utm_medium=referral">Dan Gold</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Creating a User-Defined Function (UDF) is like crafting your own special recipe, which you can reuse whenever needed. Just like your custom recipe adds new flavors to your menu, UDFs extend Snowflake’s capabilities, letting you tailor data processing to fit your specific needs. Once created, they’re ready to be called upon, enhancing your data toolkit with personalized solutions.</p><h3><strong>Creating a Snowflake UDF/UDTF</strong></h3><h4>UDF Components</h4><pre>CREATE [ OR REPLACE ] [ { TEMP | TEMPORARY } ] [ SECURE ] FUNCTION &lt;name&gt; (<br>    [ &lt;arg_name&gt; &lt;arg_data_type&gt; [ DEFAULT &lt;default_value&gt; ] ] [ , ... ] )<br>  [ COPY GRANTS ]<br>  RETURNS { &lt;result_data_type&gt; | TABLE ( &lt;col_name&gt; &lt;col_data_type&gt; [ , ... ] ) }<br>  [ [ NOT ] NULL ]<br>  LANGUAGE JAVASCRIPT<br>  [ { CALLED ON NULL INPUT | { RETURNS NULL ON NULL INPUT | STRICT } } ]<br>  [ { VOLATILE | IMMUTABLE } ]<br>  [ COMMENT = &#39;&lt;string_literal&gt;&#39; ]<br>  AS &#39;&lt;function_definition&gt;&#39;</pre><p>Given above is a definition to create a JavaScript UDF. The syntax for CREATE FUNCTION varies depending on which language you’re using as the UDF handler.</p><p>The first step is to decide what kind of function you want to create from below, which defines the RETURNS Clause in the function definition.</p><h4>Types of UDFs</h4><ol><li><strong>Scalar UDF</strong><br>Returns a single value for each input row, ideal for straightforward calculations and transformations.</li></ol><pre>create function function_name(x integer, y integer)<br>  returns integer</pre><p><strong>2. User-Defined Table Function (UDTF)</strong><br>Returns a table of values for each input row, with methods to process, initialize, and finalize partitions.</p><pre>create or replace function stock_sale(id varchar, qty number, price number(10,2))<br>returns table (id varchar, total number(10,2))</pre><p>Second, you need to specify a function handler whose code is written in one of the supported languages.</p><h4>Supported Handler Languages</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/281/1*std2ON-JTawWeLpNlGMKHg.png" /><figcaption>supported languages and handler locations</figcaption></figure><pre>create function function_name(x integer, y integer)<br>returns integer<br>LANGUAGE [ PYTHON | SCALA | JAVA | JAVASCRIPT ]<br>RUNTIME_VERSION = &#39;3.8&#39;  ##required only when language is Python or Scala##</pre><p>For SQL, you need not define the language or runtime version. You can write your code directly using AS clause.</p><pre>CREATE FUNCTION multiply1 (a number, b number)<br>  RETURNS number<br>  COMMENT=&#39;multiply two numbers&#39;<br>  AS &#39;a * b&#39;;</pre><p>Depending on the language, you can include the handler code directly in the CREATE FUNCTION statement or reference its location if it&#39;s precompiled or staged.</p><h4>Type of Handler Locations</h4><ol><li><strong>Staged Handler: </strong>in an external location</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*LrO6s9dBtgulo3moyPvBWg.png" /><figcaption>Handler in a stage location</figcaption></figure><p>For a staged handler, you need to make your dependency file available to the UDF, and for that, the file needs to be on a stage where it can be reached at runtime.</p><ol><li>Choose or create a stage for your dependency: Remote git repository/ user or named internal stage/ external stage</li><li>Upload the files required for your function handler to a stage.</li><li>Specify the dependency’s location in the IMPORTS clause of the SQL you used to create the function just like we did in the image above.</li></ol><p>Advantages:</p><ul><li>For large handler code that exceeds in-line limits, use an external source rather than pasting it into the SQL statement.</li><li>Staged code can contain multiple handler functions in which each function can be used by a different UDF or procedure.</li><li>Can use previously compiled code, such as when you already have compiled output but don’t have the source.</li></ul><p>2. <strong>Inline Handler: </strong>within the UDF code</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*LEeAv7ProPWUZB2GQc3F0g.png" /><figcaption>img 3: handler defined in the UDF itself</figcaption></figure><p>Considerations:</p><ul><li>In-line handlers are easy to manage and update directly in SQL statements; code is captured as metadata.</li><li>Use staged handlers if you prefer not to capture code as metadata; deploy using external stages.</li></ul><h4>UDF Body</h4><ul><li>The delimiters around the function definition can be either single quotes or a pair of dollar signs.</li><li>Using $$ as the delimiter makes it easier to write functions that contain single quotes.</li><li>If the delimiter for the body of the function is the single quote character, then any single quotes within function definition (e.g. string literals) must be escaped by single quotes.</li></ul><h3>Calling a UDF</h3><ul><li>You call a UDF same way that you call other functions. If it has arguments, you can specify those arguments by name or by position.</li></ul><pre>SELECT multiply1 (2,3)</pre><ul><li>You can call a UDTF the way you would call any table function. When calling a UDTF in the FROM clause of a query, specify the UDTF’s name and arguments inside the parentheses that follow the TABLE keyword.</li></ul><pre>SELECT ...<br>  FROM TABLE ( udtf_name (udtf_arguments) )</pre><h4>Required Privileges</h4><p>To perform SQL actions on a UDF or UDTF, the person performing the action must have been assigned a role that has been granted the required privileges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*PX-EcrLXMYBAObxvWs2uwg.png" /><figcaption>UDF Privileges</figcaption></figure><h4>Secure UDF</h4><p>To help ensure that sensitive information is concealed from users who should not have access to it, you can use the SECURE keyword when creating a user-defined function (UDF) and stored procedure.</p><p>For a secure function or procedure, unauthorized users won’t see the body, imports, handler name, or packages. They will still see parameter types, return type, handler language, null handling, and volatility.</p><h4>References</h4><p>If you want to understand the features and configurations available for supported handler languages for UDF, you can check the below link.</p><p><a href="https://docs.snowflake.com/en/developer-guide/udf/udf-overview.html#label-udf-supported-languages">https://docs.snowflake.com/en/developer-guide/udf/udf-overview.html#label-udf-supported-languages</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=702db33ad49c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Snowflake’s Universal Search: Simplified Data Retrieval]]></title>
            <link>https://medium.com/@deepti.agl2912/snowflakes-universal-search-simplified-data-retrieval-c84a0b41878d?source=rss-abd297954d5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c84a0b41878d</guid>
            <category><![CDATA[cortex]]></category>
            <category><![CDATA[universal-search]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Deepti Agrawal]]></dc:creator>
            <pubDate>Fri, 26 Jul 2024 10:00:41 GMT</pubDate>
            <atom:updated>2024-07-26T10:00:41.613Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W1tSU0odtowIZatwpQzR5Q.png" /></figure><p>When we have questions, we often turn to Google Search for answers. But when it comes to extracting insights from your data, where should you go?</p><p>Just as Google helps navigate the vast internet, <strong>Snowflake’s Universal Search</strong> guides you through your data landscape. It offers a seamless way to discover the information you need, making data exploration as intuitive as a web search. With Universal Search, unlocking your data’s potential has never been easier.</p><h4><strong>What can you search?</strong></h4><ul><li><strong>Objects in Your Account:</strong> Dashboards, worksheets, databases, schemas, tables, views, user-defined functions, and stored procedures.</li><li><strong>Data Products:</strong> Listings from the Snowflake Marketplace.</li><li><strong>Documentation:</strong> Snowflake docs and community knowledge base articles.</li></ul><h4>Capabilities?</h4><p>Universal Search is designed to understand your queries, even if you make typos or only type part of your search term.</p><p>You can use natural language, so phrases like “sales opportunities that are likely to close” work just as well as simple keywords. It searches through metadata, including names, comments, and tags, ensuring you can find what you need based on how things are labeled.</p><h4><strong>How to get started?</strong></h4><p>To start a search, first log in to <strong>Snowsight</strong> and click on the “Search” option in the menu. Enter your search terms and press Enter. The results will be displayed in different categories. If a category isn’t shown, it means there are no results for that category, or you may not have the required access.</p><p><strong>Note</strong>: You only see results that your currently active role has privileges to view.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3So7m_RK9feKQoX24FsXPQ.png" /><figcaption>search resulted into marketplace listings and documentation</figcaption></figure><h4>Usage Considerations</h4><ul><li>Universal Search is optimized for search terms in English.</li><li>New or recently modified objects may take a few hours to show up in the search results.</li><li>Objects that are dropped and recreated might temporarily disappear from search results.</li><li>Universal Search is not yet available in Virtual Private Snowflake (VPS) or government cloud regions.</li><li>To access Snowflake Marketplace listings, you must agree to specific terms and conditions.</li><li>Search results are secure and align with your data access permissions.</li></ul><h4>Conclusion</h4><p>Discovering data in Snowflake has never been easier. With Universal Search, you can quickly and effortlessly locate the information you need, minimizing time spent searching and maximizing time spent on analysis.</p><p>Experience a new level of efficiency in your data exploration — try Universal Search today and see how it transforms the way you interact with your data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c84a0b41878d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>