<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Prash Medirattaa on Medium]]></title>
        <description><![CDATA[Stories by Prash Medirattaa on Medium]]></description>
        <link>https://medium.com/@prashmedirattaa?source=rss-14293f8d4549------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*YhyBG5oDowAqYcwfeRRNEg.jpeg</url>
            <title>Stories by Prash Medirattaa on Medium</title>
            <link>https://medium.com/@prashmedirattaa?source=rss-14293f8d4549------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 03:25:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@prashmedirattaa/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Announcing secure AI-Assisted data science in R with Posit and Snowflake Cortex]]></title>
            <link>https://medium.com/snowflake/announcing-secure-ai-assisted-data-science-in-r-with-posit-and-snowflake-cortex-724104c6da37?source=rss-14293f8d4549------2</link>
            <guid isPermaLink="false">https://medium.com/p/724104c6da37</guid>
            <category><![CDATA[posit]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[cortex]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Prash Medirattaa]]></dc:creator>
            <pubDate>Tue, 18 Mar 2025 14:02:40 GMT</pubDate>
            <atom:updated>2025-03-18T14:02:40.735Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*F3VulX6nvswSjsLbu6IHcA.png" /><figcaption>Posit + Snowflake: AI &amp; Analytics with Cortex</figcaption></figure><blockquote><em>Credits </em>Posit Team : <a href="https://www.linkedin.com/in/chetanthapar/">Chetan Thapar</a>, <a href="https://www.linkedin.com/in/garrett-grolemund-49328411/">Garrett Grolemund,</a> <a href="https://www.linkedin.com/in/blairjm/">James Blair</a></blockquote><h4><strong>The Promise of AI in Data Science Workflows, Now Made Seamless and Secure</strong></h4><p>By now, you probably know that AI is changing software development, making developers faster and more efficient. Developers can use Large language Models (LLMs) to suggest, complete, and improve their code as they write it. You may have tried to do something like this for your R-based data science workflows, using ChatGPT or another LLM alongside your data science code. If so, you probably noticed a few short-comings:</p><ol><li>Context switching fatigue</li><li>Security concerns</li><li>Missing context, the LLM doesn’t know about your project, so it fills its suggestions with dummy variables and pseudo code that you have to strip out and replace.</li></ol><p>Is it possible to have a seamless and secure AI assistant for R-based data science? One that knows that you are trying to analyze data — not develop software, and tailors its suggestions accordingly? Yes, it is, with the Posit Native App on Snowflake SPCS, combined with the<strong> </strong><a href="https://simonpcouch.github.io/gander/"><strong><em>gander</em></strong></a><em> and</em> <strong><em>chores</em></strong> R packages, and Snowflake Cortex. These things are “Better Together,” providing a frictionless workflow for data science, with robust security and context-aware AI assistance.</p><h4><strong>Why Data Scientists Need a “Better Together” Approach</strong></h4><p>So far, there has been no AI tool dedicated to R-based data science workflows. To take full advantage of AI to write code, R users have had to combine disparate tools, which creates some big disadvantages:</p><ul><li><strong>Context Switching Fatigue:</strong> It is frustrating to move between different environments and tools to leverage LLMs. The constant copy-and-pasting requires us to be detail oriented and fastidious in the wrong places, disrupting our workflow and sapping productivity. We want code suggestions to appear directly within our source code and to accept them with the press of a key.</li><li><strong>Security and Governance Headaches: </strong>LLMs can use the information we provide to train their models, which means the information can reappear in surprising places, which is a security nightmare. It is challenging to know whether we are using an LLM in a way that complies with corporate policy.</li><li><strong>AI Knowledge Limitations:</strong> Does your AI know the column names of your data frame? The filenames within your project? Generic AI tools usually lack the context needed to provide accurate suggestions within a data science project. Instead they provide plausible but error prone suggestions: code that won’t run until you spend time adjusting it.</li><li><strong>Boilerplate Burden:</strong> Chatbots cannot do repetitive tasks that appear throughout your project, like writing documentation. To do things like this, you need to run your AI <em>inside</em> of your project, where it can be context aware and adjust your code at the source.</li></ul><p>You can avoid these limitations and do powerful AI assisted data science by combining the Posit Native App with Snowflake Cortex.</p><h4><strong>Posit + Snowflake Cortex — A Seamless Solution</strong></h4><p>This winning combination has three ingredients:</p><p>1.<strong>The Posit Native App on Snowpark Container Services (SPCS)</strong> is a unified environment for data science, directly within Snowflake. To open the app, go to Data Products &gt; Apps in your Snowflake account sidebar. Then click on the Posit Native App. If the app is not there, ask your account administrator to <a href="https://docs.posit.co/partnerships/snowflake/workbench/native-app/install.html">install it</a>.</p><p>The Posit Native App lets you get immediately up and running within SPCS as a data scientist, launching your favorite data science IDEs like RStudio Pro, VS Code, and JupyterLab, or Jupyter Notebooks. Posit Native App is ready to use “out of box” meaning faster time to value, and is a managed software meaning reduced ops headaches.</p><p>2.<strong> Snowflake Cortex</strong> gives you instant access to industry-leading LLMs trained by researchers at companies like Anthropic, Mistral, Reka, Meta, and Google.</p><p>Because these LLMs are fully hosted and managed by Snowflake, using them requires minimal setup. Your data stays within Snowflake, giving you the performance, scalability, and governance you expect.</p><p>3.<strong> Gander and chores</strong> are two R packages that use LLMs, like those provided by Snowflake Cortex, to assist data science workflows performed in the RStudio IDE. You could use gander and chores in the local RStudio Desktop IDE or in the RStudio Pro IDE provided by the Posit Native App.</p><p>a. <a href="https://simonpcouch.github.io/gander/">gander</a> is a chat and code completion experience, like Copilot, that knows how to query your R environment to write data science code that is ready to use.</p><p>b. <a href="https://simonpcouch.github.io/chores/">chores</a> provides a library of ergonomic LLM assistants that help you quickly complete repetitive, hard-to-automate tasks.</p><p>Using gander and chores with the Posit Native App in SPCS is a natural choice because:</p><p>Using gander and chores with the Posit Native App in SPCS is a natural choice because:</p><ol><li><strong>You can start using AI immediately with no friction.</strong> Both <a href="https://simonpcouch.github.io/gander/"><strong>gander</strong></a> and <a href="https://github.com/simonpcouch/pal"><strong>chores</strong></a> are pre-packaged with the Posit Native App, and pre-configured to use Snowflake Cortex from within SPCS.</li><li><strong>You stay secure and compliant because data never leaves your Snowflake ecosystem.</strong> Since gander and chores use Snowflake Cortex to access an LLM, each interaction follows Snowflake’s strict governance guardrails — ensuring no external training or unauthorized usage of data.</li><li><strong>You reduce the friction of context switching. </strong>Both gander and chores can seamlessly interact with Cortex models through the press of a keyboard shortcut in RStudio/Positron. Moreover, they make Enterprise-scale LLM capabilities available <em>within </em>your R IDE, you do not need to switch between two contexts to use them.</li><li><strong>You receive better, more targeted suggestions from your AI, which reduces the time spent on iteration and refinement. </strong>gander automatically includes relevant project context — such as code snippets and environmental details — in prompts to the LLM, leading to more accurate and useful AI-generated outputs. As a concrete example, gander provides the LLM the name and columns of each data frame, so the LLM can suggest code that works with your data, even when you haven’t yet explicitly referenced the names in your session.</li></ol><p>Together, the Posit Native App, gander and chores, and Snowflake Cortex create a frictionless, secure, and context-aware AI-assisted development workflow in R, all within the Snowflake data governance ecosystem. And since gander and chores create code as output, results remain consistent, auditable, and reproducible across teams.</p><h3>Let’s see it in action</h3><p>To see how easy it is to use these tools together, let’s use gander to generate code that creates a data visualization.</p><p>First, we open Posit Native Workbench and launch an RStudio Pro session. As we do, we click the middle box on the launch screen to provide our Snowflake Credentials.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EIhuJFKJDGBljLaJ" /></figure><p>Then we begin a data science project. We load gander and the tidyverse, which come pre-installed with the Posit Native app. Then we tell gander which LLM to use.</p><p>Many LLMs are available through Snowflake Cortex. To tell gander to use a specific LLM, like Claude 3.5 Sonnet, set the `.gander_chat` option.</p><pre>library(tidyverse)<br>library(gander)<br><br>options(.gander_chat = ellmer::chat_snowflake(model = &quot;claude-3-5-sonnet&quot;))</pre><p>Next, we use the tidyverse to connect to a Snowflake Warehouse and access a table in the `HEALTHCARE` catalog. This is a streamlined process because we are inside the Posit Native App. We don’t need to pass along Snowflake credentials; the Native App provides them for us. The Posit Native App will also automatically provide these credentials when gander accesses Snowflake Cortex making it as easy to use Cortex LLMs as it is to access Snowflake data in the native app!</p><pre><br>con &lt;- DBI::dbConnect(<br>  odbc::snowflake(),<br>  warehouse = &quot;DEFAULT_WH&quot;<br>)<br><br>heart_data &lt;- tbl(con, dbplyr::in_catalog(&quot;HEALTHCARE&quot;, &quot;PUBLIC&quot;, &quot;HEART_DATA&quot;))<br><br>heart_data |&gt; <br>  head()<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Vz8gZQoR8dplZYQ4" /></figure><p>It is time to write code to make a visualization. We move our cursor to where we want to insert code, and open gander.</p><p>The gander package adds a chat interface to the RStudio IDE as an add-in. We can access this add-in by opening the add-ins menu. We can also create a keyboard shortcut to launch the add-in (explained below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1FFRlIBBVKQ_3Ygt" /></figure><p>We can now tell gander to “make a plot of age and serum sodium using the heart data.” gander writes the code and inserts it into our file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sh8MoDoUE69rZxvx" /></figure><p>Gander receives a snapshot of our environment with every request and recognizes that we are talking about the `heart_data` dataset. It also looks up the column names `age` and `serum_sodium` and uses them within our code. gander even realizes that `heart_data` is a connection to a table (and not the table itself), and adds a `collect()` command to import the data, so it can be plotted. This is a phenomenal level of context awareness, and surpasses what is available in AI assistants like CoPilot, which are not built for data science specific use cases.</p><p>This is great. Now we want to iterate.</p><p>We highlight our plot code and ask gander to update it to include a title and annotation line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*paHhcc9PuDmB52xa" /></figure><h3>Behind the scenes</h3><p>Gander works by collecting our prompt, augmenting it, and sending the result to Snowflake Cortex. You can inspect what gander sends with `gander_peek()`, making gander a flexible and transparent AI experience.</p><pre>gander_peek()</pre><p><strong>The output of `gander_peek()` contains:</strong></p><ol><li>The number of tokens used</li><li>The system prompt gander sends to the LLM</li><li>The user prompt gander sends to the LLM, which includes your prompt, descriptions of your environment and the contents of your source file</li></ol><h3>Customize behavior</h3><p>By default, gander adds the following system prompt to your query</p><p>You are a helpful but terse R data scientist. Respond only with valid R code: no exposition, no backticks. Always provide a minimal solution and refrain from unnecessary additions. <em>Use tidyverse style and, when relevant, tidyverse packages. For example, when asked to plot something, use ggplot2, or when asked to transform data, using dplyr and/or tidyr unless explicitly instructed otherwise.</em></p><p>You can replace the section in green to change gander’s behavior with <a href="https://simonpcouch.github.io/gander/reference/gander_options.html#style-taste">the `.gander_style` option</a>. For example, running the code below</p><pre>options(.gander_style = &quot;Use base R.&quot;)</pre><p>will update the system prompt to</p><p>You are a helpful but terse R data scientist. Respond only with valid R code: no exposition, no backticks. Always provide a minimal solution and refrain from unnecessary additions. Use base R.</p><h3>Choose a model</h3><p>Many LLMs are available through Snowflake Cortex. To tell gander to use a specific LLM, like Claude 3.5 Sonnet, set the `.gander_chat` option.</p><pre>options(.gander_chat = ellmer::chat_snowflake(model = &quot;claude-3-5-sonnet&quot;))</pre><h3>Create a keyboard shortcut</h3><p>To create a keyboard shortcut that opens the gander chat interface, go to Tools &gt; Keyboard Shortcuts to add a custom keyboard shortcut to open the gander chat.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oStvvccgCUyTVr0K" /></figure><p>Now we can launch the chat interface by pressing Ctrl + Cmd + g. This shortcut will be available to us in every RStudio Pro session that we launch with the Posit Native App.</p><h3>Recap: The Sum is Greater Than Its Parts</h3><p>The combined power of Posit, gander, and Snowflake Cortex delivers a superior AI-assisted data science experience, with:</p><ul><li>Immediate access to data science AI tools within Snowflake.</li><li>Enterprise-grade security and governance.</li><li>An unparalleled frictionless AI workflow for data science</li><li>Truly context-aware and intelligent AI assistance.</li></ul><p>Accelerate development across your R-based workflows with this “better together” combo.</p><p>Go <a href="https://posit.co/use-cases/snowflake/">here</a> to request a personalized demo of Posit Native App on Snowflake and see the ‘Better Together’ story in action!</p><p><strong>Want to try this locally?</strong></p><p>Gander and chores also work in your local RStudio Desktop IDE. Follow the installation instructions <a href="https://simonpcouch.github.io/gander/#installation">here</a>.</p><h3>Links to relevant resources</h3><ul><li><a href="https://app.snowflake.com/marketplace/listing/GZTSZMCB69/posit-pbc-posit-workbench">Posit Native App homepage</a></li><li><a href="https://simonpcouch.github.io/gander/">gander R package site</a></li><li><a href="https://simonpcouch.github.io/chores/">chores R package site</a></li><li><a href="https://docs.snowflake.com/en/user-guide/snowflake-cortex/llm-functions">Snowflake Cortex documentation</a></li><li><a href="https://posit.co/conversion-general/">Posit contact form</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=724104c6da37" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/announcing-secure-ai-assisted-data-science-in-r-with-posit-and-snowflake-cortex-724104c6da37">Announcing secure AI-Assisted data science in R with Posit and Snowflake Cortex</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[In today’s data-driven world, the ability to seamlessly integrate high-performance databases with…]]></title>
            <link>https://medium.com/snowflake/in-todays-data-driven-world-the-ability-to-seamlessly-integrate-high-performance-databases-with-b0da165dd6fe?source=rss-14293f8d4549------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0da165dd6fe</guid>
            <category><![CDATA[snowflake-data-cloud]]></category>
            <category><![CDATA[singlestore]]></category>
            <category><![CDATA[snowpark-container]]></category>
            <category><![CDATA[native-app]]></category>
            <category><![CDATA[snowpark]]></category>
            <dc:creator><![CDATA[Prash Medirattaa]]></dc:creator>
            <pubDate>Thu, 27 Feb 2025 15:32:15 GMT</pubDate>
            <atom:updated>2025-02-27T15:32:15.530Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Unlocking Real-Time Analytics: Connecting SingleStore to Snowflake with Snowpark Container Services</strong></h3><p>In today’s data-driven world, the ability to seamlessly integrate high-performance databases with scalable data warehousing solutions is crucial. Enter <strong>SingleStore and Snowflake</strong> — a powerhouse combination that bridges real-time analytics with enterprise-scale data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3YYj5eU2sAucrpWM" /></figure><p>In today’s data-driven world, businesses need lightning-fast access to their data to power real-time analytics and decision-making. While <strong>Snowflake </strong>offers powerful cloud data warehousing and scalability, pairing it with <strong>SingleStore </strong>as a real-time or caching layer can take performance to the next level. By integrating SingleStore with Snowflake and your existing <strong>data lake architecture</strong>, companies can achieve millisecond query response times at scale, unlocking instant insights and driving smarter decisions.</p><p>This tutorial is designed to help you seamlessly integrate SingleStore’s real-time database capabilities with Snowflake, enabling you to query your existing data and accelerate your analytics workflows. By the end of this guide, you’ll have SingleStore up and running within Snowpark Container Services, and you’ll run SQL queries on Snowflake’s sample data using the SingleStore native app.</p><p>This integration opens the door to real-time data insights and helps bridge the gap between Snowflake’s data warehousing capabilities and SingleStore’s high-performance database functionalities. Throughout this tutorial, we’ll walk you through the following:</p><ul><li>Installing the SingleStore application</li><li>Setting up your SingleStore database cluster</li><li>Accessing SingleStore Studio</li><li>Creating a SingleStore ingestion pipeline</li><li>Querying row counts to verify data parity between SingleStore and Snowflake</li></ul><h4>Prerequisites</h4><p>Before we get started, there are a few things you need to make sure you have set up.</p><ol><li>Snowflake Account</li></ol><ul><li>You need an active Snowflake account to complete this tutorial.</li></ul><p>2. Data Preparation</p><ul><li>Familiarize yourself with loading data and creating Iceberg tables in Snowflake. If you haven’t done this yet, go through the Create your first Apache Iceberg™ table tutorial <a href="https://docs.snowflake.com/en/user-guide/tutorials/create-your-first-iceberg-table#introduction">here</a>. Complete all the steps through <a href="https://docs.snowflake.com/en/user-guide/tutorials/create-your-first-iceberg-table#load-data-and-query-the-tables">load and query data</a> (Don’t worry we’ll walk you through clean up at the end of this tutorial).</li></ul><p>3. SingleStore Native App Access</p><p>Request access to the SingleStore Native App by following these steps</p><ul><li>Visit the<a href="https://app.snowflake.com/marketplace/listing/GZTYZG4WEWA/singlestore-real-time-database-for-intelligent-applications?search=singlestore"> SingleStore Real-time Database for Intelligent Applications</a> listing on the Snowpark Marketplace.</li><li>On the product page, click <strong>Request</strong> to display a form. Fill out the required information and click <strong>Request</strong>.</li></ul><p>Once approved, the SingleStore app will appear under <strong>Data Products &gt; Apps</strong>.</p><p>4. Install the SingleStore App</p><p>Navigate to <strong>Data Products &gt; Apps &gt; Installed Apps</strong> and click <strong>Get</strong> to download the SingleStore app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/0*CjNNc9sPDnS_3WBl" /></figure><p>A confirmation dialog will appear. Select the appropriate Snowflake warehouse to install the app and click <strong>Get</strong>.</p><p>5. Grant Permissions and Activate the App</p><ul><li>You should now see a page asking you to activate the app. Review and grant the requested account privileges.</li><li>Under <strong>Allow Connections</strong>, click <strong>Review</strong>, then connect the app. Finally, click <strong>Activate</strong> to complete the setup.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*a4hiC7bWDHxnzuUc" /></figure><p>Once you’ve completed these prerequisites, you’re ready to proceed to the next steps!</p><h4>Cluster Setup</h4><p>Now that the SingleStore app is fully installed, you should see a <strong>Launch app</strong> button. Go ahead and click that and select a warehouse if prompted. After a few seconds, you should be greeted with a welcome page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*if2ScU7lDqvkNy9V" /><figcaption>Click on the <strong>Start Dev Cluster</strong> button to begin the cluster creation process. For the cluster name choose S2_DEMO.</figcaption></figure><p>For cluster size, by default we offer a <strong>Dev</strong> and <strong>Small</strong> size. You can check the documentation page <a href="https://docs.singlestore.com/db/v8.9/deploy/get-started-with-the-singlestore-snowpark-app/">here</a> for specifics on the differences between these two sizes. Larger cluster sizes are available on request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5p7-CUfYZVyqYpJK" /></figure><p>Next, set a root password for the cluster. This will be the password for the actual root database user that you will use to access and execute queries against the cluster. As the note suggests, the password cannot be reset by anyone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vSx0Ni3idua6pNRY" /><figcaption>Once you’re ready click <strong>Create Cluster</strong>. Please wait several minutes for the cluster to spin up. You will see all the services being spun up, as well as a progress bar.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7h2BWA93dRRVn5iL" /></figure><p>Once everything is done, you should see the following screen with the cluster status as well as the connection details for your cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3oxL1mvzNqj20rps" /></figure><h4>A Tour of SingleStore Studio</h4><p>On the bottom of the cluster details page you should see an <strong>Advanced </strong>tab. There you should see two links. Click on <strong>Go to SingleStore Studio</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/0*1zJ5qCtcPgG_huip" /></figure><p>It will prompt you to login with your Snowflake account credentials. If you get a connection reset error, try opening the link in private browsing mode.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X2QBUtYtU5jpbA0k" /></figure><p>Click on the SingleStore cluster you just created.</p><p>Once the page is open you will then be asked to login to the SingleStore cluster. Use “root” as the user name and the password that you created in the earlier cluster setup step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CffiNg9q1x9N-VMi" /></figure><p>You should now be looking at the SingleStore Studio dashboard. The studio provides an all-in-one place to monitor the status of the cluster, as well as execute queries against your databases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*w0v6LDFUhqi4maJV" /></figure><h4>Pipeline Configuration</h4><p>In order to ingest data from our customer_iceberg table in Snowflake, we will use <a href="https://www.singlestore.com/pipelines/">SingleStore Pipelines</a>. Pipelines are a group of SingleStore features that enable <a href="https://docs.singlestore.com/cloud/load-data/load-data-with-pipelines/">loading data</a> as it arrives from external sources. As a built-in component of the database, Pipelines can extract, shape (modify), and load external data without the need for third-party tools or middleware. Pipelines are robust, scalable, highly performant, and support fully distributed workloads.</p><p>Before we can create a pipeline, we need to get a few things ready to establish a connection to our Snowflake account and our Iceberg catalog.</p><p><strong>&lt;account-id&gt; — </strong>This is your Snowflake account identifier. It should be in the form of your account name and organization separated by a dash. You can find it at the bottom right of your Snowflake portal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XrzFHEWg_zE3iU-e" /></figure><p><strong>&lt;snowflake-user&gt; — </strong>This is your Snowflake user. You can find this by running SELECT CURRENT_USER() in the Snowflake SQL editor.</p><p><strong>&lt;snowflake-password&gt; — </strong>This is the password for your Snowflake account.</p><p><strong>&lt;snowflake-role&gt; — </strong>This is the role used by your Snowflake account. You can find this by running SELECT CURRENT_ROLE() in the Snowflake SQL editor.</p><p>Once you have the above variables, we need to give SingleStore access to the S3 bucket where our Iceberg catalog is located. We will create the following two variables.</p><p><strong>&lt;aws_access_key_id&gt; — </strong>This is the AWS key id used to access the S3 bucket where your Iceberg data is stored.</p><p><strong>&lt;aws_secret_access_key&gt; — </strong>This is the AWS secret key used to access the S3 bucket where your Iceberg data is stored.</p><p><strong>Root User:</strong></p><p>Within the AWS Console, on the top right hand corner, you should see your username. If you hover over your username, a drop down menu will appear and you should see the option for “Security Credentials”. Click on “Security Credentials” and you will be brought to a new page.</p><p>On that page, you will navigate down to where you see “Access Keys”. On the right hand side of that box, you will find and click on “Create access keys”. This will bring you to a new page that will alert you to the use of your root account. Click on the check box and continue to create the keys.</p><p>Once you’re done, you should see an option to download a CSV file with the access_key_id and the secret_access_key. Keep these handy as we’ll use them to configure our SingleStore Pipeline.</p><h4>SingleStore Table Setup</h4><p>Navigate to SingleStore’s <strong>SQL Editor</strong> page, located in the studio sidebar. Here you can execute any SQL query that SingleStore <a href="https://docs.singlestore.com/cloud/reference/sql-reference/">supports</a>. At the top you will see a dropdown where you can optionally select a database to execute queries against.</p><p>Since we just created this cluster, the only database available will be information_schema.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KdlGxhPDdXN_w32s" /></figure><p>Let’s start by creating a SingleStore database and table that we can use to pull in our Snowflake data into. We will be bringing in the customer data from the customer_iceberg table that you should have created in the prerequisite tutorial.</p><pre>CREATE DATABASE SPCS_DEMO;<br>USE SPCS_DEMO;<br><br>CREATE TABLE customer (<br>C_CUSTKEY INTEGER, <br>C_NAME TEXT, <br>C_ADDRESS TEXT, <br>C_NATIONKEY INTEGER, <br>C_PHONE TEXT, <br>C_ACCTBAL INTEGER, <br>C_MKTSEGMENT TEXT, <br>C_COMMENT TEXT<br>);</pre><p>Now that we have a customer table in our SingleStore cluster, we can create a pipeline to ingest data from our customer_iceberg table in Snowflake.</p><h4>Pipeline Execution</h4><p>Run the following queries in the SingleStore SQL Editor. Make sure to update the configuration and credential values to the values in the Pipeline Configuration section.</p><pre>SET GLOBAL enable_iceberg_ingest = ON;<br>SET global pipelines_extractor_get_offsets_timeout_ms = 600000;<br><br>CREATE OR REPLACE PIPELINE customer_iceberg AS<br>LOAD DATA S3 &#39;schema_name_from_snowflake&gt;.PUBLIC.&lt;table_name_from_snowflake&gt;&#39;<br>CONFIG &#39;{&quot;region&quot; : &quot;us-west-2&quot;,<br>        &quot;catalog_type&quot;: &quot;SNOWFLAKE&quot;,<br>        &quot;catalog.uri&quot;: &quot;jdbc:snowflake://&lt;snow acct&gt;.snowflakecomputing.com&quot;,<br>        &quot;catalog.jdbc.user&quot;:&quot;&lt;Snowflake username&quot;,<br>        &quot;catalog.jdbc.password&quot;:&quot;&lt;Snowflake password&gt;&quot;,<br>        &quot;catalog.jdbc.role&quot;:&quot;ACCOUNTADMIN&quot;}&#39;<br>CREDENTIALS &#39;{&quot;aws_access_key_id&quot; : &quot;&lt;AWS credentials&gt;&quot;,<br>             &quot;aws_secret_access_key&quot;: &quot;&lt;AWS credentials&gt;&quot;}&#39;<br>INTO TABLE customer (<br>    C_CUSTKEY &lt;- C_CUSTKEY,<br>    C_NAME &lt;- C_NAME,<br>    C_ADDRESS &lt;- C_ADDRESS,<br>    C_NATIONKEY &lt;- C_NATIONKEY,<br>    C_PHONE &lt;- C_PHONE,<br>    C_ACCTBAL &lt;- C_ACCTBAL,<br>    C_MKTSEGMENT &lt;- C_MKTSEGMENT,<br>    C_COMMENT &lt;- C_COMMENT<br>)<br>FORMAT ICEBERG;<br><br>test pipeline customer_iceberg;<br><br>start pipeline customer_iceberg;<br><br></pre><p>We first set the enable_iceberg_ingest engine variable to be ON since it’s OFF by default. Then we create a pipeline definition called customer_iceberg that uses a Snowflake Iceberg catalog stored on an AWS S3 bucket.</p><p>The body of the pipeline defines which table our data is being ingested to as well as which columns from the Iceberg table will map to which columns of our SingleStore table. For more information on value mapping, check out our Iceberg ingestion documentation <a href="https://docs.singlestore.com/cloud/load-data/load-data-with-pipelines/how-to-load-data-using-pipelines/iceberg-ingest/">here</a>.</p><p>Lastly, we start the pipeline and wait for it to finish running. The FOREGROUND at the end just means that we want the query to return only once the pipeline has ingested all the data, rather than the pipeline running asynchronously in the background.</p><h4>Validation</h4><p>Now that all our customer data has been ingested, we can verify that we properly ingested all the correct rows by running a simple count query.</p><pre>SELECT COUNT(*) FROM customer;</pre><p>You should see that the returned row count should match the original row count from the customer_iceberg table in Snowflake!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/0*Nf6fYuW2cyS7d7jI" /></figure><p>You now have a full copy of your Snowflake data inside SingleStore that you can query. One thing to note is that SingleStore uses a SQL <strong>dialect similar to MySQL</strong> so some of your existing Snowflake SQL queries may need to be modified to run correctly in SingleStore. Check out our <a href="https://docs.singlestore.com/cloud/reference/sql-reference/">SQL reference</a> for more information.</p><h4>Cleanup</h4><p>To delete all the objects created in this tutorial, simply go back to the SingleStore Native App page and click the <strong>Delete Cluster</strong> button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/0*2eQDw_J7cr4RJWQh" /></figure><p>Once it’s done you will be navigated back to the welcome page. On the Snowflake side, simply pick back up at the <a href="https://docs.snowflake.com/en/user-guide/tutorials/create-your-first-iceberg-table#delete-rows">Delete Rows step of the Snowflake Iceberg tutorial</a> where you will be guided through dropping the Iceberg table, external volume, database, and warehouse that were used.</p><h4>Conclusion</h4><p>You should have successfully installed and activated the SingleStore Native App, set up your database cluster, learned how to navigate SingleStore Studio, created an Iceberg ingestion pipeline, and verified data parity with Snowflake. These steps provide you with a robust foundation to leverage the power of SingleStore’s real-time analytics capabilities alongside Snowflake’s data warehousing features.</p><p>Stay tuned for the next part of our series, where we will explore how to build and deploy applications to SPCS that can connect to our SingleStore cluster.</p><p><strong>Further Reading:</strong></p><ul><li><a href="https://www.singlestore.com/pipelines/">SingleStore Pipelines</a></li><li><a href="https://docs.snowflake.com/en/user-guide/tutorials/create-your-first-iceberg-table">Snowflake Iceberg Tables</a></li><li><a href="https://docs.singlestore.com/cloud/reference/sql-reference/">SingleStore SQL Reference</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0da165dd6fe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/in-todays-data-driven-world-the-ability-to-seamlessly-integrate-high-performance-databases-with-b0da165dd6fe">In today’s data-driven world, the ability to seamlessly integrate high-performance databases with…</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Neo4j and Snowflake: Unlocking the Power of Graph Databases to Transform Data Insights — A True…]]></title>
            <link>https://medium.com/snowflake/neo4j-and-snowflake-unlocking-the-power-of-graph-databases-to-transform-data-insights-a-true-d6831912bc8f?source=rss-14293f8d4549------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6831912bc8f</guid>
            <dc:creator><![CDATA[Prash Medirattaa]]></dc:creator>
            <pubDate>Mon, 09 Dec 2024 21:37:48 GMT</pubDate>
            <atom:updated>2024-12-09T21:37:48.980Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Neo4j and Snowflake: Unlocking the Power of Graph Databases to Transform Data Insights — A True Game-Changer</strong></p><p><strong>Introduction</strong></p><p>Snowflake, the leading cloud data platform, and Neo4j, the premier graph database, have partnered to revolutionize data analytics. This integration, now available in the Snowflake Marketplace, combines Snowflake’s capabilities in large-scale analytics with Neo4j’s expertise in uncovering complex relationships. Together, Neo4j and Snowflake enable organizations to model and analyze complex relationships within data, uncover hidden patterns, and gain deeper insights. Graph databases like Neo4j are perfect for addressing challenges such as fraud detection, recommendation engines, supply chain optimization, network analysis, and customer behavior analysis. By efficiently managing interconnected data, Neo4j enhances Snowflake’s cloud capabilities, empowering businesses to make data-driven decisions, optimize processes, and unlock new opportunities across industries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OUzCQU8vSJYsnD-bsNglSA.png" /><figcaption>Snowflake and Neo4j</figcaption></figure><p>This integration unlocks a wide range of use cases by enabling Snowflake to natively run the GDS library across multiple platforms. The process unfolds through a series of steps, supporting most of the official Neo4j Graph Data Science <a href="https://neo4j.com/docs/graph-data-science/current/operations-reference/algorithm-references/">algorithms</a>, ensuring a seamless and efficient execution of graph analytics within Snowflake</p><p><strong>Steps in snapshot</strong></p><ol><li><strong>Download Neo4j from the Marketplace</strong></li><li><strong>Identify Your Use Case</strong></li><li><strong>Configure Neo4j </strong>: Set up your database and schema</li><li><strong>Explore GDS for Snowflake API</strong>: Overview and details</li><li><strong>Upload the Dataset</strong></li><li><strong>Run the App</strong>: Execute the use case for Data Insights in Fraud Detection</li><li><strong>Execute Algorithms</strong>: Apply Louvain, PageRank, and WCC for Entity Resolution</li><li><strong>Write Results Back to Snowflake</strong></li><li><strong>Utilize Snowflake Notebooks </strong>for <em>entity Resolution &amp; Fraud Labels</em></li><li><strong>Run ML Models</strong>: Coming soon</li></ol><p><strong><em>Step 1: Getting started download the app from marketplace</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PZa7ShSePYLokK2BZm5vmw.jpeg" /><figcaption>Searching on marketplace</figcaption></figure><p>Lets get the app</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kM3PMBpPXNwLnEQWcZT3mg.png" /><figcaption>installing the App</figcaption></figure><p>App is installed on Snowflake and ready to use with necessary privileges</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U9OtXsd6DtDf2eWk7EJZTQ.png" /></figure><p><strong><em>Step 2 : Business Problem : Data Insights for Fraud Detection</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S2xNpmCMv0mIGQFJP_7dJw.png" /></figure><p>This example shows how to use Neo4j Graph Analytics within Snowflake to analyze your data, in this case focusing on peer-to-peer (P2P) financial transaction data. Specifically, this example shows how to:</p><ol><li>Identify user clusters — i.e. customer segmentation (graph algo: louvain community detection)</li><li>Find the most influential users: who has the most money flowing through their account (graph algo:pagerank centrality)</li><li>Perform entity resolution &amp; identify fraud risk accounts (graph algo: weakly connected components)</li></ol><p><strong>Dataset</strong></p><p>The P2P Dataset has the below schema with users sending money to each other in P2P transactions. Each user can also have multiple identifiers, including credit cards, devices, and IP addresses.</p><p>We have 3 datasets we are using this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2MFbLVTBS_LHvRt2JsoYfw.png" /><figcaption><em>P2P Dataset</em></figcaption></figure><p><strong><em>Step 3 : Creating the user/ admin roles, and granting the app to them</em></strong></p><pre>--Creating the user/ admin roles, and granting the app to them <br>-- Create a consumer role for users of the GDS application<br>CREATE ROLE IF NOT EXISTS neo4j_user;<br>GRANT APPLICATION ROLE NEO4J_app.app_user TO ROLE neo4j_user;<br>-- Create a consumer role for administrators of the GDS application<br>CREATE ROLE IF NOT EXISTS neo4j_admin;<br>GRANT APPLICATION ROLE NEO4J_app.app_admin TO ROLE neo4j_admin;</pre><p><strong><em>Creating database/Schema &amp; granting the app on it</em></strong></p><pre>-- Grant access to consumer data<br>-- The application reads consumer data to build a graph object, and it also writes results into new tables.<br>-- We therefore need to grant the right permissions to give the application access.<br>create database IF NOT EXISTS neo4j_db;<br>create schema IF NOT EXISTS neo4j_db.neo4j_sch;<br><br>GRANT USAGE ON DATABASE neo4j_db TO APPLICATION NEO4J_app;<br>GRANT USAGE ON SCHEMA neo4j_db.neo4j_sch TO APPLICATION NEO4J_app;<br><br>-- required to read tabular data into a graph<br>GRANT SELECT ON ALL TABLES IN SCHEMA neo4j_db.neo4j_sch TO APPLICATION NEO4J_app;<br>-- required to write computation results into a table<br>GRANT CREATE TABLE ON SCHEMA neo4j_db.neo4j_sch TO APPLICATION NEO4J_app;<br>-- optional, ensuring the consumer role has access to tables created by the application<br>GRANT ALL PRIVILEGES ON FUTURE TABLES IN SCHEMA neo4j_db.neo4j_sch TO ROLE neo4j_user;</pre><p><strong><em>Calling ths app — App is ready to use</em></strong></p><pre>CALL NEO4J_app.gds.create_session(&#39;CPU_X64_XS&#39;);</pre><p><strong><em>Step 4: GDS for Snowflake API</em></strong></p><p>GDS for Snowflake is used via SQL procedures and functions. Neo4j_GDS.gds.create_session procedure creates the GDS Session service and service functions. The service functions are used to project graphs, run algorithms, and write back results. Most operations in GDS take a GraphName and a Configuration. The GraphName is a String and must be unique per user within the application.</p><p>The GDS application requires the CREATE COMPUTE POOL and CREATE WAREHOUSE privileges. GDS creates multiple compute pools on which GDS Sessions can be run. The compute pools are created in suspended state and will auto-suspend shortly after the GDS Session is terminated. GDS also creates a query warehouse for use by the Session service.</p><p><strong><em>Step 5 : </em>Run the App</strong>: Execute the use case for Data Insights in Fraud Detection</p><pre>CREATE OR REPLACE TABLE p2p_demo.public.P2P_AGG_TRANSACTIONS (<br> SOURCENODEID NUMBER(38,0),<br> TARGETNODEID NUMBER(38,0),<br> TOTAL_AMOUNT FLOAT<br>) AS<br><br>SELECT sourceNodeId, targetNodeId, SUM(transaction_amount) AS total_amount<br>FROM p2p_demo.public.P2P_TRANSACTIONS<br>GROUP BY sourceNodeId, targetNodeId;<br><br>-- LOUVIAN &amp; PAGERANK<br><br>USE DATABASE p2p_demo;<br>USE SCHEMA public;<br>SELECT * FROM p2p_demo.public.P2P_AGG_TRANSACTIONS;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T8IhgRrY0w2sXIXCPu2DSA.png" /><figcaption>We can See the results Source/ Target Node ID</figcaption></figure><pre>SELECT NEO4J_app.gds.graph_drop(&#39;transaction_graph&#39;, { &#39;failIfMissing&#39;: false });<br>USE DATABASE p2p_demo;</pre><p><strong><em>Lets do graph projections</em></strong></p><pre>-- Construct a graph projection from node and relationship tables<br>SELECT NEO4J_app.gds.graph_project(&#39;transaction_graph&#39;, {<br>    &#39;nodeTable&#39;: &#39;p2p_demo.public.p2p_users&#39;,<br>                                        &#39;relationshipTable&#39;: &#39;p2p_demo.public.p2p_agg_transactions&#39;});</pre><p>Graph is projected in Snowflake, without data ever leaving snowflake boundaries, using native compute. Response<strong> </strong>below</p><pre>{<br>  &quot;completed&quot;: true,<br>  &quot;error&quot;: null,<br>  &quot;result&quot;: {<br>    &quot;graphName&quot;: &quot;transaction_graph&quot;,<br>    &quot;nodeCount&quot;: 33732,<br>    &quot;nodeMillis&quot;: 897,<br>    &quot;relationshipCount&quot;: 43135,<br>    &quot;relationshipMillis&quot;: 805,<br>    &quot;totalMillis&quot;: 1702<br>  }<br>}</pre><p><strong><em>Step 6 : </em>Execute Algorithms</strong>: Apply Louvain, PageRank, and WCC for Entity Resolution</p><pre>--calculate louvain communities<br>SELECT NEO4J_app.gds.louvain(&#39;transaction_graph&#39;, {&#39;mutateProperty&#39;: &#39;community_id&#39;});</pre><p><strong><em>Snapshot of the result</em></strong></p><pre>{<br>  &quot;completed&quot;: true,<br>  &quot;error&quot;: null,<br>  &quot;result&quot;: {<br>    &quot;communityCount&quot;: 12231,<br>    &quot;communityDistribution&quot;: {<br>      &quot;max&quot;: 968,<br>      &quot;mean&quot;: 2.7579102281089036,<br>      &quot;min&quot;: 1,<br>      &quot;p1&quot;: 1,<br>      &quot;p10&quot;: 1,<br>      &quot;p25&quot;: 1,<br>      &quot;p5&quot;: 1,<br>      &quot;p50&quot;: 2,<br>      &quot;p75&quot;: 2,<br>      &quot;p90&quot;: 4,<br>      &quot;p95&quot;: 5,<br>      &quot;p99&quot;: 13,<br>      &quot;p999&quot;: 176<br>    },<br>    &quot;computeMillis&quot;: 6762,<br>    &quot;configuration&quot;: {<br>      &quot;concurrency&quot;: 12,<br>      &quot;consecutiveIds&quot;: false,<br>      &quot;includeIntermediateCommunities&quot;: false,<br>      &quot;jobId&quot;: &quot;01b8e7f3-0004-4d3d-0037-cf0702418b72:1:1:0:0&quot;,<br>      &quot;logProgress&quot;: true,<br>      &quot;maxIterations&quot;: 10,<br>      &quot;maxLevels&quot;: 10,<br>      &quot;mutateProperty&quot;: &quot;community_id&quot;,<br>      &quot;nodeLabels&quot;: [<br>        &quot;*&quot;<br>      ],<br>      &quot;relationshipTypes&quot;: [<br>        &quot;*&quot;<br>      ],<br>      &quot;seedProperty&quot;: null,<br>      &quot;sudo&quot;: false,<br>      &quot;tolerance&quot;: 1.000000000000000e-04<br>    },<br>    &quot;modularities&quot;: [<br>      0.6516474864289455,<br>      0.7738283240328684,<br>      0.8075875039464162,<br>      0.8202458543269713,<br>      0.8268084592772428,<br>      0.8300280180905033,<br>      0.8323258489750152,<br>      0.8343142338389502,<br>      0.8357988092973281,<br>      0.8371050245677109<br>    ],<br>    &quot;modularity&quot;: 0.8371050245677109,<br>    &quot;mutateMillis&quot;: 0,<br>    &quot;nodePropertiesWritten&quot;: 33732,<br>    &quot;postProcessingMillis&quot;: 29,<br>    &quot;preProcessingMillis&quot;: 2514,<br>    &quot;ranLevels&quot;: 10<br>  }<br>}</pre><p><strong><em>Calculate transaction pagerank &amp; WCC entity resolution</em></strong></p><pre>--calculate transaction pagerank<br>SELECT NEO4J_app.gds.page_rank(&#39;transaction_graph&#39;, {&#39;mutateProperty&#39;: &#39;score&#39;});<br>-- Write  to table<br>SELECT NEO4J_app.gds.write_nodeproperties(&#39;transaction_graph&#39;,<br>           {&#39;nodeProperties&#39;: [&#39;score&#39;], &#39;table&#39;: &#39;p2p_demo.public.p2p_transaction_pagerank&#39;});<br>-- Query result from table<br>SELECT * FROM p2p_transaction_pagerank ORDER BY score DESC;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*hFjvQKT4ZurVZRwjj1jVAw.png" /></figure><p><strong><em>Calculate WCC entity resolution</em></strong></p><pre>SELECT * FROM P2P_W_SHARED_CARD;<br><br>-- Construct a graph projection from node and relationship tables<br>SELECT NEO4J_app.gds.graph_drop(&#39;entity_linking_graph&#39;, { &#39;failIfMissing&#39;: false });<br><br>-- Similar to calling the functions with simple or qualified names, we have to reference the tables wither with qualified names or with simple names while USE ing the database or schema.<br>SELECT NEO4J_app.gds.graph_project(<br>               &#39;entity_linking_graph&#39;,<br>           { &#39;nodeTable&#39;: &#39;p2p_demo.public.p2p_users&#39;,<br>               &#39;relationshipTable&#39;: &#39;p2p_demo.public.P2P_W_SHARED_CARD&#39;});<br><br>-- calculate wcc<br>SELECT NEO4J_app.gds.wcc(&#39;entity_linking_graph&#39;, {&#39;mutateProperty&#39;: &#39;wcc_id&#39;});<br><br>-- Write  to table<br>SELECT NEO4J_app.gds.write_nodeproperties(&#39;entity_linking_graph&#39;,<br>           {&#39;nodeProperties&#39;: [&#39;wcc_id&#39;], &#39;table&#39;: &#39;p2p_demo.public.P2P_COMPONENTS&#39;}<br>);<br>SELECT * FROM P2P_COMPONENTS ORDER BY wcc_id;<br><br>-- create a resolved entity view based on WCC<br>CREATE OR REPLACE VIEW resolved_p2p_users AS<br>SELECT p2p_components.wcc_id,<br>       count(*) AS user_count,<br>       TO_NUMBER(SUM(p2p_users.fraud_transfer_flag)&gt;0) AS has_fraud_flag,<br>       ARRAY_AGG(p2p_users.nodeId) AS user_ids<br>FROM p2p_users JOIN p2p_components ON p2p_users.nodeId = p2p_components.nodeId<br>GROUP BY wcc_id ORDER BY user_count DESC;<br>SELECT * FROM resolved_p2p_users;</pre><p>Results — Wow we have these results mapping in snowflake</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6C3n97B5CcTnI6wrvGf6cQ.png" /></figure><p><strong><em>Step 7 : Writing back the result on snowflake</em></strong></p><pre><br>-- Write  to table<br>SELECT NEO4J_app.gds.write_nodeproperties(&#39;transaction_graph&#39;,<br>           {&#39;nodeProperties&#39;: [&#39;community_id&#39;], &#39;table&#39;: &#39;p2p_demo.public.p2p_louvain&#39;});</pre><p><strong><em>Step 8 : Entity Resolution &amp; Fraud Labels using snowflake notebook</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QP4vhxSZuN1mgXahJSMX-w.png" /></figure><pre>from snowflake.snowpark import Session<br>from snowflake.snowpark.version import VERSION<br>from snowflake.snowpark.functions import udf<br>import snowflake.snowpark.functions as F<br><br>from snowflake.snowpark.context import get_active_session<br>session = get_active_session()<br><br>snowflake_environment = session.sql(&#39;SELECT current_user(), current_version()&#39;).collect()<br>snowpark_version = VERSION<br><br># Current Environment Details<br>print(&#39;\nConnection Established with the following parameters:&#39;)<br>print(&#39;User                        : {}&#39;.format(snowflake_environment[0][0]))<br>print(&#39;Role                        : {}&#39;.format(session.get_current_role()))<br>print(&#39;Database                    : {}&#39;.format(session.get_current_database()))<br>print(&#39;Schema                      : {}&#39;.format(session.get_current_schema()))<br>print(&#39;Warehouse                   : {}&#39;.format(session.get_current_warehouse()))<br>print(&#39;Snowflake version           : {}&#39;.format(snowflake_environment[0][1]))<br>print(&#39;Snowpark for Python version : {}.{}.{}&#39;.format(snowpark_version[0],snowpark_version[1],snowpark_version[2]))<br></pre><p><strong><em>Combining Graph Features &amp; Additional Fraud Risk Labeling</em></strong></p><pre>CREATE OR REPLACE VIEW user_features AS<br>SELECT p2p_users.* RENAME nodeId AS user_id,<br>    gds_features.wcc_id,<br>    gds_features.has_fraud_flag AS fraud_risk,<br>    gds_features.user_count AS community_size,<br>    TO_NUMBER(gds_features.user_count &gt; 1) AS part_of_community,<br>    gds_features.transaction_pagerank<br>-- join users to gds features<br>FROM p2p_users JOIN (<br>    -- join resolved user view with gds algo results on wcc_id<br>    SELECT resolved_p2p_users.wcc_id, <br>        resolved_p2p_users.user_count, <br>        resolved_p2p_users.has_fraud_flag,<br>        gds_algo_results.transaction_pagerank,<br>        gds_algo_results.nodeId<br>    FROM resolved_p2p_users JOIN (<br>        -- join gds output tables on node<br>        SELECT p2p_components.nodeId, <br>            p2p_components.wcc_id, <br>            pagerank.transaction_pagerank<br>        FROM p2p_components <br>        JOIN (<br>            SELECT p2p_transaction_pagerank.nodeId, <br>                p2p_transaction_pagerank.score AS transaction_pagerank<br>            FROM p2p_transaction_pagerank<br>        ) pagerank ON p2p_components.nodeId = pagerank.nodeId<br>    ) gds_algo_results ON gds_algo_results.wcc_id = resolved_p2p_users.wcc_id<br>) gds_features ON user_id = gds_features.nodeId;<br>SELECT * FROM user_features</pre><p>Results</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lHhdzLG2l9i2DhIFLq03Eg.png" /></figure><p>Next Blog coming soon — Lets do some ML on this dataset created on Snowflake notebooks <strong>GDS For Snowflake Demo: Entity Resolution &amp; ML</strong></p><p>For more information on the Neo4j Graph Data Science library refer to the <a href="https://neo4j.com/docs/graph-data-science/current/">official documentation</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6831912bc8f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/neo4j-and-snowflake-unlocking-the-power-of-graph-databases-to-transform-data-insights-a-true-d6831912bc8f">Neo4j and Snowflake: Unlocking the Power of Graph Databases to Transform Data Insights — A True…</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Harnessing the Power of LangChain and Snowflake: Unlocking Insights with Cortex]]></title>
            <link>https://medium.com/snowflake/harnessing-the-power-of-langchain-and-snowflake-unlocking-insights-with-cortex-43402b6b8319?source=rss-14293f8d4549------2</link>
            <guid isPermaLink="false">https://medium.com/p/43402b6b8319</guid>
            <dc:creator><![CDATA[Prash Medirattaa]]></dc:creator>
            <pubDate>Mon, 21 Oct 2024 23:54:30 GMT</pubDate>
            <atom:updated>2024-10-22T00:38:32.095Z</atom:updated>
            <content:encoded><![CDATA[<p>The integration of LangChain and Snowflake offers a powerful combination of cutting-edge AI with enterprise-grade data management. LangChain, a framework designed for building applications powered by language models, enables natural language interactions and decision-making workflows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1jVG0dxixXFx_Kkr" /><figcaption>Langchain and Snowflake</figcaption></figure><p>Recently, <strong>Snowflake</strong> announced an official integration package that connects <strong>Snowflake with LangChain</strong>, marking a significant advancement for developers looking to build AI-driven applications. This package streamlines the process of connecting the powerful language model capabilities of LangChain with Snowflake’s robust data platform, enabling seamless querying, processing, and analysis of large datasets using natural language interfaces.</p><p>With this integration, users can harness LangChain’s advanced language processing for a variety of use cases, from automating complex workflows to building conversational agents. Meanwhile, Snowflake ensures that these AI applications can scale, delivering secure and high-performance access to massive amounts of data. This package simplifies the development process, making it easier for teams to integrate cutting-edge AI into their data pipelines and leverage the combined power of LangChain and Snowflake in real-world scenarios.</p><p><a href="https://pypi.org/project/langchain-snowflake/">langchain-snowflake</a></p><pre>from langchain_snowflake import CortexSearchRetriever<br><br>search = CortexSearchRetriever(<br>        role=&quot;snowflake_role&quot;,<br>        database=&quot;your_db&quot;,<br>        schema=&quot;your_schema&quot;,<br>        search_service=&quot;your_cortex_search_service_name&quot;,<br>        search_column=&quot;search_column_name&quot;,<br>)</pre><p><strong>Part of this integration,I converted these two quickstart guide </strong><a href="https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-search/tutorials/cortex-search-tutorial-1-search#introduction">Cortex Search guide</a> &amp; <a href="https://quickstarts.snowflake.com/guide/ask_questions_to_your_own_documents_with_snowflake_cortex_search/index.html?index=..%2F..index#0">Cortex RAG guide</a></p><p>Step 1 — Lets create the conda env</p><pre>conda create -n snowflake-langchain python=3.10<br>conda activate snowflake-langchain</pre><p>Step 2 — Lets create a requirement file</p><pre>snowflake-snowpark-python<br>streamlit<br>pypdf<br>langchain-snowflake<br>langchain_community<br>langchain_core</pre><p>Step 3 — Run in the Env</p><pre>pip install -r requirements.txt</pre><p>step 4 — create the credentials</p><pre>cp connect.json snow_connect.json<br>cp connect.json langchain_cs.json</pre><p>Step 5 For <a href="https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-search/tutorials/cortex-search-tutorial-1-search#introduction">Quickstart-Cortex Search</a></p><pre>snow_connect.json-- Parameters please update your id/password <br>{<br>    &quot;account&quot;: &quot;xxxx&quot;,<br>    &quot;user&quot;: &quot;xxxx&quot;,<br>    &quot;password&quot;: &quot;xxxx&quot;,<br>    &quot;role&quot;: &quot;ACCOUNTADMIN&quot;,<br>    &quot;warehouse&quot;: &quot;cortex_search_tutorial_wh&quot;,<br>    &quot;database&quot;: &quot;cortex_search_tutorial_db&quot;,<br>    &quot;schema&quot;: &quot;public&quot;,<br>    &quot;client_session_keep_alive&quot;: &quot;True&quot;<br>}</pre><p>For <a href="https://quickstarts.snowflake.com/guide/ask_questions_to_your_own_documents_with_snowflake_cortex_search/index.html?index=..%2F..index#0">Quickstart-Cortex RAG LLM</a> if you don’t have WH configured</p><pre>CREATE OR REPLACE WAREHOUSE cortex_search_tutorial_wh WITH<br>     WAREHOUSE_SIZE=&#39;X-SMALL&#39;<br>     AUTO_SUSPEND = 120<br>     AUTO_RESUME = TRUE<br>     INITIALLY_SUSPENDED=TRUE;```</pre><pre>langchain_cs.json-- Parameters please update youer id/password <br>{<br>    &quot;account&quot;: &quot;xxxx&quot;,<br>    &quot;user&quot;: &quot;xxxx&quot;,<br>    &quot;password&quot;: &quot;xxxx&quot;,<br>    &quot;role&quot;: &quot;ACCOUNTADMIN&quot;,<br>    &quot;warehouse&quot;: &quot;cortex_search_tutorial_wh&quot;,<br>    &quot;database&quot;: &quot;CC_QUICKSTART_CORTEX_SEARCH_DOCS&quot;,<br>    &quot;schema&quot;: &quot;DATA&quot;,<br>    &quot;client_session_keep_alive&quot;: &quot;True&quot;<br>}</pre><p>Step 6. Replace <a href="https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-search/tutorials/cortex-search-tutorial-1-search#step-4-create-a-streamlit-app">Streamlit App</a> to this <a href="https://github.com/Snowflake-Labs/langchain_cortex/blob/main/src/langchain_cortexsearch.py">New App</a> for this <a href="https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-search/tutorials/cortex-search-tutorial-1-search#introduction">Cortex Search guide</a></p><p>Step 7 Replace <a href="https://quickstarts.snowflake.com/guide/ask_questions_to_your_own_documents_with_snowflake_cortex_search/index.html?index=..%2F..index#4">Streamlit App</a> to this <a href="https://github.com/Snowflake-Labs/langchain_cortex/blob/main/src/Langchain_RAG.py">New App</a> for this <a href="https://quickstarts.snowflake.com/guide/ask_questions_to_your_own_documents_with_snowflake_cortex_search/index.html?index=..%2F..index#0">Cortex RAG guide</a></p><p>Step 8 Optional — Connecting Cortex <a href="https://pypi.org/project/langchain-snowflake/">https://pypi.org/project/langchain-snowflake/</a> from local env</p><pre>## In your directory <br>touch .env <br><br>### In env file <br>LANGCHAIN_API_KEY=&quot;XXXXXX&quot;<br>LANGCHAIN_PROJECT=&quot;Cortex-langchainintegration&quot;<br><br>SNOWFLAKE_ACCOUNT=XXXXX<br>SNOWFLAKE_USERNAME=XXXX<br>SNOWFLAKE_PASSWORD=XXXXX<br>SNOWFLAKE_DATABASE=XXXXX<br>SNOWFLAKE_SCHEMA=XXXXX<br>SNOWFLAKE_WAREHOUSE=XXXXX<br>SNOWFLAKE_ROLE=XXXXX</pre><p>Step 9 Connecting with Snowflake, there are multiple ways I will be using an environment file, create <a href="https://docs.smith.langchain.com/how_to_guides/setup/create_account_api_key">langchain API key </a>to track</p><pre>pip install -r requirements.txt</pre><pre>import os, getpass<br>from dotenv import load_dotenv,find_dotenv<br>from snowflake.core import Root<br>load_dotenv()<br>from langchain_core.messages import HumanMessage, SystemMessage<br>from langchain_snowflake import CortexSearchRetriever<br>from langchain_community.chat_models import ChatSnowflakeCortex</pre><pre>#### Getting the env variables<br>os.environ[&#39;LANGCHAIN_API_KEY&#39;]=os.getenv(&#39;LANGCHAIN_API_KEY&#39;)<br>os.environ[&quot;LANGCHAIN_TRACING_V2&quot;] = &quot;true&quot;</pre><p>Step 10 Create the <a href="https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-search/tutorials/cortex-search-tutorial-1-search#step-3-create-the-search-service">search service</a> will use an example</p><pre>from langchain_snowflake import CortexSearchRetriever<br><br>search_airbnb = CortexSearchRetriever(<br>        search_service=&quot;airbnb_svc&quot;,<br>        search_column=&quot;LISTING_TEXT&quot;,<br>        columns=[&quot;ROOM_TYPE&quot;, &quot;AMENITIES&quot;],<br>        filter={&quot;@eq&quot;: {&quot;ROOM_TYPE&quot;: &quot;Entire home/apt&quot;}},<br>        limit=2<br>)</pre><p>Step 11 Test the service calling the langchain framework using this <a href="https://github.com/Snowflake-Labs/langchain_cortex/blob/main/src/langchain_cortexsearch.ipynb">notebook</a></p><pre>### Invoking the query <br>query=&quot;faulty&quot;<br>result = search.invoke(query)<br>for doc in result:<br>    print(doc)</pre><pre>### Result<br>page_content=&#39;I received a faulty router, can I get it replaced?&#39; metadata={&#39;region&#39;: &#39;North America&#39;}<br>page_content=&#39;My internet has been down since yesterday, can you help?&#39; metadata={&#39;region&#39;: &#39;North America&#39;}</pre><p>Step 12 Tracking using langsmith the service</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zwwGysH0moqkSuvPRNs2YA.jpeg" /><figcaption>Different Projects — Will focus on Cortex-langchainintegration</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ZsCEVV3BikLCcJCpOcM1g.jpeg" /><figcaption>Multiple Runs</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nGxNp8DUlrn25zTEwoITKg.jpeg" /><figcaption>Tracking multiple projects</figcaption></figure><h3>Summary</h3><p>There are several workarounds to integrate Langchain with Snowflake. I’ve focused on the official PyPI package, which currently supports Cortex Search, and I’m hopeful that Cortex Analyst will be available soon.</p><p><strong>Note</strong>: These are my personal opinions</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43402b6b8319" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/harnessing-the-power-of-langchain-and-snowflake-unlocking-insights-with-cortex-43402b6b8319">Harnessing the Power of LangChain and Snowflake: Unlocking Insights with Cortex</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building AI-Powered Interaction Analytics with Symbl.ai and Snowflake]]></title>
            <link>https://medium.com/snowflake/building-ai-powered-interaction-analytics-with-symbl-ai-and-snowflake-c01e0c57c9e9?source=rss-14293f8d4549------2</link>
            <guid isPermaLink="false">https://medium.com/p/c01e0c57c9e9</guid>
            <dc:creator><![CDATA[Prash Medirattaa]]></dc:creator>
            <pubDate>Fri, 18 Oct 2024 02:52:33 GMT</pubDate>
            <atom:updated>2024-10-18T17:12:36.498Z</atom:updated>
            <content:encoded><![CDATA[<p>Co-authored with <a href="https://medium.com/u/ccae4fc06c32">Toshish</a></p><p>In today’s data-driven world, combining analyzing unstructured data in business calls such as customer calls, sales calls, interviews, internal meetings, etc. within a business with existing business data creates new opportunities for deeper insights and more informed decision-making. <a href="http://symbl.ai/">Symbl.ai</a>’s programmable APIs can be integrated with <a href="https://snowflake.com/">Snowflake</a> to create a powerful data stream of multimodal interaction data, enhancing your data warehouse capabilities and unlocking new insights.</p><ol><li>Removing Blind Spots in Predictive Analytics</li><li>Unlocking New App Experiences for Cross-Functional Teams</li><li>Improving Performance of Customer-Facing Teams</li></ol><p>In this blog post, we will show how to build a Sales analytics dashboard from sales calls and CRM data together to analyze sales efficiency, quality assurance, and performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Xk8UPT1wvUIwWijf" /></figure><h3>Why<a href="http://symbl.ai/"> Symbl.ai</a> + Snowflake?</h3><p>Integrating Symbl.ai with Snowflake helps organizations bridge the gap between unstructured conversational and structured business data. <a href="http://symbl.ai/">Symbl.ai</a> provides a variety of programmable features powered by Generative and Specialized models built for understanding human conversations. Businesses can extract insights such as call scoring, topics, action items, questions, and sentiment from customer conversations, whether those occur in real time or are pre-recorded. Once these insights are processed, they can be loaded into Snowflake, where businesses can combine them with sales data, CRM information, or product performance metrics, providing a holistic view of customer interactions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Q9cjsHHnvUiWpi9v" /><figcaption>Stitching business data with knowledge in calls and meetings in the organization using Symbl.ai and Snowflake.</figcaption></figure><h3>Connectors for Streaming and Async Interaction Data</h3><p>Symbl offers programmable APIs that can seamlessly integrate with various forms of interaction data, including voice, video, and text, whether in real-time or asynchronously. These APIs can be layered on top of Communications Platform as a Service (CPaaS) solutions or custom applications, enabling businesses to harness valuable insights from their communications. Additionally, Symbl provides pre-built connectors for popular contact center platforms such as Genesys, Five9, and more, facilitating easy integration with existing systems. By acting as a connector layer, Symbl can aggregate and <a href="https://symbl.ai/developers/blog/transform-multimodal-interaction-data-with-symbl-and-snowflake/">transform multimodal data into Snowflake</a>, allowing for contextual correlation and analysis alongside other data streams, ultimately driving comprehensive data intelligence across an organization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hayrLarrnbToy7GxjOUp1A.png" /></figure><h3>Use Case: Sales Call Analysis and Performance Dashboard</h3><p>This use case focuses on leveraging Symbl.ai’s capabilities to analyze sales calls and create a comprehensive performance dashboard. While the example uses sample call transcripts and CRM data, the solution can be applied to various audio/video sources, including recorded files, live call streaming, and phone calls across the business.</p><p><strong>Problem:</strong> Inadequate training of the team can reduce sales performance by up to 43%. For a team of 20 reps, each losing 43% of their potential $300,000 in annual sales, this could amount to $2.58 million in lost revenue. This exponentially increases for larger teams and the lack of correlation in day-to-day conversations to the sales metrics leads to inaccurate sales forecasting and missed revenue goals. To summarize:</p><ol><li>Lack of real-time performance monitoring that leads to blindspots in operations</li><li>Long time to deliver contextual feedback to sales reps which leads to poor performance</li><li>Lack of Insights from customer conversations to drive better messaging and objection handling</li></ol><p><strong>Solution:</strong> To automatically analyze the sales calls to quantify the performance of sales representatives, Symbl.ai provides the capability to score sales calls based on various pre-built criteria such as Communication, Engagement, Question Handling, Forward Motion, and Sales Process adherence. These criteria can be modified or additional custom criteria can be configured in the context of the team’s expertise, specific product focus, and other aspects. Additional metadata from the calls such as sentiments, objections raised, next steps, intents and topics discussed, etcetera provide more dimension to the analysis to correlate them with the sales metrics.</p><p>We will use these structured insights from Symbl to build a sales analysis dashboard using Streamlit that can be used to measure and analyze the performance of sales representatives thus correlating data in the sales calls and CRM. We will persist the outcomes of Symbl to Snowflake, and use Cortex Search with Symbl’s Nebula LLM for search and co-pilot functionality in the dashboard.</p><h3>Processing Calls with Symbl.ai and Snowflake</h3><p>We will use this <a href="https://github.com/symblai/snowflake-symbl-integration-python">demo application</a> to build a simple processing pipeline with Symbl’s APIs to ingest calls to extract insights, store them in Snowflake in a structured manner, and prepare an analytics dashboard using Streamlit.</p><h3>Step 1: Set up Symbl.ai and Snowflake Accounts and Environment</h3><p>To begin, ensure you have Symbl.ai and Snowflake accounts ready.</p><p>You will need Symbl.ai application credentials to process audio files using Symbl APIs -</p><ol><li>Visit the<a href="https://platform.symbl.ai/"> Symbl.ai platform</a> and log into your existing account or sign up for a new account.</li><li>Obtain your App ID and App Secret from the Symbl.ai dashboard, request Nebula access, and get the Nebula API key.</li></ol><p>Similarly, you’ll need a Snowflake instance where you can ingest, store, and analyze data -</p><ol><li><a href="https://signup.snowflake.com/">Sign up</a> for a new Snowflake account or log in if you already have one.</li><li>Make sure you have your account ID, username, and password handy.</li></ol><h3>Step 2: Setup the environment</h3><p>Clone the repository and navigate to the project directory.</p><pre><br>git clone https://github.com/symblai/snowflake-symbl-integration-python.git<br><br>cd snowflake-symbl-integration-python</pre><p>Create a Conda environment and activate it. Install <a href="https://docs.anaconda.com/miniconda/miniconda-install/#installing-miniconda">miniconda</a> if you don’t have it installed.</p><pre><br>conda create -n snowflake-symbl python=3.11<br>conda activate snowflake-symbl</pre><p>Install the required dependencies by running -</p><pre>pip install -r requirements.txt</pre><p>Copy secrets.toml.default file to a new secrets.toml file.</p><pre>cp secrets.toml.default secrets.toml</pre><p>Update secrets.toml file with your Symbl and Snowflake credentials.</p><p>Once all values are updated, run the setup script to create the necessary tables in Snowflake.</p><pre>python setup_snowflake_env.py</pre><p>This will set up the snowflake database and schema with the required tables, and cortex search services. And also copy your secrets.toml file to the streamlit/.streamlit directory so that the Streamlit app can access the required credentials.</p><h3>Step 3: Run the Call processing</h3><p>You are now ready to run the main.py script to process the files and store the results in the Snowflake.</p><p>The main.py will use the sample transcripts and CRM data provided in the <a href="https://github.com/symblai/snowflake-symbl-integration-python/blob/main/data">data</a> directory. You can also provide your data by updating the main.py script. Feel free to modify it to also use your audio files by calling the submit_audio_file function instead of the submit_transcript function, and use your own CRM data.</p><pre>python main.py</pre><p>This will start processing the sample call transcripts with Symbl.ai and store results in the Snowflake database. It will process the calls with Symbl.ai Async API, and fetch results using Conversation APIs, Call Score API, and Insights API. Async API allows submitting audio/video files, as well as text transcripts of the conversation to Symbl for processing asynchronously, and Conversation API allows us to fetch various outputs generated from the submitted call along with Call Score API, and Insights API. These results are then stored in Snowflake.</p><p>We’ll use the following features to process audio files from Symbl -</p><ul><li><strong>Automatic Speech Recognition (ASR)</strong> to transcribe the audio files.</li><li><strong>Call Score</strong> on the various criteria to generate quality of the calls along with explanations.</li><li><strong>Sentiment Analysis</strong> of the calls to capture sentiment trajectories and overall sentiment analytics, with an explanation of the analysis.</li><li><strong>Summarization</strong> of the calls to capture call summaries.</li><li>Identification of <strong>Questions and Answers</strong>, and Next Steps discussed in the calls.</li><li>Identification of key <strong>Topics</strong> of discussion from the calls.</li><li><strong>Pre-built Insights UI</strong> for out-of-the-box UI provided by Symbl to explore various Insights in conversation.</li></ul><h3>Step 4: Start Streamlit Dashboard</h3><p>Once the script has finished running, you can now start the Streamlit application to run the dashboard.</p><pre>cd streamlit<br>streamlit run app.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P9JJaTSSGaa7l8EM" /><figcaption>Sales Analysis dashboard built with Streamlit using data pushed in Snowflake</figcaption></figure><h3>Next Steps</h3><p>This was just an example with a few features of Symbl.ai and Snowflake. To further expand on this example you can -</p><ul><li>If you have a paid Snowflake account that is not a trial account, you can also install the Streamlit app to Snowflake so that it can access Symbl’s Nebula API from within your Snowflake instance ``python setup_snowflake_env.py — install_streamlit_app```</li></ul><pre>python setup_snowflake_env.py - install_streamlit_app</pre><ul><li>Expand the scope of analysis by using your own choice of LLM and Embedding from <a href="https://www.snowflake.com/en/data-cloud/cortex/">Cortex</a>.</li><li>Build dashboards for other use cases in your organization such as customer support, recruitment, internal meeting knowledge mining, etc.</li><li>Automate this pipeline with ETL Jobs using <a href="https://docs.snowflake.com/en/user-guide/tasks-intro">Snowflake’s Task</a> scheduling or tools like <a href="https://airflow.apache.org/">Airflow</a>.</li><li>Scale the processing of live calls with Symbl’s <a href="https://docs.symbl.ai/docs/streaming-api">Streaming API</a> to get instant analysis available in Snowflake.</li><li>Use <a href="https://docs.snowflake.com/en/developer-guide/udf/udf-overview">Snowflake UDFs</a> to implement the processing logic to unify your processing within your Snowflake environment.</li></ul><p>Explore and learn about Symbl.ai and Snowflake offerings -</p><ul><li>Symbl.ai Documentation: <a href="https://docs.symbl.ai/">https://docs.symbl.ai</a></li><li>Snowflake Documentation: <a href="https://docs.snowflake.com/">https://docs.snowflake.com</a></li><li>Streamlit Documentation: <a href="https://docs.streamlit.io/">https://docs.streamlit.io</a></li></ul><h3>Conclusion: Empowering Business with<a href="http://symbl.ai/"> Symbl.ai</a> and Snowflake</h3><p>By integrating<a href="http://symbl.ai/"> </a>Symbl.ai’s powerful AI with Snowflake’s scalable data infrastructure, businesses can transform raw interaction data into structured, actionable insights and unlock a new data stream for their intelligence and analytics initiatives. This not only helps improve customer service and sales but also enables data-driven decision-making across various departments. With this powerful integration, teams can better understand their customers, streamline operations, and enhance overall business intelligence.</p><p>More about Authors: <a href="https://www.linkedin.com/in/toshishjawale/">Toshish Jawale</a>, <a href="https://www.linkedin.com/in/prashmedirattaa/">Prash Medirattaa</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c01e0c57c9e9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/building-ai-powered-interaction-analytics-with-symbl-ai-and-snowflake-c01e0c57c9e9">Building AI-Powered Interaction Analytics with Symbl.ai and Snowflake</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking Machine Learning Potential: Running R Inside Snowpark Container Services]]></title>
            <link>https://medium.com/snowflake/unlocking-machine-learning-potential-running-r-inside-snowpark-container-services-3ccdb2cac896?source=rss-14293f8d4549------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ccdb2cac896</guid>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Prash Medirattaa]]></dc:creator>
            <pubDate>Sun, 10 Sep 2023 11:50:04 GMT</pubDate>
            <atom:updated>2023-09-11T17:07:39.785Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="R Integration via Snowpark Containers Empowered by Posit" src="https://cdn-images-1.medium.com/max/1024/1*cyznVrt2coW_KCFscKyMcA.png" /></figure><p>There is a big community for R globally, and for Snowflake it is growing fast. Snowpark Container services opens new opportunities for Snowflake customers to run natively, without data ever leaving the data cloud. Snowpark Container Services (SPCS) was launched at Snowflake Summit. More details can found <a href="https://www.snowflake.com/blog/snowpark-container-services-deploy-genai-full-stack-apps/">here</a>. There was some work done earlier on running R inside Hex environment as well <a href="https://medium.com/snowflake/using-hexs-new-r-support-with-snowflake-ff6c7a107cd6">here</a></p><p>In this blog you will see how easy it to spin up a instance, upload a docker image and run R inside snowflake. I will be building an end to end ML basic project for demonstartion. We will be using a basic iris dataset for this demo. Snowpark Container Services (SPCS) which is now available to a subset of customers in Private Preview. It is a fully managed container offering that allows you to easily deploy, manage, and scale containerized services, jobs, and functions, all within the security and governance boundaries of Snowflake, and requiring zero data movement. <a href="https://medium.com/snowflake/snowpark-container-services-a-tech-primer-99ff2ca8e741">More-Info</a></p><h4>How can you run this demo ?</h4><p>A Snowflake Account that supports Snowflake Services: This private preview works with Snowflake accounts that are created specifically for the purposes of evaluating Snowpark Container Services.</p><p>These accounts have Snowflake Image Registryenabled so that you can create one or more image repositories. In addition, these accounts also have access to new Snowflake features, such as services and jobs, managed by Snowpark Container Services. Please talk to your local account represaentative to activate this flag.Lastly, I have used an enterprise version of R studio which requires a license keys.</p><h4>Steps for the Demo :</h4><ol><li>Create the Infrastructure required</li><li>Downloading the docker image</li><li>Creating specification yaml</li><li>Create &amp; Run the service</li><li>Create the R Project</li><li>Connecting R-Studio with Snowflake</li><li>Bringing the data set</li><li>Creating ML project &amp; writing back to Snowflake</li></ol><h4>1. Create the Infrastructure required</h4><p>Lets create a some basic objects, such as compute pool, WH, database, stages &amp; image repository</p><pre>CREATE COMPUTE POOL yourComputepool <br>MIN_NODES = 1<br>MAX_NODES = 1<br>INSTANCE_FAMILY = standard_1;<br><br>CREATE OR REPLACE WAREHOUSE yourWH <br>WITH WAREHOUSE_SIZE=&#39;X-SMALL&#39;<br>AUTO_SUSPEND = 180<br>AUTO_RESUME = true<br>INITIALLY_SUSPENDED=false;<br><br>CREATE DATABASE yourdatabase;<br><br>CREATE SCHEMA yourschema;<br><br>CREATE OR REPLACE IMAGE REPOSITORY yourimage;<br><br>CREATE STAGE yourstage DIRECTORY = ( ENABLE = true );</pre><h4>2. Downloading the docker image</h4><p>Lets download the docker image, I am using a licensed R-Studio enterprise edition. You can very well download opensource R container. To run an enterprise edition you will be needing a license keys. It is pretty starting forward.</p><p>Now lets do some docker — Logging, tagging &amp; pushing the container in SPCS. I have named the image as rstudio:v1</p><p><a href="https://github.com/orgs/rstudio/packages/container/package/rstudio-workbench-preview">https://github.com/orgs/rstudio/packages/container/package/rstudio-workbench-preview</a></p><pre>docker login youraccount.registry.snowflakecomputing.com<br><br>docker pull image<br><br>docker tag image.registry.snowflakecomputing.com/yourdatabase/yourschema/yourimage/rstudio:v1<br><br>docker push image.registry.snowflakecomputing.com/yourdatabase/yourschema/yourimage/rstudio:v1</pre><h4>3. Creating specification R.yaml</h4><p>Creating the yaml file, pointing to the image registry &amp; uploading this to stage. You can specify multiple environment variables, but this is a basic example.</p><pre>spec:<br>  container:<br>  - name: rstudio<br>    image: image.registry.snowflakecomputing.com/yourdatabase/yourschema/yourimage/rstudio:v1<br>    env:<br>      RSW_LICENSE: &quot;XXXXXXXX&quot;<br>    resources:<br>      limits:<br>        memory: 16Gi<br>      requests:<br>        memory: 4Gi<br>  endpoint:<br>  - name: e1<br>    port: 8080<br>    public: true</pre><h4>4. Create the service</h4><p>We are creating R_notebook service here.</p><pre>CREATE SERVICE R_notebook<br>MIN_INSTANCES=1<br>MAX_INSTANCES=3<br>COMPUTE_POOL=yourComputepool<br>SPEC=@yourstage/R.yaml;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iRXcNVA0mAn73SHpzqZHVQ.png" /><figcaption>Containers are running</figcaption></figure><h4><strong>5. End Point/ Running the service</strong></h4><p>Using the end point, use the credential for rstudio. Now we have a rstudio running within Snowflake. Using the end point, use the credential for rstudio. Now we have a rstudio running within Snowflake.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gx0CdOMkLPyu3MzI97e_sA.png" /><figcaption>End point created</figcaption></figure><p>Creating a new session of R-studio using posit workbench.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*772LGfPdSXzliQKDzg8iPg.png" /><figcaption>Lets use the credentials</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iKvivt3X_63qVOpVm938Fg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_GyVZEy5WSWNN71Irk_X5g.png" /><figcaption>Lets create a RStudio &lt;&gt; SPCS Session</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xRKalVXHhb1OaygpLxTCwg.png" /><figcaption>✊ success !! you can see how easy it was to launch R Studio &lt;&gt; SPCS</figcaption></figure><h4><strong>6. Connecting R-Studio with Snowflake</strong></h4><p>Now we have R-studio running inside SPCS lets connect it to Snowflake. It can either be done by UI or by code,</p><pre>### Downlaod this library - which has dplyr <br>install.packages(&#39;tidyverse&#39;)<br>library(tidyverse)</pre><pre> con &lt;- dbConnect(odbc::odbc(), Driver=&quot;snowflake&quot;,<br>  Server = &quot;account.snowflakecomputing.com&quot;, <br>  UID = &quot;user&quot;, PWD = rstudioapi::askForPassword(&quot;Database password:&quot;),<br>   Database = &quot;yourDatabase, Warehouse = &quot;yourWarehouse&quot;, Schema = &quot;YourSchema&quot;)</pre><p>You can put these credentials, in UI/ or define it in connection.</p><pre>Server : account.snowflakecomputing.com<br>User : User <br>Database : yourDatabase<br>Warehouse : yourWarehouse<br>Schema : yourSchema</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i2qz8d4GWW7W9lcHl9svdA.png" /><figcaption>Server | Database | connection | WH | Schema</figcaption></figure><p><strong>7. Bringing the data set</strong></p><p>We will be using the iris data, and writing back training &amp; test dataset back to Snowflake. I used the project from <a href="https://github.com/dataprofessor/code/tree/master">Source-Credit</a></p><pre># Importing libraries<br>library(tidyverse)<br>library(datasets) <br>library(caret) <br># Importing the Iris data set<br>data(iris)<br># Check to see if there are missing data?<br>sum(is.na(iris))<br># To achieve reproducible model; set the random seed number<br>set.seed(16)<br># Split dataset 80/20 <br>TrainingIndex &lt;- createDataPartition(iris$Species, p=0.8, list = FALSE)<br>TrainingSet &lt;- iris[TrainingIndex,] # Training Set<br>TestingSet &lt;- iris[-TrainingIndex,] # Test Set</pre><p>Defining the objects/ and writing back to Snowflake</p><pre>#### Defining the objects<br>Database - r_studio<br>Schema - Public <br>Name - TrainingSet,TestingSet (in df)<br><br># Writing Training / Test to Snowflake <br>dbWriteTable(con, SQL(&quot;r_studio.public.TrainingSet&quot;), TrainingSet)<br>dbWriteTable(con, SQL(&quot;r_studio.public.TestingSet&quot;), TestingSet)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j29RtgeYMnY2a3PHeg8toA.png" /><figcaption>Dataset- Written back on Snowflake</figcaption></figure><p><strong>8. Creating ML project &amp; writing back in Snowflake</strong></p><p>Lets build a quick classification model on predicting iris dataset. For this example I have used SVM. Again goal of this blog is not to build the best possible model but to see show how R can run in Snowflake using <em>SPCS</em>.</p><pre>###############################<br># SVM model (polynomial kernel)<br><br># Build Training model<br>Model &lt;- train(Species ~ ., data = TrainingSet,<br>               method = &quot;svmPoly&quot;,<br>               na.action = na.omit,<br>               preProcess=c(&quot;scale&quot;,&quot;center&quot;),<br>               trControl= trainControl(method=&quot;none&quot;),<br>               tuneGrid = data.frame(degree=1,scale=1,C=1)<br>)<br><br># Build CV model<br>Model.cv &lt;- train(Species ~ ., data = TrainingSet,<br>                  method = &quot;svmPoly&quot;,<br>                  na.action = na.omit,<br>                  preProcess=c(&quot;scale&quot;,&quot;center&quot;),<br>                  trControl= trainControl(method=&quot;cv&quot;, number=10),<br>                  tuneGrid = data.frame(degree=1,scale=1,C=1)<br>)<br><br># Apply model for prediction<br>Model.training &lt;-predict(Model, TrainingSet) # Apply model to make prediction on Training set<br>Model.testing &lt;-predict(Model, TestingSet) # Apply model to make prediction on Testing set<br>Model.cv &lt;-predict(Model.cv, TrainingSet) # Perform cross-validation<br># Model performance (Displays confusion matrix and statistics)<br>Model.training.confusion &lt;-confusionMatrix(Model.training, TrainingSet$Species)<br>Model.testing.confusion &lt;-confusionMatrix(Model.testing, TestingSet$Species)<br>Model.cv.confusion &lt;-confusionMatrix(Model.cv, TrainingSet$Species)<br><br>print(Model.training.confusion)<br>print(Model.testing.confusion)<br>print(Model.cv.confusion)<br><br># Feature importance<br>Importance &lt;- varImp(Model)<br>plot(Importance)<br>plot(Importance, col = &quot;red&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y8-3Ax3XGGmrjPzPD0WZBA.png" /><figcaption>Basic ML project</figcaption></figure><p><strong>Conclusion</strong></p><p>With Snowpark Container Services, Snowflake customers are now able to host and run arbitrary containerized applications, within the data cloud. Running Rstudio powered by posit is an excellent example.</p><p>This feature is in limited private preview, and can only be activated by request for now. For more information about SPCS, contact your Snowflake account team or sales representative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/286/1*Y3S_-FAxCRGdzbZEENYlkA.png" /></figure><p>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ccdb2cac896" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/unlocking-machine-learning-potential-running-r-inside-snowpark-container-services-3ccdb2cac896">Unlocking Machine Learning Potential: Running R Inside Snowpark Container Services</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enterprise AI for the Modern Data Stack]]></title>
            <link>https://medium.com/snowflake/guide-snowflake-continual-87dceae84c8a?source=rss-14293f8d4549------2</link>
            <guid isPermaLink="false">https://medium.com/p/87dceae84c8a</guid>
            <category><![CDATA[continual]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[data-scie]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Prash Medirattaa]]></dc:creator>
            <pubDate>Sat, 02 Apr 2022 03:46:43 GMT</pubDate>
            <atom:updated>2022-04-06T02:39:26.413Z</atom:updated>
            <content:encoded><![CDATA[<p>By: <a href="https://www.linkedin.com/in/prashmedirattaa/">Prash Medirattaa</a>(Snowflake) and <a href="https://www.linkedin.com/in/brendanmmckenna/">Brendan M. McKenna </a>(Continual)</p><p><strong>Overview</strong></p><p>This guide will show you how to easily add Continual as the AI layer to your modern data stack with Snowflake at the core. The intention is to provide an introduction to using Continual on Snowflake.</p><p>We are going to demonstrate connecting Continual to Snowflake, building feature sets and models from data stored in Snowflake, and analyzing and maintaining the predictive model continuously over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AEUwUU0HEUyv6VEB" /><figcaption>Architecture</figcaption></figure><p>To keep things simple at the start, we’ll use a nicely manicured, fictitious dataset to illustrate how Snowflake and Continual combine to enable modern data teams to effectively build, deploy, and utilize production grade models. The dataset consists of customer information such as account data, demography, geographic area, and phone activity of a fictional telecommunications business. It also conveniently contains a boolean value per customer defining whether or not the person ended their contract and “churned”. While this dataset will suffice the purposes of quickly trying Continual + Snowflake, we don’t believe the telco churn dataset is the most realistic example of customer churn. Which is why we created a more comprehensive <a href="https://docs.continual.ai/customer-churn/">example you can try next</a>!</p><p><strong>What you’ll learn</strong></p><ol><li>How to connect Continual to Snowflake and do Machine Learning on your data warehouse data cloud</li><li>Create feature sets and models in Continual</li><li>Evaluate and maintain production machine learning models</li><li>Analyze model performance, input data, and features to iteratively improve performance</li><li>Writing back prediction to Snowflake</li></ol><p><strong>Prerequisites</strong></p><ol><li>Basic experience with Snowflake and SQL</li><li>Basic knowledge of Machine Learning and Data Science problems</li></ol><h3><strong>Prepare your lab environment</strong></h3><h3>Setting up Snowflake</h3><p>If you have a Snowflake account, then login using your unique credentials.If you don’t have a Snowflake account, visit <a href="https://signup.snowflake.com/">https://signup.snowflake.com/</a> and sign up for a free 30-day trial environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*edokc17aXIRLXEK7" /></figure><p>For this example, you will only need the Standard edition on <strong>AWS</strong>. But you may want to select <strong>Enterprise</strong> to try out rad features like time travel, materialized views, or database failover.</p><p>Choose <strong>US West (Oregon) </strong>for the AWS region.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*p5b4tau-34f15yHr" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t_Drc7C5EK34qoNwopT5Wg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rYGkDQtRkmlH2NTz_2TUJg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DYPoAqJ9t9aBdygBYkhRJA.jpeg" /></figure><p>Once you’ve logged in, open a new Worksheet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*3ZCAiRWGl65KeYLz" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wrzxv6heDtg8ayFePWu38A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q2cgKtO8ST7-h72Xv6lvkg.png" /></figure><p><strong>Role Creation Script</strong></p><p>Let’s create a user role, role, user, warehouse and database for use by Continual. In Worksheets, copy and paste the following SQL into your worksheet. Make sure to <strong>update the user_password</strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7780a79865318874a3eaa424bea6cbd/href">https://medium.com/media/e7780a79865318874a3eaa424bea6cbd/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yMN9bpSmCVGNTpLH" /></figure><p>In this tutorial, we will not use other databases/schemas/tables as source tables for feature sets or models. But for an actual use case, you will need to grant the continual user created above USAGE permission on any such resources. See Continual docs for more information.Setting up Continual</p><h3>Signup for trial account</h3><p>To get started, navigate to <a href="https://cloud.continual.ai/register">Continual</a> and fill in your user details to register an account. Continual has a free 30-day trial and no credit card is required.</p><p>You’ll need to verify your email address. If you don’t receive a verification email within a few minutes, check your spam folder and email <a href="/service/mailto:support@continual.ai">support@continual.ai</a>. If your link expires, you can log back into your account to send a new verification email.</p><h3>Create an organization</h3><p>Organizations allow you to share projects within a company and collaborate with team members under a shared billing account.</p><h3>Create project</h3><p>After creating your organization you will see your organization’s project dashboard with the option to create a project. Projects are isolated workspaces for feature sets and models and connect bi-directionally with Snowflake.</p><p>Go ahead and create a new project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/0*g8zmPZtQ1TU0Mq3A" /></figure><h3>Connect to Snowflake</h3><p>Continual was designed for cloud data warehouses and, consequently, connectivity is simple. Each Continual project connects bi-directionally to one Snowflake Database. Continual maintains tables and views for all your feature sets and models, as well as all predictions made by your models, inside a schema. This makes it easy to build models from your existing data and consume the predictions Continual maintains using your existing tools in Snowflake!</p><p>Click “Connect your data warehouse” and then select “Snowflake”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AuvGhYXTliC3yF1v" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1009/0*weWu3lrw8dSRMRaz" /></figure><p>Enter your snowflake account identifier, username, password, database name, warehouse name and role. Leave the schema field blank.</p><p>NOTE: the Host (Endpoint) is the <a href="https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#account-identifiers">Snowflake account identifier</a>. If you selected a region <em>other than </em>US West (Oregon) you need <a href="https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#account-identifier-formats-by-cloud-platform-and-region">additional segments depending on the region</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/0*u4Iv7WEY3MoTmt6F" /></figure><p>Test the connection and then create it. And there we have it: Continual and Snowflake are connected!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/0*1ASkzG4CpFEhHUae" /></figure><h3>Create a feature set with SQL</h3><p>Now that we’ve established our connection and can access our data in Snowflake, it’s time to prepare features for a model.</p><p>A <a href="https://docs.continual.ai/feature-sets/">feature set</a> is one of the main objects in Continual. It describes a collection of related features and the data underlying those features. You can think about it as a view or table of your data warehouse that organizes the data in a way that is easiest for the machine learning model to understand. Just as we’ll do when creating a model, we use SQL to query the data and a YAML file to define metadata.</p><p>Click “Create a feature set”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ne7D9qEMuTNBfw1f" /></figure><p>The Query Data step is where we use SQL to select the data for our feature set. To make it easy, we have an example ready to go that will copy a csv from an object store into your Snowflake database and pre-populate the query editor, configurations and metadata, and schema. You are living the good life!</p><p>Click “Use an Example” on the right-hand side and select “Predict Customer Churn”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/0*C5zYKMRoGtowWbaq" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/0*M8tlAG6EyYb9KIyD" /></figure><p><strong>Preview </strong>the data to verify the query is selecting the data required for the feature set.</p><p>Then select <strong>Configure Feature Set</strong> on the bottom right to advance to the next step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_6U6090KrahWbewg" /></figure><p>The <strong>Configure Feature Set</strong> step is where you add all the metadata to the feature set: name, description, entity, and index. An <a href="https://docs.continual.ai/feature-sets/">entity</a> is a higher level object that combines feature sets that represent common business objects such as “customers”, “products”, and “sales”. The <a href="https://docs.continual.ai/feature-sets/#feature-set">index</a> is what uniquely identifies the feature set and connects it to an entity. All feature sets in an entity have the same index. Populate the fields as shown below and create a new entity called “<strong>customer</strong>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GO90oCqQOdhKhL8p" /></figure><p>Click “<strong>Define Schema</strong>” to advance to the next step.</p><p>Notice our feature set is displayed in the Data Model graph, with all the columns, their data types, and whether they are included in this feature set.</p><p>Okay, time to review and create! Click “<strong>Review Changes</strong>” and then “<strong>Submit Changes</strong>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SIahCPe-F_gxGoS3" /></figure><p>Now, click on the “<strong>Changes</strong>” tab on the left hand side to see the action added to the activity feed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*w_uQFlKOF2HmhL7t" /></figure><p>Once the Feature Set has been created, we can see it listed on “<strong>Feature Sets</strong>” on the left vertical menu:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DNXzw3g3eJbXfoHt" /></figure><h3>Create a model</h3><p>We’ve connected to Snowflake and created a feature set for a model. Now it’s time to create a model that we will use our feature set and some additional data to predict the probability of a customer churning. The flow is very similar to creating a feature set except with some key additions.</p><p>At the configuration step, we’ll need to provide a target column to train our model against. Then we need to set policies for re-training, promotion, and running predictions. Click on “<strong>Models” </strong>on the left hand side, then “<strong>Create Model</strong>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dYvmpeMrk3YYdN6t" /></figure><p>Click “<strong>Use an example</strong>” and then select “<strong>Predict Customer Churn</strong>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/0*14XQqsqi4oe9SgZO" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/0*vnaNorK3xy0xGxG6" /></figure><p>We need to make sure our SQL query contains a unique index, features, and a target. In addition to new features we’ll define in our model spine, we want to include the feature set we previously built. We do this by including the index column of our feature set in our query and then linking it to our “customers” entity in the “Review Schema” step. Then, at model training time, Continual will join the feature set with the model to create the training data set.</p><p>We typically recommend storing your features in feature sets and connecting your models to them via <a href="https://docs.continual.ai/feature-sets/?h=entity+linking#entity">entity linking</a>, but it’s also possible to specify a list of columns in your model that represent additional features to bring into the model.</p><p>Click “<strong>Configure Model</strong>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F8ibbxQelWQGmHYJ" /></figure><p>Cool, so let’s give our model a name and description and define our model index and target column. These attributes, along with a sql query that generates the data and linked entities, forms the core of a model definition, and this is sometimes referred to as the model spine.</p><p>Click “<strong>Define Schema</strong>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6wQ-Omety5Y058j9" /></figure><p>Now it’s time to link our feature set index to our “customer” entity. Click the chain icon on the “id” row and then select “customer”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V8Yt0zEG1oweILgi" /></figure><p>Type “<strong>customer</strong>” into the pop up box:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/0*OcPlBMplSGoo4_79" /></figure><p>Then click “<strong>Link Column</strong>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/0*YafvTcazBYRcQTqt" /></figure><p>Click “<strong>Set Policies</strong>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pnrWEfi6CVPpx34_" /></figure><p>In Continual, you can configure recurring training schedules to ensure your model is updating as frequently as it needs to. You can also set advanced settings such as which performance metric to optimize for, the size of the container, and even which models to include or exclude in the experiment. While automated, Continual allows you to have control over how your model is created, optimized, deployed, and managed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cgj3qEPpHPqQKaMe" /></figure><p>You can also set how the system chooses which model to promote to production and when new predictions should be made.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_LRWs7AOvUbuq07e" /></figure><p>Go ahead and create the model by clicking “<strong>Submit Changes</strong>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pz2epOl2oWngklS0" /></figure><p>Well done! How easy was that?</p><p>All changes you make in Continual, such as creating a new feature set or editing/updating an existing model, is listed in the “<strong>Changes</strong>” tab. This gives you a lineage of your team’s work you can reference at any time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D_Ku7DtmgXO2HAve" /></figure><p>Once your model has been created and promoted it will write predictions directly back to Snowflake. Continual creates a table in your feature store for every model you create in the system that tracks all predictions made by model versions in that model over time. This table lives under <strong>&lt;feature_store&gt;.&lt;project_id&gt;.model_&lt;model_id&gt;_predictions_history</strong>. Continual additionally builds a view under <strong>&lt;feature_store&gt;.&lt;project_id&gt;.model_&lt;model_id&gt;_predictions</strong> which represents the latest prediction made for each record in your model spine.</p><p>Let’s use the latest predictions view. In Snowflake, paste the following sql statement in to view all your predictions:</p><p><strong>Open a new work sheet, name as predictions and run the code below</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/32707b175971e298d93c834e7d20ed1a/href">https://medium.com/media/32707b175971e298d93c834e7d20ed1a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bm5ZOeU5lOf1HwOStizomQ.png" /></figure><h3>MLOps: Monitoring data and models</h3><p>Back in Continual, there are many tools for monitoring your data, models, and prediction jobs. Navigate to “<strong>Models</strong>” and select the customer_churn_30days mode</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*71vI3H7ryp8LQFGQ" /></figure><p>Each time you train a model, a new version is produced and managed under the “Model Version” view.</p><p>Click “Versions” and choose a Model Version to evaluate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GiA5cTxvLvymZV86" /></figure><h3>Performance Analysis</h3><p>The “Overview” page shows the performance of the winning model, as well as each model that was tested. Continual runs a series of experiments across different model algorithms and optimizes performance across a specified performance metric.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*63j624-fMF7ljFQJ" /></figure><h3>Monitoring data</h3><p>Click on “Data Analysis” to look closer at the data used to train the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fhg_Dlts4bX5mcxY" /></figure><p>Here you can look at the <a href="https://docs.continual.ai/mlops/#correlation-matrix">correlation matrix</a> to see which two variables are correlated and <a href="https://docs.continual.ai/mlops/#data-profiling">category scores</a> to look at each feature’s profile to check if there are features with many Null values, large outliers, or unexpected distributions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8hafB4bu4dGN6ssi" /></figure><h3>Analyzing the model</h3><p>Click on “Model Insights” and look at the confusion matrix to understand what your model is getting right and what types of errors it’s making.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XjFEGG08WJvpYMfX" /></figure><p>We can also reference Feature Importance to view which features were the most impactful.</p><h3>Conclusion</h3><p>Just like that you’ve enabled machine learning on Snowflake. Continual is the AI layer for the modern data stack and designed with the shared principles of simplicity, minimal management overhead, and elasticity.</p><p>In less than 15 minutes, we connected Continual to Snowflake, created a feature set, used it as input to experiment among more than 10 models and added other relevant features, promoted the best performing model to production, wrote prediction results back to Snowflake, analyzed our features and model performance to learn what improvements we can make. We did this all in the UI but could’ve used the CLI or SDK.</p><p>This concludes the guide to quickly getting started with Continual on Snowflake. Now you’re ready for a <a href="https://docs.continual.ai/customer-churn/">more advanced example of predicting customer churn</a> with Continual and dbt on Snowflake.</p><p>After completing this tutorial, users are invited to try <a href="https://docs.continual.ai/customer-churn/">more advanced examples</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87dceae84c8a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/guide-snowflake-continual-87dceae84c8a">Enterprise AI for the Modern Data Stack</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>