<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pooja Kelgaonkar on Medium]]></title>
        <description><![CDATA[Stories by Pooja Kelgaonkar on Medium]]></description>
        <link>https://medium.com/@poojakelgaonkar?source=rss-86c81ce918a3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*n9tX09QnpiZrwJtIszp1fQ.png</url>
            <title>Stories by Pooja Kelgaonkar on Medium</title>
            <link>https://medium.com/@poojakelgaonkar?source=rss-86c81ce918a3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 23:32:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@poojakelgaonkar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Snowflake Cortex AISQL]]></title>
            <link>https://poojakelgaonkar.medium.com/snowflake-cortex-aisql-34f298c01673?source=rss-86c81ce918a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/34f298c01673</guid>
            <category><![CDATA[data-superhero]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[snowflake-data-superhero]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Pooja Kelgaonkar]]></dc:creator>
            <pubDate>Tue, 29 Jul 2025 21:49:12 GMT</pubDate>
            <atom:updated>2025-07-29T21:49:12.931Z</atom:updated>
            <content:encoded><![CDATA[<p>Thanks for reading my earlier blogs! Snowflake Cortex LLM functions enable users to use these functions interactively with SQL. Refer to my earlier blogs on Snowflake Cortex, LLM Functions, using these functions, and latest updates to LLM functions:</p><ol><li>first blog — <a href="https://medium.com/snowflake/understanding-snowflake-cortex-functions-93d6eb695ed7">https://medium.com/snowflake/understanding-snowflake-cortex-functions-93d6eb695ed7</a></li><li>Using COMPLETE() function — <a href="https://medium.com/snowflake/using-snowflake-cortex-complete-function-to-generate-responses-to-prompt-de9bbde869fa">https://medium.com/snowflake/using-snowflake-cortex-complete-function-to-generate-responses-to-prompt-de9bbde869fa</a></li><li>Using Cortex LLM Functions — <a href="https://medium.com/snowflake/using-snowflake-cortex-llm-functions-b3130ee3d40b">https://medium.com/snowflake/using-snowflake-cortex-llm-functions-b3130ee3d40b</a></li><li>Cortex LLM Updates — <a href="https://poojakelgaonkar.medium.com/snowflake-cortex-llm-functions-and-updates-4da0f78ac905">https://poojakelgaonkar.medium.com/snowflake-cortex-llm-functions-and-updates-4da0f78ac905</a></li></ol><p>All these earlier blogs and LLM functions shared can be used for structured as well as semi-structured data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yygMfybbs4OcXFyZDdSfWQ.png" /><figcaption>P.C. Snowflake quickstarts</figcaption></figure><p>Snowflake announced Cortex AISQL functions to run analytics on top of unstructured data. These functions are currently in preview and can be used to test the functionality with unstructured data using images and text. These can also be referred as the latest version of LLM functions which supports both — text and unstructured data. AISQL functions can be used to implement below use cases :</p><ul><li>Generating aggregated insights</li><li>Document parsing</li><li>Sentiment analysis</li><li>Running classification uses cases to classify content</li><li>Entity enrichment</li></ul><p>All models used to implement AI SQL functions are fully hosted in the platform. These can leverage Snowflake native features and capabilities to implement scalability, performance, security , along with data governance standards. As AISQL functions are extensions to the existing LLM functions hence these can also be categorized into two categories — Helper functions as well as SQL functions. Refer to following table for AISQL functions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rUQ4Uq9CeG0FDLTpsodgtg.png" /><figcaption>AISQL Functions</figcaption></figure><p><strong><em>Is there a way to control the unwanted or harmful responses/results while using Generative AI-based functions?</em></strong></p><p>Yes, this feature is available with Snowflake COMPLETE function as this function can be used to execute any kind of generative AI use case. We can define various parameters for COMPLETE function and guard is one of the option that can prevent from any harmful responses. You can specify ‘guardrails’: true as one of the parameter while using COMPLETE function. Read more about it here — <a href="https://docs.snowflake.com/en/sql-reference/functions/complete-snowflake-cortex#label-cortex-complete-arguments">https://docs.snowflake.com/en/sql-reference/functions/complete-snowflake-cortex#label-cortex-complete-arguments</a></p><p><strong><em>Will there be any performance impact if we use it for large inputs? What are the recommendations to get better performance?</em></strong></p><p>As these functions and models are hosted in the platform and use native capabilities to autoscale to offer optimum performance. However, its recommended to use these AISQL functions to be used as batch processing over real-time processing. These would perform better for a large SQL table or input as a batch. If these need to be implemented for a real-time then it’s recommended to use it as a REST API ( COMPLETE API) over SQL invocation.</p><p><strong><em>Does this need any additional permissions or have separate access controls other than CORTEX functions? What is the recommended method to control access?</em></strong></p><p>Snowflake offers CORTEX_USER as database role to allow users to access CORTEX AI functions. This is granted by default to the public role however as a Security Admin or Account admin , we can control the permissions or roles granted to PUBLIC role by default. You can run below SQL commands to remove permissions:</p><pre>REVOKE DATABASE ROLE SNOWFLAKE.CORTEX_USER<br>  FROM ROLE PUBLIC;<br><br>REVOKE IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE<br>  FROM ROLE PUBLIC;</pre><p>Its recommended to create a custom role to manage access to CORTEX functions and control on usage of the function as well as testing it in lower environments first. You can create a custom role and assign CORTEX_USER to the role. Later, this custom role can be granted to the users using this functionality e.g. data scientists or analysts.</p><p><strong><em>Is it possible to control the models to be used with these functions?</em></strong></p><p>Snowflake offers a variety of models that can be used and chosen while using these functions. You can control what models are allowed to be used in your environment if you want to restrict to only Snowflake offered or using llama models. You can use <strong>CORTEX_MODELS_ALLOWLIST </strong>an account-level parameter that can have a list of models that can be used in the account/environment/platform. Refer to below examples:</p><pre>/* Allow All */<br>ALTER ACCOUNT SET CORTEX_MODELS_ALLOWLIST = &#39;All&#39;;<br><br>/* Allow only selected */<br>ALTER ACCOUNT SET CORTEX_MODELS_ALLOWLIST = &#39;mistral-large2,llama3.1-70b&#39;;<br><br>/* Allow None to be chosen */<br><br>ALTER ACCOUNT SET CORTEX_MODELS_ALLOWLIST = &#39;None&#39;;</pre><p><strong>How can I track the cost associated with AISQL functions used?</strong></p><p>As you know, Snowflake metadata ACCOUNT USAGE can be used to get details of the usage. Similar to CORTEX functions, you METERING HISTORY table to filter the AI Services. Refer to below example:</p><pre>SELECT *<br>  FROM SNOWFLAKE.ACCOUNT_USAGE.METERING_DAILY_HISTORY<br>  WHERE SERVICE_TYPE=&#39;AI_SERVICES&#39;;</pre><p>Apart from the METERING HISTORY, there are a few other views dedicated for CORTEX functions to gather function usage. You can use CORTEX_FUNCTIONS_USAGE_HISTORY or CORTEX_FUNCTIONS_QUERY_USAGE_HISTORY.</p><p>You can read more about this function on snowflake documentation — <a href="https://docs.snowflake.com/en/user-guide/snowflake-cortex/aisql">https://docs.snowflake.com/en/user-guide/snowflake-cortex/aisql</a> . Get more insights into using AISQL functions:</p><ul><li><a href="https://docs.snowflake.com/en/sql-reference/functions/ai_classify">AI_CLASSIFY | Snowflake Documentation</a></li><li><a href="https://docs.snowflake.com/en/sql-reference/functions/ai_complete">AI_COMPLETE | Snowflake Documentation</a></li><li><a href="https://docs.snowflake.com/en/sql-reference/functions/ai_embed">AI_EMBED | Snowflake Documentation</a></li><li><a href="https://docs.snowflake.com/en/sql-reference/functions/ai_filter">AI_FILTER | Snowflake Documentation</a></li><li><a href="https://docs.snowflake.com/en/sql-reference/functions/ai_sentiment">AI_SENTIMENT | Snowflake Documentation</a></li><li><a href="https://docs.snowflake.com/en/sql-reference/functions/ai_similarity">AI_SIMILARITY | Snowflake Documentation</a></li></ul><p>Once you learn about these functions, you can also complete Snowflake quickstarts —</p><p><a href="https://quickstarts.snowflake.com/guide/getting-started-with-cortex-aisql/index.html#0">Getting Started with Cortex AISQL</a></p><p>Subscribe to my blog series to learn more about latest and greatest features of Snowflak, Gen AI and data domain use cases.</p><blockquote><strong>About Me :</strong></blockquote><blockquote>I am one of the Snowflake Data Superheroes 2023,2024, &amp; 2025. I am a DWBI and Cloud Architect! I am currently working as Senior Data Architect — GCP, Snowflake. I have been working with various Legacy data warehouses, data implementations, and Cloud platforms/Migrations. I am a SnowPro Core certified Data Architect as well as a Google-certified Google Professional Cloud Architect. You can reach out to me on <a href="https://www.linkedin.com/in/poojakelgaonkar/">LinkedIn</a> if you need any further help on certification, Data Solutions, and Implementations!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34f298c01673" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Snowflake Query Insights]]></title>
            <link>https://medium.com/snowflake/snowflake-query-insights-0e0aecba9224?source=rss-86c81ce918a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/0e0aecba9224</guid>
            <category><![CDATA[query-optimization]]></category>
            <category><![CDATA[data-superhero]]></category>
            <category><![CDATA[snowflake-data-superhero]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[query]]></category>
            <dc:creator><![CDATA[Pooja Kelgaonkar]]></dc:creator>
            <pubDate>Fri, 04 Jul 2025 17:20:47 GMT</pubDate>
            <atom:updated>2025-07-11T23:03:58.391Z</atom:updated>
            <content:encoded><![CDATA[<p>Snowflake’s focus on simplicity and ease of use always excites me most about the platform, features, and services. Query_insights can help to improve and optimize performance of queries even more.</p><p>Snowflake has recently added one more ACCOUNT_USAGE view that contain each row for insight query. This may not contain insights for all queries, however it can list down all queries that falls under certain conditions and that are marked for insight queries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*i--8THzHXbw3NrUpVWgsUg.png" /><figcaption>AI generated image</figcaption></figure><p><strong><em>What is query insight?</em></strong></p><p>Query insight is nothing but a message that shares information about the query that is causing the performance issue. This not only provides information on how performance is impacted but also offers recommendations to fix/improve the performance issue.</p><p><strong><em>Can all queries have insights generated for all?</em></strong></p><p>Snowflake offers insights on the query based on a few conditions, and if these conditions are met, these insights will be available in QUERY_INSIGHTS. There are a few categories where query execution may or may not lead to the resultant datasets as expected or have high data pruning or no data pruning depending on the queries.</p><p><strong><em>What are the types of checks that lead to insights?</em></strong></p><p>Snowflake queries that meet following conditions will lead to insights:</p><ol><li>Filter Not Application — the provided filter condition on query did not yield any filtered data or no records present for given condition. This also leads to full table scan or more data pruning as none of the conditions met to filter out data scanned. This is represented with QUERY_INSIGHT_INAPPLICABLE_FILTER_ON_TABLE_SCAN</li><li>Filter does not filter more records or not selective enough — in this case, the filters are present however they do not filter as expected number of records when compared with full data. This is another example where data pruned will be more and selected will be less. This is represented as QUERY_INSIGHT_UNSELECTIVE_FILTER</li><li>Filters using clustering columns — the executed query have filter conditions on columns that are used for clustering. This is represented with QUERY_INSIGHT_FILTER_WITH_CLUSTERING_KEY</li><li>Joining conditions not defined — when one or more tables are joined in the query and joins are defined without specifying the joining conditions. This may lead to the cross join yielding more data records hence pruning than needed. This is represented with QUERY_INSIGHT_JOIN_WITH_NO_JOIN_CONDITION</li><li>Query Spillage — based on the query written and executed, resultant dataset is more and need more capacity than warehouse to store, it will be pushed to the storage hence indicating data spilled to the disks. This is represented with QUERY_INSIGHT_REMOTE_SPILLAGE</li></ol><p><strong><em>Can query insights run on the platform?</em></strong></p><p>We can get insights if the queries falls into one of the conditions explained in the above section. However, there are few limitations based on the type of objects and queries. Not all can get insights, queries that uses hybrid tables and secure objects are not supported currently. Also, the explain queries are not supported and queries that are generated by Native app or queries that reuses results are not supported. Read more about reuse result here — <a href="https://docs.snowflake.com/en/user-guide/querying-persisted-results">https://docs.snowflake.com/en/user-guide/querying-persisted-results</a></p><p><strong><em>How to query insights view?</em></strong></p><p>QUERY_INSIGHTS is account_usage view and can be used same way as other usage views. This view have specific columns that share information about queries, warehouse, and three key columns — insight_type_id, message, and suggestion.</p><p>We can use query_id to check if there any insights available in the table and get recommendations to improve performance.</p><pre>SELECT query_id, insight_instance_id,insight_type_id, message, suggestions<br>  FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_INSIGHTS<br>  WHERE query_id = &#39;01bd1a3p-0203-9072-0000-06172103b023&#39;;</pre><p>We can also use some of the date and time columns to get a view of queries over a week, month, or year. We can also find specific insights for queries that are long-running queries running more than 60 minutes or more based on the application workloads overall.</p><pre>SELECT query_id, insight_instance_id,insight_type_id, message, suggestions<br>  FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_INSIGHTS<br>  WHERE start_time &gt; TO_DATE(DATEADD(DAY, -30, CURRENT_DATE()))<br>    AND total_duration &gt; 3600000;</pre><p>Hope this blog helps to get started with query insights and guides you on using insights from the view.</p><p>Thanks for reading my blog! Watch out for upcoming series to share insights of the new services, features announced. Subscribe to my blog series to learn more about latest and greatest features of Snowflake, Gen AI and data domain use cases.</p><blockquote><strong><em>About Me :</em></strong></blockquote><blockquote>I am one of the Snowflake Data Superheroes 2023, 2024, and 2025. I am a seasoned cloud and data Architect! I currently work as a Senior Data Architect and leading data practices for GCP, Snowflake. I have been working with various Legacy data warehouses, data implementations, and Cloud platforms/Migrations. I am a SnowPro Advanced Architect and SnowPro Core SME. You can reach out to me on <a href="https://www.linkedin.com/in/poojakelgaonkar/">LinkedIn</a> if you need any further help on certification, Data Solutions, and Implementations!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0e0aecba9224" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/snowflake-query-insights-0e0aecba9224">Snowflake Query Insights</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Snowflake Summit 2025 Announcements]]></title>
            <link>https://poojakelgaonkar.medium.com/snowflake-summit-2025-announcements-b971fd20626d?source=rss-86c81ce918a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/b971fd20626d</guid>
            <category><![CDATA[data-superhero]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Pooja Kelgaonkar]]></dc:creator>
            <pubDate>Fri, 06 Jun 2025 02:05:56 GMT</pubDate>
            <atom:updated>2025-06-06T02:05:56.350Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello All!</p><p>It was snowing in SFO in the month of June! I attended Snowflake summit this year and had an amazing experience listening to keynotes, sessions with lots of takeaways. It’s tough to cover everything in one blog! This is summary blog of the annoucements. Look out for upcoming blogs to get deeper insights of each of the managed services, services, and features announced this year. Following are the key areas of the announcements:</p><ol><li>Improved and Adaptive Snowflake Compute</li><li>Improved AI based expansion of Data Governance with Snowflake Horizon</li><li>Seamless collaborations with empowered innovations and sharing with Snowflake Marketplace</li><li>AI powered analytics and migrations</li><li>Simplicity and seamless data interoperatability with extended engineering managed services with Openflow, dbt, Apache icberg, and DevOps enhancements</li><li>Using data to take actions using agentic AI</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WcYigbJn48yjD_2DCeAOXw.png" /></figure><p>Here are more details of the each section and corresponding annoucements:</p><p><strong><em>Using data to take actions using Agentic AI</em></strong></p><p>Now, engineers, data analysts, and data scientists can use structured, semi-structured, and unstructured data to take actions and drive insights without any complex tooling or managing infrastructure.</p><ol><li>Snowflake Intelligence (Pub Pr) : currently in public preview, offering users to query underlying data in natural language. This can be used not to query the data in NLP but also to run complex analysis, get insights analytically to take next steps. Snowflake Intelligence is powered by Anthropic and OpenAI LLMs, running inside the Snowflake boundaries, and cortex agents (public preview) under the hood.</li><li>Cortex AISQL (Pub Pr): Currently in Public Preview that offers multimodal data processing same as SQL processing within Snowflake platform. This can be used to query unstructured data(images, video audio files, messages, documents) without need to pre-process , converting them to structured data. Snowflake AISQL is powered to handle all required pre-processing steps and query data automatically. Snowflake also introduced a new data type where these unstructured files can be stored within Snowflake storage and can be accessed from Snowsight just as structured data. this can also be used to fetch data from documents, pdfs with schemaaware tables, the table is defined based on the format of the document and store data.</li><li>Obeservability with Cortex AI ( Pub Pr) : Prepared to be generally available soon, this offers users to monitor generative AI apps without any additional coding or tooling complexities.</li></ol><p><strong><em>Simplicity and seamless data interoperatability</em></strong></p><p>Typically, we end up using multiple tools and integrators while we build a data ecosystem with ETL or ELT approach. More tools more is the complexity of the engineering process to manage. Data integrators, orchestrators, engineering pipelines often lead to large systems. With new announcements of Snowflake, engineering complexities can be reduced and stay within the same Snowflake project /perimeter.</p><ol><li>Snowflake Openflow: This is powered by Apache NiFi that offers variety of connectors/plugins that can be used to integrate with heterogeneous sources or consumers from Snowflake platform itself. This allows engineers to use single platform to build native ingestion pipelines and transform data.</li><li>dbt projects on Snowflake: dbt can be used to automate pipeline devleopment, have configuration driven workloads while we define models and build automated test cases, quality use cases. Now, dbt is available as part of Snowflake ecosystem. We can manage projects, models and pipelines within Snowflake environment through Snowsight. This gives a single platform and UI experience while engineers work on developing pipeline.</li><li>Improved support to Apache iceberg: We can create a single view of all linked catalogs, Federation streamlines data discovery and enables activation on any engine that supports Iceberg REST catalog integrations. Federation is coming to Open Catalog soon in private preview.</li></ol><p><strong><em>AI powered analytics and migrations</em></strong></p><p>Snowflake announced a few advancements to make AI and analytics faster. We can use these to migrate from legacy systems to Snowflake making migrations easier and faster.</p><ol><li>SnowConvert AI: SnowCovert is now available to be used for free for users looking to migrate to Snowflake. Convert AI is AI based automated conversions, migrations to the Snowflake. This intelligently convert existing code and runs data validations lowering risk of migration associated with large migrations.</li><li>Snowflake Semantic Views(Pub Pr) : This is currently in public preview which allow users to define a view across source, enriched data that offers a overlap of business entity with physical entities. These are beneficial for business analysts, BI analysts which offers mapping of business to physical entities along with metrics, dimensions that can be used to run analytics.</li></ol><p><strong><em>Seamless collaborations with empowered innovations</em></strong></p><p>Snowflake announced internal data sharing and data sharing powered with AI to ask questions to the data.</p><ol><li>Cortex Knowledge Extensions — This will be generally available soon and used to get real time responses from internet publications like USA TODAY, Packt, Stack Overflow etc.</li><li>Semantic model sharing (Private Preview) — This is private preview feature that offers sharing of models with internal users as well as third party users/providers. This also allows users to use natural language to talk to data across.</li><li>Agentic Snowflake native apps on Snowflake Marketplace — This is a new way to provders to develop Snowflake native apps that can refer to Cortex Agent APIs.</li></ol><p><strong><em>Improved and Adaptive Snowflake Compute</em></strong></p><p>Snowflake announced two types of Snowflake compute —</p><ol><li>Adaptive Compute — This is private preview offering where Snowflake warehouses are adaptive where performance of warehouses can be obtained with varying size of warehouses automatically</li><li>Snowflake Gen2 Warehouses: Gen2 warehouses are 2times (2X) faster than the standard warehouses. These are improved version of standard warehouses which gives faster performance to any DML operations being run on the warehouses (insert, delete, merge, update etc). Now, we can optimize the workloads 2X with Gen2 warehouses.</li></ol><p><strong><em>Improved AI based expansion of Data Governance with Snowflake Horizon</em></strong></p><p>Snowflake announced expansions to the Snowflake Horizon enabling users to make use of single platform to implement governed features, services.</p><ol><li>Horizon Catalog- Snowflake horizon now can be used to discover external data discovery for external sources like iceberg, dashboards, relational databases , semantic models etc.</li><li>Copilot for Snowflake Horizon — Use natural language to perform governance, security, and metadata discovery tasks in Snowsight.</li><li>Snowflake security extensions of trust center enforce users to enable MFA of users restricting them to have password based logon only.</li></ol><p>Thanks for reading my blog! Watch out for upcoming series to share insights of the new services, features annouced. Subscribe to my blog series to learn more about latest and greatest features of Snowflake, Gen AI and data domain use cases.</p><blockquote><strong>About Me :</strong></blockquote><blockquote>I am one of the Snowflake Data Superheroes 2023,2024, and 2025. I am data and Cloud Architect! I am currently working as Senior Data Architect — GCP, Snowflake. I have been working with various Legacy data warehouses, data implementations, and Cloud platforms/Migrations. I am a SnowPro Core certified Data Architect as well as a Google-certified Google Professional Cloud Architect. You can reach out to me on <a href="https://www.linkedin.com/in/poojakelgaonkar/">LinkedIn</a> if you need any further help on certification, Data Solutions, and Implementations!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b971fd20626d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SnowPro Advanced: Architect Certification Study Guide]]></title>
            <link>https://poojakelgaonkar.medium.com/snowpro-advanced-architect-certification-study-guide-95819e413752?source=rss-86c81ce918a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/95819e413752</guid>
            <category><![CDATA[snowflake-data-cloud]]></category>
            <category><![CDATA[snowflake-data-superhero]]></category>
            <category><![CDATA[data-superhero]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Pooja Kelgaonkar]]></dc:creator>
            <pubDate>Sat, 22 Mar 2025 03:00:42 GMT</pubDate>
            <atom:updated>2025-03-22T03:02:19.120Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*aQOLFK6ApmtukxIsetf5OA.png" /><figcaption>Snowflake Advanced Architect Badge</figcaption></figure><p>Recently, I have completed my advance architect certification and its a great experience to appear for this exam. This exam evaluates architectural and design skills with understanding of the data ecosystem. Snowflake experience is essential to start preparing for this certtification. If you are an architect and working on designing data solution using Snowflake features and services then you can plan on preparing for this certification. I would like to share my experience and references to prepare for advanced architect certification!</p><p>There are variety of certifications available for users. Snowflake offers SnowPro Associate, SnowPro Core, SnowPro Specialty, and SnowPro Advanced categories. Refer to <a href="https://learn.snowflake.com/en/certifications/">https://learn.snowflake.com/en/certifications/</a> for more details of available certifications.</p><p><strong><em>What is covered in exam?</em></strong></p><p>Snowflake Advanced Architect certification exam tests ability of architects/users to design end to end data pipelines, data consumption from source to Snowflake, design and deploy data platform design and architecture that cater to business , security, and compliance requirements. Using third party tools, partner integrations, connectors to design data integrations, ingestions, and implementing data workloads on Snowflake. Using Snowflake’s data sharing features and capabilities to share data with consumer users. Designing data platform architectures and solutions to implement data intense applications with Snowflake including data engineering, data ingestions, data transformations, data governance, data quality, data sharing, data protection, data privacy, encryption, data consumers etc.</p><p><strong><em>Who can take this exam?</em></strong></p><p>Architects and engineers with 2+ hands-on experience with Snowflake. Arcitects working on solutions and designing data applications with skills to develop SQL data workloads, experience in implementing ETL and ELT design approaches, Hands-on experience implementing data security and govenance with Snowflake. Hands-on experience working in production environments to develop, support, manage. Solution architects, data architects, system architects, cloud architects , data engineers with Snowflake experience can appear for this exam.</p><p><strong><em>What are the pre-requisites?</em></strong></p><p>2+ years of experience with Snowflake . Preferably hands-on exp to implement and manage workloads in Snowflake production environment. A valid SnowPro Core certification is mandatory to appear for this exam.</p><p><strong><em>How much this exam pattern?</em></strong></p><p>Snowflake guide shares the pattern and registration fees required to register for the exam:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2ijBU9LsP9gC5cGAGp834g.png" /><figcaption>Snowflake guide</figcaption></figure><p><strong><em>What are topics covered in exam?</em></strong></p><p>Certification exam covers following areas and weightage in exam.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ja6Gc7wen6_bB6kzLV-D6w.png" /><figcaption>P.C. Snowflake certification guide</figcaption></figure><p>There are subtopics listed for each domain that specifies list of features and services evaluated as part of each domain. You can refer to <a href="https://learn.snowflake.com/en/certifications/snowpro-advanced-architect/">https://learn.snowflake.com/en/certifications/snowpro-advanced-architect/</a> for domain breakdown.</p><p>Get detailed topics to prepare for each domain — Snowflake certification guide — <a href="https://training.snowflake.com/lmt/clmsCatalogDetails.prMain?site=sf&amp;in_offeringId=98873813&amp;in_language_identifier=en-us&amp;in_region=us&amp;in_from_module=CLMSSHARE.PRMAIN">https://training.snowflake.com/lmt/clmsCatalogDetails.prMain?site=sf&amp;in_offeringId=98873813&amp;in_language_identifier=en-us&amp;in_region=us&amp;in_from_module=CLMSSHARE.PRMAIN</a></p><p><strong><em>How to prepare for exam?</em></strong></p><p>This is one of the toughest exam to pass however you can strategize and plan on preparing for the exam. You can go through the guide and start listing down the subtopics covered in each. Based on your experience and expertise start updating bullet list to identify topics you are great at, topics that needs to be worked upon and topics that are to be started. This will help you to guage time required to prep yourself for the exam. This excercise helped me to prepare in a week’s time. I am working on designing solutions on day today basis hence week’s time was enough to prepare for this exam. Revising topics and labs that you are great at will help to gain initial confidence required to prepare for the cert.</p><p><strong><em>What are the key topics to be prepared for exam?</em></strong></p><p>Snowflake architecture is the key and all topics covered in SnowPro Core are base to start preparing for the exam. Following are the key areas and topics to prepare well for the cert exam:</p><ol><li><em>Snowflake Accounts and Editions</em> — understand what is organization and accounts setup. what all scenarios where one or multiple accounts used to setup the environments , domains and what are the prons and cons of managing one or more accounts. what features are used for business critical edition and understand use cases difference for enterprise and business critical editions.</li><li><em>Access Control and Managing custom roles </em>— setting up custom roles and assigning roles to users. Enabling MFA for users and managing federated authentication for users.</li><li><em>Data Models and Constraints</em> — understand types of data models and differences between each of them — Star, Snowflake, Data Vault etc. Different types of tables , keys and constraints supported as part of data model. how semi-structured data is handled using VARIANT, JSON data stored and accessed. Managed schema vs unmanaged schema, understand how it differs and roles access granted for existing and duture objects.</li><li><em>Parameters and roles </em>— understand Snowflake context setting, parameters supported and used at account, database objects, and user sessions. Understand primary and secondary roles with their usage, how these can be used in a session by users.</li><li><em>Security controls </em>— Network policies and implementation, ingress and egress setup, integrating with SSO (Single Sign On), Federated authentication, MFA , allowing users to connect to Snowflake instance. Securing data platform connections and onfiguring connectivity from programmatic approach, Snowsight, connectors and drivers.</li><li><em>Data Engineering </em>— ETL vs ELT approach, what services and integrations used for data ingestions. Snowflake native -COPY, Snowpipe , third party tools and integrators, partner integrators to ingest data to Snowflake. implementing medallion architecture, managing Snowflake data obejcts, change data capture, streams and trasks, dynamic tables etc.</li><li><em>Streaming Data processing </em>— understand Kafka connector, Snowpipe, Snowpipe streaming thoroughly. List down differences between these approaches and what are the benefits over each other. What scenarios used to implement streaming features. How snowpipe automate data loads, Kafka connector loads data, learn how data is treated and loaded with metadata fields like RECORD_CONTENT &amp; RECORD_METADATA. Understand Snowpipe API and using API endpoints. Choosing appropriate streaming service to integrate real time loads in Snowflake.</li><li><em>Bulk loading &amp; unloading</em> — COPY INTO is key and how its used to load data to Snowflake. COPY options is critical and how each of them is used to handle errorneous records and scenarios. VALIDATE, ERROR handling , transformations with COPY command for loading and unloading data. File formats and how file formats can be defined. What all places file formats can be defined and used.</li><li><em>Data Cloning vs Replication</em> — How data cloning works and usage of data cloning. Understand scenarios where data cloning can be used and limitations of cloning. what is the difference between cloning and replication? data replication is used in what all scenarios and understand use cases. Understand the use cases where replication can fail and what all objects supported in replication with limitations.</li><li><em>Data Sharing </em>— what is data sharing and how this is used to share data. Understand the use cases where data can be shared within same region, across regions of same cloud platform, across regions and platforms, data shared between different clouds. What are the use cases of data sharing — how data can be shared between Snowflake accounts as well as non-snowflake accounts. Usage computation and monitoring usage using account usage views.</li><li><em>Snowflake scripting and SQL</em> — Snowflake SP , UDFs, UDFTs and how these can be defined, used, shared. understand how functions can be invoked and managed in Snowflake. what are the caller permissions vs owner permissions. understand the scenarios and use cases where these permissions are used and how object permissions are implemented for caller and owner roles. what are limitations and use cases of external functions. understand the scenarios where external functions can be used.</li><li><em>Performance Optimization </em>— understand the options of performance otpimization and materialized views used to improve performance. What are the limitations of MVs. how the performance improved and clustering defined on top of MVs. use cases where MVs used by optimizer over tables.</li><li><em>Search Optimization service vs Query Acceleration Service</em> — understand how these services are used to improve performance of workloads. what types of workloads supported and scenarios where one can be picked over other as well as understand scenarios of using both these services together. understand the overall usage of these services and tracking the billing, consumption etc. functions used to check the eligibility of SQL queries for Query acceleration service (QAS).</li><li><em>External tables </em>— how external tables are defined, used, performance, partitioning, querying external tables. understand how external tables work in case of replication and sharing. how external table can be replicated if needed. How data is accessed and what are the metadata columns i.e., <em>VALUE, METADATA$FILENAME, METADATA$FILE_ROW_NUMBER used. W</em>hat is schema evolution and how its handled with external table.<em> </em>understand scenarios, use cases where external tables can be used.</li><li><em>Cost and cost Optimization</em> — understand Snowflake pricing and cost components. Understand how warehouses are charged, what are the types of warehouses, scaling — horizontal vs vertical scaling, scaling mode, storage costs, factors that impacts storage cost, how can we optimize overall cost, cloud services cost and impact to overall cost. understand optimization of the cost and what services can be used to reduce the cost. resource monitors and using them.</li><li><em>Snowflake micro-partitions and clustering </em>— understand how data is stored and partitions maintained, how data is loaded, stored, and updated imacts micro partitions, how pruning works and query profile represents pruning. understand the clustering concepts and how system function help to identify clustering depth, clustering_information, how its represented in query profile. what is data pruning and data spilled to the disk as well as to remote storage.</li><li><em>Query Optimization </em>— understand how queries can be optimized, query profile, components and sections of query profile, query_history view, how query performance is computed, how to identify queries being queued, spilled etc.</li><li><em>Snowflake Cache </em>— understand what are the cache available in platform — result cache, warehouse cache, cloud services metadata used to optimize queries. how cache is maintained and how optimizer decide on using cache, understand scenarios where query result cache is used, scenarios where warehouse cache is used and cache maintained, understand how queries can benefit from these types of cache.</li></ol><p><strong><em>What kind of quesions to be expected in exam?</em></strong></p><p>All questions are multiple choice questions. Most of the quesions are scenario based questions on design choices, implementation choices, preferences, services and features used to implement design pattern. Most of the quesions are multiple selections -choose two or three options from given choices. Expect only few questions that are straight forward and single choice questions — its like 80–20% , 20% are single choice over 80% of two or more correct choices to be selected.</p><p><strong><em>Sample questions for reference:</em></strong></p><pre>Q1. format of data/file can be defined at what all places (Choose Three)<br>1. CREATE FILE FORMAT<br>2. CREATE STAGE<br>3. CREATE STORAGE INTEGRATION<br>4. CREATE TABLE<br>5. CREATE PIPE<br>6. COPY INTO<br><br>Correct Answer (1,2,4)</pre><pre>Q2. An architect clones the prod database to a non-prod database. He observes that the task cloned is not running, what can be the reason<br>1. Task cannot be cloned as part of data cloning<br>2. The task is cloned but in suspended state and need to resume<br>3. The task is still referencing to the source database and not triggered<br>4. The task cloned contains reference to the object that is not cloned hence<br> task is not running<br><br>Correct Answer (2)</pre><pre>Q3. A table is partitioned on a column that contains unique values and there are 10 unique values and stored in different partitions. <br>There is clustering defined on top of the table then what will be the clustering depth<br>1. 0<br>2. 1<br>3. 5<br>4. 10</pre><pre>Q4. An architect defines an external table on top of an external stage. What all <br>metadata fields used to maintain the record reference for external table (Choose three)<br>1.METADATA$FILENAME<br>2.METADATA$VALUE<br>3.METADATA$FILE_ROW_NUMBER<br>4.VALUE<br>5.METADATA$FILE<br><br>Correct Answer (1,3,4)</pre><pre>Q5. There are huge source files to be loaded to Snowflake as batch data loads.<br>These files are stored on cloud storage to be loaded to Snowflake tables. <br>What option will be efficient to load volumeneous data to Snowflake?<br>1. Using ETL tool<br>2. Using partner integration tool<br>3. Define External table<br>4. Load using COPY command<br>5. Define Snowpipe<br><br>Correct Answer (4)</pre><p><strong>References to prep you up for scenario based questions:</strong></p><p>Snowflake certification guide shares references of quickstarts, documentations, reference architectures, white papers, community blogs, and books. Follow all given references carefully to prepare for various scenario based design patterns. Some of the references are as below:</p><ol><li><a href="https://www.snowflake.com/resource/7-snowflake-reference-architectures-application-builders/">https://www.snowflake.com/resource/7-snowflake-reference-architectures-application-builders/</a></li><li><a href="https://www.snowflake.com/en/solutions/departments/marketing/customer-360/">https://www.snowflake.com/en/solutions/departments/marketing/customer-360/</a></li><li><a href="https://www.snowflake.com/en/resources/white-paper/definitive-guide-to-managing-spend-in-snowflake/">https://www.snowflake.com/en/resources/white-paper/definitive-guide-to-managing-spend-in-snowflake/</a></li><li><a href="https://www.snowflake.com/guides/iot-architecture-for-iot-analytics">https://www.snowflake.com/guides/iot-architecture-for-iot-analytics</a></li><li><a href="https://www.snowflake.com/en/blog/modern-data-streaming-pipeline-architectures-industry-use-cases/">https://www.snowflake.com/en/blog/modern-data-streaming-pipeline-architectures-industry-use-cases/</a></li><li>Snowflake Documentation — <a href="https://docs.snowflake.com/">https://docs.snowflake.com/</a></li></ol><p>Snowflake documentation is key resource to cover all required topics, features, and services. Understand all basics of services and design aspects, design patterns, ingestion patterns, engineering patterns, performance, optimization, choosing one over other.</p><p>Best wishes for your upcoming exam!</p><p>Subscribe to my blog series to learn more about latest and greatest features of Snowflake, Gen AI and data domain use cases.</p><blockquote><strong><em>About Me :</em></strong></blockquote><blockquote><em>I am one of the Snowflake Data Superheroes 2023,2024, and 2025. I am data and Cloud Architect! I am currently working as Senior Data Architect — GCP, Snowflake. I have been working with various Legacy data warehouses, data implementations, and Cloud platforms/Migrations. I am a SnowPro Core certified Data Architect as well as a Google-certified Google Professional Cloud Architect. You can reach out to me on </em><a href="https://www.linkedin.com/in/poojakelgaonkar/"><em>LinkedIn</em></a><em> if you need any further help on certification, Data Solutions, and Implementations!</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95819e413752" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting started with Snowflake AI Assistant]]></title>
            <link>https://poojakelgaonkar.medium.com/getting-started-with-snowflake-ai-assistant-082a97d89d63?source=rss-86c81ce918a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/082a97d89d63</guid>
            <category><![CDATA[data-superhero]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Pooja Kelgaonkar]]></dc:creator>
            <pubDate>Fri, 10 Jan 2025 17:49:00 GMT</pubDate>
            <atom:updated>2025-01-10T17:49:00.554Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello Everyone!</p><p>Happy New Year!! Let’s begin this new year with a resolution to keep learning, sharing knowledge, experience, and growing with people!!</p><p>This year’s first blog is all about understanding the Snowflake AI Assistant — Snowflake Co-Pilot and using it to develop, optimize, get recommendations, generate sql queries and more. Lets begin with series of questions to understand more!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YZwFcJNvVWgGqYBKEnRbpQ.png" /><figcaption>AI generated Image of Co-Pilot from Canva</figcaption></figure><ol><li><strong><em>What is Snowflake Co-Pilot?</em></strong></li></ol><p>Snowflake Copilot is an AI-powered assistant designed to simplify data analysis within Snowflake while ensuring strong data governance. Running securely inside Snowflake Cortex, it respects RBAC policies and only accesses authorized datasets. Using natural language requests, it helps explore data structures, generate and optimize SQL queries, and address potential issues. Additionally, Copilot enhances SQL fluency and understanding of Snowflake features by providing guidance based on official documentation.</p><p><strong><em>2. How access is granted to users to use Co-Pilot?</em></strong></p><p>Snowflake has a database role — COPILOT_USER and this is granted to PUBLIC role by default. This means that the Co-Pilot is accessible to the users with least or Public role assigned by default. Hence, all users will see the Co-Pilot feature by default as PUBLIC role gets assigned automatically to the users as and when we create users.</p><p><strong><em>3. How to use Co-Pilot?</em></strong></p><p>You can login to the Snowflake account and open a new worksheet or open existing worksheet. You can locate the Ask Co-Pilot button on the bottom right corner of the Snowsight! if you are working on a new SQL worksheet click new worksheet, you will see the button on the screen! That’s the Co-Pilot ready to be used. Refer to below screenshot :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cDY4tu9xlxO6j8grvg1k3Q.png" /></figure><p>click on the Ask button and a window pops up as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*yPXNUW4zr2spUoOsIrKGyA.png" /></figure><p>This will also ask to set the database and schema. Once you select the database, it will show this on the screen:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*qeBuikP93fN2QcRnZyZ4Ig.png" /></figure><p>This can read the underlying tables, data and help to write queries. This can also help to understand the data and analyze the data patterns. As you can see, we can use this to generate sql queries, optimize them, get recommendations etc.</p><p>Each worksheet or notebook is a separate chat session for Co-Pilot. We must provide a database and schema for the session.</p><p><strong><em>4. What are the use cases can be solved using Co-Pilot?</em></strong></p><p>Snowflake Co-Pilot can be used to implement variety of use cases and help developers to develop applications faster by improving their productivity. Following are some of the well known use cases :</p><ul><li><strong>Analysing datasets anbd data :</strong></li></ul><p>We can ask open-ended questions to learn more about the structure and nuances of a new dataset.</p><ul><li><strong>SQL query generation:</strong></li></ul><p>We can ask questions in plain English to generate the SQL queries. If someone is new to the platform or doesnot know how to write SQL queries then they can use Co-Pilot to ask questions in natural language to get the corresponding SQL queries.</p><ul><li><strong>Query Trials:</strong></li></ul><p>As Co-Pilot is part of the ecosysem, it allow users to edit the queries suggested by Snowflake Copilot and run them before using it.</p><ul><li><strong>Developing complex queries:</strong></li></ul><p>As this interface can maintain a context, we can use it like a conversation by asking follow-up questions to refine the suggested SQL query or dig deeper into the analysis.</p><ul><li><strong>Learning Platform:</strong></li></ul><p>This can also be used to enhance the learning. As this is also trained on the Snowflake documentation, we can ask any questions about Snowflake concepts, capabilities, and features to learn more about the platform. This gives relevant references and examples to help quick learning.</p><ul><li><strong>Query Optimixation and Recommendations:</strong></li></ul><p>We can use Co-Pilot to optimize the query or share recommendations. This can also help to assess query efficiency, find optimizations, or explain what the query does.</p><p><strong><em>5. How can we share feedback for Co-Pilot or is there any way we can share the improvements?</em></strong></p><p>yes, Snowflake Co-Pilot feedback feature can help to gather the feedback on the responses. We can use the thumb button to share the feedback if the responses were expected.</p><p><strong><em>6. Can we limit the users of Co-Pilot and restrict the users access to start with?</em></strong></p><p>yes, we can set the database role not to be granted to the PUBLIC role by default. We can maintain it along with other database roles and have permissions granted accordingly. We can use following queries to implement the change:</p><pre>USE ROLE ACCOUNTADMIN;<br><br>REVOKE DATABASE ROLE SNOWFLAKE.COPILOT_USER<br>  FROM ROLE PUBLIC;<br><br>CREATE ROLE copilot_trial_role;<br>GRANT DATABASE ROLE SNOWFLAKE.COPILOT_USER TO ROLE copilot_trial_role;<br><br>GRANT ROLE copilot_trial_role TO USER poojakelgaonkar;<br></pre><p><strong><em>7. Can we refer to the table columns while providing prompts to the Co-Pilot?</em></strong></p><p>yes, we can provide the column names as we select the database and schema for the session. we can refer to the column names as parameters or references using @ to refer to specific values. we can use @ to refer to specific tables.</p><p><strong><em>8. What is the cost associated with Co-Pilot usage?</em></strong></p><p>Currently, Snowflake Co-Pilot is free to use. Snowflake team will announce the pricing and billing details before starting Co-Pilot usage charges.</p><p><strong><em>9. Are there any specific regions where Co-Pilot is available for users?</em></strong></p><p>Currently, Co-Pilot is available only on AWS and Azure with limtied regions. Following are the regions where its available:</p><ul><li>AWS us-east-1</li><li>AWS us-west-2</li><li>AWS eu-central-1</li><li>Azure eastus2</li></ul><p>You can read more about this function on snowflake documentation — <a href="https://docs.snowflake.com/en/user-guide/snowflake-copilot">https://docs.snowflake.com/en/user-guide/snowflake-copilot</a> to get started with it for your use cases.</p><p>Subscribe to my blog series to learn more about latest and greatest features of Snowflake, Gen AI and data domain use cases.</p><blockquote><strong>About Me :</strong></blockquote><blockquote>I am one of the Snowflake Data Superheroes 2023 &amp; 2024. I am a DWBI and Cloud Architect! I am currently working as Senior Data Architect — GCP, Snowflake. I have been working with various Legacy data warehouses, data implementations, and Cloud platforms/Migrations. I am a SnowPro Core certified Data Architect as well as a Google-certified Google Professional Cloud Architect. You can reach out to me on <a href="https://www.linkedin.com/in/poojakelgaonkar/">LinkedIn</a> if you need any further help on certification, Data Solutions, and Implementations!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=082a97d89d63" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Snowflake Cortex LLM Functions and updates]]></title>
            <link>https://poojakelgaonkar.medium.com/snowflake-cortex-llm-functions-and-updates-4da0f78ac905?source=rss-86c81ce918a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/4da0f78ac905</guid>
            <category><![CDATA[data-superhero]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[snowflake-data-superhero]]></category>
            <dc:creator><![CDATA[Pooja Kelgaonkar]]></dc:creator>
            <pubDate>Fri, 20 Sep 2024 17:37:57 GMT</pubDate>
            <atom:updated>2024-09-20T17:37:57.807Z</atom:updated>
            <content:encoded><![CDATA[<h3>Snowflake Cortex LLM Function updates</h3><p>Thanks for reading my earlier blogs! This is the next blog in the series about understanding and using Snowflake Cortex functions. The first blog helps you to understand the remaining Snowflake Cortex functions and sample use cases to implement them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*eCqBoiEEz-YkEY7cwCroUg.png" /></figure><p>Read the blog series here —</p><ol><li>first blog — <a href="https://medium.com/snowflake/understanding-snowflake-cortex-functions-93d6eb695ed7">https://medium.com/snowflake/understanding-snowflake-cortex-functions-93d6eb695ed7</a></li><li>Using COMPLETE() function — <a href="https://medium.com/snowflake/using-snowflake-cortex-complete-function-to-generate-responses-to-prompt-de9bbde869fa">https://medium.com/snowflake/using-snowflake-cortex-complete-function-to-generate-responses-to-prompt-de9bbde869fa</a></li><li>Using Cortex LLM Functions — <a href="https://medium.com/snowflake/using-snowflake-cortex-llm-functions-b3130ee3d40b">https://medium.com/snowflake/using-snowflake-cortex-llm-functions-b3130ee3d40b</a></li></ol><p>In this blog, we will cover the new releases, updates, and functions to Snowflake Cortex LLM functions.</p><p><strong><em>New Functions Added:</em></strong></p><ol><li>EMBED_TEXT : A new vector embedding function added — EMBED_TEXT_1024 that outputs 1024 dimensions vector. It is available in all accounts for selected regions of Cortex functions on AWS and Azure. The syntax to use this function is :</li></ol><pre>SNOWFLAKE.CORTEX.EMBED_TEXT_1024( &lt;model&gt;, &lt;text&gt; )</pre><p>This supports set of models to generate vector — nv-embed-qa-4 (English only) , multilingual-e5-large, and voyage-multilingual-2.</p><p>EMBED_TEXT_768 is also available that can be used 768 dimensions from english language only. This supports LLM models — snowflake-arctic-embed-m and e5-base-v2</p><p>2. FINETUNE : This function is available in AWS regions — AWS US West 2 (Oregon) , AWS US East 1 (N. Virginia), AWS Europe Central 1 (Frankfurt), and Azure East US 2 (Virginia). This function is used to create and manage custom functions. This is used to CREATE | SHOW | DESCRIBE | CANCEL models. Refer to following examples to create, manage, show, and describe models:</p><pre>SELECT SNOWFLAKE.CORTEX.FINETUNE(<br>  &#39;CREATE&#39;,<br>  &#39;customized_model&#39;,<br>  &#39;mistral-7b&#39;,<br>  &#39;SELECT prompt, response FROM movie_reviews&#39;<br>);</pre><p>FINETUNE — SHOW is used to specify a list of the fine-tuning jobs for which you have access.</p><pre>SELECT SNOWFLAKE.CORTEX.FINETUNE(&#39;SHOW&#39;);</pre><p>3. CLASSIFY_TEXT() — This is a new function added to Cortex LLM library. This is used to classify the text used.</p><pre>SELECT SNOWFLAKE.CORTEX.CLASSIFY_TEXT(&#39;One day I will win the master chef competition&#39;, [&#39;travel&#39;, &#39;cooking&#39;]);</pre><p>4. New models available in regions — llama3.1-405b model is available on AWS east-US region.</p><p>5. Mistral AI’s Mistral Large 2 large language model (LLM) is available for serverless inference in <a href="https://www.snowflake.com/en/data-cloud/cortex/">Snowflake Cortex AI</a>.</p><p>These functions are very easy to use and implement the Generative AI use case for your applications or on top of your application data. Your data processed wont be sent outside the Snowflake data boundaries. You dont need to move your data or bring it outside to implement any the AI use cases. Snowflake Cortex functions have enabled many users to use these features with seamless integrations.</p><p>You can read more about this function on snowflake documentation — <a href="https://docs.snowflake.com/en/user-guide/snowflake-cortex/llm-functions">https://docs.snowflake.com/en/sql-reference/functions/complete-snowflake-cortex </a>to implement it for your use cases.</p><p>Subscribe to my blog series to learn more about latest and greatest features of Snowflak, Gen AI and data domain use cases.</p><blockquote><strong><em>About Me :</em></strong></blockquote><blockquote><em>I am one of the Snowflake Data Superheroes 2023 &amp; 2024. I am a DWBI and Cloud Architect! I am currently working as Senior Data Architect — GCP, Snowflake. I have been working with various Legacy data warehouses, data implementations, and Cloud platforms/Migrations. I am a SnowPro Core certified Data Architect as well as a Google-certified Google Professional Cloud Architect. You can reach out to me on </em><a href="https://www.linkedin.com/in/poojakelgaonkar/"><em>LinkedIn</em></a><em> if you need any further help on certification, Data Solutions, and Implementations!</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4da0f78ac905" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Snowflake’s Data Metric Functions]]></title>
            <link>https://poojakelgaonkar.medium.com/using-snowflakes-data-metric-functions-60a3f6fc01d0?source=rss-86c81ce918a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/60a3f6fc01d0</guid>
            <category><![CDATA[snowflake-data-superhero]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-superhero]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Pooja Kelgaonkar]]></dc:creator>
            <pubDate>Thu, 19 Sep 2024 16:48:05 GMT</pubDate>
            <atom:updated>2024-09-19T16:48:05.943Z</atom:updated>
            <content:encoded><![CDATA[<p>Thanks for reading my earlier blogs! This blog is the next blog in the series of “Data Quality” and “Data Governance” series. Refer to my earlier blog on understanding data metrics functions — <a href="https://poojakelgaonkar.medium.com/understanding-snowflakes-data-metric-functions-f9b218f378f7">https://poojakelgaonkar.medium.com/understanding-snowflakes-data-metric-functions-f9b218f378f7</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X8oFJr8ODuhz-02SdaKK4w.png" /></figure><p>This blog focuses on learning about the system defined DMFs and using them to validate the quality of the data. There are various built in functions available that caters to the accuracy, freshness, generating stats, uniqueness and generating volume of the data. These are data quality metrics that are used to validate and verify the quality of the data.</p><ol><li><strong><em>Validating the uniqueness of the data</em></strong>: As you know, data consistency is plays a vital role in data and analytics domain. Data engineering pipelines are designed in way to avoid any duplicate data processing and data duplication at the target. Snowflake offers system DMFs to validate the data uniqueness with <em>DUPLICATE_COUNT and UNIQUE_COUNT. </em>SNOWFLAKE.CORE.DUPLICATE_COUNT is function used to validate the duplicate records present for a given field. You can use them as follows:</li></ol><pre>SELECT SNOWFLAKE.CORE.DUPLICATE_COUNT(customer_id) from customer_info;</pre><p>This function also considers the NULL values along with the field values. This can be implemented on DATE,VARCHAR, FLOAT, NUMBER, and TIMESTAMP columns.</p><p>UNIQUE_COUNT can also be used in similar fashion for same set of data types as DUPLICATE_COUNT. This gives count of unique values present in the field.</p><pre>SELECT SNOWFLAKE.CORE.UNIQUE_COUNT(zip_cd) from customer_info;</pre><p>ROW_COUNT can also be used to get the total number of rows from a table. This can be used to detect the data anomaly for critical data fields or sources. If the row count has a huge variation in comparison with accepted range then this can generate an alert as part of data anomaly detection. These can be set to run on a schedule.</p><pre>ALTER TABLE sales.sales_history SET<br>  DATA_METRIC_SCHEDULE = &#39;USING CRON 0 8 * * * UTC&#39;;<br>/* runs daily 8AM to validate the data */</pre><p>2. <strong><em>Data staleness </em></strong>is another challenge in data processing and data world. At times, the data processing gets delayed and SLAs are missed. We may encounter an issue of stale data reporting to end users as well as reports. Ensuring the data processed and available is fresh is essential before this gets fed into analytical reporting and apps. Snowflake’s built-in function UNIQUENESS can be used to determine whether data is fresh to be consumed by the users. This function returns a boolean value that can be used to validate the quality check of data freshness. As this evaluates the data freshness, this can be used only with DATE and TIMESTAMP columns.</p><pre>SELECT SNOWFLAKE.CORE.FRESHNESS(<br>  SELECT<br>    sale_ts<br>  FROM sales.sales_history<br>) &lt; 300;</pre><p>This SQL checks if the data refreshed for sales purchase is refreshed in real time within 5minutes of latency.</p><p>3. For <strong><em>data accuracy</em></strong>, Snowflake offers bunch of functions — NULL_COUNT, BLANK_COUNT, BLANK_PERCENT, and NULL_PERCENT. You can use these to validate the primary fields as well as the fields which cannot be null or blank. You can use these functions and validate the data quality.</p><p>4. Snowflake offers set of functions to <strong><em>generate the stats</em></strong> of the data using system defined DMFs — AVG, MIN, MAX, and STDDEV. These can be used to validate the data ranges, data fields where data is expected to be in a range and evaluating if its following the same. These can be used along with the business rules and set the DQ rules to evaluate the data quality. <em>We will design a data quality framework in the upcoming blog.</em></p><p>You can also view the list of DMFs present and set in your Snowflake account by using SHOW command as shown below:</p><pre>SHOW DATA METRIC FUNCTIONS IN ACCOUNT;</pre><p>This is second blog in the series, you will learn to create the custom DMFs and apply it to validate your data in next blog. Please subscribe to my blog series to learn more about latest and greatest features of Snowflak, Gen AI and data domain use cases.</p><blockquote><strong>About Me :</strong></blockquote><blockquote>I am one of the Snowflake Data Superheroes 2023 &amp; 2024. I am a DWBI and Cloud Architect! I am currently working as Senior Data Architect — GCP, Snowflake. I have been working with various Legacy data warehouses, data implementations, and Cloud platforms/Migrations. I am a SnowPro Core certified Data Architect as well as a Google-certified Google Professional Cloud Architect. You can reach out to me on <a href="https://www.linkedin.com/in/poojakelgaonkar/">LinkedIn</a> if you need any further help on certification, Data Solutions, and Implementations!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60a3f6fc01d0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Snowflake’s Data Metric Functions]]></title>
            <link>https://poojakelgaonkar.medium.com/understanding-snowflakes-data-metric-functions-f9b218f378f7?source=rss-86c81ce918a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9b218f378f7</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-superhero]]></category>
            <dc:creator><![CDATA[Pooja Kelgaonkar]]></dc:creator>
            <pubDate>Wed, 18 Sep 2024 17:00:47 GMT</pubDate>
            <atom:updated>2024-09-18T17:00:47.266Z</atom:updated>
            <content:encoded><![CDATA[<p>Thanks for reading my earlier blogs! This blog is the next blog in the series of “Data Quality” and “Data Governance” series. Refer to my earlier blog on “Data Quality Essentials” that talks about necessity of Data Quality and metrics considered to validate data quality frameworks — <a href="https://poojakelgaonkar.medium.com/unfolding-data-quality-essentials-289008a73bec">https://poojakelgaonkar.medium.com/unfolding-data-quality-essentials-289008a73bec</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*UOixbIWUQabcEZIJsCpibA.png" /></figure><p>In growing era of Data and Analytics, implementation of Data Governance is necessary and crucial aspect of data architecture design. Data Governance is one of the design pillar when it comes to the data architecture design framework. Data Governance consist of multiple aspects and framework pillars in a nutshell. Governance is one of the bigger umbrella of implementation that takes care of Data Security, Data Protection, Data Access Management, Data Quality etc. Refer to my earlier blogs to understand the Data Securtiy, Data Protection, and Data Access management implementation with Snowflake —</p><p>Data protection — <a href="https://medium.com/snowflake/snowflake-row-access-policy-c30d552ea3b5">https://medium.com/snowflake/snowflake-row-access-policy-c30d552ea3b5</a></p><p>Dynamic masking — <a href="https://medium.com/snowflake/snowflake-dynamic-data-masking-4ef7b53b414e">https://medium.com/snowflake/snowflake-dynamic-data-masking-4ef7b53b414e</a></p><p>This blog brings Snowflake features that can be used to implement Data Quality and list down specifics of the metric functions that can be used to design the “Data Quality (DQ) Framework”.</p><p>Snowflake’s Data Metric Functions (DMFs) are functions that can be used to implement data quality. These functions cater to the requirements of the quality metrics evaluations and reporting. These can be used to design an automated DQ framework for your data applications. Snowflake provides bunch of DMFs are system functions that can be readily used. You can also setup custom functions using CREATE DDL to define custom metrics functions. Let’s understand how DMFs work!</p><p><strong><em>What are built-in DMFs?</em></strong></p><p>Snowflake offers set of metrics functions for users. These can be used as a system functions as it is without modifying the definition or name of the functions. These are available as part of CORE schema of Snowflake shared database.</p><p><strong><em>What are the system DMFs available within Snowflake?</em></strong></p><p>Snowflake offers DMFs to cater to DQ need to measure the accuracy, consistency, freshness, uniqueness, volume, as well as to generate statistics. These functions can be used for any table as well as view.</p><p><strong><em>How to use DMFs(Data Metric Functions)?</em></strong></p><p>These system functions can be assigned to any table or view. These are currently supported and used for Dynamic tables, Event tables, Event tables, iceberg tables, Materialized views, Standard views, and Standard tables along with temporary as well as transient tables. These can also be used as functions with SELECT query with appropriate access and role. You can also have these functions sheduled to run using <a href="https://docs.snowflake.com/en/sql-reference/parameters.html#label-data-metric-schedule">DATA_METRIC_SCHEDULE</a> proptery of Snowflake object. You can define it while creating the object as well assign it with ALTER DDL command.</p><p><strong><em>How DMFs(Data Metric Functions) work in the background?</em></strong></p><p>Snowflake creates a dedicated event table for data metrics functions in the background. Whenever user assigns a function to a table it gets captured in the dedicated event table.</p><p><strong><em>How to generate alerts on the quality issues?</em></strong></p><p>Once you assign the DMFs to the table or view then you can use the NOTIFY feature of Snowflake to get alerts, notifications with the data issues. This integration of DMF and NOTIFY plays a crucial role to get the data quality assessed and get the issues addressed before the data gets consumed by analytics use cases. Data stewards, architects, and engineers can be part of the notification group to get alerts to rectify the data quality issues.</p><p><strong><em>How to design DQ framework with DMFs?</em></strong></p><p>Snowflake is rich in features and the native/built-in system functions allow users to setup the appropriate checks and alerts in places. You need to identify the data steward who can be an architect or senior engineer or a business analyst who can understand the data and business requiements. Stewards can convert the BR(business requirements) to tech requirements and share with the engineering team for implementation. Stewards can also work with architects to design an automated standard DQ framework for data applications. Snowflake DMF, scheduling, and alerting features make this implementation fairly simple over a customized script and bunch of configurations. Design the framework to address the DQ needs that reads the field requirements and assign DMF to the table/view to verify the quality of the data present in the field/table. Required stakeholders need to adress the data anomaly, quality issues within given SLA.</p><p>We have learnt about Snowflake’s Data metric function (DMF) and got introduced to the technique of DMF. The next blog in the series shares the implementation use case of DMF with a sample use case. Please keep reading and subscribe to the medium blogs to get new blogs/updates.</p><p>Subscribe to my blog series to learn more about latest and greatest features of Snowflak, Gen AI and data domain use cases.</p><blockquote><strong>About Me :</strong></blockquote><blockquote>I am one of the Snowflake Data Superheroes 2023 &amp; 2024. I am a DWBI and Cloud Architect! I am currently working as Senior Data Architect — GCP, Snowflake. I have been working with various Legacy data warehouses, data implementations, and Cloud platforms/Migrations. I am a SnowPro Core certified Data Architect as well as a Google-certified Google Professional Cloud Architect. You can reach out to me on <a href="https://www.linkedin.com/in/poojakelgaonkar/">LinkedIn</a> if you need any further help on certification, Data Solutions, and Implementations!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9b218f378f7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Snowflake Cortex-LLM Functions]]></title>
            <link>https://medium.com/snowflake/using-snowflake-cortex-llm-functions-b3130ee3d40b?source=rss-86c81ce918a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/b3130ee3d40b</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[data-superhero]]></category>
            <dc:creator><![CDATA[Pooja Kelgaonkar]]></dc:creator>
            <pubDate>Sat, 30 Mar 2024 12:55:56 GMT</pubDate>
            <atom:updated>2024-04-01T17:06:15.704Z</atom:updated>
            <content:encoded><![CDATA[<p>Thanks for reading my earlier blogs! This is the next blog in the series about understanding and using Snowflake Cortex functions. The first blog helps you to understand the remaining Snowflake Cortex functions and sample use cases to implement them.</p><p>Read the blog series here</p><ol><li>first blog — <a href="https://medium.com/snowflake/understanding-snowflake-cortex-functions-93d6eb695ed7">https://medium.com/snowflake/understanding-snowflake-cortex-functions-93d6eb695ed7</a></li><li>Using COMPLETE() function — <a href="https://medium.com/snowflake/using-snowflake-cortex-complete-function-to-generate-responses-to-prompt-de9bbde869fa">https://medium.com/snowflake/using-snowflake-cortex-complete-function-to-generate-responses-to-prompt-de9bbde869fa</a></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wvzpon4ggoPPxmTl7aWpNQ.png" /><figcaption>Source: Google and LinkedIn image</figcaption></figure><p><strong><em>What are the functions covered in this blog?</em></strong></p><p>We will cover — SENTIMENT, SUMMARIZE, TRANSLATE, &amp; EXTRACT_ANSWER functions as part of this blog. You can read about COMPLETE in earlier blog.</p><p><strong><em>What are the different features offered with each of the LLM functions?</em></strong></p><p>As you know, the features supported are as per the functions spefied here.</p><ul><li>EXTRACT_ANWER: Gives response to the question and unstructured data shared as input. Uses unstructured data provided and tries to answer the question.</li><li>SENTIMENT: Used to generate the sentiment score, from -1 to 1. This is used to represent positive or negative sentiments of the given text.</li><li>SUMMARIZE: Used to summarize the given text or context or input.</li><li>TRANSLATE : Used to translates the given text to any supported language.</li></ul><p><strong><em>How to calculate the usage of the functions and derive the cost?</em></strong></p><p>Snowflake Cortex LLM functions compute is based on the number of tokens processed. The cost in credits varies per 1 million tokens for each function.</p><ul><li>For functions — COMPLETE, SUMMARIZE, and TRANSLATE — both input and output tokens are counted.</li><li>For functions — EXTRACT_ANSWER and SENTIMENT — only input tokens are counted.</li><li>For EXTRACT_ANSWER, the number of billable tokens is the sum of the number of tokens in the from_text and question fields.</li></ul><p><strong><em>How to use these functions?</em></strong></p><p>You can use these functions without any additional setup, configuration, or using any other integrations to configure the foundation model , access setup etc. There are set of models that are available as part of Snowflake LLM foundation model. You can use the model as per your choice, provide it to the function as input parameter and use standard SQL statements to use these functions.</p><p><strong><em>Reference use cases to use Cortex LLM functions?</em></strong></p><p>We can use the sample MOVIE REVIEW table created in earlier blog. You can also follow below SQL statements to create one.</p><pre>/* Create table*/<br><br>CREATE TABLE MOVIE_REVIEWS(MOVIE_NAME STRING, MOVIE_REVIEW STRING, MOVIE_RATING float);<br><br>/* load sample records */<br><br>INSERT INTO MOVIE_REVIEWS VALUES(&#39;Arthur The King&#39;,&#39;Michael Light (Mark Wahlberg) has one last chance to win a championship and prove himself. Finding sponsorship and creating a team of four is challenging, given his washed-out career and a previous disastrous race. However, he assembles a team of an athlete and a social media star, Leo (Simu Liu), free climber Olivia (Nathalie Emmanuel), and Chik (Ali Sulaiman), who has a bad knee. During the race, he has a fifth and an unlikely teammate — an injured indie dog, Arthur.<br>Director Simon Cellan Jones’s offering is an account of companionship between man and dog. The story talks about loyalty, friendship, sacrifice, and survival. It also offers ample adventure through a jungle trek, rock climbing with bicycles, ziplining across a valley (a sequence that will have you on the edge of your seat), night runs, and more.<br>Cinematographer Jacques Jouffret splendidly captures the Dominican terrains, forests, mountains, rivers, and valleys. The adventurous race also looks convincing and authentic. Writers Michael Brandt and Mikael Lindnord’s narrative has all the ingredients you expect from this genre—friction amid friendship, obstacles, and the tough decisions the athletes must make. Although the story gets a tad too sappy, the narrative’s even pacing prevents it from weighing you down.<br>Mark Wahlberg delivers a powerful performance as the out-of-luck captain who wants to make a comeback no matter what. Nathalie Emmanuel, Simu Liu, and Ali Suliman are also worthy additions. Their banter and individual reasons for participating in the race add authenticity to the narrative. The movie’s real star, however, is the dog. It’s inspiring and incredulous to think this is a true story (of Michael Lindnord), and the narrative will tug at your heartstrings.<br>The movie does not end with the race or whether the Broadrail team wins and goes beyond being an adventure sports drama. Whether you are an animal lover or an adventure sports enthusiast, Arthur the King will thrill you and warm your heart. Be warned: you may tear up at this endearing story of a dog and underdogs!&#39;,3.5);<br><br>INSERT INTO MOVIE_REVIEWS VALUES(&#39;Kung Fu Panda 4&#39;,&#39;After eight years since Kung Fu Panda 3, Po the Panda is back, facing the dilemma of relinquishing his role as the Dragon Warrior to become the Spiritual Leader of the Valley of Peace. Despite resistance from his mentor, Master Shifu (Dustin Hoffman), Po hesitates to let go of his dream role. A new threat emerges in the form of the Chameleon, a master sorceress and crime lord who seeks to unlock the power of previous Kung Fu masters to take over the world. Po teams up with a skilled thief, Zhen, to combat Chameleon and protect the Staff of Wisdom. Throughout the film, Po grapples with his inner conflicts while facing formidable adversaries, navigating themes of duty, sacrifice, and embracing ones true potential. Kung Fu Panda 4 follows a predictable plot, but co-directors Mike Mitchell and Stephanie Ma Stine deliver fun and fast-paced action, which continues to be one of the franchises strong suits. The fight scenes, especially the vibrant chase sequence throughout the setting of Juniper City are performed with style and keep you engaged. The top-notch animation has vivid colours that enhance the visual experience without overwhelming the senses. The editing, too, has been deftly done, thus ensuring that the movie never drags. The only drawback is that the script, co-written by Jonathan Aibel, Glenn Berger, and Darren Lemke, leans heavily on its moral messages, occasionally feeling like its guiding the audience too overtly through its themes of duty and sacrifice. Jack Black reprises his role as Po and delivers a solid vocal performance, mentoring a protégé throughout the film. Dustin Hoffman shines again as Master Shifu, while other characters like Pos adopted father, Ping (Hong), and his biological panda father, Li Shan (Cranston), join the adventure. Viola Daviss portrayal of the menacing Chameleon adds an intimidating presence to the film, while the animation team brings the character to life with impressive visuals. Although Awkwafinas voice doesnt fit the character, Zhens characterization is fresh and unique. Ian McShanes surprising return as Tai Lung provides closure for the character, with his performance adding depth to the story.<br>Despite not reaching the heights of its predecessors, Kung Fu Panda 4 still offers an entertaining 94-minute ride, blending humour, action, and heart in a way that will please fans of the franchise.&#39;,3);</pre><ol><li>Using EXTRACT_ANSWER : You can use the table created to read the review comment and provide answers to your questions. You can ask questions as below :</li></ol><pre>SELECT SNOWFLAKE.CORTEX.EXTRACT_ANSWER(MOVIE_REVIEW,<br>    &#39;What movies do these review mention?&#39;)<br>FROM MOVIE_REVIEWS;</pre><p>Result:</p><pre>SNOWFLAKE.CORTEX.EXTRACT_ANSWER(MOVIE_REVIEW,     &#39;WHAT MOVIES DO THESE REVIEW MENTION?&#39;)<br>[   {     &quot;answer&quot;: &quot;arthur the king&quot;,     &quot;score&quot;: 0.0006233065   } ]<br>[   {     &quot;answer&quot;: &quot;kung fu panda 4&quot;,     &quot;score&quot;: 0.3261313   } ]</pre><p>2. Using SENTIMENT to get the sentiments of the viewers based on these movie reviews</p><pre>SELECT SNOWFLAKE.CORTEX.SENTIMENT(MOVIE_REVIEW), MOVIE_NAME FROM MOVIE_REVIEWS;</pre><p>Result:</p><pre>SNOWFLAKE.CORTEX.SENTIMENT(MOVIE_REVIEW) MOVIE_NAME<br>0.3396543 Arthur The King<br>0.40889353 Kung Fu Panda 4</pre><p>3. Use SUMMARIZE to get review of the movie in short lines.</p><pre>SELECT SNOWFLAKE.CORTEX.SUMMARIZE(MOVIE_REVIEW) FROM MOVIE_REVIEWS;</pre><p>Result:</p><pre>SNOWFLAKE.CORTEX.SUMMARIZE(MOVIE_REVIEW)<br>A movie titled &quot;Arthur the King&quot; follows Michael Wahlberg, an athlete with a past racing disaster, as he assembles a team for a race with challenges due to his career and a bad team. The team includes an athlete, a social media star, a free climber, and a teammate with a bad knee. During the race, they are joined by an injured indie dog named Arthur. The film explores themes of companionship, loyalty, and survival, with thrilling adventure sequences and stunning cinematography. Mark Wahlberg delivers a powerful performance, and the heartwarming story goes beyond the race, tugging at heartstrings and appealing to animal lovers and adventure sports enthusiasts.<br>Kung Fu Panda 4 follows Po the Panda as he faces the dilemma of becoming the Spiritual Leader of the Valley of Peace, while a new threat emerges in the form of the Chameleon. Po teams up with a thief, Zhen, to combat the Chameleon and protect the Staff of Wisdom. The film explores themes of duty, sacrifice, and embracing one&#39;s true potential, with solid vocal performances from Jack Black, Dustin Hoffman, and Viola Davis, among others. The animation is top-notch, and the action scenes are fast-paced and engaging, although the script leans heavily on moral messages. Overall, Kung Fu Panda 4 offers an entertaining 94-minute ride for fans of the franchise.</pre><p>4. Use TRANSLATE to get the reviews in your choice of language.</p><pre>SELECT SNOWFLAKE.CORTEX.TRANSLATE(MOVIE_REVIEW, &#39;en&#39;, &#39;fr&#39;) FROM MOVIE_REVIEWS;</pre><p>Result:</p><pre>SNOWFLAKE.CORTEX.TRANSLATE(MOVIE_REVIEW, &#39;EN&#39;, &#39;FR&#39;)<br>Michael Wollie, qui a une vraie histoire, et qui est un véritable narrateur de la course de l’animal de guerre (Michael Wollie, qui a une vraie histoire, et qui est le premier à se faire connaître par l’histoire, mais que ce que vous voulez, c’est que ce film de l’aventure et de la narration de Mark Wahlberg, que vous êtes un amant de l’histoire, que ce soit en étant un fan de l’incroyable, que ce soit en étant un fan de l’incroy<br>Après avoir adopté son père sur les animations de son film, le film de l&#39;animation de l&#39;année passée, le film de l&#39;animation de l&#39;animation de l&#39;animation de l&#39;animation de l&#39;animation de l&#39;animation de l&#39;animation de l&#39;animation de l&#39;animation de l&#39;animation de l&#39;animation de l&#39;animation de l&#39;animation de l&#39;animation de l&#39;animation de l&#39;animation de l&#39;animation de l&#39;animation de l&#39;animation</pre><p>As you notice, these functions are very easy to use and implement the Generative AI use case for your applications or on top of your application data. Your data processed wont be sent outside the Snowflake data boundaries. You dont need to move your data or bring it outside to implement any the AI use cases.</p><p>Snowflake Cortex functions have enabled many users to use these features with seamless integrations.</p><p>You can read more about this function on snowflake documentation — <a href="https://docs.snowflake.com/en/user-guide/snowflake-cortex/llm-functions">https://docs.snowflake.com/en/sql-reference/functions/complete-snowflake-cortex </a>to implement it for your use cases.</p><p>Subscribe to my blog series to learn more about latest and greatest features of Snowflak, Gen AI and data domain use cases.</p><blockquote><strong>About Me :</strong></blockquote><blockquote>I am one of the Snowflake Data Superheroes 2023 &amp; 2024. I am a DWBI and Cloud Architect! I am currently working as Senior Data Architect — GCP, Snowflake. I have been working with various Legacy data warehouses, data implementations, and Cloud platforms/Migrations. I am a SnowPro Core certified Data Architect as well as a Google-certified Google Professional Cloud Architect. You can reach out to me on <a href="https://www.linkedin.com/in/poojakelgaonkar/">LinkedIn</a> if you need any further help on certification, Data Solutions, and Implementations!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b3130ee3d40b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/using-snowflake-cortex-llm-functions-b3130ee3d40b">Using Snowflake Cortex-LLM Functions</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Snowflake Cortex — COMPLETE function to generate responses to prompt]]></title>
            <link>https://medium.com/snowflake/using-snowflake-cortex-complete-function-to-generate-responses-to-prompt-de9bbde869fa?source=rss-86c81ce918a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/de9bbde869fa</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-superhero]]></category>
            <dc:creator><![CDATA[Pooja Kelgaonkar]]></dc:creator>
            <pubDate>Fri, 22 Mar 2024 16:52:17 GMT</pubDate>
            <atom:updated>2024-03-22T21:52:12.327Z</atom:updated>
            <content:encoded><![CDATA[<h3>Using Snowflake Cortex — COMPLETE function to generate responses to prompt</h3><p>Thanks for reading my earlier blogs! This is the next blog in the series about understanding and using Snowflake Cortex functions. The first blog helps you to understand the Snowflake Cortex and its functions offered as part of the Cortex.</p><p>Read the first blog here — <a href="https://medium.com/snowflake/understanding-snowflake-cortex-functions-93d6eb695ed7">https://medium.com/snowflake/understanding-snowflake-cortex-functions-93d6eb695ed7</a> if you missed reading the blog.</p><p>This is the second blog that helps you learn more about the COMPLETE function and use it to generate responses to input prompts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qpXXYWvapKqYLS1lBumVog.png" /><figcaption>Snowflake SNOWDAY announcement photo: <a href="https://www.snowflake.com/snowday/">https://www.snowflake.com/snowday/</a></figcaption></figure><p><strong><em>What is the function- COMPLETE?</em></strong></p><p>This is the LLM function available in the preview for selected regions. This function returns responses using selected large language models (LLM).</p><p><strong><em>What are the models available to be used with COMPLETE?</em></strong></p><p>There is a set of models available to be used with this function. You can choose one from below:</p><ul><li>&#39;mistral-large&#39;</li><li>&#39;mixtral-8x7b&#39;</li><li>&#39;llama2-70b-chat&#39;</li><li>&#39;mistral-7b&#39;</li><li>&#39;gemma-7b&#39;</li></ul><p>You can use one of the models from the above list. These models are subject to different costs and quotas.</p><p><strong><em>What are the available quotas for given LLM Models?</em></strong></p><p>As shared in the first blog as well, these models are subject to limitations and quotas depending on the model, input tokens, and rows processed per minute.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m9VHJ10edeCtBrcdnYvrsg.png" /><figcaption>Model quotas — Snowflake documentation</figcaption></figure><p><strong><em>How can I grant access to the users to use this function?</em></strong></p><p>The access is to be granted to a database role, and a role can be granted to the user to use this function. Users who want to use LLM functions must use a role that has been granted to them. The custom role created must have granted Snowflake.cortex_user to the database role.</p><p><strong><em>How can I use this function?</em></strong></p><p>You can use this function similar to any other UDF or system-defined functions in Snowflake. You need to pass the input as prompt, and model as the choice of model to be used. You can also pass a bunch of optional parameters to the function. You can use the below syntax :</p><p>SELECT SNOWFLAKE.CORTEX.COMPLETE(‘model_name’, ‘input prompt’);</p><p>Additionally, you can use options to set the temperature, max tokens, and top_p as alternatives to the temperature to pass to the model.</p><p><strong><em>Can I use the COMPLETE function for conversational responses?</em></strong></p><p>Snowflake Cortex — COMPLETE function does not maintain the state of the prompts or prompt history on its own. You need to pass the prompt_history in case you need to have conversational experience using COMPLETE. You can pass all the historical or previous prompts as &lt;prompt_of_history&gt; array.</p><p><strong><em>What are the sample use cases of COMPLETE?</em></strong></p><p>There are various scenarios where you can use this function as independent prompts or conversational or run on top of any table attributes to generate responses.</p><ol><li>You can use this function to run the sample input prompts and provide responses to the prompts as part of the conversation for individual users.</li><li>You can also use this to read table data in the form of one or more table attributes and generate responses.</li></ol><p>You can also use this to generate the responses to summarize, sentiment generation, etc. however Cortex&#39;s set of functions includes these special functions to offer summarization, text generation, and sentiment analysis features to be implemented for specific use cases.</p><p><strong><em>Sample Use Cases:</em></strong></p><ol><li><strong><em>SELECT to generate prompt using LLM model</em></strong></li></ol><pre>SELECT SNOWFLAKE.CORTEX.COMPLETE(&#39;mistral-large&#39;, &#39;What is Snowflake cortex?&#39;);</pre><p><strong>Response:</strong></p><p>Snowflake Cortex is a feature of the Snowflake cloud data platform that enables customers to process data stored in external cloud storage services, such as Amazon S3, Azure Blob Storage, and Google Cloud Storage, without having to first load the data into Snowflake. Cortex uses serverless computing resources to perform data processing tasks, such as querying and transformation, on the data in its external location. This allows customers to take advantage of the scalability and cost-effectiveness of cloud storage services while still being able to use Snowflake’s powerful data processing capabilities.</p><p><strong><em>2. Run SELECT statement to generate response using different model:</em></strong></p><pre>SELECT SNOWFLAKE.CORTEX.COMPLETE(&#39;llama2–70b-chat&#39;, &#39;How can I use Snowflake cortex?&#39;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3OVhGoFT4qp62S0NkzVq6A.png" /><figcaption>Snowsight screenshot</figcaption></figure><p><strong>Reponse:</strong></p><p>Snowflake Cortex is a feature of Snowflake’s cloud-based data warehousing platform that allows you to run containerized workloads, such as data processing, data transformation, and data science tasks, inside Snowflake. This allows you to perform complex data processing tasks without having to leave the Snowflake environment, and without having to worry about managing the underlying infrastructure. Here are the general steps to use Snowflake Cortex: 1. Create a Snowflake account: If you don’t already have a Snowflake account, sign up for one at snowflake.com. 2. Create a Snowflake Cortex cluster: Once you have a Snowflake account, you can create a Snowflake Cortex cluster by going to the Snowflake web interface, clicking on the “Create” button, and selecting “Cortex Cluster” from the drop-down menu. 3. Install Docker images: Snowflake Cortex uses Docker containers to run workloads. You can install Docker images from Docker Hub or other container registries to use with Snowflake Cortex. 4. Create a Cortex job: Once you have a Cortex cluster and Docker images installed, you can create a Cortex job by going to the Snowflake web interface, clicking on the “Create” button, and selecting “Cortex Job” from the drop-down menu. 5. Define the job: When creating a Cortex job, you’ll need to specify the Docker image to use, the command to run inside the container, and any dependencies or libraries that the job needs. 6. Run the job: Once you’ve defined the job, you can run it by clicking the “Run” button in the Snowflake web interface. Snowflake Cortex will automatically provision and run the container, and you’ll be able to see the output of the job in the Snowflake web interface. 7. Monitor the job: While the job is running, you can monitor its progress in the Snowflake web interface. You can view logs, monitor resource usage, and see the status of the job. 8. Retrieve results: Once the job is complete, you can retrieve the results by clicking on the “Results” tab in the Snowflake web interface. Some common use cases for Snowflake Cortex include: * Data processing: You can use Snowflake Cortex to run data processing workloads, such as data transformation, data cleaning, and data aggregation. * Data science: Snowflake Cortex provides a flexible environment for data scientists to run machine learning models, perform statistical analysis, and visualize data. * Data integration: Snowflake Cortex can be used to integrate data from different sources, such as APIs, databases, and files, and load it into Snowflake for analysis. Overall, Snowflake Cortex provides a powerful and flexible way to run containerized workloads inside Snowflake, allowing you to perform complex data processing and analysis tasks without leaving the Snowflake environment.</p><p><strong><em>3. Running on top of table columns :</em></strong></p><p>Consider a use case where you have movie review table with movie_name, movie_review and movie_rating. You want to get summary of the review and sentiment of a viewer quickly.</p><pre>SELECT SNOWFLAKE.CORTEX.COMPLETE(<br>    &#39;mistral-large&#39;,<br>        CONCAT(&#39;Critique this review in 50 words as summary and what is sentiment of viewer:&#39;, MOVIE_REVIEW)<br>) FROM MOVIE_REVIEWS;</pre><p>You can use below table and sample records to run this use case:</p><pre>/* Create table*/<br><br>CREATE TABLE MOVIE_REVIEWS(MOVIE_NAME STRING, MOVIE_REVIEW STRING, MOVIE_RATING float);<br><br>/* load sample records */<br><br>INSERT INTO MOVIE_REVIEWS VALUES(&#39;Arthur The King&#39;,&#39;Michael Light (Mark Wahlberg) has one last chance to win a championship and prove himself. Finding sponsorship and creating a team of four is challenging, given his washed-out career and a previous disastrous race. However, he assembles a team of an athlete and a social media star, Leo (Simu Liu), free climber Olivia (Nathalie Emmanuel), and Chik (Ali Sulaiman), who has a bad knee. During the race, he has a fifth and an unlikely teammate — an injured indie dog, Arthur.<br>Director Simon Cellan Jones’s offering is an account of companionship between man and dog. The story talks about loyalty, friendship, sacrifice, and survival. It also offers ample adventure through a jungle trek, rock climbing with bicycles, ziplining across a valley (a sequence that will have you on the edge of your seat), night runs, and more.<br>Cinematographer Jacques Jouffret splendidly captures the Dominican terrains, forests, mountains, rivers, and valleys. The adventurous race also looks convincing and authentic. Writers Michael Brandt and Mikael Lindnord’s narrative has all the ingredients you expect from this genre—friction amid friendship, obstacles, and the tough decisions the athletes must make. Although the story gets a tad too sappy, the narrative’s even pacing prevents it from weighing you down.<br>Mark Wahlberg delivers a powerful performance as the out-of-luck captain who wants to make a comeback no matter what. Nathalie Emmanuel, Simu Liu, and Ali Suliman are also worthy additions. Their banter and individual reasons for participating in the race add authenticity to the narrative. The movie’s real star, however, is the dog. It’s inspiring and incredulous to think this is a true story (of Michael Lindnord), and the narrative will tug at your heartstrings.<br>The movie does not end with the race or whether the Broadrail team wins and goes beyond being an adventure sports drama. Whether you are an animal lover or an adventure sports enthusiast, Arthur the King will thrill you and warm your heart. Be warned: you may tear up at this endearing story of a dog and underdogs!&#39;,3.5);<br><br>INSERT INTO MOVIE_REVIEWS VALUES(&#39;Kung Fu Panda 4&#39;,&#39;After eight years since Kung Fu Panda 3, Po the Panda is back, facing the dilemma of relinquishing his role as the Dragon Warrior to become the Spiritual Leader of the Valley of Peace. Despite resistance from his mentor, Master Shifu (Dustin Hoffman), Po hesitates to let go of his dream role. A new threat emerges in the form of the Chameleon, a master sorceress and crime lord who seeks to unlock the power of previous Kung Fu masters to take over the world. Po teams up with a skilled thief, Zhen, to combat Chameleon and protect the Staff of Wisdom. Throughout the film, Po grapples with his inner conflicts while facing formidable adversaries, navigating themes of duty, sacrifice, and embracing ones true potential. Kung Fu Panda 4 follows a predictable plot, but co-directors Mike Mitchell and Stephanie Ma Stine deliver fun and fast-paced action, which continues to be one of the franchises strong suits. The fight scenes, especially the vibrant chase sequence throughout the setting of Juniper City are performed with style and keep you engaged. The top-notch animation has vivid colours that enhance the visual experience without overwhelming the senses. The editing, too, has been deftly done, thus ensuring that the movie never drags. The only drawback is that the script, co-written by Jonathan Aibel, Glenn Berger, and Darren Lemke, leans heavily on its moral messages, occasionally feeling like its guiding the audience too overtly through its themes of duty and sacrifice. Jack Black reprises his role as Po and delivers a solid vocal performance, mentoring a protégé throughout the film. Dustin Hoffman shines again as Master Shifu, while other characters like Pos adopted father, Ping (Hong), and his biological panda father, Li Shan (Cranston), join the adventure. Viola Daviss portrayal of the menacing Chameleon adds an intimidating presence to the film, while the animation team brings the character to life with impressive visuals. Although Awkwafinas voice doesnt fit the character, Zhens characterization is fresh and unique. Ian McShanes surprising return as Tai Lung provides closure for the character, with his performance adding depth to the story.<br>Despite not reaching the heights of its predecessors, Kung Fu Panda 4 still offers an entertaining 94-minute ride, blending humour, action, and heart in a way that will please fans of the franchise.&#39;,3);</pre><p>You can use it for any of your use cases where you may want to get details of support issues, tickets, customer sentiments etc. You can use other cortex functions that are specifically designed to be used for sentiments and summarizations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A-6geQHIleCpWJ1J89Ma8Q.png" /><figcaption>SELECT Query response</figcaption></figure><p>You can read more about this function on snowflake documentation — <a href="https://docs.snowflake.com/en/sql-reference/functions/complete-snowflake-cortex">https://docs.snowflake.com/en/sql-reference/functions/complete-snowflake-cortex</a> to learn to implement it for your use cases.</p><p>Subscribe to my blog series to learn more about next set of Cortex functions. The next blog will share SUMMARIZE function with sample use case and SQL to implement it real quick.</p><blockquote><strong>About Me :</strong></blockquote><blockquote>I am one of the Snowflake Data Superheroes 2023 &amp; 2024. I am a DWBI and Cloud Architect! I am currently working as Senior Data Architect — GCP, Snowflake. I have been working with various Legacy data warehouses, data implementations, and Cloud platforms/Migrations. I am a SnowPro Core certified Data Architect as well as a Google-certified Google Professional Cloud Architect. You can reach out to me on <a href="https://www.linkedin.com/in/poojakelgaonkar/">LinkedIn</a> if you need any further help on certification, Data Solutions, and Implementations!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de9bbde869fa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/using-snowflake-cortex-complete-function-to-generate-responses-to-prompt-de9bbde869fa">Using Snowflake Cortex — COMPLETE function to generate responses to prompt</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>