<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ravi Kumar on Medium]]></title>
        <description><![CDATA[Stories by Ravi Kumar on Medium]]></description>
        <link>https://medium.com/@ravikuma2003?source=rss-5dbe5845b1ab------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*pgpnWu5Ziy8jMGb0H0Aiwg.jpeg</url>
            <title>Stories by Ravi Kumar on Medium</title>
            <link>https://medium.com/@ravikuma2003?source=rss-5dbe5845b1ab------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 18:49:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ravikuma2003/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Unlocking Business Potential: How ALTR Complements Snowflake Horizon for Robust Data Security]]></title>
            <link>https://medium.com/@ravikuma2003/unlocking-business-potential-how-altr-complements-snowflake-horizon-for-robust-data-security-4aa7e32d07a1?source=rss-5dbe5845b1ab------2</link>
            <guid isPermaLink="false">https://medium.com/p/4aa7e32d07a1</guid>
            <dc:creator><![CDATA[Ravi Kumar]]></dc:creator>
            <pubDate>Thu, 30 Jan 2025 16:55:10 GMT</pubDate>
            <atom:updated>2025-01-30T17:02:19.303Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s data-driven world, businesses need a comprehensive solution to manage, secure, and govern their data centrally and effectively. Snowflake <a href="https://www.snowflake.com/en/data-cloud/horizon/">Horizon</a> is Snowflake’s robust framework designed to help organizations achieve just that. It provides a unified platform for data discovery, classification, access control, and auditing, enabling businesses to meet regulatory requirements and maintain data integrity across their ecosystem.</p><p><strong>Snowflake Horizon: Fortified Security for Your Data</strong></p><p>Snowflake is built with a state-of-the-art security framework that ensures data is protected at every level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*677LOOU7Cv1ZHSOZ" /></figure><p><strong>Key features include:</strong></p><ul><li><strong>End-to-End Encryption:</strong> Data is encrypted at rest and in transit. Snowflake manages encryption keys by default, but customers can also use their own keys (BYOK).</li><li><strong>Role-Based Access Control (RBAC):</strong> Granular access controls allow administrators to define roles and permissions, ensuring least privilege access.</li><li><strong>Network Security:</strong> Private connectivity options (e.g., PrivateLink, VPN) and IP allowlisting/blocklisting secure data in transit and control access.</li><li><strong>Data Masking:</strong> Dynamic data masking obscures sensitive data based on user roles.</li><li><strong>Audit and Monitoring:</strong> Comprehensive audit logs track user activity, while Snowflake’s Account Usage schema provides detailed insights into data access patterns.</li><li><strong>Multi-Factor Authentication (MFA):</strong> Adds an extra layer of security for user logins.</li><li><strong>Data Governance:</strong> Tools for data classification, tagging, and policy enforcement ensure compliance with regulations.</li><li><strong>Secure Data Sharing:</strong> Enables secure data sharing with external partners without copying or transferring data.</li></ul><p><strong>Unlocking More Power: The Snowflake Partner Ecosystem</strong></p><p>Snowflake’s partner ecosystem plays a pivotal role in enhancing the capabilities of Snowflake Horizon. By integrating with specialized third-party solutions, Snowflake Horizon becomes even more powerful, offering businesses a comprehensive and flexible platform tailored to their unique needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RRnYF7k6NnPsR-qT" /></figure><p><strong>ALTR</strong> is a key partner in this ecosystem. As a trusted provider of advanced data protection and governance solutions, ALTR complements Snowflake’s robust security framework by adding specialized features that address complex data governance, security, and compliance challenges.</p><p><strong>Amplifying the Power of Snowflake Horizon</strong></p><p>While Snowflake Horizon provides a strong foundation for data governance and security, ALTR enhances these capabilities to suit the unique needs of customers:</p><p><strong>Format-Preserving Encryption (FPE):</strong></p><ul><li>Snowflake already offers robust encryption, but ALTR’s FPE takes it a step further by encrypting sensitive data while preserving its original format. This is particularly useful for scenarios where data structure must remain unchanged, such as credit card numbers or Social Security Numbers (SSNs).</li><li>For example, a credit card number like 1234–5678–9012–3456 might be encrypted to 9876–5432–1098–7654, maintaining the same format and usability.</li></ul><p><strong>Tokenization:</strong></p><ul><li>ALTR replaces sensitive data with tokens, which can be securely mapped back to the original data when needed. This adds an extra layer of protection for highly sensitive information.</li></ul><p><strong>Real-Time Monitoring and Alerts:</strong></p><ul><li>While Snowflake provides comprehensive audit logs, ALTR enhances this with near real-time monitoring and alerts for suspicious activity, enabling organizations to respond quickly to potential threats.</li></ul><p><strong>No-Code Policy Management:</strong></p><ul><li>ALTR simplifies policy creation and enforcement with a no-code interface, making it easier for organizations to implement and manage access controls.</li></ul><p><strong>Automated Data Classification:</strong></p><ul><li>ALTR complements Snowflake’s data classification capabilities by automatically identifying and tagging sensitive data, further reducing the risk of exposure.</li></ul><p><strong>Compliance and Auditing:</strong></p><ul><li>ALTR provides detailed audit trails and reporting tools that enhance Snowflake’s native capabilities, helping organizations meet regulatory requirements more effectively.</li></ul><p><strong>Security Without Compromise on Data Format</strong></p><p>One feature that truly sets ALTR apart is its Format-Preserving Encryption (FPE). This cutting-edge technology ensures sensitive data is securely encrypted while retaining its original format, making it compatible with existing systems and applications. For instance, encrypted data like credit card numbers or SSNs can still be used in payment processing or healthcare systems without compromising security.</p><p><strong>Unlocking FPE Power: The Core Features of ALTR’s Implementation</strong></p><ul><li><strong>Data Format Preservation:</strong> Encrypted data retains the same structure, length, and type as the original data.</li><li><strong>Strong Encryption:</strong> Uses robust algorithms to ensure data security.</li><li><strong>Tokenization Support:</strong> Combines FPE with tokenization for added protection.</li><li><strong>Data Masking:</strong> Enables realistic but encrypted data for testing or development.</li><li><strong>Compliance: </strong>Helps organizations meet regulations like GDPR, HIPAA, and PCI-DSS.</li></ul><p><strong>A Look at Diverse Use Cases</strong></p><p>ALTR’s Format-Preserving Encryption (FPE) isn’t just a theoretical concept; it’s a practical solution with real-world applications. By transforming sensitive data while preserving its original format, FPE allows for data use in secure ways. Here are some key use cases illustrating where ALTR’s FPE proves invaluable:</p><ul><li><strong>Data Sharing (Internal or External): </strong>Encrypt sensitive data before sharing it internally or with external partners.</li><li><strong>Payment Processing:</strong> Encrypting emails, credit card numbers or bank account details while preserving their format for processing and storage.</li><li><strong>Healthcare Data Protection:</strong> Securing patient identifiers or medical records while maintaining compatibility with healthcare systems.</li><li><strong>Data Masking for Testing:</strong> Providing realistic but encrypted data for development or testing without exposing sensitive information.</li></ul><p><strong>Cloud Data Security: Snowflake and ALTR</strong></p><p>Snowflake provides a powerful data platform, and ALTR enhances it with advanced security features tailored for a cloud environment. Here’s how ALTR strengthens data protection within your Snowflake ecosystem:</p><ul><li><strong>Automated, No-Code Data Protection: </strong>Sophisticated data protection with near real-time monitoring.</li><li><strong>Encrypted, Private Connectivity:</strong> Ensures secure data transfer.</li><li><strong>Federated Authentication and Authorization:</strong> Enhances access control.</li><li><strong>Automated Data Tagging and Classification:</strong> Reduces the risk of data exposure.</li><li><strong>No-Code Policy Construction: </strong>Simplifies access management.</li><li><strong>Full Audit Trails: </strong>Tracks all user activity in near real time.</li></ul><p><strong>The Power of Partnership: ALTR + Snowflake for Data Security</strong></p><p>ALTR and Snowflake together provide a powerful solution for securing sensitive data in the cloud. By integrating ALTR’s advanced data protection and monitoring capabilities with Snowflake’s scalable and flexible data platform, organizations can enhance their security posture, ensure compliance, and minimize risks associated with storing and processing sensitive data.</p><p>To learn more, explore the ALTR native app on the Snowflake Marketplace. With ALTR’s Format-Preserving Encryption and other advanced features, businesses can unlock the full potential of their data while maintaining the highest standards of security and compliance.</p><p>Discover how the Snowflake Horizon partner ecosystem can further complement your data initiatives by visiting <a href="https://www.snowflake.com/en/why-snowflake/partners/horizon-partners">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4aa7e32d07a1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Quality in the Era of Data-Driven Decision-Making With Snowflake & Alation]]></title>
            <link>https://medium.com/@ravikuma2003/data-quality-in-the-era-of-data-driven-decision-making-with-snowflake-alation-d58ffb3df668?source=rss-5dbe5845b1ab------2</link>
            <guid isPermaLink="false">https://medium.com/p/d58ffb3df668</guid>
            <dc:creator><![CDATA[Ravi Kumar]]></dc:creator>
            <pubDate>Wed, 29 Jan 2025 17:54:22 GMT</pubDate>
            <atom:updated>2025-01-29T17:54:22.036Z</atom:updated>
            <content:encoded><![CDATA[<p>The value we get from data in this “information and data explosion age” really depends on how good that data is. If the data has errors, is out-of-date, or is incomplete, even the most advanced tools won’t be able to make sense of it. Organizations working with unreliable data put themselves at risk: they might make the wrong decisions, have inefficient processes, face costly mistakes, and damage their reputation. This is why data quality (DQ) has become so important. It’s not just about having neat, tidy data, it’s about making sure it’s actually fit for purpose and that we can have confidence in the insights it gives us. Good DQ is key to getting the most out of modern tech and making AI and machine learning work effectively. That way, we can get more accurate predictions, understand our business better, and achieve faster results with those tech investments.</p><blockquote>A study by Experian found that 95% of companies see negative effects from poor data quality, which goes to show its a big deal.</blockquote><p>Okay, so you’ve got mountains of data, and you need it to be not just there, but also reliable and understandable, right? That’s where Snowflake and Alation really complement each other.</p><p>Snowflake, with its <a href="https://www.snowflake.com/en/data-cloud/horizon/">Horizon Catalog</a> capabilities, gives you a built-in data catalog to discover and understand your data assets, and it provides robust governance capabilities, like tagging, policies, Data Quality Monitoring and Data Lineage Visualization. It’s basically your all-in-one Security, Collaboration, Governance solution and more. Think of Horizon as your central place to discover, control, and protect your data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IsNU40q4B-krGmw4" /></figure><p>The Snowflake Horizon ecosystem boasts a wide array of <a href="https://www.snowflake.com/en/why-snowflake/partners/horizon-partners/">partners</a>, providing solutions that enhance and complement its core capabilities to meet the unique needs of customers. And that’s where <strong>Alation</strong> comes into play and adds its magic. It doesn’t just plug into Snowflake; it consumes all that rich information about your data and its quality directly from Horizon — that includes <strong>Data Quality Metrics</strong>. It then builds on top of it, applying its powerful intelligence to give you detailed insights on data quality and showing you precisely where things can be improved. Basically, this duo offers not just a way to store data, but also a way to actively manage it for real business value, all while being easier to manage.</p><p><strong>Snowflake: Integrated Data Quality for Seamless Reliability</strong></p><p>Snowflake’s approach to data quality is built directly into the platform, aiming to provide users with a seamless and integrated experience for managing data reliability. The core idea is to make data quality a proactive and actionable process rather than an afterthought. Snowflake’s <a href="https://docs.snowflake.com/en/user-guide/data-quality-intro"><strong>Data Metric Functions</strong></a><strong> </strong>provide a powerful yet user-friendly way to monitor and manage your data quality. With the ability to use pre-built or create custom metrics, and automation of data quality evaluation, Snowflake is paving the way for consistent, trustworthy data. This not only improves your data governance posture but gives you confidence to make informed decisions from a credible, high-quality source.</p><p>The power of DMFs is that they don’t just run in the background; they give you actionable information. After scheduling DMFs, you can set up alerts to notify you when data quality changes occur. By combining the DMF and alert functionality, you can have consistent threshold notifications for data quality whenever a measurement exceeds or drops below the thresholds you set, enabling you to quickly respond to potential problems.</p><p>This can significantly enhance your data quality efforts:</p><ul><li><strong>Compliance Made Easier</strong>: By continuously monitoring the state of your data, you can demonstrate your adherence to compliance and regulatory standards more effectively, reducing risk and improving governance posture.</li><li><strong>Meet Service-Level Agreements (SLAs):</strong> DMFs provide the accurate data metrics needed to meet service-level agreements between providers, consumers, and clients, especially metrics like data freshness.</li><li><strong>Data Credibility: </strong>DMFs provide data validation, which leads to trustworthy data and better data-driven decisions.</li><li><strong>Consistent Evaluations:</strong> Using system or custom DMFs on an ongoing basis leads to more consistent data quality evaluation, ensuring your data is reliable over time.</li><li><strong>Optimize for Your Specific Needs: </strong>Custom DMFs allow data engineers to create metrics that measure data quality, allowing you to optimize data for targeted applications.</li><li><strong>Automated Measurement: </strong>After setting up a DMF on a table, the measurements are automated, saving you manual effort on measuring your data.</li><li><strong>Performance Optimized: </strong>Snowflake optimizes the evaluation of your DMF queries so you receive optimal query performance.</li></ul><h3>Alation: Ensure Trustworthy Data with Alation’s Data Quality Processor</h3><p>Alation’s Open Data Quality Initiative and Data Health API have allowed seamless integration of custom data quality tools or best-of-breed solutions from our technology partners. This means Alation integrates Data Quality (DQ) results directly from Snowflake and presents them through intuitive health scorecards, upstream alerts, and DQ overlays in lineage diagrams, making insights trustworthy, actionable, and accessible for all.</p><p>With the Data Quality Processor (DQP), Alation takes DQ one step further. This capability <strong>integrates </strong><a href="https://www.alation.com/docs/en/latest/archive/AddDataSources/Snowflake.html"><strong>natively</strong></a><strong> with Snowflake</strong>, emerging DQ capabilities to directly bring their data quality results into Alation’s Data Intelligence Platform.</p><p>Using Alation’s Data Quality Processor, companies can easily import all stock and custom DMFs in a customer’s account. With this information, users can apply scoring to the metrics returned from Snowflake, which helps business users better understand the quality of their data.</p><p><strong>How Does DQP Work ?</strong></p><p>Using a Snowflake DMF, a company can surface how many poorly formatted email addresses a table has. If this DMF returns a metric of 13, is it Good or Bad? Is a score of 13 acceptable for their data analysis, or is it out of compliance? Alation’s Data Quality Processor allows you to create these “Quality Zones” and, more importantly, push these results into Alation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*5Mz0fhS5PTX0VmwH" /></figure><p><strong>Pull DMF Results At Scale</strong></p><p>The above email rule is only a single example. DQP is designed to pull in all your DMF rules and map them to the correct data objects in Alation. As you can see in the image below, custom DMFs, like the <strong>governance.dmf s.ssn_contains_alpha</strong>, are all ingested into Alation and mapped to the correct data objects. Since there is no limit to how many rules are ingested, you can be a creative as you need to be with your DMF design.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dTtn5EKdlgzvTdjl" /></figure><p><strong>Drill For More Information</strong></p><p>Users can also drill deeper into the metadata, learn precisely what DMFs ran against the data, and see how far out of compliance the data is. They can also see all the DMFs attached to the data, giving them even more insight into the data’s quality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ktZkERBcWr0cnQy6" /></figure><p><strong>Lineage</strong></p><p>Another exciting feature of this integration is seeing Snowflake DMF metric results on a lineage path. As you can see below, DQ information can be “overlayed” on top of lineage, allowing users to see precisely where data quality issues exist.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Jwkt-32NSNl3QJpF" /></figure><h3>The Benefits Of DQP</h3><p>It is crucial to offer users easy access to data quality. Company decisions will be based on the analysis of data models or business intelligence reports. If the quality of your data is in question, how can you trust the decisions made from it? Combining Snowflake’s DMFs and Alation’s Data Quality Processor is a cost-effective and reliable way to implement data quality in your organization. Both tools are quick to implement, so your return on investment can be recognized quickly. For more information, please reach out to your Account Executive.</p><h3>In Closing</h3><p>Achieving true data quality excellence requires a holistic approach.</p><blockquote>By leveraging the powerful combination of Snowflake’s built-in data catalog, governance features, and Data Quality Metric capabilities within its Horizon platform, along with Alation’s intelligent Data Quality Processor (DQP) that enables actionable scoring and rich lineage, organizations can move beyond simply storing data to actively managing its integrity.</blockquote><p>This partnership offers a seamless solution for discovering, understanding, and improving data quality, ensuring trustworthy data that fuels confident decision-making across the entire business. To learn more about Snowflake’s Data Metric Functions and data quality capabilities, visit: <a href="https://docs.snowflake.com/en/user-guide/data-quality-intro">https://docs.snowflake.com/en/user-guide/data-quality-intro</a></p><p><em>This article was a collaborative effort and would not have been possible without the co-authorship of </em><a href="https://www.linkedin.com/in/genearnold/"><em>Gene Arnold</em></a><em> and the valuable reviews of </em><a href="https://www.linkedin.com/in/vikasjainpublic/"><em>Vikas Jain</em></a><em> and </em><a href="https://www.linkedin.com/in/anthonylempelius/"><em>Anthony Lempelius</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d58ffb3df668" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Re-Imagine Data Governance with Snowflake Horizon and Powered by AI]]></title>
            <link>https://medium.com/snowflake/re-imagine-data-governance-with-snowflake-horizon-and-powered-by-ai-9ac1ead51b6f?source=rss-5dbe5845b1ab------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ac1ead51b6f</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[snowflake-horizon]]></category>
            <dc:creator><![CDATA[Ravi Kumar]]></dc:creator>
            <pubDate>Mon, 29 Jan 2024 13:34:59 GMT</pubDate>
            <atom:updated>2024-02-07T19:34:45.106Z</atom:updated>
            <content:encoded><![CDATA[<p>In the era of data-driven decision making, establishing a robust Data Governance framework is essential. More often than not; Data Governance typically involves many manual processes across the Data Governance continuum and tends to become tedious, time-consuming, and prone to human error.</p><p><strong>Re-Thinking Data Governance: From Traditional to Transformative</strong></p><p>Snowflake’s expanding ecosystem of <a href="https://www.snowflake.com/en/why-snowflake/partners/horizon-partners/">Data Governance Partners</a> does offer comprehensive solutions for this challenge. Snowflake’s Partners integrate with Horizon and expand the scope and in addition, address wider data governance requirements across various platforms and sources.</p><p>Whereas, the objective of this blog is to examine how Snowflake Horizon’s “Data Governance” capabilities (Horizon is much broader), when combined with the strength of AI/LLM can lead to a transformative approach to Data Governance and help tackle the major obstacles faced by Data Governance teams. We’ll further explore how these capabilities help various personas within Enterprise Data Management including Data Engineers, Data Quality Analysts, Data Governors, Data Stewards, Data Scientists, Data Users, Data Ops professionals, and others, providing a holistic and inclusive approach to data governance and management.</p><p><strong>Data Governance within Enterprise Data Management</strong></p><p>Data Governance is very broad; Gartner defines Data Governance as “The <em>specification of decision rights</em> and an <em>accountability framework</em> to ensure the <em>appropriate behavior</em> in the valuation, creation, consumption and control of data and analytics”. In essence, it’s the 3P; People, Processes and Product (Technology) collaborating together.</p><p>Within Enterprise Data Management, the typical roles that engage with data include Data Engineer, Data Quality Analyst, Data Steward, User, Data Scientist, Compliance &amp; Security and others. The data and process workflow would more or less look like the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3jFdSY5CsgYYqFBDdYOu4A.png" /><figcaption>Enterprise Data Management Workflow</figcaption></figure><p>The Standards and Policies are setup by the DG Committee which, is then put into action by the rest of the personas. Once the data is loaded into the landing area, the data quality analyst analyzes, profiles, and pinpoints data quality issues that require rectification through DQ rules. Following this, a data engineer takes charge of loading the corrected data into Snowflake. Any business centric issues that fail the governance rules are then floated to a data steward who steps in to scrutinize the data for any business quality problems and coordinates with the stakeholders of the data sources to address the issues. Subsequently, the refined data becomes accessible to end-users who utilize it for their specific needs. Further, data scientists engage with the data, employing it to build and refine predictive models.</p><p><strong>About Snowflake Horizon</strong></p><p>Before we go any further, let’s introduce the star of the show — Horizon. <a href="https://www.snowflake.com/en/data-cloud/horizon">Horizon</a> is Snowflake’s built-in governance solution with a unified set of compliance, security, privacy, interoperability and access capabilities in the Data Cloud. Horizon’s governance umbrella covers a wide range of areas, with data governance being just one aspect that we will explore in this blog. Horizon is much broader and encompasses the governance of applications, models/AI and more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/629/0*6mI2EW8AK9RLo3e_" /><figcaption>Snowflake Horizon — Easy, Powerful and Actionable</figcaption></figure><p>With Horizon, Customers can:</p><ul><li>Protect and audit their data with compliance, business continuity, data quality monitoring and lineage</li><li>Secure their environment with continuous risk monitoring and protections, RBAC and granular authorization policies</li><li>Unlock the value of their sensitive data with advanced privacy policies and data clean rooms</li><li>Integrate with other Apache Iceberg-compatible catalogs and engines, in addition to data catalog and data governance partners</li><li>Classify, share, discover and immediately act on data, apps and more across regions and clouds</li></ul><p>For more information on Horizon, see <a href="https://www.snowflake.com/en/data-cloud/horizon">here</a>.</p><p><strong>Using GenAI to lead Data Governance</strong></p><p>In Snowflake, you can fine-tune and deploy your very own LLM models via Snowpark Container Services (SPCS) (PuPr). Further, SPCS wrapped with Snowflake Native App, enables customers to seamlessly access LLMs through the Snowflake Marketplace as well. For the scope of this blog however, we will use the Snowflake External Access (PrPr) feature to integrate with a 3rd party hosted model — OpenAI service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l5cm4zZREXsJ0Y2VTsBJcw.png" /><figcaption>Snowflake and LLM</figcaption></figure><p>An external LLM means that the data will be egressing Snowflake. While the demo here uses mock data; In the real world, you must ensure that robust security measures are in place before transferring sensitive data. You can find additional resources <a href="https://www.snowflake.com/dca-thankyou/generative-ai-llm-school-thank-you/">here</a></p><p>If OpenAI is not your favorite, then Snowflake has made this so easy to switch; If you would like to try with Amazon Bedrock, use instructions <a href="https://quickstarts.snowflake.com/guide/getting_started_with_bedrock_streamlit_and_snowflake/index.html?index=../..index#0">here</a>. Instead, if you want to eliminate the egress of your data, and instead would like to fine-tune using models such as Llama2 7b on Nvidia GPUs all within Snowflake leveraging Snowpark Container Services, read <a href="https://medium.com/snowflake/fine-tuning-llama2-using-snowpark-container-services-33d7194ef49c">here</a>. You will conclude that the only limit is your willingness to try more. Ok. That was cheesy.</p><p><strong>Transformative Data Governance — AI + Horizon</strong></p><p>It is important to note that the quality of responses from a large language model (LLM) depends on the quality and relevance of the training data and the context provided for analysis. Therefore, verification of responses is crucial before relying on them. However, an LLM can also serve as a valuable starting point for investigation and data analysis, helping to identify potential issues and patterns more efficiently than manual methods. By leveraging LLMs for the “Where should I begin” approach, we can provide personas with a solid starting point and accelerate the discovery of insights that might otherwise require extensive manual effort.</p><p>With that context, lets begin our journey and start with the <strong>Data Security</strong> persona. Let’s use Patient data as our example as it belongs to one of the regulated industries (HCLS) and is easy to follow. However, as you can imagine, it can easily be leveraged for other verticals as well.</p><p><strong>Data security</strong> professionals are continually monitoring for questionable data occurrences, a laborious and exhausting process that also depends heavily on the expertise of the security analyst. This is an ideal scenario for AI to contribute by providing guidance and support to alleviate the workload and enhance the process.</p><p>Below is an example where AI performs analysis on Snowflake’s Query_History and Access_History views, condensing the results into a concise summary. This proactive approach empowers the team to initiate investigations with valuable insights already in hand, rather than dedicating extensive time to sorting through raw data.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FXA4Z7s-7tgQ%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DXA4Z7s-7tgQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FXA4Z7s-7tgQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/af62137f1346b900db628550df7318a8/href">https://medium.com/media/af62137f1346b900db628550df7318a8/href</a></iframe><p>As you can see, the security analysts can easily eliminate noise and spend their time dedicated to investigating relevant events. Let’s see how easy it is to build this framework for invoking AI from within Snowflake. Once ready, this can be reused for the rest of the sections as well. For the demo purposes, I used a mock raw Patient data table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OtHO8aqUfzu2zYLtyLofpw.png" /><figcaption>Patient Table</figcaption></figure><p>Assuming that the reader has a good understanding of Snowflake concepts, let’s begin. We will create a stored procedure that takes in 2 inputs — a table name and a prompt/question; and invokes openAI.</p><p>Lets set the environment first. Set your database, schema, warehouse. Create one if you don’t have an existing one.</p><pre>--Set your database, schema and Warehouse.  Create one if it does not exist for your demo.<br>USE DATABASE DG_DB;<br>USE SCHEMA DATA;<br>USE WAREHOUSE LLM_WH;</pre><p>Create the Network Rule and the Secret for the External Access.</p><pre>--CREATE A NETWORK RULE FOR YOUR EXTERNAL ACCESS INTEGRATION<br>CREATE OR REPLACE NETWORK RULE openai_network_rule<br>  MODE = EGRESS<br>  TYPE = HOST_PORT<br>  VALUE_LIST = (&#39;api.openai.com&#39;);<br><br>--CREATE A SECRET WITH YOUR OPEN AI KEY<br>CREATE OR REPLACE SECRET openai_api_key<br>  TYPE = GENERIC_STRING<br>  SECRET_STRING = &#39;xx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&#39;;<br><br>--CREATE THE EXTERNAL ACCESS INTEGRATION<br>CREATE OR REPLACE EXTERNAL ACCESS INTEGRATION openai_access_integration<br>ALLOWED_NETWORK_RULES = (openai_network_rule)<br>ALLOWED_AUTHENTICATION_SECRETS = (openai_api_key)<br>ENABLED = true;</pre><p>Now, Lets now create a stored proc that calls the openAI</p><pre>--Stored Proce that returns the data in a table as a text<br>CREATE OR REPLACE PROCEDURE get_table_data_as_string(table_name STRING)<br>RETURNS STRING<br>LANGUAGE JAVASCRIPT<br>AS<br>$$<br>    // Fetch and format column names from the table<br>    // Iterate over rows and append data<br>    return resultString;<br>$$;<br><br>--The stored procedure to invoke OpenAI<br>CREATE OR REPLACE PROCEDURE SP_INVOKE_OPENAI(table_name STRING, prompt_ques STRING)<br>returns STRING<br>LANGUAGE PYTHON<br>RUNTIME_VERSION = 3.9<br>HANDLER = &#39;SP_INVOKE_OPENAI&#39;<br>EXTERNAL_ACCESS_INTEGRATIONS = (openai_access_integration)<br>PACKAGES = (&#39;snowflake-snowpark-python&#39;,&#39;aiohttp&#39;, &#39;aiosignal&#39;, &#39;async-timeout&#39;, &#39;attrs&#39;, &#39;certifi&#39;,<br>&#39;charset-normalizer&#39;, &#39;colorama&#39;, &#39;frozenlist&#39;, &#39;idna&#39;, &#39;multidict&#39;, &#39;requests&#39;, &#39;tqdm&#39;, &#39;urllib3&#39;, &#39;yarl&#39;)<br>SECRETS = (&#39;openai_api_key&#39;=openai_api_key)<br>AS<br>$$<br>import _snowflake<br>import logging<br>import openai<br>import sys<br>import openai<br>from snowflake.snowpark.functions import listagg<br>    <br>def (session,table_name,prompt_ques)-&gt;str:<br><br>    ret_text = session.sql(f&quot;call [[get_table_data_as_string(&#39;{table_name}&#39;)&quot;).collect()<br>    entire_payload = prompt_ques + &#39;\n&#39; + ret_text[0][0]<br>    <br>    openai.api_key = _snowflake.get_generic_secret_string(&#39;openai_api_key&#39;)<br>    <br>    response = openai.ChatCompletion.create(<br>                    model=&quot;gpt-3.5-turbo-0613&quot;,<br>                    messages=[{&quot;role&quot;: &quot;user&quot;,&quot;content&quot;:entire_payload}]<br>                    ,temperature=1,)<br><br>    return response.choices[0].message.content<br>$$;</pre><p>Great! Now you can execute the stored procedure and engage OpenAI by simply providing the table name and a clear prompt specifying the assistance you require. Let’s try an example to create the glossary on a table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*od-3rE8sksAJ7XIO1NQKvw.png" /></figure><p>As seen, the glossary for the table with the definition of each field is provided. By running the procedure on all key tables, you’ll have a comprehensive Glossary for all your essential assets. This task is typically labor-intensive for <strong>Data Stewards or SMEs</strong>, especially when dealing with hundreds of production tables, where the workload increases exponentially. Further, you can easily visualize the results in Streamlit, creating a personalized UI by calling the same procedure.</p><p>Let’s visualize this data on Streamlit below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0FJMwerNZIMGBM9OQ66qFg.png" /><figcaption>Glossary of Assets</figcaption></figure><p>Speaking of <strong>Data SME</strong> or <strong>Analysts</strong>, let’s get the trend summary of the patient table. This gives a good understanding to the user before they start analyzing the data further.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p8rGGSsu_jWw_JOWjjPoGA.png" /><figcaption>Trend Summary</figcaption></figure><p>In the realm of data analysis, there are roles that focus solely on analysis without requiring SQL expertise. These analysts can still tap into the insights within Snowflake tables, thanks to AI assistance. By posing questions in plain English, they can leverage AI to translate their queries into SQL, empowering them to explore data without needing to write code. Below is a scenario where an analyst wants to extract a list of all patients who reside in the state of Massachusetts. AI responds back with the SQL query.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GOjk0URLos8gKC9VBI1DGQ.png" /><figcaption>Convert English to SQL</figcaption></figure><p>Is this getting interesting ? Let’s dive deeper and explore the possibilities to see just how fascinating it can get!</p><p>Let’s now assume the role of a <strong>Data Quality Analyst</strong>, responsible for ensuring data conformity to the DG committee’s standards. They must detect and address errors or data quality issues. Typically, they begin by profiling the data to identify patterns and ensure quality compliance. Let’s leverage AI to automate the initial step of identifying DQ challenges in a table.</p><p>To execute this in your environment, you would need to call the stored procedure SP_INVOKE_API with the appropriate prompt as an input, and the AI will take care of the rest. See below.</p><pre>call SP_INVOKE_OPENAI(&#39;YOUR TABLE&#39;,&#39;Summarize the DQ challenges in the name field&#39;)</pre><p>On Streamlit, this would render as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OikQpAotxe8rSDzCqvoADA.png" /></figure><p>The output effectively identified most of the data quality issues embedded in the dataset, providing a comprehensive starting point for further investigation, profiling, and assessment by a <strong>DQ Analyst</strong>.</p><p>As a follow up to the above, Let’s ask to generate a custom regex that addresses the data issues. This will produce tailored regular expressions that standardizes the table’s data, proving a valuable tool for the <strong>Data Quality Analyst</strong> to ensure data consistency and accuracy. See below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*MUSHKAVB6LC7o4iXwVMGug.png" /><figcaption>Regex generated by AI</figcaption></figure><p>Now that we’ve explored AI-driven capabilities, let’s shift focus to data-driven approaches. Let’s enhance our Data Governance strategy by incorporating the features of Snowflake Horizon.</p><p>As discussed before, Snowflake Horizon’s capabilities fall under Compliance, Security, Privacy, Interoperability, and Access. Let’s start with <strong>Compliance</strong>.</p><p>In fact, recall that our initial <strong>Security persona</strong> example leveraged data from Snowflake’s Compliance pillar, specifically the Access_History view. This view offers granular insights into data access patterns within Snowflake, serving as a valuable tool for auditing and monitoring. It empowers administrators and authorized users to monitor and analyze data access, ensuring comprehensive oversight.</p><p>Horizon’s Compliance pillar offers <strong>Data Quality Monitoring</strong>(PrPr), which helps evaluate quality of the data in a Snowflake table. <strong>Data Metric Functions (DMFs)</strong> are used to measure quality, serving <strong>Data Engineers, Data Quality Analysts, and Data Stewards</strong>. Snowflake provides pre-built system DMFs in SNOWFLAKE.CORE for common metrics, and users can define custom DMFs for precise measurements. These DMFs are stored in the chosen database and schema, enabling tailored data quality assessments.</p><p>Regardless of whether you utilize system DMFs, custom DMFs, or a combination of both, once assigned to a table or view, Snowflake stores the DMF results in the event table. You can schedule how often the DMF is executed, such as running DMFs on the PATIENT table three times daily. You can adjust the schedule as needed to meet your internal data quality requirements. All DMFs assigned to a table follow the same scheduled cadence.</p><p>Let’s begin by utilizing the pre-built System DMF for evaluating the quality of the Email field. Currently, the following System DMFs are available for use, as noted in this blog post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7buweZPzJI8Ssl7YB5nkgQ.png" /></figure><p>Let’s utilize a few of the available System DMFs to demonstrate their functionality such as NULL count and Unique data count for Emails.</p><pre>--Associate the System DMF to the Patient Table.<br>alter table PATIENT add data metric function snowflake.core.null_count on (email);<br>alter table PATIENT add data metric function snowflake.core.unique_count on (email);<br><br>--Run the DMF to obtain the metrics<br>select snowflake.core.null_count(select email from PATIENT);<br>select snowflake.core.unique_count(select email from PATIENT);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X6v0T06u6S5uTVdIWiHZKA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nth71-Py7ny_rHgKiDCerA.png" /></figure><p>This indicates that all email fields contain a value, and there are 126 unique email addresses present in the Email field.</p><p>Next, let’s create a custom Data Metric Function (DMF) to assess the validity of the Email field. We’ll need to define the DMF, link it to a table, and schedule its execution based on our desired frequency to measure the metric.</p><pre>-- Create a custom DMF that checks for validity of email with a regex<br>CREATE DATA METRIC FUNCTION IF NOT EXISTS<br>invalid_email_count (ARG_T table(ARG_C1 STRING))<br>returns number as<br>&#39;SELECT COUNT_IF(FALSE = (ARG_C1 regexp &#39;&#39;^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,4}$&#39;&#39;)) FROM ARG_T&#39;<br>;<br><br>-- association the DMF to the table<br>alter table PATIENT add data metric function invalid_email_count on (email);<br><br>-- Set the schedule for the DMF on the table<br>alter table PATIENT set data_metric_schedule = &#39;5 minute&#39;; -- for demo purpose, use 5 minute</pre><p>Let’s query to find the number of invalid email records as calculated by the DMF.</p><pre>select /* show 1 */ invalid_email_count (select email from customers);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C0_kZxJxj0G1mVtLcyujvA.png" /></figure><p>So, there are 176 email records that did not pass the regex for email.</p><p>Alternatively, you can retrieve historical metrics from the events table (filter by snow.data_metric.name is not null). In the example below, DATA_METRIC_EVALUATION_HISTORY is a view built on the events table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_rjLL0LIVkdFyQVflUyXzw.png" /></figure><p>Horizon’s Compliance feature also includes the <strong>Data Lineage Visualization UI</strong> (PrPr), a valuable tool for <strong>Data Stewards</strong>. This user interface visualizes the table and column lineage, tracing the flow of data from a source table to a target table, making it easier to understand data relationships and provenance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aFCjcXHN-KrluIiCSRIkQQ.png" /></figure><p>Let’s explore a couple of features from Horizon’s <strong>Security</strong> pillar. If you have been using Snowflake for sometime, then you must have surely come across the features like Row Access Policies, Masking Policies, RBAC, Dynamic Data Masking, Conditional Masking etc. All these (and more) fall within the Security pillar of Horizon. These features are critical for the security of Patient’s PII and sensitive information. If you would like to test drive these features, then use the <a href="https://quickstarts.snowflake.com/guide/getting_started_with_pii/index.html?index=..%2F..index#0">instructions</a> here.</p><p>Another key feature of Horizon’s Security pillar is the <strong>Trust Center</strong>(PrPr) that allows the <strong>Security</strong> persona to evaluate and monitor your account for security risks. When the Trust Center detects violations of the Center for Internet Security (CIS) Snowflake Benchmarks, it creates statistics of these security risks for you to analyze. The Trust Center also reports the individual violations and provides statistics about when and how many login failures occurred on specific dates, allowing you to find patterns that might indicate security attacks. The statistics also provide you with details about authentication types that were used during the login attempts, helping you determine whether weak credentials are being used to authenticate.</p><p>Moving on to Snowflake Horizon’s <strong>Interoperability</strong> pillar, which supports <strong>Data Stewards</strong> and Data Governance initiatives by facilitating integrations with Data Catalogs and a growing network of <a href="https://www.snowflake.com/en/why-snowflake/partners/data-governance-accelerated/">Data Governance partners</a>. These partners enable the consolidation of metadata from various enterprise data stores into a single, unified view, promoting comprehensive governance and oversight.</p><p>Interoperability also allows customers to choose how they integrate Snowflake into their architecture by allowing them to store data in Iceberg, an open-source data table format. This gives <strong>Data Engineers</strong> the flexibility to select the compute engine they prefer to use. Iceberg Tables can now be configured to use either Snowflake or an external service like AWS Glue as the catalog to track metadata. This is made possible by a new account-level object called a Catalog Integration (in public preview), which allows Snowflake to securely read metadata from AWS Glue and object stores.</p><p>Next, Horizon’s <strong>Privacy</strong> pillar enables you to harness the potential of your sensitive data while maintaining advanced privacy standards and utilizing secure data clean rooms.</p><p>Data Clean Rooms (DCRs) are secure environments that enable multiple organizations (or divisions of an organization) to bring data together for joint analysis under the governed and defined guidelines and restrictions that keep the data secure. These guidelines control what data comes into the clean room, how the data within the clean room can be joined with other data in the clean room, the kinds of analytics that can be performed on the clean room data, and what data — if any — can leave the clean room environment. Click here to get your hands dirty with <a href="https://quickstarts.snowflake.com/guide/create_a_native_data_clean_room/index.html?index=..%2F..index#0">DCR</a>.</p><p>As part of Privacy, Horizon enables users to identify and tag PII and other sensitive data with Data Classifiers. Classification is a multi-step process that associates Snowflake-defined tags (i.e. system tags) to columns by analyzing the cells and metadata for personal data; this data can now be tracked by a <strong>Data Engineer </strong>that can protect the column containing personal or sensitive data with a masking policy or the table containing this column with a row access policy.</p><p>System tags are tags that Snowflake creates, maintains, and makes available in the shared <a href="https://docs.snowflake.com/en/sql-reference/snowflake-db">SNOWFLAKE database</a>. There are two Classification system tags, both of which exist in the SNOWFLAKE.CORE schema:</p><ul><li>SNOWFLAKE.CORE.SEMANTIC_CATEGORY</li><li>SNOWFLAKE.CORE.PRIVACY_CATEGORY</li></ul><p>The following table summarizes the relationship between each classification category and system tag, and the string values for each classification system tag.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*0Il84W1RFmfgR1Rv4F6yog.png" /></figure><p>Below is sample code to classify the columns in the table PATIENT.</p><pre>--Classify the Data.  This function analyzes columns in a table and <br>--outputs the possible categories and associated metadata.<br>select extract_semantic_categories(&#39;DG_DB.DATA.PATIENT&#39;);<br><br>call associate_semantic_category_tags(&#39;DG_DB.DATA.PATIENT&#39;,<br>extract_semantic_categories(&#39;DG_DB.DATA.PATIENT&#39;));</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*koE9ZXmrK33hqgaqMvEjvQ.png" /></figure><p>When the table definition is viewed via Snowsight, the applied tags can be viewed</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rUwyU9q_Vf7TKDF5l_-QYQ.png" /></figure><p><strong>Data Engineers</strong> can further extend the data classification capabilities with regular expressions using the CUSTOM_CLASSIFIER class (PrPr) in the SNOWFLAKE.DATA_PRIVACY schema. These include classification categories for:</p><ul><li>Bank account numbers</li><li>Passport identifiers</li><li>Diagnostic codes</li><li>Internal tracking identifiers that are relevant to your business</li></ul><p>Sample code to use custom classifier</p><pre>--Create a Custom classifier class<br>create or replace snowflake.data_privacy.custom_classifier demo_instance();<br><br>--Add regex for custom classificition<br>--Expression for Master Card <br>Call demo_instance!add_regex(&#39;MC_PAYMENT_CARD&#39;,&#39;identifier&#39;,<br>&#39;^(?:5[1-5][0-9]{2}|222[1-9]|22[3-9][0-9]|2[3-6][0-9]{2}|27[01][0-9]|2720)[0-9]{12}$&#39;);<br><br>--Expression for Visa<br>Call demo_instance!add_regex(&#39;VS_PAYMENT_CARD&#39;,&#39;identifier&#39;,<br>&#39;\b([4]\d{3}[\s]\d{4}[\s]\d{4}[\s]\d{4}|[4]\d{3}[-]\d{4}[-]\d{4}[-]\d{4}|[4]\d{3}[.]\d{4}[.]\d{4}[.]\d{4}|[4]\d{3}\d{4}\d{4}\d{4})\\&#39;);<br><br>--Expression for Amex card<br>Call demo_instance!add_regex(&#39;AMX_PAYMENT_CARD&#39;,&#39;identifier&#39;,&#39;^3[47][0-9]{13}$&#39;);<br><br>--Apply the custom classifier<br>CALL SYSTEM$CLASSIFY(<br>  &#39;crm_data.public.hr_data&#39;,<br>  {&#39;auto_tag&#39;: true, &#39;custom_classifiers&#39;: [&#39;crm_data.public.demo_instance&#39;]}<br>  );</pre><p>And the last pillar of Horizon is <strong>Access</strong>. As part of Access, the features include the ability to Share, Discover and Immediately act on data, Apps and more.</p><p>With Data Share &amp; Collaboration, you can efficiently collaborate across teams and business units without ETL or integrations by sharing direct access to live data and with Snowflake Native Apps. You can also Discover the rich set of content within your organization, as well as data and apps available on Snowflake Marketplace, with a single Universal Search (PrPr). Further, You can use natural language powered by LLMs to easily interact with and query data using Snowflake Copilot(PrPr)</p><p><strong>Pushing The Envelope</strong></p><p>We’ve explored how Horizon’s capabilities can enhance a Data Governance program and accelerate time to value with AI-driven insights for various Enterprise Data Management personas. Now, let’s make it even more exciting — leveraging AI to generate custom code tailored to your specific data, making data processing and transformation more efficient and effortless.</p><p>When prompted, AI generated a Python stored procedure tailored to the snowflake table’s data, specifically designed to cleanse the “Name” field. Additionally, it provided step-by-step instructions for execution, making it a game-changer for Data Quality Analysts who can now efficiently tackle data cleansing tasks with ease.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tYZLHdmu3jtVGs2r55Xi8g.png" /><figcaption>Name Cleansing Python Code</figcaption></figure><p>For Master Data Management, it is essential to de-duplicate the master data to ensure there is no ambiguity. Below is a Python stored procedure code snippet that AI generated that is tailored to the patient table data, which de-duplicates patient data using fuzzy matching. Further, it provided clear instructions for implementation, making it an invaluable tool for <strong>Data Quality Analysts / Data Engineers </strong>to streamline their data cleansing processes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ybdb8U9WYazR7dlS-OUd3g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DFn9fC4ph4VEQ7YFZNBp_w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aiTcqo8OaUmJQqlnNWXMLQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ur5V7z2tnhiv2DyA-UwW4w.png" /></figure><p><strong>In Closing</strong></p><p>In closing, our journey through the transformative capabilities for Data Governance using Snowflake Horizon, powered by the innovative integration of AI, is very promising. By reimagining traditional processes through the lens of AI-driven and data-driven methodologies, we’ve seen how the once tedious and error-prone tasks of data governance can evolve into a more streamlined, efficient, and accurate practice.</p><p>Horizon’s comprehensive suite, from compliance and security to privacy and interoperability, combined with AI’s analytical prowess, empowers data professionals across the spectrum. Whether you’re a Data Engineer, Quality Analyst, or Data Steward, the fusion of Snowflake Horizon and AI not only simplifies your workflow but also elevates the strategic value of data governance within your organization. As we step into the future, the potential for further innovations in this space is boundless, promising a more secure, compliant, and insightful data-driven decision-making environment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ac1ead51b6f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/re-imagine-data-governance-with-snowflake-horizon-and-powered-by-ai-9ac1ead51b6f">Re-Imagine Data Governance with Snowflake Horizon and Powered by AI</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking the Potential of Snowflake Alerts & PagerDuty Operations Cloud: Enhancing Data Operations]]></title>
            <link>https://medium.com/@ravikuma2003/unlocking-the-potential-of-snowflake-alerts-pagerduty-operations-cloud-enhancing-data-operations-66511a25b8af?source=rss-5dbe5845b1ab------2</link>
            <guid isPermaLink="false">https://medium.com/p/66511a25b8af</guid>
            <dc:creator><![CDATA[Ravi Kumar]]></dc:creator>
            <pubDate>Fri, 09 Jun 2023 19:43:15 GMT</pubDate>
            <atom:updated>2023-06-13T19:18:24.254Z</atom:updated>
            <content:encoded><![CDATA[<p>Snowflake, known as the Data Cloud company, has introduced a feature called Snowflake Alerts &amp; Notifications. This feature empowers users to send notifications or trigger specific actions when predefined conditions are met in their Snowflake data.</p><p>In this article, Manuraj Rajasekharan and Jaswinder Paul from PagerDuty explore the powerful combination of Snowflake Alerts &amp; Notifications and PagerDuty Operations Cloud is designed to effectively manage, triage, and communicate operational issues. These issues may range from data quality errors in procedures to AI/ML/Data Science errors or urgent administration notifications. By implementing this integration, you can elevate your operations and glean insights from real-world examples of how PagerDuty’s Data team has harnessed this solution to enhance their workflows.</p><p>PagerDuty Operations Cloud manages urgent and unplanned work across the enterprise. The <a href="https://www.pagerduty.com/">PagerDuty Operations Cloud</a> is an essential infrastructure for critical work. It automatically detects and diagnoses disruptive events, mobilizes the right team members to respond, and automates infrastructure and workflows across your digital operations.</p><p>By leveraging a comprehensive incident management tool like PagerDuty, users can employ best practices from DevOps, and utilize the same tool already being used for DataOps and control center operations. The integration brings productivity gains through features such as grouping similar incidents, making managing incidents easier and minimizing downtime.</p><p>Overall, by utilizing Snowflake Alerting and PagerDuty Operations Cloud together, organizations can improve their operational efficiency, reduce downtime, and ensure that they effectively communicate with their team and leadership on important metrics that matter in real-time.</p><p><strong>How do Snowflake and PagerDuty Integrate?</strong></p><p>Snowflake notifications and alerts set up are explained here in this <a href="https://docs.snowflake.com/en/user-guide/notifications-alerts">article</a>. Snowflake sends email notifications through a new built-in stored procedure: SYSTEM$SEND_EMAIL. This stored procedure can be <a href="https://docs.snowflake.com/en/user-guide/email-stored-procedures">called by user-created stored procedures</a>, or as the <a href="https://docs.snowflake.com/en/user-guide/alerts">result of an alert action</a>.</p><p>Step 1 : Set up PagerDuty service with email Integration in PagerDuty and escalation policy as you normally use PagerDuty.</p><p>Step 2 : Create an <a href="https://docs.snowflake.com/en/user-guide/email-stored-procedures">Email Notification</a> object within Snowflake to send email from Snowflake to PagerDuty by using the PagerDuty email generated in Step 1 while creating the PagerDuty service.</p><p><em>create or replace notification integration alert_email</em></p><p><em>type=email</em></p><p><em>enabled=true</em></p><p><em>allowed_recipients=(‘</em><a href="/service/mailto:xxxxxx@abc.com"><em>xxxxxx@abc.com</em></a><em>’); — this is PagerDuty email generated via the email integration while creating the service in PagerDuty.</em></p><p>You are all set !</p><p><strong>Use Cases</strong></p><p>PagerDuty’s DataDuty team manages data infrastructure, performs data engineering, and delivers business insights and data science capabilities to improve business operations and meet customer demands for operational data. We also use PagerDuty for various other use cases, such as customer success, security, and business applications to quickly respond to issues and incidents. The team leveraging PagerDuty ensures that customers receive top-notch service and support.</p><p>Here are three use cases where we used Snowflake alerts &amp; PagerDuty together to great success</p><ol><li>Data Apps/Data Science/AI/ML</li><li>Metric/Data Quality Alerts</li><li>Snowflake Warehouse Usage Alert</li></ol><p>Now, lets dig deeper.</p><ol><li><strong>Data Apps/AI/ML Operations</strong></li></ol><p>Data applications and AI/ML models/Operations require identification of data drift, and actions must be taken to ensure that models operate within a certain threshold. Model features that are relevant may change, resulting in basic assumptions about the model changing as well. Please refer to this <a href="https://www.pagerduty.com/blog/minimizing-data-science-model-drift-by-leveraging-pagerduty/">article</a> for more information on how the DataDuty team at PagerDuty manages data drift notifications.</p><p>When AI/ML models become critical infrastructure for an organization, where accuracy and uptime are crucial, operations teams should consider adding tests that monitor model drift and alert the appropriate stakeholders at the first sign of trouble. Suppose you are using Snowpark to build AI/ML models. In that case, whether in Python, Java, or Scala, you can embed snowflake alerts and notifications and use the same PagerDuty Operations Cloud to manage your data operations and increase productivity.</p><p><strong>2. Metric/Data Quality Alerts</strong></p><p>Ensuring data quality is paramount to all stakeholders. Identifying data quality issues after the data has been ingested into critical systems is too late for action. With data now moving through real-time systems, data science models, recommendation engines, etc., in a matter of seconds or minutes, it is essential to have alerts and notifications for data quality issues in place. PagerDuty managing these alerts and surfacing them as incidents is critical for effective data management.</p><p>DataDuty monitors PagerDuty’s financial systems to track daily revenue numbers, as ARR is an important metric for a SaaS company. This monitoring extends from Snowflake to a plethora of SaaS Business Applications like Salesforce, NetSuite, Coupa, etc. Any significant swings in ARR must be monitored closely. In case of issues at the order management systems or any source system, the operations team is alerted in near real-time, allowing them to manage the data incident and take proactive measures to ensure that all stakeholders have access to the right data at the right time.</p><p>Here is an example of a script to set up a data quality alert to notify the operational team of a spike in arr(revenue) on a daily basis. The same can be used for any aggregation procedure.</p><p><em>CREATE or replace ALERT dw_tmp.TOTAL_ARR_DIFF_ALERT</em></p><p><em>&lt;Body of Procedure&gt;</em></p><p><em>from dw.agg_table</em></p><p><em>where &lt;Clause&gt;</em></p><p><em>group by 1</em></p><p><em>order by 1</em></p><p><em>))</em></p><p><em>where arr_pcnt&gt;0.005 or arr_pcnt&lt;-0.005 — Threshold defined here</em></p><p><em>))</em></p><p><em>THEN</em></p><p><em>call system$send_email(</em></p><p><em>‘ALERT_EMAIL’,</em></p><p><em>‘xxxxxx@abc.com’,</em></p><p><em>‘Email Alert: Daily ARR difference is more than 0.5%’,</em></p><p><em>‘ARR End Increased/Decreased by more than 0.5%’</em></p><p><em>);</em></p><p><strong>3. Snowflake Warehouse Usage Alert</strong></p><p>Effective management of credit usage in real-time( or near real-time) is a crucial concern for Snowflake administrators and practitioners. The improper execution of ad hoc queries, long-running jobs, incorrect joins by business users, and time-consuming data science extractions can all contribute to unplanned credit usage. The recent surge in the use of AI tools for reinforcement learning, which needs large amounts of data extraction, has further compounded the problem.</p><p>In all these cases, delay in taking corrective action can prove costly. Every second counts and the team must be alerted immediately. Therefore, it is essential to have effective measures in place to monitor credit usage and identify potential issues as soon as they arise.</p><p>Snowflake <a href="https://docs.snowflake.com/en/user-guide/resource-monitors">resource monitors</a> are one method to be notified of these issues. Suppose your operations team runs on PagerDuty already. In that case, you can use it to meet your standard operating procedures by integrating these alerts on credit usage and utilizing PagerDuty features around escalation &amp; notifications. In PagerDuty, we put the monitoring script to monitor all warehouses used for self-service or in warehouses where there are concerns about a spike in credit usage by comparing utilization against previous usage. A small XS warehouse was used for running these against snowflake metadata. By implementing this, teams cantake real-time action against rogue queries, extracts, plan to optimize queries, improve performance, and reduce credit consumption, ultimately saving time and resources.</p><p>Here is an example of a script to set up alerts in snowflake to track warehouse usage and notify via the procedure. The same can be used in multiple scenarios.</p><p><em>CREATE or replace ALERT dw_tmp.PROD_AIRFLOW_WH_ALERT</em></p><p><em>WAREHOUSE = WHNAME</em></p><p><em>SCHEDULE = ‘1 minute’</em></p><p><em>IF( EXISTS(</em></p><p><em>select (total_credits_used/prev_credits)*100 as credit_pcnt from (</em></p><p><em>select START_MONTH,total_credits_used,</em></p><p><em>lag(total_credits_used) over (order by START_MONTH) as prev_credits from (</em></p><p><em>SELECT DATE_TRUNC(‘month’, START_TIME) AS START_MONTH,</em></p><p><em>SUM(credits_used) as total_credits_used</em></p><p><em>FROM snowflake.account_usage.warehouse_metering_history</em></p><p><em>WHERE warehouse_name=’WHNAME’</em></p><p><em>AND start_time &gt;= date_trunc(month,add_months(current_date,-1))</em></p><p><em>GROUP BY 1)</em></p><p><em>group by 1,2)</em></p><p><em>where prev_credits is not null</em></p><p><em>and credit_pcnt&gt;90</em></p><p><em>))</em></p><p><em>THEN</em></p><p><em>call system$send_email(</em></p><p><em>‘ALERT_EMAIL’,</em></p><p><em>‘snowflake-alert-email-email@pdt-circular.PagerDuty.com’,</em></p><p><em>‘Email Alert: Warehouse ‘WHNAME’ Monthly approach limits’,</em></p><p><em>‘Warehouse ‘WHNAME’ is reaching your limit soon 90% credit used ‘</em></p><p><em>);</em></p><p>This script is aimed at the administrators and involves taking action when credit usage exceeds a certain threshold. In this scenario, the administrators work alongside warehouse owners and users to address the high credit usage issue. The course of action may include killing a query, requesting that users optimize their queries, and communicating with the warehouse owner regarding potential overages.</p><p>The above example can be extended to other use cases around taking quick action to terminate a query that exceeds the credit limit.(A snowflake query running against the snowflake.account_usage.query_history with filter condition on elapsed time continuously tracks this). This alert enables the operations team and warehouse owner to run a playbook, process, or automation as soon as the notification is received while also notifying the user running the query.</p><p>Another area where we found it useful is managing BI tools, specifically when there is a surge of users, particularly on Mondays, along with ad hoc queries run by these tools. Depending on the configuration, this surge can impact Snowflake credit usage and the queuing at BI tools. Rather than purchasing additional processing power, implementing this mechanism can help track, monitor, and take quick action to meet the front-end tool’s SLA.</p><p><strong>Conclusion</strong> : <em>By leveraging Snowflake and PagerDuty, your team can quickly identify and address issues related to data infrastructure, data quality, AI Operations, and manage software costs. This solution ensures your team has more time to focus on business teams to help you gain the trust of your stakeholders, increase the productivity of your existing resources, manage credit usage, and provide trusted data to your stakeholders.</em></p><p>Want to get started on Snowflake? Visit <a href="http://signup.snowflake.com/">signup.snowflake.com</a> to start your 30-day free Snowflake trial that includes $400 worth of free usage. Want to learn more about PagerDuty, visit <a href="http://www.pagerduty.com/">www.pagerduty.com</a>. If you want to try a 14 day trial, click <a href="https://www.pagerduty.com/nlp/trial-sign-up/?utm_medium=blog&amp;utm_source=snowflake&amp;utm_campaign=Global-Tr-FreeTrial-Partner&amp;utm_content=SnowFlake%20Blog">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66511a25b8af" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Modernizing Enterprise SOC’s: Anvilogic’s Automated Detection Engineering On Snowflake Security…]]></title>
            <link>https://medium.com/@ravikuma2003/modernizing-enterprise-socs-anvilogic-s-automated-detection-engineering-on-snowflake-security-d26461aa1747?source=rss-5dbe5845b1ab------2</link>
            <guid isPermaLink="false">https://medium.com/p/d26461aa1747</guid>
            <dc:creator><![CDATA[Ravi Kumar]]></dc:creator>
            <pubDate>Tue, 06 Jun 2023 15:18:46 GMT</pubDate>
            <atom:updated>2023-06-06T15:18:46.695Z</atom:updated>
            <content:encoded><![CDATA[<h3>Modernizing Enterprise SOC’s: Anvilogic’s Automated Detection Engineering On Snowflake Security Data Lake Using Generative AI</h3><h3>Snowflake: The Modern Data Lake For Security Analytics</h3><p>The Snowflake Data Cloud offers an excellent security data lake with low cost storage and on-demand compute capabilities. It offers a significantly lower cost platform for security data lakes compared to existing SIEM’s based on how the data cloud architecture decouples storage and compute. Traditional SIEM’s have bundled the data platform with their proprietary detection and response platform slowing down innovation on both dimensions. By unlocking the data platform from the threat detection and response platform, the Snowflake data cloud allows Enterprise SOC’s to take advantage of modern innovations in storage architectures, and, simultaneously, avail of a choice of specialized detection and response vendors.</p><h3>Challenges In Enterprise SOC’s</h3><p>Enterprise SOC’s struggle with challenges in detection engineering and managing false positives, and perform investigations and triage on the rapidly rising alert counts.</p><ul><li><strong>The complexity of threat detection and response:</strong> Detection analysts today require knowledge of attacked tradecraft(TTP’s), deep understanding of the event and alert data sets collected in their data lake, and be able to code up rule logic in the SIEM-specific language. Automation and the power of generative AI offers great opportunities in empowering detection analysts. Similar challenges and opportunities apply during the investigation and response phase.</li><li><strong>Limited subject matter expertise and skills shortage:</strong> Building and operating a detection engineering takes training and subject matter expertise, which can fall on a handful of people to research, develop, test, tune and maintain. There are simply not enough analysts who can successfully operate a high efficacy detection and response program.</li><li><strong>High cost of legacy SIEM’s:</strong> Legacy SIEM’s have been consuming an increasing share of SOC budgets, and enterprises are responding by introducing modern data lakes such as Snowflake in their SOC’s in an incremental manner which does not require rip and replace of existing legacy SIEM’s. Enterprises are thoughtfully augmenting their existing legacy SIEM’s with Snowflake and using a single detection and response platform to manage detections across their existing SIEM and their Snowflake security data lake.</li></ul><h3>Unbundling The SIEM: The Anvilogic Detection And Hunt Platform</h3><p>The Anvilogic platform offers a highly automated solution for detection engineering, hunt and alert investigations that help accelerate threat detection and hunting in the SOC. The platform integrates with Snowflake’s security data lake and connected application model, which allows SOC’s to adopt Snowflake as a security data lake alongside their existing data lakes. By unbundling the SIEM, Enterprises can benefit from Snowflake’s rapid innovation in data lakes, coupled with Anvilogic’s innovations in threat detection and response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f6ll_svA87snQRg8C9_Mmg.png" /></figure><h3>Applying Automation To Threat Detection And Hunt</h3><p>Anvilogic, Powered by Snowflake, simplifies and automates the detection engineering process through the following steps:</p><ol><li>Know Your TTP’s. It keeps track for your critical TTP’s and platforms that reflect your highest detection priorities based on your threat landscape and attack surfaces.</li><li>Know Your Data. It automatically analyzes the events and alerts that you have collected across all your data lakes, and identifies TTP’s that can be detected using those.</li><li>Automated Identification And Deployment of Detection Rules. Based on your TTP detection priorities, and feed assessment it recommends detections from the Anvilogic detection repository of over a thousand detection rules. There are specific technical challenges with making the external rule logic work in a unique Enterprise data lake environment which are addressed by innovations in the Anvilogic detection rule architecture.</li><li>A ingest pipeline for parsing and normalizing events and alerts into the data model that is based on CIM.</li></ol><h3>Threat Research And AI Underpinning Detection Automation</h3><p>These capabilities are based on the following technical underpinnings in the Anvilogic platform:</p><ol><li>Threat Research. The Anvilogic threat research team are constantly tracking attacker TTP’s, vulnerabilities across platforms and adversaries, and releasing new detections in response.</li><li>AI-Powered Recommendation and Scoring Systems. The Anvilogic automation platform is built around AI-powered recommendations based on automated assessments of events and alerts, and detection rules and generates rule tuning recommendations and prioritizes alerts.</li><li>Low Code/No Code with Generative AI. With Anvilogic’s new feature Unified Detect that includes a generative AI capability for detection code generation, analysts can quickly build detections using drag-and-drop features, eliminating the need to rely on proprietary languages.</li></ol><h3>Automating Detection Engineering on Snowflake: Unified Detect With Generative AI</h3><p>Unified Detect in the Anvilogic Threat Detection and Hunting platform includes the following</p><ol><li>Raw events and alerts ingest and normalize pipeline into Snowflake. Where enterprises have developed alternative means for ingesting events into Snowflake, Unified Detect can work with those SQL capable with requiring rule logic changes.</li><li>Native Threat Detection against events and alerts in the Snowflake data lake. Anvilogic threat detection rules built in SQL are executed in a Snowpark execution framework that is deployed on the enterprise;</li><li>Using Generative AI to develop new threat detections on Snowflake. As some analysts are not familiar with SQL for threat detection, Anvilogic has introduced a generative AI capability where analysts can query this service with English language detection questions and get built out SQL queries that are customized for the event schema’s in the Snowflake data lake for the Enterprise.</li><li>Detection Rule tuning based on ML-powered assessment of detection rule performance</li><li>Alert Prioritization</li></ol><p>By leveraging AI and low/no-code logic, it reduces the time and effort involved in creating and maintaining detections for threat detection. With its platform integration and drag-and-drop user experience, it is a powerful solution for improving detection coverage and security posture.</p><h3>Force Multiplier For the Enterprise SOC</h3><p>Anvilogic’s Unified Detect allows Enterprise SOC’s to confidently take advantage of a modern cost-performant and scalable security data lake powered by Snowflake by offering a best-in-class detection and hunt platform optimized for Snowflake.</p><h3>Applying Generative AI In Detection Engineering</h3><p>With the integration of Generative AI, the initial iteration of Unified Detect lowers the barriers for SOC analysts to use SQL as their query language. Anvilogic has developed an assistant to help analysts generate SQL queries through a plain English language interface.</p><h3>Using Low Code/No Code Builders for Rules and Enrichments</h3><p>The second key differentiator is the low/no-code detection builder. While Anvilogic has over a thousand detection rules in their detection repository, enterprises may need to develop custom detections. Unified Detect offers a drag-and-drop interface, enabling users to develop detections quickly and efficiently. This feature democratizes the threat detection content process helping novice analysts act as power users.</p><h3>Enabling a Hybrid Data Lake Architecture For Enterprise SOC’s</h3><p>Lastly, Unified Detect is designed to be data platform and search language agnostic, allowing enterprises to use the right data lake for the right problem. By allowing enterprise SOC’s to leave their data in the most appropriate data lake, the Anvilogic detection and hunt platform offers a consistent interface for detection and hunts analysts across these hybrid data lakes. In early deployments of Snowflake in enterprise SOC’s, Anvilogic have seen some enterprises standardize entirely on Snowflake as their security data lake, while others are using it in conjunction with their existing SIEM’s. The Anvilogic platform serves as the primary detect and hunt platform for the analysts across these data lakes.</p><h3>Detection Engineering In Action: Searching Crowdstrike FDR data in Snowflake Security Data Lake</h3><p>Lets see how with a low/no-code builder and Generative AI, Unified Detect simplifies and automates the detection engineering process across security data lakes. For this demo, we will create a simple detection rule to search Crowdstrike FDR data in Snowflake Security Data Lake.</p><p>To start building detections, start with the Search tab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*nrMhbAf3mQRJ0a5_yjwTQA.png" /></figure><h3>Choose Your Data Lake</h3><p>Select Snowflake as your data repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Cfuh3fx-CsptSSm4A7YqQ.png" /></figure><p>This takes you to the canvas where you’ll begin to build your detection by simply dragging and dropping the building blocks (“components”). To start, we’ll drag and drop the “Gather Data” component to the workspace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*YAHVd6mHRAoMIrY3AxKcZA.png" /></figure><h3>Select Normalized Data Sources From Snowflake Data Lake</h3><p>Select the data you want to search. In this case, we’ll be searching Crowdstrike FDR data in our Snowflake repo. These Gather Data macros can be versioned and managed like all other detection engineering content on the Anvilogic Platform. Normalization is implemented during ingestion when the enterprise chooses to use the Anvilogic pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MAMN9tnrlQo0YkuU6NngCA.png" /></figure><h3>Developing a SQL Query For Suspicious Events</h3><p>Now, let’s write a code block</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qkjjiNQuPHJjrKuVF0VxeQ.png" /></figure><p>In case you’re not a SQL expert, you can leverage our OpenAI integration to help you out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ITOlUsGAkqorG0Akb3rG0w.png" /></figure><p>Simply ask the chatbot a detection rule question, and it returns the sql query in the response. The prompt incorporates the required enterprise context such as the normalized schema to make the query work against the SQL tables hosting the events.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*um-xjQxOTJJ3uF7xV4tJgw.png" /></figure><p>For our demo, let’s write a sql to retrieve all events where the process field contains the text powershell or has an ip.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l7lPJ6nvJdeoQSpuBvVs2w.png" /></figure><p>And the chatbot will return the SQL code results to you so you can copy and paste them into your Code Block.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p7_CA-X1oHy-TgD6_76lTA.png" /></figure><p>Now, you can add the sql code to the code block</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zJOCjiffYUFjFStmZp-YQw.png" /></figure><h3>Analyze Events To Improve The Detection</h3><p>To preview the output of your detection on your security data, select the “Find Events” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2-OR-PXzenitM_VuwmxnmA.png" /></figure><p>And we see the returned events from our search.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WZ2Bl1RY4tVB-pgbVTRv2w.png" /></figure><h3>Save In The Detection Repository</h3><p>When you’re ready, you can save this detection as a Threat Identifier (or an alert), so you can deploy it as a use case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L35syOoiTo0G1EH5HqIabQ.png" /></figure><h3>Deploy As A Scheduled Detection</h3><p>Now you’re ready to deploy and start detecting on this use case!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zM6fdQnKrNg7BaFs1LVLKQ.png" /></figure><h3>The Modern SOC Is Here And Now</h3><p>Anvilogic’s Unified Detect with AI-powered detection engineering enables cybersecurity analysts to accelerate the development, deployment and tuning of threat detection rules, and alert prioritization and investigation. Enterprises can confidently lower SEIM costs by adopting Snowflake as a security data lake in conjunction with the Anvilogic platform. By supporting additional data lakes, Anvilogic accelerates adoption of Snowflake in the Enterprise SOC as an additional data lake that does not require rip and replace of existing data lakes. The partnership between Snowflake and Anvilogic creates value for customers by providing a platform that is flexible, easy to use, and accelerates time to detection and response.</p><blockquote>According to Tim Yip, Head of Cybersecurity Services, Crypto.com, “Anvilogic Unified Detect significantly reduces the learning curve when building SQL-based detections and has instilled greater confidence in our detection engineering process.”</blockquote><p>Want to see <a href="https://www.anvilogic.com/solutions/search-and-detect">Unified Detect</a> in action? Visit <a href="http://anvilogic.com/trial">anvilogic.com/trial</a> to request a two-week free trial with a low-lift effort from your team. In addition, you can attend the Anvilogic session at our Snowflake Summit <a href="https://www.snowflake.com/summit/sessions/?agendaPath=session/1115938">here</a>. Want to get started on Snowflake? Visit <a href="http://signup.snowflake.com/">signup.snowflake.com</a> to start your 30-day free Snowflake trial that includes $400 worth of free usage.</p><p><em>Sincere Thanks to Deb Banerjee, Co-founder and CTO at Anvilogic; Kelly Huang, Product Marketing Manager at Snowflake and Ashley Frizzell, Senior Partner Marketing Manager at Snowflake for their contributions to this blog.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d26461aa1747" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How To Achieve Security Operations at Scale with Panther & Snowflake]]></title>
            <link>https://medium.com/@ravikuma2003/how-to-achieve-security-operations-at-scale-with-panther-snowflake-5a6bfc85092?source=rss-5dbe5845b1ab------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a6bfc85092</guid>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Ravi Kumar]]></dc:creator>
            <pubDate>Mon, 03 Oct 2022 13:06:27 GMT</pubDate>
            <atom:updated>2022-10-03T13:06:27.906Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Sub: Get real-time security monitoring and threat detection for Snowflake with zero operational overhead.</em></p><p>Fortunately, Snowflake and Panther Labs have been working together to make this possible. With the Panther security monitoring solution working in tandem with your Snowflake account, you get fast, flexible and scalable threat detection and response capabilities and true end-to-end security visibility that can grow with your business.</p><h3>Focus on Security That’s Scalable, Practical, and Easy to Use</h3><p>Panther and Snowflake were built leveraging cloud-native principles to solve problems of scale. As more applications and SaaS solutions are brought into your business, your data, and attack surface, grows exponentially. Server-based solutions just can’t keep up. Snowflake’s forward-thinking approach of creating a cloud-based data lake enables businesses to not just store all their data in a single place, but to put that data to use and make sense of it without the hurdles of managing hot and cold storage or other administrative headaches. With the separation of storage and querying capabilities, Snowflake enables Panther to scale with businesses, no matter how many terabytes of data they need to analyze to protect their attack surface. A Snowflake data lake can be spun up in minutes (seriously) and Snowflake takes care of all provisioning, upgrades, and management, so you can focus on gathering and analyzing security data, not provisioning infrastructure.</p><p>Snowflake has mastered scalability for ever-expanding data needs. And, Panther has approached the problem of scale for security purposes in a similar way and with a shared mindset of building cloud-native, agile technology to keep up with exploding data volumes. Panther’s security monitoring solution leverages Snowflake to give security teams the ability to ingest and retain petabytes of logs and query any size data set in seconds with zero operational overhead.</p><p>With Panther analyzing and alerting on potential threats on top of your Snowflake data lake, you gain the security visibility you need to focus on preventing breaches rather than just reacting to them. While being cloud-native is one step toward making security and data management scalable, it’s just the one piece of the puzzle. Panther also incorporates software development principles and workflows to help security teams simplify and improve operations and save time.</p><h3>Virtually Unlimited Scalability</h3><p>Traditional SIEM platforms have not kept pace with the demands of today’s cloud workloads, resulting in poor performance, exorbitant licensing costs and heavy operational burdens for security teams. Many of these challenges stem from a server-based architecture which requires significant administrative overhead to increase storage capacity or increase compute resources which limits security teams’ ability to access and analyze the data they need, when they need it.</p><p>Panther was founded by a team of veteran security practitioners who faced these challenges first-hand and built a platform to solve them. Using Snowflake to power cloud architecture enables Panther to ingest and normalize terabytes of data per day, and transform raw logs into a structured security data lake to power threat-hunting and investigations.</p><p>Cloud architecture separates where the data lives (storage) from the resources needed to load and query the data (compute) which reduces storage costs dramatically, and makes it fast and easy to spin up additional compute resources to power massive query workloads during an investigation, and then quickly return to normal operations. This flexibility and responsiveness is simply not possible with a traditional, server-based SIEM platform. Even if it was possible, the cost of keeping petabytes of data available for querying on a moments’ notice would be astronomical. But, this is the kind of scale modern security teams actually need, particularly for threat investigations that span months in time and require extensive historical data analysis.</p><p>With Panther, security teams get real-time security monitoring and threat detection with zero operational overhead at a fraction of the cost of traditional solutions, even when ingesting and analyzing terabytes of data per day.</p><h3>Flexibility, Automation, and Customization with Detection-As-Code</h3><p>Detection-as-Code is a modern, flexible, and structured approach to writing detections that applies software engineering best practices to security. By treating detections as code that can be tested, checked into source control, and code-reviewed by peers, teams can produce higher-quality alerts that reduce false positives and accurately flag suspicious activity.</p><p>By embracing Detection-as-Code and a Python-based rules engine, Panther helps security teams build scalable processes for writing and hardening detections which improves their ability to identify sophisticated threats across rapidly expanding environments.</p><p>Additionally, Panther offers 200+ pre-built detections mapped to specific threats, suspicious activity and security frameworks like <a href="https://attack.mitre.org/">MITRE ATT&amp;CK</a>. These built-in rules can be customized as needed to fit a specific environment, or security teams can craft their own flexible and expressive detections in Python.</p><p>For more information about Panther’s Detection-as-Code approach and to see an example of how to write custom detections, check out: https://panther.com/cyber-explained/detections-as-code/?utm_source=snowflake&amp;utm_medium=tech_ptnr</p><h3>Reduce Complexity with Connected Application Model</h3><p>If you’re already a Snowflake customer and you’re looking to build a security program that takes advantage of the efficiency, performance and flexibility of a security data lake, you’ll be glad to know that Panther offers a connected application model. Essentially, you can bring Panther’s SaaS environment directly into your Snowflake account to begin the data normalization process.</p><p>Think of this option as an easy entry into next-generation cloud security and end-to-end security visibility, built right into your existing Snowflake account. If your team is already used to working with Snowflake, this option can help streamline and simplify deployment of Panther, while giving you real-time alerting, analytics, and pre-built security and threat detections.</p><p>One additional (and significant) benefit of adding Panther to your Snowflake environment is that you can enrich your data without moving it or dealing with an ETL process or solution. You’ll have the ability to easily and quickly enrich your Snowflake data with threat intelligence, asset data, and other feeds and data sources to get the relevant business context you need for proper security investigations.</p><h3>Enforce Snowflake Security Best Practices with Panther</h3><p>In addition to being powered by Snowflake, Panther also recently introduced “<a href="https://panther.com/product/panther-for-snowflake/?utm_source=snowflake&amp;utm_medium=tech_ptnr">Panther for Snowflake</a>”, a version of the product with detections developed in collaboration with Snowflake security experts. These purpose-built detections alert security teams of potential tampering or unwanted changes to security settings and monitor for incidents or security risks in their Snowflake Data Cloud. With Panther for Snowflake, Snowflake event logs are aggregated, normalized and analyzed along with security events from other systems so security teams have greater visibility into potential threats across their entire IT infrastructure from cloud to on-premises.</p><h3><strong>Summary</strong></h3><p>Security is a big-data problem, and modern security teams need a solution for threat detection and response that treats it as such. Panther has partnered with Snowflake to solve this problem with a cloud-native platform, capable of transforming massive volumes of raw security data into a structured security data lake. With this solution, security teams can analyze data in real-time, get answers quickly during investigations, and conduct on-going threat-hunting to secure their organizations.</p><p>Learn more at <a href="https://panther.com/?utm_source=snowflake&amp;utm_medium=tech_ptnr">https://panther.com/?utm_source=snowflake&amp;utm_medium=tech_ptnr</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a6bfc85092" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>