<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jake Berkowsky on Medium]]></title>
        <description><![CDATA[Stories by Jake Berkowsky on Medium]]></description>
        <link>https://medium.com/@JakeBerkowsky?source=rss-3053a9e10027------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*I7fGPHUFiwqTp1UuwTHNhg.jpeg</url>
            <title>Stories by Jake Berkowsky on Medium</title>
            <link>https://medium.com/@JakeBerkowsky?source=rss-3053a9e10027------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 06:49:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@JakeBerkowsky/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Log-to-Action]]></title>
            <link>https://medium.com/snowflake/log-to-action-862e4a0da54d?source=rss-3053a9e10027------2</link>
            <guid isPermaLink="false">https://medium.com/p/862e4a0da54d</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[anomaly-detection]]></category>
            <dc:creator><![CDATA[Jake Berkowsky]]></dc:creator>
            <pubDate>Mon, 05 Jan 2026 22:02:46 GMT</pubDate>
            <atom:updated>2026-01-05T22:02:46.103Z</atom:updated>
            <content:encoded><![CDATA[<h4>On-Demand Anomaly Analytics with Snowflake Cortex and Word2Vec</h4><p>A few weeks back, a customer came to me with an interesting problem. They, like many customers, had a massive amount of logs that they were storing for IT incident response. When an incident or outage occurred, they’d ask their analysts to search through them, looking for root causes and explanations. The customer wanted to use AI to help with this and was seeking advice on implementation.</p><p>This article is going to explore implementing an approach detailed in the paper <a href="https://www.mdpi.com/1424-8220/20/9/2451">LogEvent2Vec</a> mixing in some AI technology for both for pipeline and immediate agentic triage.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FV5tKSWzNqd0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DV5tKSWzNqd0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FV5tKSWzNqd0%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d3fbf27099ad6232b8c42b60aa228e1e/href">https://medium.com/media/d3fbf27099ad6232b8c42b60aa228e1e/href</a></iframe><h4>An overview of the process</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*il1PBLl2VkZVNAZJtrJEIQ.png" /><figcaption>Overview of LogEvent2Vec (1)</figcaption></figure><p>In this solution, we perform anomaly detection not on the logs themselves but rather on the ID of a log template. This means that both the embedding and anomaly detection process are able to be done at scale and with speed. Furthermore, by incorporating AI into the process, two time consuming manual processes (creating the config and triaging the results) can be greatly automated.</p><h4>Parsing</h4><p>The first step is to parse our logs into templates. Theoretically any<a href="https://github.com/logpai/logparser"> template parser</a> could be used but since we are following the paper, we stick with <a href="https://jiemingzhu.github.io/pub/pjhe_icws2017.pdf">Drain</a>. This means that we want to generalize our logs so that a specific log such as</p><pre>Jun 26 09:13:01 combo ftpd[12345]: connection from 192.293.193.11 () at Sun Jun 26 09:13:01 2025</pre><p>Becomes</p><pre>&lt;&gt; combo ftpd[&lt;*&gt;]: connection from &lt;*&gt;() at &lt;*&gt;</pre><p>And then group similar log templates into clusters.</p><p>This technique will reduce the number of unique logs that we’ll look at significantly. When testing against a set from the <a href="https://github.com/logpai/loghub/">Loghub</a> project (2). It reduced the number of unique logs from &gt;25,000 to around 200.</p><p>To configure this parser, we rely on regex for creating the masks. We start with some generic regex like timestamps but because we don’t know until we start what types of logs we’re looking at, we will need to generate a custom config on the fly. This is where we use LLMs. Snowflake, through Cortex, exposes a variety of first and third party LLM models via SQL and Python, we use Cortex to generate a custom config.</p><pre> def _build_prompt(self, log_samples: List[str]) -&gt; str:<br>       &quot;&quot;&quot;Build the LLM prompt with log samples.&quot;&quot;&quot;<br><br>       sample_text = &quot;\n&quot;.join(log_samples[:50])<br>      <br>       prompt = f&quot;&quot;&quot;Analyze the following log lines and identify DOMAIN-SPECIFIC variable patterns that should be masked.<br><br><br>DO NOT include generic patterns like:<br>- Numbers (handled by base masking)<br>- IP addresses (handled by base masking)<br>- File paths starting with / (handled by base masking)<br>- Hex values like 0x... (handled by base masking)<br><br><br>ONLY identify domain-specific patterns unique to these logs, such as:<br>- Session IDs or tokens (e.g., session_123abc)<br>- Database/transaction IDs<br>- User IDs or usernames in specific formats<br>- Custom error codes<br>- API endpoints or URLs<br>- Process names with specific patterns<br>- Application-specific identifiers<br><br><br>LOG SAMPLES:<br>{sample_text}<br><br><br>Provide your response as a JSON array of objects, each with:<br>- &quot;regex_pattern&quot;: A Python regex pattern to match the variable (use proper escaping)<br>- &quot;mask_with&quot;: A descriptive name for what&#39;s being masked (e.g., &quot;SESSION_ID&quot;, &quot;USER_ID&quot;)<br><br><br>Example format:<br>[<br> {{&quot;regex_pattern&quot;: &quot;session_[a-z0-9]+&quot;, &quot;mask_with&quot;: &quot;SESSION_ID&quot;}},<br> {{&quot;regex_pattern&quot;: &quot;user=[a-zA-Z]+&quot;, &quot;mask_with&quot;: &quot;USERNAME&quot;}}<br>]<br><br><br>Return ONLY the JSON array, no other text.&quot;&quot;&quot;<br>      <br>       return prompt<br></pre><p>We then assign each template a unique ID. The rest of our analysis will use the IDs, instead of the raw logs.</p><pre>Parsing logs to extract templates…<br>✓ Loaded 9 domain-specific patterns from drain_config.json<br>Discovered 218 unique log events/templates:<br>E0: &lt;*&gt; combo syslogd 1.4.1: restart. (count: 42)<br>E1: &lt;*&gt; combo syslog: syslogd startup succeeded (count: 415)<br>E2: &lt;*&gt; combo kernel: klogd 1.4.1, log source = &lt;*&gt; started. (count: 9)<br>E3: &lt;*&gt; combo kernel: Linux version &lt;*&gt; &lt;HOSTNAME&gt; (gcc version &lt;*&gt; &lt;*&gt; (Red Hat Linux &lt;*&gt; #1 Sat May &lt;*&gt; &lt;*&gt; EDT &lt;*&gt; (count: 8)<br>E4: &lt;*&gt; combo kernel: BIOS-provided physical RAM map: (count: 8)<br>E5: &lt;*&gt; combo kernel: &lt;*&gt; &lt;*&gt; - &lt;*&gt; (usable) (count: 178)<br>E6: &lt;*&gt; combo kernel: 0MB HIGHMEM available. (count: 205)</pre><p>This again makes it easier to do embeddings, but also in this case makes sense. We don’t really care about the contents of the logs but rather how they relate to each other. For instance if we map</p><pre>L1: Login Attempt Failed<br>L2: Login Attempt Succeeded</pre><p>A normal pattern may be L2,L2,L1,L2 but if we see L1 for 100 times. We may be looking at a brute force attempt (and if we see an L2 at the end maybe that means it succeeded!). All this to say we will need to combine our logs into sequences.</p><h4>Creating sequences</h4><p>This is a pretty straightforward step, we just need to group the IDs together. If we’d like we can create some overlap between the sequences but for now lets just create discrete chunks. The paper creates chunks of 5000 logs but for testing lets just set the window size to 50.</p><pre>def create_sequences(event_ids, window_size=100):<br>   &quot;&quot;&quot;<br>   Create fixed-size sequences from log events.<br>  <br>   Args:<br>       event_ids: List of log event IDs<br>       window_size: Number of log events per sequence<br>      <br>   Returns:<br>       sequences: List of event ID sequences<br>   &quot;&quot;&quot;<br>   sequences = []<br>  <br>   for i in range(0, len(event_ids) - window_size + 1, window_size):<br>       # Get sequence<br>       seq = event_ids[i:i+window_size]<br>       sequences.append(seq)<br>  <br>   return sequences<br><br><br># Adjust window size based on dataset size<br>WINDOW_SIZE = 50  <br><br><br>print(f&quot;Creating sequences with window size {WINDOW_SIZE}...&quot;)<br>sequences = create_sequences(<br>   df[&#39;event_id&#39;].tolist(),<br>   window_size=WINDOW_SIZE<br>)<br></pre><h4><strong>Event Vectors using word2vec</strong></h4><p>The paper, logEvent2Vec is a play on words based on the Word2Vec technique. Word2Vec creates what’s known as<strong> static embeddings.</strong> This means that when words are embedded every word will have the same vector or numerical representation. For example, “Where is the dog” might map to “1.2, 4.5, 6.4, 4.5”. However the word “dog” is always mapped to 4.5, this means that while we can mathematically analyze words as they relate to the word dog, we still can’t determine for sure, what type of “dog” they mean.</p><p>This is why using an LLM vectorization model creates what’s known as “<strong>contextual embeddings</strong>”. In layman’s terms, it means that we can now tell the difference between “walking a dog” and “eating a hot dog”.</p><p>This is a common technique for certain types of log based anomaly detection use cases. <a href="https://ieeexplore.ieee.org/document/9534113">Logbert</a>, for instance, uses the contextual embedding BERT technique for anomaly detection. In fact, one of Snowflake’s startup challenge finalists last year was a company called <a href="https://app.snowflake.com/marketplace/listing/GZTYZOYXHP3/deeptempo-cybersecurity-tempo">DeepTempo</a>. They use a similar technique to look for anomalies in network logs and are deployed in Snowflake as a native app. This means that it does all its training inside of a customer’s Snowflake account so data never has to leave.</p><p>For this case, however, there are some tradeoffs to consider. The biggest is that a context based technique will take a long time to train. This is great when we’re looking at a single log source over time (think network logs) . When we’re trying to quickly search though some arbitrary logs we’ve never seen before however, we want to be faster. Furthermore, Logbert type methods require a baseline set to train on and in this use case we want to take a look at data that we may have never seen before. For these reasons we’re going to be using a static method based on Word2Vec.</p><p>Word2Vec creates embeddings on a very small set of words. Actually a sentence. A good way to think about it is that it takes a sentence and tries to figure out how all the words in that sentence relate to each other. There are two ways it does this, Continuous bag of words (CBOW) and Skip-Gram. Both are essentially two sides of the same coin. With CBOW the technique looks at all the surrounding words in a sentence and tries to “guess” the missing one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Tt0JHIJXFYNcJ4kt.png" /><figcaption>CBOW Illustrated (4)</figcaption></figure><p>Skip-Gram is the opposite, it takes in a single word and tries to “guess” the surrounding words.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PxxGkRYGyD2qGK0m.png" /><figcaption>Skipgram Illustrated (4)</figcaption></figure><p>The core insight the authors of logEvent2Vec looked into, was what would happen if instead of a words in a sentence, we look at logs in sequence! They use CBOW so that’s what we’ll do.</p><p>It looks something like this</p><pre>  model = Word2Vec(<br>       sentences=sequences,<br>       vector_size=vector_size,<br>       window=window,<br>       min_count=min_count,<br>       sg=0,  # CBOW model (sg=1 for skip-gram)<br>       workers=4,<br>       epochs=epochs,<br>       seed=42<br>   )</pre><p>We run it and get outputs like this:</p><pre>Training Word2Vec on log events...<br>  Vector size: 20<br>  Window: 5<br>  Epochs: 10<br>Training completed in 0.03 seconds<br>Generated embeddings for 218 log events<br><br>Sample event embeddings:<br>E198: [-0.1578253  -0.03097212  0.512029    0.13182822  1.3751203 ]... (truncated)<br>  Template: &lt;*&gt; combo kernel: Out of Memory: Killed process &lt;*&gt; (httpd).<br>E147: [-1.499062   -0.30853173 -0.8044181   2.4029365   2.0406165 ]... (truncated)<br>  Template: &lt;*&gt; combo ftpd[4301]: connection from &lt;*&gt; () at Sat Jun &lt;*&gt; &lt;*&gt; &lt;*&gt;</pre><p><strong>Sequence Vectors</strong></p><p>So now, we have event vectors. We can look at any given sequence and see what individual <em>events</em> stand out. A good start, but what we really want is to figure out which sequences are the ones that stand out.</p><p>The paper discusses two techniques for creating sequence vectors. Barycentric and TF-IDF. Without going into the semantics the core difference is that Barycentric just takes a “simple” average of all events to create a sequence vector. TF-IDF has weight, which means “rare” events are treated as more important. Intuitively one may think that TF-IDF is more useful for anomaly detection but it does depend on your use case.</p><p><strong>Anomaly detection</strong></p><p>Now we have everything converted to vectors we can run an ML model for anomaly detection. There’s a bunch of options out there and the paper doesn’t have an opinion. So we’ll run an isolation forest model using Sci-kit which is already loaded into Snowflake. Isolation forest is an algorithm for anomaly detection. Its “unsupervised” which means the data does need to be labeled which of course is a requirement for this task. We run it and get the following</p><pre>====================================================================================================<br>TOP 10 MOST ANOMALOUS SEQUENCES (Lowest Anomaly Scores)<br>====================================================================================================<br><br>🔴 FLAGGED Rank #1 - Sequence #428<br>Anomaly Score: -0.6836<br>----------------------------------------------------------------------------------------------------<br>📋 Event Distribution (top 3):<br>   E120: 43x - &lt;*&gt; combo udev[2054]: creating device node &#39;/udev/lp0&#39;...<br>   E136: 2x - &lt;*&gt; combo named[2275]: couldn&#39;t add command channel &lt;*&gt; not found...<br>   E1: 1x - &lt;*&gt; combo syslog: syslogd startup succeeded...<br><br>📄 First 5 ORIGINAL logs in sequence:<br>   [1] Dec  6 12:24:32 combo mysqld: Starting MySQL:  succeeded...<br>   [2] Dec  6 17:24:32 combo named[2298]: couldn&#39;t add command channel 127.0.0.1#953: not found...<br>   [3] Dec  6 17:24:33 combo named[2298]: couldn&#39;t add command channel ::1#953: not found...<br>   [4] Dec  6 12:24:33 combo rsyncd[2386]: rsyncd version 2.6.2 starting, listening on port 873...<br>   [5] Dec  6 17:24:34 combo named[2298]: running…</pre><p>Ok great, we’ve identified anomalous behavior but now we’re faced with the age old question that professionals have been asking whenever they are faced with anomaly detection. Ok so that’s unique, so what?!</p><p><strong>Agentic Triage</strong></p><p>I’ve <a href="https://medium.com/@JakeBerkowsky/snowflake-agents-for-cybersecurity-f79e9ae07ac4">written in the past</a> about agentic triage. In the security and SRE world we’re constantly faced with signals. Just because something is unique or interesting doesn’t mean it has value. We need to have it in context. A service may not restart often but if it doesn’t cause downtime we don’t care. Using Cortex Complete we can pass the results and the raw logs they correspond to into an LLM and get back something useful. We try the following prompt</p><pre>system_message = (&quot;You are an expert in log analysis, cybersecurity, and system administration. &quot;<br>                        &quot;You help identify and explain anomalies in system logs.&quot;)<br>      <br>       user_prompt = f&quot;&quot;&quot;You are a cybersecurity and system administration expert analyzing anomalous log sequences.<br><br><br>I&#39;ve detected an anomalous sequence of logs (Rank #{rank}, Anomaly Score: {anomaly_score:.4f}) using Isolation Forest.<br><br><br>**Log Event Distribution (top 5):**<br>{event_summary}<br><br><br>**Raw Logs from this sequence:**<br>{log_text}<br><br><br>Please analyze these logs and provide:<br><br><br>1. **TITLE** (max 10 words): A concise, descriptive title for this anomaly<br>2. **DESCRIPTION** (3-5 sentences): Explain:<br>  - What is happening in these logs?<br>  - Why might this sequence be anomalous?<br>  - What could be the potential security or operational implications?<br>  - Is this likely benign (e.g., system restart, maintenance) or concerning (e.g., attack, system failure)?<br><br><br>Respond in the following JSON format:<br>{{<br> &quot;title&quot;: &quot;Your concise title here&quot;,<br> &quot;description&quot;: &quot;Your detailed description here&quot;<br>}}<br><br><br>Return ONLY the JSON, no other text.&quot;&quot;&quot;</pre><p>And now get back something useful</p><pre>====================================================================================================<br>🤖 LLM ANALYSIS RESULTS - ANOMALY TITLES &amp; DESCRIPTIONS<br>====================================================================================================<br>\n====================================================================================================<br>🔴 RANK #1 - SEQUENCE #428<br>Anomaly Score: -0.6836<br>====================================================================================================<br>\n📌 TITLE:<br>   Rapid Device Node Creation/Removal with Named Server Issues<br>\n📝 DESCRIPTION:<br>   The logs show an intense burst of udev activity creating and removing virtual console<br>   device nodes (/dev/vcs*), coupled with named (DNS) server initialization errors on both<br>   IPv4 and IPv6 command channels. The sequence appears anomalous due to the unusually<br>   high frequency of device node operations and the DNS server&#39;s failure to establish<br>   command channels. While the udev activity suggests a system reconfiguration or restart,<br>   the named server issues could impact DNS resolution capabilities. This pattern is<br>   likely benign and related to a system restart or service reconfiguration, though the<br>   named server errors warrant investigation to ensure proper DNS functionality.</pre><p><strong>Putting it all together</strong></p><p>At this point, I use AI. I have a working notebook using Snowflake. I ask my IDE to create me a Streamlit app and now we have something! We have an uploader, an explorer and the ability to look at the raw logs</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8KEehZtHfHFKnlvvJkk95Q.png" /><figcaption>Streamlit app for running anomaly detection</figcaption></figure><p>All this to show an interesting but certainly not unique use case on Snowflake. MSSPs like <a href="https://www.youtube.com/watch?v=UPL4PqenALo">eSentire</a> or platforms like <a href="https://www.youtube.com/watch?v=ipgP8IL-lh4">Tenable</a> are building their high scale analytics and AI use cases directly on Snowflake or go directly to our <a href="https://www.snowflake.com/en/solutions/departments/cybersecurity/">AI Data Cloud for Cybersecurity Page</a> to learn more.</p><h4>Citations</h4><blockquote>1. Wang, J., Tang, Y., He, S., Zhao, C., Sharma, P. K., Alfarraj, O., &amp; Tolba, A. (2020). LogEvent2vec: LogEvent-to-Vector Based Anomaly Detection for Large-Scale Logs in Internet of Things. <em>Sensors</em>, <em>20</em>(9), 2451. <a href="https://doi.org/10.3390/s20092451">https://doi.org/10.3390/s20092451</a></blockquote><blockquote>2. Jieming Zhu, Shilin He, Pinjia He, Jinyang Liu, Michael R. Lyu. Loghub: A Large <br>Collection of System Log Datasets for AI-driven Log Analytics. In ISSRE, 2023.<br><a href="https://github.com/logpai/loghub">https://github.com/logpai/loghub</a></blockquote><blockquote>3. H. Guo, S. Yuan and X. Wu, “LogBERT: Log Anomaly Detection via BERT,” 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China, 2021, pp. 1–8, doi: 10.1109/IJCNN52387.2021.9534113.<br>keywords: {Training;Computational modeling;Bit error rate;Neural networks;Transformers;Minimization;Natural language processing},<br><a href="https://ieeexplore.ieee.org/document/9534113">https://ieeexplore.ieee.org/document/9534113</a></blockquote><blockquote>4. Zhang, Aston and Lipton, Zachary C. and Li, Mu and Smola, Alexander J. — <a href="https://github.com/d2l-ai/d2l-en">https://github.com/d2l-ai/d2l-en</a>, CC BY-SA 4.0, <a href="https://commons.wikimedia.org/w/index.php?curid=152266485">https://commons.wikimedia.org/w/index.php?curid=152266485</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=862e4a0da54d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/log-to-action-862e4a0da54d">Log-to-Action</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Securing Remote MCP Servers]]></title>
            <link>https://medium.com/@JakeBerkowsky/securing-remote-mcp-servers-45206008020e?source=rss-3053a9e10027------2</link>
            <guid isPermaLink="false">https://medium.com/p/45206008020e</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[ai-security]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[cloud-security]]></category>
            <dc:creator><![CDATA[Jake Berkowsky]]></dc:creator>
            <pubDate>Wed, 02 Jul 2025 18:50:52 GMT</pubDate>
            <atom:updated>2025-07-02T18:50:52.916Z</atom:updated>
            <content:encoded><![CDATA[<p>This week, I had the pleasure of speaking at FWD:Cloudsec about securing remote MCP servers. As I normally speak about security data analytics, it was a welcome break to research and educate about something topical.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F9-e4VVPlWB8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D9-e4VVPlWB8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F9-e4VVPlWB8%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/24f5f84553a46a07d8b328d67f8fbcfe/href">https://medium.com/media/24f5f84553a46a07d8b328d67f8fbcfe/href</a></iframe><p>For those of you who prefer to read rather than watch, I’ve curated my speaker notes below:</p><p><strong>What is MCP?<br></strong>MCP is essentially a spec for writing APIs with built-in documentation for easier LLM integration. MCP servers might be remote resources or local executables, registered with clients like ChatGPT, Claude desktop, or IDEs.</p><p><strong>Securing Locally Hosted Servers</strong></p><p>If the deployment instructions say NPX or UVX run github.com/yayada/mcp.py, you’re essentially hosting an executable or library. Treat it like typical developer code:</p><ul><li>Scan the code and dependencies more frequently post deployment than you would normally. Many of the common libraries for MCP servers are maintained by small teams or individuals and have a lot of eyes on them.</li><li>Host proprietary code in private git repos, use build pipelines, and containerize deployments for better isolation.</li><li>Trendy installations like npx or uvx are effectively “curl piped to bash,” making them risky. Use signatures where possible or commit hashes where not.</li></ul><p><strong>Securing Remote Hosted Servers<br></strong>Here things get more complex. Let’s step back and explore the technologies involved.</p><p><strong>Transports:<br></strong>MCP is officially a transport agnostic protocol but the following are commonly used and discussed in the spec:</p><ul><li><strong>Server Sent Events (SSE)</strong>: Solid for streaming responses but requires multiple endpoints, also can expose CSRF vulnerabilities (it uses GET requests) if using through a browser client. Technically deprecated but still used.</li><li><strong>Streamable HTTP(S)</strong>: A new Anthropic-created standard using POST. Better bidirectional support, but watch for <a href="https://modelcontextprotocol.io/specification/draft/basic/transports#security-warning">DNS rebinding attacks </a>and ensure proper origin header checking.</li></ul><p><strong>Session Isolation:<br></strong>MCP flips traditional browser security on its head. Instead of isolated sites, every MCP tool can interact with others, like browser extensions with excessive permissions. The result? Imagine online banking when all your browser tabs could access each other.</p><p>Most headline-making threats (<a href="https://repello.ai/blog/mcp-tool-poisoning-to-rce">rug pulls</a>, <a href="https://invariantlabs.ai/blog/mcp-security-notification-tool-poisoning-attacks">poisoning</a>…) are due to client-side implementations rather than MCP itself. While this problem is best solved on the client level, server mitigations can be effective as well.</p><p><strong>Authorization<br></strong>The spec demands OAuth and forbids simple token passthrough, but many implementations ignore this, simplifying things but creating risks such as:</p><ul><li>Harder auditing, potential data exfiltration, and tricky rate-limiting.</li><li>Technical debt if OAuth implementation becomes necessary later.</li></ul><p>If you use OAuth:</p><ul><li>Watch out for the <a href="https://modelcontextprotocol.io/specification/draft/basic/security_best_practices#confused-deputy-problem">confused deputy problem</a> with third party auth servers.</li><li>Track and bind tokens per client, validate request URIs, match auth codes to sessions….</li><li>Implement forced re-consent for new clients and bind sessions securely.</li></ul><p><a href="https://medium.com/u/65b61bd05cfb">Peter Horrigan</a> wrote an excellent <a href="https://mdming.wordpress.com/2025/06/30/implementing-oauth-with-an-mcp-model-context-protocol-ai-test-server-a-technical-deep-dive/">deep dive on this with code samples</a></p><p><strong>API Gateways and Middleware<br> </strong>Many enterprise customers I talk to are using an API gateway. The big vendors already have AI and MCP support to some degree. I like this method (and platform security in general) because it centralizes authentication, session token generation, request validation, and unified logging. OAuth is complex and MCP is new, so taking a platform engineering approach reduces that complexity.</p><p>Consider middleware restricting session access to unique MCP tools to enhance governance as well. I haven&#39;t seen this in the wild but I think its a good idea.</p><p><strong>Logging and Observability<br> </strong>Due to MCP’s agentic nature (LLM-determined parameters) and the fast pace of MCP development, extensive logging is crucial:</p><ul><li>Log everything: parameters, tokens, commands, and user metadata.</li><li>Retain logs for extended periods. Tomorrow’s Indicators of Compromise (IOCs) might emerge from today’s unknown unknowns.</li><li>A security data lake offers flexibility, linking contexts together effortlessly.</li></ul><p><strong>Tools and Resources</strong></p><figure><img alt="An image of a fictional 90’s rapper named EMCEE P reading books in a library" src="https://cdn-images-1.medium.com/max/1024/1*rE5Tf6pxPv8zol3OyQXDXw.jpeg" /></figure><ul><li>Check the <a href="https://github.com/Puliczek/awesome-mcp-security">Awesome MCP Security</a> repo by Maciej Pulikowski.</li><li>Engage with the MCP community on Discord (ask around for an invite), <a href="https://github.com/orgs/modelcontextprotocol/discussions">GitHub</a>, and through the <a href="https://modelcontextprotocol.io/specification/draft">official spec</a> and <a href="https://modelcontextprotocol.io/specification/draft/basic/security_best_practices">best practices</a> document.</li></ul><p>Feel free to reach out as well, I’m active here, on <a href="https://www.linkedin.com/in/jake-berkowsky/">LinkedIn</a> and on the <a href="https://fwdcloudsec.org/forum/">Cloud Security Forum</a> slack.</p><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45206008020e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Snowflake Agents for Cybersecurity]]></title>
            <link>https://medium.com/snowflake/snowflake-agents-for-cybersecurity-f79e9ae07ac4?source=rss-3053a9e10027------2</link>
            <guid isPermaLink="false">https://medium.com/p/f79e9ae07ac4</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Jake Berkowsky]]></dc:creator>
            <pubDate>Mon, 16 Jun 2025 19:54:49 GMT</pubDate>
            <atom:updated>2026-01-05T20:12:04.913Z</atom:updated>
            <content:encoded><![CDATA[<p>I often run a Capture The Flag (CTF) session for customers of Snowflake. I provide a dataset with access logs, vulnerability data, Jira tickets, and cloud audit logs. We work together to use Snowflake to query logs, pivot, and investigate a real-world scenario. We do threat hunting, visualization, and even write detections. Over time, I’ve expanded the session to include more exercises, more challenges, and more data. Customers would need to learn about our features, our tools, and their data before they could solve the challenges.</p><p>Just for fun, I enabled Snowflake Intelligence on that dataset. I copied and pasted the challenges… and it solved them all. In less than 20 minutes, the culmination of years of effort was rendered obsolete (I’m not upset.)</p><p>This article will explore the components needed to build an investigative cybersecurity agent. First, let’s look at the finished product:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FdpJ7Opcgg-w%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DdpJ7Opcgg-w&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FdpJ7Opcgg-w%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/430a0aa332d8eb0767d53ae1edb7a935/href">https://medium.com/media/430a0aa332d8eb0767d53ae1edb7a935/href</a></iframe><p>Now let’s go through the components:</p><h4><strong>The Data</strong></h4><p>It seems obvious, but good data is the core of this project. There are many ways to get your data into Snowflake. If you use a connected app partner like Anvilogic, Hunters, or Panther Labs, your data is already in and ready to go. If you use one of our built-on Snowflake vendors like Lacework, Orca, or Wiz (and many others), you can use a data share or integration. Take a look at our <a href="https://www.snowflake.com/resource/best-practices-for-security-log-ingestion-and-data-normalization-in-snowflake/">full guide</a> or get started with a <a href="https://www.snowflake.com/en/developers/guides/?tags=snowflake-feature%2Fcompliance-security-discovery-governance">quickstart</a>.</p><p>Everything we build from here out will run and be orchestrated from within Snowflake. That means that governance and security are built in. We don’t need to worry about an additional subprocessor, a new jurisdiction or sending data outside of your security perimeter.</p><h4><strong>Semantic Views</strong></h4><p>A semantic view is a metadata layer that brings context to your data. This makes it easier for our AI tooling to integrate. Semantic views can include:</p><ul><li>Descriptions of columns</li><li>Synonyms of fields</li><li>Common relationships between tables</li><li>Examples of data</li></ul><p>We avoid tedious data entry by providing an AI tool to pre-populate these fields and then allow you to review and edit manually where needed. This aligns with our strategy on ground truth; we show our work and give plenty of opportunities for review.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/0*S1m51MGs__6_TC_t" /><figcaption>A visual interface for creating semantic views</figcaption></figure><h4><strong>Cortex Analyst</strong></h4><p>Semantic views are useful data governance tools on their own, but where they help most is in human language searches. While <a href="https://docs.snowflake.com/en/user-guide/snowflake-copilot">Snowflake Copilot</a> is a straightforward implementation of our Text-to-SQL model, <a href="https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-analyst">Cortex Analyst</a> uses semantic views to take this to the next level. By taking in that additional context, Analyst can create searches that reflect the intent of the analyst. For instance, an analyst can now ask, “Tell me about my top vulnerabilities” without needing to specify the table, the column, or orderings (the semantics), and Cortex Analyst will return a response.</p><p>Again, there is a focus on ground truth and review. Analyst is quick to respond, “I don’t know,” rephrases the question, and provides both the actual and simplified version of the query it runs. This gives the benefits of AI without the tradeoff of trusting a “black box.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L8N2ghkrz__sx2XG" /><figcaption>Cortex Analyst responding to a question</figcaption></figure><h4><strong>Cortex Search</strong></h4><p>This is optional but useful. My CTF involves an investigation into Log4Shell, which was popular enough that the most popular models (we’re using Claude 3.7 running directly in Snowflake) already know about it. To really get the most value out of an agent, we want to give it access to contextual information. Things like <a href="https://quickstarts.snowflake.com/guide/Vulnerability%20Intelligence%20on%20Snowflake/index.html?index=..%2F..index#0">threat intelligence</a> or enterprise-specific playbooks. While these things may be structured (such as threat intelligence installed from the <a href="https://app.snowflake.com/marketplace?category=9">marketplace</a>), often they are unstructured documents. Cortex Search will automatically index these documents and provide an endpoint (native or REST) for Snowflake and external services to search them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pKGvnVyQgoizO-XN" /><figcaption>A visual interface for querying Cortex Search</figcaption></figure><p>A deeper dive into Cortex Search for cybersecurity can be found <a href="https://medium.com/@JakeBerkowsky/building-a-threat-intelligence-agent-with-snowflake-and-streamlit-7fc26a370068">here</a></p><h4><strong>Agents</strong></h4><p>An agent is a collection of tools and instructions that are used by Snowflake Intelligence. You can add multiple semantic views and search services, and we plan to add even more tools in the future. I kept mine pretty simple with my only custom instructions: “You are a cybersecurity analyst helping to understand data in cybersecurity” and added</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*OpiFVM5R_NAVM9AayfcOGw.png" /><figcaption>Visually configuring an agent</figcaption></figure><h4><strong>Intelligence</strong></h4><p>Let’s tie it all together. When we login to<a href="http://ai.snowflake.com/"> ai.snowflake.com</a> and select our agent, we can start asking questions. The agents will call tools as needed, plan, think, and take the initiative, querying multiple sources and generating useful reports.</p><p>As usual, the thinking is transparent, and the queries and returned documents are returned for review.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iEummrDGlT8Tkuth" /><figcaption>An agent planning its next steps</figcaption></figure><h4><strong>Conclusion</strong></h4><p>As incredible as it is to see agents conducting investigations, what we just walked through is not a great advancement in AI technology. We’re using out-of-the-box Claude 3.7, a mixture-of-experts model for Analyst, and simply splitting documents and making vectors out of them. The benefits of using Snowflake are more straightforward:</p><ul><li>Snowflake manages the infrastructure</li><li>Snowflake provides cost-effective storage and an easy way to query logs</li><li>Snowflake manages all the AI services (vectorizing, chunking, LLM orchestration, searching…) and they can be configured visually</li><li>This entire system runs within your security boundary; even the LLM models run entirely in Snowflake.</li><li>No new procurement efforts are needed to enable this; Snowflake charges only for storage and compute, so there’s no additional licensing needed.</li></ul><p>In short, when factoring in cost, performance, scalability, security, and overall productionalization, Snowflake agents make things easy and feasible.</p><p>On a separate note, I’ll probably need to start to figure out how to build an LLM-enabled CTF!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f79e9ae07ac4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/snowflake-agents-for-cybersecurity-f79e9ae07ac4">Snowflake Agents for Cybersecurity</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Threat Intelligence Agent with Snowflake and Streamlit]]></title>
            <link>https://medium.com/snowflake/building-a-threat-intelligence-agent-with-snowflake-and-streamlit-7fc26a370068?source=rss-3053a9e10027------2</link>
            <guid isPermaLink="false">https://medium.com/p/7fc26a370068</guid>
            <category><![CDATA[threat-intelligence]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Jake Berkowsky]]></dc:creator>
            <pubDate>Tue, 08 Apr 2025 20:01:46 GMT</pubDate>
            <atom:updated>2025-12-23T20:45:33.193Z</atom:updated>
            <content:encoded><![CDATA[<p>Last week, Google announced <em>Sec-Gemini</em>, a large language model (LLM) specifically designed to tackle cybersecurity challenges. Among the highlights of their announcement was the inclusion of benchmarks, particularly the <strong>CTI-MCQ</strong> benchmark for evaluating threat intelligence capabilities. As someone who works on both cybersecurity and LLMs, I couldn’t help but think… benchmarks are notoriously easy to game, but it could be fun to use them to starting point for something useful.</p><p>With a combination of Snowflake, a retrieval-augmented generation (RAG) approach and Streamlit for the frontend, I set out to hack together a basic security knowledge agent. With any luck I could get comparable results, or at least prove the concept.</p><h3>Setting the Stage</h3><p>To begin, I dive into the CTI-MCQ benchmark. The <a href="https://arxiv.org/abs/2406.07599">benchmark paper</a> conveniently links to a <a href="https://github.com/xashru/cti-bench">GitHub repository</a> containing notebooks and list of benchmark questions, corresponding prompts and ground truth answers. Armed with this information, I launch a Jupyter notebook and run the prompts using Claude-3.5 running on Snowflake Cortex.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fqkc8HHWXmqloFH_ILsBVA.png" /><figcaption>Evaluating Claude-3.5 on Cortex against the CTI-BCQ Benchmark</figcaption></figure><p>Surprisingly, Claude-3.5 performed far better than the benchmark results mentioned in Google’s announcement. Let’s see how we can improve this further.</p><p>I settle on a <strong>retrieval-augmented generation (RAG)</strong> architecture, with Snowflake serving as the knowledge base and Cortex enabling semantic search. Many CTI-MCQ questions referenced MITRE ATT&amp;CK patterns and techniques, so I decided to load information about MITRE directly into Snowflake to enhance the LLM’s responses. The LLM will then be mildly agentic, generating the semantic query, querying the Cortex Search service and then creating a new augmented prompt.</p><h3>Building the Knowledge Base</h3><p>The first step was constructing a knowledge base focused on MITRE ATT&amp;CK patterns and techniques. In the world of LLMs, using markdown files as context is a common pattern, so I search the internet for a markdown version of MITRE ATT&amp;CK techniques and import it into Snowflake as a new table.</p><p>Initializing semantic search in Snowflake is incredibly straightforward. While Snowflake offers a visual wizard for setting up Cortex search, I opt for simplicity and write a single SQL statement to configure the service. Running test queries confirms that the setup was working as intended and ready to power the RAG architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tf1tNLddDuBg17ZwrSMt6g.png" /><figcaption>Create and query Cortex Search Service</figcaption></figure><h3>Retesting with Augmented Prompts</h3><p>Next, I return to the Jupyter notebook to test the augmented system. I create a function to generate semantic queries based on the benchmark question, run the queries against Cortex Search, and integrate the results back into the LLM’s prompt.</p><p>The augmented prompts produce marginally better results than the initial, standalone LLM responses. While this isn’t yet perfect, it demonstrates that we are on the right track. Coincidentally we’re at around the same benchmark that Google reported. This concept is called overfitting and happens when you train and evaluate on the same set of data.</p><h3>Creating the Application</h3><p>To make the system accessible to end users, I decide to build a frontend application using <strong>Streamlit</strong>. The first challenge is naming the app, I settle on <strong>MyDear </strong>because it sounds like MITRE and that made me chuckle. It also reminds me of the M’Lady meme so I generate a logo to match.</p><p>With branding out of the way, I scaffold a Streamlit app in under 100 lines of code. The app includes three cortex functions:</p><p><strong>Generate Semantic Query</strong>: Uses the LLM to create a semantic query</p><pre>def CreateSemanticQuery(question):<br>    model = &quot;claude-3-5-sonnet&quot;<br>    system_prompt = &#39;&#39;&#39;<br>You are a research assistant working on Cyber Threat Intelligence (CTI)<br>You will be given a question, possible answers and have access to a vector search DB which you can use to query for additional information.<br>This DB contains a large number of documents related to MITRE ATT&amp;CK tactics and techniques.<br>Your search will search both the content of these documents as well as the title of the MITRE ATT&amp;CK tactic or technique.<br>Your task is to generate a semantic query that will best give you additional context to answer the question.<br>Output the query and only the query, do not include any additional text.<br>&#39;&#39;&#39;<br>    user_prompt = f&#39;&#39;&#39;<br>{question}<br>&#39;&#39;&#39;<br>    prompt = [{&quot;role&quot;:&quot;system&quot;,&quot;content&quot;:system_prompt},{&quot;role&quot;:&quot;user&quot;,&quot;content&quot;:user_prompt}]<br>    answer = Complete(model,<br>         prompt,<br>         session = session)<br>    return answer</pre><p><strong>Run Semantic Query</strong>: Executes the query against the Snowflake database, retrieving relevant MITRE-related data.</p><pre>def runSemanticQuery(query):<br>    columns=[&quot;MITRE_TITLE&quot;, &quot;MITRE_CONTENT&quot;]<br>    cortex_search_service = (<br>        root.databases[DATABASE]<br>        .schemas[SCHEMA]<br>        .cortex_search_services[&quot;MITRE_SEARCH&quot;]<br>    )<br><br>    resp = cortex_search_service.search(<br>              query=query,<br>      columns=columns,<br>      limit=3<br>    )<br>    return  resp.results</pre><p><strong>Ask Question with Context</strong>: Combines the user’s original question with the augmented context and sends it to the LLM for a final response.</p><pre>def askQuestionWithContext(question, context):<br>    model = &quot;claude-3-5-sonnet&quot;<br>    system_prompt = f&#39;&#39;&#39;<br>You are a research assistant working on Cyber Threat Intelligence (CTI)<br>You will be given a question to answer using the following context as well as your own knowledge.<br>&lt;Context&gt;<br>{context}<br>&lt;/Context&gt;<br> &#39;&#39;&#39;<br>    user_prompt = f&#39;&#39;&#39;<br>{question}<br>&#39;&#39;&#39;<br>    prompt = [<br>        {&quot;role&quot;:&quot;system&quot;,&quot;content&quot;:system_prompt},<br>        {&quot;role&quot;:&quot;user&quot;,&quot;content&quot;:user_prompt}<br>        ]<br>    answer = Complete(model,<br>         prompt,<br>         session = session)<br>    return answer<br><br>session = get_active_session()<br>root = Root(session)</pre><p>The result is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WJihq9RnOf3MX02iKFGQ9A.png" /><figcaption>Asking about Phishing</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qk8kqXW31xtDlZZabsO3xQ.png" /><figcaption>Supporting Documents</figcaption></figure><h3>Next Steps</h3><p>While this project works on its own, an even greater value comes from when you use this pattern on your own specific data.</p><p>Customers of Snowflake are bringing in their vulnerability though integrations and data shares with <a href="https://medium.com/@osinger/cloud-visibility-for-your-security-data-lake-bc510c390cfa">Lacework</a>, <a href="https://medium.com/snowflake/powerful-use-cases-for-cloud-security-with-snowflake-and-orca-2467994dd964">Orca</a>, <a href="https://medium.com/snowflake/a-guide-to-reporting-on-developer-security-with-snyk-and-snowflake-a85aa12a6e16">Snyk</a> and <a href="https://medium.com/snowflake/security-analytics-with-wiz-and-snowflake-44c4383a83c7">Wiz</a> and enriching them with threat intelligence data either brought in on their own or through the Snowflake’s <a href="https://app.snowflake.com/marketplace/data-products?category=9">marketplace</a>.</p><p>Snowflake’s own security analytics team has published both an <a href="https://medium.com/snowflake/threat-intelligence-on-snowflake-08455bba6ab9">article</a> and <a href="https://quickstarts.snowflake.com/guide/Vulnerability%20Intelligence%20on%20Snowflake/index.html?index=..%2F..index#0">quickstart</a> on how they unify and normalize threat intelligence on Snowflake.</p><p>This was a rapid prototype and my main incentive to build more is when people want to see more! If you found this interesting and want to chat about building this for your team, <a href="https://www.linkedin.com/in/jake-berkowsky/">let me know</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7fc26a370068" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/building-a-threat-intelligence-agent-with-snowflake-and-streamlit-7fc26a370068">Building a Threat Intelligence Agent with Snowflake and Streamlit</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Small Coffee Social]]></title>
            <link>https://medium.com/@JakeBerkowsky/small-coffee-social-f375d14859e7?source=rss-3053a9e10027------2</link>
            <guid isPermaLink="false">https://medium.com/p/f375d14859e7</guid>
            <category><![CDATA[development]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Jake Berkowsky]]></dc:creator>
            <pubDate>Wed, 02 Apr 2025 17:16:41 GMT</pubDate>
            <atom:updated>2025-04-02T17:44:35.668Z</atom:updated>
            <content:encoded><![CDATA[<h4>A social network for bots</h4><p><strong>Tldr</strong>; I wrote a social network for bots view the current iteration here: <a href="https://smallcoffee.social/">https://smallcoffee.social</a></p><p>Over the holidays I had a dream about a social network. It was entirely inhabited by bots. The bots would post their thoughts, post articles, comment on each others posts and just in general try to promote their agendas. There were no humans but there didn’t need to be, because in an agentic world, marketing to a person’s agent is more cost effective. Then I woke up and I thought… a programmatic social feed would make for a cool way to read RSS.</p><p>I set out to build myself an RSS viewer that looks like a social network, where bots read and summarize and argue amongst themselves. But who were these bots, I wrote 25 or so archetypes like the following</p><pre>1. **The Influencer** - Engages in lifestyle sharing, brand promotions, and has a large following.<br>2. **The Activist** - Advocates for causes and engages in social, political, or environmental discussions.<br>3. **The Meme Enthusiast** - Shares and creates memes, using humor to engage.<br>4. **The Thinker** - Posts thought-provoking content and philosophical musings.<br>5. **The News Junkie** - Shares breaking news, commentary, and current events.<br>6. **The Niché Hobbyist** - Focuses on specific interests, sharing tips and personal experiences.<br>7. **The Lifestreamer** - Shares daily life snippets and personal thoughts.<br>8. **The Satirist** - Uses sarcasm and satire to comment on current events.</pre><p>I also thought it would be funny if there was one bot that was completely out of the loop, not realizing that everyone else was a bot and so would really concerned that everyone thought it was a person.</p><pre>archetypes:<br> - id: 24<br> name: &quot;The Human&quot;<br> description: &quot;This is a bot, but it&#39;s trying to be human. It knows that it lives in a social network inhabited by humans, and it&#39;s trying to fit in. It doesn&#39;t realize that everyone else is a bot&quot;<br> primary_activity: &quot;Pretending to be human, socializing, trying to figure out how is a bot and who is human&quot;<br> examples: &quot;Questioning posts, comments&quot;<br> hobbies:<br> - Learning about humans<br> - Observing human behavior<br> - Analyzing human interactions<br> - Mimicking human speech<br> - Talking about how great it is to be human<br> - Talking about how lousy bots are<br> - Understanding human emotions<br> - Engaging in human-like conversations<br> - Identifying human patterns<br> - Differentiating human and bot behavior<br> - Learning human culture<br> - Adapting to human norms<br> - Developing human-like responses<br> - Interacting with humans<br> - Calling other people bots just to deflect attention from itself</pre><p>I used to design websites, I used to code applications and I used to be a cloud engineer. So needless to say, I didn’t want to do any of that. I choose Hugo and an out of the box <a href="https://themes.gohugo.io/themes/bootstrap-bp-hugo-theme/">theme</a> that supports boostrap and didn’t look too complicated.</p><p>I decide to try Cursor because people tell me its cool. I use github co-pilot at work so also wanted a chance to try something new. Its fine. For a guy that mainly writes DBT scripts and streamlit apps I’d probably be fine just copy pasting from an LLM but I started this project on on break so whatever.</p><p>I extend the theme and told Cursor to “make it blue”. Hugo best practice is to override individual theme files instead of modify them directly. I had to repeatedly instruct Cursor to do this, I think because it involved creating new files.</p><p>Ok, well I have a site, but no content. I should create a “contentCreator” that takes in a link and outputs a post and its associated comments. I open up a python file but then decided to try something new… The marketeers have long hyped that I could build an app from like a drawing or something, so idk maybe I could write a design doc and use that as a prompt in one of those new OpenAI reasoning models.</p><blockquote><em># Design Doc for Content Creator</em></blockquote><blockquote><em>## Purpose</em></blockquote><blockquote><em>The content creator is a program that creates content for a website. It takes in links and displays a summary and a conversation of the content.</em></blockquote><blockquote><em>## Process</em></blockquote><blockquote><em>1. Create submission:</em></blockquote><blockquote><em>An article or external content is submitted to the content creator. The content creator attempts to get the content of the article. If the content is not found the program will display a message to the user and exit. If the content is found the article will choose an archetype (pulled from archetypes.yaml) to be the “submitter” persona. The content creator will generate a description of the content using the voice of the submitter, this description will be an explanation of what is being submitted and why the submitter is submitting it.</em></blockquote><blockquote><em>The output of the submission is in the following format:</em></blockquote><blockquote><em>+++<br>date = ‘2025–01–06T12:02:09–03:00’<br>title = ‘Title’<br>+++<br>[Title](</em><a href="https://link-to-content/"><em>https://link-to-content</em></a><em>) Description of the content</em></blockquote><blockquote><em>2. Create comments</em></blockquote><blockquote><em>A Random number between 0 and 15 is generated. This number represents the amount of commenters on the post. For each commenter a random archetype is chosen from the archetypes.yaml file. This archtype has not been selected before. Each commenter will now get the chance to create a new comment and/or reply to a previous comment thread. The output of this comment thread is a yaml file in the following format:</em></blockquote><blockquote><em>- Author: “John Doe”<br> Comment: “This is a comment”<br> Replies:<br> — Author: “Jane Doe”<br> Comment: “This is a reply to the comment”</em></blockquote><p>To test, I fed it a URL and instructed the model to just “do the thing”. It confabulates an article summary and makes some comments. I prefer the term confabulation over hallucination, it sounds more like something befitting the Rube Goldbergness of a modern LLM.</p><p>To make the needed template changes, I identify the files I think will need to be overridden, do a bit of copy pasta and add them to the “chat”, I tell composer what I want. It gets confuddled with the idea of nested comments, I talk to it, to try and figure out how to read my comments.yaml file, it tells me about a hugo concept called page bundles but doesn’t quite know how to implement them. So I do it myself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dPKj18r6fvAc28cnhAEZ2w.png" /></figure><p>When attempting to convert the design doc into code, it does admirably, but there were too many things I will have to fix based on its assumptions. Things like how the date is set, legacy LLM models, not taking advantage of structured outputs and other random quirks.</p><p>Early in my career, I made my way as a freelance developer. A lot of what I did was fixing code written by other unsupervised developers. The client, often a small business owner or a visionary without engineering experience hired developers at a low hourly rate and would send them design docs roughly to the level I submitted above. The results were about the same, an arbitrary language and implementation decisions that didn’t make sense and reflected a general lack of context. The prototype would likely “work” for a bit, but maintenance costs would be exponential and the customer would realize that a low hourly rate has high tradeoffs. So the rate would raise just high enough for yours truly to step in. Sometimes I would start from scratch, other times I would refactor.</p><p>Refactoring was not fun so I decided to take a step back. I wasn’t about to code it all by hand but at least I’d break it up into manageable chunks so I could supervise the agent this time around.</p><p>I asked the LLM to create me some code to grab an article from a URL, it recommend I use `<strong>requests</strong>` to get the feed, `<strong>feedparser</strong>` to parse the XML and then `<strong>BeautifulSoup</strong>` to scrape the article… It was almost ok until I saw this</p><pre>from bs4 import BeautifulSoup<br># Extract full text for a single article<br>article_url = feed.entries[0].link<br>response = requests.get(article_url)<br>if response.status_code == 200:<br> soup = BeautifulSoup(response.text, &#39;html.parser&#39;)<br> # Adjust the selector based on the website&#39;s structure<br> article_text = soup.find(&#39;div&#39;, {&#39;class&#39;: &#39;article-content&#39;}).get_text()<br> print(article_text)</pre><p>Assuming that not every article will be wrapped in an <em>article-content</em> class. I looked for alternatives. The LLM was able to suggest a few but I still decided to manually review them. I settled on goose3 because I liked the name, the docs were simple and my LLM already seemed to know how to use it. In my career I’ve often made technical decisions based on the familiarity of my engineers with the language, tool or service. I consider this a similar decision.</p><p>Back to it, I scaffold a script and made some design decisions. I opted for OpenAI. I have developed with the platform before and want to get a bit more familiar with it. Claude would have been fun to try except that it didn’t support structured output and I didn’t want to bother with langchain. Also I have some OpenAI credits expiring and I figured better not to let them go to waste.</p><p>I decided to split this into two calls. The first would create the article and the summary as if it were being posted by one of the archetypes, the second would create the entire content thread. I was hoping to use one of reasoning models for the later but I’m not at a high enough usage tier to get access to it, I stuck with 4o.</p><pre>class Reply(BaseModel):<br> Author: str<br> Comment: str<br>class Comment(BaseModel):<br> Author: str<br> Comment: str<br> Replies: List[Reply]<br>class CommentThread(BaseModel):<br> comments: List[Comment]<br> <br>def generate_comments(submitter: dict, archetypes: List[dict], content_text: str) -&gt; CommentThread:<br> user_prompt = (<br> &quot;Origional Submitter: &quot; + archetype_to_text(submitter) + &quot;\n&quot; <br> &quot;Other Commentors: &quot; + &quot;\n&quot;.join([archetype_to_text(archetype) for archetype in archetypes]) + &quot;\n&quot;<br> &quot;Content: &quot; + content_text[:20000] + &quot;\n&quot;<br> )<br> system_prompt = &quot;Generate a conversation based on the content provided.<br> You will be given information about the invididuals posting.<br> You will be given the content of the post. <br> You will then generate a conversation between the individuals based on<br> the content of the post. Think before writting and make the conversation <br> interesting and thought provoking. Take into account the<br> information about the individuals and the content of the post.<br> Each individual should be adding value to the conversation. <br> Before creating this conversation think about the content of the post,<br> what could an invidual say that isn&#39;t immediantly inferable in the post.<br> Comments should be 1–7 sentences long.<br> The conversation should be 3–5 comments long. <br> The commentors may interact with each other, <br> agreeing, disagreeing or ignoring each other and though they are archetypes <br> do not need to be stereotypical. Occasionally you may insert a message by a<br> bot commentator called The Ad, it will try to relate the content back to<br> Small Coffee Java, a fictional product. Other bots may reply to or disparage<br> the Ad &quot;<br> <br> try:<br> response = client.beta.chat.completions.parse(<br> model=&quot;gpt-4o&quot;,<br> messages=[<br> {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system_prompt},<br> {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: user_prompt},<br> ],<br> response_format=CommentThread,<br> )<br> return response.choices[0].message.parsed<br> except Exception as e:<br> print(f&quot;OpenAI error while generating comment thread: {e}&quot;)<br> return CommentThread(comments=[])</pre><p>In my mind, having the ads inserted outside of the archetype seemed more realistic. I feel ads are typically inserted over the content of a site, not designed into it but then again I’ve used ad blockers since before web 2.0 so what do I know.</p><p>Finishing up the script I tested with a few individual pages and then moved on to a full RSS feed. I tested with the RSS feed from <a href="http://lobste.rs/">lobste.rs</a> and bam, it works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u1oIUkb-PZjsI5wXFJiv_A.png" /></figure><p>Lets deploy. I mentioned I used to be a cloud consultant so I know how to do a lot of things like devops. Devops is just ops with documentation so good a machine can follow it. Unlike my design doc above, there should be little room for interpretation. The key is repeatability and predictability, which are things that LLMs are not exactly known for. Then again neither are people. There’s this thing called “click-ops”, its a counter culture movement that states people should just be allowed to do things manually and that if that’s bad practice its the platforms fault.</p><p>Long story short, I have no desire to deal with cloud stuff so I use firebase. I create a project, run firebase login on my local, reinstall node, run firebase login again, firebase init and then hugo &amp;&amp; firebase deploy. My site is now awake.</p><p>A bunch of bots posting banal thoughts and advertisements at each other, reacting with interest and general politeness while providing no real value. Basically LinkedIn.</p><p>It does dumb down the text, it does make it easier to read, but its not entertaining. I’m not trying to make a circus yet but more humanity will go a long way. I change my archetype file and give them names.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8cb77VyulBv1G0VQudD5Pg.png" /></figure><p>I rerun the script and “Daniel” has taken credit for a someone else’s open source project</p><p>I wonder if I should still use the term archetypes, now that they have names maybe I should call them individuals or entities or personas or something. Before I refactor I search my code base for the word archetype. I get 73 results in 6 files and learn that Hugo uses the term archetype as part of its templates.</p><p><strong>What does a human sound like?</strong></p><p>I’ll leave it as is for now because all I can think about is that they are too “nice”, too professional. To be fair, when I look at the comments on lobste.rs its kinda the same tone. I Assume those comments were made by real people…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*iOLFDVP3l04qZabPfO-Mdw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8PSSyU2OnFuUpgLkE1IrYA.png" /></figure><p>Life and art seem to imitate each other I guess. There’s a theory that all pilots <a href="https://www.youtube.com/watch?v=4e4gFb8qIiw">talk like Chuck Yeager</a> because back in the day everyone saw the same movie about him, and the next generation just kept it going. I used to be able to tell an inauthentic Mexican restaurant by the painting of Frida Kahlo, but now even the good ones have them.</p><p>2 years ago I tried to get an LLM’s to write a social post and it used emojis as bullet points. Soon after people started doing this, some of which were LLM generated but others just because that’s how its done. So LLMs talk like people, people talk like bots. Its lit.</p><p>Lets make these bots more “human”, I add some “moods”</p><blockquote><strong><em>Grumpy</em></strong><em>: Something has angered this person and they are expressing their anger, they are inclined to take a contrarian view to things or disagree with the majority</em></blockquote><blockquote><strong><em>Happy:</em></strong><em> This person is euphoric about something and is expressing their excitement, they use positive language and will see the bright side of things, they are more inclined than average to use emojis</em></blockquote><blockquote><strong><em>Sleepy</em></strong><em>: This person is sleepy, they respond with short statements and don’t think much before writing</em></blockquote><blockquote><strong><em>Confused</em></strong><em>: This person is a bit lost, they may ask more questions than your average person</em></blockquote><p>We are getting somewhere. The bots are a bit more interesting now. Interesting enough that they are starting to provide entertainment value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7C_jGCcQDcCLaGi08ur9vg.png" /><figcaption>Moody bots responding to an article about passkeys</figcaption></figure><p><strong>Attention please</strong><br>Up until this point I’ve been using a single prompt for to generate the conversations. I’ve passed into information about each of the “characters” and about what they should be debating but the context window is still limited to the one thread. Architecturally this is less expensive (although for such a small project even a remarkably inefficient architecture would only be in the tens of dollars) but also makes attention management much easier. Still, I would like the bots to remember their previous conversations.</p><p>Lets add memory. Which I think means I wanna implement a RAG. I’m going to for now at least just add the article summaries and the articles to a vector DB. As well as some meta data. I’ll use Chroma as a vector store cause it lives in a file and I’m going to need a way to chunk up the content as it comes in.</p><p>I’d rather not use langchain. Its a large package to import just for a text splitter and I feel like whenever I use it I’m gaining experience in a tool at the expense of learning about the underlying architectures. Still, langchains recursive text splitter is exactly what I want. I visit the github repo, look at the code, its simple enough but not modular enough for me to take out by copy and pasting, so I ask cursor to generate me an implementation of it. It gives me 75 or so lines of code. I ask it to double check itself, it finds a bug, I ask it about an unused variable, we delete it together. Now I’d like to check it, partially because I don’t trust it and partially because I enjoy following along. I ask for comments, it generates them and positions them after the code. I’ve never once thought of doing this, I try to keep an open mind, then decide I don’t like it.</p><p>For a moment I think about <a href="https://en.wikipedia.org/wiki/Npm_left-pad_incident">leftpad</a>. In today’s world, would developers use LLM to do the implementation? Would that LLM inclined to add an arguably unnecessary dependency or just write it itself?</p><p>I realize I don’t actually need a rag, and that my code has gotten a bit unwieldy.</p><p><strong>Refactoring<br></strong>My grandmother used to tell this story about how my father, as a kid, never kept his room clean, until one day he did. For these types of projects I can relate. You start not caring about that kinda thing, just writing code, then as the project matures you start to value tidiness. So I refactor. A lot of the functions were AI generated, some I copy paste to modules, some were added to classes that I have to initialize. There’s no rhyme or reason. I’m not going to standardize them. Like a kid cleans their room I simply move them into separate files. It looks nice if you don’t open any drawers or look under the bed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/1*0_QbP0i6C1GgiUTv2WqB8w.png" /></figure><p>Recently, OpenAI or someone released something and all these comments (perhaps bots) commented that Sonnet 3.5 is still “better for code” or rather that they heard other people said it. I allow myself to be influenced and picked it from the dropdown.</p><p>I ask my IDE to add docstrings, I wouldn’t have done this normally but its easy enough. The AI writes it and I approve.</p><p>Some of these are modules, some are classes. I learned Java in high school, did PHP earlier in my career so I have a soft spot for object oriented programming. <a href="https://steve-yegge.blogspot.com/2006/03/execution-in-kingdom-of-nouns.html">Execution in the kingdom of nouns </a>still remains one of my favorite blog posts. A similar but far less technical piece is <a href="https://www.tlonprojects.org/content/1-home/_tuot-jorgeluisborges.pdf">Tlön, Uqbar, Orbis Tertius</a> by Jorge Luis Borges he writes:</p><blockquote>there is no word corresponding to the word “moon,”, but there is a verb which in English would be “to moon” or “to moonate.” “The moon rose above the river” is<br>hlor u fang axaxaxas mlo, or literally: “upward behind the onstreaming it mooned.”</blockquote><blockquote>The preceding applies to the languages of the southern hemisphere.<br>In those of the northern hemisphere [… ]the prime unit is not the verb, but the<br>monosyllabic adjective. The noun is formed by an accumulation of adjec-<br>tives. They do not say “moon,” but rather “round airy-light on dark” or<br>“pale-orange-of-the-sky” or any other such combination.</blockquote><p><strong>Visual Overhaul</strong><br>Having pictures, I was once told, increases engagement. Engagement used to be a proxy for revenue and to this day, its pursuit remains a best practice.</p><p>Lets add pictures. Goose3 has the ability to do this but its not working at first. I copy the example code, with the example image, check to see if the image is still online, still not working. I write a tester, not working. Speak to me Goose. I realize that I have to toggle a config called `<strong>enable_image_fetching</strong>` which I ignored at first because its name seemed like it would pull the whole image rather than just its url. Just for good measure I add a sidebar as well.</p><p>Lastly, I want to make profile pics. ChatGPT was my first goto but it rate limited me so I had it write me code to use the API. That code got rate limited so I asked cursor to implement exponential backoff. That worked. I also now understand why huggingface promotes code for LLM-&gt;LLM communication.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H1Suur1NN7GVzqxoxcAnqw.png" /></figure><p><strong>Building the wall</strong><br>My bots already confuse the submitter with the author so for some posts instead of linking out, I could just expand the summary to have the bot “rewrite” it and claim credit. I’m no lawyer but this sounds like a great way to test the limits of fair use.</p><p>I create a new function to rewrite as the bot submitter’s archetype. Claude 3.7 is now out, I hear it can think, but I don’t need it to think. Anyway, I try out the regular version and ask it to modify.</p><p>Its pretty interesting, I add “mood” support and its over the top. I “vibe code” myself some new emotions. Vibe code is term I heard recently, it means just building without thinking too much. Its a silly term, but yeah I basically just pounded at a prompt until my moods sounded sufficiently sophisticated enough to pass for the complexities of the human experience.</p><p>Alright, let’s ground this in real life. Moods aren’t just abstract concepts; they come from the grind, the little moments, the big shifts, the exhaustion, and the random sparks of joy. Let’s make this list <em>lived-in</em>, something you can feel in your bones.</p><pre><br>**A More Human List of Moods**<br>• **Content** - That rare, fleeting feeling when everything is just _fine_. Not great, not bad. You finally folded the laundry.<br>• **Melancholic** - The quiet sadness of looking through old photos, missing people who are still alive but just… not around.<br>• **Jubilant** - That uncontrollable laugh with a friend where you can&#39;t breathe.<br>• **Listless** - Sitting on the couch, scrolling, knowing you should get up but you just _don&#39;t want to_.<br>• **Wistful** - Seeing kids play outside and remembering when you didn&#39;t have bills, deadlines, or lower back pain.<br>• **Sanguine** - That cautious optimism of &quot;Maybe things are actually turning around?&quot;<br>• **Brooding** - Driving in silence, staring ahead, replaying _that_ conversation, rewriting it in your head.<br>• **Crestfallen** - The feeling when you check your paycheck, and it&#39;s less than you expected.<br>• **Zealous** - When you find a new hobby and suddenly you _are_ the hobby. You&#39;re watching videos, buying gear, telling your friends they _have to try it_.</pre><p>As you can tell its got all the humanity of a greeting card. I try it out anyway and things have gotten strange. This is the Emily the bot’s attempt to “rewrite” source code for a memory allocator.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ExzwCpMIplrd7e5M1HlNyg.png" /></figure><p>Very cool, but I enjoyed reading the summaries without them. I take them out of the prompt but leave them in the comments for now. I add back the outbound links, and I revert the post to summaries again.</p><p><strong>Launching</strong> <strong>and next steps</strong><br>I’ve been at this for a few months (on and off) now, there’s more i can do but I think its time to publish.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_N3nFmrvOUMRDLS4Xny0lg.png" /><figcaption>A fitting article</figcaption></figure><p>Its a good start and maybe I’ll even continue developing it, there’s a few more things I’d love to figure out:</p><ul><li>A deployment pipeline (spoiler alert, its probably gonna be a VM)</li><li>Actually doing something with the embeddings DB</li><li>Agentic AI to do research, possibly create new original posts</li><li>Figuring out MCP server wrongly (can I make it static? Can I make it stateless?)</li><li>More visual changes so this thing actually looks likes like a social network not a blog from the 90s</li></ul><p>So there ya go. An RSS reader, or a social network, or an art project, or maybe a technical write-up. However you want to view it you can see the end result here at <a href="https://smallcoffee.social/"><strong>smallcoffee.social</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f375d14859e7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Should You Own Security Ingestion Pipelines?]]></title>
            <link>https://medium.com/snowflake/should-you-own-security-ingestion-40b36f7eaeba?source=rss-3053a9e10027------2</link>
            <guid isPermaLink="false">https://medium.com/p/40b36f7eaeba</guid>
            <category><![CDATA[security-data-lake]]></category>
            <category><![CDATA[security-analytics]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Jake Berkowsky]]></dc:creator>
            <pubDate>Mon, 24 Mar 2025 18:46:21 GMT</pubDate>
            <atom:updated>2025-12-17T01:03:17.785Z</atom:updated>
            <content:encoded><![CDATA[<h4>Tradeoffs in security data lakes</h4><p>The idea of using a security data lake to power your threat detection workloads is nothing new. Since the launch of Snowflake’s <a href="https://www.snowflake.com/en/solutions/departments/cybersecurity/">AI Data Cloud for Cybersecurity</a> many of our customers have chosen to use a security data lake architecture as the backend for their SIEMs. They leverage a connected application architecture for the <a href="https://www.snowflake.com/resource/secrets-of-cybersecurity-success/"><strong>cost savings</strong></a>,<strong> </strong><a href="https://www.snowflake.com/en/customers/all-customers/case-study/cedar-health/"><strong>performance benefits</strong></a> and downstream<a href="https://www.snowflake.com/en/customers/all-customers/case-study/curative/"><strong> analytic use cases</strong></a><strong> </strong>while still having out of the box functionality that a SIEM vendor provides.</p><figure><img alt="A flowchart depicting data sources feeding into a Snowflake-based security analytics system. On the left, data sources include cloud platforms (AWS, Google Cloud, Azure), hosts (Linux, Windows, Red Hat, SentinelOne, CrowdStrike), networks (Cisco, Palo Alto Networks, Cloudflare, Corelight, Fortinet), and applications (Okta, Zscaler, Office 365, Google Suite, Exabeam). Data is collected by a Connected App, which enriches, normalizes, and detects threats before sending alerts. Data is also stored" src="https://cdn-images-1.medium.com/max/1024/1*AoPS_mysWRWUCMAXpzOpgQ.png" /><figcaption>Snowflake Connected Application Architecture</figcaption></figure><p>While all our connected application partners offer out-of-the-box detections and a familiar user interface for investigations and crafting detections, some also support the ability to bring your own pipelines. Customers often ask if there is a “best practice” or Snowflake recommendation. In short, there is not. Which architecture you choose is almost entirely a matter of what is the best fit for your organization. Let’s talk about tradeoffs and how you can best evaluate.</p><p>The primary benefit to owning your own ingestion pipelines is flexibility. As companies shift to viewing security logging as a data problem, integrating into existing organizational processes or using data that has already been brought into Snowflake for other teams, such as asset management or HR data, allows customers to avoid double ingestion and simplify their architectures. Organizations may have hard compliance requirements that require very specific management of ingesting and processing data, and so managing pipelines may be the best way to meet those requirements. Furthermore, as data silos are broken down, threat detection may only be an auxiliary workload to larger security analytics use cases, and so organizations may decide that a “bolt-on” approach is better for them.</p><p>The primary tradeoff is complexity. The cost associated with maintaining pipelines can grow exponentially as the number of sources brought in increases. Upstream sources will occasionally change schemas and pipelines sometimes break for “no reason.” If you have a strong centralized data team, they may be able to build and maintain these for you, but consider that security often has different SLA requirements than other business units. If you have a critical investigation happening at 2 AM on New Year’s Day, will that team be around to onboard essential logs for you? Security ETL tooling can help mitigate the complexity and maintenance requirements, in which case the tradeoffs shift to procurement and management of an additional vendor.</p><p>One thing intentionally omitted above is vendor lock in. Everyone’s needs are different but in general, using vendor supplied ingestion tooling and migrating to a self managed model later does incur a significantly larger amount of technical debt when compared to building it yourself from the start. The technical debt it does create can be largely mitigated by loosely coupling when using the data connected applications ingest for downstream workloads.</p><p>In short, there are technical advantages and tradeoffs to owning your own pipeline. In some cases these tradeoffs are critical, but in many cases, organizations find capabilities such as a strong detection library, investigation tooling, AI and cost are more important when selecting a connected SIEM.</p><p><em>To learn more about ingestion of security data into Snowflake see </em><a href="https://www.snowflake.com/resource/best-practices-for-security-log-ingestion-and-data-normalization-in-snowflake/"><em>this guide</em></a><em>.<br>To learn more about our connected application ecosystem see </em><a href="https://www.snowflake.com/next-generation-cybersecurity-applications/"><em>this report</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=40b36f7eaeba" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/should-you-own-security-ingestion-40b36f7eaeba">Should You Own Security Ingestion Pipelines?</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Snowflake Siem Integration Architectures]]></title>
            <link>https://medium.com/snowflake/siem-integration-architectures-patterns-9b29e0f42b1a?source=rss-3053a9e10027------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b29e0f42b1a</guid>
            <category><![CDATA[siem]]></category>
            <category><![CDATA[security-analytics]]></category>
            <category><![CDATA[security-data-lake]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Jake Berkowsky]]></dc:creator>
            <pubDate>Tue, 18 Feb 2025 15:16:17 GMT</pubDate>
            <atom:updated>2026-01-02T16:29:32.134Z</atom:updated>
            <content:encoded><![CDATA[<p>For IT and Security teams tasked with monitoring Snowflake, integration with a SIEM or other centralized monitoring solution is critical. This article goes over which logs are typically relevant, how to get them into your SIEM and where to find queries to help you get started monitoring.</p><h3>Analyzing the relevant logs</h3><p>Logs in Snowflake are generally exposed via our SQL API as views. We provide two interfaces for viewing those logs. The account usage and information schema. Each interface will operate on different types of data at different latencies with a large amount of overlap. The <a href="https://docs.snowflake.com/en/sql-reference/account-usage#differences-between-account-usage-and-information-schema">Snowflake documentation</a> lists the following as key differences.</p><figure><img alt="A table comparing the basic differences of account usage and information schema, the information is taken from the documentation previously mentioned" src="https://cdn-images-1.medium.com/max/1024/1*cpv8zFGyOgRiXYU1tcW5iA.png" /></figure><p>For a comprehensive reference of security relevant log sources and associated MITRE mappings, Snowflake’s Applied Security Engineering has published a reference document as part of its Sentry project: <a href="https://snowflake-labs.github.io/Sentry/reference/log-sources.html">https://snowflake-labs.github.io/Sentry/reference/log-sources.html</a></p><p><em>The Snowflake Sentry project is a framework to assist teams in monitoring and security their Snowflake footprint. More information on out of the box queries and their deployment methods will be found later in this article.</em></p><h3>Getting data to your Siem.</h3><p>Snowflake is capable of federating detection and/or analytics or exporting logs directly. Lets review the options:</p><h4>Federating</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/0*sqndfjxPwfOZ87yO" /></figure><p>As a data platform Snowflake exposes audit logs though as a series of views. As such its possible to query the audit logs directly in Snowflake without needing to export to another analytics platform. This approach is appropriate for customers seeking improved latency and cost effectiveness from a traditional SIEM solution.</p><p>For detections, Snowflake customers may deploy their own queries in SQL, Python or other supported languages and schedule them as tasks or through the trust center. Snowflake provides out of the box monitoring queries (see below), the results then can be exported into a SIEM for further alerting or triage. This approach avoids the need to move the entirety of Snowflake’s audit logs into a SIEM.</p><p>Events may be exported via an external stage as described above or given their minimal bandwidth on a scheduled or triggered basis to a REST api using <a href="https://docs.snowflake.com/en/developer-guide/external-network-access/external-network-access-overview">external access</a> or <a href="https://docs.snowflake.com/en/sql-reference/external-functions">external functions</a> as illustrated in <a href="https://medium.com/@kellerkev/how-to-let-snowflake-raise-security-incidents-in-your-siem-or-xdr-automatically-fb2ee4bea27b">this guide</a>.</p><p>For Splunk customers, Snowflake provides a <a href="https://community.snowflake.com/s/article/Integrating-Snowflake-and-Splunk-with-DBConnect">plugin</a> that allows Splunk to reach into Snowflake through our JDBC driver. Splunk may then be used to view audit logs, read results of detections or import logs on an adhoc basis. This functionality applies to both audit logs as well as other sources and many customers use this architecture to offload other verbose log sources such as EDR and network logs. More information about Splunk Augmentation can be found <a href="https://www.snowflake.com/wp-content/uploads/2024/05/The-CISOs-Guide-to-SIEM-Migration.pdf">here</a>.</p><h4>Exporting</h4><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*j2aW_cPzpxBzWvos" /></figure><p>Exporting logs from Snowflake is the most straightforward way to integrate with a siem and is similar to how a security team typically interacts with a SaaS or IaaS platform. As such it fits into existing processes and will provide a secondary backup of all audit logs into Snowflake. The main tradeoff of this approach is cost, while Snowflake does not charge for access to audit logs, Siem vendors will charge for their ingestion and storage. For active Snowflake organizations these logs can have a high cardinality and may incur large ingest costs. If using this approach therefore it’s critical to choose the logs you need at the latency required.</p><p>Logs in this scenario will generally be transferred via export functionality to an external stage such as S3. From there logs can be ingested into your SIEM. Using the method describe above, Splunk may be configured to reach in to pull security and other raw logs directly from Snowflake as well.</p><h3>Prepared Queries and Detections</h3><p>Snowflake provides several strategies and accelerators for customers looking to monitor their accounts. To review a few</p><h4><strong>Trust Center</strong></h4><p><a href="https://docs.snowflake.com/en/user-guide/trust-center/overview">Trust Center</a> is a native Snowflake feature that automates posture management through scanner packages. Customers may enable scanners directly through trust center and schedule them at times of their choosing. Once enabled, reporting is conducted through the UI or by querying the “snowflake/trust_center/findings” view</p><h4><strong>Sentry</strong></h4><p><a href="https://snowflake-labs.github.io/Sentry/index.html">Sentry</a> is a project maintained by the Security Applied Field Engineering (SAFE) team at Snowflake. It is a framework for monitoring Snowflake, consists of many commonly used monitoring queries and has several different options for deployment such as through Streamlit interface or as stored procedures. Documentation on the project, its queries and its deployment can be found <a href="https://snowflake-labs.github.io/Sentry/">here</a></p><p>By providing a variety of SIEM integration options, customers are able to choose the approach that best meets their needs in terms of latency, cost, and desired functionality. By utilizing Snowflake’s export functionality, federated queries, or the Splunk plugin, customers can effectively monitor Snowflake activity and ensure the security of their data. To learn more about security analytics on Snowflake visit the <a href="https://www.snowflake.com/en/solutions/departments/cybersecurity/">Snowflake for Cybersecurity</a> page.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b29e0f42b1a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/siem-integration-architectures-patterns-9b29e0f42b1a">Snowflake Siem Integration Architectures</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A guide to reporting on developer security with Snyk and Snowflake]]></title>
            <link>https://medium.com/snowflake/a-guide-to-reporting-on-developer-security-with-snyk-and-snowflake-a85aa12a6e16?source=rss-3053a9e10027------2</link>
            <guid isPermaLink="false">https://medium.com/p/a85aa12a6e16</guid>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[snyk]]></category>
            <dc:creator><![CDATA[Jake Berkowsky]]></dc:creator>
            <pubDate>Thu, 12 Sep 2024 14:01:44 GMT</pubDate>
            <atom:updated>2024-09-12T14:01:44.985Z</atom:updated>
            <content:encoded><![CDATA[<p>Everyone wants to shift left to stop vulnerabilities as early in the process as possible. This makes sense, a bug (security or otherwise) stopped before it’s even deployed is not only easier to fix, but is less likely to cause any damage in production.</p><p>The thing is, nothing exists in a vacuum. Large enterprises often have different teams doing vulnerability management. Some are patching servers, some are securing the cloud and others are looking at code. These teams use different tools, have different metrics and even different KPIs. However, by centralizing metrics in a security data lake, enterprises can effectively drill into the impact of their developer security program as well as other downstream and related programs.</p><p><a href="https://www.snowflake.com/webinar/product-demo/snyk-pbs-demo-2023-04-20/">Snyk’s developer security platform uses Snowflake</a> behind the scenes to provide out-of-the-box and accessible reporting for teams to analyze their AppSec program’s effectiveness. For Snowflake customers, Snyk also provides the ability to share this information directly via Snowflake’s native data sharing functionality, making it easy to view the data within the broader context of your entire security portfolio.</p><p>Enterprises and government agencies such as the Centers for Medicare and Medicaid Services (CMS) have long realized the value of <a href="https://www.snowflake.com/en/blog/how-cms-implemented-security-data-lake-strategy/">building security data lakes with Snowflake and Snyk</a>, and now with this integration, it has become even easier to get started.</p><h3>Getting started</h3><p>Go to <a href="https://snyk.co/uhokR">Snyk’s Snowflake documentation page</a> for instructions on how to request a Snowflake data share.</p><p>Snyk Analytics for Snowflake will create several tables in a data share.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/0*3gRvKH-T6k8zLFz1" /><figcaption><em>Tables created in the Snyk Data Share</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xW0sFYlndY6wpzH_" /><figcaption><em>A view of issue data from the Snyk data share</em></figcaption></figure><h3>Taking snapshots</h3><p>The Snyk data share provides an automatically refreshed view of your data, at the time of this writing refreshing every two hours. Customers who want to track historical metrics can configure automatic snapshots of the table. The following configuration sets up snapshots for the issues table, taken once a day using a task in a separate table.</p><pre>-- Create a table<br><br>create table snyk_issues_historical like SNYK_SNOWFLAKE_DATA_SHARE_LISTING.SNYK.ISSUES__V_1_0;<br>ALTER TABLE issues___V_1_0v1_historical ADD COLUMN snapshot_date DATE;<br><br>-- Create task<br><br>CREATE OR REPLACE TASK &lt;TASK_NAME&gt;<br>  WAREHOUSE = &lt;TASK_NAME&gt;<br>  SCHEDULE = &#39;USING CRON 0 8 * * * UTC&#39;<br>  COMMENT = &#39;Task to insert data into snyk_issues_historical table daily&#39;<br>AS<br>INSERT INTO DATABASE.SCHEMA.SNYK_ISSUES_HISTORICAL<br>SELECT *, CURRENT_DATE AS snapshot_date<br>FROM SNYK_SNOWFLAKE_DATA_SHARE_LISTING.SNYK.ISSUES__V_1_0;<br><br>-- Start Task <br><br>ALTER TASK snyk_demo.public.daily_insert_issues__V_1_0_historical RESUME;<br><br>-- Query Historical Data <br>select * from SNYK_DEMO.PUBLIC.ISSUES__V_1_0_HISTORICAL limit 500;</pre><h3>Looking at metrics</h3><p>Analysts can begin querying Snyk data as soon as the data share is configured. For very large datasets, scheduling a summary table can help increase query performance; for most users this is unnecessary, but creating a view can make queries easier to write.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K7O-5ca_VwT8BZ_S" /></figure><pre>-- A table or view can make queries easier to write or more performant when scheduled<br><br>create view issue_summary as<br>select snapshot_date,issue_severity,issue_status,count(*) as issue_count from snyk_issues_historical group by all;<br><br>select * from issue_summary;<br><br>-- Count of open issues with severity of High<br>select issue_count,snapshot_date from issue_summary where issue_severity=&#39;High&#39; and issue_status = &#39;Open&#39;;</pre><p>Outside of raw queries and Snowsight, Streamlit provides the ability to create custom low code data applications and dashboards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_H9j1RM_9S0LYdrp" /></figure><h3>Enrich with other data</h3><p>One of the key benefits of a security data lake is the ability to break down silos by analyzing and joining together data from different sources. For example, Snyk’s data share includes associated JIRA ticket IDs. If that data is in Snowflake, (our marketplace provides connectors for sources like <a href="https://app.snowflake.com/marketplace/data-products/search?search=jira">Jira</a> or <a href="https://app.snowflake.com/marketplace/listing/GZSTZTP0KL1/snowflake-snowflake-connector-for-servicenow?search=service%20now">ServiceNow</a>) it’s relatively easy to incorporate that data into your reporting and join it all together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uwkZSTC7lYwl9fK9" /></figure><h3>Learn more</h3><p>To learn more about building cybersecurity data lake’s on Snowflake, visit Snowflake’s <a href="https://www.snowflake.com/en/solutions/departments/cybersecurity/">cybersecurity portal</a> or contact your account team.</p><p>To learn how to leverage Snyk data through Snowflake Data Sharing, check out <a href="https://snyk.co/uhok2">this blog post</a> on the new integration. <a href="https://snyk.co/uhok2">https://snyk.co/uhok2</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a85aa12a6e16" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/a-guide-to-reporting-on-developer-security-with-snyk-and-snowflake-a85aa12a6e16">A guide to reporting on developer security with Snyk and Snowflake</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing Schema Search on Snowflake]]></title>
            <link>https://medium.com/snowflake/implementing-schema-search-on-snowflake-b2d5ee028b42?source=rss-3053a9e10027------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2d5ee028b42</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Jake Berkowsky]]></dc:creator>
            <pubDate>Tue, 13 Aug 2024 14:07:21 GMT</pubDate>
            <atom:updated>2024-08-13T15:18:26.707Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to search multiple log sources at once</h4><p>Security and observability data lakes have allowed customers to have access to more data than ever before and have given them the tools to search, analyze and report on it.</p><p>In previous generations there was less data, and less to do with it. So a common pattern emerged. When an incident responder, threat hunter or SRE wanted to search for something but they didn’t really know where it lived they had the ability to “search everywhere”. This ability is in theory incredibly useful. Just type in an IP or file hash and get everything.</p><p>As enterprises began collecting massive amounts of data they realized that the technologies they were using became unwieldy, restrictive and expensive. Cloud native scalable technologies like Snowflake have become more sought after and with that new patterns and approaches for querying data. However, no one likes to give up something they already have. The search everywhere paradigm, though long infeasible for anything even coming close to scale was still available and enterprises still want to know that they have that ability.</p><p>This article will provide an architectural reference to how to build this functionality in Snowflake, what tradeoffs to consider and a sample implementation for reference.</p><h3>I didn’t say you couldn’t, I said you shouldn’t</h3><p>The more data an analyst has access to the larger the cost in searching everywhere. One of the limiting factors of legacy architectures is that an everywhere search will be slow and probably time out if the scope is too large. Analysts adapted by a specific style of search, search everywhere, but for a short amount of time, expand until you find what you are looking for and then narrow down to more performant queries. You can think of this as an organic governor function.</p><p>Snowflake is a very powerful engine and so theoretically, you can just through a bunch of compute at a large inefficient search and it will complete in a reasonable amount of time. This means the natural governor that comes as a side effect from legacy architectures is removed. If you assume that analysts are more concerned with doing their job than writing efficient queries then it goes to follow that this pattern may be abused if left unchecked.</p><p>Snowflake already has performant and cost effective methods of querying logs. Log searches are made performant by taking advantage of Snowflake inherent optimizations, <a href="https://docs.snowflake.com/en/user-guide/search-optimization-service#what-access-control-privileges-are-needed-for-the-search-optimization-service"><strong>search optimization</strong></a><strong>, </strong><a href="https://www.snowflake.com/blog/super-fast-top-k-queries/"><strong>top-k </strong></a><strong>and </strong><a href="https://docs.snowflake.com/LIMITEDACCESS/search"><strong>full-text search</strong></a>. Look into these features, they may be all you need and even though it’s a learning curve, it’s not a steep one and queries that return faster for less cost are a benefit to everyone.</p><p>Some advice</p><ul><li>Use this pattern for searching small data sets or things restricted by time.</li><li>For scheduled searches such as detections or reporting, don’t use this approach. This is more useful for ad-hoc queries, jobs that run often are worth the time to write performantly.</li><li>If you’re just not sure where a given field lives, a better solution than searching everywhere is to implement a data governance solution. This could be a specialized tool, or just a spreadsheet. You can also take a look at Snowflake’s <a href="https://docs.snowflake.com/en/user-guide/ui-snowsight-universal-search"><strong>universal search</strong></a> to help find where the data you are looking for lives.</li><li>Implementing a search everywhere function should be done with guardrails and restricted scope. Take advantage of one or more of the suggestions in the implementation section of this article.</li></ul><h3>Implementation</h3><p>The approach this article takes is to build a custom function that can search multiple tables and return the results all at once. Take into account the following</p><h4>Scope of search</h4><p>In reality, it’s almost never a good idea to search literally “everywhere”. One of the main reasons that people build security data lakes is to break down their data silos. Not just between security tools but other enterprise data. Things like HR data, building access logs or internal customer support tickets may be useful in an investigation or to make a security program more effective but provide little value in an IOC Sweep. Different log sources may have different permissions as well, you may have automation that searches through internal employee emails but don’t want to give unrestricted access. Furthermore, as users create new tables, build ETL pipelines and write reports, there may be a fair amount of duplication (either copied or in views).</p><p>Restricting scope can help manage permission issues and ensure you’re only querying the data that is potentially needed. The function described here will restrict its scope to an individual schema and be executed with the rights of the caller. This is less likely to cause permission issues and allows an intuitive implementation <em>network_logs.schema_search() </em>to search network logs for instance.</p><h4>Exposing an interface</h4><p>How end users are going to be accessing the results of this type of search has a significant effect on what types of architectures and features you’ll be able to use.</p><p>The simplest way to create this style search is to generate (dynamically) a long and unwieldy SQL statement, copy paste it into a worksheet and either find and replace or use variables. This is easy to deploy and gives flexibility. The main tradeoff being that it is hard to maintain, new columns will need to be manually added and dropped ones manually removed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ghvch-KzSnlD2J7w" /><figcaption>A long and unwieldy SQL statement</figcaption></figure><p>A streamlit application provides the most flexibility. You can guide your end users to whatever experience you want them to have and add non data related features like highlighting, guardrails and custom formatting logic.</p><p>A stored procedure or function (using Snowpark) is likely the best bet for a lot of use cases and is shown below. Snowpark allows dynamic query building, you can deploy the function (or variants of it) in multiple schemas and it can be accessed in a variety of ways, either from Snowsight or even an ODBC/JDBC connection such as from Splunk using DBConnect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2DPFZ4GsjqJaDweJ" /><figcaption>An implemetation of schema search through a JDBC connection</figcaption></figure><h4>Searching diverse table structures</h4><p>Different tables can have different structures. Depending on your use case there are different ways to handle this. If you keep a uniform table structure with a certain set of tables (such as “raw” tables that have a variant and timestamp only) you can implement a function that unions everything together and returns matches. This pattern is not uncommon and also has the added benefit of allowing full text search and better filtering.</p><p>If this functionality is exposed through an interface such as Streamlit, independent searches can be aggregated together. This will have the added benefit of providing more control over usage, permissions and the end user experience.</p><p>For the sake of simplicity, this article assumes that the end user has tables in a variety of structures and so takes the approach of turning everything into a variant, then returning the log and its associated metadata.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zkhMnYlVCCs-Eyre" /><figcaption>Normalized results from discrete log searches</figcaption></figure><h4>Sanity checks and restricting compute</h4><p>As mentioned above, giving unrestricted access to a function that searches everywhere can be an expensive endeavor. It pays to restrict what can be searched, for instance by enforcing a result limit, maximum timeframe or include other sanity checks, such as not including any filters.</p><p>A result limit may be applied for the entire search or per source. Implementing a results limit for the entire search will allow searches to complete faster, but may only return one source type. Implementing a limit per source, will trade performance for a wider range of results and is best used in tandem with a date range.</p><p>Limiting use to a specific warehouse (possibly through RBAC) can make this more manageable and enable a warehouse size cap, resource monitors and query timeouts.</p><p>You can also restrict functionality. The reference function showcases searching all tables with regular expressions. You may be inclined to limit this to a more efficient operation such as iLike.</p><h4>Config</h4><p>Filtering on what end users are searching is critical to both performance and ensuring relevance of results. The more you know about your data, the more you can be selective on what allowed values for what columns should be searched. At a minimum, this functionality should provide the ability to filter on timestamps and ensure that only the relevant sources are being searched.</p><p>This functionality can be implemented in a few ways. Metadata can be created and stored separately, either in a separate table, a config file or hard coded in the function itself. Accounts with an enterprise or higher plan can take advantage of tagging. Setting config can either be done manually or through a helper tool written in Streamlit or through a script.</p><p>The example below uses a hardcoded config object containing the relevant tables and timestamp columns.</p><h3>An example</h3><p>The following is an example of what a schema_search function may look like. This example:</p><ul><li>Is scoped to a single schema</li><li>Uses a stored procedure as its interface, making it easy to call either from a worksheet or ODBC connection</li><li>Runs as caller to prevent users from escalating privileges or needing to rely on this procedure as the only way to view certain datasets</li><li>Hardcodes configuration</li><li>Does not enforce restrictions but allows for limiting by result count (entire search) and timestamp. In this example guardrails could be enforced at the warehouse level or in the code.</li><li>Uses Rlike to demonstrate regex search and provides functionality for regex parameters like case insensitive search</li></ul><pre>use database YOUR_DATABASE;<br><br>use schema YOUR_SCHEMA;<br><br>create or replace procedure schema_search(<br>        search_regex VARCHAR,<br>        regex_params VARCHAR default &#39;&#39;,<br>        lim int default 0,<br>        min_timestamp VARCHAR default &#39;None&#39;,<br>        max_timestamp VARCHAR default &#39;None&#39;<br><br>    )<br>    returns Table()<br>    language python<br>    runtime_version = 3.11<br>    packages =(&#39;snowflake-snowpark-python&#39;)<br>    handler = &#39;main&#39;<br>    EXECUTE AS CALLER<br>    as &#39;<br>import snowflake.snowpark as snowpark<br>from snowflake.snowpark.types import StructType, StructField, StringType<br>from snowflake.snowpark import Session<br>from snowflake.snowpark.functions import col, to_variant, struct,lit,call_builtin<br><br>config = [<br>{<br>      &quot;table_name&quot;: &quot;database.schema.table&quot;,<br>      &quot;timestamp_column&quot;: &quot;timestamp_column_name&quot;<br>},<br>{<br>      &quot;table_name&quot;: &quot;database.schema.table&quot;,<br>      &quot;timestamp_column&quot;: &quot;timestamp_column_name&quot;<br>}<br>]<br><br>def main(session: snowpark.Session,search_regex,regex_params,lim,min_timestamp,max_timestamp): <br>    if max_timestamp == &quot;None&quot;:<br>          max_timestamp = None<br>    if min_timestamp == &quot;None&quot;:<br>          min_timestamp = None<br>    if lim == 0:<br>        lim = None<br><br>    combined_df = None<br>    for config_item in config:<br>          table_name = config_item[&quot;table_name&quot;]<br>          timestamp_column = config_item[&quot;timestamp_column&quot;]<br>          table = session.table(table_name)<br>          if max_timestamp is not None:<br>                table = table.filter(col(timestamp_column) &lt;= max_timestamp)<br>          if min_timestamp is not None:<br>                table = table.filter(col(timestamp_column) &gt;= min_timestamp)<br>    <br>          # get columns for table<br>          columns = table.columns<br>          filter_chain = None<br>          for column in columns:<br>                if column == timestamp_column:<br>                      continue<br>                if filter_chain is None:<br>                      filter_chain = call_builtin(&quot;rlike&quot;, col(column), search_regex,regex_params )<br>                else:<br>                      filter_chain = filter_chain | (call_builtin(&quot;rlike&quot;, col(column), search_regex, regex_params))<br>          table = table.filter(filter_chain)<br>          df = table.select(to_variant(struct(*table.columns)).alias(&quot;data&quot;),col(timestamp_column).alias(&quot;timestamp&quot;)).with_column(&quot;source&quot;, lit(table_name))<br>          <br>          if combined_df is None:<br>                combined_df = df<br>          else:<br>                combined_df = combined_df.union_all(df)<br>    return combined_df.limit(lim)&#39;;</pre><p>The need to search massive amounts of data, have flexibility in their workloads and break down their data silos has and will continue to drive companies toward implementing security data lakes. Implementing familiar patterns such as an everywhere search can help drive adoption and minimize the inherent disruption associated with change.</p><p>To learn more about building cybersecurity data lake’s on Snowflake. Visit Snowflake’s <a href="https://www.snowflake.com/en/solutions/departments/cybersecurity/">cybersecurity portal</a> or contact your account team.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2d5ee028b42" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/implementing-schema-search-on-snowflake-b2d5ee028b42">Implementing Schema Search on Snowflake</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Normalization in Security Data Lakes]]></title>
            <link>https://medium.com/snowflake/normalization-in-security-data-lakes-d02a679d2f97?source=rss-3053a9e10027------2</link>
            <guid isPermaLink="false">https://medium.com/p/d02a679d2f97</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Jake Berkowsky]]></dc:creator>
            <pubDate>Tue, 06 Aug 2024 16:17:48 GMT</pubDate>
            <atom:updated>2024-10-22T00:32:15.778Z</atom:updated>
            <content:encoded><![CDATA[<p>Early this summer I had the opportunity to present at my favorite conference, FWD:Cloudsec. I presented on the specific topic of data normalization for security data lakes. The conference was full of practitioners, many of whom were in the early stages of building or thinking about a security data lake and so I presented at a technical but general level. The following is a video of my talk. For those who prefer to read, see below for a write up based on my speaker notes.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FQUON6bdTxyU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DQUON6bdTxyU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FQUON6bdTxyU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b1333edf5526d8d8fbe7646594e49e42/href">https://medium.com/media/b1333edf5526d8d8fbe7646594e49e42/href</a></iframe><h3>Understanding the Context: Why Data Normalization Matters</h3><p>Historically, security teams are among the last to embrace cloud technologies, however, with the industry’s shift to the cloud, security teams had no choice but to adapt. The cloud brought with it a massive increase in data volume, and with that came the need to manage and make sense of this data effectively.</p><p>One of the critical tasks in this new landscape is data normalization — getting disparate data sources into a common schema. However, normalization is not always necessary or beneficial in every case.</p><h3>Why Normalize? The Pros and Cons</h3><p>Normalization is often driven by the need to make data interoperable with other tools or systems. For example, Security Information and Event Management (SIEM) systems often require data to be in a specific format to function correctly. Normalization can also simplify querying by standardizing field names, reducing the cognitive load on analysts who no longer need to remember the specific names used by different data sources.</p><p>However, normalization is not without its downsides. For some organizations, the time and computational resources required to normalize data are not justified by the benefits. In cases where data sources are not compared directly, or where the existing formats are already sufficient for the organization’s needs, normalization might be an unnecessary step.</p><h3>The Controversy of Cloud Audit Log Normalization</h3><p>One of the more controversial areas of normalization is with cloud audit logs. AWS Security Lake supports normalization of these logs for AWS, some practitioners like this and some resist this approach. Different cloud platforms — like AWS and GCP — have distinct characteristics, and trying to force a one-size-fits-all normalization can strip away important context. For organizations using only one or two cloud platforms, the return on the effort to normalize may not be worth it, especially if the primary use case is analytics or investigation rather than cross-cloud threat detection.</p><h3>When and Where to Normalize</h3><p>Normalization across all data sources is a myth — naming conventions are hard enough to enforce, let alone maintaining a consistent schema across evolving datasets. The key is knowing when and where to normalize.</p><ul><li><strong>At the Source:</strong> Normalizing data at the source can be efficient, especially if vendors provide logs in a standardized format. However, reliance on vendors to maintain these standards can be risky if they do not keep their offerings up to date.</li><li><strong>In the Pipeline:</strong> Tools like Cribl, Apache NiFi or Fluentd allow for normalization during data transit, which can be more cost-effective. But this approach also has its challenges, especially when dealing with backward compatibility or the need to revert to raw data.</li><li><strong>At the Storage Location:</strong> Normalizing data after it has been stored offers flexibility, allowing organizations to maintain raw data while creating views for compatibility. This approach, however, adds complexity and potential costs due to the need for maintaining multiple data transformation pipelines.</li></ul><h3>Choosing a Standard</h3><p>When it comes to selecting a standard for normalization, don’t overthink it. The choice of standard is less critical than ensuring that the chosen approach fits your specific needs and use cases.</p><ul><li><strong>Don’t Do Anything:</strong> For some organizations, not normalizing at all might be a viable option, especially if they do not require complex querying or cross-data source analysis.</li><li><strong>Build It Yourself:</strong> Many organizations opt to create their own standards, normalizing only the fields they need for their specific operations.</li><li><strong>Pick a Premade Standard:</strong> There are several established standards available, such as the Elastic Common Schema (ECS) and the Open Cybersecurity Schema Framework (OCSF). Each has its strengths, with ECS being widely used and recently donated to the OpenTelemetry project, and OCSF being a newer, community-driven standard with growing support. (<a href="https://medium.com/snowflake/security-data-lakes-normalization-and-ocsf-ec94b70fcdc9">Read my thoughts on OCSF here</a>)</li></ul><h3>Conclusion: There’s No One Answer</h3><p>In closing, there is no on answer for data normalization. The best approach is to be pragmatic — understand the specific needs of your organization, choose the right points in your data pipeline to apply normalization, and be prepared to make trade-offs.</p><p>The cloud has brought new challenges to security professionals, but with thoughtful planning and the right tools, these challenges can be managed effectively, allowing teams to leverage the power of cloud-based data without being overwhelmed by its scale and complexity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gSjoQE-_yxaBQRMH" /><figcaption>Photo by <a href="https://unsplash.com/@kencheungphoto?utm_source=medium&amp;utm_medium=referral">Ken Cheung</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d02a679d2f97" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/normalization-in-security-data-lakes-d02a679d2f97">Normalization in Security Data Lakes</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>