<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by XFIL Security Solutions on Medium]]></title>
        <description><![CDATA[Stories by XFIL Security Solutions on Medium]]></description>
        <link>https://medium.com/@xfilsecuritysolutions?source=rss-6126c8710ac4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ABkS_UMOneZ2guixecDdPQ.png</url>
            <title>Stories by XFIL Security Solutions on Medium</title>
            <link>https://medium.com/@xfilsecuritysolutions?source=rss-6126c8710ac4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 08:16:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@xfilsecuritysolutions/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Threat Hunting Starts Before the Hunt]]></title>
            <link>https://xfilsecuritysolutions.medium.com/threat-hunting-starts-before-the-hunt-6c63f2a58238?source=rss-6126c8710ac4------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c63f2a58238</guid>
            <category><![CDATA[mitre-attack]]></category>
            <category><![CDATA[threat-hunting]]></category>
            <category><![CDATA[blue-team]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[XFIL Security Solutions]]></dc:creator>
            <pubDate>Sat, 18 Jul 2026 01:23:47 GMT</pubDate>
            <atom:updated>2026-07-18T01:23:47.734Z</atom:updated>
            <content:encoded><![CDATA[<p>By Justin Vaicaro (@H3dTr1p) | <a href="https://www.xfilsecurity.io/">xfilsecurity.io</a></p><p>Tags: Threat Hunting · Blue Team· MITRE ATT&amp;CK · Automation · APT Mapping</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AuiE-kJtlbYZD47mEQoPsw.png" /></figure><p><em>The planning phase encompasses deciding which adversaries matter to your organization and mapping the techniques they use. This critical step is what makes a threat hunt </em><strong><em>relevant</em></strong><em>. It is also the phase most teams often skip because it is slow and research intensive. We built the </em><strong><em>APT TTP Mapper</em></strong><em> tool to turn an expertise-intensive process into an automated workflow. This reduces APT-group mapping from </em><strong><em>weeks to hours</em></strong><em> eliminating repetitive manual effort and accelerating the threat hunting campaign planning phase.</em></p><h3>The Critical Phase Often Skipped</h3><p>Ask ten security teams how they run a threat hunt and most will describe the fun part: querying logs, chasing anomalies, pivoting through telemetry. Ask them how they decided <strong>what</strong> to hunt for, and the answers get vague. That gap is the problem. The planning phase, deciding which adversaries matter to your organization and mapping the specific techniques they use, is what makes a hunt <strong>relevant</strong>. By skipping this phase you are hunting on instinct instead of intelligence-driven guidance.</p><p>Planning gets skipped for two honest reasons. First, it is <strong>slow</strong>. Profiling a cluster of APT groups, pulling their techniques from <a href="https://attack.mitre.org/">MITRE ATT&amp;CK</a> and open-source reporting, and reconciling where they overlap can consume days or weeks of an analyst’s time before a single query is written. Second, it is <strong>expertise-intensive</strong>. It assumes the hunter already knows which groups are relevant, where to find authoritative technique data, and how to weigh conflicting vendor reporting. For a junior hunter that is a tall order; for a busy senior hunter it is a tax on time they would rather spend hunting.</p><p>So the phase that most determines whether a hunt succeeds is also the one most likely to be cut. The industry knows this matters. Splunk’s <a href="https://www.splunk.com/en_us/blog/security/peak-threat-hunting-framework.html">PEAK framework</a> (“Prepare, Execute, Act with Knowledge”) puts <strong>Prepare</strong> first, describing it as where hunters “select topics, conduct research, and generally plan out their hunt.” The question is not whether to plan. It is how to plan without burning weeks doing it.</p><p>⚠️ <strong>Skipping planning has a cost:</strong> hunts drift toward whatever data is easy to query rather than whatever threat is most relevant, and hunter hours (your most expensive and scarce resource) get spent chasing low-value leads.</p><p>That bottleneck is what we set out to automate. Our <strong>APT TTP Mapper</strong> tool compresses the group-mapping portion of hunt planning from <strong>weeks into hours</strong>, turning a research marathon into a repeatable, evidence-backed process any hunter can run in a fraction of the time.</p><h3>Tool Primer: What the APT TTP Mapper Tool Actually Does</h3><p>This is a high-level tour, not a tool how-to overview. In short: you give the tool a cluster of up to five APT groups, it maps every technique each group is known to use onto MITRE ATT&amp;CK, then finds where those groups <strong>overlap</strong>. The more groups that share a technique, the higher its priority for your hunt because one detection can counter several adversaries at once. Everything is sourced from authoritative data (MITRE ATT&amp;CK) or explicitly cited public research and reporting.</p><p>The output comes in three forms, each aimed at a different audience.</p><h3>1. Deep-dive Hunt Packages</h3><p>For every technique shared by two or more groups, the tool generates a JSON formatted analysis file. The specific groups involved, the cited source for each mapping, a cloud-versus-on-prem methodology note, and ready-to-adapt <strong>KQL and SPL hunting queries</strong>. This is the bridge from “who should we hunt” to “here is a base query to start from.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BFFOAwndaKiJZmaDfBkxAg.jpeg" /><figcaption>Deep-dive output — overlapping techniques with cited sources and ready-to-adapt KQL/SPL queries.</figcaption></figure><h3>2. An Aggregated ATT&amp;CK Navigator Heatmap</h3><p>The same mapping renders as a <a href="https://mitre-attack.github.io/attack-navigator/">MITRE ATT&amp;CK Navigator</a> layer. Each group gets its own color and a combined heatmap scores every technique by how many groups use it, which adjusts from light amber to bright red so the highest-overlap TTPs are easily identified from the matrix. For a hunter, this is the fastest way to see at a glance where a cluster of adversaries converges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e8yvbDwFZ3ZaGTEB6qmDIA.jpeg" /><figcaption>Aggregated Navigator heatmap — overlap intensity from amber (one group) to red (all five).</figcaption></figure><h3>3. A Leadership Dashboard</h3><p>Finally, an Excel workbook translates the analysis for decision-makers. A master sheet of every technique, one sheet per overlap tier, and a <strong>Leadership Dashboard</strong> with charts showing how techniques distribute across the cluster. It is the artifact you hand a hunt lead or a CISO to justify where the next sprint of effort should go.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VXjKt2uMtuXA26Q7uNAUtA.jpeg" /><figcaption>The Excel Leadership Dashboard — overlap tiers and per-group technique distribution for briefings.</figcaption></figure><p>The underlying framework: the same rigorous mapping serves a junior analyst (starter queries), a hunt lead (the heatmap), and leadership (the dashboard) without anyone re-doing the research.</p><h3>The Numbers: Why Structured Planning Pays Off</h3><p>This is not just a workflow preference, the field is measurably moving toward structure. In the <a href="https://www.sans.org/blog/the-evolution-of-enterprise-threat-hunting-detailed-insights-from-the-sans-2024-survey">SANS 2024 Threat Hunting Survey</a>, the share of organizations with a <strong>formally documented hunting methodology rose from 35% to 51% in a single year</strong>. Teams are moving into structured Threat Hunting Program development and the ones that do are increasingly proving the payoff. Organizations that <strong>measure hunt effectiveness climbed from 34% to 64%</strong> year over year.</p><p>The same survey names the constraint. <strong>50% of organizations still cite a shortage of skilled hunters </strong>as a top obstacle (down from 73%, but still the leading challenge). That is exactly where planning automation earns its keep. When the expensive, research intensive step is automated and repeatable, a smaller or more junior team can produce a senior-grade hunt plan. You are not replacing the hunter, you are removing the weeks of arduous research that stand between them and the actual hunt.</p><p>The technical ROI. If mapping a five-actor cluster takes an experienced analyst the better part of a week and the tool produces the same cited prioritized output in an afternoon, the planning phase stops being the reason a hunt gets postponed or possibly even missed. Multiply that across every threat hunting campaign in a year and the compounding time savings, plus the consistency of always hunting the right techniques, is where the real value lives.</p><p>💡 <strong>The point isn’t speed for its own sake.</strong> Faster planning means planning is now actually taking place for every threat hunting campaign and on every hunt, not just the ones there was time for.</p><h3>Plan First, Hunt Better</h3><p>The threat hunting campaign planning step should not be thought of as an administrative burden towards the overall threat hunting activities, it is the necessary step that decides whether everything downstream of it is relevant. Without it, hunts are degraded before they begin and precious operational hours are spent on the wrong things. Through this automation the planning becomes a repeatable strength instead of a bottleneck you skip.</p><p>The APT TTP Mapper tool rapidly accomplishes the tasks associated with <strong>Phase 1</strong> of the threat hunting campaign building. It tells you which techniques matter most across the adversaries you care about and hands every level of hunter an artifact they can act on. It turns a week of research into an afternoon and it does it against verifiable threat intelligence data.</p><p>If you would like to schedule a threat hunting assessment or try the tool in your environment please contact the XFIL Security Solutions Sales team at <em>sales@xfilsecurity.io</em>.</p><p><strong>Next in this threat hunting automation series</strong> we will explore how to automate the individual threat hunt building itself, which takes the prioritized output from this Phase 1 planning and turning it into fully constructed ready-to-run hunts.</p><h3>Resources &amp; References</h3><p>🔗 <a href="https://attack.mitre.org/">MITRE ATT&amp;CK</a> — The adversary technique knowledge base</p><p>🔗 <a href="https://www.splunk.com/en_us/blog/security/peak-threat-hunting-framework.html">Splunk PEAK Framework</a> — Prepare, Execute, Act with Knowledge</p><p>🔗 <a href="https://www.sans.org/blog/the-evolution-of-enterprise-threat-hunting-detailed-insights-from-the-sans-2024-survey">SANS 2024 Threat Hunting Survey</a> — Field data on hunting maturity</p><p>🔗 <a href="https://www.detectionengineering.net/">Detection Engineering Weekly</a> — Zack Allen — detection &amp; hunting newsletter</p><p>🔗 <a href="https://dispatch.thorcollective.com/">THOR Collective Dispatch</a> — Community threat-hunting insights</p><p>🔗 <a href="https://www.xfilsecurity.io/blog">XFIL Security Blog</a> — More guides like this one</p><p>Originally published at <a href="https://www.xfilsecurity.io/blog">xfilsecurity.io/blog</a>. For threat hunting support, visit <a href="https://www.xfilsecurity.io/threat-hunting">XFIL Security Threat Hunting</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c63f2a58238" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Free Local Splunk Detection Lab]]></title>
            <link>https://xfilsecuritysolutions.medium.com/building-a-free-local-splunk-detection-lab-c4d5200e8d8c?source=rss-6126c8710ac4------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4d5200e8d8c</guid>
            <category><![CDATA[detection-engineering]]></category>
            <category><![CDATA[splunk]]></category>
            <category><![CDATA[mitre-attack]]></category>
            <category><![CDATA[blue-team]]></category>
            <category><![CDATA[incident-response]]></category>
            <dc:creator><![CDATA[XFIL Security Solutions]]></dc:creator>
            <pubDate>Sat, 30 May 2026 01:37:37 GMT</pubDate>
            <atom:updated>2026-05-30T01:37:37.795Z</atom:updated>
            <content:encoded><![CDATA[<p>By Justin Vaicaro (@H3dTr1p) | <a href="https://www.xfilsecurity.io/">www.xfilsecurity.io</a></p><p><em>Tags: Splunk · Detection Engineering · Attack Range · MITRE ATT&amp;CK · Blue Team</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iKgMgBT89YgruHGTudd5Uw.jpeg" /></figure><p><em>One of the biggest challenges for a junior detection engineer is getting hands-on experience without access to an expensive cloud lab. In this guide, we walk through building a free local Splunk detection lab using a local virtual Splunk SIEM instance and open-source attack datasets — on your own hardware, zero cloud costs, using VMware Fusion Pro and Ubuntu.</em></p><h3>Introduction</h3><p>By the end of this post you will be able to:</p><ul><li>Set up a portable local virtual Splunk detection lab using VMware Fusion Pro</li><li>Replay real-world attack telemetry into an existing Splunk instance</li><li>Troubleshoot common HTTP Event Collector (HEC) connectivity issues</li><li>Fix XML field parsing with the right Splunk Technology Add-ons</li><li>Write and validate production-quality detections for T1003.003 (NTDS Credential Dumping)</li></ul><p>Want hands-on guidance beyond this guide? Check out our <a href="https://www.xfilsecurity.io/cybersecurity-training">Cybersecurity Training</a> program or reach out about <a href="https://www.xfilsecurity.io/purple-teaming">Purple Teaming</a> engagements where we bring detection engineering into your real environment.</p><h3>What is the Splunk Attack Range?</h3><p>The <a href="https://github.com/splunk/attack_range">Splunk Attack Range</a> is an open-source project maintained by Splunk’s Threat Research Team. It provisions a fully configured lab environment containing:</p><ul><li>A Splunk Enterprise instance pre-wired for detection</li><li>Windows / Linux victim machines with Sysmon and Universal Forwarders</li><li>An optional Kali Linux attacker machine</li><li>Integration with Atomic Red Team and CALDERA for attack simulation</li></ul><p>The good news: you do not need AWS or Azure. With some creativity you can create your Attack Range completely local simply using virtualization software. This maps directly to the kind of <a href="https://www.xfilsecurity.io/threat-hunting">threat hunting</a> and <a href="https://www.xfilsecurity.io/red-team-attack-simulation">red team attack simulation</a> capabilities XFIL Security uses in production engagements.</p><p>While the complete Splunk Attack Range supports the full build-out of victim machines and attacker machines, this guide focuses exclusively on two components: standing up the Splunk Enterprise SIEM instance and leveraging the Splunk Attack Data repository to replay pre-recorded attack telemetry directly into it. This streamlined approach gets you writing and validating detections faster, without the overhead of provisioning a full multi-machine lab environment.</p><h3>Part 1 — Setting Up the Local Portable Lab (Using VMware Fusion Pro)</h3><h3>Prerequisites</h3><p>This lab was built and validated on the following host environment. macOS and Windows hosts are both supported — the VMware Fusion Pro + Ubuntu 24.04 approach works on macOS, while Windows users can substitute VMware Workstation Pro (free for personal use) to achieve an identical setup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*936BOS35_D4qb-d4aa9Aiw.jpeg" /></figure><blockquote>VMware Fusion Pro &amp; Workstation Pro are now free for personal use. Download Fusion Pro at vmware.com — no license key required for non-commercial use.</blockquote><h3>Splunk License — Dev License vs. Free License</h3><p>This lab uses a Splunk Enterprise Developer License, which provides 10 GB/day of data ingestion. This is a significant advantage over the standard free license:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/1*TgswJgSMIhnY5hNuaFQmaQ.jpeg" /></figure><p>The Developer License is free and available to anyone who registers a <a href="https://dev.splunk.com/">Splunk Developer account</a>. With 10 GB/day you can replay large attack datasets, ingest multiple data sources simultaneously, and run this lab at full scale without hitting ingestion caps.</p><h3>Step 1 — Install VMware Fusion Pro and Create Your Ubuntu VM</h3><p>Download VMware Fusion Pro (macOS) or VMware Workstation Pro (Windows):</p><ul><li>macOS: <a href="https://www.vmware.com/products/desktop-hypervisor/workstation-and-fusion">vmware.com/products/desktop-hypervisor</a></li><li>Windows: Same link — select Workstation Pro</li></ul><p>Create a new Ubuntu 24.04 Server VM with these settings:</p><p>VM Settings</p><pre>RAM:   16 GB (allocate half your host RAM)<br>CPU:   4 cores<br>Disk:  100 GB (thin provisioned)<br>OS:    Ubuntu 24.04 Server (minimal install)</pre><blockquote>Important: Use the Server ISO, not Desktop. The server edition is leaner, has no GUI overhead, and is better suited for running Splunk as a service.</blockquote><p>Download Ubuntu 24.04 Server ISO: <a href="https://ubuntu.com/download/server">ubuntu.com/download/server</a></p><h3>Step 2 — Install Splunk Enterprise on Ubuntu</h3><p>SSH into your Ubuntu VM, then run:</p><p>Ubuntu VM</p><pre># Download Splunk Enterprise (get the latest .deb from splunk.com)<br>wget -O splunk.deb &#39;https://download.splunk.com/products/splunk/releases/9.x.x/linux/splunk-9.x.x-linux-2.6-amd64.deb&#39;</pre><pre># Install<br>sudo dpkg -i splunk.deb</pre><pre># Start Splunk and accept license<br>sudo /opt/splunk/bin/splunk start --accept-license</pre><pre># Enable Splunk to start at boot<br>sudo /opt/splunk/bin/splunk enable boot-start</pre><p>Access Splunk at http://&lt;your-vm-ip&gt;:8000 from your Mac.</p><h3>Step 3 — Apply Your Splunk Developer License</h3><ol><li>Log in to Splunk Web → Settings → Licensing</li><li>Click Add License</li><li>Upload your .lic file downloaded from <a href="https://dev.splunk.com/">dev.splunk.com</a></li><li>Restart Splunk:</li></ol><p>Ubuntu VM</p><pre>sudo /opt/splunk/bin/splunk restart</pre><p>You should now see 10 GB/day capacity in the licensing dashboard.</p><h3>Step 4 — Clone the Attack Data Repository</h3><p>Splunk maintains a dedicated <a href="https://github.com/splunk/attack_data">attack_data repository</a> with pre-recorded JSONL/log files from real attack simulations — organized by MITRE technique.</p><blockquote>Note: The following steps are performed directly on the Ubuntu server where your Splunk instance is installed . SSH into that machine before proceeding.</blockquote><p>Ubuntu VM</p><pre># Install Git LFS first — critical for pulling real attack datasets<br>sudo apt install git-lfs python3-pip -y<br>git lfs install</pre><pre># Clone the Splunk Attack Data repository<br>git clone <a href="https://github.com/splunk/attack_data">https://github.com/splunk/attack_data</a> ~/attack_data</pre><pre># Pull the actual dataset files via Git LFS (not just pointer files)<br>cd ~/attack_data<br>git lfs pull</pre><blockquote>Do not skip git lfs pull . Without it, the dataset files will be stub pointer files (~130 bytes) rather than the real log data. See Part 2 — Issue 4 for how to identify and fix this.</blockquote><p>Install the Python dependencies required by replay.py:</p><p>Ubuntu VM</p><pre>pip3 install requests</pre><h3>Step 5 — Set Persistent Environment Variables for Splunk HEC</h3><p>Rather than using a config file, this lab uses persistent shell environment variables to pass the Splunk host and HEC token to replay.py. Setting these once in ~/.bashrc means every future replay command just works — no config file to maintain or accidentally commit to version control.</p><p>First, create your HEC token in Splunk if you haven’t already:</p><ol><li>Splunk Web → Settings → Data Inputs → HTTP Event Collector</li><li>Global Settings → Enable All Tokens → Save</li><li>New Token → name it attack_data_replay → set default index to attack_data</li><li>Copy the generated token (format: a1b2c3d4-1234-5678-abcd-ef1234567890)</li></ol><p>Then set the environment variables permanently:</p><p>Ubuntu VM — ~/.bashrc</p><pre># Add to ~/.bashrc for persistence across sessions<br>echo &#39;export SPLUNK_HOST=&quot;192.168.x.x&quot;&#39; &gt;&gt; ~/.bashrc        # IP of your Splunk Ubuntu VM<br>echo &#39;export SPLUNK_HEC_TOKEN=&quot;your-hec-token-here&quot;&#39; &gt;&gt; ~/.bashrc</pre><pre># Reload to apply immediately<br>source ~/.bashrc</pre><p>Verify both variables are set:</p><p>Ubuntu VM</p><pre>echo &quot;Host:  $SPLUNK_HOST&quot;<br>echo &quot;Token: $SPLUNK_HEC_TOKEN&quot;</pre><p>Verify HEC is reachable:</p><p>Ubuntu VM</p><pre>curl -k https://&lt;SPLUNK_HOST IP&gt;:8088/services/collector/health<br># Expected: {&quot;text&quot;:&quot;HEC is healthy&quot;,&quot;code&quot;:17}</pre><h3>Step 6 — Locate replay.py in the Attack Data Repository</h3><p>The replay.py script used in this lab is the one bundled inside the attack_data repository itself — not the Attack Range.</p><p>You will find the script in the following path:</p><p>Script Location</p><pre>&lt;download_directory&gt;/attack_data/bin/replay.py</pre><p>Confirm it’s there and check the available arguments:</p><p>Ubuntu VM</p><pre>ls &lt;download_directory&gt;/attack_data/bin/<br>python3 &lt;download_directory&gt;/attack_data/bin/replay.py --help</pre><p>This version of replay.py is designed specifically to work with the attack datasets in that same repo and reads connection details directly from environment variables — no config file needed.</p><h3>Step 7 — Run the Replay</h3><blockquote>Pro Tip: If you already have a Splunk instance running (like Marvel Labs ), you can skip the full Attack Range build entirely and just replay pre-recorded attack datasets directly. This is the fastest path to writing and testing detections.</blockquote><p>With your environment variables set and replay.py located, run the replay:</p><p>Ubuntu VM — T1003.003 Replay</p><pre>python3 (download directory)/attack_data/bin/replay.py (download directory)/attack_data/datasets/attack_techniques/T1003.003/</pre><blockquote>Tip: Because SPLUNK_HOST and SPLUNK_HEC_TOKEN are set in ~/.bashrc , you can reuse this exact command for every future dataset replay — just swap out the — dataset path for a different MITRE technique folder.</blockquote><p>To replay a different technique, simply point --dataset at another folder:</p><p>Ubuntu VM — Different Technique</p><pre># Example: replay a different technique<br>python3 (download directory)/attack_data/bin/replay.py (download directory)/attack_data/datasets/attack_techniques/T1059.001/</pre><h3>Step 8 — Verify the Replay Output and Confirm Data in Splunk</h3><p>After the replay completes, verify the execution was successful by checking the terminal output. You should see log lines confirming events were sent to the HEC endpoint without errors, similar to the output shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kvO3scOcUKuyh5gQ.jpg" /></figure><p>Next, log in to Splunk Web and confirm the attack data has been successfully ingested. Navigate to the Search &amp; Reporting app and run a quick search against the attack_data index to validate events are present:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nVIMq3YU0BZB2DQi.jpg" /></figure><blockquote>Note: The ingested logs will be fully normalized — with proper field extraction — if the Splunk Add-on for Windows and the Splunk Add-on for Sysmon are installed in your environment. Without these Add-ons, events will arrive as raw unparsed XML. See Part 2 — Issue 5 for installation instructions.</blockquote><h3>Part 2 — Troubleshooting Common Issues</h3><h3>Issue 1 — SPLUNK_HEC_TOKEN is required but not set</h3><p>The replay script uses Splunk’s HTTP Event Collector (HEC) to send data. You need to create a token first.</p><p>Create a HEC Token in Splunk:</p><ol><li>Settings → Data Inputs → HTTP Event Collector</li><li>Global Settings → Enable All Tokens → Save</li><li>New Token → name it attack_data_replay → set default index to attack_data</li><li>Copy the generated token (format: a1b2c3d4-1234-5678-abcd-ef1234567890)</li></ol><p>Set the environment variable:</p><p>Ubuntu VM</p><pre># Temporary (current session)<br>export SPLUNK_HEC_TOKEN=&quot;your-token-here&quot;</pre><pre># Permanent<br>echo &#39;export SPLUNK_HEC_TOKEN=&quot;your-token-here&quot;&#39; &gt;&gt; ~/.bashrc<br>source ~/.bashrc</pre><p>Verify HEC is healthy:</p><p>Ubuntu VM</p><pre>curl -k https://$SPLUNK_HOST:8088/services/collector/health<br># Expected: {&quot;text&quot;:&quot;HEC is healthy&quot;,&quot;code&quot;:17}</pre><h3>Issue 2 — [Errno 111] Connection Refused on Port 8088</h3><p>This means HEC is either disabled or not listening. Work through this checklist:</p><p>Ubuntu VM</p><pre># 1. Confirm Splunk is running<br>sudo /opt/splunk/bin/splunk status</pre><pre># 2. Check if anything is listening on 8088<br>sudo ss -tlnp | grep 8088</pre><pre># 3. Test TCP connectivity from replay machine<br>nc -zv $SPLUNK_HOST 8088</pre><pre># 4. Open firewall if needed (Ubuntu UFW)<br>sudo ufw allow 8088/tcp</pre><h3>Issue 3 — Splunk Not Listening on the Right IP</h3><p>By default Splunk may bind to 127.0.0.1 (loopback only). To bind HEC to your network IP, edit inputs.conf:</p><blockquote>Always edit the file under system/local/ — never system/default/ as that gets overwritten on upgrades.</blockquote><p>Ubuntu VM</p><pre># Find all local config files<br>find / -path &quot;*/splunk/etc/system/local/*.conf&quot; 2&gt;/dev/null</pre><pre># Edit inputs.conf<br>sudo nano /opt/splunk/bin/../etc/system/local/inputs.conf</pre><p>inputs.conf</p><pre>[http]<br>port = 8088<br>host = 192.168.1.198     # bind to your specific IP</pre><p>web.conf</p><pre>[settings]<br>server.socket_host = 192.168.1.198<br>mgmtHostPort = 192.168.1.198:8089</pre><p>Ubuntu VM — Verify</p><pre>sudo /opt/splunk/bin/splunk restart<br>sudo ss -tlnp | grep splunk<br># Should now show 192.168.1.198:8088</pre><h3>Issue 4 — Git LFS Pointer Files Replayed Instead of Real Data</h3><p>Symptom: Events in Splunk show content like this instead of real logs:</p><p>Splunk Event (Bad)</p><pre>version https://git-lfs.github.com/spec/v1<br>oid sha256:ee1c7cd9fa20013c82da17f8cb1998197c...<br>size 12883992</pre><p>Cause: The attack_data repo uses Git LFS for large log files. If LFS isn&#39;t installed before cloning, you only get pointer files (~130 bytes) instead of the actual data (several MBs).</p><p>Fix:</p><p>Ubuntu VM</p><pre># Install Git LFS<br>sudo apt install git-lfs<br>git lfs install</pre><pre># Pull the real files<br>cd ~/attack_data<br>git lfs pull</pre><pre># Verify file size (should be MBs, not bytes)<br>ls -lh datasets/attack_techniques/T1003.003/atomic_red_team/windows-sysmon.log</pre><p>Clear the bad data from Splunk and re-replay:</p><p>Ubuntu VM</p><pre>/opt/splunk/bin/splunk clean eventdata -index attack_data</pre><pre>python3 (download directory)/attack_data/bin/replay.py (download directory)/attack_data/datasets/attack_techniques/T1003.003/</pre><h3>Issue 5 — Events Arrive as Unparsed Raw XML</h3><p>Symptom: Events show raw XML in Splunk. Cause: Splunk is missing the Technology Add-ons (TAs) needed to extract fields from Windows Event Log and Sysmon XML.</p><p>Fix — Install the Required TAs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*7-S7nsoJD1zH0yu-_3Ufog.jpeg" /></figure><p>Ubuntu VM — Manual TA Install</p><pre>cp splunk-add-on-for-sysmon_*.tgz /opt/splunk/etc/apps/<br>cd /opt/splunk/etc/apps/<br>tar -xzf splunk-add-on-for-sysmon_*.tgz<br>sudo /opt/splunk/bin/splunk restart</pre><p>After installing TAs, clear the index and re-replay so events get parsed on fresh ingest. Then verify field extraction:</p><p>Splunk Search — Verify Fields</p><pre>index=attack_data sourcetype=XmlWinEventLog earliest=-24h<br>| fieldsummary<br>| where count &gt; 0<br>| table field, count, distinct_count<br>| sort -count</pre><h3>Part 3 — Writing Splunk Detections for T1003.003</h3><h3>What is T1003.003 — NTDS Credential Dumping?</h3><p>NTDS credential dumping is a technique where attackers extract the NTDS.dit file from a Windows Domain Controller. This file contains the password hashes for every account in Active Directory. Combined with the SYSTEM registry hive, an attacker can decrypt and crack every password in the domain offline.</p><p>The most common methods involve abusing Volume Shadow Copies to bypass the OS file lock on NTDS.dit:</p><ol><li>Create a shadow copy of the C: drive</li><li>Use copy, xcopy, or robocopy to copy NTDS.dit from the shadow copy</li><li>Or create a mklink symlink to the shadow copy directory</li></ol><p>Your replayed dataset contains two data sources covering this technique:</p><ul><li>Sysmon EventID 1 — Process creation with full CommandLine (via XmlWinEventLog:Microsoft-Windows-Sysmon/Operational)</li><li>Windows Security EventID 4688 — Process creation (via XmlWinEventLog:Security)</li></ul><h3>Credential Dumping via Copy from Shadow Copy</h3><p>Detection Query</p><pre>index=attack_data sourcetype=XmlWinEventLog<br>| spath input=_raw<br>| search EventID IN (&quot;1&quot;, &quot;4688&quot;)<br>| eval CommandLine = coalesce(CommandLine, ParentCommandLine)<br>| where match(CommandLine, &quot;(?i)(\system32\config\(sam|security|system)|\windows\ntds\ntds\.dit)&quot;)<br>| where match(CommandLine, &quot;(?i)(copy|xcopy|robocopy|esentutl)&quot;)<br>| table _time, Computer, User, CommandLine, ParentCommandLine, EventID<br>| sort -_time</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eW4NDqgmmGcb8oVt.jpg" /></figure><h3>Symlink Creation to Shadow Copy</h3><p>Detection Query</p><pre>index=attack_data sourcetype=XmlWinEventLog<br>| spath input=_raw<br>| search EventID IN (&quot;1&quot;, &quot;4688&quot;)<br>| eval CommandLine = coalesce(CommandLine, ParentCommandLine)<br>| where match(CommandLine, &quot;(?i)mklink&quot;)<br>| where match(CommandLine, &quot;(?i)HarddiskVolumeShadowCopy&quot;)<br>| table _time, Computer, User, CommandLine, ParentCommandLine, EventID<br>| sort -_time</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3loV5yjDeVEkcCUy.jpg" /></figure><h3>Shadow Copy Creation (Precursor Activity)</h3><p>Detection Query</p><pre>index=attack_data sourcetype=XmlWinEventLog<br>| spath input=_raw<br>| search EventID IN (&quot;1&quot;, &quot;4688&quot;)<br>| eval CommandLine = coalesce(CommandLine, ParentCommandLine)<br>| where match(CommandLine, &quot;(?i)(vssadmin|wmic|powershell)&quot;)<br>| where match(CommandLine, &quot;(?i)(shadow|shadowcopy|create)&quot;)<br>| table _time, Computer, User, CommandLine, ParentCommandLine, EventID<br>| sort -_time</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mTIX5wr2Y1zeJss-.jpg" /></figure><h3>Broad Hunt Query Across All Sources</h3><p>Hunt Query</p><pre>index=attack_data<br>| spath input=_raw<br>| eval CommandLine = coalesce(CommandLine, CmdLine, CommandHistory, ParentCommandLine)<br>| eval Process = coalesce(Image, NewProcessName, ParentBaseFileName)<br>| search<br>    CommandLine IN (&quot;*ntds.dit*&quot;, &quot;*HarddiskVolumeShadowCopy*&quot;, &quot;*\config\sam*&quot;,<br>                    &quot;*\config\system*&quot;, &quot;*\config\security*&quot;)<br>    OR (CommandLine=&quot;*vssadmin*&quot; AND CommandLine=&quot;*shadow*&quot;)<br>    OR (CommandLine=&quot;*mklink*&quot; AND CommandLine=&quot;*shadow*&quot;)<br>| eval technique = &quot;T1003.003 - NTDS Credential Dumping&quot;<br>| table _time, sourcetype, host, Process, CommandLine, technique<br>| sort -_time</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*q17dCb6B9E9Dgix4.jpg" /></figure><h3>Detection Fidelity Summary</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*gw9gLD_5Xx_Tfef6QdZr2g.jpeg" /></figure><h3>Summary</h3><p>As this guide demonstrates, you don’t need prior lab-building experience or an expensive cloud environment to get started with detection engineering. With a few free tools and a couple of hours, you can have a fully functional Splunk detection lab up and running on your own hardware. Beyond this initial setup, the lab is highly scalable and modular — you can replay any of the hundreds of techniques available in the Splunk Attack Data repository, extend the environment with additional data sources, or layer in new tooling as your skills grow. Whether you’re just starting out in the blue team space or looking to sharpen your threat hunting instincts, this kind of hands-on practice against real adversary telemetry is one of the most effective ways to level up your detection engineering craft.</p><p>Here is the full end-to-end workflow covered in this guide:</p><p>End-to-End Steps</p><pre> 1. Install VMware Fusion Pro (macOS) or Workstation Pro (Windows)<br> 2. Create Ubuntu 24.04 Server VM<br> 3. Install Splunk Enterprise + apply Developer License (10 GB/day)<br> 4. SSH into Ubuntu VM — clone attack_data repo + run git lfs pull<br> 5. Set persistent SPLUNK_HOST and SPLUNK_HEC_TOKEN env variables in ~/.bashrc<br> 6. Enable HEC in Splunk → create token → verify connectivity<br> 7. Configure Splunk to listen on correct IP via inputs.conf (if needed)<br> 8. Locate replay.py at (download directory)/attack_data/bin/replay.py<br> 9. Install Sysmon + Windows TAs for field parsing<br>10. Replay T1003.003 dataset using replay.py + env variables<br>11. Run detections and validate they fire</pre><p>Want to take these skills further? XFIL Security offers <a href="https://www.xfilsecurity.io/cybersecurity-training">hands-on detection engineering training</a> as well as <a href="https://www.xfilsecurity.io/purple-teaming">purple team engagements</a> where we validate your detections against real adversary techniques. <a href="https://www.xfilsecurity.io/contact">Get in touch</a> to learn more.</p><h3>Resources &amp; References</h3><p>🔗 <a href="https://github.com/splunk/attack_range">Splunk Attack Range</a> — GitHub — full lab environment</p><p>🔗 <a href="https://github.com/splunk/attack_data">Splunk Attack Data</a> — GitHub — pre-recorded datasets</p><p>🔗 <a href="https://attack.mitre.org/techniques/T1003/003/">MITRE ATT&amp;CK T1003.003</a> — NTDS Credential Dumping</p><p>🔗 <a href="https://splunkbase.splunk.com/app/5709">Splunk Add-on for Sysmon</a> — Splunkbase — app/5709</p><p>🔗 <a href="https://splunkbase.splunk.com/app/742">Splunk Add-on for Windows</a> — Splunkbase — app/742</p><p>🔗 <a href="https://github.com/redcanaryco/atomic-red-team/blob/master/atomics/T1003.003/T1003.003.md">Atomic Red Team T1003.003</a> — Red Canary — test cases</p><p>🔗 <a href="https://www.xfilsecurity.io/cybersecurity-training">XFIL Security Training</a> — Hands-on detection engineering</p><p>🔗 <a href="https://www.xfilsecurity.io/purple-teaming">XFIL Security Purple Teaming</a> — Validate detections in your environment</p><p>Originally published at <a href="https://www.xfilsecurity.io/blog">www.xfilsecurity.io/blog</a>. For training, visit <a href="https://www.xfilsecurity.io/cybersecurity-training">XFIL Security Training</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4d5200e8d8c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>