<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rakesh Mukherjee on Medium]]></title>
        <description><![CDATA[Stories by Rakesh Mukherjee on Medium]]></description>
        <link>https://medium.com/@rakesh.mkhrj?source=rss-b30089a2ab01------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*2uEcLJlkMUE8D7r_VTkn6w.png</url>
            <title>Stories by Rakesh Mukherjee on Medium</title>
            <link>https://medium.com/@rakesh.mkhrj?source=rss-b30089a2ab01------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 01:46:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rakesh.mkhrj/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Querying Microsoft Defender’s Hidden Flight Recorder with osquery]]></title>
            <link>https://medium.com/@rakesh.mkhrj/querying-microsoft-defenders-hidden-flight-recorder-with-osquery-aa2bbd4aa8ee?source=rss-b30089a2ab01------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa2bbd4aa8ee</guid>
            <category><![CDATA[microsoft-defender]]></category>
            <category><![CDATA[incident-response]]></category>
            <category><![CDATA[osquery]]></category>
            <category><![CDATA[threat-hunting]]></category>
            <category><![CDATA[dfir]]></category>
            <dc:creator><![CDATA[Rakesh Mukherjee]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 11:16:05 GMT</pubDate>
            <atom:updated>2026-11-16T11:16:05.559Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Turning Defender’s encrypted </em><em>mpenginedb.db into SQL tables you can hunt across a fleet, building on Protokol-7&#39;s research.</em></p><p>Every Windows machine running Microsoft Defender Antivirus keeps a small, quiet record of what it has been watching. It isn’t the event log, and it isn’t Defender’s detection history. It’s a SQLite database called mpenginedb.db, which Defender internally calls &quot;MetaVault&quot;:</p><pre>C:\ProgramData\Microsoft\Windows Defender\Scans\mpenginedb.db<br>C:\ProgramData\Microsoft\Windows Defender\Scans\mpenginedb.db-wal</pre><p>Inside it is the kind of context that investigators usually have to reconstruct by hand:</p><ul><li><strong>Per-process attack scoring.</strong> Defender tracks which MITRE ATT&amp;CK techniques it has attributed to each process (by PID and process start time), for example T1547.001 against a specific rundll32.exe instance.</li><li><strong>Path tags set by signatures.</strong> These include NewPECreatedNoCert, ExecutedPENoCert, NewlyCreatedZip and BM_IsPotentialSideLoad: Defender&#39;s own notes about unsigned executables being dropped and run, fresh archives, and likely DLL side-loading.</li><li><strong>A hash cache.</strong> SHA-256 hashes (plus MD5, SHA-1 and ssdeep) tied to NTFS MFT entry numbers, which can include files that no longer exist on disk.</li><li><strong>Ransomware rollback tracking.</strong> Files and processes Defender is watching, with NextUSN values you can correlate with the $UsnJrnl.</li><li><strong>ASR blocks, recent behaviour detections, tamper-protection state, Controlled Folder Access paths</strong> and a <strong>software inventory</strong> with CPE and package-URL identifiers matched to CVEs.</li></ul><p>It holds roughly two weeks of history, and some parts (such as ASR block history) go back further.</p><h3>Credits: Protokol-7’s research</h3><p>None of what follows would exist without the work of <strong>Protokol-7</strong>. Their write-up, <a href="https://protokol-7.github.io/dfir/2026/11/10/defenders-hidden-flight-recorder-mpenginedb.html"><strong>Defender’s Hidden Flight Recorder: mpenginedb.db</strong></a>, is the reason this database is usable at all.</p><p>mpenginedb.db is a SQLite file, but standard tools refuse to open it because every page is encrypted. Protokol-7 showed that:</p><ul><li>Defender encrypts each page with <strong>RC4 using a fixed 256-byte key</strong> stored in mpengine.dll. The &quot;empty key&quot; passed to SQLite&#39;s codec hook is a decoy.</li><li>RC4 restarts for every page, so <strong>every page uses the same keystream</strong>, with no nonce and no integrity protection. Page 1 even leaves bytes 16–23 in the clear.</li><li>Because unused space in SQLite pages is zero-filled, <strong>the keystream can be rebuilt from the evidence itself</strong>, without the DLL.</li><li>The WAL’s checksums are computed over the <em>encrypted</em> bytes, so decryption has to happen before SQLite validates frames.</li></ul><p>They also released excellent tooling (MIT licensed), which I’d recommend to anyone doing offline forensics:</p><ul><li><a href="https://github.com/Protokol-7/mpenginedb_parser"><strong>mpenginedb_parser</strong></a>: a standard-library Python parser that decrypts the database, replays the WAL, recovers deleted rows and produces a full timeline.</li><li><a href="https://github.com/Protokol-7/MpEngineDb_SQLECmd"><strong>MpEngineDb_SQLECmd</strong></a>: a PowerShell decryptor and a SQLECmd map.</li></ul><p>The RC4 key, the keystream-derivation approach and the ASR rule list used in my extension all come from their work, and their MIT notice ships with it. If you work in DFIR, read their post first. This article only makes sense on top of it.</p><h3>The gap: from one disk image to a whole fleet</h3><p>Protokol-7’s tools are built for the forensic workflow: you collect the Scans folder from a machine, then parse it on an analysis box. That&#39;s exactly right for deep-dive investigations, deleted-row recovery and court-ready output.</p><p>But a lot of security work starts with a different question:</p><blockquote><em>“On which of our 5,000 endpoints did Defender attribute T1003.001 to a process in the last hour?”</em></blockquote><blockquote><em>“Show me every machine where an unsigned PE was created and executed from a user’s Downloads folder.”</em></blockquote><p>That’s a fleet question, and in many environments the answer to fleet questions is <a href="https://osquery.io/"><strong>osquery</strong></a>: SQL over endpoint state, deployed through tools like Fleet, osctrl or Kolide. So I built an osquery extension that exposes MetaVault as ordinary tables.</p><p><a href="https://github.com/rakeshrm/osquery-defender-extension"><strong>osquery-defender-extension</strong></a> (MIT, written in Go)</p><h3>What it looks like</h3><p>Load the extension and Defender’s database becomes a set of defender_mpengine_* tables:</p><pre>-- Which techniques has Defender attributed to which processes?<br>SELECT pid, datetime(process_start, &#39;unixepoch&#39;) AS started,<br>       technique_id, queue_type<br>FROM defender_mpengine_rolling_queues<br>WHERE queue_type IN (&#39;techniques&#39;, &#39;inherited_techniques&#39;);</pre><pre>-- Unsigned executables created/executed, archives, side-loading candidates<br>SELECT context, file_path, datetime(insert_time, &#39;unixepoch&#39;) AS tagged<br>FROM defender_mpengine_persist_context<br>WHERE context IN (&#39;NewPECreatedNoCert&#39;, &#39;ExecutedPENoCert&#39;,<br>                  &#39;NewlyCreatedZip&#39;, &#39;BM_IsPotentialSideLoad&#39;);</pre><pre>-- ASR blocks with human-readable rule names<br>SELECT datetime(time, &#39;unixepoch&#39;) AS blocked, process_path, target_path,<br>       asr_rule_name, is_audit<br>FROM defender_mpengine_process_block_history;</pre><pre>-- Hashes Defender saw for files that may already be gone<br>SELECT sha256, md5, sha1, mft_entry, volume_serial,<br>       datetime(instance_time, &#39;unixepoch&#39;) AS seen<br>FROM defender_mpengine_file_hashes;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*HGKdEEQ0V_QOsVlXInoyTQ.png" /><figcaption>osqueryi on Windows Server 2022, reading Defender’s live database: the built-in key matched, the WAL replayed cleanly, and Defender’s path tags come back as rows.</figcaption></figure><p>There are 17 tables in total. Fourteen are typed tables (rolling queues, path tags, hash cache, ransomware rollback, LowFi cloud lookups, behaviour-monitored processes with command lines, firewall rules created by network protection, atomic counters, Controlled Folder Access paths, AMSI cache, schema history and software inventory). The other three are:</p><ul><li>defender_mpengine_status: decryption health (which key matched, WAL state, quick_check, errors).</li><li>defender_mpengine_tables: every table in the database, with row counts and schema.</li><li>defender_mpengine_raw: any table, cell by cell, for everything the typed tables don&#39;t cover yet.</li></ul><p>The decoding details from Protokol-7’s research are handled for you. FILETIMEs become Unix timestamps (the raw value is kept). GUID blobs are decoded with Windows byte order. NTFS file IDs are split into MFT entry and sequence. Defender’s schema quirk of storing <strong>MD5 in the column named </strong><strong>SHA1 and SHA-1 in the column named </strong><strong>MD5</strong> is corrected.</p><h3>Offline evidence works too</h3><p>Every table has a path column, so the same SQL works on a collected copy:</p><pre>SELECT * FROM defender_mpengine_rolling_queues<br>WHERE path = &#39;D:\case\host42\Scans&#39;;</pre><p>And if you need a plain SQLite file for other tools (including Protokol-7’s SQLECmd map), the binary has a CLI mode:</p><pre>defender-mpengine.ext.exe --decrypt D:\case\host42\Scans --out D:\case\host42\mpengine_plain.db</pre><h3>Who it’s for</h3><p><strong>Threat hunters.</strong> Defender has already done a lot of behavioural reasoning on every endpoint. This lets you query its conclusions (techniques per process, side-load tags, recent behaviour detections) across the fleet with the same SQL you use for everything else in osquery, and schedule it as a pack.</p><p><strong>Incident responders.</strong> On a single host you get a timeline of what Defender noticed, hashes of files that were deleted, and NextUSN values to pivot into the USN journal, without pulling a disk image first. When you do need the deep dive (deleted-row carving, per-commit WAL history), hand the same evidence to mpenginedb_parser.</p><p><strong>Detection engineers.</strong> Run ASR rules in audit mode and query what they <em>would</em> have blocked, with rule names attached. Check tamper-protection state history. Look at what the Defender inventory thinks is vulnerable.</p><p><strong>Security researchers.</strong> defender_mpengine_tables and defender_mpengine_raw make it easy to explore the schema as Defender evolves. For example, I found that the software inventory uses <strong>two different layouts on different hosts, even on the same engine version</strong>: MetaInventoryIdentifiers on one, a newer InventoryEntries table on another. The extension now reads both.</p><h3>How it works</h3><ul><li><strong>Decrypt.</strong> Read the database and its WAL. Generate the RC4 keystream from the known key and check it against page 1. Decrypt every page, including the page-1 special case Protokol-7 documented. Unknown engine builds fall back to keystream derivation from the evidence.</li><li><strong>Replay the WAL</strong> exactly as SQLite would. Validate salts and the checksum chain on the <em>stored</em> (encrypted) bytes, apply only committed frames, and stop at the first break.</li><li><strong>Load the plaintext image into memory.</strong> Using SQLite’s deserialize API, the decrypted database never touches disk. It’s opened read-only and cached briefly so a pack of queries decrypts once.</li><li><strong>Serve osquery tables</strong> through declarative, schema-tolerant column specs. If Defender adds, drops or renames something, missing columns come back empty instead of breaking the table.</li></ul><h3>Building something that runs as SYSTEM, carefully</h3><p>An osquery extension runs inside osqueryd, usually as SYSTEM, on every endpoint. That raises the bar, so a good part of the work went into making it boring and safe:</p><ul><li><strong>Never write decrypted data to disk</strong> (apart from the explicit --decrypt output). <strong>Never write to Defender&#39;s files.</strong> They&#39;re opened read-only with full sharing, so Defender keeps working normally.</li><li><strong>Hostile-input hardening.</strong> An independent review fuzzed the decryptor and tables with about 1,500 corrupted databases. It also built a malicious database whose recursive SQL view could exhaust memory. The fixes: ignore views and virtual tables, add query deadlines, row caps, value-size caps and file-size limits, keep SQLite temp storage in memory, and reject UNC or device paths and arbitrary SQLite files in the path column. A derived key is only accepted if the decrypted database passes SQLite&#39;s integrity check.</li><li><strong>An independent oracle.</strong> The test fixtures were generated by a separate, standard-library Python implementation, and the Go decryptor’s output is <strong>byte-for-byte identical</strong> to it.</li><li><strong>Live Windows testing.</strong> On Windows Server 2022 and 2025 under osquery 5.20, running as SYSTEM against Defender’s live database, the built-in key decrypted engines <strong>1.1.26090.9 and 1.1.26100.4</strong>; Defender actually updated mid-test. The extension has also run for long stretches under osqueryd with its extensions watchdog on, while Defender scanned, updated and rewrote the database. If osqueryd ever drops it, the extension notices, exits and is relaunched, instead of sitting there unregistered. A GitHub Actions job now runs this smoke test weekly on windows-latest, as both admin and SYSTEM, to catch engine updates that change the key or schema.</li></ul><h3>Limitations</h3><ul><li>The format is verified on the engine builds above. Other builds may change the key, in which case the extension falls back to best-effort derivation and may refuse to serve data rather than guess. defender_mpengine_status tells you which happened.</li><li>This exposes only the <strong>current</strong> state (database plus committed WAL). For deleted-row recovery and per-commit history, use Protokol-7’s mpenginedb_parser.</li><li>Reading the Scans folder requires SYSTEM or administrator rights, as you&#39;d expect.</li><li>The data is sensitive (paths, command lines, user names, hashes). Treat query results accordingly, and don’t attach real mpenginedb.db files to public bug reports.</li><li>It’s a lens on Defender’s telemetry, not a replacement for Defender, Microsoft Defender for Endpoint, or your EDR.</li></ul><h3>Try it</h3><p>Grab a release (Windows, Linux and macOS builds; Linux and macOS are for offline analysis of collected evidence):</p><p><a href="https://github.com/rakeshrm/osquery-defender-extension/releases"><strong>github.com/rakeshrm/osquery-defender-extension/releases</strong></a></p><pre>osqueryi --extension defender-mpengine-windows-amd64.ext.exe --allow-unsafe ^<br>  --extensions_require=defender_mpengine ^<br>  &quot;SELECT key_source, quick_check, wal_commits, error FROM defender_mpengine_status;&quot;</pre><p>The release also ships defender_mpengine.conf, an osquery pack with scheduled hunting queries for attack-scoring techniques, suspicious path tags, tamper protection, ASR blocks, ransomware detections and vulnerable software.</p><p>Contributions are very welcome, especially reports from engine builds and hosts I haven’t seen. The repo includes a smoke-test script that collects versions, status and schema (never row contents) so you can share them safely.</p><p><em>Thanks again to </em><strong><em>Protokol-7</em></strong><em> for the original research and tooling: </em><a href="https://protokol-7.github.io/dfir/2026/11/10/defenders-hidden-flight-recorder-mpenginedb.html"><em>the write-up</em></a><em>, </em><a href="https://github.com/Protokol-7/mpenginedb_parser"><em>mpenginedb_parser</em></a><em> and </em><a href="https://github.com/Protokol-7/MpEngineDb_SQLECmd"><em>MpEngineDb_SQLECmd</em></a><em>. This extension stands entirely on their shoulders.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa2bbd4aa8ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>