Can AI Explain a Malware PCAP? Lessons from 20 AdversaryGraph Tests
Real packet captures, short investigation reports, and an honest comparison with published answers.
A security report can list hundreds of addresses and still leave the analyst asking: what actually happened?
That was the problem I wanted to test in AdversaryGraph. Could my platform turn packet evidence, recovered files, and indicator enrichment into a short, understandable investigation story — and get that story right?
I tested 20 historical Malware-Traffic-Analysis.net exercises through the local platform. These were public training captures; no recovered malware was executed.
What the tests showed
The workflow combined packet analysis, file recovery and hashing, approved enrichment, and a dedicated report-writing stage. The original model outputs were preserved before retrieving the published answers for comparison.
All 20 captures decoded. Sixteen final short reports were saved, with a median length of 179 words. But completion was not accuracy:
- 12 cases: partial explanations.
- 4 cases: missed or misinterpreted scenarios.
- 4 cases: reports withheld rather than saved as successful results.
None was counted as a complete reference-level investigation. Some published answers used additional evidence unavailable in the PCAP-only input. This was an iterative engineering study, not an untouched blind benchmark or a measured competition against human analysts.
The full research on 1200km.com documents the methodology, actual model identifiers, timing, token accounting, and every case comparison.
Where the story broke down
Steelcoffee illustrates the gap. AdversaryGraph identified the affected workstation and recovered a Windows executable delivered through a PNG-looking URL. Its SHA-256 matched the publisher’s sample.
Get Andrey Pautov’s stories in your inbox
Join Medium for free to get updates from this writer.
Yet the native report did not identify Qakbot or reconstruct its later activity. That family identification came from the separate reference-assisted review — not a successful live reputation lookup.
Enrichment had limits too: VirusTotal produced no successful verdicts in this run, while MalwareBazaar returned one direct Gozi hash match. An unavailable lookup is not a clean result. An observed IP address is not automatically an indicator of compromise.
The lesson is practical: readable prose, valid citations, and correct hashes can coexist with an incomplete incident explanation.
Conclusion
AdversaryGraph produced useful evidence and more readable drafts, but the results do not support autonomous investigation or near-perfect accuracy. Better reporting cannot compensate for missing attack stages or incorrect attacker–victim roles.
I published the failures alongside the useful findings. The public evidence collection includes all 20 platform screenshots, available native reports, separate reference-assisted reviews, answer comparisons, and enrichment records.
My conclusion: let automation organise the evidence and draft the account. Keep the analyst responsible for validating what happened.
References
- Full twenty-case study and published-answer comparisons.
- Screenshots, reports, and downloadable evidence.
Follow My Work
I publish practical cybersecurity research, CTI workflows, detection engineering notes, malware-analysis projects, AI-security research, open-source tools, labs, and technical guides.
