I Build Agent Memory. Raw Retrieval Is the Baseline to Beat.

Memory can solve new questions and still make the system worse. That’s the part worth measuring.

The uncomfortable baseline

I’m building BrainAPI, a memory layer for AI agents. Before asking anyone to adopt another layer, there’s a question worth answering: what does it do better than a good retrieval pipeline over the original history?

Memory Tax: Dissecting the Net Value of Agent Memory, by Xuan He and colleagues, makes that question difficult to avoid.

Constructing memory can make useful information easier to find. It can also discard evidence, distort it, or preserve it in a form the answering system struggles to retrieve.

The interesting measurement is therefore not just what memory helps the agent answer. It’s also what the agent could answer before — and can’t anymore.

That performance tax is separate from the computational bill.

MemoryTax graph.

What “90 out of 99” actually means