Member-only story
Context Engineering Is Dead. Long Live Cache Engineering.
What happens when we give the agent direct control over its active KV cache?
When the context gets crowded, the harness trims older messages or asks the model to summarize them, and the next inference request processes the revised history.
That cleanup step carries a systems cost that most harnesses never account for.
Editing an early message invalidates the KV cache for everything after it. A coding agent can throw away one obsolete tool response and still pay to recompute a long suffix full of useful reasoning and tool calls.
We can extend inference engines for an agent to remove selected cache entries and compact token positions without rewriting the conversation or recomputing the cache states it keeps.
Today’s project provides 27.7% lower average inference cost and a 1.7 percentage-point gain in benchmark score. In the serving evaluation, it reaches 1.18x peak output throughput and 1.51x faster P95 request latency.
If you build long-running coding agents, research agents, or multi-tool workflows, the idea deserves a close look at both the API and the infrastructure level.
Let’s dive in.
