REASONEX · SAME 18 QUESTIONS, SAME THREE CONDITIONS, TWO MODELS
Two models. Two different stories.
We had a simple intuition: More context doesn’t necessarily mean more useful context.
So we ran the same context-optimization experiment across two very different models.
18 questions. Three conditions:
A – Raw retrieval
B – Verbose context
C – Compact / structured context
The results diverged.
Mercury (diffusion-based)
- Raw retrieval: 88.9% accuracy
- • Verbose: 83.3%
- • Compact / structured: 77.8%
Qwen3 (autoregressive)
- Raw retrieval: 94%
- • Verbose: 100%
- • Compact / structured: 89%
The accuracy leader changed.
Get Wagied Davids’s stories in your inbox
Join Medium for free to get updates from this writer.
Then we ran a compression sweep on the structured context, from 100% down to 5% retention.
Mercury: 83% → 76% → 61% → 61% → 61%
Qwen3: 94% → 83% → 89% → 89% → 94%
The interesting part isn’t simply that “less context works.”
It’s that accuracy was not a monotonic function of context retained.
For Qwen3, performance dropped at 50% retention, recovered at 25%, and returned to its full-retention accuracy at 5%.
That suggests useful signal in the context may be concentrated rather than evenly spread – with real redundancy elsewhere.
But there’s an important caveat.
It’s tempting to attribute the difference to diffusion vs. autoregressive generation. We can’t.
The models also differ in training data, scale, instruction tuning, and serving stack.
And this is only 18 questions, one run per model.
So the claim we can make today is narrower:
The winning context condition wasn’t stable across models, and compression behavior wasn’t monotonic.
The next question is more interesting:
Which parts of context actually matter?
Context optimization may be less about adding more information and more about identifying and preserving the structural signal that actually matters.
#RAG #ContextEngineering
