I've been reading through dev communities lately, and this exact topic keeps showing up in different forms: context windows hit a million tokens, so is memory even necessary anymore. Enough people are arguing both sides that I wanted to actually dig into it and put together my own take instead of just picking whichever post I read most recently.
Just How Big Are They Now?
A few years ago, a few thousand tokens felt generous. Now, 1 million is the baseline. Meta released the 10-million-token Llama 4 Scout last year, and a startup called Magic built a 100-million-token model (LTM-2-mini). That means about 10 million lines of code or 750 novels can fit into a single prompt. At this point, you have to wonder what is left for a separate memory system to do.
Camp 1: It's Already Enough
Fabio Akita pointed out that looking at the leaked Claude Code source code, Anthropic's own coding agent doesn't use a vector DB at all; it just uses the file system and grep. By his math, a 200,000-token query costs about $0.63 including caching, which is cheaper long-term than maintaining a vector DB pipeline. Meta made the same bet, marketing the Llama 4 Scout as capable of holding years of chat history "without a vector store."
It's an attractive argument, and honestly, the cost aspect is the strongest part of this camp. But I think the question this camp is answering is narrower than reality. "Do we need a vector DB?" and "Do we need memory?" are different questions. Akita's point is really about search complexity, using grep instead of embeddings, not about whether state needs to persist between sessions. Meta's marketing also conveniently skips over the fact that a session with 10-million-token chat history eventually ends, and the next session starts at zero.
Camp 2: Memory Solves a Completely Different Problem
Mem0 compares the context window to RAM rather than storage: the moment a session ends, everything inside vanishes. Redis puts it more sharply: agents don't fail because a single invocation lacks space, but because they lack continuity, they can't carry what they learned in one session over to the next. No window size fixes this. Even a 100-million-token model completely forgets you the moment you open a fresh chat.
The most reliable evidence here is Chroma's "context rot" research. Testing 18 models (including GPT-4.1, Claude 4, Gemini 2.5, Qwen3), they found that performance degrades as inputs get longer, well before hitting the model's actual limits. A single piece of irrelevant distractor info noticeably drops accuracy. In some tests, a scrambled mess of info actually performed better than an organized one. This is what genuinely convinced me: fitting into a window and a model actually utilizing it well are two different things, and sellers of larger windows have plenty of incentive to blur that boundary.
And Camp 3: The Window Is Simply Too Small
Factory.ai points out that current 1-million to 2-million-token models are already smaller than their enterprise customers' codebases. CloudGeometry goes a step further, arguing that even if you gave them 100 million tokens tomorrow, it wouldn't be enough because codebases are graph structures while context windows are linear, no matter how large they get, that structural topology disappears.
What's fascinating is that Factory and CloudGeometry draw the exact same fact, "therefore, we need better search," while Magic and Meta go with, "therefore, let's build bigger windows." Same observation, opposite prescriptions. It's also worth noting that the "bigger window" crowd usually sells models, while the "better search" crowd usually builds things on top of other people's models.
My Conclusion
Context windows need to keep growing, I'm not arguing against that. What I disagree with is treating "bigger windows will fix memory problems" as a given. These are two separate investments, and relying entirely on one to substitute for the other doesn't work. Why? Because a bigger window solves exactly one problem: the fitting problem.
It does nothing for session amnesia, accuracy loss from irrelevant tokens, or the cost of reprocessing the exact same context on every single invocation. Memory has to evolve on its own to solve these three, and scaling the context window won't do that work for it. So the real paradigm isn't "context versus memory." Rather, while context handles one problem, memory must solve the remaining three through its own evolution, not by waiting for someone else's context window to get bigger.
The Strongest Counter-Argument to My Point
I don't want to give memory a free pass, so let's briefly counter my own argument. Memory poisoning is a real attack where someone sneaks mundane content into an agent's long-term memory to plant false info, which the agent then retrieves and trusts in a completely unrelated situation later on. MINJA, presented at NeurIPS 2025, achieved this with just a few mundane queries.
Add in silent failures (memory retrieves the wrong thing, yet the model speaks with absolute confidence and no errors) and stale information (studies measure that forcing answers yields stale info 15 to 40 percent of the time), and you have genuine reasons to doubt anyone saying, "just throw memory at it."
Yet my answer doesn't change, and here is why. Silent failures and stale info are mistakes, not malicious acts. They are mostly fixed by displaying confidence scores and recency on retrieved memories, and automatically invalidating old info when new data arrives. This isn't a design flaw, it's engineering maturity, and it's precisely what improves as memory technology matures. Security, however, never fully goes away, just like defense and offense evolving together, much like anti-spam filtering never truly ends. But saying "this requires ongoing work" is different from saying "memory is a dead end." It's simply a trade-off you must accept the moment you ask an agent to persist anything, via memory or otherwise.
Wrapping Up
So let's sum it up: if your agent needs to remember anything past the current session, waiting for context windows to get bigger is betting on the wrong lever.
Change my mind. What is actually breaking in your production environment right now, massive contexts, RAG, structured memory, or something entirely different?
Top comments (11)
The context rot part is the one that changed how I work. Long threads get worse before they get full, and the failure looks like a confident wrong answer, so you only notice it much later.
The one I keep hitting is reprocessing. Same project, same files, paid for again every session. That is not a fitting problem and no bigger window fixes it.
Reprocessing is the one I underrated while writing this. Caching helps but it expires, and the moment anything early in the context changes you pay for the whole thing again, so the cost isn't really per session, it's per edit. And the confident wrong answer part is what makes context rot so hard to argue about, since there's no error to point at. If it threw an exception at 60 percent fill, everyone would have fixed it years ago.
The exception line is the useful part. You can build that error yourself, roughly. Keep a few questions with known answers, re-ask them as the thread grows, and watch where the answers start drifting.
Per edit rather than per session also explains why a long chat feels cheap right up until you go back and fix one thing near the top. That edit is the expensive keystroke and nothing tells you.
The canary idea is good and I hadn't thought to run it live. A few questions with known answers, re-asked as the thread grows, gives you the error signal you otherwise don't get, and it costs almost nothing. My guess is the drift point isn't fixed either, it moves depending on how much unrelated stuff is sitting in the thread, which would make it worth re-running rather than measuring once. The keystroke point is the part that stuck with me though. Editing near the top is the one action where the cost is immediate and invisible at the same time, and every tool I use presents it as if it's free.
Re-running rather than measuring once is right, and there is a catch inside it. The canary is part of the thread too, so every re-ask adds to the thing you are measuring. Keep it short, word for word identical each time, and read it as a trend rather than a number.
On the keystroke, the reason it looks free is that the counter you can see is tokens and what you actually spent is cache. Nothing in the interface shows the second one, so there is nothing on screen that could have warned you.
The canary being part of the thread is the bit I'd missed. In an offline setup you dodge it, since you can ask each question against a clean copy of the same context and nothing accumulates. Live you can't, so your third re-ask is measuring a slightly different thread than your first. Trend rather than number is the right way to read that. Word for word identical matters for a second reason too, since even a small rewording changes what the question is testing, and then you can't tell drift from a different question. On the cache thing, what gets me is that the visible number isn't just incomplete, it points the wrong way. Token count climbs slowly and smoothly, and the cost of an edit near the top is a spike that never shows up anywhere. A counter that moves calmly while the bill jumps is worse than no counter at all.
That last bit is fixable, and cheaply. The usage payload already comes back split into cached and uncached tokens, most people just add them together into one number and lose the only signal that matters.
Plot the uncached share on its own instead of the total. It sits near flat while you are appending, and jumps the moment you touch anything above the cache point. The spike is the edit you made, and it tracks the bill instead of tracking the thread length.
The RAM-vs-storage framing is the right one, and the "next session starts at zero" point is the one the 10M-token marketing skips. A context window is working memory: it's there for the duration of the invocation and then it's gone. Nothing about it persists.
Where I hit this in practice is the VRAM wall, not the token limit. On a 2x3090 the context size that actually fits fully in VRAM is a hard number — push past it and the weights start offloading to system RAM and throughput collapses. So "how big can I make the context" has a concrete answer on my hardware, and I measure it rather than guess. The benchmark tab in homelab-monitor runs each model across a context ladder and reports the largest context that still fits in VRAM — that's the cap I actually set, and it's the number that decides how much of "memory" I can keep in-context vs. have to persist.
The VRAM wall is a nicer problem to have than the one most people in this thread are describing, because it fails loudly. Throughput collapsing is impossible to miss. The ceiling everyone else is hitting is a quality one, and it sits well below the advertised limit with nothing announcing it. Your context ladder is the same shape as something another commenter suggested here, which was keeping a few questions with known answers and re-asking them as the thread grows to see where the answers start drifting. One measures where it stops fitting, the other measures where it stops being useful, and my guess is the second number is lower than the first on your setup. Have you ever run the ladder with an accuracy check on top, rather than just fit and throughput?
The uncached share spike is a diagnostic for what should have been in persistent memory instead of the context window. Facts that don't change between sessions don't need to be reprocessed every time something upstream shifts.
We built DataGrout's Logic tools around this. Facts stored outside context, retrieved per task. The reprocessing cost disappears because those tokens never entered the window in the first place.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.