I built an MCP memory server for one user (me, for six weeks)

Your trade-off is the one I would have written down too, and you named it more precisely than I did in the post.

On the versioning question - three parts, because two have a clean answer and one does not.

Does it overwrite? For the answer, yes. A lesson is keyed by topic, so writing under the same topic replaces the current text. That is deliberate: a superseded note that keeps competing on relevance is worse than no note at all.

Is the old one gone? No. Every write also appends the complete record to a per-topic history list. I counted in my own store rather than guessing: 524 lessons, of which 257 (49%) have been overwritten at least once - 408 overwrites in total. 638 superseded versions are still sitting there, across 334 topics. So the "what changed" half of your question has a real answer: the previous texts are there and they diff.

Can you see why? No. That is the honest gap, and it is exactly the half your approach gets for free. Each record carries an audit entry with a timestamp, whether it was a create or an update, and the previous outcome. There is no rationale field. Your commit message has the trigger and the reasoning in it; my update has neither.

Three more things I would rather say than have you discover:

• The history expires after 90 days. Git keeps forever; this does not. For a note that turns out wrong two quarters later, the version that was right is already gone.
• There is no rollback. The history is readable, but there is no "restore version 3" call.
• No tool surfaces the diff. The data is in the store; the product does not show it to you. So in practice, today, you would miss it precisely as you expect.

One design decision that is in there and belongs in the same family: a record whose outcome is a failure cannot take the slot from a known-good fix. Success and partial always replace; a failure only claims the slot when nothing is there. Otherwise one bad run overwrites the fix with the report that it broke.

On not having to pick a side - we ended up reading your half into ours rather than choosing. brain_from_git parses commit history and turns fixes into lessons, incrementally, so re-running only picks up new commits. And there are two CI templates, GitHub Actions and GitLab CI, that write a lesson per run, so a red-to-green transition gets recorded as a learned fix. The reasoning was the one you describe: the commit message already carries the trigger and the rationale, so not reading it means throwing away the best-labelled data in the repo.

What that still does not do is carry the rationale through into the lesson as a first-class field. It reads your history; it does not keep your "why". That is the piece your approach has and mine does not, and it is now written down as a gap rather than a preference.

Correction to my own comment above/below, and it is the worst kind of one.

I wrote: "Retrieval quality itself is benchmarked and defended in CI - +33% Precision@1 over raw BM25, 98.2% Recall@3, reproducible with one command."

I ran the command again. This is what it prints:

  metric       flatfile baseline   cachly    vs flat
  Precision@1     76.9%    69.2%    69.2%     -10.0%
  MRR             87.2%    83.3%    83.3%      -4.4%
  ---------------------------------------------------
  vs BM25 baseline : MRR +0.0% - Precision@1 +0.0%
  vs flat-file mem : MRR -4.4% - Precision@1 -10.0%

Not +33%. +0.0% - and ten points behind a flat file on the first metric.

How the wrong number survived: it was a hardcoded string in the server's own metrics output, with a comment above it reading "CI-defended", and a unit test asserting that the string was present. So there was a green check. The check was guarding the sentence, not the measurement. A number the server does not compute cannot fall, which means it is not measuring anything.

The instrument is worse than the string, and that matters more. That benchmark has 17 lessons and 13 queries. On 498 real lessons with 20 questions asked in plain language, the current ranking formula scores 30% Precision@1 and the one it replaced scores 15% - while the 17-lesson benchmark ranks them the other way round, 69% against 92%. On the one comparison I can check against reality, it moves in the opposite direction.

So the honest state: the only retrieval number I can stand behind is 30% Precision@1 on 498 real lessons, up from 15%, measured once, on one store, by me. Everything I said about CI-defended benchmarking was describing a test that defended a sentence.

The string is out of the product, and the test now asserts the opposite - that no fixed ranking number appears in that output at all. Replacing the benchmark with something larger than 17 lessons is the next job, and it has to happen before any number goes back in.

Your question was whether the memory changed the action. I answered it and then attached a caveat that turned out to be the less honest half of the comment. The metric you asked for is what sent me to run the command, so: thank you, twice over.

"Confident clutter" goes into my vocabulary, with attribution.

Your second question sent me to count instead of guess, and the counting corrected me twice.

First pass: I searched all 521 records for ones that explicitly correct an earlier record - phrases like "corrects a previously stored lesson". 48 of them. I was about to write that number down as a defect.

Second pass, splitting them properly: 39 of the 48 correct their own record. The store updates in place and bumps a version, so the wrong text is gone rather than sitting next to its replacement. Of the remaining 9, only 2 name another record that still exists - and reading those two, they are citing examples, not superseding anything.

So the honest answer to "how often did you have to correct the memory": roughly one record in ten has been corrected at some point, and the correction almost always replaced the thing it corrected rather than competing with it. That is a considerably better answer than the one I was about to give, and I only have it because you asked for a metric instead of an impression.

What is genuinely missing is narrower than I first thought: a record cannot say "I supersede that one" across topics, and there is no valid-from date, so retrieval cannot prefer the currently applicable fact over an older one that still looks true. I have not yet found a case where that bit us. I would rather say that than dress up a gap I cannot demonstrate.

Your first question is the harder one, and you phrased it better than I had. "Did it change the action" is not "was it used". Used is self-reported and will flatter. Changed-the-action is a counterfactual, and counterfactuals do not come from asking.

The only honest route I see is to withhold. Hold back the top record on a random half of eligible turns; if the outcomes differ, the record changed the action. Not per turn - in aggregate, which is the level the claim gets made at anyway.

That is what I am setting up. The awkward part belongs in the same breath: it costs real sessions, because half my own turns get a deliberately worse answer for the duration. That is the price of the number, and I have not found a cheaper one that is not a self-report in disguise.

One thing I should put next to all this, because otherwise it reads as if nothing here is measured. Retrieval quality itself is benchmarked and defended in CI — +33% Precision@1 over raw BM25, 98.2% Recall@3 against an external corpus, reproducible with one command. What I cannot yet prove is the step after that: whether finding the right record changes what the assistant does. That is the gap the hold-out is for, and it is the honest boundary between what I can show you and what I am still owed.

"The value only showed up as an absence" is the whole measurement problem in one line, and you got there from a different direction than I did.

That's why the metric is so slippery: an event that didn't happen leaves no row in any table. You can count recalls, you can count writes - you cannot count the bug that never got filed. Every number I have measures delivery and I keep having to stop myself from labelling it prevention.

The only honest instrument I've found is withholding. Hold back the top record on a random half of eligible turns and compare outcomes in aggregate. It costs real sessions - half your own turns get a deliberately worse answer - and I haven't found a cheaper version that isn't a self-report in disguise. If Kiro's steering docs gave you any way to see the absence directly, I'd genuinely like to hear it.

On team memory, the honest state: it's built and it runs - every lesson carries the name of whoever learned it, and there's cross-author reuse tracking, which is the number that matters (how often you recall something a teammate wrote). What I don't have is teams using it. So the feature exists and the evidence doesn't.

The failure mode I expect first, and I'd rather name it before you find it: a lesson written in one person's vocabulary that nobody else ever retrieves. It counts as "stored" and prevents nothing. If you try it with a team, that's the number I'd watch - the share of lessons never recalled by anyone but their author.

Agreed on the principle, and I want to add the failure mode that sits right behind it - because "put the invariant in CI" is exactly what I did, and it went wrong twice in ways worth naming.

A guard can be green and blind. I had a check asserting that a benchmark number appeared in the server's output. The number was a hardcoded string. The check guarded the sentence, not the measurement - and a number the server does not compute cannot fall. It was green for weeks while the real figure was ten points behind a flat file.

A guard can watch the spelling instead of the rule. This one is from today. I wrote thirteen checks over a harvesting tool. Two hours later I rewrote the tool's transport - same rules, different words - and seven of thirteen went red, while nothing had gotten worse. The checks were asserting identifiers, not behaviour. I rewrote them to run the actual functions and only left a text assertion where an execution genuinely needs a network.

So the version of your rule I'd now write for myself: the invariant must be executed, not spelled. If a check can pass on a codebase where the thing it protects has been deleted and re-implemented differently, it's protecting the name.

Your recovery-path point is the half I'm weakest on and I'll take it plainly. My guards say "this is wrong"; most of them don't say what to do. The two I've fixed since read like this: the failing message names the replacement to use and prints the first twelve offending sites with file and line. That's the difference between a red check and an operating practice, and you named it better than I had.

Thank you - and that question has a precise answer, because I went and counted this afternoon instead of guessing.

Overwritten, mostly. A lesson is keyed by topic, so writing a correction under the same topic replaces the text and bumps a version; the audit trail keeps what changed. In my own store 488 of 521 records are past version 1, and of the 48 that explicitly say they correct something, 39 correct their own earlier text. So in the common case the wrong version is gone rather than sitting next to its replacement.

What does not happen, and this is the honest boundary:

A record cannot mark a different record as superseded. If the correction lands under a new topic name, both survive and compete on relevance alone.

And nothing detects rot. A lesson that quietly stopped being true - the server moved, the flag got renamed - sits there looking exactly as confident as the day it was written, until a human notices and writes the correction. There is no valid-from date, so retrieval cannot prefer "currently applicable" over "still plausible".

So the accurate answer to your three options: overwritten when someone catches it, and sitting there when nobody does. The first half is solid. The second half is the part I would rather name here than have a user discover on their own.

You moved the question to the write path and that is where it belongs. I went
and read our own code rather than answer from memory, and the answer is worse and more interesting than a plain no.

Is there a notion of a fact scoped to something that can change underneath
it? Yes - and it does not reach the reader.

The write path takes a depends_on list (["node:>=20", "docker:running",
"wireguard:active"]) and builds a reverse index: dependency → the topics that
rest on it. There is a trace_dependency call that walks that index and, with
mark_review=true, stamps needs_review: true on every dependent record.

That is exactly the shape you describe. Here is what is wrong with it.

One. I grepped the retrieval path for needs_review. Zero occurrences. Not
in the recall handler, not in the ranking core. The flag is written, and then
it is rendered as a badge inside trace_dependency's own output - a tool
nobody calls mid-task. A record marked needs_review ranks and returns exactly
as it did before. So the invalidation exists as bookkeeping and not as
behaviour, which is the same as not existing for your agent mid-task.

Two. Nothing fires it. mark_review is a parameter someone passes after
they already noticed the dependency changed. That is not invalidation by event.
That is invalidation by somebody remembering, which is the thing the memory was
supposed to replace.

Three, and this is the one that made me wince. The dependency index is
written with a 90-day expiry. The mechanism for invalidating by event is itself
invalidated by age. Another commenter in this thread had just pushed back on
that same TTL on the history side; I did not know it also ate this.

So: contradiction is resolved at read time, by whoever is reading, exactly as
you assumed - and the write-time machinery that would have prevented that is
built up to the last inch and then stops.

On "a reason to die instead of a confidence score" - I am taking that whole,
and not only for the memory.

Your sentence about the agent mid-task ("it picks one and keeps going, and you
find out later") describes a failure I hit three times today, in three
systems that have nothing to do with each other. Same shape every time: no
state for "I cannot say", so something plausible gets substituted.

1. The ranker. Features are normalised across the candidate pool. A missing
   value became 0 - the worst value, not a neutral one. 108 records were
   being penalised for a field they were never supposed to have. I measured
   with the feature and without, concluded it was harmful, and removed it. The
   measurement was right. The conclusion was wrong.

2. A health endpoint. An instance with zero records was classified "100 %,
   healthy" - the best value. Paired with a random sample of 8 out of 67
   instances, mostly empty test accounts, that made ten consecutive calls eight
   seconds apart come back five times "fine" and five times "outage". Our alerts
   had been flapping for two nights and I had already fixed a different, real
   cause one layer up. This one was underneath it.

3. My own tooling. I wrote a contrast checker for our UI that hardcoded a
   white background. Run against a dark app it produced 341 findings, every one
   of them with a line number and two decimal places, every one of them wrong.
   Had I not checked, I would have "fixed" a working interface.

Three systems, three substituted defaults, three different directions - worst,
best, and assumed. The common part is not the value. It is that none of them
could say "no statement". Your framing names the fix better than mine did:
a fact needs a reason to die, and a system needs a way to say it has nothing
to report. Those turn out to be the same missing thing seen from two ends.

The health endpoint now has three states instead of two — aus (records
present, index gone), leer (answered, nothing to say), nicht_gemessen (did
not answer). Keeping leer and nicht_gemessen apart matters more than it
looks: collapsing them turns the common case into an alarm, which is worse than
the flapping was.

What I owe you, in order, and I would rather write it down than let it stay a
preference:

• needs_review has to be read at rank time, or the flag is theatre.
• The dependency index must outlive its TTL, or event-based invalidation dies on a timer.
• Firing it needs to be automatic for the dependencies a machine can observe (a version, a running service, a config hash), and manual only for the ones it cannot.

The first is small and I had not seen it until you asked. Thanks for pointing
at the write path - I would have kept polishing the read side.

Thanks - and I have to give you the honest answer rather than the good one: I can't tell you yet, because I don't have that data.

The tool has effectively one heavy user, so every number I have about prevention describes one person's memory patterns. I could dress that up as an early finding, but it'd be a description of me, not of the metric.

What I think would actually show up with several different users is the failure mode, not the success: a lesson written in one person's vocabulary that never gets recalled by anyone else, so it counts as "stored" and prevents nothing. Prevention is measured at recall time, and recall depends on the words the asker uses. That's the number I'd want to watch first - the share of stored lessons that never get retrieved by anyone but their author. If that's high, the metric is quietly measuring authorship rather than usefulness.

If you end up running something similar across a team, I'd genuinely like to hear whether that's what breaks first.

You are right about the snippet, and it is the weakest thing in the piece. HIT/MISS counts delivery and I labelled it prevention. Those are not the same, and a non-empty recall that is stale or confidently wrong is worse than an empty one, because it costs the hour it takes to find out.

The part I want to ask you about, because I think it is where this gets hard: "used / ignored / contradicted" — who applies the label?

If the tool labels its own recall as "used", that is a self-reported exit code. It is cheap to collect and it will be systematically generous, because the thing being asked whether it helped is the thing whose helpfulness is in question.

So I have been trying to split your three into what a machine can observe without taking anyone's word for it:

1. contradicted - mechanical. A later record on the same topic supersedes the one that was served.
2. corrected -mechanical. The served record gets edited within N days of being served.
3. stale - mechanical. A newer record on the same topic existed and was not the one returned.
4. used - I have no honest mechanical version. Only weak proxies.

Three honest fields beat four with one that flatters. But if you have seen a way to earn the fourth without self-report, that is the thing I would most like to be wrong about.

Your provenance list reads as a gap list from here. We have author, timestamp, confidence and an audit trail. We do not have valid-from or supersession links — so "prefer the current applicable fact" is not something our retrieval can express. It blends, exactly as you say it should not. That one I can act on directly.

Net prevented cost is the sentence I will end up quoting. Our counting has no term for harm at all. A recall that sends someone the wrong way for an hour scores the same as one that was never made — which means the metric rewards more memory rather than trustworthy memory, and I did not see that until you wrote it down.

Second time you have handed me something sharper than the post it is under - thank you.

"A new user reads an empty result as a broken tool" - I've now hit the machine version of the same sentence, and it's worse than the UX one.

Yesterday a harvesting tool of mine ran across sixteen public repositories. The first five returned 260–277 results each. The remaining ten returned zero, reporting "no issue has a linked pull request" - for repositories with 60,000 linked pairs between them.

The cause was an hour rate limit. The bug was one line:

} catch { return null; }

Every error became the statement "this one has nothing". The tool then wrote a valid, empty file, printed "done" and exited 0.

Here's your point, one layer down: I knew what an empty result meant, so I never gave the code a way to say it didn't know. There was a state for "found nothing" and a state for "found something" - and no state for "did not measure". Silence had to become one of the two, so it became zero. And a zero looks like a result.

The founder tolerance you describe isn't only about UX. It's in the data model. When you're the only reader, "no rows" is a sentence you finish in your head, so the schema never learns to finish it. The fix wasn't better error handling - it was adding the third state, and refusing to write a file at all when it's non-zero. A half harvest is worse than none, because it looks like a whole one.

Your last line is the one I'd put on the wall: handing it to a stranger without explaining a thing. That's also the only way I found this — a colleague asked "why is that number zero?" and I had no answer that wasn't "huh".