Title : Runbook Truth
Runbook Truth - Agent logic
I Used Hindsight to Turn Incident Memory Into Recommendations
I didn’t want our incident-response agent to stop at "I found a similar incident." That is useful, but it still leaves the engineer to
interpret the history and decide what to try next. I wanted the agent to use previous outcomes as evidence: which fixes worked,
which failed, and how certain the recommendation should be.
That became the core of my work on Runbook Truth.
I built the recommendation layer around recalled incident memories. Hindsight provides the memory layer, while my logic analyzes
those memories, compares outcomes, produces a recommendation, assigns a confidence level, explains the reasoning, and stores the
result of a newly resolved incident for future use.
The important distinction is simple: remembering an incident is not the same as learning from it.
What the system does
The incident flow starts with a new error. The system asks the memory layer for similar past incidents. Hindsight is responsible for
retaining and recalling that experience; my part begins when those memories reach the recommendation layer.
The flow is:
New incident
↓
Hindsight recall
↓
Similar past incidents
↓
Analyze success / failure
↓
Recommend a fix
↓
Calculate confidence
↓
Explain the recommendation
↓
Resolve the incident
↓
Store the outcome
I used Hindsight on GitHub as the memory component and designed the recommendation logic around the outcomes contained in
recalled incidents. The broader idea is consistent with Hindsight’s documentation: memory should help an agent use previous
experience rather than simply retrieve old text.
The first useful step was counting outcomes
A recalled incident is only useful to my recommendation logic if I can determine what was tried and what happened afterward.
I represented the useful part of an incident as three fields:
{
"error": "Redis connection timeout",
"fix": "Increase Redis timeout",
"result": "success"
}
Then I grouped memories by the fix and counted successes and failures:
def analyze_memories(memories):
stats = {}
for memory in memories:
fix = memory["fix"]
result = memory["result"]
if fix not in stats:
stats[fix] = {"success": 0, "failure": 0}
stats[fix][result] += 1
return stats
For the Redis timeout case, my test memories produced a result like:
Restart Redis
Success: 0
Failure: 2
Increase Redis timeout
Success: 2
Failure: 0
This small transformation is important. Instead of treating recalled incidents as a list of unrelated text entries, the recommendation
layer turns them into evidence that can be compared.
Get Ankithanalla’s stories in your inbox
Join Medium for free to get updates from this writer.
Choosing the recommendation
Once the outcomes are grouped, the next question is straightforward: which previous fix has the strongest successful history?
My current recommendation function deliberately stays simple:
def recommend_fix(memories):
if not memories:
return None
stats = analyze_memories(memories)
best_fix = None
best_score = -1
for fix, result in stats.items():
if result["success"] > best_score:
best_score = result["success"]
best_fix = fix
return best_fix
For example, if restarting Redis failed twice while increasing the Redis timeout succeeded twice, the recommendation becomes:
Recommended Fix: Increase Redis timeout
I intentionally kept this layer understandable. The recommendation is not a mysterious model-generated sentence. It is traceable to
the outcomes that were recalled.
That makes debugging much easier. If an engineer disagrees with the recommendation, I can inspect the underlying memories and see
why the logic selected it.
Confidence is separate from the recommendation
A recommendation without uncertainty can be misleading.
Suppose a fix succeeded once. I can technically recommend it, but I don’t want the system to communicate that result with the same
confidence as a fix that succeeded repeatedly.
I therefore calculate a success rate for the selected fix:
success_rate = success / total
if success_rate >= 0.75:
return "HIGH"
elif success_rate >= 0.50:
return "MEDIUM"
else:
return "LOW"
In the Redis example, two successes and zero failures produce:
Confidence: HIGH
The confidence value is intentionally simple. It is not a statistical guarantee and it is not a prediction of how the next incident will
behave. It is a compact representation of the evidence available in similar past incidents.
That distinction matters in incident response. A system should be able to say "this worked before" without pretending that the same
action is guaranteed to work again.
The explanation comes from the same evidence
I also wanted the agent to explain its recommendation.
Returning only this:
Increase Redis timeout
forces the engineer to ask another question: why?
The explanation function uses the same outcome counts:
def explain_recommendation(memories, recommended):
stats = analyze_memories(memories)
result = stats[recommended]
return (
f"Recommend '{recommended}' because it succeeded "
f"{result[’success’]} times and failed "
f"{result[’failure’]} times."
)
That produces an explanation such as:
Recommend 'Increase Redis timeout' because it succeeded
2 times and failed 0 times in similar past incidents.
This is one of the design decisions I found useful: the recommendation and explanation share the same source of truth. I don’t need a
separate explanation system to invent a reason after the recommendation has already been made.
Connecting the recommendation layer to Hindsight
The recommendation logic becomes useful when it is connected to actual recalled memory.
The integrated flow starts by calling the memory layer:
raw_memories = recall_similar_incidents(error, limit=8)
memories = convert_hindsight_memories(raw_memories)
recommended = recommend_fix(memories)
The recalled information is converted into the structured form expected by the recommendation functions. From there, the flow is
deterministic: analyze the outcomes, select a fix, calculate confidence, and build the explanation.
This separation keeps responsibilities clear.
Hindsight answers:
> What similar experience do I remember?
My recommendation layer answers:
> Given that experience, what should I recommend?
That separation is closely related to the broader idea of agent memory. Memory becomes more useful when an agent has a clear
mechanism for turning recalled experience into its next action.
No memory is a valid result
One of the easiest mistakes in an incident-response system is to assume that there will always be a useful previous incident.
There won’t.
If no similar memory exists, my code returns an explicit response:
def handle_no_memory(memories):
if not memories:
return {
"recommendation": "No known fix from past incidents",
"confidence": "LOW",
"explanation":
"No similar past incident was found. "
"Follow the standard runbook.",
"past_incidents_used": []
}
The important behavior here is not the wording. It is the decision not to manufacture certainty.
No relevant history means no evidence-based recommendation from memory. The system falls back to the standard runbook instead.
Conflicting outcomes need a different response
Past incidents can also disagree with each other.
For example:
Restart database
Success: 2
Failure: 2
In this situation, recommending the fix without qualification would hide useful information.
I handle that case explicitly:
if result["success"] > 0 and result["failure"] > 0:
return {
"recommendation": recommended,
"confidence": "LOW",
"explanation": (
f"’{recommended}' has mixed results: "
f"{result[’success’]} success and "
f"{result[’failure’]} failure."
)
}
The resulting response makes the uncertainty visible:
