Hindsight Ended My LLM’s Villain Arc
“Why is it asking me to upload my resume again — I literally did this yesterday.” That one message from a tester during our first internal demo sent me back to main.py to rebuild the entire memory layer from scratch. Turns out "the model knows things" and "the system remembers things" are two completely different problems.
I’m on the backend of Team Ignis’s AI Career & Internship Advisor, built for Microsoft Hackathon 2026. The problem statement was straightforward: build an AI agent that helps students land internships. The platform we shipped does skill gap analysis against a job description, generates a personalized learning roadmap, runs a proctored 10-question voice interview, evolves your resume with AI-verified skill badges, and fetches live internship listings scored against your actual profile. All of it runs through a single FastAPI server in backend/main.py, talking to LLaMA 3.3 70B via Groq, with Serper API for job search and ReportLab for PDF export.
The LLM integration wasn’t the hard part. Memory was.
What the System Does and How It Hangs Together
Fifteen endpoints, one core flow. A student uploads their resume and a job description as PDFs. /api/analyze-gap extracts the text, sends both to LLaMA acting as a technical recruiter, and returns structured JSON — skills found, skills missing, reasoning. /api/generate-roadmap reads those gap entries and builds a 3-step learning path with curated resources from YouTube, Udemy, Coursera, and NPTEL. /api/chat runs the proctored interview with progressive difficulty and evaluates at question 11. /api/evolve-resume reads verified skills from memory and rewrites the resume with an AI-Verified Certifications section. /api/search-jobs and /api/match-job chain LLM query generation with live Serper results and per-listing match scoring.
Everything talks through one file: memory.json. That single architectural choice created most of the interesting problems I had to solve — and most of the lessons I'm writing about here.
The Memory Problem
My first pass at memory was a flat append-only list. Load it on every request, write new entries at the end:
def load_memories():
if os.path.exists(MEMORY_FILE):
with open(MEMORY_FILE, "r") as f:
data = json.load(f)
if isinstance(data, list):
return data
return []This worked for a single session. Gap analysis wrote "Gap Identified: Docker" and "CANDIDATE_NAME: Pruthvi". The roadmap endpoint read those entries and injected them into the LLM prompt. The interview endpoint pulled the candidate name for personalization. Simple, fast, worked on the first demo.
The problem appeared the moment a second tester tried the platform after the first one had already used it. Everything was keyed off a global flat list. There was no concept of a user record — no roadmap history, no verified skills attached to a specific person, no way to restore state on login. The second tester’s session started with the first tester’s gaps still sitting in memory. Our “personalized” AI career advisor was serving someone else’s career data.
The tester’s message — “why is it asking me to upload my resume again” — was actually the polite version of what was happening. The impolite version: our agent had learned nothing about anyone, and was pretending it had.
The Dual-Layer Store
I needed a user persistence layer without breaking the six endpoints already depending on the flat list. The solution was migrating memory.json from a list to a keyed dict, wrapping the flat list inside it, and handling the migration transparently on every read:
def _read_full_store() -> dict:
try:
if os.path.exists(MEMORY_FILE):
with open(MEMORY_FILE, "r") as f:
data = json.load(f)
if isinstance(data, list):
# Migrate old flat list transparently
return {"_legacy": data, "users": {}}
return data
except Exception:
pass
return {"_legacy": [], "users": {}}Old format comes in as a list, gets wrapped on the way out. New format passes through. Every existing endpoint kept working because load_memories() still reads from _legacy. The new user records sit in users, keyed by a sanitized username:
def _make_user_id(name: str) -> str:
return re.sub(r'[^a-z0-9]+', '_', name.lower()).strip('_')[:40]'Bhavana K.C.' becomes 'bhavana_kc'. Safe dict key, no special characters, 40-char cap. The write path went atomic after one crash corrupted the file mid-write during a stress test:
def _write_full_store(data: dict):
tmp = MEMORY_FILE + ".tmp"
try:
with open(tmp, "w") as f:
json.dump(data, f, indent=2)
os.replace(tmp, MEMORY_FILE) # atomic on all platforms
except Exception as e:
print(f"Store write error: {e}")
try: os.unlink(tmp)
except: passWrite to .tmp, rename to real file. os.replace() is atomic on both Linux and Windows — the file is either fully written or untouched, never half-written. I wish I'd added this before the crash rather than after it. This is the same durability principle the Hindsight documentation emphasizes for persistent agent memory — writes need to be safe and non-destructive because every downstream LLM behavior depends on what's stored.
How Hindsight Runs Through Every Endpoint
The architectural pattern I landed on: every major API action writes something to memory, and every downstream endpoint reads from it. The Hindsight framework describes this as making memory a first-class part of the agent loop rather than a logging side effect — which is exactly what I had to retrofit here after building it the wrong way first.
Gap analysis writes gaps and the candidate name, then persists to the user record:
save_memory(f"CANDIDATE_NAME: {gap_data.get('name', 'Candidate')}")
for skill in gap_data.get("skills_missing", []):
save_memory(f"Gap Identified: {skill}")upsert_user(
gap_data.get('name', 'Candidate'),
resume_summary=resume_text[:500],
gaps=gap_data.get("skills_missing", []),
)When a mock interview exam is passed, the verified skill writes back into both layers:
if result.get("passed"):
save_memory(f"VERIFIED_MASTERY: {target_topic}")
user = load_user(_make_user_id(cname))
if user:
verified = user.get("verified_skills", [])
if target_topic not in verified:
verified.append(target_topic)
upsert_user(cname, verified_skills=verified)That VERIFIED_MASTERY entry is what the resume evolution endpoint reads three steps later to inject certified skills into the exported PDF. One write, three downstream effects across three separate endpoints. This is the pattern the Vectorize agent memory page calls "memory-driven personalization" — where stored context actively changes what the agent does next, rather than just providing background.
Get Pooja M’s stories in your inbox
Join Medium for free to get updates from this writer.
The prefixes — CANDIDATE_NAME:, Gap Identified:, VERIFIED_MASTERY: — are not logging. They're the query interface that connects the entire backend. I named them carelessly at first, then had to refactor every endpoint when I changed one prefix. Treat them like an API contract from day one.
The Interview Endpoint: One Endpoint, Two Behaviors
The proctored interview was the trickiest piece. Ten questions, progressive difficulty, the interviewer never reacts to answers mid-exam, evaluation only at the end. I almost split this into /api/interview and /api/evaluate. One endpoint with behavioral branching turned out to be cleaner — one URL, one auth check, one error handler, one place to debug:
user_msgs = [m for m in request.messages
if isinstance(m, dict) and m.get('role') == 'user']
q_num = len(user_msgs) + 1if q_num > 10:
# Evaluation mode — review full history, return pass/fail
...
else:
if q_num <= 3:
level = "Foundational/Easy (Syntax, definitions, core concepts)"
elif q_num <= 9:
level = "Application-based/Moderate"
else:
level = "Deep Logic/Hard (Optimization, edge cases)"The system instruction enforces exam silence. Without this explicit constraint, LLaMA would praise correct answers and hint at wrong ones — completely undermining the proctoring and making the evaluation meaningless:
"CRITICAL — THIS IS AN EXAM: Do NOT give any feedback, hints, corrections, "
"praise, or commentary on the candidate's previous answer. "
"Immediately ask the next question without reacting to their response. "
"Silence on correctness is mandatory until the exam ends."State is derived entirely from message history length. No server-side session tracking, no separate state store. The tradeoff: if the frontend sends a malformed history, the question count is wrong. That’s a client contract problem, not a server problem — and it’s easier to enforce on the client than to add server-side session tracking for one piece of state.
LLM Output Is Not JSON Even When You Ask for JSON
Every endpoint that calls LLaMA runs this before parsing:
clean_json = re.sub(
r'^```json\s*|```$', '',
raw_response,
flags=re.MULTILINE
).strip()
result = json.loads(clean_json)Without it, roughly 40% of responses throw a parse error. LLaMA wraps JSON in markdown code blocks even when the prompt says “RETURN ONLY JSON with no extra text.” This is not a LLaMA-specific quirk — it happens across models. The fix is one line. The debugging time before I added it across six endpoints was around three hours total.
For /api/match-job I switched to response_format={"type": "json_object"} which Groq supports natively and eliminates the problem entirely. I should have used that everywhere from the start. The reason I didn't: I didn't know Groq supported it until halfway through the build.
How It Behaves Now
A returning user logs in. The backend calls check_user, finds their record, returns their stored roadmap, gap list, and verified skills. The frontend restores their previous state. The welcome header says "Welcome back, Pruthvi" because the backend returned is_new: false with the stored name attached.
A new user uploads resume and job description. Five gap entries and a candidate name write to _legacy. A user record creates in users. They follow the roadmap, complete a module, sit the interview, pass. VERIFIED_MASTERY: Docker appears in memory. Next session, resume evolution reads it and Docker shows up in the AI-Verified Certifications section of the exported PDF.
The limitation that remains: everything runs in one process against one file. Two concurrent gap analyses will race on memory.json writes. The atomic write prevents corruption but not interleaving — if two requests read the store simultaneously before either writes back, one overwrites the other's changes. For a hackathon with limited concurrent users this is acceptable. For production it needs per-user files, a write queue, or a real database.
Lessons Learned
Atomic writes aren’t optional when a file is your database. The .tmp → os.replace() pattern costs nothing and prevents file corruption. Add it before your first crash, not after.
Prefixes in a flat list are a query language — treat them that way. CANDIDATE_NAME:, Gap Identified:, VERIFIED_MASTERY: sound like log lines. They're actually the contract connecting six endpoints. Name them carefully and don't change them without updating every reader.
LLM outputs lie about their format. Always strip markdown fences before JSON parsing. Better yet, use response_format: json_object where the API supports it. Find out whether your API supports it on day one, not day five.
One endpoint with behavioral branching beats two endpoints with shared state. The interview’s dual behavior — examiner for questions 1–10, evaluator after — is cleaner than splitting. Less surface area, fewer moving parts.
Memory design is product design. Every save_memory() call I added required thinking about what three other endpoints would do with that data later. The memory layer isn't infrastructure — it's the contract the entire agent's behavior is built on. Get it wrong early and you refactor everything. Get it right and the agent actually learns.
Screenshots
Required Links
- 🔗 Project GitHub: github.com/pruthvibc/ai-career-internship-advisor
Every feature the frontend shows — the personalized welcome, the gap badges, the verified skill certificates, the live job matches — exists because the backend quietly wrote something to memory at the right moment.
That’s the job no one sees. That’s also the job that makes everything else possible.
