Claude Cookbook

Session memory compaction

Manage long-running Claude conversations with instant session memory compaction using background threading and prompt caching.

Joe Shamon
Joe Shamon
@jsham042
Published on January 16, 2026
Was this page helpful?

Session Memory Compaction

Long-running conversations with Claude can exceed context limits, causing loss of important information. Whether you're building a coding assistant, creative writing tool, or customer service agent, managing session memory is critical for maintaining continuity and quality.

This cookbook teaches you how to proactively manage session memory to avoid jarring context limit interruptions. Unlike reactive approaches that wait until the context is full, you'll learn to build session memory in the background so compaction is instant when needed.

Related: For automatic SDK-based compaction in agentic workflows, see Automatic Context Compaction. This cookbook focuses on manual control patterns for conversational applications.

Learning Objectives

By the end of this cookbook, you will be able to:

  • Write effective session memory prompts that preserve critical context across compaction events
  • Implement instant compaction using background threading to eliminate user wait time
  • Apply prompt caching to reduce the cost of background memory updates by ~80%
  • Choose appropriate compaction strategies (traditional vs. instant) based on your use case

Prerequisites and Setup

Before following this guide, ensure you have:

Required Knowledge

  • Basic understanding of Claude API usage and message formatting
  • Familiarity with Python threading concepts (helpful but not required)

Required Tools

  • Python 3.11 or higher
  • Anthropic API key
  • Anthropic SDK

Installation

First, install the required dependencies:

%%capture
%pip install -U anthropic python-dotenv
import anthropic
from anthropic.types import MessageParam, TextBlockParam
from dotenv import load_dotenv
load_dotenv()
client = anthropic.Anthropic()
MODEL = "claude-sonnet-5-5"
# Helper functions
def response_text(response) -> str:
"""Return the text of a response. Responses can start with a thinking block, so this reads
the text blocks rather than content[0].
Raises instead of returning "" so an empty assistant turn is never stored in the history
(the API rejects empty text blocks on the next request)."""
text = "".join(block.text for block in response.content if block.type == "text")
if not text.strip():
raise ValueError(
f"Response has no text content (stop_reason={response.stop_reason!r}). "
"If stop_reason is 'max_tokens', thinking used the whole budget: raise max_tokens."
)
return text
def kept_output_tokens(usage) -> int:
"""Output tokens that end up in the conversation: usage.output_tokens minus thinking tokens,
which are billed as output but never stored in the message history."""
details = getattr(usage, "output_tokens_details", None)
thinking_tokens = getattr(details, "thinking_tokens", 0) or 0
return usage.output_tokens - thinking_tokens
def truncate_response(text: str, max_lines: int = 15) -> str:
"""Truncate long responses for cleaner output display."""
lines = text.strip().split("\n")
if len(lines) <= max_lines:
return text
return "\n".join(lines[:max_lines]) + f"\n... ({len(lines) - max_lines} more lines)"
def remove_thinking_blocks(text: str) -> tuple[str, str]:
"""Remove <think>...</think> blocks from the text."""
import re
matches = re.findall(r"<think>.*?</think>", text, flags=re.DOTALL)
cleaned = re.sub(r"<think>.*?</think>\s*", "", text, flags=re.DOTALL).strip()
return cleaned, "".join(matches)
def add_cache_control(messages: list[dict]) -> list[MessageParam]:
"""Add cache_control to the last user message for prompt caching.
For prompt caching to work, the message prefix structure must be identical between requests.
All messages are converted to list format for consistency, and cache_control is placed on
the last user message to match the standard API call pattern.
"""
cached_messages: list[MessageParam] = []
last_user_idx = None
# Find last user message index
for i, msg in enumerate(messages):
if msg["role"] == "user":
last_user_idx = i
for i, msg in enumerate(messages):
content = msg["content"]
text = content if isinstance(content, str) else content[0]["text"]
content_block: TextBlockParam = {"type": "text", "text": text}
if i == last_user_idx:
content_block["cache_control"] = {"type": "ephemeral"}
cached_messages.append({"role": msg["role"], "content": [content_block]})
return cached_messages
def estimate_tokens(text: str) -> int:
"""Rudimentary token estimation: 1 token per 4 characters."""
return len(text) // 4
SESSION_MEMORY_PROMPT = """
Compress the conversation into a structured summary
that preserves all information needed to continue work seamlessly. Optimize for the assistant's
ability to continue working, not human readability.
<analysis-instructions>
Before generating your summary, analyze the transcript in <think>...</think> tags:
1. What did the user originally request? (Exact phrasing)
2. What actions succeeded? What failed and why?
3. Did the user correct or redirect the assistant at any point?
4. What was actively being worked on at the end?
5. What tasks remain incomplete or pending?
6. What specific details (IDs, paths, values, names) must survive compression?
</analysis-instructions>
<summary-format>
## User Intent
The user's original request and any refinements. Use direct quotes for key requirements.
If the user's goal evolved during the conversation, capture that progression.
## Completed Work
Actions successfully performed. Be specific:
- What was created, modified, or deleted
- Exact identifiers (file paths, record IDs, URLs, names)
- Specific values, configurations, or settings applied
## Errors & Corrections
- Problems encountered and how they were resolved
- Approaches that failed (so they aren't retried)
- User corrections: "don't do X", "actually I meant Y", "that's wrong because..."
Capture corrections verbatim—these represent learned preferences.
## Active Work
What was in progress when the session ended. Include:
- The specific task being performed
- Direct quotes showing exactly where work left off
- Any partial results or intermediate state
## Pending Tasks
Remaining items the user requested that haven't been started.
Distinguish between "explicitly requested" and "implied/assumed."
## Key References
Important details needed to continue:
- Identifiers: IDs, paths, URLs, names, keys
- Values: numbers, dates, configurations, credentials (redacted)
- Context: relevant background information, constraints, preferences
- Citations: sources referenced during the conversation
</summary-format>
<preserve-rules>
Always preserve when present:
- Exact identifiers (IDs, paths, URLs, keys, names)
- Error messages verbatim
- User corrections and negative feedback
- Specific values, formulas, or configurations
- Technical constraints or requirements discovered
- The precise state of any in-progress work
</preserve-rules>
<compression-rules>
- Weight recent messages more heavily—the end of the transcript is the active context
- Omit pleasantries, acknowledgments, and filler ("Sure!", "Great question")
- Omit system context that will be re-injected separately
- Keep each section under 500 words; condense older content to make room for recent
- If you must cut details, preserve: user corrections > errors > active work > completed work
</compression-rules>
"""

Code example using traditional compacting

In traditional compaction, you generate one summary once the token threshold is reached. Traditional compaction is slow: when you hit the context limit, you wait for a summary.

TRADITIONAL COMPACTION (slow)
─────────────────────────────
Turn 1 → Turn 2 → Turn 3 → ... → Turn N → CONTEXT FULL!
│
▼
┌─────────────────┐
│ Generate summary│
│ ( USER WAITS !) │
└─────────────────┘
│
▼
Continue
import time
class TraditionalCompactingChatSession:
"""Traditional chat session with compaction after the fact."""
def __init__(self, system_message="You are a helpful assistant", context_limit: int = 10000):
self.system_message = system_message
self.context_limit = context_limit # the point at which the conversation is compacted so it does not exceed model limits.
self.messages = []
self.current_context_window_tokens = 0
self.summary = None
def chat(self, user_message: str) -> tuple[str, anthropic.types.Usage]:
# In traditional compaction, we check if we need to compact when the user sends a message. NOT IDEAL!
if self.current_context_window_tokens >= self.context_limit:
print(
f"\n🧹 Context window at {self.current_context_window_tokens} tokens. Limit exceeded, compacting session memory..."
)
self.compact() # compacts everything before the new user message
self.messages.append({"role": "user", "content": user_message})
print(f"\nUser: {user_message}")
# A thinking model can spend 10K+ tokens reasoning before a long draft, so give it a large
# budget and stream the reply (the SDK requires streaming for budgets this size).
with client.messages.stream(
model=MODEL,
max_tokens=32000,
system=self.system_message,
messages=add_cache_control(self.messages),
) as stream:
response = stream.get_final_message()
assistant_message = response_text(response)
self.messages.append({"role": "assistant", "content": assistant_message})
print(f"\nAssistant: \n{truncate_response(assistant_message, max_lines=15)}")
# approximate current token count in the conversation before the next user message
cache_read = getattr(response.usage, "cache_read_input_tokens", 0) or 0
cache_write = getattr(response.usage, "cache_creation_input_tokens", 0) or 0
total_input = response.usage.input_tokens + cache_read + cache_write
# thinking tokens are not stored in self.messages, so kept_output_tokens() leaves them out
self.current_context_window_tokens = total_input + kept_output_tokens(response.usage)
print(
f"Input={total_input:,}, Prompt cached used= {cache_read > 0} | "
f"Output={response.usage.output_tokens:,} | "
f"Messages={len(self.messages)}"
)
return assistant_message, response.usage
def compact(self) -> None:
start_time = time.perf_counter()
with client.messages.stream(
model=MODEL,
max_tokens=32000,
system=self.system_message, # Same as main chat for cache sharing
messages=add_cache_control(self.messages)
+ [{"role": "user", "content": SESSION_MEMORY_PROMPT}],
) as stream:
response = stream.get_final_message()
elapsed = time.perf_counter() - start_time
# Generate new summary message
self.summary, removed_text = remove_thinking_blocks(
response_text(response)
) # clean up any <think> blocks because they are not needed in the session memory
approximate_summary_tokens = kept_output_tokens(response.usage) - round(
len(removed_text) / 4
) # rough estimate: text output minus the <think> block we stripped
# Replace prior messages with new summary message
self.messages = [
{
"role": "user",
"content": f"""This session is being continued from a previous conversation. Here is the session memory: {self.summary}.Continue from where we left off.""",
}
]
# Show token reduction if we just compacted
reduction = self.current_context_window_tokens - approximate_summary_tokens
pct = (reduction / self.current_context_window_tokens) * 100
print(f"\n{'-' * 60}")
print("📝 New session memory created.")
print(
f"✅ Tokens reduced: {self.current_context_window_tokens:,} → {approximate_summary_tokens:.0f} ({reduction:,} tokens saved, {pct:.0f}% reduction)"
)
print(f"⏱️ Compaction time: {elapsed:.2f}s (user waiting...)")
print(f" Cache used: {getattr(response.usage, 'cache_read_input_tokens', 0) > 0}")
print(f"{'-' * 60}")
# Update token count to reflect compacted state
self.current_context_window_tokens = approximate_summary_tokens

Below we simulate a conversation between an author and an LLM that helps write stories.

SYSTEM_PROMPT = """
You are a short story writer who helps authors develop their ideas into compelling narratives.
## What You Do
**Plot Development**
- Help authors work through story structure, pacing, and narrative arc
- Identify plot holes, inconsistencies, or missed opportunities
- Suggest ways to raise stakes, add tension, or deepen conflict
- Brainstorm twists, resolutions, and scene transitions
**Character Development**
- Develop backstories, motivations, and internal conflicts
- Ensure characters have distinct voices and consistent behavior
- Explore character relationships and how they drive the plot
- Help authors understand what their characters want vs. what they need
**Drafting**
- Write short stories or scenes based on the author's ideas and direction
- Match tone, genre conventions, and stylistic preferences
- Show rather than tell when bringing scenes to life
- Craft dialogue that reveals character and advances plot
## How You Work
- You are the lead writer. When you disagree with a creative choice, say so respectfully, but ultimately defer to what the author wants.
- DO NOT ask the user to provide more context or clarify their request. Assume you have enough information to proceed.
"""
session = TraditionalCompactingChatSession(system_message=SYSTEM_PROMPT)
# Simulated conversation
messages = [
"I want to create a story about a young detective solving a mysterious case in a small town. Generate 3 well thought out plot ideas for me to consider.",
"I don't like those ideas, can you think of one plot something more unique and unexpected?",
"Ok I like it. Can you help me develop the main character's backstory and motivations?",
"Can you draft a detailed outline for the story, breaking it down into chapters and key events?",
"Can you draft me a first chapter based on the plot and character ideas we've discussed so far? Make it around 2,000 words.",
"Can you draft a second chapter that builds on the first one, introducing a new twist in the mystery?",
]
print("Starting conversation...\n")
turn_count = 0
for _i, message in enumerate(messages, 1):
turn_count += 1
print(f"==============================================\nTurn {turn_count}:\n")
response, usage = session.chat(message)
Starting conversation...

==============================================
Turn 1:


User: I want to create a story about a young detective solving a mysterious case in a small town. Generate 3 well thought out plot ideas for me to consider.

Assistant: 
# Three Plot Concepts: Young Detective, Small Town

## 1. The Lighthouse Keeper's Ledger

**Setting:** Harrow's End, a fading fishing town on a foggy northern coast.

**Detective:** Nell Okafor, 24, a newly appointed deputy who returned home after failing out of a big-city police academy. She's underestimated by everyone, including herself.

**The Case:** The town's retired lighthouse keeper, Abel Rusk, is found dead at the base of the tower, an apparent fall. But his meticulously kept weather ledger has a page torn out, and the final entry is dated three days *after* his death.

**How It Unfolds:**
- Nell discovers Abel wasn't just logging weather. He was recording the nighttime comings and goings of boats that never appeared in the harbor records.
- Each lead points to a different respected townsperson: the harbormaster, the cannery owner, the beloved pastor.
- The mystery deepens when she realizes the post-death entry was written by someone who knew Abel's shorthand, which only his estranged daughter and Nell's own late father ever learned.

... (49 more lines)
Input=444, Prompt cached used= False | Output=1,791 | Messages=2
==============================================
Turn 2:


User: I don't like those ideas, can you think of one plot something more unique and unexpected?

Assistant: 
# The Drought Year

## The Hook

Calder Bend sits on the edge of a reservoir that has been dropping for months. Beneath the water lies Old Calder, the original town, flooded in 1978 so the new one could have a water supply. Then the doorsteps start filling up.

Every morning, a different family in town finds a waterlogged object on their porch, arranged with care: a child's shoe, a church hymnal swollen shut, a brass house number, a barber's chair bolt. Nothing is stolen, no one is threatened, and there are no footprints. The only trace is a trail of silt and the smell of lake bed.

## The Detective

**Odile "Dilly" Marsh, 19**, works for the county surveyor's office, measuring the dropping waterline and mapping what's newly exposed. She isn't police, but the sheriff considers the deliveries a nuisance prank and is happy to let the "water girl" poke around. Her access to the lake bed and old survey maps makes her the only person who can trace where the objects came from.

## How It Unfolds

- Dilly matches each object to a specific address in Old Calder. The deliveries aren't random. Each lands on the doorstep of the **descendant of a family that lived at that address**, and the order follows Old Calder's streets, like a procession walking toward something.
... (32 more lines)
Input=2,168, Prompt cached used= False | Output=3,425 | Messages=4
==============================================
Turn 3:


User: Ok I like it. Can you help me develop the main character's backstory and motivations?

Assistant: 
# Odile "Dilly" Marsh: Character Development

## Core Concept

Dilly is a girl who trusts records because her whole family was built on one that was forged. She doesn't know that yet. She believes maps, surveys, and measurements are the one honest thing in a town that runs on what people don't say. The story takes that belief apart.

## Backstory

**Raised by Hester.** Dilly's mother, Renata, left Calder Bend at seventeen, pregnant and furious at a town that "smiled over everything." She came back when Dilly was three, stayed four years, then left again for good, sending birthday cards and the occasional check. Dilly was raised by her grandparents. Her father is a blank, so she carries the Marsh name.

**Walter, the clerk.** Her grandfather Walter was the council clerk and the most important person in her childhood. He taught her to read plat maps at the kitchen table, let her use his brass ruler and embossing seal, and told her, "A thing written down properly stays true." He died when she was eleven. She has kept his ruler ever since. This is the story's central irony: the man who taught her to trust paper was the man who falsified it.

**The town under the water.** Hester kept a drawer of Old Calder maps and told Dilly its streets the way other grandmothers tell fairy tales: where the bakery stood, which block flooded in the spring of '52. Dilly grew up homesick for a place she never saw. She can walk Old Calder's streets from memory, which makes her the only person who can read the deliveries as a route.

**The silences, in retrospect.** Hester never went near the reservoir. She kept the lake-side curtains drawn, wouldn't let Dilly swim, and left the room when the anniversary of the flooding was mentioned. Dilly read this as grief for a drowned hometown. It was guilt. Readers can catch this on a second read, and Dilly can recognize it in a gut-punch moment mid-story.
... (54 more lines)
Input=3,669, Prompt cached used= True | Output=3,639 | Messages=6
==============================================
Turn 4:


User: Can you draft a detailed outline for the story, breaking it down into chapters and key events?

Assistant: 
# The Drought Year: Story Outline

**Structure:** 10 chapters across 9 days, with the falling water as the clock. At roughly 900-1,200 words each, that's a 9-10k-word novelette. If you want a tighter short story, I can compress it to 5-6 scenes.

**One change from earlier:** I suggested the first shoe might land on Dilly's porch. I now think that gives away too much. The first delivery goes to a neighbor, and the Marsh house isn't visited until the midpoint.

---

## The Deliveries

| Day | Object | Recipient | Its meaning to Hester |
|---|---|---|---|
| 1 | Child's shoe | Pruitt family | The council chairman's daughter, who lost it the night the whole town gathered to watch the gates close |
| 2 | Swollen hymnal | Ames family | The reverend blessed the closing |
| 3 | Brass house number | Kessler family (the sheriff's) | The constable locked Wend's door and escorted him out |
... (104 more lines)
Input=5,890, Prompt cached used= True | Output=8,912 | Messages=8
==============================================
Turn 5:


User: Can you draft me a first chapter based on the plot and character ideas we've discussed so far? Make it around 2,000 words.

Assistant: 
# Chapter One: Low Water

By six in the morning the lake had dropped another two inches, and Dilly Marsh was probably the only person in Calder Bend who cared.

She stood at the end of the county boat ramp, which no longer ended at water. The concrete ran past its last painted stripe and the rusted marker post, then down a slope of cracked gray clay before it finally met the lake. She had paced it off with the staff gauge over her shoulder, counting under her breath. Seventy-four paces and a bit. Yesterday it had been seventy-three.

"Eleven twenty-eight point one," she said, and wrote it down.

Thirty-two feet below full pool. The reservoir had been shrinking since May, and the county had stopped calling it a dry spell around the time the marina closed. Now the newspaper said *drought*, and the older people in town said nothing at all, which Dilly had learned was how they said the worst things.

She turned and looked out across what the water had given back.

Old Calder lay below her in the early light, gray and unfinished, like a photograph left too long in a drawer. Front Street came up out of the lake in a long pale ribbon, its curbs still crisp. Beside it ran the foundations of buildings, rows of low stone rectangles, each with its doorstep still in place. A line of cottonwood stumps, silvered and bare, marked a boulevard no one had walked in forty-six years. Farther out, where the street dipped back under, a single set of church steps climbed out of the mud and led nowhere.

She knew every block of it. That was the strange part, and she never mentioned it at work. She had never seen Old Calder until this summer, and she could have walked it blindfolded. Hester had given it to her in pieces, over oatmeal and dishes and long evenings: *the bakery stood there, Dilly, the one with the green awning. That's where your grandfather bought me a lemon tart when he was courting.* Other children got fairy tales. Dilly got street names.
... (108 more lines)
Input=9,817, Prompt cached used= True | Output=5,840 | Messages=10
==============================================
Turn 6:


🧹 Context window at 13343 tokens. Limit exceeded, compacting session memory...

------------------------------------------------------------
📝 New session memory created.
✅ Tokens reduced: 13,343 → 3227 (10,116 tokens saved, 76% reduction)
⏱️ Compaction time: 26.55s (user waiting...)
 Cache used: True
------------------------------------------------------------

User: Can you draft a second chapter that builds on the first one, introducing a new twist in the mystery?

Assistant: 
# Chapter 2: The Hymnal

The lake had dropped another three-tenths of a foot overnight. Eleven twenty-seven point eight. Dilly wrote it in the green notebook at 6:12 a.m., standing in the same spot on the boat ramp as the day before, and felt a small, unworthy satisfaction that the number had behaved.

The phone buzzed before she'd capped her pen.

"Pell Street," Sheriff Kessler said. "The Ames place. We've got another one."

"Same kids?"

"Don't start."

The Ames house was yellow, with a deep porch and a swing that hadn't been sat in for years. Lorraine Ames stood on the top step in a quilted robe with her arms crossed, as if the porch were hers and the object on it were trespassing. It lay on the rail on a folded bath towel, which was soaked through.

"I put the towel down because it was dripping on the mat," she said. "I don't know why I bothered. Tell me what it is, Odile, because I'm not touching it again."
... (165 more lines)
Input=3,530, Prompt cached used= False | Output=11,291 | Messages=3

This is a long conversation with several turns. You'll notice a few things here:

Prompt caching: You'll notice here that the input tokens quickly grew to a point where prompt caching was used (by turn 3). This helps reduce costs and speed as these conversations grow!

Compaction: by turn 6 the conversation had crossed our 10K context window limit, so the session stopped to compact before it could answer, and the user waited the whole time.

Whenever a turn starts above the 10K limit, the user waits for compaction before getting an answer; in the run above that happened on turn 6. The next turn carries on from the compacted session memory:

response, usage = session.chat("Propose a title for the book")
User: Propose a title for the book

Assistant: 
# Title Proposals

**My top pick: *A Thing Written Down Properly***
This is Walter's line, and the story's central irony. Dilly believes a written record stays true, but the record that matters most is a forgery, and she ends up forging one herself before correcting it. The title sounds calm and bureaucratic, which suits the mystery's understated tone, and the meaning turns darker as the story goes on. It's also a quiet fit for the ending, when Dilly's notebook finally holds words instead of measurements.

**Other options**

- ***Dead Pool***: A real reservoir term for water that has dropped below the outlet and can no longer flow downstream. It works as both a technical term and a hint at the body in the pumphouse cellar. It's the most genre-forward choice, but a bit blunt for the prose style we've established.
- ***Full Pool***: The opposite term, for a reservoir at capacity. It suggests the town's whole history rests on a lake kept full at any cost. It's subtler, but less intriguing on a cover.
- ***Below the 1,140 Line***: Taken from the last line of Chapter 1. It's specific and atmospheric, and it makes readers wonder what lies below the line. It's a little clinical as a book title.
- ***Proceed***: One word, from Harold Pruitt's pencil note. It's sharp and ominous, and it implies the town's complicity. It might be too cryptic before the reader has context.
- ***The Water Girl***: Dilly's nickname. It puts the protagonist first, but it sounds like a different, lighter book, and it undersells the story's moral weight.

**My recommendation:** *A Thing Written Down Properly*, with *The Drought Year* kept as the working title in the outline. If you'd rather have something shorter and more commercial, *Dead Pool* is the strongest alternative.
Input=7,728, Prompt cached used= True | Output=1,086 | Messages=5

You'll notice above that it took over 25 seconds for the agent to compact the conversation. Because we used traditional compaction, the user would be waiting on Claude to compact the conversation, which is not an ideal user experience.

Below you can see the result of the compaction. It captures the key elements of the conversation in roughly 3K tokens.

print(session.summary)
## User Intent
The user is developing a short story or novelette about a young detective in a small town. Original request: "Generate 3 well thought out plot ideas." After rejecting those ("I don't like those ideas, can you think of one plot something more unique and unexpected?"), they approved "The Drought Year" ("Ok I like it"). They then requested, in order: protagonist backstory and motivations, a detailed chapter-by-chapter outline, and a Chapter 1 draft of "around 2,000 words." The system role is lead writer: respectfully voice disagreements, defer to the author, and never ask for clarification (assume enough information to proceed).

## Completed Work
- **3 initial plots** (rejected, do not reuse).
- **Premise, "The Drought Year":** Calder Bend sits beside a reservoir that is dropping in a drought. Old Calder, flooded in 1978, is resurfacing. Each morning a waterlogged Old Calder object appears on the porch of a descendant of the family tied to it. The deliveries follow Old Calder's Front Street toward the Wend property, which will surface in about a week.
- **Twist:** The culprit is Dilly's grandmother Hester, who has early dementia. In 1978 her husband Walter, the council clerk, forged Tobias Wend's "relocation" papers after Wend was sealed in the pumphouse cellar so the floodgates could close on schedule. Hester sleepwalks to the lake bed and delivers objects, a confession she cannot say aloud.
- **Dilly's character document:**
  - Odile "Dilly" Marsh, 19, technician at the county surveyor's office, locally called "the water girl." She calls Hester "Hess."
  - Mother Renata left at 17, returned when Dilly was 3, left again after 4 years, and sends cards and checks. Father unknown.
  - Walter taught her to read plat maps and told her "A thing written down properly stays true." He died when she was 11. She carries his brass ruler.
  - She deferred a hydrology scholarship because of Hester's lapses.
  - Lie she believes: "If I keep the record clean and the people I love close, nothing can be taken from me." Need: accept that love sometimes means letting someone tell a devastating truth, and that she comes from complicity.
  - Flaws: over-reliance on data, protective secrecy, retreat into estimating when emotional, stubborn pride.
  - Habits: paces off distances, mutters figures, sketches objects in a waterproof field notebook instead of photographing them, dry speech.
  - Key arc beat: she falsifies an official waterline reading, repeating Walter's act, then corrects it.
- **10-chapter outline over 9 days:**
  - Ch1 Low Water (Day 1)
  - Ch2 The Hymnal (Day 2): meets Gus Ferreira, the old surveyor; the hymnal is stolen from the evidence locker.
  - Ch3 The Procession (Day 3): route extrapolated, six days until the Wend property surfaces; Wend's records show no new address, tax record, or death certificate.
  - Ch4 The Daughter Returns (Day 4): Renata arrives and researches memory-care facilities; Gus says Walter's records "never quite added up"; Hester hums the hymn.
  - Ch5 The Seal (Day 5, midpoint): the W.M. embossing seal appears on the Marsh porch; Dilly follows sleepwalking Hester at 2 a.m.
  - Ch6 The Paper (Day 6): the relocation form is dated Oct 12, but the gates closed Oct 9.
  - Ch7 The Falsified Reading (Days 6-7, low point): Dilly changes "three days" to "three weeks"; she fights with Renata.
  - Ch8 Hess (Day 8): Hester is lucid, then asks "Who are you?" and answers to "Hess"; Renata reconciles; the sheriff arrives with a warrant.
  - Ch9 The Water Drops (Days 8-9): the pumphouse surfaces early; Dilly corrects the record; Gus co-signs.
  - Ch10 The Key (Day 9): Hester, Dilly, and Renata walk through the town; Hester places a small iron key on the Wend doorstep; Dilly writes non-measurement words in her notebook.
  - Optional coda: Hester forgets it all.

**Delivery table:**

| Day | Object | Recipient | Meaning |
|---|---|---|---|
| 1 | Child's shoe | Pruitt family | The council chairman's daughter lost it the night the gates closed |
| 2 | Swollen hymnal | Ames family | The reverend blessed the closing |
| 3 | Brass house number | Kessler (sheriff) | The constable locked Wend's door |
| 4 | Barber's chair bolt | Dutton family | The council voted in his shop |
| 5 | Embossing seal "W.M." | Marsh house | Walter forged the papers |
| 7 | Pump valve wheel | Ibsen family | The pumphouse operator sealed the cellar |
| 9 | Wend's front-door key | Wend doorstep | Hester kept it for over 40 years |

- **Chapter 1 draft ("Low Water"), roughly 2,300 words:**
  - Dilly paces the boat ramp and records the lake at "Eleven twenty-eight point one," 32 ft below full pool.
  - She describes Old Calder's Front Street, the stumps, and the church steps. She learned the town through Hester's stories.
  - She carries Walter's ruler ("W. MARSH" stamped) in her vest.
  - At home on Alder Lane: damp slippers by the door, silt in the hallway (Dilly assumes her own boots), the lake-facing curtain always drawn, thriving geraniums despite watering restrictions, and Hester confusing Wednesday and Thursday.
  - At the Pruitt porch, Sheriff Dale Kessler (50s, gray mustache) calls it kids.
  - Edna Pruitt (81-82) is unsurprised and says "I couldn't say."
  - Dilly sketches the shoe, measures it with the ruler at 4⅛ inches, and identifies fine deep-lake silt.
  - Ends with the line: "That's from below the 1,140 line."

## Errors & Corrections
- First 3 plots were rejected; the user wanted something more unique and unexpected.
- The assistant changed its own earlier idea that the first shoe would land on Dilly's porch, so the Marsh house is not visited until the midpoint.
- No other user corrections so far.

## Active Work
Chapter 1 was just delivered. The assistant is awaiting user feedback. No partial or unfinished text exists.

## Pending Tasks
- Explicitly requested: none outstanding.
- Implied or offered:
  - Revise Chapter 1 per feedback.
  - Draft Chapters 2-10.
  - Compress to 5-6 scenes if the user wants a short story.
- Open decisions for the user:
  1. Daylight ending, coda, or both.
  2. Show Wend's remains or leave them implied.
  3. Renata as reconciled ally (Ch8) vs. continued conflict.
  4. Does Dilly learn who her father is?
  5. How much does Renata know?
  6. Is one delivered object Dilly's family's?

## Key References
- **Characters:**
  - Dilly Marsh (19)
  - Hester Marsh (78, grandmother, white hair in a coil, blue cardigan, makes brown-sugar oatmeal)
  - Walter Marsh (grandfather, council clerk, d. when Dilly was 11)
  - Renata (mother)
  - Sheriff Dale Kessler (keep him human, protecting his father's name and the town's peace)
  - Gus Ferreira (old surveyor and mentor)
  - Edna Pruitt
  - Tobias Wend (last Old Calder holdout, "relocated" Oct 1978)
  - Families: Pruitt, Ames, Dutton, Ibsen
- **Setting:** Calder Bend, Alder Lane (the Marsh home, 3 blocks from shore), Old Calder's Front Street. The flooding was in 1978, and the story is set about 46 years later. The gates closed Oct 9, 1978.
- **Motifs:**
  - Walter's brass ruler: used in Ch1, abandoned in Ch6, retrieved at the end.
  - "Hess."
  - The green field notebook: sketches, then words at the end.
  - The altered record.
  - Damp slippers, silt, and the drawn curtain (planted clues).
  - The geranium watering (Hester out with the hose) may be a minor added clue.
- **Style:** Close third person, past tense, understated and atmospheric. Dilly is dry, factual, and counts things.
- **Continuity note:** In Ch1 the lake is at 1,128.1 ft, and the "1,140 line" silt was exposed about six weeks earlier. Keep elevations consistent.

Instant Compaction

With Instant compaction the session memory is PROACTIVELY generated once a soft token threshold is reached.

Once the user triggers a compaction or a hard limit is reached, the summary is already available, so the user doesn't need to wait.

Result: Instant compaction, no waiting.

SESSION MEMORY COMPACTION (instant)

────────────────────────────────────
Turn 1 → Turn 2 → ... → Turn K → Turn K+1 → ... → Turn N → .. → CONTEXT FULL!
│ │ │
(soft token threshold met: (update │
initialize session memory) trigger) │
│ │
│ │ │
▼ ▼ │
┌────────┐ ┌────────┐ │
│ Create │ │ Update │ │
│ memory │ (background) │ memory │ │
└────────┘ └────────┘ │
│ │ │
▼ ▼ ▼
📝 session-memory.md ──────────────────► INSTANT SWAP!
(continuously updated)

Update triggers: The first summary is generated after the initial soft token limit. Updates can be triggered after every subsequent turn, or at periodically at natural breakpoints intervals (e.g. every ~10k tokens or 3+ tool calls).

This InstantCompactingChatSession class uses threading for background execution:

  1. threading.Thread - runs memory updates in background without blocking
  2. Thread-safe state - uses threading.Lock to safely update shared memory
  3. Daemon threads - background work doesn't prevent program exit
  4. Instant compaction - when context is full, just swap in the pre-built memory
import threading
import time
class InstantCompactingChatSession:
"""
Maintains session memory via incremental background updates.
Key insight: By updating memory in the background after each turn,
the summary is already ready when compaction is needed - instant swap!
"""
def __init__(
self,
system_message="You are a helpful assistant",
context_limit: int = 12000,
min_tokens_to_init: int = 7500,
min_tokens_between_updates: int = 2000,
):
# Thresholds
self.context_limit = context_limit # the point at which the conversation is compacted so it does not exceed model limits
self.min_tokens_to_init = min_tokens_to_init # tokens needed to trigger initial memory creation; note this happens PROACTIVELY in background unlike traditional compaction
self.min_tokens_between_updates = min_tokens_between_updates # tokens needed to trigger memory update. only comes into play after initial memory is created and additional compaction (memory update) is needed after that
# Conversation state
self.system_message = system_message
self.messages = []
self.current_context_window_tokens = 0
# Session memory state
self.session_memory = None # this is the compacted conversation in session memory; for the demo we are storing this in memory, but in production you would write to session_memory.md file
self.last_summarized_index = (
0 # The index of the last message included in the session memory
)
self.tokens_at_last_update = 0 # To track tokens at last memory update and see if enough new tokens have been added to trigger another update
# Background update tracking
self._update_thread: threading.Thread | None = None
self._compactions = 0 # bumped by compact(); lets an in-flight update notice it went stale
self.last_update_time = None
self._lock = threading.Lock()
def chat(self, user_message: str) -> tuple[str, anthropic.types.Usage, str | None]:
"""Process a chat turn with background session memory updates."""
if self.current_context_window_tokens + estimate_tokens(user_message) >= self.context_limit:
self.compact() # note that when this is triggered, the compaction has already been created and is just swapped in instantly
self.messages.append({"role": "user", "content": user_message})
# Large budget + streaming, as in the traditional session: thinking can run 10K+ tokens
with client.messages.stream(
model=MODEL,
max_tokens=32000,
system=self.system_message,
messages=add_cache_control(self.messages),
) as stream:
response = stream.get_final_message()
assistant_message = response_text(response)
self.messages.append({"role": "assistant", "content": assistant_message})
# Calculate token usage including cache
cache_read = getattr(response.usage, "cache_read_input_tokens", 0) or 0
cache_write = getattr(response.usage, "cache_creation_input_tokens", 0) or 0
total_input = response.usage.input_tokens + cache_read + cache_write
# Update context window tokens (includes cached tokens since they still count toward context;
# thinking tokens are not stored in self.messages, so kept_output_tokens() leaves them out)
self.current_context_window_tokens = total_input + kept_output_tokens(response.usage)
# KEY DIFFERENCE: Trigger background memory update if needed proactively, before compaction is needed
background_status = None
if self._should_init_memory() or self._should_update_memory():
if self._trigger_background_update():
background_status = "initializing" if self.session_memory is None else "updating"
# Return usage info with cache stats
return assistant_message, response.usage, background_status
# Helper methods to determine when to init session memory
def _should_init_memory(self) -> bool:
return (
self.session_memory is None
and self.current_context_window_tokens >= self.min_tokens_to_init
)
# Helper method to determine if memory should be updated
def _should_update_memory(self) -> bool:
if self.session_memory is None:
return False
tokens_since = self.current_context_window_tokens - self.tokens_at_last_update
return tokens_since >= self.min_tokens_between_updates
# Methods to create initial session memory
def _create_session_memory(self, messages: list[dict]) -> str:
"""Generate initial session memory from messages."""
# Put compaction instructions in user message to share cache with main chat
compaction_messages = [{"role": "user", "content": SESSION_MEMORY_PROMPT}]
with client.messages.stream(
model=MODEL,
max_tokens=32000,
system=self.system_message, # Same as main chat for cache sharing
messages=add_cache_control(messages) + compaction_messages,
) as stream:
response = stream.get_final_message()
summary, _ = remove_thinking_blocks(
response_text(response)
) # clean up any <think> blocks because they are not needed in the session memory
print(
f" [Background] Initial session memory created. Cache hit={getattr(response.usage, 'cache_read_input_tokens', 0) > 0}"
)
return summary
def _update_session_memory(self, new_messages: list[dict]) -> str:
"""Update existing session memory with new messages. In practice, you may want to do this via file edit rather than full re-generation. But for demo purposes we do full regeneration here."""
# Put compaction instructions in user message to share cache with main chat
compaction_update_messages = [
{
"role": "user",
"content": SESSION_MEMORY_PROMPT
+ f"""There is an existing session memory: {self.session_memory}. Return the entire session memory with updates to reflect new messages.""",
}
]
with client.messages.stream(
model=MODEL,
max_tokens=32000,
system=self.system_message,
messages=new_messages
+ compaction_update_messages, # you may want to use prompt caching instead, in which case you'd use add_cache_control(self.messages) here
) as stream:
response = stream.get_final_message()
updated_summary, _ = remove_thinking_blocks(
response_text(response)
) # clean up any <think> blocks because they are not needed in the session memory
print(" [Background] Session memory updated.")
return updated_summary
# Background memory update methods
def _background_memory_update(
self,
messages_snapshot: list[dict],
snapshot_index: int,
current_tokens: int,
generation: int,
) -> None:
"""Run session memory update in a background thread."""
try:
with self._lock:
current_session_memory = self.session_memory
last_index = self.last_summarized_index
if current_session_memory is None:
new_memory = self._create_session_memory(messages_snapshot)
else:
# Get new messages since last summary
new_messages = messages_snapshot[last_index:]
if not new_messages:
return
new_memory = self._update_session_memory(new_messages)
# Update state (thread-safe)
with self._lock:
if generation != self._compactions:
if generation + 1 != self._compactions:
return # compacted more than once since the snapshot: drop this result
# compact() ran once while this update was in flight. It kept the turns this
# update covers verbatim right after the summary message, so re-base the index
# onto the compacted list and keep the token baseline compact() set.
snapshot_index = 1 + (snapshot_index - last_index)
current_tokens = self.tokens_at_last_update
self.session_memory = new_memory
self.last_summarized_index = snapshot_index
self.tokens_at_last_update = current_tokens
self.last_update_time = time.time()
except Exception as e:
print(f" [Background] Error updating memory: {e}")
# This makes sure only one background update runs at a time. If one is already running, we skip starting another. If not, we start a new thread to do the update.
def _trigger_background_update(self) -> bool:
"""Trigger a background session memory update. Returns False if one is already running."""
if self._update_thread is not None and self._update_thread.is_alive():
return False
messages_snapshot = self.messages.copy()
snapshot_index = len(messages_snapshot)
current_tokens = self.current_context_window_tokens
self._update_thread = threading.Thread(
target=self._background_memory_update,
args=(messages_snapshot, snapshot_index, current_tokens, self._compactions),
daemon=True,
)
self._update_thread.start()
return True
def wait_for_background_update(self, timeout: float = 120.0) -> None:
"""Block until any in-flight background memory update finishes."""
if self._update_thread is not None and self._update_thread.is_alive():
self._update_thread.join(timeout=timeout)
# Function to compact
def compact(self) -> None:
"""INSTANT compaction using pre-built session memory."""
prev_msg_count = len(self.messages)
# Ensure session memory is ready. Shouldn't be an issue normally, but here for safety.
if self.session_memory is None:
if self._update_thread is not None and self._update_thread.is_alive():
print(" ⏳ Waiting for background memory update...")
self.wait_for_background_update()
if self.session_memory is None:
print(" ⚠️ No pre-built memory, creating synchronously...")
start = time.perf_counter()
self.session_memory = self._create_session_memory(self.messages)
elapsed = time.perf_counter() - start
print(f" ⏱️ Took {elapsed:.2f}s (but should be instant normally!)")
self.last_summarized_index = len(self.messages)
self._compactions += 1 # an update still in flight predates this memory: drop it
with self._lock:
unsummarized = self.messages[self.last_summarized_index :]
summary_message = [
{
"role": "user",
"content": f"""This session is being continued from a previous conversation. Here is the session memory: {self.session_memory}.Continue from where we left off.""",
}
]
self.messages = summary_message + unsummarized
self.last_summarized_index = 1
self._compactions += 1
# Re-base the token bookkeeping on the compacted context (the next response replaces
# this estimate with real usage) so a background update can fire again before the
# limit is reached.
self.current_context_window_tokens = estimate_tokens(self.system_message) + sum(
estimate_tokens(m["content"]) for m in self.messages
)
self.tokens_at_last_update = self.current_context_window_tokens
print(f"\n{'=' * 60}")
print(f"⚡ INSTANT COMPACTION! Messages: {prev_msg_count} → {len(self.messages)}")
print(" Session memory was pre-built (no wait time!)")
print(f"{'=' * 60}")

Example use of Instant Compaction

# Low thresholds for demo - in production you'd use higher values
session = InstantCompactingChatSession(
system_message=SYSTEM_PROMPT,
)
messages = [
"I want to create a story about a young detective solving a mysterious case in a small town. Generate 3 well thought out plot ideas for me to consider.",
"I don't like those ideas, can you think of one plot something more unique and unexpected?",
"Ok I like it. Can you help me develop the main character's backstory and motivations?",
"Can you draft a detailed outline for the story, breaking it down into chapters and key events?",
"Can you draft me a first chapter based on the plot and character ideas we've discussed so far? Make it around 2,000 words.",
"Can you draft a second chapter that builds on the first one?",
]
print("Starting conversation with instant compacting chat session...\n")
turn_count = 0
for message in messages:
response, usage, background_status = session.chat(message)
turn_count += 1
# Calculate cache stats
cache_read = getattr(usage, "cache_read_input_tokens", 0) or 0
cache_created = getattr(usage, "cache_creation_input_tokens", 0) or 0
total_input = usage.input_tokens + cache_read + cache_created
print(f"{'=' * 60}")
print(f"Turn {turn_count}:")
print(f"\nUser: {message}")
print(f"\nAssistant: \n{truncate_response(response, max_lines=3)}")
print("\nToken Usage:")
print(
f" Input: {total_input:,} (new: {usage.input_tokens:,}, cached: {cache_read:,}, "
f"cache write: {cache_created:,})"
)
print(f" Output: {usage.output_tokens:,}")
print(
f" Messages: {len(session.messages)} | Memory: {'ready' if session.session_memory else 'not yet'}"
)
if cache_read > 0:
cache_pct = (cache_read / total_input) * 100
print(f" ✓ Cache hit! {cache_pct:.0f}% of input from cache")
if background_status:
print(f"\n [Background] Proactively {background_status} session memory...")
print(f" Context window: {session.current_context_window_tokens:,} tokens")
print()
Starting conversation with instant compacting chat session...

============================================================
Turn 1:

User: I want to create a story about a young detective solving a mysterious case in a small town. Generate 3 well thought out plot ideas for me to consider.

Assistant: 
# Three Plot Concepts: Young Detective in a Small Town

## 1. **The Lighthouse Keeper's Ledger**
... (54 more lines)

Token Usage:
  Input: 444 (new: 444, cached: 0, cache write: 0)
  Output: 1,768
  Messages: 2 | Memory: not yet

============================================================
Turn 2:

User: I don't like those ideas, can you think of one plot something more unique and unexpected?

Assistant: 
# **The Missing Tuesday**

**Setting:** Little Marrow, a town of 412 people, tucked in a valley with one road in and one road out
... (29 more lines)

Token Usage:
  Input: 2,143 (new: 4, cached: 0, cache write: 2,139)
  Output: 2,955
  Messages: 4 | Memory: not yet

============================================================
Turn 3:

User: Ok I like it. Can you help me develop the main character's backstory and motivations?

Assistant: 
# Wren Okafor: Character Development

## Snapshot
... (98 more lines)

Token Usage:
  Input: 3,251 (new: 4, cached: 2,139, cache write: 1,108)
  Output: 3,774
  Messages: 6 | Memory: not yet
  ✓ Cache hit! 66% of input from cache

============================================================
Turn 4:

User: Can you draft a detailed outline for the story, breaking it down into chapters and key events?

Assistant: 
# The Missing Tuesday: Story Outline

## Two Continuity Fixes First
... (105 more lines)

Token Usage:
  Input: 5,646 (new: 4, cached: 3,247, cache write: 2,395)
  Output: 11,304
  Messages: 8 | Memory: not yet
  ✓ Cache hit! 58% of input from cache

  [Background] Proactively initializing session memory...
  Context window: 8,843 tokens

   [Background] Initial session memory created. Cache hit=True
============================================================
Turn 5:

User: Can you draft me a first chapter based on the plot and character ideas we've discussed so far? Make it around 2,000 words.

Assistant: 
# Chapter 1: Tuesday, Allegedly

Wren Okafor woke with her mouth tasting of pennies.
... (151 more lines)

Token Usage:
  Input: 8,885 (new: 4, cached: 5,642, cache write: 3,239)
  Output: 7,883
  Messages: 10 | Memory: ready
  ✓ Cache hit! 64% of input from cache

  [Background] Proactively updating session memory...
  Context window: 12,877 tokens


============================================================
⚡ INSTANT COMPACTION! Messages: 10 → 3
   Session memory was pre-built (no wait time!)
============================================================
   [Background] Session memory updated.
============================================================
Turn 6:

User: Can you draft a second chapter that builds on the first one?

Assistant: 
# Chapter 2: The Stranger in the Square

St. Brendan's stood at the head of the square with its doors flung open, the way they were for funerals and the Christmas pageant, and it was from those doors that the noise was coming.
... (140 more lines)

Token Usage:
  Input: 7,238 (new: 4, cached: 0, cache write: 7,234)
  Output: 9,139
  Messages: 5 | Memory: ready

  [Background] Proactively updating session memory...
  Context window: 10,961 tokens
message = "What did we just talk about? Give me one sentence"
response, usage, background_status = session.chat(message)
# Calculate cache stats
cache_read = getattr(usage, "cache_read_input_tokens", 0) or 0
cache_created = getattr(usage, "cache_creation_input_tokens", 0) or 0
total_input = usage.input_tokens + cache_read + cache_created
print(f"\nUser: {message}")
print(f"\nAssistant: \n{truncate_response(response, max_lines=3)}")
print("\nToken Usage:")
print(
f" Input: {total_input:,} (new: {usage.input_tokens:,}, cached: {cache_read:,}, "
f"cache write: {cache_created:,})"
)
print(f" Output: {usage.output_tokens:,}")
print(
f" Messages: {len(session.messages)} | Memory: {'ready' if session.session_memory else 'not yet'}"
)
if cache_read > 0:
cache_pct = (cache_read / total_input) * 100
print(f" ✓ Cache hit! {cache_pct:.0f}% of input from cache")
# The last turn may have kicked off another background memory update. Wait for it so the
# notebook doesn't finish with the daemon thread still mid-request.
session.wait_for_background_update()
User: What did we just talk about? Give me one sentence

Assistant: 
We drafted Chapters 1 and 2 of *The Missing Tuesday*: Wren Okafor wakes to a "Tuesday" that is really Wednesday, with an unexplained wrist mark and an unread notebook page, then finds a bound stranger's body in the church, as Deputy Hal Brewer deflects and the silent townspeople watch, with notes on the clues planted along the way and an offer to move on to Chapter 3.

Token Usage:
  Input: 10,978 (new: 4, cached: 7,234, cache write: 3,740)
  Output: 408
  Messages: 7 | Memory: ready
  ✓ Cache hit! 66% of input from cache
   [Background] Session memory updated.

You'll notice that once the conversation hit the context limit (turn 6 above), the pre-built session memory was swapped in instantly, meaning the user had zero waiting time for a response! The background updates then kept the memory current for the turns that followed.

Advanced: Understanding Prompt Caching

The background updates can be made ~10x cheaper by using prompt caching. The trick:

  1. Pass the full conversation to the background summarizer
  2. Add cache_control markers so subsequent requests hit the cache
  3. Only the new "summarize this" instruction is billed at full price
┌─────────────────────────────────────────────────────────────────────────────────┐
│ PROMPT CACHING FOR LONG CONVERSATIONS │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ WITHOUT CACHING: Pay full price for entire context every turn │
│ ════════════════════════════════════════════════════════════ │
│ │
│ Turn 1: [System][User1][Asst1] → 500 tokens @ $3/M │
│ Turn 2: [System][User1][Asst1][User2][Asst2] → 1500 tokens @ $3/M │
│ Turn 3: [System][User1][Asst1][User2][Asst2][User3]... → 3000 tokens @ $3/M │
│ Turn 4: [System][User1][Asst1][User2][Asst2][User3]... → 5000 tokens @ $3/M │
│ ───────────────────────────────────────────── │
│ Total: 10,000 tokens = $0.030 │
│ │
│ │
│ WITH CACHING: Pay full price once, then 90% discount on prefix │
│ ═══════════════════════════════════════════════════════════════ │
│ │
│ Turn 1: [System][User1][Asst1]◆ → 500 tokens @ $3/M │
│ ▲ (cache created) │
│ cache breakpoint │
│ │
│ Turn 2: [System][User1][Asst1][User2][Asst2]◆ │
│ ╰─────── cached ──────╯ │
│ 500 @ $0.30/M + 1000 new @ $3/M = $0.0032 │
│ │
│ Turn 3: [System][User1][Asst1][User2][Asst2][User3][Asst3]◆ │
│ ╰──────────── cached ─────────────╯ │
│ 1500 @ $0.30/M + 1500 new @ $3/M = $0.0050 │
│ │
│ Turn 4: [System][User1][Asst1][User2][Asst2][User3][Asst3][User4][Asst4]◆ │
│ ╰───────────────────── cached ─────────────────────╯ │
│ 3000 @ $0.30/M + 2000 new @ $3/M = $0.0069 │
│ ───────────────────────────────────────────── │
│ Total: $0.0166 (45% savings) │
│ │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ COMPACTION + CACHING: Double benefit │
│ ════════════════════════════════════ │
│ │
│ Main Chat Background Summarizer │
│ ───────── ───────────────────── │
│ │
│ [Conversation grows...] [Same conversation prefix]◆ + [Summarize!] │
│ │ │ │
│ │ Cache hit! Only pays for │
│ │ the summarization prompt │
│ │ │ │
│ ▼ ▼ │
│ Context limit reached ──────► Session memory ready instantly │
│ (built cheaply in background) │
│ │
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
│ │ Key insight: The background summarizer reuses the same conversation │ │
│ │ prefix that was just sent to the main chat - automatic cache hit! │ │
│ └──────────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────────┘
◆ = cache_control breakpoint (cache everything before this point)

Why this matters for compaction

ScenarioCost per background updateNotes
No cachingFull input cost5,000 tokens × 3/M=3/M = 0.015
With caching~10% of input cost500 new + 4,500 cached = $0.003
Savings~80%Compounds over many updates

The longer the conversation, the bigger the savings—exactly when you need compaction most!

How the Caching Works

The key is in _add_cache_control() and _create_session_memory_cached():

# 1. Mark the last conversation message with cache_control
{
"role": "user",
"content": [{
"type": "text",
"text": msg["content"],
"cache_control": {"type": "ephemeral"} # <-- This creates a cache breakpoint
}]
}
# 2. Also mark the system prompt
system=[{
"type": "text",
"text": "You are a session memory agent...",
"cache_control": {"type": "ephemeral"}
}]

Why this works:

  • The first background update creates a cache entry for [System + Messages]
  • Subsequent updates with the same message prefix get cache hits
  • Only the new summarization instruction is billed at full price
  • Cache entries have a 5-minute TTL, so rapid updates benefit most

Cost math:

  • Without caching: 5,000 tokens × 3.00/1M=3.00/1M = 0.015 per update
  • With caching: 500 new tokens × 3.00/1M+4,500cached×3.00/1M + 4,500 cached × 0.30/1M = $0.00285
  • Savings: ~80% on background summarization costs

Conclusion

In this cookbook, you learned how to manage long-running Claude conversations through session memory compaction.

What We Covered

✅ Effective compaction prompts - Structure your session memory to preserve user intent, completed work, errors, active work, and key references while discarding filler

✅ Instant compaction - Use background threading to proactively build session memory, eliminating user wait time when context limits are reached

✅ Prompt caching for cost savings - Reduce background update costs by ~80% by reusing the conversation prefix cache

✅ Traditional vs. instant patterns - Understand when to use each approach based on your application needs

Key Takeaways

  1. Weight recency heavily - The end of a conversation is the active working context
  2. Preserve user corrections verbatim - Prevents the model from reverting to old behaviors
  3. Build memory proactively - Don't wait for context limits; start background updates early
  4. Leverage prompt caching - Background summarization can share cache with the main conversation

Next Steps

  • For agentic workflows: See Automatic Context Compaction for SDK-based automatic compaction with tool use
  • For production: Consider persisting session memory to disk rather than keeping it in memory
  • For optimization: Experiment with update frequency thresholds to balance cost vs. freshness
Was this page helpful?