The Components of Agent Memory
Søren Kierkegaard wrote that life can only be understood backward, but it must be lived forward. AI agents face the exact same bind: they have to decide what is worth remembering in the moment, before they know what comes next. Here is a 4-minute visual explainer and engineering breakdown of Anthropic’s research on agent memory and offline “Dreaming.”
1. The Biological Division of Labor & The 4-Layer Stack
By agent memory, Anthropic refers specifically to knowledge an agent gains from experience and retains between sessions as readable context—excluding weight updates, single-conversation in-context learning, and context compaction summaries.
Just as the human brain splits memory formation between fast online capture in the hippocampus and offline consolidation during sleep replay in the neocortex, effective agent memory relies on a division of labor between active sessions and background sleep-time compute (“Dreaming”):
Fast Capture & Salience Predictions
- Immediate availability: Writes salient rules, observations, and current goals to a persistent memory folder during the active task.
- Memory as a prediction: Every note written in-session is a bet that a future session will need that verified fact or rule without re-deriving it.
- Trade-off: Writing detailed notes in the moment pulls attention and tokens away from the primary task, and early hypotheses can be wrong.
“Dreaming” Consolidation
- Ground-truth auditing: Replays raw session transcripts offline to verify in-session notes and fix hallucinated or drifted claims.
- Knowledge representations: Synthesizes scattered notes and files into linked Knowledge Wikis and repairs broken cross-references.
- Hierarchy maintenance: Promotes recurring rules into the top-level injected index while pushing verbose details down into linked files.
Across sessions, agent memory is organized into a four-layer hierarchy from raw ground truth at the bottom to preemptively injected context at the top:
Injected Context (MEMORY.md Header)
Preemptively loaded into every new session so the agent knows what exists without searching. Capped at 200 lines / 25 KB in Claude Code.
Memory Representations (Knowledge Wikis)
Interconnected topic pages built and maintained across sessions via Dreaming, giving the agent a navigable concept map.
Memory Notes
Verified rules, trap warnings, and goal dashboards (e.g., global.md) written by the agent during active sessions.
Session Transcripts
Complete, searchable raw event logs of every turn, action, and tool output—used as ground truth during offline Dreaming.
2. Memory as Prediction & Judging Salience
On Continual Learning Bench (CL-Bench) 1.0, Claude answers a sequence of database questions where each question starts in a fresh session and only a persistent memory directory carries over. While older models like Sonnet 4.6 wrote scattered files of untested guesses (rising only from 0.330 to 0.364), Claude Opus 5 (0.564 → 0.828) and Opus 5.5 (0.655 → 0.838) closed the full learning loop: recording failures, testing hypotheses, and saving validated rules with explicit trap warnings.
The Pokémon Red Salience Test (10,000 Steps)
To test how models judge what is worth saving over long horizons, Anthropic compared the memory directories of Claude Sonnet 3.7 and Claude Fable 5 at step 10,000 of Pokémon Red:
- Claude Sonnet 3.7 (Indiscriminate Hoarding — 413 Files, 225 Folders, 977,930 Chars → 1 Badge): Assumed everything might matter, creating separate files for every battle, item, and NPC. Because game state changes every turn, its detailed records went continuously out of date. At step 10,000, it spent 17 consecutive steps reorganizing a fourth “master” folder while taking zero actions in the game.
- Claude Fable 5 (Lean Dashboard — 10 Files, 2 Folders, 43,406 Chars → 4 Badges): Stored only non-obvious knowledge that could not be read off the screen. A top-level
global.mdfile acted as a goal dashboard, whilelocations/andgameplay/held tested heuristics (such as varying button-press wait timing after three Poké Balls in a row failed on a Magikarp). Opus 5.5 used this same lean pattern to beat the entire game.
3. Dreaming I: Auditing Notes Against Ground-Truth Transcripts
Because in-session notes are written before the agent sees how events play out, a single wrong note can poison future sessions. In Pokémon Red, Claude used the PokéCenter outside Mt. Moon as a landmark, but mistakenly wrote in its notes that the PokéCenter was on the east exit rather than the west entrance.
Walking out of the building, Claude announced “Mt Moon is behind me,” updated its note to Mt Moon DONE, and stepped straight off a one-way trap ledge that dropped it back onto the route it came from. Trusting its corrupted note, Claude jumped the one-way ledge three times, gave up (“I’m going to stop rather than keep looping”), and retreated all the way back to Pewter City.
When Anthropic ran an offline Dreaming pass on that memory directory, Claude checked its note against the raw session transcript, caught the east/west reversal, and rewrote the note. With the corrected memory, Claude walked straight back into Mt. Moon and reached Cerulean City at action 1,935.
4. Dreaming II: Knowledge Wikis, The Borges Map Trap & Parallel Collisions
Beyond fixing mistakes, Dreaming can synthesize raw files and notes into interconnected Knowledge Wikis:
- Scaling across 420M tokens (350,000 internal Stainless files): Scaling sequential Opus 5 Dreaming from 2 sessions to 128 sessions expanded the wiki from
103 pages / 766 linksto222 pages / 12,893 links. Answering from the wiki alone rose from13%to48%, while combining Wiki + Search reached61%(beating search alone at53%and saving ~20,000 tokens per question). - When Wikis beat search (Harvey LAB legal benchmark): On merger questions requiring multi-document synthesis and arithmetic, consulting the wiki jumped accuracy from
73%to87%. When answers lived in a single document, direct text search was already sufficient. - Pitfall 1 — The Borges Map Trap: On one task, a
115 KB(10-document) corpus bloated into a203 KB(68-page) wiki—1.77×larger than the raw files, with unlinked orphan pages and zero abstraction, like Jorge Luis Borges’s map as large as the empire it represented. - Pitfall 2 — Multi-Agent Write Collisions: When four parallel agents wrote contract summaries into a single shared
provisions.mdtable, they overwrote one another’s saves, retaining only 41 of 57 graded facts (72%). After reflecting on its failure, Claude rewrote the wiki instructions so each contract got its own dedicated page with verbatim quotes—preserving 57 of 57 facts (100%). A second Dreaming pass then repaired 9 broken links (repointingescrow-agreementtobuyer-form-escrow-agreement) and built an index page.
5. Summary: The Agent Memory Builder’s Playbook
| Architectural Component | Key Failure Mode | Engineering Best Practice |
|---|---|---|
| In-Session NotesLayer 2 · Fast Capture | Hoarding rapidly expiring screen state (Sonnet 3.7’s 413 files) or over-constraining frontier models with rigid rules (−0.234 drop). |
Maintain a compact global.md goal dashboard + tested heuristics. Tell frontier models what memory is for and let them judge salience. |
| Offline Dreaming AuditLayer 1 → Layer 2 | In-session predictions that are wrong from the start (Mt. Moon east/west reversal) or drift as the environment changes. | Schedule background Dreaming passes to validate frequently accessed or volatile notes against ground-truth session transcripts. |
| Knowledge WikisLayer 3 · Representations | Borges bloat (wiki growing 1.77× larger than source docs), parallel agent overwrite collisions, and dead cross-links. |
Pair Wikis with raw search for multi-document synthesis; assign 1 file per parallel agent; run a second Dreaming pass to repair links. |
| Injected ContextLayer 4 · Preemptive Index | Bloated instruction files that burn tokens on every turn and reduce overall instruction adherence. | Cap injected MEMORY.md at 200 lines / 25 KB as a lightweight routing index, pushing detailed notes down into linked Wiki files. |