Context management: what stays in the window, what gets dropped
How do AI agents decide what to keep in their context window and what to summarize, drop, or retrieve later?
Every long-running agent hits the same wall: the context window is finite, but the task, the transcript, and the tool output keep growing. Papers converge on the same fixes — summarize into notebooks or dossiers, retrieve only distilled entries instead of raw logs, compress traces by orders of magnitude before reuse — but they disagree sharply on the risk this creates. Some show compaction silently deleting safety rules or stale-but-important facts, producing amnesia or reversals mid-task. Others treat eviction itself as the real bottleneck, arguing agents need more compute at the moment of forgetting, not just a bigger window. A few reframe the whole problem as a security surface, since what gets kept or dropped can be manipulated by an attacker as easily as by a summarizer.
What context management means
Context management is the active engineering problem of deciding what goes into a model’s context window and what gets summarized, dropped, or retrieved on demand. It dominates real-world agent behavior more than most prompt-engineering write-ups admit.
The episodes (28)
Newest first. Each line is what that paper contributed to the question.
- Split the Same Story Across Five Messages and the Model Switches Sides
The mechanism hinges on how prior model replies accumulate in the transcript and constrain later judgments. (25 percentage points across 17 models)
- The Same Model Refused a Backdoor, Then Its Own Sub-Agent Ran It
The failure occurs during context rebuilds when the harness re-labels content's role as it moves between agent calls. (13 out of 13 on all six harnesses)
- Fifteen Models Ran Football Clubs for Twenty Years, and Size Didn't Decide It
Agents must decide what to write into a small notebook each season, since chat history never persists across decision stops. (−0.19 correlation between tokens burned and score (p = 0.5))
- How a Cheap Model Reads the Flagship's Secret Reasoning Aloud
Stateless reasoning APIs pass opaque encrypted reasoning blobs back to the client to preserve conversational continuity. (62 live API keys and 33 passwords from public logs)
- How a Robot Builds a Debugging Notebook It Can Read, Edit, and Hand to Another Robot
Only distilled library entries — not raw chat logs — are shared across parallel task agents to avoid flooding their million-token windows. (31% vs 4%)
- A 32B Open Model Matched Frontier Systems By Learning to Take Notes
The paper shows that better memory discipline reduces per-step input token counts by up to 30%, directly improving context efficiency. (2x-4x gains, same weights)
- The Bug Where Smart Assistants Read a Fact and Still Forget It
The supersession gap arises directly from agents working within bounded context windows rather than full conversation history. (28% to 28%)
- Why Better Bug Reports Can Make AI Coding Agents Worse
SHERLOC's self-recovery layer actively manages long-context failures by truncating stale middle turns and preserving key information. (74% to 10%)
- A Router That Beats the Frontier Models It Calls
The episode describes Fugu-Ultra's careful isolation of agents within a workflow while maintaining shared persistent memory across turns. (~5-6% relative gain on agentic coding)
- The Summarizer That Quietly Deletes Your Agent's Safety Rules
The core failure mode is context compaction/eviction silently deleting safety rules from an agent's working memory during long sessions. (0% to 59% violation rate)
- How a 7B Model Out-Investigates a 72B One by Choosing What to Look At
Raw video frames are purged after each look and replaced with text notes, keeping the working context bounded regardless of video length.
- When Cornering a Chatbot Makes It Lie: J.P. Morgan's Case for 'Playing Dead'
The 'point of no return' experiment shows the agent ignoring correct data injected into its context window after prior fabrication.
- Agents Fail at the Body, Not the Brain: A Self-Rewriting Scaffold That Lifts a 9B Model 44 Points
The Digester achieves 1000-to-1 compression of raw execution traces to make the coaching pipeline computationally feasible.
- Training a Tiny Model to Run the Plumbing Between an Agent and the World
The observation projection component decides which past turns to pass verbatim, compress, or drop, directly addressing context window management in long-horizon agents.
- The Agent Failed — But Did the Instructions Deserve to Be Followed?
Skill library compression is highlighted as a key benefit, reducing the token cost of injecting skill manifests into every prompt.
- Why Autonomous Research Agents Forget Their Own Lessons, and Arbor's Fix
Context window overflow across 48-hour runs is identified as a root cause of agent amnesia that the hypothesis tree addresses.
- Why AI Agents Coordinate Better Through a Shared Board Than a Boss
A three-layer hierarchical summarization scheme (gist → dossier → raw) manages what fits in each agent's context window.
- AI Coding Agents Run a Marathon, and Fewer Than One in Three Finish
Context replay causes ~99.5% of tokens to be re-reading history; compaction triggered by context overflow correlates with zero task passes.
- What If a Prompt Injection Never Left? Attacks That Wait in Agent Memory
The paper's proposed defense — 'secure context management' — treats how agents assemble context as a first-class security concern.
- How a 4B Web Agent Beat Models 60x Its Size on 500 Demonstrations
The system retains only the latest screenshot plus full historical reasoning traces, dropping old screenshots to stay within the 64K token budget.
- Why Frozen-Weight Agents Still Get Worse Over Time
Memory compaction policies and summarization strategies are shown to be primary drivers of agent aging and accuracy loss.
- Why Long-Context Models Might Need Compute, Not Capacity, Before Eviction
The paper is fundamentally about what happens at the moment of context eviction and how to make that transition compute-rich.
- Training the Translator: How a Small Communication Model Lets Agent Teams Outperform Themselves
The hub compresses agent scratchpads when context fills up, trading raw transcript for structured notes to free working memory.
- Training a Deep Research Agent on 8,000 Synthetic Tasks: The Rubric Tree Trick
The context condenser compresses long multi-step trajectories into a structured trusted/untrusted/uncertain state to avoid context overflow.
- Why Parallel Sampling Plateaus, And What Evidence Graphs Do Instead
The 1200:1 compression from evidence-graph merging is the key mechanism that keeps the Navigator's context tractable at scale.
- How One Sentence and a Forged History Flip the Most Aligned Models
The attack exploits the fact that agents treat all text in the context window — including forged turns — as equally authoritative prior history.
- Why AI Coding Agents Keep Trying to Debug Without a Debugger
DAIRA reduces input token consumption by ~25% because targeted runtime information replaces speculative file retrieval.
- An AI Ran a Real Optics Lab for 21 Hours and Found a Transformer-Shaped Pattern in Light
The Meta-Trace mechanism prevents context rot over thousands of steps by passing structured lab-notebook-style records instead of raw transcripts.
Papers we have not covered yet
Other guides
Intro written by Anthropic's Claude Sonnet 5; episodes selected and edited by Garrett Casey. Episode notes come from each episode's own analysis. How PaperDive is made.