ML//agent//context engineering//context compaction

2026-10-28

Context compaction is replacing a long conversation history with a much shorter representation that keeps the state of the task (decisions taken, files touched, open threads), so that an agent can keep working once its context window is nearly full. It is one of four ways an agent harness handles a full context, next to truncating the oldest messages, retrieving only the fragments it needs (RAG) and keeping decisions and instructions in files it reads back when needed (agent memory). Coding agents combine the last three: Claude Code and Codex both compact, by hand or automatically near the limit, and both load project rules from a file (CLAUDE.md, AGENTS.md).

Compaction is an investment, not a saving at the moment it runs. Summarizing 100,000 tokens into 5,000 means one extra call that reads the 100,000 and writes the summary; it pays back only over the turns that follow with the short context, and it costs whatever the summary leaves out.

With illustrative numbers and no caching, 20 more turns cost 2M tokens processed without compaction and about 205,000 with it (105,000 once, then 5,000 a turn). With two turns left and the history already in the provider's cache, where reading costs a tenth of the input price, continuing is usually cheaper.

The old KV cache cannot be turned into the cache of the summary, and the cache of the messages kept after the summary is no longer valid either: their keys and values were computed while attending to the original history, and their positions move. Only the untouched prefix (system prompt, tools) is reused; the rest goes through prefill again.

The summarizing call itself should share the conversation's exact prefix so that it reads the history from the prefix cache. Claude Code sends it with the same system prompt, tools and history as the conversation and appends the instruction to summarize as the last message; a request with its own system prompt would change the first tokens and recompute everything.

OpenAI's API offers server-side compaction that returns an opaque compacted item rather than a readable summary, so mechanisms differ between providers.

Truncation is the cheap alternative and has a hidden cost of its own: cutting from the front loses the oldest material (usually the assignment and its constraints) and breaks the prefix match for everything after the cut.