ML//inference//KV cache//prefix caching

Prefix caching is the reuse, across requests, of the KV cache computed for the beginning of a prompt, so that a new request starting with exactly the same tokens skips that part of prefill and computes only its new tail. It is what keeps chat and agent loops affordable: every call resends the whole conversation (system prompt, history, documents, the new question), and without the cache every call would reread all of it.

Prefix caching is the reuse, across requests, of the KV cache computed for the beginning of a prompt, so that a new request starting with exactly the same tokens skips that part of prefill and computes only its new tail. It is what keeps chat and agent loops affordable: every call resends the whole conversation (system prompt, history, documents, the new question), and without the cache every call would reread all of it.

The reuse is exact because of causal masking: a token's keys and values at every layer depend only on the tokens before it, so two prompts that agree on their first 10,000 tokens have identical keys and values for those 10,000. The match must hold token by token from the very first one. Changing one word of the system prompt, reordering the tools or inserting a line near the top changes the keys and values of everything after the edit, because every later token attended to the changed one.

Context should grow by appending: stable material first, the variable question last, and nothing already sent edited in place. Rewriting the beginning to save tokens throws away the cache for everything after it.

It shortens prefill and does nothing for the decoding steps, which still read the weights once per new token and the cache of the whole context.

Engines keep recent caches in memory. vLLM does it automatically, and Ollama reuses the cache of the last request in a loaded model (a benchmark whose prompts all began with the same text measured prefill at 7,000 to 9,000 tokens a second for that reason). The llama.cpp server can also save a slot's cache to disk and restore it later, which turns a minute of prefill into a fraction of a second read from the SSD.

APIs bill it. Anthropic charges 1.25 times the input price to write a prefix into its five-minute cache and a tenth of it or less to read it back, so a long cached prefix can cost less than a short fresh one, and counting tokens is a poor guide to the bill.

The cache lives as long as the model stays loaded (keep-alive) or the provider's time window lasts; a cold start rebuilds it from scratch.