ML//inference//KV cache
The KV cache is the store of attention keys and values that a decoder keeps for every earlier token, at every layer (and, in multi-head attention, for every key-value head), so that generating the next token costs one pass for that token instead of a pass over the whole sequence. It is what makes autoregressive generation affordable, and its size is what limits how long and how many conversations a server can hold. It belongs to inference alone: training processes whole sequences in parallel and keeps no cache, so a saving in the cache lowers the cost of serving and leaves the training cost untouched.
The KV cache is the store of attention keys and values that a decoder keeps for every earlier token, at every layer (and, in multi-head attention, for every key-value head), so that generating the next token costs one pass for that token instead of a pass over the whole sequence. It is what makes autoregressive generation affordable, and its size is what limits how long and how many conversations a server can hold. It belongs to inference alone: training processes whole sequences in parallel and keeps no cache, so a saving in the cache lowers the cost of serving and leaves the training cost untouched.
It works only because of causal masking: a token's representation at any layer depends only on itself and the tokens before it, so appending a token changes nothing that is already there. With the context The cat and the new token sleeps, only KsleepsK_{\text{sleeps}}Ksleeps and VsleepsV_{\text{sleeps}}Vsleeps are computed at each layer; KTheK_{\text{The}}KThe, VTheV_{\text{The}}VThe, KcatK_{\text{cat}}Kcat and VcatV_{\text{cat}}Vcat are read from the cache.
Without masking (BERT-style), every new token would change all previous values, and everything would have to be recomputed at each step.
What it stores is the enriched state of earlier tokens: the key and value at layer 60 come from the token's column as it stood at layer 60, after the attention and MLP blocks below it. The new token starts from its plain embedding and, at each layer, reads what every earlier token had become at that depth. This is the difference from an RNN, which squeezes the whole past into one fixed-size state; the cache keeps one state per token and so grows with the context.
Its size is 2×nlayers×nkv heads×dhead×b2 \times n_{\text{layers}} \times n_{\text{kv heads}} \times d_{\text{head}} \times b2×nlayers×nkv heads×dhead×b bytes per token, with bbb the bytes per number. For GPT-3 (96 layers, 96 heads of 128, 16-bit) that is about 4.7 MB per token; for Llama 3 8B, which shares each key and value set among four query heads (GQA), 128 KB per token, so a 131,072-token context takes 16 GiB, about as much as the weights.
The cache trades memory for compute. It speeds up decoding, which is limited by memory reads, and adds its own reads that lengthen with the context, so for long contexts or many concurrent sequences its capacity and bandwidth become the main serving constraints: a longer context window in use enlarges the cache with the weights unchanged, and inference batching multiplies it by the number of sequences held at once. Shared heads, a quantized cache and paged allocation (vLLM) are the usual answers.
A server can keep the cache of a prompt between requests and reuse it for any later request that starts with exactly the same tokens (prefix caching). It shortens the reading of the prompt and leaves generation as it was, and a change anywhere in the prefix invalidates everything after it.