ML//inference//KV cache

During autoregressive generation, the KV cache stores attention keys and values for previous tokens instead of recomputing them at every step.

During autoregressive generation, the KV cache stores attention keys and values for previous tokens instead of recomputing them at every step.

Only works because of causal masking: previous tokens' representations are "closed". They don't change when a new token arrives. Without masking (BERT-style), every new token would change all previous values → must recompute everything.

Example: context ["El", "gato"], new token "duerme" → only compute K_duerme, V_duerme. K_El, V_El, K_gato, V_gato already cached.

The cache reduces repeated computation, but its memory grows with active sequence length, batch size, layer count and representation precision.

The tradeoff

Trades memory for compute. For long contexts or many concurrent sequences, cache capacity and bandwidth can become major serving constraints.

Another reason not to save enriched embeddings across passes: you'd need to recompute everything each time, the RNN-style dilution problem.