ML//inference//KV cache//KV cache quantization

2026-10-23

KV cache quantization stores the keys and values of the KV cache with fewer bits than the 16 they are computed in, so that long contexts and many concurrent conversations fit in less memory. It is set separately from the weights: a model whose weights are 4-bit can keep a 16-bit cache, and the reverse.

The cache grows with every token, at 2×nlayers with KV×nkv heads×dhead×b2 \times n_{\text{layers with KV}} \times n_{\text{kv heads}} \times d_{\text{head}} \times b2×nlayers with KV×nkv heads×dhead×b bytes per token, with bbb the bytes per number. For Qwen3.6 35B-A3B (10 attention layers, 2 key-value heads of 256 numbers) that is 20 KB per token in 16-bit, 5.4 GB at its maximum context of 262,144 tokens. In llama.cpp and Ollama the types are f16, q8_0 (8-bit integers plus one 16-bit scale per block of 32 numbers, about half the memory) and q4_0 (4-bit, a little over a quarter of the 16-bit size).

Quantization of the weights rounds the model, the same for every request; quantizing the cache rounds the context, the document as the model has read it. q8_0 costs little measurable quality; q4_0 deserves a test on one's own tasks before it is trusted.

In Ollama it is the server variable OLLAMA_KV_CACHE_TYPE, and it requires OLLAMA_FLASH_ATTENTION=1, because the quantized cache is read by the blockwise Flash Attention kernels.

It saves memory first and time second: a smaller cache is also fewer bytes read at every decoding step, which matters only once the context is long enough for cache reads to compete with weight reads.

The first lever on cache memory is usually the reserved context rather than the number format, since the cache is allocated for the whole requested context when the model loads, used or not. Shared key-value heads (GQA) and layers that keep no cache at all (Gated DeltaNet) shrink it at the architecture level.