ML//inference//KV cache//KV cache offloading
KV cache offloading is moving the KV cache of idle conversations out of GPU memory, into CPU RAM or storage, and copying it back when the conversation returns, instead of keeping it in scarce HBM or recomputing it with a new prefill. CPU RAM is slower and further away than HBM; it is used because copying keys and values that already exist can cost much less than running the tokens through every layer again.
KV cache offloading is moving the KV cache of idle conversations out of GPU memory, into CPU RAM or storage, and copying it back when the conversation returns, instead of keeping it in scarce HBM or recomputing it with a new prefill. CPU RAM is slower and further away than HBM; it is used because copying keys and values that already exist can cost much less than running the tokens through every layer again.
Bringing the cache back pays when Ttransfer<TrecomputeT_{\text{transfer}} < T_{\text{recompute}}Ttransfer<Trecompute. The comparison is between a copy and a computation, not between two memories.
A worked case: a 70B model with grouped-query attention (80 layers, 8 KV heads of 128) in 16-bit stores 80×2×8×128×280 \times 2 \times 8 \times 128 \times 280×2×8×128×2 bytes per token, about 3.3 GB for 10,000 tokens. Over PCIe at an effective 24 GB/s that is about 0.14 s, plus latency. Recomputing it is about 2×70B×10,000=1.42 \times 70\text{B} \times 10{,}000 = 1.42×70B×10,000=1.4 PFLOP, around 0.35 s on eight H100s at half their peak. Copying wins by more than two to one, and the conversation did not occupy HBM while it was away.
The balance can flip: a small model, a slow link or a short context make recomputation cheaper.
The second benefit is capacity. After the weights, eight H100s leave roughly 500 GB, about 150 caches of 10,000 tokens or 15 of 100,000; offloading idle ones lets a server keep many more conversations warm.
Transfers can overlap with computation on other requests. vLLM supports asynchronous offloading, and LMCache keeps caches in CPU memory or storage and reports large speedups over recomputation in its own benchmarks.
It extends prefix caching beyond what fits in GPU memory; the reuse rules are the same, so a cache brought back is valid only for the exact prefix it was computed with.