ML//inference//KV cache//KV cache stitching

KV cache stitching is reusing KV cache fragments that were computed in different contexts (each retrieved document on its own, for example) by joining them into one sequence, and it is a research line because ordinary caches cannot be joined that way. A token's keys and values at every layer above the first are projections of its residual stream, which has already attended to everything before it, so the cache of a segment C depends on what preceded it: \(KV(C \mid A, B) \neq KV(C \mid A, B')\) even when the text of C is identical.

KV cache stitching is reusing KV cache fragments that were computed in different contexts (each retrieved document on its own, for example) by joining them into one sequence, and it is a research line because ordinary caches cannot be joined that way. A token's keys and values at every layer above the first are projections of its residual stream, which has already attended to everything before it, so the cache of a segment C depends on what preceded it: KV(C∣A,B)≠KV(C∣A,B′)KV(C \mid A, B) \neq KV(C \mid A, B')KV(C∣A,B)=KV(C∣A,B′) even when the text of C is identical.

Cache blocks are independent for storage, not for computation. PagedAttention can scatter blocks anywhere in memory, but a block is only valid after the exact prefix it was computed with.

Pasting old blocks after a new prefix raises no error: the shapes match and the GPU computes. The model then runs on an incoherent internal state, with keys and values that still carry the removed context and encode old positions (RoPE rotates keys by their position when they are computed). The output can read well and still differ from processing the new sequence from scratch.

That is why compacting a conversation invalidates the cache of the recent messages kept after the summary, and why prefix caching keys each block by a hash chained through every block before it.

Stitching methods accept an approximation to make reuse possible. CacheBlend (2024) joins the precomputed caches of several documents and recomputes a small fraction of tokens to restore the cross-attention the documents never had with each other; other approaches design representations meant to be computed independently.

Exact reuse remains prefix caching: identical beginning, identical result.