ML//transformer//attention//MLA

MLA (Multi-head Latent Attention) is a variant of multi-head attention that stores, for each token and layer, one small compressed vector instead of a key and a value per head, and rebuilds the keys and values from it when they are needed; it is used in DeepSeek V2 and V3 to shrink the KV cache far more than GQA does while keeping every head's keys and values distinct.

MLA (Multi-head Latent Attention) is a variant of multi-head attention that stores, for each token and layer, one small compressed vector instead of a key and a value per head, and rebuilds the keys and values from it when they are needed; it is used in DeepSeek V2 and V3 to shrink the KV cache far more than GQA does while keeping every head's keys and values distinct.

Each token's input is projected down to a latent vector (512 numbers in DeepSeek-V3), which is what the cache keeps. Two learned up-projections turn the latent back into the keys and the values of all heads. At inference those up-projections can be folded into the query and output matrices, so the latent is used directly and the full keys and values are never materialised.

The saving is large. DeepSeek-V3 has 128 heads of 128 dimensions, so standard attention would store 2×128×128=32,7682\times128\times128=32{,}7682×128×128=32,768 numbers per token and layer; MLA stores 576 (the 512 of the latent plus 64 for a separate positional key), about 57 times less.

RoPE does not commute with the folding trick, because the rotation depends on the position. MLA therefore keeps a small extra key that carries the rotation, shared by all heads, and applies RoPE only to it.

GQA saves memory by making heads share keys and values; MLA by compressing them. The first loses some diversity between heads, the second adds projections and a more complex kernel.