ML//transformer//attention//linear attention

Linear attention is a family of sequence layers that replace attention's comparison of each token with every earlier one by a fixed-size state updated once per token, so that a new token costs the same however long the context is and the layer keeps no growing KV cache. It is used to make long contexts cheap in memory and time, usually mixed with ordinary attention layers rather than replacing all of them.

Linear attention is a family of sequence layers that replace attention's comparison of each token with every earlier one by a fixed-size state updated once per token, so that a new token costs the same however long the context is and the layer keeps no growing KV cache. It is used to make long contexts cheap in memory and time, usually mixed with ordinary attention layers rather than replacing all of them.

Standard attention keeps the key and the value of every earlier token and reads all of them for each new one, so its memory and its read per step grow with the context. A linear-attention layer folds each new key and value into a matrix of fixed size and reads only that matrix, the way a recurrent network carries its state (RNN). The price is recall: a fixed state cannot keep every token apart, so retrieving one exact detail from far back is harder for it.

Hybrid models keep a few full-attention layers for exact recall and make the rest linear, so only those few grow a cache. Qwen3.6 35B-A3B has full attention in one layer of four (10 of 40) and Gated DeltaNet in the other thirty, which makes its KV cache a quarter of what a pure transformer of the same shape would need.

Sliding-window attention answers the same cost differently: it keeps exact keys and values, but only for the last few thousand tokens.

The fixed state does not depend on the context length (about 2 MB per layer in Qwen3.6), so in a memory budget it is a constant rather than a cost per token.