ML//transformer//attention//sliding window attention

Sliding window attention is a sparse attention pattern (the Longformer approach) in which each token attends only to its \(w\) nearest neighbors plus a few designated global tokens, and it is used to make very long sequences affordable.

Sliding window attention is a sparse attention pattern (the Longformer approach) in which each token attends only to its www nearest neighbors plus a few designated global tokens, and it is used to make very long sequences affordable.

It reduces the attention cost from O(n2)O(n^2)O(n2) to O(nw)O(nw)O(nw), which makes very long sequences feasible.

The tradeoff is that tokens far apart can only communicate indirectly, through chains of local windows across layers.

Global tokens (e.g. [CLS]) still attend to everything and serve as anchor points for aggregation.

It does not give truly infinite context. Like the other long-context approaches, it is a compromise between cost and attention capacity.

Mistral uses sliding window attention as default.