ML//transformer//positional encoding//ALiBi

ALiBi (Attention with Linear Biases) is an alternative to RoPE and sinusoidal positional encoding that adds no position information to the embeddings at all. Instead, it subtracts from the attention scores a linear bias proportional to the distance between tokens.

ALiBi (Attention with Linear Biases) is an alternative to RoPE and sinusoidal positional encoding that adds no position information to the embeddings at all. Instead, it subtracts from the attention scores a linear bias proportional to the distance between tokens.

Farther tokens are penalized more, which gives a natural recency bias without learned position embeddings.

It extrapolates to longer sequences than those seen during training, because the linear penalty is simple enough to generalize.

It adds no parameters, only a fixed penalty schedule applied during the attention computation.