ML//transformer//layer normalization//RMSNorm

RMSNorm is a normalization layer for transformers that divides each token's vector by its root mean square and multiplies it by a learned gain, and it is used in place of layer normalization in almost every recent open model (LLaMA, Mistral, Qwen) because it does the same job with less work. It drops the two steps of layer norm that turned out to matter least: it never subtracts the mean, and it has no bias.

RMSNorm is a normalization layer for transformers that divides each token's vector by its root mean square and multiplies it by a learned gain, and it is used in place of layer normalization in almost every recent open model (LLaMA, Mistral, Qwen) because it does the same job with less work. It drops the two steps of layer norm that turned out to matter least: it never subtracts the mean, and it has no bias.

RMSNorm(x)=g⊙x1d∑i=1dxi2+ϵ\text{RMSNorm}(x)=g\odot\frac{x}{\sqrt{\tfrac{1}{d}\sum_{i=1}^{d}x_i^2+\epsilon}}RMSNorm(x)=g⊙d1​∑i=1d​xi2​+ϵ​x​

Here xxx is one token's vector of ddd numbers, ggg a learned vector of the same size and ϵ\epsilonϵ a small constant that avoids dividing by zero. The output has a fixed scale but keeps its direction, which is what the next projection reads.

It is cheaper because there is one statistic per token instead of two, and one learned vector instead of two. The parameters are negligible either way (a few thousand per norm against billions in the matrices); the gain is time, because a normalization is element-wise work that reads and writes the whole vector and runs twice in every layer.

It sits where layer norm sat in a pre-norm model: before attention, before the feed-forward block, and once more before the LM head. The residual stream itself is never normalized.

Skipping the mean is an empirical choice, not a theorem: models trained with it reach the same quality, which suggests the rescaling is what keeps training stable and the recentring adds little.