ML//transformer//layer normalization

Layer normalization is a normalization layer in neural networks that rescales activations across the feature dimension of each example, independently of the batch. It stabilizes training by keeping values from exploding or vanishing across layers.

Layer normalization is a normalization layer in neural networks that rescales activations across the feature dimension of each example, independently of the batch. It stabilizes training by keeping values from exploding or vanishing across layers.

Where it sits matters: pre-norm normalizes before attention or the FFN, post-norm after them, and pre-norm is the more stable of the two for deep models.