ML//neural network//batch normalization

2026-02-28

Batch normalization is a normalization layer for neural networks that rescales each layer's activations across the mini-batch to zero mean and unit variance and then applies a learnable scale and shift, and it is used to keep the training of deep networks stable. Introduced in 2015, it was the first normalization technique that made deep networks reliably trainable.

Why it works is debated. The original paper claimed it reduces internal covariate shift (each layer's input distribution changes as the previous layers update), and later research showed this explanation is incomplete. The more likely reason is that it smooths the loss function landscape, which makes gradient descent more stable.

Its weak point is the dependency on the batch. The normalization statistics (mean, variance) are computed over the current mini-batch during training, and running averages from training are used at inference time. Small batches make the statistics noisy and hurt performance, which is why batch norm struggles with batch sizes under 32.

Layer normalization replaced batch norm in Transformers because language models process sequences of varying length with small batch sizes. Layer norm normalizes across features within a single example, not across the batch. No batch dependency, no batch size sensitivity.

It is still dominant in CNNs and ResNet architectures. ResNet's success (training 100+ layer networks) was partly enabled by batch norm stabilizing gradient flow through very deep convolution stacks.

The learnable scale (γ\gammaγ) and shift (β\betaβ) parameters are critical: they let the network undo the normalization if needed. The model can learn γ\gammaγ equal to the original standard deviation and β\betaβ equal to the original mean to recover the unnormalized activations for layers where normalization hurts.