ML//neural network//residual connection

A residual connection is a shortcut in a neural network that adds a block's input to its output, \(h^{(\ell)}=h^{(\ell-1)}+F(h^{(\ell-1)})\), so that the block learns a correction to what it receives instead of a whole new representation; it is what lets a network stack tens or hundreds of layers and still train. It was introduced by ResNet in 2015 for image CNNs, and every transformer inherits it. (The word is unrelated to the residual of fault diagnosis, a measurement minus its prediction.)

A residual connection is a shortcut in a neural network that adds a block's input to its output, h(ℓ)=h(ℓ−1)+F(h(ℓ−1))h^{(\ell)}=h^{(\ell-1)}+F(h^{(\ell-1)})h(ℓ)=h(ℓ−1)+F(h(ℓ−1)), so that the block learns a correction to what it receives instead of a whole new representation; it is what lets a network stack tens or hundreds of layers and still train. It was introduced by ResNet in 2015 for image CNNs, and every transformer inherits it. (The word is unrelated to the residual of fault diagnosis, a measurement minus its prediction.)

Why it works shows in the derivative. The Jacobian of a residual block is I+JFI+J_FI+JF​: the identity gives the gradient a direct highway back through every layer, so even when JFJ_FJF​ is small the product over many blocks does not collapse toward zero (vanishing gradient). At the start of training, with every FFF near zero, a deep residual network behaves like a shallow one, and it grows into its depth as the blocks learn.

In a transformer it creates the residual stream, a persistent vector that flows through all layers and accumulates additive deltas from the attention and MLP blocks, and it keeps gradients flowing through more than 100 layers. Every transformer block has two, one after attention and one after the FFN, and together with layer norm they form the structural backbone of each attention and MLP block.