ML//neural network//vanishing gradient

The vanishing gradient is the exponential shrinking of the gradient as backpropagation carries it back through many layers or time steps, and its twin, the **exploding gradient**, is the exponential growth of the same quantity; together they are why deep and recurrent networks were hard to train, and why modern architectures look the way they do. The chain rule multiplies one layer Jacobian per layer, \(J_L\cdots J_1\), and a product of many matrices is ruled by their singular values. Systematically below 1, the gradient reaching the first layers is tiny and they stop learning; systematically above 1, it blows up and training diverges into NaN. A factor of 0.9 repeated over 50 layers multiplies the signal by about 0.005, a factor of 1.1 by about 117.

The vanishing gradient is the exponential shrinking of the gradient as backpropagation carries it back through many layers or time steps, and its twin, the exploding gradient, is the exponential growth of the same quantity; together they are why deep and recurrent networks were hard to train, and why modern architectures look the way they do. The chain rule multiplies one layer Jacobian per layer, JL⋯J1J_L\cdots J_1JL​⋯J1​, and a product of many matrices is ruled by their singular values. Systematically below 1, the gradient reaching the first layers is tiny and they stop learning; systematically above 1, it blows up and training diverges into NaN. A factor of 0.9 repeated over 50 layers multiplies the signal by about 0.005, a factor of 1.1 by about 117.

This is the stability of a discrete system read from the learning side. A recurrent network applies the same matrix WWW at every step, so backpropagation through time multiplies by it hundreds of times, and the gradient vanishes or explodes according to its eigenvalues, exactly as xk+1=Wxkx_{k+1}=Wx_kxk+1​=Wxk​ decays or diverges according to whether its spectral radius is below or above 1 (discrete-time stability).

Vanishing and exploding gradients are an eigenvalue problem.

The fixes all keep the per-layer factor near 1: a ReLU with slope 1, an initialization that preserves the scale of the activations, normalization (layer norm) and above all residual connections, whose Jacobian I+JFI+J_FI+JF​ gives the gradient a highway. Explosions are also capped bluntly by clipping the gradient's norm.

It is why deep nets could not train before ResNet skip connections, and why RNNs could not handle long sequences before LSTM gates: the hidden state carries memory from previous steps, but information from the beginning dilutes exponentially. The LSTM updates its cell by addition instead of multiplication, which keeps that factor close to 1.

It is also the fundamental reason enriched embeddings are not saved as persistent state: that is exactly what RNNs did, and the hidden state corrupted over time.

The same eigenvalue test reappears across engineering (the step limit of gradient descent, the conditioning of a fit, the stability of a loop), collected in spectral reading.