ML//neural network//backpropagation

Backpropagation is the algorithm that computes the gradient of a network's loss function with respect to every weight, by applying the chain rule from the output backwards and reusing the intermediate results of the forward pass; it is how every neural network gets the derivatives its training needs. It is the chain rule with good bookkeeping and nothing more. The learning itself is done by the optimizer (SGD, Adam), which takes those gradients and moves the weights.

Backpropagation is the algorithm that computes the gradient of a network's loss function with respect to every weight, by applying the chain rule from the output backwards and reusing the intermediate results of the forward pass; it is how every neural network gets the derivatives its training needs. It is the chain rule with good bookkeeping and nothing more. The learning itself is done by the optimizer (SGD, Adam), which takes those gradients and moves the weights.

The forward pass stores, for every layer, the input of its nonlinearity z(ℓ)=W(ℓ)h(ℓ−1)+b(ℓ)z^{(\ell)}=W^{(\ell)}h^{(\ell-1)}+b^{(\ell)}z(ℓ)=W(ℓ)h(ℓ−1)+b(ℓ) and its output h(ℓ)h^{(\ell)}h(ℓ). The backward pass then computes the error signal δ(ℓ)=∂L/∂z(ℓ)\delta^{(\ell)}=\partial L/\partial z^{(\ell)}δ(ℓ)=∂L/∂z(ℓ), how much the loss moves if that input is touched, from the last layer down to the first.

δ(ℓ)=ϕ′ ⁣(z(ℓ))⊙(W(ℓ+1)Tδ(ℓ+1)),∂L∂W(ℓ)=δ(ℓ) h(ℓ−1)T\delta^{(\ell)}=\phi'\!\left(z^{(\ell)}\right)\odot\left(W^{(\ell+1)\mathsf T}\delta^{(\ell+1)}\right),\qquad \frac{\partial L}{\partial W^{(\ell)}}=\delta^{(\ell)}\,h^{(\ell-1)\mathsf T}δ(ℓ)=ϕ′(z(ℓ))⊙(W(ℓ+1)Tδ(ℓ+1)),∂W(ℓ)∂L​=δ(ℓ)h(ℓ−1)T

A layer's blame is the next layer's blame returned through the same weights transposed and attenuated by the slope ϕ′\phi'ϕ′ of the activation (⊙\odot⊙ is the elementwise product); the derivative of each weight is the blame of the neuron it feeds times the activation it comes from.

An audit that splits the error among everyone involved, in proportion to their influence.

When a weld-inspection network misses a crack, backprop hands every one of its millions of weights its share of the miss in a single pass backwards.

The gradient for every parameter costs a small multiple of one forward pass, about two to three times, whether the network has a thousand parameters or a billion. That is what makes deep learning possible. The price is memory: every activation is kept until the backward pass, so training needs far more memory than running.

Backprop is chained linearization. Around the current point each layer behaves as its Jacobian Jℓ=diag⁡(ϕ′(z(ℓ))) W(ℓ)J_\ell=\operatorname{diag}(\phi'(z^{(\ell)})),W^{(\ell)}Jℓ​=diag(ϕ′(z(ℓ)))W(ℓ) and the network as the product JL⋯J1J_L\cdots J_1JL​⋯J1​ of local linearizations; backprop multiplies them starting from the scalar loss, so each product is a vector times a matrix and never a matrix times a matrix. A product of many matrices is ruled by its singular values, which is the vanishing gradient (linearize-solve-repeat).

Nobody writes it by hand: frameworks implement it as reverse-mode automatic differentiation (loss.backward() in PyTorch). For a physical model with five constants, finite differences or a gradient-free optimizer are enough; reverse mode pays from hundreds of parameters up.

It runs only in training. During inference the weights stay fixed and no backward pass happens, so a model that retrieves documents, calls tools or reasons longer before answering improves that answer without a single gradient, which is a different thing from learning.

In standard fine-tuning and DPO it touches all parameters (attention and MLP), and nothing is frozen by default. In LoRA the gradients only flow through the small adapter matrices: the base weights are frozen and backprop skips them.