mathematics//numerical methods//automatic differentiation

Automatic differentiation is a technique that computes the exact derivatives of a function written as a program by applying the chain rule to every elementary operation the program performs, and it is what lets an engineer write only the forward computation (a loss, a cost, a dynamics model) and get its gradient or Jacobian for free. It is what PyTorch does when `loss.backward()` is called, and what JAX's `grad` and `jacfwd` do. The derivatives are exact to machine precision: there is no step size to choose, unlike finite differences, and no formula to derive by hand, unlike symbolic differentiation.

Automatic differentiation is a technique that computes the exact derivatives of a function written as a program by applying the chain rule to every elementary operation the program performs, and it is what lets an engineer write only the forward computation (a loss, a cost, a dynamics model) and get its gradient or Jacobian for free. It is what PyTorch does when loss.backward() is called, and what JAX's grad and jacfwd do. The derivatives are exact to machine precision: there is no step size to choose, unlike finite differences, and no formula to derive by hand, unlike symbolic differentiation.

The program is seen as a chain of simple operations (additions, products, a sine, an exponential), each with a known derivative. Forward mode carries the derivative along with each value as the program runs, and costs about one extra pass per input; it suits a function with few inputs and many outputs, such as the Jacobian of a 12-state drone model with respect to its 4 motor commands. Reverse mode runs the program forwards storing every intermediate value, then walks backwards accumulating how much the output depends on each one; it gives the derivative of one scalar output with respect to all inputs at once.

Reverse mode makes the gradient of one number with respect to a billion parameters cost a small multiple of computing that number.

In practice it costs about two to three forward passes, whatever the number of parameters, and backpropagation is reverse mode applied to a neural network.

Its price is memory. Reverse mode must keep every intermediate value of the forward pass until the backward pass consumes it, which is why training a network needs far more memory than running it, and why long recurrent unrolls and long simulations differentiated end to end run out of it (checkpointing recomputes parts of the forward pass to trade compute back for memory).

It pays off with many parameters or when machine precision matters. For a function of three variables evaluated once a second, a finite difference is simpler and good enough; for a training loop, a nonlinear MPC solver that needs exact gradients of a trajectory cost, or an EKF whose hand-derived Jacobian keeps being wrong, automatic differentiation removes a whole class of bugs.

It differentiates the program exactly as written. A branch, a clamp or a lookup table gives the derivative of the branch actually taken, which can be zero or undefined exactly where an optimizer needs a slope, and a numerically unstable formula yields an equally unstable derivative.