ML//neural network//optimizer//Adam
Adam is a gradient-based optimizer that keeps an exponential moving average of the gradients and another of the squared gradients, and uses them to give each parameter its own adaptive update scale. It is used to train neural networks, where it serves as a strong default: with a learning rate of \(\eta=10^{-3}\) it works acceptably on many problems without any tuning.
Adam is a gradient-based optimizer that keeps an exponential moving average of the gradients and another of the squared gradients, and uses them to give each parameter its own adaptive update scale. It is used to train neural networks, where it serves as a strong default: with a learning rate of η=10−3\eta=10^{-3}η=10−3 it works acceptably on many problems without any tuning.
Each step moves a parameter by η m^/(s^+ϵ)\eta,\hat m/(\sqrt{\hat s}+\epsilon)ηm^/(s^+ϵ), where mmm is the running mean of its gradient, sss the running mean of the squared gradient, the hats mark a start-up correction and ϵ\epsilonϵ avoids dividing by zero. A parameter whose gradient is large and nervous takes short steps; one whose gradient is small and steady takes long ones. In a long, narrow valley, where plain gradient descent crawls along the floor and momentum overshoots at the bends, Adam advances with steps of similar size in both directions.
Adam is a cheap diagonal preconditioner.
It rescales each axis of the valley by its own gradient history, without ever computing the Hessian, which is why it tolerates badly scaled problems that make plain SGD zigzag.
The first moment behaves like momentum; the second normalizes coordinates with persistently different gradient magnitudes. Bias correction matters early, when both averages still begin near zero.
Adam uses the gradients and does not compute them: backpropagation produces ∇θL\nabla_\theta L∇θL, and Adam decides how far each parameter moves. The phrases trained with Adam and trained with backprop describe the same run from two sides, and every matrix of a transformer (WQW_QWQ, WKW_KWK, WVW_VWV, the feed-forward weights) is updated this way, AdamW being the usual choice for language models.
AdamW decouples weight decay from the adaptive gradient update: it shrinks every weight toward zero by a fixed fraction per step, the network version of ridge regression. That distinction is easy to hide in an optimizer flag and can materially change regularization; torch.optim.AdamW(params, lr=1e-3, weight_decay=1e-4) is a common starting point.
Adam often reaches a useful solution quickly and tolerates rough scaling better than plain SGD, which makes it a strong default (a default, never a theorem). Memory cost (two extra numbers per parameter), final generalization, batch regime, and architecture can favor other optimizers.