ML//neural network//optimizer//learning rate
The learning rate is the step size of gradient descent: how much the weights are adjusted per update. Too high and training diverges (the loss explodes); too low and it takes forever or gets stuck in shallow minima.
The learning rate is the step size of gradient descent: how much the weights are adjusted per update. Too high and training diverges (the loss explodes); too low and it takes forever or gets stuck in shallow minima.
Its upper limit is set by the curvature of the loss. Near a minimum every smooth loss looks quadratic with Hessian HHH, and the update becomes the linear map θk+1=(I−ηH) θk\theta_{k+1}=(I-\eta H),\theta_kθk+1=(I−ηH)θk, which converges only if η<2/λmax\eta<2/\lambda_{\max}η<2/λmax, with λmax\lambda_{\max}λmax the largest eigenvalue of HHH. The steepest direction caps the step and the flattest one sets how slow progress is, so a badly conditioned loss forces a small rate and a long training. A training that diverges is an unstable loop, and it is diagnosed like one.
It is the single most important hyperparameter in deep learning. Everything else (architecture, data, optimizer) matters, but a wrong learning rate can make the best setup fail completely. It is rarely held constant: a schedule changes it over the course of training.
Warmup starts with a tiny learning rate and increases it linearly over the first 1 to 5% of training. Without warmup, early gradients are noisy (the weights are random) and large steps destabilize training. Adam adapts the effective step per parameter from its momentum and second moments, so the rate that is set is really a maximum step it scales down from the gradient history; that partially mitigates the problem, but warmup still helps.
Cosine decay follows warmup and lowers the learning rate along a cosine curve down to near zero. The model makes large exploratory steps early and fine-grained adjustments late; it is the standard schedule for pre-training large models.
Scaling laws interact directly: when the batch size grows, the learning rate is typically scaled in proportion (the linear scaling rule). Distributed training across more GPUs means a larger effective batch size, and so an adjusted schedule.
For fine-tuning, the learning rate is typically 10 to 100 times smaller than in pre-training. The weights are already near a good solution, and large steps would destroy what the model learned. This is why LoRA works: it adds small trainable deltas instead of updating all weights with a small learning rate.
The same knob appears outside networks: in Q-learning (reinforcement learning) the rate α\alphaα says how far each new temporal-difference error moves the stored value, with the same trade between fast adaptation and a noisy estimate.