ML//training//early stopping

Early stopping is a regularization technique that ends training when validation performance stops improving, rather than when training loss bottoms out. It is cheap and effective for LSTM and gradient boosting alike.

Early stopping is a regularization technique that ends training when validation performance stops improving, rather than when training loss bottoms out. It is cheap and effective for LSTM and gradient boosting alike.

Training loss keeps falling while validation loss bottoms out early and then drifts up; the gap is overfitting, and training stops at the validation minimum. That requires a held-out fold that model selection never trains on, and under grouped data the stopping fold must respect the same cross-validation split.

It is a regularizer in disguise. Gradient descent starting from small weights fits the large, well-supported directions of the data first and the small, noisy ones last, so stopping early leaves the poorly determined directions near zero, much as ridge regression damps them with a penalty; the number of steps plays the role of 1/λ1/\lambda1/λ (regularization lists the rest of the family).

In practice it is set with a patience: training continues a fixed number of epochs past the best validation score, in case the curve was only noisy, and the weights of the best epoch are restored at the end. In boosting the same rule picks the number of trees.

The stopping point is chosen on the validation fold, so that fold has been used for a decision and is no longer an unbiased measure; the final error is read once on a separate test set (train-validation-test split).