ML//generalization//overfitting

Overfitting is a failure of a trained model in which it captures patterns specific to its training sample that do not generalize to the target distribution: it fits the noise along with the signal, and so changes its mind with every new sample. Memorization can contribute, but memorization and generalization are not strict opposites in large models.

Overfitting is a failure of a trained model in which it captures patterns specific to its training sample that do not generalize to the target distribution: it fits the noise along with the signal, and so changes its mind with every new sample. Memorization can contribute, but memorization and generalization are not strict opposites in large models.

The textbook picture is a polynomial through a few noisy points. As the degree rises, the training error falls without limit while the error on fresh points falls, bottoms out and shoots up; at a high degree, a new sample of the same process produces a completely different curve. That instability is the variance of the bias-variance trade-off, and overfitting is its high-variance end. It applies to every model family, a deep decision tree as much as a network or an unpenalized regression with many features.

A widening gap between training and validation is evidence, not a direct meter: leakage, distribution mismatch, label noise, and an unrepresentative validation set can produce similar symptoms. A validation score that is too good is more often data leakage than a well-fitted model.

More representative data, regularization, augmentation, early stopping, and appropriate capacity can help. A simpler model is better only relative to the evidence available.

The real test is performance on data drawn from the conditions where the model will actually be used, which is stricter than performance on data the model has not seen (generalization, out-of-distribution).