ML//evaluation//cross-validation

Cross-validation estimates out-of-sample performance by rotating which fold is held out and averaging the results: the data are cut into \(k\) folds (five or ten), the model is trained \(k\) times, each time on all folds but one and scored on the one left out, and the \(k\) scores give a mean and a spread. It is used when data are too scarce to sacrifice a fixed validation set, which with industrial failure data is almost always, and it costs \(k\) trainings: minutes for a gradient boosting, days for a large network.

Cross-validation estimates out-of-sample performance by rotating which fold is held out and averaging the results: the data are cut into kkk folds (five or ten), the model is trained kkk times, each time on all folds but one and scored on the one left out, and the kkk scores give a mean and a spread. It is used when data are too scarce to sacrifice a fixed validation set, which with industrial failure data is almost always, and it costs kkk trainings: minutes for a gradient boosting, days for a large network.

The spread is half the information. Five folds scoring 0.81, 0.79, 0.62, 0.84 and 0.80 say that one slice of the data (one machine, one season) behaves differently, which a single split would have hidden or exaggerated.

In grouped (session-level) cross-validation a whole group goes entirely into train or into test, never both, to avoid leakage from near-duplicate rows; in a fleet, the group is the machine.

For time series the rotation runs forward: each fold trains on the past and tests on the next stretch, with a gap between them as long as the feature windows plus the horizon (train-validation-test split). Shuffled folds on a series measure memory.

The folds are stratified so that rare positives do not all pile into one of them.

The effective sample size is the number of independent groups, not the number of rows (effective sample size).

Choosing hyperparameters by cross-validation and then reporting the same cross-validated score is optimistic, since the choice was made on those folds. The honest version nests a second loop inside the first, or keeps a final test set untouched.