ML//evaluation//data leakage

Data leakage is a flaw in the evaluation of a model in which information the model would not have at prediction time sneaks into training, inflating scores that then collapse in the real world. Its symptom is always the same, results that are too good: a first model that is right 99 % of the time is a leak until proven otherwise.

Data leakage is a flaw in the evaluation of a model in which information the model would not have at prediction time sneaks into training, inflating scores that then collapse in the real world. Its symptom is always the same, results that are too good: a first model that is right 99 % of the time is a leak until proven otherwise.

Leakage does not need to look like cheating, and it comes in four classic forms.

Temporal leakage uses the future. A centred moving average reaches forward in time, and a normalization computed over the whole history hands every training row a little of tomorrow.

Label leakage comes from a feature that is a consequence of the answer. A feature such as hours since the last intervention predicts failures beautifully, because the intervention was made because of the failure; any post-hoc feature, one not available at prediction time, is the same leak.

Split leakage puts near-identical consecutive windows on both sides at random: the model sees the minute 14:02 and is examined on 14:03. A clean random split leaks whenever rows come in families (the same session, the same engine, adjacent time windows).

Duplicate leakage is the same test exported twice, one copy landing on each side.

The shuffled split is the trap that catches everyone.

By default train_test_split shuffles, and with sensors sampled every second two consecutive rows are almost twins, so the test measures memory. If the metric falls from 0.97 to 0.70 on moving to a temporal split, 0.70 was the real number from the start.

It is the reason to split by unit, never by rows: all the samples of an engine, a patient or a user go to one side, through a grouped split or grouped cross-validation (train-validation-test split).

Its control-engineering cousin is validating an identified model by one-step-ahead prediction. At high sample rates same as before is an excellent predictor, so a model can learn persistence and score a tiny error without capturing the dynamics; free-run simulation is the honest test (model validation).

A negative control, a feature built to carry no signal, catches leaks the eye misses, and a leak found late in a project usually lived in the first table that was built (industrial labels).