ML//evaluation
Evaluation is the part of machine learning practice that estimates how a model will perform on data it has not seen, with a metric that reflects what each error really costs, while keeping track of the ways that number can lie. A model that scores 97 % in a notebook and 70 % in the plant was measured on a question nobody needed answered.
Evaluation is the part of machine learning practice that estimates how a model will perform on data it has not seen, with a metric that reflects what each error really costs, while keeping track of the ways that number can lie. A model that scores 97 % in a notebook and 70 % in the plant was measured on a question nobody needed answered.
Two choices make an evaluation. The first is the split, which decides which question the number answers: data are set aside as training, validation and test sets (train-validation-test split), rotated through folds when they are scarce (cross-validation), and cut by time or by machine so that nothing the model will lack at prediction time reaches training (data leakage). The second is the metric. In regression the MAE (mean absolute error) reads in the units of the variable, half a degree or two bar; the RMSE (root mean squared error) punishes large errors more, so it is the one to watch when a single big miss is what hurts.
MAE=1N∑i=1N∣yi−y^i∣,RMSE=1N∑i=1N(yi−y^i)2\text{MAE}=\frac1N\sum_{i=1}^{N}\lvert y_i-\hat y_i\rvert,\qquad \text{RMSE}=\sqrt{\frac1N\sum_{i=1}^{N}(y_i-\hat y_i)^2}MAE=N1i=1∑N∣yi−y^i∣,RMSE=N1i=1∑N(yi−y^i)2
Here yiy_iyi is the true value and y^i\hat y_iy^i the prediction. RMSE is never below MAE, and a large gap between them says that a few errors dominate.
In classification every metric is a ratio of the four counts of the confusion matrix. Precision and recall read them in the language of alarms; ROC-AUC and PR-AUC summarize every threshold at once; with rare failures, class imbalance makes accuracy useless and ROC-AUC optimistic.
The threshold is chosen afterwards, as a decision: it comes from the cost of each error (decision threshold) and works only with probabilities that mean what they say (probability calibration).
Controls test the pipeline itself. A negative control is a feature that should carry no signal, and a model that leans on it reveals a leak. Once predictions trigger actions, the deployed model changes the data it is judged on (performative prediction), and only experiments in which the action varies can tell what the model got right.