ML//evaluation//ROC-AUC

2026-07-04

ROC-AUC is the area under the ROC curve, the evaluation metric that summarizes a classifier or a detector over all its thresholds, and it equals the probability that the model ranks a random positive above a random negative. The curve behind it draws the trade-off every detector lives with: moving the threshold exchanges one error for the other, false alarms (type I) against misses (type II), and no threshold minimizes both (type I and type II errors).

The ROC curve plots the detection rate, or recall, against the false alarm rate as the threshold sweeps from alarming on everything to alarming on nothing. A value of 0.5 is a coin and 1.0 a perfect oracle; the metric reads cleanly and is threshold-independent, which is both its use (comparing two detectors before the costs are known) and its limit (the plant runs one threshold, never all of them). It is deceptive under class imbalance, because it rewards ranking even when almost every alarm is a false positive, so a model can be directionally right (ROC 0.72) and operationally useless at the same time.

The point on the curve is chosen by costs: the threshold where the cost of a false alarm times its rate, plus the cost of a miss times its rate, is lowest (decision threshold).

In sequential detection, where a detection threshold watches a residual sample after sample, the same trade-off is drawn in other currencies: mean time between false alarms against detection delay. For a CUSUM the first grows roughly exponentially with the threshold and the second only linearly, so ten times fewer false alarms cost a few samples of delay.

A false alarm rate per sample misleads. A 3σ3\sigma3σ threshold on Gaussian noise gives 0.27 % per sample, which at 100 Hz is almost a thousand false alarms an hour, and operators count alarms per hour (alarm management).

Advice26 · jul 04Read notePair it with PR-AUC: ROC-AUC rewards a good ranking even when almost every alarm is false, and PR-AUC shows the precision it hides.