mathematics//statistics//bootstrap

The bootstrap is a resampling method that estimates the uncertainty of a statistic by drawing the data with replacement many times and looking at the spread of the results, and it is used to put a confidence interval on a score without assuming a formula for its distribution. It applies to any statistic computed from data, a model's edge over a baseline included, and it answers one practical question: how much would this number change with other data?

The bootstrap is a resampling method that estimates the uncertainty of a statistic by drawing the data with replacement many times and looking at the spread of the results, and it is used to put a confidence interval on a score without assuming a formula for its distribution. It applies to any statistic computed from data, a model's edge over a baseline included, and it answers one practical question: how much would this number change with other data?

The recipe is mechanical. From NNN readings, draw NNN at random with replacement (some appear twice, some not at all), recompute the statistic, and repeat about a thousand times; the spread of the thousand results is the standard error of the statistic, and their 2.5th and 97.5th percentiles give a rough 95 % confidence interval. It earns its place on quantities with no formula at hand: a percentile of a latency, a maximum, a ratio of two means, the area under a ROC curve.

It gives a confidence interval without assuming a formula. If the interval on a model's edge over a baseline crosses zero, the edge is not established.

Resample the independent unit: for grouped data (sessions, patients, machines), resample whole groups, not rows, or the interval is falsely narrow. For a time series, resample contiguous blocks long enough to hold the memory of the signal (autocorrelation).

It cannot invent what the sample never contained. Extremes beyond the largest observation, a maximum, or a very high percentile from a few hundred points come out too narrow, because every resample is built from the same few tail values (heavy-tailed distribution).

It belongs offline. A thousand recomputations of a statistic take seconds on a PC; a microcontroller inside a loop keeps only running moments (Welford's algorithm).

It pairs naturally with cross-validation to put error bars on a fold-averaged score.