mathematics//statistics//heavy-tailed distribution

2022-03-20

A heavy-tailed distribution is a probability distribution whose extreme values are far more frequent than a normal distribution with the same centre and spread would allow, and recognising one changes how a system is designed: the mean misleads, and what must be planned for is the tail. Network and operating-system latencies, the durations of traffic jams in a fleet, the sizes of blackouts and the repair times of a plant all behave this way.

A general-purpose operating system shows the shape in one number pair: a mean latency of 2 ms and, now and then, 80 ms. Under a bell, a value 40 times the mean with a spread of a millisecond would never be seen; here it shows up every hour. The cause is structural. A bell arises from many small independent effects adding up (central limit theorem); heavy tails arise when one effect can dominate (a cache miss cascade, a garbage collector pausing the process), when effects multiply instead of adding, or when components are coupled so that one failure loads the next (cascading failure).

With heavy tails the mean and the standard deviation become unstable, and the high percentiles say much more.

The sample mean converges slowly, or never (a Cauchy distribution has no mean at all), and one rare event moves it more than a million ordinary ones. Describe and specify such quantities by their p99 or p99.9 (percentile); in a real-time computing loop, what kills is the worst case.

A Gaussian test is the first casualty. A 3σ3\sigma3σ alarm threshold that should fire 0.27 % of the time on healthy data fires several times more often when the noise has heavy tails, and a chi-square gate lets through fewer of the good readings than it promises (chi-square distribution). Robust thresholds (the median and the MAD instead of the mean and σ\sigmaσ) hold up (robust statistics).

Many rare events are a sign worth reading. A Gaussian exceeds 6σ6\sigma6σ with probability about 2⋅10−92\cdot10^{-9}2⋅10−9, so a 1 kHz signal can honestly show one such event a week; dozens a day mean the noise is not Gaussian, not independent, or not noise.

Large systems manufacture tails. Queues near full utilisation, coupled grids and fleets sharing corridors produce events whose size has no natural scale; the historical average of blackouts underestimates the risk, and design moves to margins, buffers and weak coupling (complex system, tails over means).

The extreme of the extreme is the event with no record at all (black swan); a heavy tail is still a distribution, fitted from data, with percentiles that can be estimated given enough of them.