ML//Training//dataset//tail distribution

The rare events at the edges of a probability distribution: the 0.1% cases that standard training underweights.

The rare events at the edges of a probability distribution: the 0.1% cases that standard training underweights.

In language: unusual phrasings, rare factual combinations, edge-case reasoning chains. In safety: jailbreaks, adversarial inputs, novel failure modes.

Why they matter: robustness lives in the tails. A model that handles the average case perfectly but fails on tails is fragile, and failures in tails are often the most consequential.

Model collapse erodes tails first because they have the weakest signal: the model "forgets" what it barely learned.

Reward hacking exploits tail blindness: if the RM hasn't seen a particular adversarial pattern, the model can optimize toward it without penalty.

Data augmentation and targeted collection of rare cases are the main defenses, but you can't augment what you don't know exists.