ML//training//dataset//model collapse

Model collapse is what happens when models are trained on their own outputs (synthetic data): each generation forgets a little more of what made the original interesting, an ouroboros in gradient space. The tail distribution erodes first, then the whole distribution narrows.

Model collapse is what happens when models are trained on their own outputs (synthetic data): each generation forgets a little more of what made the original interesting, an ouroboros in gradient space. The tail distribution erodes first, then the whole distribution narrows.

1Generation 1plausible but slightly averaged outputs2Generation 2trained on generation 1, less variance3Generation Na bland, mode-seeking average

The tails are the first casualty: rare events, unusual phrasings, minority perspectives, edge cases. The model forgets what it never saw enough of.

It is especially dangerous for reasoning, because the most sophisticated reasoning chains live in the tail of the pretraining distribution. Once lost, no amount of RL or extended thinking recovers them: pretraining is the ceiling.

It differs from mode collapse in GANs, where the generator ignores modes during training. In model collapse it is the training data that loses modes across generations.

As AI-generated content floods the internet, future models trained on web data inherit compounding distortions, and curating the dataset becomes existential: verified human-generated data is needed in the mix to anchor the distribution.