ML//training//exposure bias

Exposure bias is the gap between what a model sees in training and what it sees at inference. During SFT, the model always sees correct context (human-written training data); at inference, it sees its own previous outputs, which may contain errors.

Exposure bias is the gap between what a model sees in training and what it sees at inference. During SFT, the model always sees correct context (human-written training data); at inference, it sees its own previous outputs, which may contain errors.

It is the student who only ever practiced with answer keys, then had to take the exam using its own messy drafts as reference.

The model was never trained on its own mistakes as context. When it generates a wrong token, that error becomes context for the next token, pushing toward more errors, a distribution the model never saw during training.

Exposure bias creates path dependency in the residual stream. Once the model starts generating in a wrong direction, each incorrect token shifts the context vector further from familiar territory. It is an attractor, hard to escape.

This is why extended thinking has a dark side: more generated tokens give an error more opportunities to compound into a cascade through that path dependency. For the same reason, longer reasoning chains give exposure bias more room to accumulate, which relates it to overthinking.

Three approaches mitigate it: scheduled sampling (mix model outputs into training), RLHF or DPO (train on model-generated outputs with a reward signal), and tree search (explore multiple branches to escape bad paths).