ML//Inference//distributional shift

The probability distribution over next tokens **changes** depending on what's already in the context window — different context = different basin of attraction in probability space.

The probability distribution over next tokens changes depending on what's already in the context window — different context = different basin of attraction in probability space.

During pretraining, the model saw two kinds of text: direct answers and reasoned-then-concluded text. These form distinct regions in latent space

After reasoning tokens, the model enters the basin where pretraining data had coherent conclusions following explicit thought. After no reasoning, it's in the "cold answer" basin — less reliable.

Random text as context doesn't help: it positions the model in a region with no semantic structure. Structured reasoning activates a specific basin because that pattern co-occurred systematically with correct conclusions during pretraining.

The technical effect: reasoning tokens reduce entropy of the output distribution — fewer plausible continuations, and the remaining ones tend to be more correct.

This is why extended thinking works mechanistically: intentional distributional shift toward a useful basin of attraction

In RLHF/RLAIF: models learned that humans rate reasoned responses higher. The thinking reproduces that high-evaluation structure even when hidden.