ML//inference//distributional shift

Distributional shift is the change in the probability distribution over next tokens that follows from what is already in the context window: a different context puts the model in a different basin of attraction in probability space. oBpGr85I is the other face of the same phenomenon: in both, the context changes what the model predicts. Elsewhere in machine learning the same words name a gap between the data a model was trained on and the data it meets, which this wiki calls out-of-distribution for a single unfamiliar input and data drift for a population that changes over time.

Distributional shift is the change in the probability distribution over next tokens that follows from what is already in the context window: a different context puts the model in a different basin of attraction in probability space. In-context learning is the other face of the same phenomenon: in both, the context changes what the model predicts. Elsewhere in machine learning the same words name a gap between the data a model was trained on and the data it meets, which this wiki calls out-of-distribution for a single unfamiliar input and data drift for a population that changes over time.

During pretraining the model saw two kinds of text, direct answers and text that reasons and then concludes, and they form distinct regions in latent space.

After reasoning tokens, the model enters the basin where the pretraining data had coherent conclusions following explicit thought. Without reasoning it stays in the less reliable basin of the cold answer.

Random text as context does not help, because it places the model in a region with no semantic structure. Structured reasoning activates a specific basin because that pattern co-occurred systematically with correct conclusions during pretraining.

The technical effect is that reasoning tokens reduce the entropy of the output distribution: fewer continuations remain plausible, and the remaining ones tend to be more correct.

This is why extended thinking works mechanistically: it is an intentional distributional shift toward a useful basin of attraction.

In RLHF and RLAIF, models learned that humans rate reasoned responses higher, and the thinking reproduces that highly rated structure even when it is hidden.