ML//inference//extended thinking//overthinking

Overthinking is the empirically documented effect by which forcing long extended thinking on simple problems makes a model worse instead of better. On simple questions, where the direct answer has high probability from the start, thinking introduces three risks:

Overthinking is the empirically documented effect by which forcing long extended thinking on simple problems makes a model worse instead of better. On simple questions, where the direct answer has high probability from the start, thinking introduces three risks:

It opens more opportunities for self-consistency bias: the model prefers branches consistent with its own pretraining biases, not necessarily the most correct ones.

The model can talk itself out of a correct answer, because reasoning generates context that activates irrelevant complexity.

It drifts toward regions of latent space associated with unnecessary complexity.

Papers show o1 performs worse than GPT-4 on simple common-sense questions precisely for this reason: the thinking generates context that triggers a distributional shift to the wrong region. Token budgets (such as Anthropic's) are a partial solution, serving as cost control and also keeping the model from thinking too much on problems that do not need it. The deeper problem is that the model has no reliable metacognition: it cannot assess whether it needs to think more, and it simply generates tokens until told to stop.