ML//inference//test-time compute//self-consistency

Self-consistency is a decoding strategy in which a language model samples several chains of thought for the same question at nonzero temperature and returns the final answer that most of them reach, and it is used to buy accuracy on questions with one short, checkable answer (a number, a multiple-choice letter, a class label) without any scorer or verifier. Wang and colleagues proposed it in 2022 as a replacement for the single greedy chain, and benchmark tables report it as **maj@k**, the majority answer of \(k\) samples.

Self-consistency is a decoding strategy in which a language model samples several chains of thought for the same question at nonzero temperature and returns the final answer that most of them reach, and it is used to buy accuracy on questions with one short, checkable answer (a number, a multiple-choice letter, a class label) without any scorer or verifier. Wang and colleagues proposed it in 2022 as a replacement for the single greedy chain, and benchmark tables report it as maj@k, the majority answer of kkk samples.

The intuition is that there are many ways to reason to the right answer and they converge, while mistakes scatter across different wrong answers. If each chain alone is right 60% of the time and the wrong ones disagree with each other, five chains put the right answer ahead of any single wrong one most of the time; even in the worst case of one shared wrong answer, a majority of five independent chains is right about 68% of the time. The reasoning text is thrown away: only the final answers are counted.

The vote needs answers that can be compared.

Each chain's final answer is extracted and normalised before counting (3/4, 0.75 and 75% are one answer; units, spacing and capitalisation are stripped). A summary, an essay or a piece of code has no majority to count, which is where best-of-N and its scorer take over.

Its gains stop where the chains stop being independent. Samples of one model share its blind spots, so when the model is consistently wrong the vote returns its bias with more confidence, and accuracy flattens after a few tens of samples (the same vote that redundant controllers take, majority voting, assumes channels that fail independently, which sampled chains are not).

The agreement fraction is a free confidence signal. Nine chains of ten on the same answer is weak evidence that it is right, three of ten a reason to abstain or escalate; it measures the model's stability on the question, never its correctness.

It costs NNN times the tokens of one chain, and on a server the NNN chains decode together as a batch, so the latency grows far less than the bill (inference batching). It is one of the simplest forms of test-time compute.