ML//inference//greedy decoding

Greedy decoding is a decoding strategy that picks, at each decoding step, the single token with the highest probability from the logits, and it is the deterministic choice for tasks where reproducibility matters more than variety. It always takes the best-looking option right now, in the hope that the sequence works itself out, with no randomness and no exploration.

Greedy decoding is a decoding strategy that picks, at each decoding step, the single token with the highest probability from the logits, and it is the deterministic choice for tasks where reproducibility matters more than variety. It always takes the best-looking option right now, in the hope that the sequence works itself out, with no randomness and no exploration.

It is fast (one forward pass per token, no branching) and deterministic (the same input always produces the same output), which makes it useful for benchmarking and for tasks where reproducibility matters.

Its fundamental problem is that locally optimal choices do not add up to globally optimal sequences. Picking the best token at each step can lead to a worse overall sequence than choosing a slightly less probable token early on.

For seq2seq tasks (translation, summarization), greedy decoding produces acceptable but mediocre output. Beam search, of which greedy decoding is the case with beam width 1, consistently outperforms it by exploring KKK paths instead of one, trading compute for better global sequences.

For open-ended LLM generation, greedy decoding produces repetitive, boring text. It gravitates toward high-frequency phrases because these are always the most probable next tokens. Sampling with temperature and top-p solves this by introducing controlled randomness: greedy picks the maximum of the distribution, sampling draws from it.

In practice, greedy decoding is the hidden default: a temperature of 0 in an API call requests greedy decoding, and any temperature above 0 is sampling. It is the safe mode for tasks where creativity is not wanted (code generation, factual Q&A).