ML//inference//sampling

Sampling is the step of text generation that chooses the next token from the probability distribution over the vocabulary, and it introduces stochasticity: the same input can produce different outputs, each a new path on the manifold. The token picked becomes the next step on that manifold: a low temperature hugs the most likely path, a high one explores branches.

Sampling is the step of text generation that chooses the next token from the probability distribution over the vocabulary, and it introduces stochasticity: the same input can produce different outputs, each a new path on the manifold. The token picked becomes the next step on that manifold: a low temperature hugs the most likely path, a high one explores branches.

TemperatureScales the logits before the softmax; low is deterministic, high is creative

Top-kConsiders only the kkk highest-probability tokens

Top-p (nucleus)Considers only the tokens whose cumulative probability exceeds ppp

The temperature is cranked up when generating candidate outputs for RLHF or DPO, which need diversity.