ML//inference//sampling//top-p
Top-p, or nucleus sampling, is a method for choosing the next token that keeps only the most probable tokens, added until their cumulative probability reaches \(p\) (e.g. 0.95), and samples among them. It is adaptive: for a confident prediction it considers few tokens, for an uncertain one many.
Top-p, or nucleus sampling, is a method for choosing the next token that keeps only the most probable tokens, added until their cumulative probability reaches ppp (e.g. 0.95), and samples among them. It is adaptive: for a confident prediction it considers few tokens, for an uncertain one many.
Because it adapts to the shape of the distribution, it is generally preferred over top-k.