ML//benchmark//Perplexity

2026-02-25

Perplexity is the standard metric for language model quality, the exponential of the cross-entropy loss. A perplexity of 10 means the model is as confused as if it had to choose uniformly between 10 equally likely options at each token.

It is derived directly from cross-entropy: perplexity=exp⁡(loss)\text{perplexity}=\exp(\text{loss})perplexity=exp(loss), or 2loss2^{\text{loss}}2loss when the loss is counted in bits. A lower loss means a lower perplexity and a better model, and knowing one gives the other.

Perplexity is used instead of the loss because it is more interpretable. A loss of 2.3 means nothing intuitive, while a perplexity of 10 says that on average the model narrows each token prediction down to about 10 candidates. The number only compares within one test set and one tokenizer: the same model scores differently on Wikipedia and on forum posts, and a model with a larger vocabulary has fewer, harder tokens to predict, so two papers' perplexities on different setups say nothing about which model is better.

Scaling laws are usually plotted as loss against compute, but perplexity is the metric practitioners report. A figure such as 8.5 perplexity on WikiText-103 is a benchmark result.

Perplexity depends on the tokenizer. A model with BPE-50k and a model with BPE-100k have different perplexities on the same text because they're predicting different token sequences. Comparing perplexity across models with different tokenizers is misleading. The same holds across evaluation texts: perplexity is a property of a model on one dataset, the exponential of its mean log loss there, so two numbers measured on different corpora say nothing about which model is better.

Perplexity is not useful for evaluating SFT or RLHF models directly: a helpful chatbot might have higher perplexity than a base model because it generates less predictable (more diverse, more specific) responses. Downstream benchmarks like MMLU or GPQA measure what actually matters.