ML//transformer//logits

Logits are the raw scores a model assigns over the entire vocabulary before normalization: the normalized output multiplied by \(W_U\) gives the logits. Each value is the dot product between the context vector and one row of the LM head, and measures how aligned the output is with each possible next token.

Logits are the raw scores a model assigns over the entire vocabulary before normalization: the normalized output multiplied by WUW_UWU​ gives the logits. Each value is the dot product between the context vector and one row of the LM head, and measures how aligned the output is with each possible next token.

Logits are not probabilities: they can be negative or arbitrarily large, and softmax converts them into a proper distribution.

Temperature divides the logits before softmax, logits/T\text{logits}/Tlogits/T. A low TTT gives a peaky distribution (confident), a high TTT a flat one (creative).

In knowledge distillation the student learns from the teacher's logits (softened through a temperature) rather than from its single chosen token, because the scores of the runner-up tokens carry what the teacher knows about which alternatives were nearly right.

In DPO the model computes logits for both the preferred and the rejected outputs, and training widens the gap between them.