ML//neural network//loss function//cross-entropy

Cross-entropy is a loss function that compares the probabilities a classifier predicts with the target distribution, and it is the quantity a classification model is trained to minimize. For a target distribution \(y\) and predicted probabilities \(p\) it reads as follows.

Cross-entropy is a loss function that compares the probabilities a classifier predicts with the target distribution, and it is the quantity a classification model is trained to minimize. For a target distribution yyy and predicted probabilities ppp it reads as follows.

H(y,p)=−∑iyilog⁡piH(y,p)=-\sum_i y_i\log p_iH(y,p)=−i∑​yi​logpi​

For a one-hot classification target this reduces to −log⁡py-\log p_y−logpy​: confident and wrong predictions receive a large penalty, while confident correct predictions approach zero loss. For two classes it is the negative log-likelihood of a Bernoulli variable, −[ylog⁡p+(1−y)log⁡(1−p)]-[y\log p+(1-y)\log(1-p)]−[ylogp+(1−y)log(1−p)], the loss of logistic regression: saying 0.01 when the answer was 1 costs −log⁡0.01≈4.6-\log 0.01\approx4.6−log0.01≈4.6, saying 0.5 costs 0.69. With logistic regression it is convex, a single valley with no closed form; inside a deep network it is one term of a non-convex loss landscape.

Its reading in information theory explains the name. It splits as H(y,p)=H(y)+DKL(y ∥ p)H(y,p)=H(y)+D_{\mathrm{KL}}(y,|,p)H(y,p)=H(y)+DKL​(y∥p), and since the entropy of the labels is fixed, minimizing cross-entropy minimizes the KL divergence between the true and the predicted distribution, what you pay for believing ppp when the truth is yyy. A large language model is trained on exactly this, one token at a time.

It trains the relative probabilities of all the classes along with the winning one, so two models with equal accuracy can have very different cross-entropy. Minimizing it does not guarantee calibrated probabilities under distribution shift: a model can rank classes well and still be confidently wrong in the real world (probability calibration, out-of-distribution).