ML//neural network//loss function//cross-entropy

Cross-entropy compares a target probability distribution \(y\) with predicted probabilities \(p\):

Cross-entropy compares a target probability distribution yyy with predicted probabilities ppp:

H(y,p)=−∑iyilog⁡piH(y,p)=-\sum_i y_i\log p_iH(y,p)=−i∑​yi​logpi​

For a one-hot classification target this reduces to −log⁡py-\log p_y−logpy​: confident and wrong predictions receive a large penalty, while confident correct predictions approach zero loss.

It trains relative probabilities, not merely the winning class. Two models with equal accuracy can have very different cross-entropy.

Minimizing it does not guarantee calibrated probabilities under distribution shift. A model can rank classes well and still be confidently wrong in the real world.