ML//neural network//loss function

A loss function is a function that scores a prediction against its target, and it is the signal a model is trained to minimize. Training minimizes its aggregate value through backpropagation, but the loss is a chosen proxy for desired behavior rather than an objective measurement of how wrong the system is.

A loss function is a function that scores a prediction against its target, and it is the signal a model is trained to minimize. Training minimizes its aggregate value through backpropagation, but the loss is a chosen proxy for desired behavior rather than an objective measurement of how wrong the system is.

Every loss is also a noise model. When the loss is minus the logarithm of the likelihood, minimizing its average over the data (empirical risk minimization) is maximum likelihood estimation with a very flexible model. Squared error is the maximum-likelihood loss for Gaussian, independent noise of constant variance; weighting each point by 1/σi21/\sigma_i^21/σi2​ gives weighted least squares, the same weights sensor fusion uses; noise with spikes more frequent than the Gaussian allows (a Laplace distribution) gives the sum of absolute values, whose minimum in the simplest case is the median. Calibrating a cheap temperature probe against a reference with squared error assumes Gaussian errors, and one stuck reading of 850 °C drags the whole fit, where an absolute loss would barely move.

Every loss implies a noise assumption, known or not.

Changing the loss changes what the model believes about its errors, and the right loss follows from how the sensor or the label actually goes wrong.

Cross-entropy is common for classification and next-token prediction; squared error is common for regression.

The loss determines which mistakes look equivalent to the optimizer. If an important real-world cost is absent or badly weighted, the model has no gradient telling it to care: a missed bearing fault and a false alarm cost the plant very different amounts, and a symmetric loss treats them alike.

Evaluation metrics and training losses serve different jobs: the metric describes success, while the loss must also provide a useful, differentiable learning landscape (loss landscape).

Choosing a loss is therefore a small act of governance: it translates human priorities into geometry.