ML//neural network//loss landscape

The loss landscape is the surface obtained by plotting the loss function value against all possible combinations of a neural network's weights. Each point on the surface represents one specific configuration of weights and how badly the network performs with that configuration. The loss function sets the vertical axis, so a different one (cross-entropy, MSE) gives a different surface over the same weights. The gradient descent optimizer moves across this surface, always trying to go downhill, and the shape of the terrain decides whether it converges quickly, gets stuck or oscillates.

The loss landscape is the surface obtained by plotting the loss function value against all possible combinations of a neural network's weights. Each point on the surface represents one specific configuration of weights and how badly the network performs with that configuration. The loss function sets the vertical axis, so a different one (cross-entropy, MSE) gives a different surface over the same weights. The gradient descent optimizer moves across this surface, always trying to go downhill, and the shape of the terrain decides whether it converges quickly, gets stuck or oscillates.

A typical network has millions of parameters, so the landscape lives in millions of dimensions and cannot be seen directly. Visualization requires dimensionality reduction (projecting onto a 2D plane using random orthogonal directions, then plotting loss as the third axis). These projections are imperfect (like a photograph of a 3D scene) but preserve enough structure to generate useful insights about the morphology of the training process.

Key features of loss landscapes include minima (valleys where the optimizer converges), saddle points (points that look like minima in some directions but not others), plateaus where the gradient almost vanishes, and the overall smoothness or ruggedness of the terrain (which correlates with how well the network generalizes, flat vs sharp minima).

Network losses are non-convex, unlike the single valley of logistic regression. It frightens less than it seems: in large networks many minima are about equally good, and saddles and plateaus slow training more than bad minima trap it (mode connectivity).

The practical consequence is seed variance. Two trainings that differ only in the random seed (initial weights, order of the minibatches) end in different networks, with similar average accuracy and different mistakes on individual cases. A model that must be reproduced, audited or certified is one specific set of weights, never the recipe that produced it; and the disagreement among networks trained from several seeds is a cheap uncertainty signal (out-of-distribution).