ML//neural network//loss landscape//saddle point

A saddle point is a critical point of a function, here a point in the loss landscape, where the gradient is zero but the point is neither a minimum nor a maximum, because the surface curves upward in some directions and downward in others, like the center of a horse saddle. The textbook case is \(f(x,y)=x^2-y^2\) at the origin: a minimum along \(x\), a maximum along \(y\), and a gradient of exactly zero.

A saddle point is a critical point of a function, here a point in the loss landscape, where the gradient is zero but the point is neither a minimum nor a maximum, because the surface curves upward in some directions and downward in others, like the center of a horse saddle. The textbook case is f(x,y)=x2−y2f(x,y)=x^2-y^2f(x,y)=x2−y2 at the origin: a minimum along xxx, a maximum along yyy, and a gradient of exactly zero.

In high-dimensional spaces, saddle points vastly outnumber true local minima. A point is a local minimum only if curvature is positive in every dimension; a saddle point only needs one negative curvature direction. With millions of dimensions, the probability that all curvatures happen to be positive is vanishingly small, so what looks like a local minimum in a low-dimensional projection is often a saddle point.

Mathematically, saddle points are identified by the Hessian matrix having a mix of positive and negative eigenvalues (222 and −2-2−2 for the example above). Around one, the gradient is tiny over a wide region, so the loss curve shows a plateau that is easily mistaken for convergence. Modern optimizers like Adam handle saddle points better than vanilla SGD because their momentum terms carry them through flat regions where the gradient is near zero, and the noise of stochastic gradient descent itself nudges the weights off the ridge.