ML//neural network//loss landscape//mode connectivity

Mode connectivity is the finding that different local minima in the loss landscape are connected by paths along which the loss stays low. These paths are curved (Bezier curves, polygonal chains) rather than straight, but they exist, and training and test accuracy remains nearly constant along them.

Mode connectivity is the finding that different local minima in the loss landscape are connected by paths along which the loss stays low. These paths are curved (Bezier curves, polygonal chains) rather than straight, but they exist, and training and test accuracy remains nearly constant along them.

Garipov, Izmailov and colleagues discovered it (NeurIPS 2018). The implication is fundamental: the loss landscape of deep networks is not a disconnected archipelago of isolated valleys. It is more like a mountain range with passes between valleys. The optimizer does not need to find the right minimum; it needs to find any minimum in a connected basin, and from there, low-loss paths lead to other good solutions.

This has practical applications in model ensembling (averaging weights of models along these connecting paths produces better predictions than either endpoint) and in understanding why different random initializations converge to solutions of similar quality.