ML//neural network//loss landscape//flat vs sharp minima

A flat minimum is a region of the loss landscape where loss stays low across a wide range of weight configurations. A sharp minimum is a narrow valley where loss rises steeply with small weight perturbations.

A flat minimum is a region of the loss landscape where loss stays low across a wide range of weight configurations. A sharp minimum is a narrow valley where loss rises steeply with small weight perturbations.

The hypothesis: flat minima generalize better because they tolerate the small differences between training and test data. A model sitting in a sharp minimum is fragile, having memorized a narrow configuration that works on the training data (overfitting); tiny distribution shifts push it up the walls. A model in a flat minimum barely notices.

The evidence is mixed. Dinh et al. (2017) showed that sharpness can be manipulated by reparameterization without changing generalization, arguing the connection is not as clean as it seems. Other work (Jiang et al., 2019) found that flatness correlates well with generalization when measured correctly. The practical consensus leans toward flat being better.

Small batch sizes tend to find flat minima (the noise in small batches acts as implicit regularization, bouncing the optimizer out of sharp valleys; batch normalization adds noise of its own through its batch statistics, which may help in the same way). Large batch sizes converge to sharper minima.