ML//neural network//dropout

Dropout is a regularization technique for neural networks that randomly masks activations with probability \(p\) during training and rescales the survivors, and it is used to keep a network from relying on fragile combinations of features: switch off neurons at random so that none is indispensable. At inference the full network is used without random masks.

Dropout is a regularization technique for neural networks that randomly masks activations with probability ppp during training and rescales the survivors, and it is used to keep a network from relying on fragile combinations of features: switch off neurons at random so that none is indispensable. At inference the full network is used without random masks.

The noise discourages fragile co-adaptation, because a feature cannot assume that every collaborating feature will always be present. Dropout can be interpreted loosely as averaging many subnetworks that share weights, although it is not identical to training an explicit ensemble.

It is one of four regularizers carried over from classical ML (regularization): weight decay (ridge for networks, the decoupled form in AdamW), early stopping, dropout and, usually the most effective, data augmentation, which must respect the physics of the signal.

Dropout was crucial in early deep networks. In many modern architectures, abundant data, normalization, augmentation, and weight decay reduce how much it helps; regularization is workload-dependent, not ceremonial.

Forgetting to switch it off at inference (the model.eval() call in PyTorch) makes the predictions random from run to run, one of the plain deployment bugs that separate a training script from a production model.