ML//generalization//regularization
Regularization is the family of techniques that prevent overfitting by charging a model for its complexity, so that it keeps from memorizing its training data: an explicit penalty added to the loss, or a constraint on training such as dropout, weight decay (L2), data augmentation or batch normalization. It is the art of making the model slightly worse on the training data so that it is better on everything else.
Regularization is the family of techniques that prevent overfitting by charging a model for its complexity, so that it keeps from memorizing its training data: an explicit penalty added to the loss, or a constraint on training such as dropout, weight decay (L2), data augmentation or batch normalization. It is the art of making the model slightly worse on the training data so that it is better on everything else.
The explicit form adds a rent for every parameter to what is minimized:
minw ∑i=1N(yi−w⊤xi)2+λ Ω(w)\min_{w}\;\sum_{i=1}^{N}\big(y_i-w^{\top}x_i\big)^2+\lambda\,\Omega(w)wmini=1∑N(yi−w⊤xi)2+λΩ(w)
Here Ω\OmegaΩ measures complexity and λ\lambdaλ sets its price. With Ω=∥w∥22\Omega=\lVert w\rVert_2^2Ω=∥w∥22, the sum of the squared weights, it is ridge regression, which shrinks every weight and stabilizes ill-conditioned problems; with Ω=∥w∥1\Omega=\lVert w\rVert_1Ω=∥w∥1, the sum of their absolute values, it is lasso, which drives many weights exactly to zero and so selects features. λ\lambdaλ is a hyperparameter, chosen on validation data (train-validation-test split).
Regularization as a prior. A ridge or Tikhonov penalty is exactly a Gaussian prior on the parameters: the MAP estimate with a prior centred at zero and spread τ\tauτ is ridge with λ=σ2/τ2\lambda=\sigma^2/\tau^2λ=σ2/τ2, where σ\sigmaσ is the noise. Regularizing is having prior opinions and admitting it; with many data the fit dominates, with few the prior keeps the model from believing the noise.
Other levers regularize without a penalty term: pruning a decision tree, stopping training when the validation error starts to rise (early stopping), and above all more data, which lowers variance without adding bias (bias-variance trade-off).