ML//training//dataset//data augmentation

2026-02-25

Data augmentation is a training technique that creates new examples by transforming existing ones (flipping an image, adding noise, paraphrasing text, masking tokens), so that the effective size of the dataset grows without collecting new data.

It is the primary defense against overfitting when more data cannot be had. Showing the model different views of the same example forces it to learn invariances (a cat is still a cat when flipped) instead of memorizing specific pixel patterns.

In vision the usual transformations are random cropping, rotation, color jitter and cutout (masking random patches), all standard for CNN training. Convolution's translation equivariance handles some invariances architecturally, but augmentation covers the rest.

Every transformation is a claim that the label does not change under it, so it encodes an invariance, and the invariance must be true of the physics. Adding sensor noise or small gain errors to a vibration window is safe; reversing it in time is not, because the response to an impact rises sharply and decays slowly, and a time-reversed one grows and cuts off, which no machine produces. Shifting a spectrogram along the frequency axis is wrong for the same reason: 100 Hz is not 200 Hz. In networks it is usually the most effective regularizer of all, above dropout and weight decay.

In NLP the transformations are back-translation (translating to another language and back), synonym replacement and random insertion or deletion. They are less natural than in vision, because small changes to a text can alter its meaning.

Masked language modeling (BERT) is arguably a form of augmentation: randomly masking tokens gives each training pass a different view of the same sentence, and the model must reconstruct the original from partial information.

Augmentation transforms real examples and preserves their ground truth, while synthetic data generates entirely new examples from a model or from procedural rules. Augmentation is safer because the transformations are meaning-preserving by design, while synthetic data can introduce model collapse if the generation model has systematic biases.