ML//training//fine-tuning//catastrophic forgetting

Catastrophic forgetting is the loss of the knowledge a model learned during pre-training when fine-tuning on new data overwrites it, and it is the failure that adapter methods and parameter freezing exist to prevent. The model gets better at the new task but forgets everything else. The weights that encoded general knowledge get overwritten by task-specific gradients. It is often confused with model collapse, which loses information at the level of the data (training on synthetic data drops the tails) where forgetting loses it in the weights; both are irreversible.

Catastrophic forgetting is the loss of the knowledge a model learned during pre-training when fine-tuning on new data overwrites it, and it is the failure that adapter methods and parameter freezing exist to prevent. The model gets better at the new task but forgets everything else. The weights that encoded general knowledge get overwritten by task-specific gradients. It is often confused with model collapse, which loses information at the level of the data (training on synthetic data drops the tails) where forgetting loses it in the weights; both are irreversible.

This is the core problem that LoRA, QLoRA, and ReFT were designed to solve. Instead of updating all parameters (which risks overwriting pre-trained knowledge), they freeze most weights and only train small adapters or low-rank deltas. LoRA, the dominant one, never updates the original weights at all, so they are preserved exactly.

Parameter freezing is the simplest defense: freeze early layers (which encode general features like syntax and semantics) and only fine-tune later layers (which encode more task-specific patterns). The trade-off: less forgetting but also less adaptation.

Mechanistically, it happens because neural networks are distributed representations: the same weights participate in many different capabilities. When training optimizes for task A, the gradient does not know which weight updates will break task B.

Transfer learning depends on not forgetting: the whole point is to transfer knowledge from pre-training to a new task. Catastrophic forgetting is the failure mode where the transfer goes backward.

Not every fine-tune forgets badly. How much is lost depends on the data (how far it is from the original distribution), the method (full update or adapter) and the size and number of the updates; a light fine-tune on in-domain data may cost little, a long one on narrow data a lot. Measure old skills after training rather than assume either way.

Learning task after task without forgetting is the problem of continual learning, and no fully general solution exists. Practical mitigations include experience replay (mixing old data with new), elastic weight consolidation (penalizing changes to important weights), adapter methods, and keeping new knowledge outside the weights (agent memory).