ML//training

Training adjusts model parameters to reduce an objective over examples or interactions. For modern language models it may include pre-training, instruction-focused SFT and preference or reward optimization through methods such as RLHF and DPO.

Training adjusts model parameters to reduce an objective over examples or interactions. For modern language models it may include pre-training, instruction-focused SFT and preference or reward optimization through methods such as RLHF and DPO.

Training spends large amounts of compute to create or modify the reusable weights. Inference then amortizes that investment across many outputs. The split is economically important: an organization may tolerate an expensive training run while requiring every future token to be cheap.

Rough evolution: pre-training → SFT → RLHF/DPO

Data format is algorithm-agnostic: same (prompt, chosen, rejected) triplets feed DPO, PPO, GRPO. The difference is on-policy vs off-policy consumption.

Catastrophic forgetting is real. Training only on new data overwrites old knowledge; solution: replay buffers, mixing a % of old data during new training.

Taxonomic annotation (labeling) is a data enrichment step that can feed into any method: SFT training pairs, constitution enhancement for DPO, pre-training data, or RAG