ML//training

Training is the process of adjusting a model's parameters to reduce an objective over examples or interactions, and it produces the reusable weights that inference then puts to use. For modern language models it may include pre-training and then the stages of post-training: instruction-focused SFT and preference or reward optimization through methods such as RLHF and DPO. What a run costs, and why that is a different number from a lab's budget or a product's serving bill, is training cost.

Training is the process of adjusting a model's parameters to reduce an objective over examples or interactions, and it produces the reusable weights that inference then puts to use. For modern language models it may include pre-training and then the stages of post-training: instruction-focused SFT and preference or reward optimization through methods such as RLHF and DPO. What a run costs, and why that is a different number from a lab's budget or a product's serving bill, is training cost.

Training spends large amounts of compute to create or modify the reusable weights. Inference then amortizes that investment across many outputs: training pays to change the weights once, inference pays to use them each time. The split is economically important: an organization may tolerate an expensive training run while requiring every future token to be cheap. The rough evolution of a language model's training runs in three stages:

1Pre-training2SFT3RLHF or DPO

Outside language models the loop is the same at a smaller scale. A network is trained by stochastic gradient descent on minibatches of 32 to a few thousand examples, and one full pass over the training data is an epoch; a typical run takes tens of epochs, with a validation fold watched after each one to decide when to stop (early stopping). Most classical models of supervised learning skip the epochs altogether: a regression is one linear solve and a boosted forest trains in minutes on a laptop CPU.

Federated learning trains one model across many devices that keep their data: each drone or phone computes an update on its own records and only the updates are pooled. It answers privacy and bandwidth limits; for learned controllers across a fleet it is still mostly research.

The data format is algorithm-agnostic: the same (prompt, chosen, rejected) triplets feed DPO, PPO and GRPO. The difference is whether they are consumed on-policy or off-policy.

Catastrophic forgetting is real: training only on new data overwrites old knowledge. The remedy is a replay buffer, mixing a share of old data into the new training.

Taxonomic annotation (labeling) is a data enrichment step that can feed any method: SFT training pairs, constitution enhancement for DPO, pre-training data or RAG.