ML//training//post-training//SFT

SFT (supervised fine-tuning) is a training method that teaches a model to follow a script: for each input \(X\), the output \(A\) it should produce. A set of such pairs is called an instruction tuning dataset. Pre-training teaches the language; SFT teaches the instruction-following format on top of it. The loss is the same next-token cross-entropy, computed only on the desired answer: given *Summarize this article in three sentences* and a reviewed summary, training raises the probability of exactly that summary. It is the first rung of post-training, and it names the objective while leaving open how many weights move: SFT can update every weight or run through LoRA with the base frozen.

SFT (supervised fine-tuning) is a training method that teaches a model to follow a script: for each input XXX, the output AAA it should produce. A set of such pairs is called an instruction tuning dataset. Pre-training teaches the language; SFT teaches the instruction-following format on top of it. The loss is the same next-token cross-entropy, computed only on the desired answer: given Summarize this article in three sentences and a reviewed summary, training raises the probability of exactly that summary. It is the first rung of post-training, and it names the objective while leaving open how many weights move: SFT can update every weight or run through LoRA with the base frozen.

It creates a rigid ceiling, which DPO does not have: SFT teaches the format, DPO the quality. Through cross-entropy loss and SGD, AAA acts as a massive gravity attractor (singular attractor learning), so the model stops exploring, and valid, beautiful synonyms (BxB_xBx​, ByB_yBy​, BzB_zBz​) are sacrificed at the altar of literal imitation.

The model becomes a helpful companion, obedient but uncreative.

It carries exposure bias. During SFT the model always sees perfect human-written context, but at inference it sees its own outputs, and errors compound because it never trained on its own mistakes as context.

Iterative SFT (rejection sampling, or RFT) turns the dataset into a loop that the model feeds itself:

1Generate many answers2Pick the best3Treat it as ground truth4Train on it

Its downside is an echo chamber that amplifies biases and slop until the model loses edge and variety, and it is average in, average out: if the best of 100 answers is a 7/10, the training teaches mediocrity. Synthetic SFT is the umbrella for RFT, Constitutional AI outputs used for SFT, distillation (a big model writes, a small model learns) and self-instruct (the model generates the questions too).