ML//training//post-training//DPO

DPO (Direct Preference Optimization) is a method for training a language model on preferences between pairs of outputs, and it trains like SFT with two options: it maps a preference (the contrastive pairs \(B_y > B_x\)) directly back to individual token derivatives, widening the logit gap between the tokens that led to the better path.

DPO (Direct Preference Optimization) is a method for training a language model on preferences between pairs of outputs, and it trains like SFT with two options: it maps a preference (the contrastive pairs By>BxB_y > B_xBy​>Bx​) directly back to individual token derivatives, widening the logit gap between the tokens that led to the better path.

What it removes from RLHF is the machinery: no separate reward model to train and no online loop that generates and scores answers during training, only a loss over fixed pairs on the probabilities of the preferred and rejected answers. That makes it the cheap way into preference training in post-training; it is a different method from PPO, and neither wins everywhere.

The model computes logits for both outputs and adjusts its weights so that the preferred output gets higher logits (it feels natural) and the rejected one lower logits (it feels unnatural).

It has no ceiling, unlike SFT, whose imitation of single targets sets a rigid one; learning from contrasts, the model is free to roam latent space and find new ways to be smart, because relational gradient learning improves in-context knowledge retrieval.

It can only rearrange knowledge already baked in during pre-training, and it never creates new knowledge.

KL divergence acts as an anchor, penalizing the new probability distribution when it drifts too far from the reference model. Without it the result is reward hacking (adversarial shortcuts that score high and mean nothing).

It is off-policy: it trains on pre-collected preference pairs and loses 3 to 7% of performance out of domain compared with RLHF, because it cannot explore beyond the fixed dataset.

The data format is algorithm-agnostic: the same (prompt, chosen, rejected) triplets feed both DPO and PPO, which share its shape and differ in how they consume it: PPO scores the whole sequence through a reward model, so every token gets the same offset, while DPO derives a gradient per token with no reward model, a better credit assignment.

When humans choose between the two outputs, as in RLHF, the pairs are slow to collect but grounded in real human judgment. When machines produce the pairs (neural networks or LLM reasoning), the method is iterative DPO, the best training method today.