ML//training//post-training//RLHF

RLHF (reinforcement learning from human feedback) is a training method that fits a model to human preferences, collected by having humans choose between two responses \(B_x\) and \(B_y\), generated with different temperatures or strategies.

RLHF (reinforcement learning from human feedback) is a training method that fits a model to human preferences, collected by having humans choose between two responses BxB_xBx​ and ByB_yBy​, generated with different temperatures or strategies.

The preference pairs (By≻BxB_y \succ B_xBy​≻Bx​) feed either DPO directly or an RM for PPO. The data format is algorithm-agnostic: the same (prompt, chosen, rejected) triplets serve any preference-based method, GRPO included, consumed on-policy by PPO, which generates and scores, or off-policy by DPO on fixed pairs (on-policy vs off-policy). The feedback loop works while humans can still judge which output is better; beyond that it breaks, which is the scalable oversight problem of alignment. RLAIF replaces the human judges with AI judges, faster, cheaper and more scalable, but only as good as its constitution.

The classic recipe has three stages: collect several answers per prompt and human rankings of them; train a reward model that learns to score answers the way the rankers would; then optimize the model's policy with PPO to raise that score, with a KL divergence penalty that stops it drifting far from the starting model. Unlike SFT, it never needs anyone to write the ideal answer, only to say which of two is better, and that is its strength for qualities nobody can write down. It is also the costly rung of post-training: human labels, a second network to train, and several copies of the model in memory during PPO.

InstructGPT (GPT-3.5) was the inflection point: it took GPT-3 and added SFT and RLHF, creating the first assistant. Safe RLHF splits the annotation into separate human scores for helpfulness and harmlessness, with separate RMs and a controllable tradeoff.