ML//training//post-training//reward model

A reward model is a neural network trained to score AI outputs on a scalar, answering how good a response is.

A reward model is a neural network trained to score AI outputs on a scalar, answering how good a response is.

It is trained on preference pairs of the form Bx≻ByB_x \succ B_yBx​≻By​ (response xxx preferred to response yyy) from RLHF or RLAIF, and PPO uses it to provide the reward signal during training; DPO skips it entirely, mapping preferences directly to token gradients.

It is a learned reward function: where a game or a unit test gives the reward by rule, here a network fitted to human comparisons stands in for a goal (helpful, honest, harmless) that no rule can state. Training it costs its own labelled data and compute, one of the reasons RLHF is the expensive rung of post-training.

The quality ceiling of PPO is the quality ceiling of the reward model: every blind spot becomes a reward hacking target.

A reward model is also a standalone product beyond training. At inference time it can score and filter outputs, rerank candidates or detect policy violations without retraining.

Safe RLHF trains two separate RMs (helpfulness and harmlessness) to avoid conflating the two axes into one ambiguous scalar.