ML//Training//reward model

A neural network trained to score AI outputs on a scalar: "how good is this response?"

A neural network trained to score AI outputs on a scalar: "how good is this response?"

Trained using "Bx > By" preference pairs from RLHF or RLAIF

Used by PPO to provide reward signal during training.

The quality ceiling of PPO is the quality ceiling of the reward model: every blind spot becomes a reward hacking target.

Not just a training artifact: a standalone product. You can use an RM at inference time to score and filter outputs, rerank candidates, or detect policy violations without retraining.

Safe RLHF trains TWO separate RMs (helpfulness and harmlessness) to avoid conflating the two axes into one ambiguous scalar.