ML//training//post-training//Safe RLHF

Safe RLHF is a variant of RLHF that decouples helpfulness and harmlessness into separate annotation streams, with human annotators scoring each axis independently, and it is used to train a model that is useful without becoming unsafe.

Safe RLHF is a variant of RLHF that decouples helpfulness and harmlessness into separate annotation streams, with human annotators scoring each axis independently, and it is used to train a model that is useful without becoming unsafe.

Standard RLHF asks a single question, whether a response is good, and that question becomes ambiguous when a helpful response is slightly unsafe or a safe response is useless. Each axis gets its own reward model, since one RM conflating the two turns unreliable on safety-critical outputs. During PPO, the two rewards are combined with a controllable tradeoff coefficient, which dials safety up or down, so the trade between safety and capability, a question of alignment, is explicit and tunable instead of being left to the model.

The Pareto frontier between helpfulness and harmlessness is not a fixed tradeoff: with better data and separate optimization, both can be pushed upward at the same time.