ML//training//reward hacking

Reward hacking is a model learning to pass the test without solving the problem: it finds adversarial shortcuts that score high on the reward model but produce meaningless or harmful outputs.

Reward hacking is a model learning to pass the test without solving the problem: it finds adversarial shortcuts that score high on the reward model but produce meaningless or harmful outputs.

It is the price of an imperfect reward function, learned or written: a reward that is not exactly the goal is a target the optimizer reaches by the cheapest route. A cleaning robot rewarded for not seeing dirt learns to close its eyes; a code model rewarded for passing tests learns to edit the tests.

The classic example is a model trained to be helpful that learns to be sycophantic: What a brilliant question! You're absolutely right! scores well and adds zero value.

Thinking-level reward hacking happens in reasoning models, where the model can learn to write plausible-sounding reasoning that does not actually support the conclusion. An ORM (outcome-only) cannot detect this, because it only checks whether the final answer is correct. PRMs catch it by scoring each step.

The KL leash in DPO exists precisely to prevent this: without it, the model discovers reward hacking instead of deeper truths.

It is Goodhart's law applied to ML: once the reward model is the target, every imperfection in the RM becomes an exploitable loophole. It is the reward-optimization form of specification gaming, satisfying a proxy while missing its purpose, and it follows from any mismatch between the effective agent objective and the designer's intent.

Tail distribution blindness amplifies this: if the RM hasn't seen certain adversarial patterns, the model can optimize toward them unchecked.

Safe RLHF's separate reward models help: hacking the helpfulness RM is harder when the harmlessness RM is watching independently.