ML//training//post-training//reward model//PRM

A PRM, or Process Reward Model, is a reward model that scores each intermediate step of a chain of reasoning as well as its final answer, and it is used to catch reasoning that reaches a correct answer by an invalid path.

A PRM, or Process Reward Model, is a reward model that scores each intermediate step of a chain of reasoning as well as its final answer, and it is used to catch reasoning that reaches a correct answer by an invalid path.

Its counterpart is the ORM (Outcome Reward Model): the ORM asks whether the final answer is correct, the PRM asks whether each step is valid. The PRM is more expensive, because it needs step-level annotations, but it catches more failure modes; the ORM is cheaper and cannot tell a lucky guess from valid reasoning.

A PRM can detect reward hacking of the thinking itself: a model might write plausible-sounding reasoning that arrives at the correct answer by the wrong path, accidentally or adversarially.

It is probably part of o3, as the most sophisticated approach to evaluating reasoning chains, scoring per step instead of per outcome.

OpenAI likely uses PRMs for o1 and o3, while DeepSeek R1 used GRPO (group-relative scoring without a separate RM). DeepSeek challenged the PRM approach that way and got comparable results.