ML//training//post-training//PPO

PPO (**Proximal Policy Optimisation**) is a reinforcement learning method in which a policy learns through exploration, in small *proximal* steps.

PPO (Proximal Policy Optimisation) is a reinforcement learning method in which a policy learns through exploration, in small proximal steps.

In robotics the policy learns in a physical environment, such as NVIDIA Omniverse. In LLMs the environment becomes a reward model (RM) trained through RLHF, and PPO is on-policy: it generates outputs during training and scores them in real time with the RM, so it needs no contrastive pairs upfront.

Beyond everything DPO uses, PPO adds the reward model, a critic that reduces variance (actor-critic) by predicting scores, and the LLM actor with a reference model for the KL divergence.

The word proximal names the safety mechanism: each update is clipped so the new policy's probability for an action stays within a small ratio (typically 20 %) of the old one, which keeps one lucky batch from wrecking a policy that took days to train. It is the same reason a robot policy trained with PPO learns smoothly instead of collapsing.

Its reward signal for SGD comes from a sequence-level scalar rather than from token-level logit derivatives. That is a credit assignment problem: every token receives the same offset.

It presumes that the holistic properties of a response require sequence-level scoring. Papers argue that PPO captures properties of the whole response that DPO, mapping preferences straight to token-level gradients with no reward model, loses; but attention already encodes the dependencies between tokens before the logits reach the loss.

It is faster than DPO in theory (no humans clicking or robots reasoning beforehand), yet its on-policy behaviour (scoring during training) makes it slower in practice.

It counts as RLAIF when the reward model was trained from RLHF data.