ML//training//on-policy vs off-policy

On-policy and off-policy are the two regimes of training a model on scored or preferred outputs, distinguished by where those outputs come from: in on-policy training the model generates data during training and then learns from its own outputs, while in off-policy training it learns from a fixed dataset of pre-collected examples. PPO is on-policy, scoring outputs that the current policy produced; DPO is off-policy, training on preference pairs (prompt, chosen, rejected) generated beforehand.

On-policy and off-policy are the two regimes of training a model on scored or preferred outputs, distinguished by where those outputs come from: in on-policy training the model generates data during training and then learns from its own outputs, while in off-policy training it learns from a fixed dataset of pre-collected examples. PPO is on-policy, scoring outputs that the current policy produced; DPO is off-policy, training on preference pairs (prompt, chosen, rejected) generated beforehand.

The data format itself is algorithm-agnostic: the same (prompt, chosen, rejected) pairs can feed both RLHF with PPO and DPO. The difference is when and how the data is generated and consumed, a distinction that cuts across every training method.

On-policy training always learns from data of the model's current distribution, with no stale examples. PPO pays for it: generating outputs mid-training is expensive, so it is slower in practice despite its theoretically faster convergence.

Off-policy training needs no generation during training, so each step is much faster and the data can be reused across runs.

Off-policy training has its weakness too: DPO shows a 3 to 7% performance drop out of domain compared with RLHF, because a model trained on fixed data cannot explore regions the dataset did not cover.