ML//training//post-training//Constitutional AI

Constitutional AI is a training method in which an expert writes a constitution (a set of rules) and the AI improves itself against it. The model \(X\) answers, reflects on its answer with the rules in hand, and answers again, better:

Constitutional AI is a training method in which an expert writes a constitution (a set of rules) and the AI improves itself against it. The model XXX answers, reflects on its answer with the rules in hand, and answers again, better:

1AI[X] produces BxB_xBx​2AI[X + rules] writes a reflection R on BxB_xBx​3AI[X + R] produces ByB_yBy​better than BxB_xBx​

It replaces the human annotators of RLHF with the AI itself (a form of RLAIF), trading human judgment for speed and scale.

The AI crafts ByB_yBy​ through reasoning, where iterative DPO only chooses between BxB_xBx​ and ByB_yBy​.

The pairs with By>BxB_y > B_xBy​>Bx​ feed DPO directly, which is far wiser than training an RM for PPO.

The reasoning output is sometimes reused for SFT, because it contains good structures and core ideas.

Its risk is Goodhart's curse: the gap between what the constitution says and what its authors meant is where catastrophic misalignment lives.