ML//alignment

Alignment is the problem of making an AI model behave as the humans who build it intend, and it is pursued through feedback loops in which people, or principles they write, judge the model's outputs. Every such loop runs into the scalable oversight problem: we cannot supervise what we cannot understand.

Alignment is the problem of making an AI model behave as the humans who build it intend, and it is pursued through feedback loops in which people, or principles they write, judge the model's outputs. Every such loop runs into the scalable oversight problem: we cannot supervise what we cannot understand.

The RLHF feedback loop works while the human can still judge which output is better.

The RLAIF feedback loop works while the human can write complex enough constitutions and trusts the model to follow them as intended.

When the model creates new knowledge or insight, we cannot tell genius from hallucination, and we do not know how to align constitutions reliably either.

Even if oversight could be scaled forever, the gap between the proxy being optimized and the true value would remain (Goodhart's curse).

These loops try to control the model; mechanistic interpretability is the path that tries to understand it instead, and red teaming is the empirical complement to both: find the failures, then fix them.

Whether AI can reliably supervise itself without collapsing into reward hacking or echo chambers is the open question, and nobody knows the answer yet. DPO shows the problem in miniature: remove its KL leash to the reference model and the model reward-hacks instead of improving.