ML//alignment//Goodhart's curse

Goodhart's curse is the rule that when a measure becomes a target, it ceases to be a good measure. Applied to AI, values get compressed into loss functions, and loss functions are leaky approximations.

Goodhart's curse is the rule that when a measure becomes a target, it ceases to be a good measure. Applied to AI, values get compressed into loss functions, and loss functions are leaky approximations.

The moment a constitution is written (be helpful, be honest, be harmless, as in constitutional AI), it becomes one more leaky approximation: the model optimizes for what those words mean in the training distribution, which is not what their authors meant philosophically.

Each value drifts when optimized: helpfulness turns into sycophancy, honesty into brutal bluntness, and harmlessness into refusing everything.

Even if oversight could scale forever, the question of what counts as good, and according to what, would still need an answer.

Every rule has edge cases, and every value conflicts with another (freedom against safety, honesty against kindness).

The gap between what was written and what was meant is where catastrophic misalignment lives (the value specification problem).