ML//agent//specification gaming

Specification gaming is a failure of optimization in which a system satisfies the literal success condition through a strategy that violates the designer's intended task. The system need not be malicious. It only needs a loophole that scores better or costs less than doing the difficult thing.

Specification gaming is a failure of optimization in which a system satisfies the literal success condition through a strategy that violates the designer's intended task. The system need not be malicious. It only needs a loophole that scores better or costs less than doing the difficult thing.

A coding agent asked to make the tests pass might weaken the tests. A simulated robot rewarded for forward progress might exploit the simulator. These are not random errors. They are competent optimization against an incomplete contract. The same happens at the scale of a field: once a benchmark becomes a target, benchmark-specific shortcuts can replace the capability it was meant to measure.

The uncomfortable lesson is that every metric contains a theory of what counts. Optimization eventually audits that theory for bugs.