ML//AI progress//compute bottleneck

A compute bottleneck is a limit on the speed of AI research set by the compute available for experiments rather than by the number or quality of the researchers, and it is the main argument against a purely software-driven intelligence explosion. Automated researchers run on the same chips their experiments need, so adding researchers takes compute away from testing their ideas.

A compute bottleneck is a limit on the speed of AI research set by the compute available for experiments rather than by the number or quality of the researchers, and it is the main argument against a purely software-driven intelligence explosion. Automated researchers run on the same chips their experiments need, so adding researchers takes compute away from testing their ideas.

The argument is usually written with a CES function of research labour and experiment compute. If the elasticity of substitution is below 1, as in manufacturing, output has a ceiling with compute fixed, and the speed of software progress is capped at a few times today's, between about 2 and 100 depending on the value assumed.

The counter-argument is empirical. If frontier-scale experiments were the binding input, algorithmic progress should have slowed as training runs grew to take a large share of the world's compute, and it has not, which narrows the bottleneck to experiments near the frontier.

The bottleneck is weakest where a small, cheap experiment predicts what a large one would show, and strongest where behaviour appears only at scale.