ML//AI progress//algorithmic progress

Algorithmic progress is the reduction in the compute needed to reach a given level of performance thanks to better methods, architectures and data, and it is measured as a compute-equivalent multiplier per year. For language models, the most cited estimate (Ho and colleagues, 2024) finds the compute needed for a fixed performance halving about every eight months, roughly a threefold gain a year.

Algorithmic progress is the reduction in the compute needed to reach a given level of performance thanks to better methods, architectures and data, and it is measured as a compute-equivalent multiplier per year. For language models, the most cited estimate (Ho and colleagues, 2024) finds the compute needed for a fixed performance halving about every eight months, roughly a threefold gain a year.

Estimates disagree far more than that single number suggests. A 2026 review by Epoch AI puts its best guess near tenfold a year, with an interval from 2 to 50, and other studies range from 3 to about 20.

The gain depends on the scale at which it is measured. Ablations of a decade of innovations explain only a small part of the total gain at small scale, and one change, the switch to the transformer, accounts for most of what they can explain, with a value that grows with model size.

That concentration in single changes is why algorithmic progress is the least forecastable input: its history is mostly a few discontinuities (black swans) plus a slope.

It is the input that recursive self-improvement would speed up first, since it is made of ideas rather than of matter.