ML//scaling laws

Scaling laws are power laws that relate a model's performance to its compute, data and parameters (Kaplan et al., 2020), and labs use them to forecast capability from the training budget before the experiment runs.

Scaling laws are power laws that relate a model's performance to its compute, data and parameters (Kaplan et al., 2020), and labs use them to forecast capability from the training budget before the experiment runs.

The predictability changed how labs plan, from architecting smarter models to scaling bigger ones, and Chinchilla then refined the recipe.

What the laws relate is the pre-training loss, the error of next-token prediction, and three inputs: the number of parameters, the number of training tokens and the compute spent, which for a dense transformer is roughly six times parameters times tokens in floating-point operations. Loss falls as a power law in each, so every tenfold increase buys a roughly constant improvement, and the curves held over many orders of magnitude.

Parameters alone do not pay. For a fixed compute budget there is a best split between model size and data; Kaplan's first fits favoured big models on modest data, and Chinchilla showed that most models of the time were undertrained, needing about 20 tokens per parameter.

Reinforcement learning is a different axis. The reasoning gains of 2024 and 2025 came from post-training with verifiable rewards and from spending more compute at inference, which the original laws do not describe; it is not interchangeable with parameters or FLOPs, and its own scaling is still being measured (reasoning model).

A smooth loss curve does not mean smooth capabilities. Particular skills can appear abruptly as loss falls, and loss says nothing about cost per token, which is set by architecture and serving.

The laws describe a fixed method; better methods shift the whole curve down (algorithmic progress), which is why a cheaper model can match a bigger one from a year before.