ML//scaling laws//Chinchilla

Chinchilla is the DeepMind model and paper (Hoffmann et al., 2022) that showed most LLMs were undertrained relative to their size: for a fixed compute budget, it pays to use more data with fewer parameters than previously thought. Chinchilla 70B matched Gopher 280B by training on 4× more tokens.

Chinchilla is the DeepMind model and paper (Hoffmann et al., 2022) that showed most LLMs were undertrained relative to their size: for a fixed compute budget, it pays to use more data with fewer parameters than previously thought. Chinchilla 70B matched Gopher 280B by training on 4× more tokens.

The result shifted the industry from ever bigger models to more data and right-sized models.