ML//emergent behavior

Emergent behaviors are capabilities that appear suddenly at scale, such as arithmetic, chain-of-thought reasoning and translation to unseen languages. They are absent in small models and switch on past a compute threshold.

Emergent behaviors are capabilities that appear suddenly at scale, such as arithmetic, chain-of-thought reasoning and translation to unseen languages. They are absent in small models and switch on past a compute threshold.

They are controversial. Wei et al. (2022) catalogued them, and Schaeffer et al. (2023) argued that they are a mirage of the choice of metric. Still, the subjective experience of watching a model suddenly get something is hard to dismiss.