ML//model//GPT//GPT-3//in-context learning

In-context learning is a model adapting its behavior to what is in its context window, without any weight updates. Examples given in the prompt (few-shot learning) are one way to trigger it, and it also covers implicit pattern matching with zero examples.

In-context learning is a model adapting its behavior to what is in its context window, without any weight updates. Examples given in the prompt (few-shot learning) are one way to trigger it, and it also covers implicit pattern matching with zero examples.

Circuits of attention heads find previous occurrences of a pattern and copy what followed: given The cat sleeps... The cat ____, the model predicts sleeps.

These circuits are the induction heads, its mechanistic substrate. During training, when they emerge there is a phase transition: the loss drops abruptly as the model suddenly learns to follow context patterns.

It is still not fully understood beyond induction heads. It might be gradient descent running inside the forward pass, or more complex pattern matching; most likely there are multiple mechanisms.

The same mechanisms explain why RAG, chain of thought and prompt engineering work: circuits are reading the context and propagating patterns to new tokens.