ML//transformer//attention//induction head

An induction head is an attention head that copies what followed an earlier occurrence of the current token: if \([A][B]\) appears in the context and later \([A]\) appears again, the model predicts \([B]\) with high probability. Given *The cat sleeps. The dog barks. The cat ____*, it predicts *sleeps*. That copying is part of why few-shot prompting, RAG and chain of thought work: the heads read the examples or the retrieved passages and propagate their patterns to new inputs. It is one of the most important discoveries of mechanistic interpretability: concrete, verifiable and functionally understood.

An induction head is an attention head that copies what followed an earlier occurrence of the current token: if [A][B][A][B][A][B] appears in the context and later [A][A][A] appears again, the model predicts [B][B][B] with high probability. Given The cat sleeps. The dog barks. The cat ____, it predicts sleeps. That copying is part of why few-shot prompting, RAG and chain of thought work: the heads read the examples or the retrieved passages and propagate their patterns to new inputs. It is one of the most important discoveries of mechanistic interpretability: concrete, verifiable and functionally understood.

The circuit is two attention heads in consecutive layers working together. The first, the prefix matching head, lets the current token look back and find tokens similar to itself in the past; the second, the induction head proper, copies what came after that previous occurrence.

There is no special architecture. They are regular heads with regular query, key and value matrices, initialized randomly; the gradient pushed them into this specialization because it reduced the loss.

Large models have multiple induction circuits, some more specialized, some for different pattern types. They are a subset of the 96 heads per layer, not all of them.

During training there is a phase transition: when induction heads emerge, the loss drops abruptly, and the model suddenly gets much better at following context patterns.

They are the first worked example of understanding transformers from the inside: proof that attention heads form interpretable circuits rather than opaque weights.