ML//alignment//mechanistic interpretability//activation patching
Activation patching is an intervention technique of mechanistic interpretability that runs a model on a clean input and on a corrupted one and copies an activation from one run into the other at a single chosen component, and it is used to find which components carry the information behind a specific behaviour. The component can be the output of one attention head, the residual stream at one layer and token position, or the output of one MLP layer. Because the experimenter sets the activation instead of observing it, the result is causal evidence rather than correlation (causal inference).
Activation patching is an intervention technique of mechanistic interpretability that runs a model on a clean input and on a corrupted one and copies an activation from one run into the other at a single chosen component, and it is used to find which components carry the information behind a specific behaviour. The component can be the output of one attention head, the residual stream at one layer and token position, or the output of one MLP layer. Because the experimenter sets the activation instead of observing it, the result is causal evidence rather than correlation (causal inference).
The two inputs differ only in the fact under study. The Eiffel Tower is located in the city of makes the model answer Paris; the corrupted run swaps the subject for another landmark, or adds noise to its embeddings, so Paris is no longer favoured. A single number scores the behaviour, usually the logit difference between the right answer and a wrong one. Say it is 4.0 on the clean run and 0.0 on the corrupted run. If the corrupted run with the clean output of one head patched in reaches 3.0, that head restores 75% of the behaviour. Sweeping the patch over layers and positions maps where the information lives.
restored=mpatched−mcorruptedmclean−mcorrupted\text{restored} = \frac{m_{\text{patched}} - m_{\text{corrupted}}}{m_{\text{clean}} - m_{\text{corrupted}}}restored=mclean−mcorruptedmpatched−mcorrupted
Here mmm is the metric on each run: 1 means the patch restores the full clean behaviour, 0 that it changes nothing.
The direction of the patch decides the question. Clean into corrupted asks whether the component is enough to restore the behaviour, a test of sufficiency. Corrupted into clean asks whether breaking the component destroys it, a test of necessity, and is a form of ablation whose replacement value comes from the corrupted input.
The result belongs to the pair of inputs and the metric. Swapping the subject, swapping names or adding noise each localise a different piece of information, so a high score says what the component carries for that contrast, not what it does in general.
A high score does not mean the component computes the behaviour alone. Patching the residual stream at one point carries everything computed upstream of it at that position; finer variants patch only along chosen paths between components, which is how a circuit is traced.
The name varies between papers. The ROME work on factual recall (Meng and colleagues, 2022) calls the procedure causal tracing, and work on causal abstraction calls a single patch an interchange intervention. Activation steering also edits activations, but it adds a direction to change behaviour instead of copying one run into another to explain it.