ML//alignment//mechanistic interpretability//ablation

Ablation, in mechanistic interpretability, is an intervention that disables one component of a trained model during a forward pass (a neuron, an attention head, an MLP layer) by overwriting its output, and it is used to test whether a behaviour depends on that component. Nothing is retrained: the model runs as usual except for the one replaced activation, and the behaviour is measured with and without it. It is often confused with an *ablation study*, where a part of an architecture or training recipe is removed and the model retrained to see what that part contributed; that measures a design choice, this measures a component of one fixed model.

Ablation, in mechanistic interpretability, is an intervention that disables one component of a trained model during a forward pass (a neuron, an attention head, an MLP layer) by overwriting its output, and it is used to test whether a behaviour depends on that component. Nothing is retrained: the model runs as usual except for the one replaced activation, and the behaviour is measured with and without it. It is often confused with an ablation study, where a part of an architecture or training recipe is removed and the model retrained to see what that part contributed; that measures a design choice, this measures a component of one fixed model.

The replacement value is the main decision. Zero ablation sets the output to zero. Mean ablation sets it to its average over a reference set of inputs, which removes what is specific to this input but keeps a typical magnitude. Resample ablation takes the component's output from a different, randomly chosen input. Zero is the simplest, but often a value the component never produces, so later layers see something off-distribution and part of the damage comes from that shock rather than from the missing information. Mean and resample ablation keep the input to later layers realistic, which is why they are usually preferred.

Ablation tests necessity. If ablating three heads drops the logit difference between the right and wrong answer to near zero, the behaviour needs those heads. It does not show that they are enough to produce it, which is the question activation patching answers when it patches a component into a corrupted run.

A small effect does not prove a component unimportant, because models contain redundancy. In the indirect object identification circuit of GPT-2 small (Wang and colleagues, 2022), ablating the heads that copy the right name makes other heads, backup name movers, take over part of their role, so the drop understates what the original heads did. Ablating several components together, or comparing with patching, exposes it.

Ablating a single neuron is a coarse cut because of superposition: the neuron takes part in many features, and removing it touches all of them.

A result is relative to a task and a metric. Finding a circuit means ablating many components over a dataset of prompts built for one behaviour and keeping the components whose removal breaks it.