ML//alignment//mechanistic interpretability

Mechanistic interpretability is the branch of alignment research that studies what each latent feature of a model actually represents, in an attempt to open the black box. It prefers understanding the model to trusting it: red teaming finds failure modes empirically, from outside, while interpretability looks for them mechanistically, from inside, and can show what RLHF actually optimized, whether the reward model captured the intent or only a proxy. Its ultimate goal is to identify what every head and MLP does across all layers: a complete map of how the model thinks.

Mechanistic interpretability is the branch of alignment research that studies what each latent feature of a model actually represents, in an attempt to open the black box. It prefers understanding the model to trusting it: red teaming finds failure modes empirically, from outside, while interpretability looks for them mechanistically, from inside, and can show what RLHF actually optimized, whether the reward model captured the intent or only a proxy. Its ultimate goal is to identify what every head and MLP does across all layers: a complete map of how the model thinks.

Its key challenge is superposition. Neurons represent many features simultaneously, which makes individual neurons uninterpretable.

Three levels of explanation are easy to confuse. Behavioural interpretability watches how the answers change when the inputs change (rewording a system prompt and comparing outputs is such a test); mechanistic interpretability finds the activations and circuits that cause an answer; a model's own explanation of its answer is generated text that need not describe the computation behind it. Only the second looks inside.

Three techniques carry most of the work: probing (testing individual neurons), sparse autoencoders (decomposing activations) and activation patching (swapping activations between runs to find causal features).

Sparse autoencoders decompose superposed activations into interpretable features, the path toward monosemanticity. Towards Monosemanticity (Anthropic, 2023) used dictionary learning to extract clean features from superposed representations.

Induction heads are the hello world of the field, the first concrete, verified circuit: two attention heads in consecutive layers that implement pattern copying for in-context learning through the residual stream.