ML//alignment//mechanistic interpretability//probe

2026-10-27

A probe is a small classifier trained on a model's internal activations to predict one property of the input, and it is used to test whether that property can be read out of a given layer. The model stays frozen; only the probe learns. Its input is the activation vector at one layer and position (usually the residual stream), its output a label such as singular or plural, part of speech, or whether a statement is true.

A language model reads The cat sleeps. A logistic regression trained on the layer-12 activation of cat across thousands of sentences labels the subject singular or plural; say it is right 97% of the time on sentences it never saw, where chance is 50%. The conclusion is that number is linearly decodable at that layer. Repeating the fit layer by layer maps where a property appears and where it fades, which is how probing studies place more syntax in middle layers and more semantics in later ones.

A probe measures presence, not use. High accuracy says the information is in the activations; it does not say the model reads it to produce its output.

Presence is not use. A model can compute a property on the way to something else and never consult it, and the probe reads it all the same. Use is tested by intervening: removing or editing the direction the probe found (ablation, activation patching) and checking whether the behaviour changes, as in subject-verb agreement failing once number is erased. Without that step the result is a correlation (causal inference).

A powerful probe can do the work itself. A deep probe with enough data can learn the task from features that only loosely encode it, so its accuracy measures the probe as much as the model. This is why probes are kept linear and compared against a baseline: the same probe on a randomly initialised model, or on a control task with random labels that only memorisation could fit (the gap between real and control accuracy is the probe's selectivity).

A probe only finds what it is asked about. It needs labels chosen in advance, so it confirms or rejects a hypothesis but cannot discover features nobody thought of; sparse autoencoders work the other way, without labels.