Encoding model · Lobeworks/17

An encoding model predicts the activity recorded from a brain, voxel by voxel or electrode by electrode, from a description of the stimulus the person is receiving, and it is the standard way neuroscience tests a theory of what a region represents: the better it predicts activity for stimuli it never saw, the better th

Encoding model. An encoding model predicts the activity recorded from a brain, voxel by voxel or electrode by electrode, from a description of the stimulus the person is receiving, and it is the standard way neuroscience tests a theory of what a region represents: the better it predicts activity for stimuli it never saw, the better the theory.

Most modern encoding models take the description from a trained network. A word in its context, an image or a sound goes into a language, vision or audio model, an internal layer gives a vector of features, and a linear map fitted for each person turns those features into predicted activity. The score is the correlation between predicted and recorded activity on held-out data, often divided by how much the noise of the recording leaves predictable at all. Run backwards, the same model becomes a decoder: search for the stimulus whose predicted activity best matches what was recorded.

A high score says the features carry what the region responds to, in a form a linear map can reach. It says nothing about whether the region computes those features the way the network does, and networks with random weights already score fairly well, so a score needs control models beside it to mean anything.

The linear map is the personal part and the feature network the shared part. That split is why decoders built this way can be fitted to a new person from hours of recordings: the expensive knowledge is learned once, outside any head.

Predicting well is weaker than sharing a geometry. The map can stretch some directions and shrink others, so whether brain and model place the same things near each other is a separate claim, tested separately.

Questions: How does a model that predicts brain activity become one that reads it? An encoding model maps a stimulus to the activity it should produce. Decoding runs it backwards: a generator proposes candidate stimuli (words from a language model, images from an image model), the encoding model predicts the activity each one would cause, and the candidate whose prediction best matches the recording wins. The 2023 fMRI decoder of continuous language worked this way, so its output can only be as faithful as the encoding model is accurate, and anything the recording does not distinguish is filled in by the generator.