Neural decoder · Grey Matter

A neural decoder is the model inside a brain-computer interface that takes recorded activity, usually spike counts from a few hundred electrodes in bins of a few tens of milliseconds, and returns what the user intends: a cursor velocity, a click, a phoneme or a word.

Neural decoder. A neural decoder is the model inside a brain-computer interface that takes recorded activity, usually spike counts from a few hundred electrodes in bins of a few tens of milliseconds, and returns what the user intends: a cursor velocity, a click, a phoneme or a word.

It is trained on neural activity recorded while the user watches a cursor move to cued targets or attempts cued sentences, and then it runs in real time. For cursors the classic choice is the Kalman filter, used in people with tetraplegia by 2008: it treats each neuron's firing as roughly linear in the intended velocity and blends that evidence with the previous estimate. In 2012 ReFIT halved the time monkeys took to reach targets by retraining the filter on its own closed-loop data, with each recorded velocity rotated to point at the target. Speech decoders are recurrent networks that output phoneme probabilities, followed by an n-gram model and a large language model that rescore the candidate sentences; in the speech neuroprosthesis of 2024 that stack reached 97.5 % word accuracy.

Offline fit predicts closed-loop control poorly. The user sees the output and corrects it, so decoders compared on recorded data rank differently once a person drives them.

Two learners share the loop. Adapting the decoder while the brain adapts to it produced skilled, stable control in monkeys and a set of neurons dedicated to the task.

Signals drift. Electrodes shift and lose neurons from day to day, so decoders are recalibrated, often at the start of each session, or kept adapting while they are in use.

Pretrained generalists are arriving. NDT3, a transformer trained on 2,000 hours of spiking from more than 30 monkeys and humans, improved decoding with little new data, though its authors found that the differences between implants remained a limit at any scale.

A decoder is half of a control loop.

The person adjusts to its errors as it runs, so it is judged by how well someone controls it, and a fit to recorded data says only part of that.

Questions: Is learning to control a brain-computer interface like learning a motor skill? In monkeys it behaves much like one. When the decoder was held fixed for days, control improved with practice until the recorded neurons settled into a stable pattern that was recalled quickly the next day and resisted interference, and a second pattern for a different decoder could be learned and kept alongside it. Letting the decoder adapt at the same time made learning faster and still left a stable network for the task. Human systems mostly recalibrate the decoder instead, so the user's own learning carries less of the load there, though it still plays a part. Why can the decoder that fits recorded data best lose once someone drives it? Because the user sees the cursor and corrects it. Offline, a decoder is scored on how well it reconstructs movements from activity that was recorded with no cursor to steer; in use, the person compensates for its errors as they happen. When nine decoders were compared in monkeys, the assumption that weighed most offline, about how the neurons' preferred directions are spread, produced directional biases the animals simply corrected for, and how the decoder smoothed the cursor became the largest difference. ReFIT drew the lesson from the other side: it was retrained on closed-loop data and halved the time to reach targets. Why does a brain-computer interface for speech need a language model? A few hundred electrodes see only part of what the speech cortex does, so the neural decoder's guesses at each sound are often wrong or uncertain. A language model knows which sequences of words are likely in English and picks, among the many sentences compatible with those noisy guesses, the one that makes sense, much as a listener understands a mumbled sentence. In the 2024 system this last stage helped take word accuracy to 97.5 %; the price is that rare names and unusual phrasing are harder to get through.