ML//SSM//Mamba

Mamba is a selective state space model (Gu and Dao, 2023) whose key innovation is to make state transitions input-dependent (selective) instead of fixed. It is the strongest alternative to transformer attention, linear in the sequence length, \(O(n)\) against \(O(n^2)\), and it matches transformer quality on language benchmarks at smaller scale.

Mamba is a selective state space model (Gu and Dao, 2023) whose key innovation is to make state transitions input-dependent (selective) instead of fixed. It is the strongest alternative to transformer attention, linear in the sequence length, O(n)O(n)O(n) against O(n2)O(n^2)O(n2), and it matches transformer quality on language benchmarks at smaller scale.

What selection means is concrete. In an S4 layer the matrices Bˉ\bar BBˉ, CCC and the discretization step are the same at every position, so every input is written into the state the same way. Mamba computes them from the current input, so the model decides from what it sees how much of each new element to store and how fast to forget the old ones, the way a technician skims a log and stops at the alarm lines. The layer is then no longer time-invariant, which costs it the convolution that let S4 train in one parallel pass (SSM); it trains with a parallel scan over the recurrence instead, and still runs at constant cost per step.

It has not dethroned transformers at frontier scale. Hybrid architectures that mix Mamba layers with attention layers seem to be the pragmatic path, and whether Mamba is an interesting dead end or still in its early innings remains open. For an engineer it is a research tool today, with long sensor sequences on small hardware as its natural industrial promise.