ML//SSM

State space models (SSMs), in machine learning, are sequence layers built from a discretized linear state-space model whose matrices are learned, and they are an alternative to transformers that process sequences through a fixed-size hidden state, at \(O(n)\) cost instead of \(O(n^2)\). They come straight from control theory: continuous-time dynamics, discretized for sequences.

State space models (SSMs), in machine learning, are sequence layers built from a discretized linear state-space model whose matrices are learned, and they are an alternative to transformers that process sequences through a fixed-size hidden state, at O(n)O(n)O(n) cost instead of O(n2)O(n^2)O(n2). They come straight from control theory: continuous-time dynamics, discretized for sequences.

sk+1=Aˉ sk+Bˉ uk,yk=C sk+D uks_{k+1}=\bar A\,s_k+\bar B\,u_k,\qquad y_k=C\,s_k+D\,u_ksk+1​=Aˉsk​+Bˉuk​,yk​=Csk​+Duk​

Each layer is the equation every control course writes, with sss the internal state, uuu the input sequence and yyy the output; nonlinearities sit between the layers, so the network as a whole is nonlinear.

A linear time-invariant layer can be evaluated two ways.

Its output is the input convolved with its impulse response (CBˉ, CAˉBˉ, CAˉ2Bˉ,… )(C\bar B,,C\bar A\bar B,,C\bar A^2\bar B,\dots)(CBˉ,CAˉBˉ,CAˉ2Bˉ,…), the Markov parameters of system identification, so it trains in parallel as one long convolution; and it runs as a recurrence with constant cost per step. RNNs have only the second property and transformers only the first.

The two members show the trade. S4 keeps the layer linear and time-invariant, with a structured Aˉ\bar AAˉ built for long memory, and so keeps the convolution. Mamba makes the matrices depend on the input, which lets it choose what to remember and costs it the convolution.

Against a plain RNN, the recurrence is linear in the state, which keeps gradients through long sequences under control by the eigenvalues of Aˉ\bar AAˉ instead of leaving them to a learned nonlinear loop (vanishing gradient).

Their promise is long-range dependencies without the quadratic cost of attention, attractive for long sensor records on edge hardware. In production they are still mostly research; for industrial series, persistence, ARIMA and boosting come first (time series).