ML//RNN
An RNN (recurrent neural network) is a neural network for sequence data that processes a sequence one step at a time, combining each new input with a summary of everything before it, its hidden state; it is used to forecast sensor series, as a virtual sensor that estimates a quantity no instrument measures, and as a learned model of a plant's dynamics for control.
An RNN (recurrent neural network) is a neural network for sequence data that processes a sequence one step at a time, combining each new input with a summary of everything before it, its hidden state; it is used to forecast sensor series, as a virtual sensor that estimates a quantity no instrument measures, and as a learned model of a plant's dynamics for control.
ht=ϕ(Wht−1+Uxt+b),y^t=Chth_t=\phi\left(Wh_{t-1}+Ux_t+b\right),\qquad \hat y_t=Ch_tht=ϕ(Wht−1+Uxt+b),y^t=Cht
Here hth_tht is the hidden state, xtx_txt the input at step ttt and WWW, UUU, CCC learned matrices. Set beside a state-space model, xk+1=f(xk,uk)x_{k+1}=f(x_k,u_k)xk+1=f(xk,uk), yk=h(xk)y_k=h(x_k)yk=h(xk), it is the same structure with fff learned from data. The difference is in what the state means: a physical state is a position, a velocity, a winding temperature, while the state of an RNN need not mean anything physical, nor be observable, nor minimal. It is whatever served the optimizer.
An architecture passing notes to itself.
It tried to remember that way, one step at a time, then forgot what it wrote three steps ago.
The hidden state carries the memory of previous steps, exactly what saving enriched embeddings between forward passes would give.
It is trained by backpropagation through time, which crosses one Jacobian per step, so information from the beginning of the sequence dilutes exponentially (the vanishing gradient problem); and the network cannot be parallelized, because each step depends on the previous one, so it is slow to train.
LSTM and GRU gates partially mitigated the vanishing gradient, but couldn't match transformers at scale. Transformers solved this by replacing recurrence with attention: every token sees every other token directly, no dilution chain.
In industry the recurrent family lives on in niches: predicting a motor's winding temperature ten minutes ahead from the current and ambient temperature of the last hour, or a soft sensor in a process plant. Try an autoregressive model or gradient boosting on lag features first; the recurrent network pays with many series, long dependencies and several inputs.
The linear, structured version of the same idea came back as the SSM layer, which trains in parallel and still runs step by step.