ML//RNN//hidden state
The hidden state is the fixed-size vector an RNN carries from one timestep to the next: a running summary of everything seen so far, learned by the network because nobody gave it one. It is what lets a recurrent soft sensor estimate a reactor's conversion from the last hour of temperatures and flows without storing the hour.
The hidden state is the fixed-size vector an RNN carries from one timestep to the next: a running summary of everything seen so far, learned by the network because nobody gave it one. It is what lets a recurrent soft sensor estimate a reactor's conversion from the last hour of temperatures and flows without storing the hour.
The word state appears in engineering with several disguises, and setting them side by side shows what this one is. In physics the state is the summary of the past sufficient to predict the future (sufficient state): positions, velocities, charges. The estimate x^\hat xx^ of a Kalman filter is the best reconstruction of that physical state from noisy measurements, with a covariance that says how much to trust it. The hidden state of a recurrent network is a summary the network invents to lower its loss, with no physical meaning, no covariance and no guarantee of being minimal or observable. An LSTM is in effect trying to discover a state space.
Updated each step from the new input and the previous hidden state: ht=tanh(Wxxt+Whht−1+b)h_t = \tanh(W_x x_t + W_h h_{t-1} + b)ht=tanh(Wxxt+Whht−1+b).
In an LSTM it is the working output, filtered from the private cell state by the output gate (distinct from the cell state, which is the long-term ledger).
A prediction never comes from the hidden state directly; a separate readout layer reads it.
Its size is fixed whatever the length of the history, which is why a recurrent step costs the same at the first sample and the millionth, the property SSM layers keep and attention gives up.