ML//RNN//LSTM

2018-04-12

An LSTM (Long Short-Term Memory) is a recurrent network whose memory cell is updated by addition and guarded by learned gates that decide what to forget, what to write and what to read, and it was for two decades the standard way to learn from long sequences: speech, text, sensor forecasting and process control. The forget, input and output gates are learned filters on the information flow, each a sigmoid between 0 (closed) and 1 (open).

The cell update is the whole trick. With ftf_tft the forget gate, iti_tit the input gate and c~t\tilde c_tc~t the candidate content computed from the new input, the cell becomes ct=ft⊙ct−1+it⊙c~tc_t=f_t\odot c_{t-1}+i_t\odot\tilde c_tct=ft⊙ct−1+it⊙c~t. A plain RNN multiplies its state by a matrix at every step, and the gradient multiplied hundreds of times vanishes; the LSTM adds to its cell, and while the forget gate stays near 1 the gradient crosses many steps nearly intact (vanishing gradient). The output gate then decides how much of the cell to show as the hidden state.

It solved the vanishing gradient for sequences and dominated NLP before the transformer (1997 to 2017). The GRU is its simplified sibling with two gates.

In industry it remains in niches: forecasting process series, soft sensors, learned dynamics models for control, keyword spotting on small devices. A table of lagged features with gradient boosting often ties with it on short single-series forecasts with less data.

Its hidden state is a learned summary of the past, so an LSTM is in effect trying to discover a state space for the process it watches, without any guarantee that the state it finds is physical or minimal (state-space model).