ML//RNN//backpropagation through time

Backpropagation through time is backpropagation applied to an RNN by unrolling the loop across timesteps and treating it as one deep feedforward net whose layers all share the same weights; it is how every recurrent network, from a soft sensor to a speech model, is trained.

Backpropagation through time is backpropagation applied to an RNN by unrolling the loop across timesteps and treating it as one deep feedforward net whose layers all share the same weights; it is how every recurrent network, from a soft sensor to a speech model, is trained.

Unrolling makes the depth equal to the sequence length. A window of 10 seconds from a sensor sampled at 1 kHz is a network 10,000 layers deep, and every one of those steps contributes a Jacobian ∂ht/∂ht−1=diag⁡(ϕ′(zt)) W\partial h_t/\partial h_{t-1}=\operatorname{diag}(\phi'(z_t)),W∂ht​/∂ht−1​=diag(ϕ′(zt​))W to the chain. The gradient that reaches the start of the window is the product of all of them.

The error at the end of a sequence flows backward through every timestep, multiplying by the recurrent matrix once per step. Those repeated multiplications are exactly what cause the vanishing gradient (and exploding gradient) problem: the gradient shrinks or grows according to the eigenvalues of WWW, as the free response of xk+1=Wxkx_{k+1}=Wx_kxk+1​=Wxk​ decays or diverges (discrete-time stability).

The memory cost grows with the window, since every hidden state of the sequence is kept for the backward pass.

Truncated BPTT caps how many steps back the gradient travels, trading memory of the distant past for stable training and bounded memory. The window should still cover the slowest time constant the model has to learn, or the dependence it needs is cut off before the gradient can see it.