mathematics//numerical methods//numerical stability
Numerical stability is the property of an algorithm that keeps its own errors bounded as it runs, so that a decaying system stays decaying in the computation and small roundings do not grow into large ones, and it is the first thing to check when a simulation, an estimator or a training run misbehaves. It belongs to the method and the step, and it is separate from the stability of the system being computed: a perfectly stable plant can be simulated into an explosion.
Numerical stability is the property of an algorithm that keeps its own errors bounded as it runs, so that a decaying system stays decaying in the computation and small roundings do not grow into large ones, and it is the first thing to check when a simulation, an estimator or a training run misbehaves. It belongs to the method and the step, and it is separate from the stability of the system being computed: a perfectly stable plant can be simulated into an explosion.
The test case is the simplest decaying system, x˙=λx\dot x=\lambda xx˙=λx with λ<0\lambda<0λ<0. Explicit Euler turns it into xk+1=(1+λΔt) xkx_{k+1}=(1+\lambda\Delta t),x_kxk+1=(1+λΔt)xk, and the computed solution grows whenever
∣1+λ Δt∣>1,|1+\lambda\,\Delta t|>1,∣1+λΔt∣>1,
while the true one decays. For a real λ\lambdaλ that means a step above 2/∣λ∣2/|\lambda|2/∣λ∣. A thermal sensor with a 20 ms time constant (λ=−50\lambda=-50λ=−50 per second) integrated with a 50 ms step oscillates with growing amplitude in the simulation. Every method has its own stability region, the set of values of λΔt\lambda\Delta tλΔt for which it stays bounded; explicit methods have a finite one, and some implicit methods contain the whole left half of the complex plane (A-stability), so they stay bounded for any step on any decaying mode.
Discretization error disguises itself as physics.
A diverging simulation may be the integrator and not the system, and a simulated oscillator that gains energy looks exactly like an unstable controller. Before redesigning the controller, compare the step with the fastest time constant in the model and run again with a step ten times smaller; if the behaviour changes, it was numerical.
Stability matters as much as accuracy. Accuracy says how close each step is; stability says whether the errors of all the steps add up or multiply. A method with a small error per step but outside its stability region produces a smooth, convincing and completely wrong result (numerical integration).
The same multiplication governs other iterations. Gradient descent on a quadratic multiplies each direction by 1−ηλi1-\eta\lambda_i1−ηλi and diverges above η=2/λmax\eta=2/\lambda_{\max}η=2/λmax; a consensus update among drones diverges above an analogous bound on its gain; a long product of layer Jacobians in a deep network vanishes or explodes. Each is a discrete system whose factor left the unit circle.
In linear algebra the word means something related: an algorithm is stable when its rounding errors are not amplified beyond what the problem's own condition number forces. Forming an explicit inverse or the normal equations is less stable than factorizing, which is why solve beats inv (linear system of equations).
When a fast mode forces the step down far below what the slow dynamics need, the problem has a name and a remedy: stiffness and implicit methods.