control//state estimation//Kalman filter//Kalman filter limits

2026-10-01

A Kalman filter in the field fails in recognisable ways, and each way traces back to one assumption the real world did not honour. Knowing the list turns a misbehaving filter from a mystery into a diagnosis: the symptom in the plots of the estimate, its band and its innovation points to the broken assumption, and each broken assumption has a standard remedy. The Kalman filter itself states the core assumptions; this is the catalogue of what happens when they fail, together with the practical ones that no textbook derivation mentions. The model in violet, what the sensor brings in copper, the filter's uncertainty in green.

Assumption Symptom Remedy

Linear model (rotations, angles, ranges to beacons) Biased estimates, divergence EKF, UKF

Gaussian noise (multipath, faults, quantization) Jumps after an outlier Gating, robust update, particles

White noise (drift, slow biases) Absurdly small P\ca{P}P, a deaf filter Augment the state, decimate (colored noise)

Q\cd{Q}Q and R\cc{R}R known A nervous or a lazy filter Tuning, adaptive R and Q, the NIS

No bias (miscalibrated sensors) Converges confidently to the wrong place Calibration, bias in the state (sensor bias)

The model is right (a car brakes, a drone takes a gust) Lag in every manoeuvre Larger Q\cd{Q}Q, IMM

P\ca{P}P ignores the data An absurd reading shrinks P\ca{P}P all the same Gating, watch the NIS

Exact arithmetic (float32 on a microcontroller) P\ca{P}P loses symmetry or turns negative Joseph form, square-root filters (estimate covariance)

Observability One direction of P\ca{P}P never shrinks More sensors or references (observability)

Readings on time (latency, separate clocks) Errors in fast manoeuvres Timestamps, out-of-sequence fusion

Known correlations between sources Overconfidence when fusing Covariance intersection (correlated measurements)

Affordable cost, O(n3)O(n^3)O(n3) in the state size Infeasible with millions of variables Ensemble Kalman filter

Most of these are failures of honesty rather than of accuracy: the filter is wrong and does not know it, its band too narrow for the error it carries. That is why the first check of any filter is whether its surprises are the size it predicted, and only then whether its estimate is good; and why every remedy that inflates something (Q\cd{Q}Q, R\cc{R}R, a floor on P\ca{P}P) is a trade of a little accuracy for honesty (process noise).

Time is an assumption too. A reading stamped when it arrived rather than when it was taken is a reading of the past fused as if it were the present; at speed, a 100 ms latency on a GPS is metres of error. Production filters buffer readings with their true timestamps and fuse each at its proper moment.

Cost decides what can be filtered at all. The covariance of a state with nnn variables has n2n^2n2 entries and its propagation costs about n3n^3n3 operations, which is nothing for a drone and impossible for the atmosphere, whose models carry millions of variables. That is where the ensemble Kalman filter comes in, one of the methods weather centres use: it propagates tens to a few hundred full forecasts and computes the covariance from their spread, never storing the full matrix; Canada's weather service has initialized its ensemble forecasts this way since 2005.

Offline, with the whole recording available, every estimate can also use the readings that came after it, through a backward pass over the stored run (RTS smoother).