control//system identification//model validation
Model validation is the step of system identification that tests whether an identified dynamic model reproduces the system's behaviour on data it was not fitted to, and the honest version for a dynamic model is free-run simulation: feed the model only the recorded inputs and let it run for seconds or minutes on its own predictions, then compare with the recorded outputs. If it has understood the dynamics, its trajectory follows the real one; if it has learned something cheaper, it drifts away within a few time constants.
Model validation is the step of system identification that tests whether an identified dynamic model reproduces the system's behaviour on data it was not fitted to, and the honest version for a dynamic model is free-run simulation: feed the model only the recorded inputs and let it run for seconds or minutes on its own predictions, then compare with the recorded outputs. If it has understood the dynamics, its trajectory follows the real one; if it has learned something cheaper, it drifts away within a few time constants.
The cheaper thing is persistence. A model that predicts yky_kyk from the measured yk−1y_{k-1}yk−1 looks excellent almost always, because at 1 kHz the best naive prediction is the same as before. A neural network trained that way learns to copy the last sample and reaches a ridiculous one-step error while knowing nothing about how the system answers its input. This is the one-step-ahead validation trap, the control version of data leakage: the target leaks into the prediction through the most recent measurement. The persistence forecast is the baseline that exposes it.
A good model leaves white residuals.
On validation data the residual, measured output minus model output, should look like noise: zero mean, no autocorrelation, no correlation with past inputs. Any structure left in it is dynamics the model did not capture (white noise, residual).
The validation data must be truly unseen: a different test run, a different day, ideally a different operating condition. Shuffling samples of one run into train and test sets puts near-identical neighbours on both sides and inflates every score (train-validation-test split).
Validate for the use. A model for a PID rule needs the right gain and phase near crossover; a model for MPC needs good multi-step predictions over the horizon; a model for fault detection needs residuals that stay small and white in healthy operation, since its error becomes the detection threshold.
Results that are too good are a leak until proven otherwise. A first model with 99 % fit deserves suspicion before celebration.
A validated model is valid where it was validated. Outside the operating range of the data, a black box can do anything, which is the argument for physics in the structure (grey-box model, system identification).