ML//evaluation//model robustness

Model robustness is the property of a machine learning model, and of the system around it, that keeps its performance when the inputs depart from the clean data it was evaluated on: typos and messy documents, ambiguous instructions, sensor noise, inputs from conditions never seen in training, and inputs crafted to make it fail. It is what decides whether a model that scored well in the lab still works in production, where a camera gets dirty, an operator writes in shorthand and someone tries to talk the chatbot out of its rules.

Model robustness is the property of a machine learning model, and of the system around it, that keeps its performance when the inputs depart from the clean data it was evaluated on: typos and messy documents, ambiguous instructions, sensor noise, inputs from conditions never seen in training, and inputs crafted to make it fail. It is what decides whether a model that scored well in the lab still works in production, where a camera gets dirty, an operator writes in shorthand and someone tries to talk the chatbot out of its rules.

Robustness is measured by deliberately disturbing the test set and watching the score fall. Each disturbance answers a different question: random noise and corruption (blur, dropped sensor values, misspellings) say how the model degrades gracefully; data from another site, season or machine say how it handles distribution shift; worst-case inputs searched on purpose (adversarial examples, jailbreaks, prompt injection) say how it behaves against an opponent. A model can be strong on one axis and fragile on another.

Robustness belongs to the system more than to the model.

Input validation, refusing what is out of range, a confidence check that hands doubtful cases to a person, permissions that limit what a manipulated model can do: these protect a deployment even when the model itself can be fooled.

In control, robust control means guaranteed stability and performance for every plant inside a stated uncertainty set, proved from a model. Robustness in ML is mostly empirical: tested on the disturbances someone thought of, with no guarantee for the rest.

The usual price is a little clean accuracy. Training with noise, data augmentation or adversarial examples buys robustness on the disturbances it saw, and tends to cost a point or two on clean data.

A drone's vision model that never saw rain, glare or motion blur is the industrial version of the problem; the remedies are collecting those conditions, simulating them (domain randomization), and a fallback when confidence drops.

A robust model is not automatically calibrated: it may keep its accuracy on shifted data while its confidence stays falsely high, which is worse than a model that knows it is lost.