ML//neural network//adversarial example

An adversarial example is an input crafted by someone who has studied a model, changed by a small and often imperceptible amount so that the model gives a wrong output with confidence; it is the reason a neural network deployed where someone wants it to fail is an attack surface of its own. A defect classifier, a pedestrian detector or a drone's landing-pad recognizer can all be pushed to a chosen wrong answer by a pattern that a human inspector would not notice.

An adversarial example is an input crafted by someone who has studied a model, changed by a small and often imperceptible amount so that the model gives a wrong output with confidence; it is the reason a neural network deployed where someone wants it to fail is an attack surface of its own. A defect classifier, a pedestrian detector or a drone's landing-pad recognizer can all be pushed to a chosen wrong answer by a pattern that a human inspector would not notice.

The mechanism is the network's own geometry. Its output can have very large local slopes with respect to the input, so a perturbation aimed along the gradient of the loss, spread thinly over thousands of pixels or samples, moves the output far while each input value barely changes. Random noise of the same size does almost nothing, because it is not aimed. That is the difference between nature and an adversary: the wind does not know you exist, while an attacker observes your model and adapts, which is the setting of game theory (an adversarial example is the move of a player who has read your strategy).

It differs from an out-of-distribution input, which is natural (new lighting, fog, another machine) and hurts by accident, and from shortcut learning, a flaw of training that no one exploits on purpose. All three end the same way, a wrong answer at full confidence.

With an adversary in play it belongs to cyber-physical security, beside false data injection and GNSS spoofing. The defences are the same in spirit: cross-check the network against an independent sensor or a physical relation it cannot fake, limit what its output may command, and keep a simpler verified path ready (safety filter).

Training on crafted examples (adversarial training) makes a network harder to fool at some cost in clean accuracy and training time; no current method makes a large network immune.

The large local slopes that make it fragile are also a loop-gain problem: a network inside a control loop with steep input-output slopes acts as a high gain, the short road to instability (learning-based control).