ML//neural network//universal approximation theorem

The universal approximation theorem is the result of neural network theory stating that a neural network with a single hidden layer, wide enough and with a nonlinear activation function, can approximate any continuous function on a bounded region to arbitrary precision (Cybenko 1989, Hornik 1991). It is the reason nobody doubts that a network can represent the map from a drone's state to the right thrust, or from a vibration window to a fault class.

The universal approximation theorem is the result of neural network theory stating that a neural network with a single hidden layer, wide enough and with a nonlinear activation function, can approximate any continuous function on a bounded region to arbitrary precision (Cybenko 1989, Hornik 1991). It is the reason nobody doubts that a network can represent the map from a drone's state to the right thrust, or from a vibration window to a fault class.

It says much less than it seems. It is an existence proof: a good weight setting exists, but the theorem says nothing of how many neurons are enough, how to find the weights, or what happens outside the region. A giant lookup table is also a universal approximator; what separates a useful model from a table is its behaviour on data it has not seen.

Fitting is cheap; generalizing is the question.

A model that can fit anything has told you nothing about the point it has not seen, which is where an engineer needs it.

Outside the data the network extrapolates by its construction, with full confidence. A ReLU network is piecewise linear and continues its last slope in a straight line; a model of a drone trained only on calm days has never seen wind and may answer anything. A grey-box model degrades gently there because its physics still holds (grey-box model, out-of-distribution, model versus data).

Depth, in practice, is what makes the search tractable: deep networks reach the same functions with far fewer parameters than one wide layer.

It explains why nets are expressive, but neither why training via backpropagation finds good solutions nor why they generalize; that is the subject of generalization.