ML//neural network

A neural network is a parametric function built from layers of weighted sums with an elementwise nonlinearity between them, fitted to data by gradient descent, and it is the model an engineer reaches for when the input is an image, a sound, a raw signal or text and the useful features cannot be written by hand. Each layer multiplies its input by a weight matrix, adds a bias and applies an activation function; at the bottom it is all matrix multiplications running on ALUs. A logistic regression, \(\sigma(w^{\mathsf T}x+b)\), is a single neuron; put several in parallel and you have a layer, chain layers and you have a multilayer perceptron, the plainest network of all.

A neural network is a parametric function built from layers of weighted sums with an elementwise nonlinearity between them, fitted to data by gradient descent, and it is the model an engineer reaches for when the input is an image, a sound, a raw signal or text and the useful features cannot be written by hand. Each layer multiplies its input by a weight matrix, adds a bias and applies an activation function; at the bottom it is all matrix multiplications running on ALUs. A logistic regression, σ(wTx+b)\sigma(w^{\mathsf T}x+b)σ(wTx+b), is a single neuron; put several in parallel and you have a layer, chain layers and you have a multilayer perceptron, the plainest network of all.

Depth is what changes the game. Each layer builds on the features of the one before (edges, then textures, then crack in a weld photo), so the network learns the features that classical ML needs an engineer to write (physics-based features). That is its advantage and its cost: it needs data to learn what you could have told it.

A network is a composition of weighted sums and nonlinearities.

Remove the hidden layers and logistic regression comes back. Convolution, recurrence and attention only decide which weights are shared and which inputs each unit may see.

Training is three separate pieces. The loss function says what counts as wrong, backpropagation computes its gradient for every weight at a few times the cost of one forward pass, and the optimizer (SGD, Adam) uses it, with a step set by the learning rate. The loss is non-convex (loss landscape), so two runs with different seeds give two different networks.

Depth has a numerical price. The gradient crosses a product of layer Jacobians, and when their singular values sit below or above 1 it vanishes or explodes. The remedies are ReLU, scale-preserving initialization (weights drawn so that the variance of the activations stays the same from layer to layer), normalization (layer norm, batch norm) and above all residual connections.

The architecture is an assumption about the world, its inductive bias. A CNN assumes a pattern means the same wherever it appears (inspection images, spectrograms), an RNN or an SSM that a state evolves in time (virtual sensors, forecasting), a GNN that relations are local (fleets, grids, pipe networks), and the attention of a transformer that any element may matter to any other (text). An autoencoder learns what normal data looks like, for anomaly detection.

A network has to earn its rent. On a table of process data, gradient boosting with forty well-chosen features usually wins at a fraction of the cost; the network pays when the data is image, sound, raw signal or text, or when a pretrained model already did the hard work (applied ML, flyswatter rule). The universal approximation theorem says it can fit anything, which is also true of a giant table.

It fails with confidence. An out-of-distribution input gets the usual certainty, and its softmax output is not a calibrated probability. A fixed-size network has nearly constant latency, which suits a microcontroller, but in a loop its outputs move the states it will see next (imitation learning), so in critical systems it sits behind a safety filter.