ML//neural network//activation function

An activation function is the elementwise nonlinearity applied to the weighted sum of each layer of a neural network, and it is what lets a network represent curved relations at all. Without it, stacking linear layers collapses into a single linear transform: \(W_2(W_1x)\) is just \((W_2W_1)x\), so a hundred layers equal one and cannot learn a curve. The output vector of a layer after the nonlinearity is its **activation**, the quantity backpropagation stores on the way forward.

An activation function is the elementwise nonlinearity applied to the weighted sum of each layer of a neural network, and it is what lets a network represent curved relations at all. Without it, stacking linear layers collapses into a single linear transform: W2(W1x)W_2(W_1x)W2​(W1​x) is just (W2W1)x(W_2W_1)x(W2​W1​)x, so a hundred layers equal one and cannot learn a curve. The output vector of a layer after the nonlinearity is its activation, the quantity backpropagation stores on the way forward.

The choice decides the shape of what the network can draw and how well gradients survive depth. Tanh, tanh⁡z\tanh ztanhz, is smooth and saturating, squashes into (−1,1)(-1,1)(−1,1) and gives curved decision boundaries; its slope tends to zero for large inputs, which feeds the vanishing gradient. ReLU, max⁡(0,z)\max(0,z)max(0,z), has slope 1 wherever it is active, so gradients cross many layers intact, and it makes the network piecewise linear with straight-edged boundaries. The sigmoid maps to (0,1)(0,1)(0,1) and survives where a value must read as a probability or a valve opening (the gates of an LSTM, the output of a binary classifier).

ReLU replaced the sigmoid as the default in hidden layers, and GELU, a smooth version, took over in transformers.

The activation sets how the network extrapolates. Outside the data, a ReLU network continues its last slope in a straight line and a tanh network flattens out; neither knows it has left the region it was trained on (universal approximation theorem).

It is a different thing from softmax, which normalizes a whole vector into weights that sum to 1 instead of acting on each entry alone.