ML//neural network//activation function//ReLU

ReLU (rectified linear unit) is the activation function \(\max(0,z)\), which passes a positive input unchanged and sets a negative one to zero, and it is the default nonlinearity in the hidden layers of most networks and the threshold gate in MLP layers. A network built from it is piecewise linear: each unit folds the input space along one hyperplane, so the decision boundaries it draws have straight edges and corners, and a hidden layer of three units can already enclose a triangle around a ring of points.

ReLU (rectified linear unit) is the activation function max⁡(0,z)\max(0,z)max(0,z), which passes a positive input unchanged and sets a negative one to zero, and it is the default nonlinearity in the hidden layers of most networks and the threshold gate in MLP layers. A network built from it is piecewise linear: each unit folds the input space along one hyperplane, so the decision boundaries it draws have straight edges and corners, and a hidden layer of three units can already enclose a triangle around a ring of points.

It solved the vanishing gradient problem for deep nets, because its gradient is either 0 or 1 and never shrinks on the active side. In transformers it acts as a feature compatibility threshold: if the input aligns with a feature direction above zero, that feature fires, and below zero it is suppressed. Its downside is dead neurons: once a unit's input stays negative for every example, its gradient is zero and it never recovers. GELU, a smooth approximation, is the modern fix.

Piecewise linear means linear extrapolation. Outside the region covered by the data a ReLU network keeps its last slope with the same confidence it has inside, so a learned model of a motor's temperature trained up to 80 % load predicts a straight line at 120 % (universal approximation theorem, out-of-distribution).

It costs one comparison per unit, which matters on a microcontroller running an int8 network in a fast loop.