ML//neural network//activation function//sigmoid

The sigmoid is an activation function that squashes any real number into \((0,1)\) through \(\sigma(z)=1/(1+e^{-z})\), and it is used wherever a score has to read as a probability or an opening between closed and open. It was the classic choice in neural networks before ReLU.

The sigmoid is an activation function that squashes any real number into (0,1)(0,1)(0,1) through σ(z)=1/(1+e−z)\sigma(z)=1/(1+e^{-z})σ(z)=1/(1+e−z), and it is used wherever a score has to read as a probability or an opening between closed and open. It was the classic choice in neural networks before ReLU.

Its input zzz is the logit, the logarithm of the odds p/(1−p)p/(1-p)p/(1−p), so z=0z=0z=0 means even odds, z=2.2z=2.2z=2.2 about 90 % and z=4.6z=4.6z=4.6 about 99 %. That is the whole of logistic regression: a weighted sum of features passed through σ\sigmaσ, with the decision boundary at z=0z=0z=0 a hyperplane. The probability that a drone battery fails its capacity test, from its cycle count, its operating temperature and its internal resistance, is a typical case.

The output reads naturally as a probability or a valve setting (0 closed, 1 open). That is why LSTM gates use it, and why it is still standard for the final unit of a binary classifier, trained with cross-entropy.

It saturates at both ends: for large magnitudes the slope σ(1−σ)\sigma(1-\sigma)σ(1−σ) is near 0, and its maximum is only 0.25, so a stack of sigmoids shrinks the gradient at every layer and feeds the vanishing gradient. Hidden layers moved to ReLU and GELU for that reason.

A number in (0,1)(0,1)(0,1) is not yet a trustworthy probability: whether 0.9 means right nine times in ten is a separate property (probability calibration).