ML//neural network//activation function//GELU

GELU (Gaussian Error Linear Unit) is an activation function that smoothly approximates ReLU: it computes \(x\,\Phi(x)\), where \(\Phi\) is the Gaussian CDF.

GELU (Gaussian Error Linear Unit) is an activation function that smoothly approximates ReLU: it computes x Φ(x)x,\Phi(x)xΦ(x), where Φ\PhiΦ is the Gaussian CDF.

It is used in Transformers, BERT and GPT, and is slightly better than ReLU for language tasks.