RELU

REctifier Linear Units.

Used as an activation function of a neuron in a neural network.

  • Rectifier since it removes

GELU

Gaussian Error Linear Unit

Where is the CDF of the standard Normal Distribution. The GELU function can be approximated by the following two approximations, the second one with less accuracy:

  • where is the CDF of the logistic distribution