RELU
REctifier Linear Units.
Used as an activation function of a neuron in a neural network.
- Rectifier since it removes
GELU
Gaussian Error Linear Unit
Where is the CDF of the standard Normal Distribution. The GELU function can be approximated by the following two approximations, the second one with less accuracy:
- where is the CDF of the logistic distribution