Softmax

Softmax function takes in a vector $z$ of dimension $K$ and normalizes it into a probability distribution with $K$ different outcomes:

$$z=\left[\begin{array}{c}{z}_1\\ z_2\\ ⋮\\ z_n\end{array}\right]\to \sigma (z{)}_i=\frac{e^{z_i}}{{\sum }_{j=1}^K{e}^{z_j}}\text{for each i}$$

Numerical Instability §

When calculating softmax in floating point, when the values get big there are instabilities. Instead, observe:

$$\begin{array}{rl}\text{softmax}(x{)}_i& =\frac{e^{x_i}}{{\sum }_j{e}^{x_j}}\\ & =1\cdot \frac{e^{x_i}}{{\sum }_j{e}^{x_j}}\\ & =\frac{C}{C}\frac{e^{x_i}}{{\sum }_j{e}^{x_j}}\\ & =\frac{C{e}^{x_i}}{{\sum }_{j}C{e}^{x_j}}\\ & =\frac{e^{x_i+\log C}}{{\sum }_j{e}^{x_j+\log C}}\end{array}$$

Which means we can add any constant $\log C$ to prevent overflow/underflow and thus infs.