Softmax and Sigmoid

def. Sigmoid Function is a simple S-shaped function.

$$\sigma (z):=\frac{1}{1+e^z}$$

def. Softmax function takes in a vector $z$ of dimension $K$ and normalizes it into a probability distribution with $K$ different outcomes:

$$z=\left[\begin{array}{c}{z}_1\\ z_2\\ ⋮\\ z_n\end{array}\right]\to \sigma (z{)}_i=\frac{e^{z_i}}{{\sum }_{j=1}^K{e}^{z_j}}\text{for each i}$$

Sometimes if we input each $z_i$ separately we see it as a function that already knows $z$:

$$\sigma (z_k|z_1,\dots ,z_n)=\frac{\exp (z_k)}{{\sum }_{i=1}^n\exp (z_i)}$$

and it is the single-element (scalar) ${\sigma }_k$, i.e. probability of class $k$.

Numerical Instability §

When calculating softmax in floating point, when the values get big there are instabilities. Instead, observe:

$$\begin{array}{rl}\text{softmax}(x{)}_i& =\frac{e^{x_i}}{{\sum }_j{e}^{x_j}}\\ & =1\cdot \frac{e^{x_i}}{{\sum }_j{e}^{x_j}}\\ & =\frac{C}{C}\frac{e^{x_i}}{{\sum }_j{e}^{x_j}}\\ & =\frac{C{e}^{x_i}}{{\sum }_{j}C{e}^{x_j}}\\ & =\frac{e^{x_i+\log C}}{{\sum }_j{e}^{x_j+\log C}}\end{array}$$

Which means we can add any constant $\log C$ to prevent overflow/underflow and thus infs.