Information Theory

A Short Introduction to Entropy, Cross-Entropy and KL-Divergence - YouTube

Motivation. Suppose a weather station is sending you information about the current weather. There is 50% chance of sun, and 50% chance of rain. Then the weather station can send you just one bit of information to sum this information: 1 if sunny, 0 if rainy.

def. Shannon Information.¹ Given a random variable $X$, the information given in a particular realization $x$ of $X$ is:

$$I_X(x):=-{\log }_b(p(x))$$

where $b$, the base, determines the units (either bits when $b=2$ or nats when $b=e$)

def. Shannon Entropy.² Given a random variable $X$, the entropy of this random variable is:

$$\begin{array}{rl}H(X)& :=-\sum _{\forall x}p(x)I_X(x)={\mathbb{E}}_X[I_X(x)]\end{array}$$

Intuition. Entropy is the average amount of information transmitted in total.

Example. In our weather station example, $X$ is the random variable:

$$X=\{\begin{array}{ll}1& \text{if sunny with }p=0.5\\ 0& \text{if rainy with }p=0.5\end{array}$$

When $1:\text{sunny}$, transmitted is $I_X(1)=-{\log }_{2}0.5=1\text{ bit}$ of information; same when $0:\text{rainy}$ is transmitted. Entropy of $X$ is:

$$H(X)=-\left(0.5{\log }_2(0.5)+0.5{\log }_2(0.5)\right)=1\text{ bit}$$

which matches our intution that entropy is the average amount of information transmitted.

def. Cross Entropy. The cross-entropy of the distribution $q$ relative to distribution $p$ is:^bic1ka

$$H(p,q)=-\underset{\text{ expectation }}{\underbrace{E_p}}[\log q]$$

Intution. Cross-entropy is a measure of a “distance” between two distributions. Remark. Often used in Deep Learning but it doesn’t have anything to do with information theory, but simply that coincidentally they had to calculate the “distance between two distributions.”

def. Kullback-Leibler Divergence (KL-Divergence). Another measure of “distance” between two distributions

$$\begin{array}{rl}{D}_{\text{KL}}(p\Vert q)& :=\sum _{x\in \mathcal{X}}p(x)\ln \left(\frac{p(x)}{q(x)}\right)\\ & ={\mathbb{E}}_{x\sim p(x)}[\ln p(x)-\ln q(x)]\end{array}$$

Example. KL Diverge between two multivariate gaussians $p(x)\sim \mathcal{N}({\mu }_p,{\Sigma }_p)$ and $q(x)\sim \mathcal{N}({\mu }_q,{\Sigma }_q)$ both of dimension $k$ can be derived³ to be:

$$D_{\text{KL}}(p\Vert q)=\frac{1}{2}\left(\ln \frac{|{\Sigma }_q|}{|{\Sigma }_p|}-k+({\mu }_p-{\mu }_q{)}^{\top }{\Sigma }_q^{-1}({\mu }_p-{\mu }_q)+\text{tr}\{{\Sigma }_q^{-1}{\Sigma }_p\}\right)$$

Footnotes §

1. Information content - Wikipedia ↩

2. Entropy (information theory) - Wikipedia ↩

3. KL Divergence between 2 Gaussian Distributions ↩