Information Theory

A Short Introduction to Entropy, Cross-Entropy and KL-Divergence - YouTube

Motivation. Suppose a weather station is sending you information about the current weather. There is 50% chance of sun, and 50% chance of rain. Then the weather station can send you just one bit of information to sum this information: 1 if sunny, 0 if rainy.

def. Shannon Information.¹ Given a random variable $X$, the information given in a particular realization $x$ of $X$ is:

$$I_X(x):=-{\log }_b(p(x))$$

where $b$, the base, determines the units (either bits when $b=2$ or nats when $b=e$)

def. Shannon Entropy.² Given a random variable $X$, the entropy of this random variable is:

$$\begin{array}{rl}H(X)& :=-\sum _{\forall x}p(x)I_X(x)={\mathbb{E}}_X[I_X(x)]\end{array}$$

Intuition. Entropy is the average amount of information transmitted in total.

Example. In our weather station example, $X$ is the random variable:

$$X=\{\begin{array}{ll}1& \text{if sunny with }p=0.5\\ 0& \text{if rainy with }p=0.5\end{array}$$

When $1:\text{sunny}$, transmitted is $I_X(1)=-{\log }_{2}0.5=1\text{ bit}$ of information; same when $0:\text{rainy}$ is transmitted. Entropy of $X$ is:

$$H(X)=-\left(0.5{\log }_2(0.5)+0.5{\log }_2(0.5)\right)=1\text{ bit}$$

which matches our intution that entropy is the average amount of information transmitted.

def. Cross Entropy.

Footnotes §

1. Information content - Wikipedia ↩

2. Entropy (information theory) - Wikipedia ↩