Fisher Information

Fisher information helps us find better estimators.

1. Cramer-Rao Lower Bound shows what the best estimators can do with their precision
2. Reaching the CRLB means it’s finite-sample efficient
3. The Maximum Likelihood Estimator is also a very good estimator.

def. Fisher Information is the amount of information we have about the unknown parameter. It’s the Expected Value of score.

• Given $f_X(x;\theta )$, if $X$ has a high peak we may assume that $X$ carries a lot of information about $\theta$.
• If $X$ is spread out a lot, we may assume that $X$ carries little information. Thus:

$$\begin{array}{rl}\mathcal{I}(\theta )& :=\text{Var}[s]\\ & =E[s^2]-E[s{]}^2\\ & =E[s^2]\end{array}$$

• (2) → (3) as we know that $E[s^2|{\theta }_{gt}]=0$

thm. Addition of fisher information. if $X_1,\dots ,X_n\stackrel{iid}{\sim }{f}_{X_i}(x_1,\dots ,x_n|\theta )$, then:

$${\mathcal{I}}_n=n\cdot \mathcal{I}$$

thm. Score and Fisher Information. if $f_X(\theta ;x)$ is twice differentiable wrt $\theta$, and under certain regularity conditions:

$$\mathcal{I}(\theta )=-E[\frac{{\partial }^2}{\partial {\theta }^2}\ln f_X(X;\theta )]$$

• Knowing this we also know that $E[s’]=-E[s^2]$.