Maximum Likelihood Estimator

def. The Maximum Likelihood (Statistics) Estimator is an estimator $\hat{\theta }$ that maximizes $\mathcal{L}$.

• It also works for log likelihood, because the natural log is a monotonic function:

$$\begin{array}{rl}{\hat{\theta }}_{\text{MLE}}& =\underset{\theta }{\max }{\mathcal{L}}_n(\theta ;x_1,...,x_n)\\ & =\underset{\theta }{\max }\ln {\mathcal{L}}_n(\theta ;x_1,...,x_n)\end{array}$$

Under certain regularity conditions, we can find the MLE by finding stationary points in the log likelihood. These are called the likelihood equation:

$$\frac{\partial \ln {\mathcal{L}}_n(\theta )}{\partial \theta }\stackrel{\text{set to}}{=}0$$

To consider whether this stationary point is the maximum (as opposed to a miminum) either:

• take the second derivative…
• …or find out via other means

Properties of MLEs:

• MLEs are always a function of a sufficient statistic.
• MLEs are not necessarily unbiased.
• MLEs may not reach the CRLB in variance.

thm. Functional Equivariance of MLE: Given parameter $\theta$ and let $\alpha =g(\theta )$. Then:

$$g({\hat{\theta }}_{\text{MLE}})\stackrel{g\text{invertible}}{=}{\hat{\alpha }}_{\text{MLE}}$$

⇒ the estimator for any function over the parameter can then be found easily.

thm. Asymptotic Normality of MLE. [=Fisher’s Approximation]

let data generated by a univariate single parameter distribution $X_1,\dots ,X_n\stackrel{iid}{\sim }f(x_1,\dots ,x_n,\theta )$.

let also that ${\hat{\theta }}_{\text{MLE}}$ is found by the likelihood equation $\frac{\partial s}{\partial \theta }=0$. Then both are equivalently true:

$${\hat{\theta }}_{\text{MLE}}\stackrel{n\to \infty }{⟶}\mathcal{N}({\theta }_0,\frac{1}{\mathcal{I}({\theta }_0)})$$ $$\sqrt{n}({\hat{\theta }}_{\text{MLE}}-{\theta }_0)\stackrel{n\to \infty }{⟶}\mathcal{N}(0,\frac{1}{\mathcal{I}({\theta }_0)})$$