Confidence Intervals

def. Confidence Interval. For $X$ distributed with an unknown parameter $\theta$, a $\gamma$-level confidence interval is an interval in which the probability of ${\theta }_0$ being in the interval is $\gamma$:

$$\begin{array}{rl}& P(L<{\theta }_0<U)=\gamma \\ & ⟹\frac{L+U}{2}\pm \frac{U-L}{2}\\ & ⟹\text{Confidence Interval}=[L,U]\end{array}$$

where $L,U$ are both random variables (=a statistic) from $X$.

def. Observed Information. For $X_1,\dots ,X_n\stackrel{iid}{\sim }f(x_1,\dots ,x_n;\theta )$, observed information is:

$$J(\theta )=-l^{\prime \prime }(X_i;\theta )$$

And for $X_1,\dots ,X_n\stackrel{iid}{\sim }f(x;\theta )$

$$J_n(\theta )=\sum _{i=1}^n-l^{\prime \prime }(X_i;\theta )$$

Remark. This is Fisher Information, but gathered on data ($X_i$)

thm. (approximating a confidence interval using Fisher Information) In general, if ${\hat{\theta }}_{\text{MLE}}$ can be found and the log-likelihood is twice-differentiable, an approximate CI of level $\gamma =1-\alpha$ can be approximated by:

$${\hat{\theta }}_{\text{MLE}}\pm \frac{z_{1-\frac{\alpha }{2}}}{\sqrt{J_n(\hat{\theta })}}$$

thm Delta Method. Let unknown parameter $\theta$ of r.v. $X_1,\dots ,X_n$’s distribution. If Fisher’s approximation [=asymptotically efficient and asymptotically unbiased] conditions are satisfied, the following holds for all $g$ which $g’(\theta )\ne 0$:

$$\sqrt{n}({\hat{g}}_{\text{MLE}}-g_0)\stackrel{n\to \infty }{\underset{d}{⟶}}\mathcal{N}(0,[g^{\prime }(\theta ){]}^2\cdot {\sigma }^2)$$

and due to Fisher’s Approximation (asymptotic normality):

$$\sqrt{n}({\hat{g}}_{\text{MLE}}-g_0)\stackrel{n\to \infty }{\underset{d}{⟶}}\mathcal{N}(0,\frac{g^{\prime }({\theta }_0{)}^2}{\mathcal{I}({\theta }_0)})$$

Example. To construct a standard normal distribution’s confidence interval of level $\gamma =1-\alpha$:

$$P(z_{\frac{\alpha }{2}}<Z<z_{1-\frac{\alpha }{2}})=1-\alpha$$