Confidence Intervals

Simple Definition §

def. Confidence Interval of the parameter $\theta \in [f(X,\psi ),g(X,\psi )]$ in the probability distribution of a random variable $X\sim \text{pdf}(\theta ,\psi )$ is defined as the following:

$$\text{C.I.}=\gamma :=\mathbb{P}[f(X,\psi )<\theta <g(X,\psi )]$$

• where functions $f(X),g(X)$ are able to be derived from the random variable $X$.

The most common use of confidence interval problems is with CLT, where a parameter of a binomial distribution is estimated. e.g.: let $X\sim \text{Binom}(n,p)$ where $p$ is the parameter to be estimated.

Formal Definition §

def. Confidence Interval. For $X_i$ distributed with an unknown parameter $\theta$, a $\gamma$-level confidence interval:

• …is an interval in which the probability of ${\theta }_0$ being in the interval is $\gamma$ [$=1-\alpha$-level CI]
  • think of $\gamma$ big is good, $\alpha$ small is good.
• …is formalized as the following where $L,U$ is a statistic of $X$:

$$\begin{array}{rl}& P(L<\theta <U)=\gamma \\ & ⟹\frac{L+U}{2}\pm \frac{U-L}{2}\\ & ⟹\text{Confidence Interval}=[L,U]\end{array}$$

def. Observed Information. For $X_1,\dots ,X_n\stackrel{iid}{\sim }f(x_1,\dots ,x_n;\theta )$, observed information is:

$$J(\theta )=-l^{\prime \prime }(X_i;\theta )$$

And for $X_1,\dots ,X_n\stackrel{iid}{\sim }f(x;\theta )$

$$J_n(\theta )=\sum _{i=1}^n-l^{\prime \prime }(X_i;\theta )$$

Remark. This is Fisher Information, but gathered on data ($X_i$)

thm. (approximating a confidence interval using Fisher Information) In general, if ${\hat{\theta }}_{\text{MLE}}$ can be found and the log-likelihood is twice-differentiable, an approximate CI of level $\gamma =1-\alpha$ can be approximated by:

$${\hat{\theta }}_{\text{MLE}}\pm \frac{z_{1-\frac{\alpha }{2}}}{\sqrt{J_n(\hat{\theta })}}$$

thm Delta Method. Let unknown parameter $\theta$ of r.v. $X_1,\dots ,X_n$’s distribution. If Fisher’s approximation [=asymptotically efficient and asymptotically unbiased] conditions are satisfied, the following holds for all $g$ which $g’(\theta )\ne 0$:

$$\sqrt{n}({\hat{g}}_{\text{MLE}}-g_0)\stackrel{n\to \infty }{\underset{d}{⟶}}\mathcal{N}(0,[g^{\prime }(\theta ){]}^2\cdot {\sigma }^2)$$

and due to Fisher’s Approximation (asymptotic normality):

$$\sqrt{n}({\hat{g}}_{\text{MLE}}-g_0)\stackrel{n\to \infty }{\underset{d}{⟶}}\mathcal{N}(0,\frac{g^{\prime }({\theta }_0{)}^2}{\mathcal{I}({\theta }_0)})$$

Example. To construct a standard normal distribution’s confidence interval of level $\gamma =1-\alpha$:

$$P(z_{\frac{\alpha }{2}}<Z<z_{1-\frac{\alpha }{2}})=1-\alpha$$