Normal Distribution

def. Normal Distribution. A random variable $X$ distributed over a Normal distribution with mean $\mu$ and standard deviation $\sigma$ is denoted:

$$\begin{array}{rl}X& \sim \text{Normal}(\mu ,\sigma )\\ f_X(x)& =\frac{1}{\sigma \sqrt{2\pi }}\cdot e^{-\frac{1}{2}(\frac{x-\mu }{\sigma }{)}^2}\\ P(a<X<b)& ={\int }_a^b\frac{1}{\sigma \sqrt{2\pi }}\cdot e^{-\frac{1}{2}(\frac{x-\mu }{\sigma }{)}^2}dx\end{array}$$

def. Cumulative Distribution Function (CDF) of a Normal Distribution

$$\Phi (z):={\int }_{-\infty }^z\frac{e^{-t^2/2}}{\sqrt{2\pi }}dt$$

Observe:

• $z\to +\infty ,\Phi (z)\to 1$
• $z\to -\infty ,\Phi (z)\to 0$
• $\Phi (-z)=1-\Phi (z)$

def. Standard Normal Distribution. A standard normal distribution is a normal distribution where $\mu =0,\sigma =1$

$$X\sim {\text{Normal}}_{std.}(0,1)$$

Tip

You can approximate a bunch of distributions using the Normal.

rmk. Linear Transformation of Normal Distribution. If $X\sim \text{N}[\mu ,{\sigma }^2]$, then

• $\mathbb{E}(aX+b)=a\mathbb{E}(X)+b=a\mu +b$
• $\text{Var}(aX+b)=a^2\text{Var}(X)=a^2{\sigma }^2$
• • ⇒ Thus $(aX+b)\sim \text{Normal}(a\mu +b,a^2{\sigma }^2)$

remark. Exponentiating Transformation of Normal Distribution. If $X\sim N(\mu ,{\sigma }^2)$

$$E[e^X]=e^{\mu +\frac{{\sigma }^2}{2}}$$

(using Law of Unconscious Statistician)

rmk. Standardizing the Normal Distribution. Given $X\sim N(\mu ,{\sigma }^2)$:

• $Y=\frac{X-\mu }{\sigma }$ has the standard normal distribution
• The pdf is as follows:

$$P(a<X<b)={\int }_a^b\frac{1}{\sigma \sqrt{2\pi }}\cdot e^{-\frac{1}{2}(\frac{x-\mu }{\sigma }{)}^2}dx\Rightarrow {\int }_{\frac{a-\mu }{\sigma }}^{\frac{b-\mu }{\sigma }}\frac{e^{-x^2/2}}{\sigma \sqrt{2\pi }}dz$$

• See also Standardizing a Random Variable

rmk. Empirical Rule: Rule of thumb for calculating probabilities (integrals) of normal distributions

• ! Generalized version: Chebyshev’s Inequality
• 1 std. dev. away is $\approx 66\%$; 2 std.dev. away is $\approx 0.95\%$

Box-Mueller Transform §

Motivation. Computers can easily sample from a uniform distribution, but it cannot randomly generage a normal distribution. The Box-Mueller Transform is a method of transforming a uniform unit random variable into a standard normal random variable. (From Problem 3) thm. Box-Mueller Transform. Let uniform distrubtions $U_1\sim \text{Unif}[0,1],U_2\sim \text{Unif}[0,1]$. Then let:

$$\begin{array}{rl}R& =\sqrt{-2\ln U_2}\\ \Theta & =2\pi U_1\end{array}$$

And then let:

$$\begin{array}{rl}{Z}_1& =R\cos \Theta \\ Z_2& =R\sin \Theta \end{array}$$

Then $Z_1,Z_2$ are standard normal distributions. proof. First, $U_1,U_2$ to $R,\Theta$. Consider that

• $U_1=\frac{\Theta }{2\pi }$, $\frac{d{U}_1}{d\Theta }=\frac{1}{2\pi }$
• $U_2=\exp \left(-\frac{R^2}{2}\right)$, $\frac{d{U}_2}{dR}=R\exp \left(-\frac{R^2}{2}\right)$ Using Change of Variable (Probability)s we have:
• $f_{\Theta }(\theta )=\frac{1}{2\pi }$
• $f_R(r)=r\exp \left(-\frac{r^2}{2}\right)$ Then, from $R,\Theta$ to $Z_1,Z_2$: Note that:

1. $Z_1^2+Z_2^2=R^2({\cos }^2\Theta +{\sin }^2\Theta )=R^2$ thus $R=\sqrt{Z_1^2+Z_2^2}$
   • $\frac{dR}{d{Z}_1}=Z_1(Z_1^2+Z_2^2{)}^{-1/2}=\frac{Z_1}{R}$
   • Symmetrically $\frac{dR}{d{Z}_2}=\frac{Z_2}{R}$
2. $\frac{Z_2}{Z_1}=\frac{R\sin \Theta }{R\cos \Theta }=\tan \Theta$ thus $\Theta =\arctan \left(\frac{Z_2}{Z_1}\right)$
   • $\frac{d\Theta }{d{Z}_1}=\frac{1}{1+\frac{Z_2^2}{Z_1^2}}\cdot Z_2(-1)Z_1^{-2}=-\frac{Z_2}{Z_1^2+Z_2^2}=-\frac{Z_2}{R^2}$
   • Symmetrically $\frac{d\Theta }{d{Z}_2}=\frac{Z_1}{Z_1^2+Z_2^2}=\frac{Z_1}{R^2}$ Now, calculate the jacobian for a multivariate change of variables:

$$\begin{array}{rl}|J|& =\left|\begin{array}{cc}\frac{\partial r}{\partial z_1}& \frac{\partial r}{\partial z_2}\\ \frac{\partial \theta }{\partial z_1}& \frac{\partial \theta }{\partial z_2}\end{array}\right|\\ & =\left|\begin{array}{cc}\frac{Z_1}{R}& \frac{Z_2}{R}\\ -\frac{Z_2}{R^2}& \frac{Z_1}{R^2}\end{array}\right|\\ & =\frac{Z_1^2}{R^3}+\frac{Z_2^2}{R^3}\\ & =\frac{1}{R}\end{array}$$

And thus the join probability being:

$$\begin{array}{rl}{f}_{\Theta ,R}(\theta ,r)& =f_{\Theta }(\theta )\cdot f_R(r)\cdot |J|\\ & =\frac{1}{2\pi }r\exp \left(-\frac{r^2}{2}\right)\cdot \frac{1}{r}\\ & =\frac{1}{2\pi }\exp \left(-\frac{z_1^2+z_2^2}{2}\right)\\ & =\frac{1}{\sqrt{2\pi }}\exp \left(-\frac{z_1^2}{2}\right)\cdot \frac{1}{\sqrt{2\pi }}\exp \left(-\frac{z_2^2}{2}\right)\\ & =f_{Z_1}(z_1)\cdot f_{Z_2}(z_2)\end{array}$$

This show both that:

1. $f_{Z_1},f_{Z_2}$ is the standard normal pdf
2. $Z_1,Z_2$ are independent because the joint pdf is a simple product of each pdf. ∎

Estimators §

let

• $X\sim \mathcal{N}(\mu ,{\sigma }^2)$
• $X_1,\dots ,X_n\stackrel{iid}{\sim }\mathcal{N}(\mu ,{\sigma }^2)$ ⇒ Log likelihood:

$$\ln {\mathcal{L}}_n(\mu ,{\sigma }^2|x_1,...,x_n)=-\frac{n}{2}\ln (2\pi )-\frac{n}{2}\ln ({\sigma }^2)-\frac{1}{2{\sigma }^2}\sum _{i=1}^n(x_i-\mu {)}^2$$

Score §

One R.V.	Multiple Data
$s(\mu )=\frac{x-\mu }{{\sigma }^2}$	$s_n(\mu )=\frac{n\mu -{\sum }_{i=1}^n{x}_i}{{\sigma }^2}$
$s(\sigma )=\frac{(x-\mu {)}^2}{{\sigma }^3}-\frac{1}{\sigma }$	$s_n(\sigma )=\frac{{\sum }_{i=1}^n(x_i-\mu {)}^2}{{\sigma }^3}-\frac{n}{\sigma }$

MLEs §

$$\hat{\mu }=\frac{1}{n}\sum _{i=1}^n{x}_i$$ $${\hat{\sigma }}^2=\frac{1}{n}\sum _{i=1}^n(x_i-\bar{x}{)}^2$$

• ! divisor for variance MLE is not $\frac{1}{n-1}$ as opposed to Besset Correction

Fisher Information §

• Unknown $\mu$, known ${\sigma }^2$

$$\begin{gathered} I(\mu )=\frac{1}{{\sigma }^2} \\ {I}_n(\mu )=\frac{n}{{\sigma }^2} \end{gathered}$$

• Known $\mu$ unknown ${\sigma }^2$

$$\begin{gathered} I({\sigma }^2)=\frac{1}{2{\sigma }^2} \\ {I}_n({\sigma }^2)=\frac{n}{2{\sigma }^2} \end{gathered}$$

Multivariate Normal Distribution §

def. Multivariate Normal Distribution.¹

$$\underset{\text{ scale down }}{\underbrace{\frac{1}{(2\pi {)}^{\frac{d}{2}}|\Sigma {|}^{\frac{1}{2}}}}}{e}^{-\frac{1}{2}(X-\mu {)}^T{\Sigma }^{-1}(X-\mu )}$$

where

• $d$ is the dimension
• $\Sigma$ is the covariance matrix

The image shows part of a larger equation or expression, as indicated by the curly brace on the right side, but the complete right-hand side is not visible in this image.

Footnotes §

1. Multivariate Normal (Gaussian) Distribution Explained - YouTube ↩