Moment (Probability)

def. Moment Generating Functions:

$$M_X(t)=E[\exp (tX)]$$

…where $M_X$ is differentiable $k$-times around zero to be able to generate the $k$-th moment. You can build an MGF from a pmf of pdf:

• for pmfs: $M_X(t)={\sum }_{}^{}{e}^{t{x}_i}p(x_i)$
• for pdfs: $M_X(t)={\int }_{-\infty }^{\infty }{e}^{tx}f(x)dx$ It can generate the $k$-th moment like such:

$$E(X^k)=\frac{{\partial }^k}{\partial x^k}{M}_X(0)=M_X^{(k)}(0)$$

thm. Linear Combination of MGFs. Given r.v. $W=X_1+\cdots +X_n$, and $X_i$ which are iid,:

$$M_W(t)=M_{X_1}\cdot M_{X_2}\cdots M_{X_n}$$

thm. Uniqueness Theorem of MGF. let $X,Y$ have CDFs $F_X,F_Y$ then:

$$M_X(t)=M_Y(t)⟺F_X=F_Y$$

i.e. this moment generating function fully characterizes the distribution of the random variable. thm. Taylor Expansion of MGF. For RV $X$

$$\begin{array}{rl}{M}_X(t)& =\mathbb{E}[e^{tX}]\\ & =1+t\mathbb{E}[X]+\frac{t^2}{2!}\mathbb{E}[X^2]+\frac{t^3}{3!}\mathbb{E}[X^3]+\cdots \end{array}$$

Intuition for Moments §

Motivation. Moments are a convenient way to characterize a distinction. The foundation is from this realization (1) The Uniqueness Theorem and (2) Taylor expansion. This means that a linear combination of $\mathbb{E}[X],\mathbb{E}[X^2],\mathbb{E}[X^3],\dots$ uniquely describes the distribution of $X$. It’s also nice that there’s a nice visual explanation for each.

Types of Moments §

For RV $X$:

1. Raw Moments: $\mathbb{E}[X],\mathbb{E}[X^2],\mathbb{E}[X^3]\dots$
2. Central Moments: $\mathbb{E}[X]=:\mu ,\mathbb{E}[(X-\mu {)}^2],\mathbb{E}[(X-\mu {)}^3],\dots$
3. Standardized Moments: $\mathbb{E}[X],\mathbb{E}[(X-\mu {)}^2],\mathbb{E}\left[{\left(\frac{X-\mu }{\sigma }\right)}^3\right]$

Special Moments §

• $0$th moment $\mathbb{E}[X^0]=1$ i.e. second probability anxiom
• 1st moment: $\mathbb{E}[X]=\mu$, the mean, location of the pdf
• 2nd (central) moment: $\mathbb{E}[(X-\mu {)}^2]={\sigma }^2$, the variance, the spread of the pdf
  • squaring disregards signs, so the farther mass is from $\mu$ the higher variance
• 3rd (standardized) moment: $\mathbb{E}\left[{\left(\frac{X-\mu }{\sigma }\right)}^3\right]$ the skew, relative tailed-ness of the pdf
  • cubing preserves signs, so the left-tails negatively contribute, and right-tails positively contribute
• 4th (standardized) moment: $\mathbb{E}\left[{\left(\frac{X-\mu }{\sigma }\right)}^4\right]$ the kurtosis, the absolute tailed-ness of the pdf
  • 4th power disregards signs, so the farther mass is from the $\mu$ of higher variance. Similar to variance, but higher punishment.

Method of Moments §

Motivation. An alternative way to get an estimator quick and dirty, as opposed to MLEs. alg. Method of Moments (MOM): let $X_1,\dots ,X_n\stackrel{iid}{\sim }f(x;{\theta }_1,\dots ,{\theta }_p)$. Then to get estimators for ${\theta }_1,\dots ,{\theta }_p$:

1. let ${\mu }_i=\mathbb{E}[X^i]=g_i({\theta }_1,\dots ,{\theta }_n)$
2. gather data on ${\bar{X}}^i$ to get an empirical estimate $m_i$ Observe that because $E[X^i]\approx {\bar{X}}^i$, this means that ${\mu }_i\approx m_i$
3. let function $g_i$ which maps ${\theta }_1,\dots ,{\theta }_n\stackrel{g_i}{\mapsto }{m}_i$ can be inverted
4. Get system of equations for as many $i$’s as necessary
5. Solve the system of equations as ${\hat{\theta }}_i=g_i^{-1}(m_1,\dots )$