Likelihood (Statistics)

Likelihood:

• instead of asking “given parameter $\theta$ what is the probability of r.v. $X=x$”…
• …ask: “given data about $X$, what is the likelihood of parameter $\theta$ being within an interval?” This is the best way to evaluate an estimator.

Likelihood Function §

def. The likelihood function for $X_1,\dots ,X_n\stackrel{iid}{\sim }{f}_{X_i}(x_1,\dots ,x_n|\theta )$ is the likelihood that given the data, the liklihood of parameter to be that value:

$$\begin{gathered} \mathcal{L}(\theta )=f_X(\theta ;x) \\ {\mathcal{L}}_n(\theta )=f_n(\theta ;x_1,...,x_n)\stackrel{iid}{=}\prod _{i=1}^n{f}_{X_k}^{\text{univar}}(\theta ;x_k) \end{gathered}$$

where for the liklihood function, the variable is $\theta$ and the parameters are $x_1,\dots ,x_n$.

• The domain of the likelihood function is the parameter space.

Info

Alternatively, understanding $\mathcal{L}$ to be the (mythical) value of the pdf is another way to think of it. 확률(Probability) vs 가능도(Likelihood)

def. the Log liklihood is simply the natural log of the likelihood function. It exists because it’s just easy to manipulate.

$$\begin{gathered} l(\theta )=\log [\mathcal{L}(\theta )]=\log [f_{X_i}(\theta ;x)] \\ {l}_n(\theta )=\log [{\mathcal{L}}_n(\theta )]=\log [\prod _{i=1}^n{f}_{X_k}^{\text{univar}}(\theta ;x_k)] \end{gathered}$$

Score §

def. the Score is the derivative of the log likelihood. It measures how close the estimator $\hat{\theta }$ is to the actual value of $\theta$.

$$s(\theta )=\frac{\partial \log \mathcal{L}(\theta )}{\partial \theta }$$

• Score is best when 0, and the absolute value measures how far away $\hat{\theta }$ is from actual $\theta$. Signed for direction.
• $\mathbb{E}[s|{\theta }_{gt}]=0$ ← under regularity conditions. Obviously, if we know real $\theta$, then the score is perfect.