Estimator

def. Statistic. let $X_1,\dots ,X_n$ observable random variables [= data of an experiment]. then statistic $T$ is:

$$T=\delta (X_1,...,X_n)$$

• $\delta :\{X_1,\dots ,Xn\}\to R$ i.e. is a real-valued function
• $\delta$ cannot contain unknown variables

def. Estimator [= point estimate] is a statistic used to estimate the parameter of the model we think the data is showing. Note the following notation convention:

$$\hat{\theta }=\delta (X_1,...,X_n)$$

• Assume $X$ as an r.v. of an experiment, whose model includes parameter $\theta$.
• To estimate ground truth parameter $\theta$, we can use an estimator r.v. $\hat{\theta }(X)$
• A specific estimate for a particular observed value $x_1$ is denoted $\hat{\theta }(x_1)$
• An estimator has to be a function of known variables & data only.
• $Var(\hat{\theta })=E[(\hat{\theta }-E\hat{\theta }{)}^2]$, NOT $E[(\hat{\theta }-\theta {)}^2]$ ← This is MSE

How Good is Your Estimator? §

1. Accuracy is higher. Increased as Bias (Statistics) is decreased
2. Precision is higher. Increased as Variance $V(\hat{\theta })$ is decreased
3. Efficiency (Statistics) is higher. If estimators ${\hat{\theta }}_1,{\hat{\theta }}_2$ have the same accuracy, but $\mathbb{V}({\hat{\theta }}_1)<\mathbb{V}[{\hat{\theta }}_2]$ then the former is more efficient than the latter.
4. Consistency.
5. Mean Squared Error is lower.
6. Likelihood (Statistics) is higher.

→ In general, making sure to reduce bias of estimators is important. Note that:

• If you can write down what the bias is mathematically [= characterize the bias], then you can make a new estimator that doesn’t have the bias.
• Bias usually decreases as the data points increase

Example

let $X_1,\dots ,X_n\stackrel{iid}{\sim }\mathcal{N}(\mu ,{\sigma }^2)$ and let estimator $\hat{\theta }:={\sum }_i^n{a}_i{X}_i$ where

• $a_1,\dots ,a_n$ are weights that sum to 1. [= weighted average]
• $\hat{\theta }$ is estimating $\mu$. $\sigma$ is known.

---

1. How accurate is $\hat{\theta }$? [=what is the bias?]

$$>\begin{array}{rl}>\mathbb{E}[\hat{\theta }]& =\mathbb{E}[\sum a_i{X}_i]\\ >& =\sum \mathbb{E}[a_i{X}_i]\\ >& =\sum a_i\mathbb{E}[X_i]\\ >& =\mu \\ >\mathbb{B}(\hat{\theta })& =0>\end{array}>>>$$

1. How precise is $\hat{\theta }$? What are the best $a_i,\dots ,a_n$?

$$>\begin{array}{rl}>\mathbb{V}(\hat{\theta })& =\mathbb{V}[\sum a_i{X}_i]\\ >>& =\mathbb{V}[a_1{X}_1]+\cdots \mathbb{V}[a_n{X}_n]\\ >& =a_1^2\mathbb{V}[X_1]+\cdots a_n^2\mathbb{V}[X_n]\\ >& =a_1^2{\sigma }^2+\cdots a_n^2{\sigma }^2\\ >& ={\sigma }^2\sum a_i^2>>\end{array}>>>$$

→ Thus $\mathbb{V}[\hat{\theta }]$ is minimized when $a_i=\frac{1}{n}$.