Bivariate Ordinary Least Squares Regression

Motivation. Let’s say that there is a relationship between GDP per capita and life expectancy. Maybe god has declared a perfect formula describing this relationship:

$$\text{GDP Per Capita}=200\cdot \text{Life Expectancy}+1000+\text{Noise}$$

While we humans may never truly know the parameters of the formula, $200$ and $1000$, we can still make a good guess about it. Therefore, assuming this is a linear relationship, we have the Bivariate Ordinary Least Squares Model.

$$Y_i={\beta }_0+{\beta }_1{X}_i+{ϵ}_i$$

where $(X_1,Y_1),\dots ,(X_n,Y_n)$ are observations (=regressors). The OLS algorithm will minimize the squared sum of residuals:

$${\text{min}}_{\widehat{{\beta }_1},\widehat{{\beta }_0}}\sum _{i=1}^N{\widehat{{ϵ}_i}}^2$$

where $\widehat{{ϵ}_i}:=Y_i-\underset{=\hat{Y}}{\underbrace{(\widehat{{\beta }_0}+\widehat{{\beta }_1}{X}_i)}}$. Note the square.

thm. Parameter OLS Estimator for $N$ observations $(X_i,Y_i)$

$$\begin{array}{rl}\widehat{{\beta }_1}& :=\frac{{\sum }_{i=1}^N(X_i-\bar{X})(Y_i-\bar{Y})}{{\sum }_{i=1}^N(X_i-\bar{X}{)}^2}\\ & :=\frac{{\sigma }_{XY}}{{\sigma }_X^2}={\rho }_{XY}\frac{{\sigma }_Y}{{\sigma }_X}\\ & \\ \widehat{{\beta }_0}& :=\bar{Y}-\widehat{{\beta }_1}\bar{X}\end{array}$$

Properties.

• Predictor: $\widehat{Y_i}=\widehat{{\beta }_0}+\widehat{{\beta }_1}{X}_i$
• Residual: $\widehat{ϵ}:=Y_i-\hat{Y}$ is the estimator for the error term, i.e. how good the predictor is.
  • $\frac{1}{N}{\sum }_i^N\widehat{{ϵ}_i}=0$ i.e. the mean of residuals is zero in OLS algorithm.
• OLS results in assuming that $X$ is exogenous, i.e. ${\rho }_{X,ϵ}=0$.
  • $\rho$
• Regression Variance: ${\hat{\sigma }}^2=\frac{{\sum }_{i=1}^N{\widehat{{ϵ}_i}}^2}{N-k}=\frac{{\sum }_{i=1}^N(Y_i-\bar{Y}{)}^2}{N-k}$ where $k$ is the number of parameters ($k=2$ in this case)
  • basically the Mean Squared Error. The lower the better.
  • $\hat{\sigma }$ is also the standard error of the residuals.
  • In Stata, $\hat{\sigma }$ is called the Root MSE

Evaluation of Estimators.

• Mean of $\widehat{{\beta }_1}$: $\mathbb{E}(\widehat{{\beta }_1})={\beta }_1+{\rho }_{X,ϵ}\frac{{\sigma }_{ϵ}}{{\sigma }_X}$
  • Thus bias is ${\rho }_{X,ϵ}\frac{{\sigma }_{ϵ}}{{\sigma }_X}$
  • If ${\rho }_{X,ϵ}=0$ then exogenous (this is the definition of exogeniety)
  • If ${\rho }_{X,ϵ}>0$ then there’s some 3rd factor positively correlated with $X$, thus bias is positive.
  • If ${\rho }_{X,ϵ}<0$ then v.v.
  • & Thus the bias characterizes exogeniety
• Variance of $\widehat{{\beta }_1}$: $\text{Var}(\widehat{{\beta }_1})=\frac{{\hat{\sigma }}^2}{N\cdot \text{Var}(X)}$
  • This is also called precision
  • $\sqrt{\text{Var}(\widehat{{\beta }_1})}$ is also called standard error of $\widehat{{\beta }_1}$.
  • ! For random variables, $\sqrt{\text{Var}(X)}$ is called standard deviation. For estimator random variables, it is called standard error. An abuse of terminology.

Other Data when Regression is Run §

There are a bunch of other variables that may matter, that is not included in the above core set of variables of the regression. Here are a few:

def. Coefficient of Determination (R-Squared). Intuition: The proportion of the variation in $\hat{Y}$ that can be determined from $X$. We define it as:

$$R^2:=1-\frac{S{S}_{res}}{S{S}_{tot}}$$

where

• $S{S}_{res}:={\sum }_i{\widehat{{ϵ}_i}}^2={\sum }_i(Y_i-\widehat{Y_i}{)}^2$ residual sum of squares
• $S{S}_{tot}:={\sum }_i(Y_i-\bar{Y}{)}^2$ total sum of squares

Example. A $R^2$ of $0.67$ means that around $67\%$ of the variation in $\hat{Y}$ is explained by $X$. Therefore, the higher it is, the better the regression is (=has more explanatory power).

Remark. It can also be shown that:

$$R^2={\rho }_{Y,\hat{Y}}^2$$

Measurement Error §

Motivation. There are measurement errors in every data; it can be both in the independent variable $X$ or the dependent variable $Y$. We can characterize measurement error (in this case a measurement error in $X$) as:

$$X_i=X_i^{\ast }+v_i$$

where $v_i$ is randomly distributed with $\mathbb{E}(v_i)=0$ and std. dev. ${\sigma }_v$, which is the measurement error. In this case, the regression in the model $\widehat{Y_i}=\widehat{{\beta }_0}+\widehat{{\beta }_1}{X}_i+{\widehat{ϵ}}_i$ will also change, into:

$$\widehat{{\beta }_1}{=}_{n\to \infty }{\beta }_1\frac{{\sigma }_{X^{\ast }}^2}{{\sigma }_v^2+{\sigma }_{X^{\ast }}^2}$$

We can extract a couple of facts from this relationship:

• Attenuation bias: the greater the ${\sigma }_v$, the closer $\widehat{{\beta }_1}$ is to zero.
• if ${\sigma }_v=0$ then $\widehat{{\beta }_1}={\beta }_1$, i.e. no measurement error

Alternatively, if there is a measurement error in $Y_i$, then:

1. Measurement error: $Y_i=Y_i^{\ast }+v_i$ where $\mathbb{E}(v_i)=0$ and std. dev. ${\sigma }_v$
2. Error term $v_i$ is absorbed by error term $\widehat{{ϵ}_i}$
3. This will increase $\hat{\sigma }$ (variance of regression) and thus $\text{Var}(\widehat{{\beta }_1})$, but does not bias the estimator.

Issues to Watch out for §

Heteroskedasticity is when the variance of data is different for some subsets of data than other subsets. ⇒ Use heteroskedatic-consistent standard errors by using robust in stata. This does not affect value of $\widehat{{\beta }_1}$ (how!) and does not bias it.

Autocorrelation often occurs in time series data where the error terms are sticky. For example, attendance at a NY yankees game will be sticky, because people who watched a good year will probably come back next year, even if the yankees aren’t as good as the year before. → used lagged variables