Multivariate Ordinary Least Squares Regression

$$Y_i={\beta }_0+{\beta }_1{X}_{1i}+{\beta }_2{X}_{2i}+{\beta }_3{X}_{3i}+{ϵ}_i$$

Omitted Variables in the regression. Refer to the document in that case.

Variance of Parameter Estimators §

In this multivariate model the variance of $\widehat{{\beta }_1}$ is:

$$\text{Var}(\widehat{{\beta }_1})=\frac{{\hat{\sigma }}^2}{N\cdot \text{Var}(X_1)\cdot (1-R_1^2)}$$

where

• ${\hat{\sigma }}^2$ is the variance of the regression
• $R_1$ is the coefficient of determination in the auxiliary regression $X_{1i}={\gamma }_0+{\gamma }_1{X}_{2i}+{\gamma }_2{X}_{3i}+{ϵ}_i$. It measures multicolinearity
  • & It measures how much of $X_1$ can be explained by $X_2,X_3$
  • In perfect multicolinearity $R_1=1$ then $X_2,X_3$ will perfectly determine $X_1$, thus $X_1$ is not relevant anymore.
  • & Equivalently $R_2$ is the coefficient of determination from $X_{2i}={\gamma }_0+{\gamma }_1{X}_{1i}+{\gamma }_2{X}_{3i}+{\epsilon }_i$, etc. etc.
• $\frac{1}{1-R_1^2}$ is known as the variance inflation factor. The higher the multicollinearity, the more inflated is the variance of parameter estimators.
• Multicollinearity is not a problem iff:
  • it occurs only between control variables
  • $\text{Var}(\widehat{{\beta }_1})$ is small enough; i.e. $\widehat{{\beta }_1}$ is statistically significant.
• $N$ is the sample size

Coefficient of Determination §

In multivariate situations (as opposed to bivariate) the coefficient of determination of the whole regression ($R^2$) is more troublesome because

1. Adding more variables will only increase $R^2$
2. Thus one is incentivized to increase the number of (possibly irrelevant) independent variables.
3. To mitigate this, we report the adjusted $R^2$ values instead, with a penalty for each additional independent variables used.

Hypothesis Testing regarding Coefficients §

Standardizing Coeffiefficients §

Example. In the model $\text{GDP}={\beta }_0+{\beta }_1\text{Life Expectancy}+{\beta }_2\text{Literacy Rate}$, if we want to compare the effects of life expectancy and literacy, we cannot simply compare the values of ${\beta }_1,{\beta }_2$. This is because their units are different, i.e. ${\beta }_1$ is in $\frac{\text{Dollars}}{\text{Year}}$ and literacy is in $\frac{\text{Dollars}}{\text{Percentage Point}}$. Thus we need to standardize them:

$$\widehat{{\beta }_i^{\text{std}}}=\frac{\widehat{{\beta }_i}-\mathbb{E}(\widehat{{\beta }_i})}{\sqrt{\text{Var}(\widehat{{\beta }_i})}}$$

which shows “how much $Y$ increase in units of ${\sigma }_Y$ does one unit ${\sigma }_{X_i}$ increase in $X$ cause?”

Remark. Only $X_1$ and $X_2$ need be standardized to compare $\widehat{{\beta }_1}$ and $\widehat{{\beta }_2}$’s effects. $Y$ need not be standardized.

Hypothesis Testing about Coefficients §

Let the model $Y_i={\beta }_0+{\beta }_1{X}_{1,i}+{\beta }_2{X}_{2,i}+{\beta }_3{X}_{3,i}+{ϵ}_i$. Sometimes we may want to check if $\widehat{{\beta }_1}{=}^{?}\widehat{{\beta }_2}$, or $\widehat{{\beta }_1}=\widehat{{\beta }_2}{=}^{?}0$. In these cases we use a hypothesis test. Let $R_{\text{unrestricted}}^2$ be the $R^2$ of this regression. Now, before we do anything we need to…

X_{1},X_{2}.

Case 1: $H_0:\widehat{{\beta }_1}=\widehat{{\beta }_2}=0$. Then the model under null would change to:

$$Y_i={\beta }_0+{\beta }_3{X}_{3,i}+{ϵ}_i$$

We run regression on this new model and get $R_{\text{restricted}}^2$ Remark. This is not equivalent to running a t-test on $H_A:\widehat{{\beta }_1}\ne 0\hat{\wedge }{\beta }_2\ne 0$ because $X_1,X_2$ may be multicollinear.

Case 2: $H_0:\widehat{{\beta }_1}=\widehat{{\beta }_2}$. Then the model under null would change to:

$$Y_i={\beta }_0+{\beta }_1(X_{1,i}+X_{2,i})+{\beta }_3{X}_{3,i}$$

We run regression on this to also get $R_{\text{restricted}}^2$.

We can observe that in both cases, $R_{\text{unrestricted}}^2>R_{\text{restricted}}^2$, always, because “restricting” the model will lead only to less (coefficient of-) determination. Now, the bigger this difference is, the more likely that the null is false. We formalize this using the $F$-test:

def. F-Test. For the F-statistic defined as:

$$F_{q,N-k}:=\frac{(R_{\text{unres.}}^2-R_{\text{restr.}}^2)/q}{(1-R_{\text{unres}}^2)/(N-k)}$$

where

• $q$ is “how many equal signs in null hypothesis”
• $k$ is the degrees of freedom (=number of coefficients in the _un_restricted model) Then:

$$\{\begin{array}{ll}{H}_0& \text{else}\\ H_1& \text{if }F>K\end{array}$$

where $K$ is the critical value. The critical values are:

• $K=3.00$ in case 1 ($H_0:\widehat{{\beta }_1}=\widehat{{\beta }_2}=0$)
• $K=3.84$ in case 2 ($H_0:\widehat{{\beta }_1}\ne \widehat{{\beta }_2}$)