Published on

# Deriving the OLS Estimator

Authors
•  Name
Tails Azimuth

# Introduction

Parsing and display of math equations is included in this blog template. Parsing of math is enabled by remark-math and rehype-katex. KaTeX and its associated font is included in _document.js so feel free to use it in any pages. 1

Inline math symbols can be included by enclosing the term between the $ symbol. Math code blocks is denoted by $$. The dollar signal displays without issue since only text without space and between two $ signs are considered as math symbols.2

Inline or manually enumerated footnotes are also supported. Click on the links above to see them in action.

# Deriving the OLS Estimator

Using matrix notation, let $n$ denote the number of observations and $k$ denote the number of regressors.

The vector of outcome variables $\mathbf{Y}$ is a $n \times 1$ matrix,

\mathbf{Y} = \left[\begin{array}
{c}
y_1 \\
. \\
. \\
. \\
y_n
\end{array}\right]

$\mathbf{Y} = \left[\begin{array} {c} y_1 \\ . \\ . \\ . \\ y_n \end{array}\right]$

The matrix of regressors $\mathbf{X}$ is a $n \times k$ matrix (or each row is a $k \times 1$ vector),

\mathbf{X} = \left[\begin{array}
{ccccc}
x_{11} & . & . & . & x_{1k} \\
. & . & . & . & .  \\
. & . & . & . & .  \\
. & . & . & . & .  \\
x_{n1} & . & . & . & x_{nn}
\end{array}\right] =
\left[\begin{array}
{c}
\mathbf{x}'_1 \\
. \\
. \\
. \\
\mathbf{x}'_n
\end{array}\right]

$\mathbf{X} = \left[\begin{array} {ccccc} x_{11} & . & . & . & x_{1k} \\ . & . & . & . & . \\ . & . & . & . & . \\ . & . & . & . & . \\ x_{n1} & . & . & . & x_{nn} \end{array}\right] = \left[\begin{array} {c} \mathbf{x}'_1 \\ . \\ . \\ . \\ \mathbf{x}'_n \end{array}\right]$

The vector of error terms $\mathbf{U}$ is also a $n \times 1$ matrix.

At times it might be easier to use vector notation. For consistency I will use the bold small x to denote a vector and capital letters to denote a matrix. Single observations are denoted by the subscript.

## Least Squares

Start:
$y_i = \mathbf{x}'_i \beta + u_i$

Assumptions:

1. Linearity (given above)
2. $E(\mathbf{U}|\mathbf{X}) = 0$ (conditional independence)
3. rank($\mathbf{X}$) = $k$ (no multi-collinearity i.e. full rank)
4. $Var(\mathbf{U}|\mathbf{X}) = \sigma^2 I_n$ (Homoskedascity)

Aim:
Find $\beta$ that minimises sum of squared errors:

$Q = \sum_{i=1}^{n}{u_i^2} = \sum_{i=1}^{n}{(y_i - \mathbf{x}'_i\beta)^2} = (Y-X\beta)'(Y-X\beta)$

Solution:
Hints: $Q$ is a $1 \times 1$ scalar, by symmetry $\frac{\partial b'Ab}{\partial b} = 2Ab$.

Take matrix derivative w.r.t $\beta$:

\begin{aligned}
\min Q           & = \min_{\beta} \mathbf{Y}'\mathbf{Y} - 2\beta'\mathbf{X}'\mathbf{Y} +
\beta'\mathbf{X}'\mathbf{X}\beta \\
& = \min_{\beta} - 2\beta'\mathbf{X}'\mathbf{Y} + \beta'\mathbf{X}'\mathbf{X}\beta \\
\text{[FOC]}~~~0 & =  - 2\mathbf{X}'\mathbf{Y} + 2\mathbf{X}'\mathbf{X}\hat{\beta}                  \\
\hat{\beta}      & = (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf{Y}                              \\
& = (\sum^{n} \mathbf{x}_i \mathbf{x}'_i)^{-1} \sum^{n} \mathbf{x}_i y_i
\end{aligned}

\begin{aligned} \min Q & = \min_{\beta} \mathbf{Y}'\mathbf{Y} - 2\beta'\mathbf{X}'\mathbf{Y} + \beta'\mathbf{X}'\mathbf{X}\beta \\ & = \min_{\beta} - 2\beta'\mathbf{X}'\mathbf{Y} + \beta'\mathbf{X}'\mathbf{X}\beta \\ \text{[FOC]}~~~0 & = - 2\mathbf{X}'\mathbf{Y} + 2\mathbf{X}'\mathbf{X}\hat{\beta} \\ \hat{\beta} & = (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf{Y} \\ & = (\sum^{n} \mathbf{x}_i \mathbf{x}'_i)^{-1} \sum^{n} \mathbf{x}_i y_i \end{aligned}

1. For the full list of supported TeX functions, check out the KaTeX documentation
2. Here's $10 and$20.