Least Squares Characterization for Linear Regression (Normal Equations)

Let $m$ and $p$ be natural numbers. Let $X=(X_{\alpha i})$ be an $m\times p$ matrix with real entries (the design matrix), acting on vectors by the matrix-vector product, and let $y\in\mathbb{R}^m$ be a point of Euclidean space (the observation vector). Let $X^{\top}$ denote the transpose of $X$ (a $p\times m$ matrix), let $d_E$ denote the Euclidean distance, and for points $u,z$ of the same Euclidean space let $u-z$ denote their difference.

For $\beta\in\mathbb{R}^p$ define the residual sum of squares

A vector $\hat\beta\in\mathbb{R}^p$ is called a least squares estimator if $\mathrm{RSS}(\hat\beta)\le\mathrm{RSS}(\beta)$ for every $\beta\in\mathbb{R}^p$. Write $0_p=(0,\dots,0)\in\mathbb{R}^p$ and $0_m=(0,\dots,0)\in\mathbb{R}^m$ for the points all of whose coordinates are the real number $0$. Then:

1. (Normal equations.) A vector $\hat\beta\in\mathbb{R}^p$ is a least squares estimator if and only if

2. (Uniqueness of fitted values.) If $\hat\beta$ and $\hat\gamma$ are both least squares estimators, then $X\hat\beta=X\hat\gamma$.
3. (Uniqueness of the estimator.) If the only $v\in\mathbb{R}^p$ with $Xv=0_m$ is $v=0_p$, then there is at most one least squares estimator.

This is the ordinary least squares problem of linear regression: the vectors $X\beta$ range over the fitted values of the linear model with design matrix $X$, and part 1 identifies the least squares estimators of the regression coefficients as exactly the solutions of the normal equations.