Inverse Matrix of X^T X: Necessary and Sufficient Conditions
Theorem
When $m \ge n$, a necessary and sufficient condition for the inverse matrix of the matrix $X \in \mathbb{R}^{m \times n}$ to exist is that $X$ has full rank. $$ \exists \left( X^{T} X \right)^{-1} \iff \text{rank} X = n $$
- $X^{T}$ is the transpose of $X$.
Explanation
The reason this fact is important is because in overdetermined systems like $\mathbf{y} = X \beta$ in multiple regression analysis, there are… actually, a lot of times when calculations such as $$ \beta = \left( X^{T} X \right)^{-1} X^{T} \mathbf{y} $$ have to be done through the least squares method. Not knowing this fact, or even knowing but not remembering it in actual analysis can easily lead to problems.
Proof 1
$(\implies)$
If the inverse matrix of $X^{T} X \in \mathbb{R}^{n \times n}$ exists, then $\text{rank} X^{T} X = n$. $$ \begin{align*} n =& \text{rank} X^{T} X \\ \le & \text{rank} X \\ \le & \min \left\{ n , m \right\} \\ \le & n \end{align*} $$ For the above inequality to hold, it must be $\text{rank} X = n$.
$(\impliedby)$
Let’s consider some $\mathbf{u} \in \mathbb{R}^{n}$ such that $$ X^{T} X \mathbf{u} = \mathbf{0} $$ If $\mathbf{u} = \mathbf{0}$, then the existence of $\left( X^{T} X \right)^{-1}$ is equivalent to proving that $\mathbf{u}$ is the zero vector. Since $\mathbb{R}^{n}$ is an inner product space, one can calculate the vector inner product $\left< \cdot , \cdot \right>$ of $X^{T} X \mathbf{u}$ and $\mathbf{0}$. Since it’s the inner product of zero vectors, naturally, the result is $0$, and $$ \begin{align*} 0 =& \left< X^{T} X \mathbf{u} , \mathbf{u} \right> \\ =& \left( X^{T} X \mathbf{u} \right)^{T} \mathbf{u} \\ =& \mathbf{u}^{T} X^{T} X \mathbf{u} \\ =& \left( X \mathbf{u} \right)^{T} X \mathbf{u} \\ =& \left< X \mathbf{u} , X \mathbf{u} \right> \end{align*} $$ is obtained. This implies $X \mathbf{u} = \mathbf{0}$, and since we assumed that $X$ has full rank, it must be $\mathbf{u} = \mathbf{0}$.
■