Inverse Matrix of X^T X: Necessary and Sufficient Conditions 📂Matrix Algebra

Inverse Matrix of X^T X: Necessary and Sufficient Conditions

Theorem

Suppose that the matrix $X \in \mathbb{R}^{m \times n}$ is given and $m \ge n$, then the following holds: $$ \exists \left( X^{T} X \right)^{-1} \iff \text{rank} X = n $$ In other words, the necessary and sufficient condition for the inverse matrix of $X^{T} X$ to exist is that $X$ has full rank.

$X^{T}$ is the transpose of $X$.

Explanation

The reason this fact is important is that in overdetermined systems like $\mathbf{y} = X \beta$ in multiple regression analysis, you often need to perform calculations such as $$ \beta = \left( X^{T} X \right)^{-1} X^{T} \mathbf{y} $$ …not just often, but really a lot. Situations where one encounters problems due to not knowing this fact, or forgetting it during actual analysis, are also quite common.

Proof ¹

$(\implies)$

If the inverse of $X^{T} X \in \mathbb{R}^{n \times n}$ exists, then it is $\text{rank} X^{T} X = n$. $$ \begin{align*} n =& \text{rank} X^{T} X \\ \le & \text{rank} X \\ \le & \min \left\{ n , m \right\} \\ \le & n \end{align*} $$ For the above inequality to hold, it must be $\text{rank} X = n$.

$(\impliedby)$

For some $\mathbf{u} \in \mathbb{R}^{n}$, let $$ X^{T} X \mathbf{u} = \mathbf{0} $$ Let us assume that $\mathbf{u} = \mathbf{0}$, which is equivalent to the existence of $\left( X^{T} X \right)^{-1}$. We need to show that $\mathbf{u}$ is a zero vector. Since $\mathbb{R}^{n}$ is an inner product space, we can calculate the vector dot product of $X^{T} X \mathbf{u}$ and $\mathbf{0}$: $\left< \cdot , \cdot \right>$. As this is the inner product of zero vectors, it is obviously $0$, and we obtain $$ \begin{align*} 0 =& \left< X^{T} X \mathbf{u} , \mathbf{u} \right> \\ =& \left( X^{T} X \mathbf{u} \right)^{T} \mathbf{u} \\ =& \mathbf{u}^{T} X^{T} X \mathbf{u} \\ =& \left( X \mathbf{u} \right)^{T} X \mathbf{u} \\ =& \left< X \mathbf{u} , X \mathbf{u} \right> \end{align*} $$ This implies $X \mathbf{u} = \mathbf{0}$. Given that $X$ is assumed to have full rank, it must be $\mathbf{u} = \mathbf{0}$.

■