Inverse Matrix of X^T X: Necessary and Sufficient Conditions 📂Matrix Algebra

Inverse Matrix of X^T X: Necessary and Sufficient Conditions

Theorem

Suppose that the matrix $X \in \mathbb{R}^{m \times n}$ is given and $m \ge n$ , then the following holds: $\exists \left( X^{T} X \right)^{-1} \iff \text{rank} X = n$ In other words, the necessary and sufficient condition for the inverse matrix of $X^{T} X$ to exist is that $X$ has full rank.

$X^{T}$ is the transpose of $X$ .

Explanation

The reason this fact is important is that in overdetermined systems like $\mathbf{y} = X \beta$ in multiple regression analysis, you often need to perform calculations such as $\beta = \left( X^{T} X \right)^{-1} X^{T} \mathbf{y}$ …not just often, but really a lot. Situations where one encounters problems due to not knowing this fact, or forgetting it during actual analysis, are also quite common.

Proof ¹

$(\implies)$

If the inverse of $X^{T} X \in \mathbb{R}^{n \times n}$ exists, then it is $\text{rank} X^{T} X = n$ . $\begin{align*} n =& \text{rank} X^{T} X \\ \le & \text{rank} X \\ \le & \min \left\{ n , m \right\} \\ \le & n \end{align*}$ For the above inequality to hold, it must be $\text{rank} X = n$ .

$(\impliedby)$

For some $\mathbf{u} \in \mathbb{R}^{n}$ , let $X^{T} X \mathbf{u} = \mathbf{0}$ Let us assume that $\mathbf{u} = \mathbf{0}$ , which is equivalent to the existence of $\left( X^{T} X \right)^{-1}$ . We need to show that $\mathbf{u}$ is a zero vector. Since $\mathbb{R}^{n}$ is an inner product space, we can calculate the vector dot product of $X^{T} X \mathbf{u}$ and $\mathbf{0}$ : $\left< \cdot , \cdot \right>$ . As this is the inner product of zero vectors, it is obviously $0$ , and we obtain $\begin{align*} 0 =& \left< X^{T} X \mathbf{u} , \mathbf{u} \right> \\ =& \left( X^{T} X \mathbf{u} \right)^{T} \mathbf{u} \\ =& \mathbf{u}^{T} X^{T} X \mathbf{u} \\ =& \left( X \mathbf{u} \right)^{T} X \mathbf{u} \\ =& \left< X \mathbf{u} , X \mathbf{u} \right> \end{align*}$ This implies $X \mathbf{u} = \mathbf{0}$ . Given that $X$ is assumed to have full rank, it must be $\mathbf{u} = \mathbf{0}$ .

■