📂Matrix Algebra

Inverse and Square Root of Positive Definite Matrices

Formulas ¹

Let’s say the eigenpairs $\left\{ \left( \lambda_{k} , e_{k} \right) \right\}_{k=1}^{n}$ of a positive definite matrix $A$ are arranged in order $\lambda_{1} > \cdots > \lambda_{n} > 0$. Regarding the orthogonal matrix $P = \begin{bmatrix} e_{1} & \cdots & e_{n} \end{bmatrix} \in \mathbb{R}^{n \times n}$ and the diagonal matrix $\Lambda = \diag \left( \lambda_{1} , \cdots , \lambda_{n} \right)$, the inverse matrix $A^{-1}$ and the square root matrix $\sqrt{A}$ of $A$ are as follows: $$ \begin{align*} A^{-1} =& P \Lambda^{-1} P^{T} = \sum_{k=1}^{n} {{ 1 } \over { \lambda_{k} }} e_{k} e_{k}^{T} \\ \sqrt{A} =& P \sqrt{\Lambda} P^{T} = \sum_{k=1}^{n} \sqrt{\lambda_{k}} e_{k} e_{k}^{T} \end{align*} $$

---

• $X^{T}$ is the transpose matrix of the matrix $X$.

Derivation

Spectral Decomposition:

Spectral Theory: It is equivalent that $A$ is a Hermitian matrix and can be unitarily diagonalized: $$ A = A^{\ast} \iff A = Q \Lambda Q^{\ast} $$

In particular, in statistics, covariance matrices are often positive definite matrices and positive definite matrices are Hermitian matrices. Not just covariance matrices but also with respect to a design matrix $X$, $X^{T} X$ is a symmetric matrix, especially if $X \in \mathbb{R}^{m \times n}$, it becomes a Hermitian matrix again. Under these conditions, according to spectral theory, $A$ can obtain $Q$, which consists of orthonormal eigenvectors $e_{1} , \cdots , e_{n}$, restated as follows: $$ \begin{align*} & A \\ = & Q \Lambda Q^{\ast} \\ = & Q \begin{bmatrix} \lambda_{1} & 0 & \cdots & 0 \\ 0 & \lambda_{2} & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & \lambda_{n} \end{bmatrix} \begin{bmatrix} e_{1}^{\ast} \\ e_{2}^{\ast} \\ \vdots \\ e_{n}^{\ast} \end{bmatrix} \\ = & \begin{bmatrix} e_{1} & e_{2} & \cdots & e_{n} \end{bmatrix} \begin{bmatrix} \lambda_{1} e_{1}^{\ast} \\ \lambda_{2} e_{2}^{\ast} \\ \vdots \\ \lambda_{n} e_{n}^{\ast} \end{bmatrix} \\ = & \lambda_{1} e_{1} e_{1}^{\ast} + \lambda_{2} e_{2} e_{2}^{\ast} + \cdots + \lambda_{n} e_{n} e_{n}^{\ast} \\ = & \sum_{k=1}^{n} \lambda_{k} e_{k} e_{k}^{\ast} \end{align*} $$

Since $\Lambda$ is a diagonal matrix, there’s nothing special to derive. Since $P$ is an orthogonal matrix, $$ \begin{align*} A^{-1} =& \left( P \Lambda P^{T} \right)^{-1} \\ =& P^{-T} \Lambda^{-1} P^{-1} \\ =& P \Lambda^{-1} P^{T} \end{align*} $$ and the following verification yields $\sqrt{A}$ with respect to the identity matrix $I$: $$ \begin{align*} & \left( P \sqrt{\Lambda} P^{T} \right) \left( P \sqrt{\Lambda} P^{T} \right) \\ =& P \sqrt{\Lambda} P^{T} P \sqrt{\Lambda} P^{T} \\ =& P \sqrt{\Lambda} I \sqrt{\Lambda} P^{T} \\ =& P \sqrt{\Lambda} \sqrt{\Lambda} P^{T} \\ =& P \Lambda P^{T} \\ =& A \end{align*} $$

■