📂Matrix Algebra

Spectral Decomposition

Definition ¹

In spectral theory, the statement that $A$ is a Hermitian matrix is equivalent to it being unitarily diagonalizable: $$A = A^{\ast} \iff A = Q \Lambda Q^{\ast}$$

The term $A = Q \Lambda Q^{\ast}$ as mentioned in spectral theory is referred to as Spectral Decomposition, and is expressed as a series of eigenpairs $\left\{ \left( \lambda_{k} , e_{k} \right) \right\}_{k=1}^{n}$. $$A = \sum_{k=1}^{n} \lambda_{k} e_{k} e_{k}^{\ast}$$

Description

Especially in statistics, covariance matrices are often positive definite matrices, and positive definite matrices are Hermitian matrices. Not just covariance matrices, but also for design matrices $X$, $X^{T} X$ becomes a symmetric matrix, specially if $X \in \mathbb{R}^{m \times n}$ then again becomes a Hermitian matrix. Under such conditions, according to spectral theory, $A$ can achieve $Q$, composed of orthonormal eigenvectors $e_{1} , \cdots , e_{n}$, which can be rewritten as follows. $$\begin{align*} & A \\ = & Q \Lambda Q^{\ast} \\ = & Q \begin{bmatrix} \lambda_{1} & 0 & \cdots & 0 \\ 0 & \lambda_{2} & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & \lambda_{n} \end{bmatrix} \begin{bmatrix} e_{1}^{\ast} \\ e_{2}^{\ast} \\ \vdots \\ e_{n}^{\ast} \end{bmatrix} \\ = & \begin{bmatrix} e_{1} & e_{2} & \cdots & e_{n} \end{bmatrix} \begin{bmatrix} \lambda_{1} e_{1}^{\ast} \\ \lambda_{2} e_{2}^{\ast} \\ \vdots \\ \lambda_{n} e_{n}^{\ast} \end{bmatrix} \\ = & \lambda_{1} e_{1} e_{1}^{\ast} + \lambda_{2} e_{2} e_{2}^{\ast} + \cdots + \lambda_{n} e_{n} e_{n}^{\ast} \\ = & \sum_{k=1}^{n} \lambda_{k} e_{k} e_{k}^{\ast} \end{align*}$$