📂Mathematical Statistics

Sum of Squares Decomposition Represented by Quadratic Form of a Random Vector

Formula

For a random vector $\mathbf{X} = \left( X_{1} , \cdots , X_{n} \right)$, an identity matrix $I_{n} \in \mathbb{R}^{n \times n}$, and an all-ones matrix $J_{n} \in \mathbb{R}^{n \times n}$ whose elements are all $1$, the following holds: $$\mathbf{X}^{T} \left( I_{n} - {\frac{ 1 }{ n }} J_{n} \right) \mathbf{X} = ( n - 1 ) S^{2}$$ Here, $S^{2}$ is the sample variance.

Derivation

$$\begin{align*} \overline{X} =& {\frac{ 1 }{ n }} \sum_{k=1}^{n} X_{k} \\ S^{2} =& {\frac{ 1 }{ n - 1 }} \sum_{k=1}^{n} \left( X_{k} - \overline{X} \right)^{2} \end{align*}$$ Let the sample mean be $\overline{X}$ and the sample variance as above. The $\left( I_{n} - J_{n} / n \right)$ given in the theorem is a symmetric matrix whose diagonal elements are all $1 - 1/n$, and off-diagonal elements are all $-1/n$, making it a quadratic form of a random vector and can be expressed as follows: $$\mathbf{X}^{T} \left( a_{ij} \right) \mathbf{X} = \sum_{i=1}^{n} a_{ii} X_{i}^{2} + \sum_{i \ne j} a_{ij} X_{i} X_{j}$$ Substituting $a_{ii} = 1 - 1/n$ and $a_{ij} = -1/n$ into the equation above yields: $$\begin{align*} & \mathbf{X}^{T} \left( a_{ij} \right) \mathbf{X} \\ =& \sum_{i=1}^{n} \left( 1 - {\frac{ 1 }{ n }} \right) X_{i}^{2} + \sum_{i \ne j} \left( - {\frac{ 1 }{ n }} \right) X_{i} X_{j} \\ =& \sum_{i=1}^{n} X_{i}^{2} - {\frac{ 1 }{ n }} \sum_{i=1}^{n} X_{i}^{2} - {\frac{ 1 }{ n }} \sum_{i, j} X_{i} X_{j} \\ =& \sum_{i=1}^{n} X_{i}^{2} - {\frac{ 1 }{ n }} \sum_{i=1}^{n} X_{i}^{2} - {\frac{ 1 }{ n }} \sum_{i, j} X_{i} X_{j} \\ =& \sum_{i=1}^{n} X_{i}^{2} - {\frac{ 1 }{ n }} \sum_{i=1}^{n} X_{i} \sum_{i=1}^{n} X_{i} \\ =& \sum_{i=1}^{n} X_{i}^{2} - n \overline{X}^{2} \\ =& ( n - 1 ) S^{2} \end{align*}$$

■