📂Mathematical Statistics

Proof of the Hogg-Craig Theorem

Theorem

Let sample $\mathbf{X} = \left( X_{1} , \cdots , X_{n} \right)$ follow iid Normal distribution like $X_{1} , \cdots , X_{n} \overset{\text{iid}}{\sim} N \left( 0, \sigma^{2} \right)$. Consider a symmetric matrix $A_{1} , \cdots , A_{k} \in \mathbb{R}^{n \times n}$ and a random variable $Q_{1} , \cdots , Q_{k}$ represented as a random vector quadratic form $Q_{i} := \mathbf{X}^{T} A_{i} \mathbf{X}$. Define symmetric matrix $A$ and random variable $Q$ as follows: $$ \begin{align*} A =& A_{1} + \cdots + A_{k} \\ Q =& Q_{1} + \cdots + Q_{k} \end{align*} $$ If $Q / \sigma^{2}$ follows a Chi-squared distribution $\chi^{2} \left( r \right)$ and for $i = 1 , \cdots , k-1$ it is $Q_{i} / \sigma^{2} \sim \chi^{2} \left( r_{i} \right)$, and $Q_{k} \ge 0$ is true, then $Q_{1} , \cdots , Q_{k}$ is independent, and $Q_{k} / \sigma^{2}$ follows a Chi-squared distribution with degrees of freedom $r_{k} = r - r_{1} - \cdots - r_{k-1}$ $\chi^{2} \left( r_{k} \right)$.

Explanation

It might initially seem strange that not $Q / n \sigma^{2}$ but $Q / \sigma^{2}$ follows a Chi-squared distribution, but it is precise to discuss $Q / \sigma^{2}$ because the addition involved is not of samples, but of a matrix as follows: $$ \begin{align*} Q =& Q_{1} + \cdots + Q_{k} \\ =& \mathbf{X}^{T} A_{1} \mathbf{X} + \cdots + \mathbf{X}^{T} A_{k} \mathbf{X} \\ =& \mathbf{X}^{T} \left( A_{1} + \cdots + A_{k} \right) \mathbf{X} \\ =& \mathbf{X}^{T} A \mathbf{X} \end{align*} $$

This theorem is used in the proof of Cochran’s theorem.

Proof ¹

We will prove it by mathematical induction. First, let $k = 2$.

Equivalence condition for Chi-squaredness of normal distribution random vector quadratic forms: Let sample $\mathbf{X} = \left( X_{1} , \cdots , X_{n} \right)$, and follow iid Normal distribution like $X_{1} , \cdots , X_{n} \overset{\text{iid}}{\sim} N \left( 0, \sigma^{2} \right)$. For a symmetric matrix with rank $r \le n$ $A \in \mathbb{R}^{n \times n}$, if we set the random vector quadratic form as $Q = \sigma^{-2} \mathbf{X}^{T} A \mathbf{X}$, the following holds. $$ Q \sim \chi^{2} (r) \iff A^{2} = A $$

Since $Q / \sigma^{2}$ follows a Chi-squared distribution, $A$ is an idempotent matrix.

Eigenvalues of an idempotent matrix: The eigenvalues of an idempotent matrix are only $0$ or $1$.

Since $A$ is symmetric and a real matrix, it is diagonalizable, and the eigenvalues of $A$ are only $0$ and $1$, so there exists an orthogonal matrix which satisfies the following regarding the sizes of identity matrix $I_{r} \in \mathbb{R}^{r \times r}$ and zero matrix $O$. $$ \Gamma^{T} A \Gamma = \begin{bmatrix} I_{r} & O \\ O & O \end{bmatrix} $$

If you derive $A = A_{1} + A_{2}$, it is as follows. $$ \begin{bmatrix} I_{r} & O \\ O & O \end{bmatrix} = \Gamma^{T} A_{1} \Gamma + \Gamma^{T} A_{2} \Gamma $$

Positive definitiveness and eigenvalues: The necessary and sufficient condition for $A$ to be positive definite is that all eigenvalues of $A$ are positive.

Since it was assumed $Q_{2} \ge 0$, the matrix $A_{2}$ is positive semidefinite, and since $A$ and $A_{1}$ are idempotent, the eigenvalues are only $0$ and $1$, so they are also positive semidefinite according to the equivalence condition of semidefiniteness. Naturally, $\Gamma^{T} A \Gamma$, $\Gamma^{T} A_{1} \Gamma$, and $\Gamma^{T} A_{2} \Gamma$, which have orthogonal matrices multiplied on the front and back, are also positive semidefinite.

Properties of diagonal elements of positive definite matrices: Given a positive definite matrix $A = \left( a_{ij} \right) \in \mathbb{C}^{n \times n}$. The sign of diagonal elements $a_{ii}$ of $A$ is the same as the sign of $A$. Suppose a symmetric matrix $A \in \mathbb{R}^{n \times n}$ made up of real numbers is positive semidefinite.

When a positive semidefinite matrix is composed of real numbers and has symmetry, if any among the diagonal elements is $0$, all rows and columns of that element would be $0$. According to this, the following expression is possible for certain $G_{r} \in \mathbb{R}^{r \times r}$ and $H_{r} \in \mathbb{R}^{r \times r}$. $$ \begin{align*} \Gamma^{T} A \Gamma = & \Gamma^{T} A_{1} \Gamma + \Gamma^{T} A_{2} \Gamma \\ \implies \begin{bmatrix} I_{r} & O \\ O & O \end{bmatrix} =& \begin{bmatrix} G_{r} & O \\ O & O \end{bmatrix} + \begin{bmatrix} H_{r} & O \\ O & O \end{bmatrix} \end{align*} $$

Since $Q_{1} / \sigma^{2} \sim \chi^{2} \left( r_{1} \right)$, $A_{1}$ is also an idempotent matrix, and the following is obtained. $$ \left( \Gamma^{T} A_{1} \Gamma \right)^{2} = \Gamma^{T} A_{1} \Gamma = \begin{bmatrix} G_{r} & O \\ O & O \end{bmatrix} $$ Multiplying both sides of $\Gamma^{T} A \Gamma = \Gamma^{T} A_{1} \Gamma + \Gamma^{T} A_{2} \Gamma$ by $\Gamma^{T} A_{1} \Gamma$ results in the following: $$ \begin{align*} \begin{bmatrix} I_{r} & O \\ O & O \end{bmatrix} =& \begin{bmatrix} G_{r} & O \\ O & O \end{bmatrix} + \begin{bmatrix} H_{r} & O \\ O & O \end{bmatrix} \\ \implies \begin{bmatrix} I_{r} & O \\ O & O \end{bmatrix} \Gamma^{T} A_{1} \Gamma =& \Gamma^{T} A_{1} \Gamma \cdot \Gamma^{T} A_{1} \Gamma + \begin{bmatrix} H_{r} & O \\ O & O \end{bmatrix} \Gamma^{T} A_{1} \Gamma \\ \implies \begin{bmatrix} I_{r} & O \\ O & O \end{bmatrix} \begin{bmatrix} G_{r} & O \\ O & O \end{bmatrix} =& \Gamma^{T} A_{1} \Gamma + \begin{bmatrix} H_{r} & O \\ O & O \end{bmatrix} \begin{bmatrix} G_{r} & O \\ O & O \end{bmatrix} \\ \implies \begin{bmatrix} G_{r} & O \\ O & O \end{bmatrix} =& \begin{bmatrix} G_{r} & O \\ O & O \end{bmatrix} + \begin{bmatrix} G_{r} H_{r} & O \\ O & O \end{bmatrix} \\ \implies \begin{bmatrix} O & O \\ O & O \end{bmatrix} =& \begin{bmatrix} G_{r} H_{r} & O \\ O & O \end{bmatrix} \\ \implies G_{r} H_{r} =& O \\ \implies \Gamma^{T} A_{1} \Gamma \Gamma^{T} A_{2} \Gamma =& O \\ \implies A_{1} A_{2} =& O \end{align*} $$

Craig’s theorem: Let sample $\mathbf{X} = \left( X_{1} , \cdots , X_{n} \right)$ follow iid Normal distribution like $X_{1} , \cdots , X_{n} \overset{\text{iid}}{\sim} N \left( 0, \sigma^{2} \right)$. For a symmetric matrix $A, B \in \mathbb{R}^{n \times n}$, if random variables $Q_{1}$ and $Q_{2}$ are defined as random vector quadratic forms $Q_{1} := \sigma^{-2} \mathbf{X}^{T} A \mathbf{X}$ and $Q_{2} := \sigma^{-2} \mathbf{X}^{T} B \mathbf{X}$, the following holds. $$ Q_{1} \perp Q_{2} \iff A B = O_{n} $$

Addition of random variables: $X_i \sim \chi^2 ( r_{i} )$ then $$ \sum_{i=1}^{n} X_{i} \sim \chi ^2 \left( \sum_{i=1}^{n} r_{i} \right) $$

According to Craig’s theorem, $Q_{1}$ and $Q_{2}$ are independent, and $Q_{2}$ follows a Chi-squared distribution with degrees of freedom $\left( r - r_{1} \right)$.

---

It suffices to show that it holds for $k = 3$. Suppose $A_{3}$ is a positive semidefinite matrix satisfying the following: $$ A = A_{1} + \left( A_{2} + A_{3} \right) = A_{1} + B_{1} $$ By factoring $B_{1} := A_{2} + A_{3}$, $B_{1}$ is still a positive semidefinite matrix, and if we apply the result when $k = 2$ to $A = A_{1} + B_{1}$, we obtain $A_{1} B_{1} = O$ as follows: $$ \begin{align*} A = A^{2} =& \left( A_{1} + B_{1} \right)^{2} \\ =& A_{1}^{2} + A_{1} B_{1} + B_{1} A_{1} + B_{1}^{2} \\ =& A_{1} + O + B_{1}^{2} \\ \implies B_{1}^{2} =& A - A_{1} = B_{1} \end{align*} $$ On the other hand, applying the result when $k = 2$ to $B_{1} = A_{2} + A_{3}$ itself gives $A_{2} A_{3} = O$ and $A_{3}^{2} = A_{3}$. Repeating this factoring process for $B_{1}$ applied to $A = A_{2} + \left( A_{1} + A_{3} \right)$ gives $A_{1} A_{3} = O$, and continuing this completes the proof.

■