📂Mathematical Statistics

Proof of Student's Theorem

Theorem¹

If random variables $X_{1} , \cdots , X_{n}$ are iid following the normal distribution $N\left( \mu,\sigma^{2} \right)$, then

• (a): $$ \overline{X} \sim N\left( \mu , { {\sigma^2} \over {n} } \right) $$
• (b): $$ \overline{X} \perp S^2 $$
• (c): $$ (n-1) { {S^2} \over {\sigma^2} } \sim \chi^2 (n-1) $$
• (d): $$ T = { {\overline{X} - \mu } \over {S / \sqrt{n}} } \sim t(n-1) $$

---

• The sample mean $\overline{X}$ and the sample variance $S^{2}$ are random variables defined as follows.

$$ \overline{X} := {{ 1 } \over { n }} \sum_{k=1}^{n} X_{k} \\ S^{2} := {{ 1 } \over { n-1 }} \sum_{k=1}^{n} \left( X_{k} - \overline{X} \right)^{2} $$

Explanation

People who do statistics use this as if it were obvious, but in fact it too has a name. The fact that it is split into four parts, making it hard to cite specifically, has also played a part.

(b) is a fact that is obvious if you call it obvious and strange if you call it strange: even though both the sample mean and the sample variance come from exactly the same data, they are nevertheless independent.

Inference on the Population Mean for Small Samples

The proof of Student’s theorem is itself the derivation of the small-sample hypothesis test for the population mean.

Proof

(a)

Since $\displaystyle \overline{X} = { { (X_1 + X_2 + \cdots + X_n )} \over {n}}$, considering the sum of random variables following the normal distribution gives $$ \overline{X} \sim N \left( \mu, {{1} \over {n}} \sigma^2 \right) $$

■

(b)

• $\mathbf{0}$ denotes the zero vector.
• $\mathbf{1} = (1, \cdots , 1) = \begin{bmatrix} 1 \\ \vdots \\ 1 \end{bmatrix}$ denotes the vector all of whose components are $1$.
• $I$ denotes the identity matrix.
• $A^{T}$ denotes the transpose of the matrix $A$.

---

Let $\displaystyle \mathbf{v} := {{ 1 } \over { n }} \mathbf{1}$.

The random vector $X := \left( X_{1} , \cdots , X_{n} \right)$ follows the multivariate normal distribution, and $$ \begin{align*} \overline{X} =& {{ 1 } \over { n }} \left( X_{1} + \cdots + X_{n} \right) \\ &= {{ 1 } \over { n }} \begin{bmatrix} 1 & \cdots & 1 \end{bmatrix} \begin{bmatrix} X_{1} \\ \vdots \\ X_{n} \end{bmatrix} \\ =& {{ 1 } \over { n }} \mathbf{1}^{T} \mathbf{X} \\ =& \mathbf{v}^{T} \mathbf{X} \end{align*} $$

Now, defining the random vector $\mathbf{Y} := \left( X_{1} - \overline{X} , \cdots , X_{n} - \overline{X} \right)$, some random vector $\mathbf{W}$ can be represented as follows. $$ \mathbf{W} = \begin{bmatrix} \overline{X} \\ \mathbf{Y} \end{bmatrix} = \begin{bmatrix} \mathbf{v}^{T} \\ I - \mathbf{1} \mathbf{v}^{T} \end{bmatrix} \mathbf{X} $$

Since $\mathbf{W}$ is a linear transformation of random vectors following the multivariate normal distribution, it still follows the multivariate normal distribution, and its population mean vector, obtained by taking the expectation of the equation above, is $$ E \mathbf{W} = \begin{bmatrix} \mathbf{v}^{T} \\ I - \mathbf{1} \mathbf{v}^{T} \end{bmatrix} \mu \mathbf{1} = \begin{bmatrix} \mu \\ \mathbf{0}_{n} \end{bmatrix} $$ and the covariance matrix $\Sigma$ is $$ \begin{align*} \Sigma =& \begin{bmatrix} \mathbf{v}^{T} \\ I - \mathbf{1} \mathbf{v}^{T} \end{bmatrix} \sigma^{2} I \begin{bmatrix} \mathbf{v}^{T} \\ I - \mathbf{1} \mathbf{v}^{T} \end{bmatrix}^{T} \\ =& \sigma^{2} \begin{bmatrix} 1/n & \mathbf{0}_{n}^{T} \\ \mathbf{0}_{n} & I - \mathbf{1} \mathbf{v}^{T} \end{bmatrix} \end{align*} $$ From this we can see that $\overline{X}$ is independent of $\mathbf{Y}$, and since $$ S^{2} = {{ 1 } \over { n-1 }} \sum_{k=1}^{n} \left( X_{k} - \overline{X} \right)^{2} = {{ 1 } \over { n-1 }} \mathbf{Y}^{T} \mathbf{Y} $$ we have $\overline{X} \perp S^{2}$.

■

(c)

If we let $\displaystyle V = \sum_{i=1}^{n} \left( { {X_{i} - \mu } \over {\sigma} } \right) ^2 $, then since $\displaystyle { {X_{i} - \mu } \over {\sigma} } \sim N(0,1)$, it must be that $V \sim \chi^2 (n)$, and

$$ \begin{align*} V =& \sum_{i=1}^{n} \left( { {X_{i} - \mu } \over {\sigma} } \right) ^2 \\ =& \sum_{i=1}^{n} \left( { { ( X_{i} -\overline{X} ) + ( \overline{X} - \mu ) } \over {\sigma} } \right) ^2 \\ =& \sum_{i=1}^{n} \left( { { X_{i} -\overline{X} } \over {\sigma} } \right) ^2 + \left( { { \overline{X} - \mu } \over {\sigma / \sqrt{n} } } \right) ^2 \end{align*} $$

Here, $$ \sum_{i=1}^{n} \left( { { X_{i} -\overline{X} } \over {\sigma} } \right) ^2 = { {n-1} \over {\sigma^2} } \sum_{i=1}^{n} { { ( X_{i} -\overline{X} ) ^ 2 } \over {n-1} } = (n-1) { {S^2} \over {\sigma^2} } $$

Rearranging gives $$ V = (n-1) { {S^2} \over {\sigma^2} } + \left( { { \overline{X} - \mu } \over {\sigma / \sqrt{n} } } \right) ^2 $$

Since $V \sim \chi^2 (n)$, by (a) of Student’s theorem, $$ \left( { { \overline{X} - \mu } \over {\sigma / \sqrt{n} } } \right) \sim N(0,1) $$ and since the square of a standard normal distribution follows the chi-squared distribution, $$ \left( { { \overline{X} - \mu } \over {\sigma / \sqrt{n} } } \right)^2 \sim \chi^2 (1) $$

Since (b) of Student’s theorem showed that $\overline{X}$ and $S^2$ are independent of each other, arranging both sides into the form of moment generating functions gives $$ (1-2t)^{-n/2} = E \left\{ \exp \left( (n-1) { {S^2} \over {\sigma^2} } t \right) \right\} (1-2t)^{-1/2} $$

Therefore, the moment generating function of $\displaystyle (n-1) { {S^2} \over {\sigma^2} }$ is $(1-2t)^{-(n-1)/2}$.

■

(d)

Derivation of Student’s t-Distribution from the Normal and Chi-Squared Distributions: If $W \sim N(0,1)$ and $V \sim \chi^2 (r)$, then $$ T = { {W} \over {\sqrt{V/r} } } \sim t(r) $$

$$ T = { {\overline{X} - \mu } \over {S / \sqrt{n}} } = { {( \overline{X} - \mu ) / (\sigma / \sqrt{n}) } \over { \sqrt{ (n-1) S^2 / ( \sigma^2 ( n-1 ) ) } } } $$ Since (a) of Student’s theorem showed that $\displaystyle \overline{X} \sim N\left( \mu , { {\sigma^2} \over {n} } \right) $ and (c) showed that $\displaystyle (n-1) { {S^2} \over {\sigma^2} } \sim \chi^2 (n-1) $, $$ T \sim t(n-1) $$

■