📂Mathematical Statistics

Proof of the Central Limit Theorem

Theorem ¹

If $\left\{ X_{k} \right\}_{k=1}^{n}$ are iid random variables that follow the probability distribution $\left( \mu, \sigma^2 \right) $, then as $n \to \infty$ $$ \sqrt{n} {{ \overline{X}_n - \mu } \over {\sigma}} \overset{D}{\to} N (0,1) $$

---

• $\overset{D}{\to}$ means convergence in distribution.

Explanation

Along with the law of large numbers, it is counted as a theorem of truly great renown in statistics. It is a theorem we hear about and use countless times, yet its proof is something we may or may not attempt just once while studying mathematical statistics. In reality, however, apart from its usefulness, the proof itself is fun, which makes it an all the more valuable theorem.

Proof

Strategy: We use a trick employing the moment generating function and Taylor’s theorem.

---

First, we must assume that the moment generating function $M(t) = E(e^{t Y}), -h<t<h$ of $\displaystyle Y := \sqrt{n} {{ \overline{X}_{n} - \mu } \over { \sigma }}$ exists. Here, if we define a new function $m(t) := E[e^{t(X-\mu)}] = e^{-\mu t} M(t)$, then $$ \begin{align*} M(t) =& E \left( e^{ t \sqrt{n} {{ \overline{X}_n - \mu } \over {\sigma}} } \right) \\ =& E \left( e^{ t {{ \sum_{i=1}^{n} X_i - n \mu } \over {\sigma \sqrt{n} }} } \right) \\ =& E \left( e^{ t {{ X_1 - \mu } \over {\sigma \sqrt{n} }} } \right) E \left( e^{ t {{ X_2 - \mu } \over {\sigma \sqrt{n} }} } \right) \cdots E \left( e^{ t {{ X_n - \mu } \over {\sigma \sqrt{n} }} } \right) \\ =& E \left( e^{ t {{ X - \mu } \over {\sigma \sqrt{n} }} } \right) E \left( e^{ t {{ X - \mu } \over {\sigma \sqrt{n} }} } \right) \cdots E \left( e^{ t {{ X - \mu } \over {\sigma \sqrt{n} }} } \right) \\ =& { \left\{ E \left( e^{ t {{ X - \mu } \over {\sigma \sqrt{n} }} } \right) \right\} }^n \\ =& { \left\{ m \left( { {t} \over {\sigma \sqrt{n} } } \right) \right\} } ^{n} \qquad , -h < { {t} \over {\sigma \sqrt{n} } } < h \end{align*} $$

Taylor’s theorem: If a function $f(x)$ is continuous on $[a,b]$ and $n$-times differentiable on $(a,b)$, then for $x_{0} \in (a,b)$ there exists $\xi \in (a,b)$ satisfying $\displaystyle f(x) = \sum_{k=0}^{n-1} {{( x - x_{0} )^{k}\over{ k! }}{f^{(k)}( x_{0} )}} + {(x - x_{0} )^{n}\over{ n! }}{f^{(n)}(\xi)}$.

Applying Taylor’s theorem for $n=2$, there exists at least one $\xi$ in $(-t,0)$ or $(0,t)$ satisfying the following. Therefore $m(t)$ can be expressed as $$ m(t) = m(0) + m ' (0)t + { {m '' (\xi) t^2} \over {2} } $$ Meanwhile, since $$ \begin{cases} m(0)=1 \\ m ' (0) = E(X-\mu) = 0 \\ m '' (0) = E[(X-\mu)^2] = {\sigma}^2 \end{cases} $$ we have $\displaystyle m(t) = 1 + { {m '' (\xi) t^2} \over {2} }$. Here comes the trick: adding and then subtracting $\displaystyle {{\sigma^2 t^2} \over {2}}$ on the right-hand side gives $$ m(t) = 1 + { { \sigma^2 t^2} \over {2} } + { { [ m '' (\xi) - \sigma^2 ] t^2} \over {2} } $$ In other words, $$ M(t) = { \left\{ m \left( { {t} \over {\sigma \sqrt{n} } } \right) \right\} } ^{n} = { \left\{ 1 + { { t^2} \over {2n} } + { { [ m '' (\xi) - \sigma^2 ] t^2} \over {2n \sigma^2 } } \right\} } ^{n} $$

By Taylor’s theorem, $\xi$ lies between $\displaystyle \left( -{ {t} \over {\sigma \sqrt{n} } },0 \right)$ or $\displaystyle \left( 0,{ {t} \over {\sigma \sqrt{n} } } \right) $, so as $n \to \infty$ we have $\xi \to 0$, and therefore $ m '' (\xi) \to m '' (0) = \sigma^2$. Removing the terms that thus converge to $0$,

$$ \lim _{n \to \infty} M(t) = \lim _{n \to \infty} \left( 1 + { { t^2} \over {2n} } \right)^{n} = e^{t^2 / 2} $$

and since $e^{t^2 / 2}$ is the moment generating function of the standard normal distribution,

$$ \sqrt{n} {{ \overline{X}_n - \mu } \over {\sigma}} \overset{D}{\to} N (0,1) $$

■