📂Mathematical Statistics

The Variance of an Unbiased Estimator Given a Sufficient Statistic is Minimized

Theorem ¹

Let’s say we have a parameter $\theta$. $U$ is an unbiased estimator, $T_{1}$ is a sufficient statistic, and $T_{2}$ is a minimal sufficient statistic, defined as follows: $$\begin{align*} U_{1} :=& E \left( U | T_{1} \right) \\ U_{2} :=& E \left( U | T_{2} \right) \end{align*}$$ it holds that: $$\operatorname{Var} U_{2} \le \operatorname{Var} U_{1}$$

Explanation

Whether $T_{1}$ or $T_{2}$ is given, $U$ being an unbiased estimator means it hits $\theta$ in expectation, but roughly speaking, it does so with less fluctuation when the minimal sufficient statistic is given. It’s easy to remember that the minimality of the sufficient statistic leads to the minimality of the variance of the unbiased estimator.

Proof

Definition of the Minimal Sufficient Statistic: A sufficient statistic $T \left( \mathbf{X} \right)$ is called a minimal sufficient statistic if it can be represented as a function of every other sufficient statistic $T ' \left( \mathbf{X} \right)$, denoted by $T \left( \mathbf{x} \right)$ being a function of $T ' \left( \mathbf{x} \right)$.

According to the definition of the minimal sufficient statistic, since $T_{2}$ can be represented as a function of $T_{1}$, $$\begin{align*} E \left( U_{1} | T_{2} \right) =& E \left( E \left( U | T_{1} \right) | T_{2} \right) \\ =& E \left( U | T_{2} \right) \\ =& U_{2} \end{align*}$$

Property of Conditional Variance: $$\operatorname{Var}(X) = E \left( \operatorname{Var}(X | Y) \right) + \operatorname{Var}(E(X | Y))$$

Following the property of conditional variance, for $U_{1}$ and $T_{2}$ we have

$$\begin{align*} \operatorname{Var} U_{1} =& E \operatorname{Var} \left( U_{1} | T_{2} \right) + \operatorname{Var} E \left( U_{1} | T_{2} \right) \\ =& E \operatorname{Var} \left( U_{1} | T_{2} \right) + \operatorname{Var} U_{2} \end{align*}$$

This holds for any other sufficient statistic $T_{1}$, so the variance of the expected value of the unbiased estimator $U$ given the minimal sufficient statistic $T_{2}$ is minimized.

■