Proof of the Mathematical Properties of Representative Values
Theorem
Suppose we are given data $X = \left\{ x_{1} , \cdots , x_{n} \right\}$.
- [0]: The $\theta$ that minimizes $\displaystyle h(\theta)=\sum_{i=1}^{n} {|x_i - \theta|}^{0}$ is $$ \argmin_{\theta} h \left( \theta \right) = \text{mode}(X) $$
- [1]: The $\theta$ that minimizes $\displaystyle h(\theta)=\sum_{i=1}^{n} {|x_i - \theta|}^{1}$ is $$ \argmin_{\theta} h \left( \theta \right) = \text{median}(X) $$
- [2]: The $\theta$ that minimizes $\displaystyle h(\theta)=\sum_{i=1}^{n} {|x_i - \theta|}^{2}$ is $$ \argmin_{\theta} h \left( \theta \right) = \text{mean}(X) $$
Explanation
Put in the difficult terminology of linear algebra, it reads as follows:
- [0]: What minimizes the $l^{0}$-norm is the mode.
- [1]: What minimizes the $l^{1}$-norm is the median.
- [2]: What minimizes the $l^{2}$-norm is the mean.
The theorems above, in themselves, provide the mathematical grounds for why representative values deserve to be called representative values. In particular, since [2] implies that the representative value minimizing the variance is the mean, it may also serve as an answer to the long-standing question of ‘why is variance defined this way’.
Proof
Mode
Strategy: The $l^{0}$-norm is a norm that counts not the degree of inequality but the number of unequal entries.
$$ \left| x_{i} - \theta \right|^{0} := \begin{cases} 1 & , \theta \ne x_{i} \\ 0 & , \theta = x_{i} \end{cases} $$ Therefore, the $\theta$ that minimizes $\displaystyle h(\theta)=\sum_{i=1}^{n} {|x_i - \theta|}^{0} = 1 + 0 + 1 + \cdots 1+ 1$ is $\text{mode}(X)$
■
Median
Strategy: Following the definition of absolute value, first unpack it so that the computation becomes easy. Pair the largest and smallest terms of the data one by one to eliminate the unknown and turn them into constant terms. Then it is simple to make only the remaining term in the unknown as small as possible.
Let $ x_{(1)} \le x_{(2)} \le \cdots \le x_{(n)}$.
Part 1. $\theta \in [x_{(1)} , x_{(n)} ]$ If we assume $\theta < x_{(1)}$, then $\theta$ is smaller than every $x_{(i)}$, so $$ h(\theta)=\sum_{i=1}^{n} {\left( x_{(i)} - \theta \right) } > \sum_{i=1}^{n} { \left( x_{(i)} - x_{(1)} \right) } $$ If we assume $ x_{(n)} < \theta$, then $\theta$ is larger than every $x_{(i)}$, so $$ h(\theta)=\sum_{i=1}^{n} { \left( \theta - x_{(i)} \right) } > \sum_{i=1}^{n} { \left( x_{(n)} - x_{(i)} \right) } $$ Therefore, whatever $\theta$ may specifically be, it must at least satisfy $\theta \in [x_{(1)} , x_{(n)} ]$.
Part 2.
For $\theta_{0} \in [x_{(1)} , x_{(n)} ]$, $$ \begin{align*} h(\theta_{0}) =& \sum_{i=1}^{n} | x_{(i)} - \theta_{0} | \\ =& \sum_{i=2}^{n-1} | x_{(i)} - \theta_{0} | + ( \theta_{0} - x_{(1)} ) + ( x_{(n)} - \theta_{0} ) \\ =& \sum_{i=2}^{n-1} | x_{(i)} - \theta_{0} | + ( x_{(n)} - x_{(1)} ) \end{align*} $$
For $\theta_{1} \in [x_{(2)} , x_{(n-1)} ] \subset [x_{(1)} , x_{(n)} ]$, $$ \begin{align*} h(\theta_{1}) =& \sum_{i=1}^{n} | x_{(i)} - \theta_{1} | \\ =& \sum_{i=2}^{n-1} | x_{(i)} - \theta_{1} | + ( x_{(n)} - x_{(1)} ) \\ =& \sum_{i=3}^{n-2} | x_{(i)} - \theta_{1} | + ( x_{(n-1)} - x_{(2)} ) + ( x_{(n)} - x_{(1)} ) \end{align*} $$
In this way, each time we choose a suitable $\theta_{k} \in [x_{(1+k)} , x_{(n-k)} ]$, the term $( x_{(n-k)} - x_{(1+k)} )$ can be pulled out of the sigma. Since the data $X$ is fixed, these terms are constant terms, and for convenience let us denote their sum as follows. $$ C_{k} : = \sum_{j=0}^{k} \left( x_{(n-j)} - x_{(j+1)} \right) $$
Part 3.
Case 3-1. $ n$ is odd
- By Part 2., $$ \begin{align*} h ( \theta ) =& \sum_{i=1}^{n} | x_{(i)} - \theta | \\ =& \sum_{i=1+k}^{n-k} | x_{(i)} - \theta | + C_{k} \\ =& \left| x_{\left( {{n+1} \over {2}} \right)} - \theta \right| + C_{{{n-1} \over {2}} - 1} \end{align*} $$ Therefore, the value that makes $h( \theta )$ smallest is $\theta = x_{\left( {{n+1} \over {2}} \right)}$.
Case 3-2. $ n$ is even
- By Part 2.,
$$
\begin{align*}
h ( \theta ) =& \sum_{i=1}^{n} | x_{(i)} - \theta |
\\ =& \sum_{i=1+k}^{n-k} | x_{(i)} - \theta | + C_{k}
\\ =& \left| x_{\left( {{n} \over {2}} \right)} - \theta \right| + \left| x_{\left( {{n} \over {2}} + 1 \right)} - \theta \right| + C_{{{n} \over {2}} - 2}
\end{align*}
$$
In this case, every $\displaystyle \theta \in \left[ x_{ \left( {{n} \over {2}} \right)} , x_{ \left( {{n} \over {2}} + 1 \right)} \right]$ makes $h ( \theta )$ smallest.
In the end, whether $n$ is even or odd, the $\theta$ that makes $h ( \theta)$ smallest is the median of $X$.
■
Mean
Strategy: It can be derived simply by differentiation.
$$ {{ d } \over { d \theta }} \sum_{i=1}^{n} \left( x_{i} - \theta \right) = \sum_{i=1}^{n} 2 \left( x_{i} - \theta \right) = 0 $$ Since the $\theta$ satisfying the above equation minimizes $\displaystyle h(\theta)=\sum_{i=1}^{n} {|x_i - \theta|}^{2}$, $$ \displaystyle\sum_{i=1}^{n} 2 \left( x_{i} - \theta \right) = 0 \implies \sum_{i=1}^{n} x_{i} = n \theta \implies \theta = {{ 1 } \over { n }} \sum_{i=1}^{n} x_{i} $$
■