Conditional Expectation Minimizes the Sum of Squared Deviations
Summary
The following holds true:
$$ \begin{equation} E\left[ Y | X \right] = \argmin_{f(X)} E\left[ (Y - f(X))^{2} | X \right] \end{equation} $$
$$ \begin{equation} E\left[ Y | X \right] = \argmin_{f(X)} E\left[ (Y - f(X))^{2} \right] \end{equation} $$
Proof
(1)
$$ \begin{align*} & \argmin_{f(X)} E\left[ (Y - f(X))^{2} | X \right] \\ &= \argmin_{f(X)} E\left[ Y^{2} - 2Yf(X) + f(X)^{2} | X \right] \\ &= \argmin_{f(X)} \left( E\left[ Y^{2} | X \right] - 2E\left[ Yf(X) | X \right] + E\left[ f(X)^{2} | X \right] \right) \\ &= \argmin_{f(X)} \left( - 2E\left[ Yf(X) | X \right] + E\left[ f(X)^{2} | X \right] \right) \\ \end{align*} $$
The fourth equality holds because the first term is independent of $f(X)$.
The smoothing property of conditional expectation $(\text{Pulling out known factors})$
[a] $E[f(X)Y | X] = f(X) E[Y | X]$
By the properties of conditional expectation, the equation can be rearranged as follows.
$$ \begin{align*} & \argmin_{f(X)} \left( - 2E\left[ Yf(X) | X \right] + E\left[ f(X)^{2} | X \right] \right) \\ &\overset{[a]}{=} \argmin_{f(X)} \left( - 2f(X) E\left[ Y| X \right] + f(X)^{2}E\left[ 1 | X \right] \right) \\ &= \argmin_{f(X)} \left( - 2f(X) E\left[ Y| X \right] + f(X)^{2} \right) \\ \end{align*} $$
Differentiating to find the minimum value of $f(X)$, we get
$$ \dfrac{d (- 2f(X) E\left[ Y| X \right] + f(X)^{2})}{d f(X)} = -2E\left[ Y | X \right] + 2f(X) = 0 $$ $$ \implies f(X) = E[Y | X] $$
■
(2)1
Subtract and add $E\left[ Y | X \right]$, then wrap it front and back.
$$ \begin{align*} \argmin_{f(X)} E\left[ (Y - f(X))^{2} \right] &= \argmin_{f(X)} E\left[ (Y - E\left[ Y | X \right] + E\left[ Y | X \right] - f(X))^{2} \right] \\ &= \argmin_{f(X)} E\left[ \Big( (Y - E\left[ Y | X \right]) + (E\left[ Y | X \right] - f(X)) \Big)^{2} \right] \end{align*} $$
Since this forms a perfect square identical to $(a + b)^{2} = a^{2} + b^{2} + 2ab$, by the linearity of expectation, we get
$$ \begin{align*} & \argmin_{f(X)} E\left[ (Y - f(X))^{2} \right] \\ &= \argmin_{f(X)} E\left[ \Big( (Y - E\left[ Y | X \right]) + (E\left[ Y | X \right] - f(X)) \Big)^{2} \right] \\ &= \argmin_{f(X)} \Big( E\left[ (Y - E\left[ Y | X \right])^{2} \right] + E\left[ (E\left[ Y | X \right] - f(X))^{2} \right] \\ &\qquad\qquad\qquad + 2 E\Big[ (Y - E\left[ Y | X \right])(E\left[ Y | X \right] - f(X)) \Big] \Big) \end{align*} $$
The first term is independent of $f(X)$, so it can be rearranged as follows.
$$ \begin{align} & \argmin_{f(X)} E\left[ (Y - f(X))^{2} \right] \nonumber \\ &= \argmin_{f(X)} \Big( E\left[ (E\left[ Y | X \right] - f(X))^{2} \right] - 2 E\Big[ (Y - E\left[ Y | X \right])(E\left[ Y | X \right] - f(X)) \Big] \Big) \end{align} $$
Let’s compute the second term.
The smoothing property of conditional expectation $(\text{Pulling out known factors})$
[a] $E[f(X)Y | X] = f(X) E[Y | X]$
The property of conditional expectation $(\text{Law of total expectation})$
[b] $E_{X}\left[ E_{Y}\left[ Y | X = x \right] \right] = E_{Y}\left[ Y \right]$
Applying the above properties to the second term of $(3)$, we get
$$ \begin{align*} & E\Big[ (Y - E\left[ Y | X \right])(E\left[ Y | X \right] - f(X)) \Big] \\ &\overset{[b]}{=} E \left[ E\Big[ (Y - E\left[ Y | X \right])(E\left[ Y | X \right] - f(X)) \Big| X \Big] \right] \\ &\overset{[a]}{=} E \left[ (E\left[ Y | X \right] - f(X)) E\Big[ (Y - E\left[ Y | X \right]) \Big| X \Big] \right] \\ &\overset{\text{linearity}}{=} E \left[ (E\left[ Y | X \right] - f(X)) \left( E[ Y | X ] - E \left[ E\left[ Y | X \right] \Big| X \right] \right) \right] \end{align*} $$
From the formula above, the last term is calculated as $E \left[ E\left[ Y | X \right] \Big| X \right] \overset{[a]}{=} E\left[ Y | X \right] E \left[ 1 | X \right] = E\left[ Y | X \right] $,
$$ \begin{align*} & E\Big[ (Y - E\left[ Y | X \right])(E\left[ Y | X \right] - f(X)) \Big] \\ &= E \Big[ (E\left[ Y | X \right] - f(X)) \left( E[ Y | X ] - E\left[ Y | X \right] \right) \Big] \\ &= E \Big[ (E\left[ Y | X \right] - f(X)) \cdot 0 \Big] \\ &= 0 \end{align*} $$
Therefore, the second term of $(3)$ is $0$ and it is rearranged as follows.
$$ \argmin_{f(X)} E\Big[ (E\left[ Y | X \right] - f(X))^{2} \Big] $$
Therefore,
$$ \argmin_{f(X)} E\left[ (Y - f(X))^{2} \right] = \argmin_{f(X)} E\Big[ (E\left[ Y | X \right] - f(X))^{2} \Big] = E\left[ Y | X \right] $$
■