logo

Conditional Expectation Minimizes the Sum of Squared Deviations 📂Mathematical Statistics

Conditional Expectation Minimizes the Sum of Squared Deviations

Summary

The following holds true:

E[Y∣X]=arg min⁡f(X)E[(Y−f(X))2∣X] \begin{equation} E\left[ Y | X \right] = \argmin_{f(X)} E\left[ (Y - f(X))^{2} | X \right] \end{equation}

E[Y∣X]=arg min⁡f(X)E[(Y−f(X))2] \begin{equation} E\left[ Y | X \right] = \argmin_{f(X)} E\left[ (Y - f(X))^{2} \right] \end{equation}

Proof

(1)

arg min⁡f(X)E[(Y−f(X))2∣X]=arg min⁡f(X)E[Y2−2Yf(X)+f(X)2∣X]=arg min⁡f(X)(E[Y2∣X]−2E[Yf(X)∣X]+E[f(X)2∣X])=arg min⁡f(X)(−2E[Yf(X)∣X]+E[f(X)2∣X]) \begin{align*} & \argmin_{f(X)} E\left[ (Y - f(X))^{2} | X \right] \\ &= \argmin_{f(X)} E\left[ Y^{2} - 2Yf(X) + f(X)^{2} | X \right] \\ &= \argmin_{f(X)} \left( E\left[ Y^{2} | X \right] - 2E\left[ Yf(X) | X \right] + E\left[ f(X)^{2} | X \right] \right) \\ &= \argmin_{f(X)} \left( - 2E\left[ Yf(X) | X \right] + E\left[ f(X)^{2} | X \right] \right) \\ \end{align*}

The fourth equality holds because the first term is independent of f(X)f(X).

The smoothing property of conditional expectation (Pulling out known factors)(\text{Pulling out known factors})

[a] E[f(X)Y∣X]=f(X)E[Y∣X]E[f(X)Y | X] = f(X) E[Y | X]

By the properties of conditional expectation, the equation can be rearranged as follows.

arg min⁡f(X)(−2E[Yf(X)∣X]+E[f(X)2∣X])=[a]arg min⁡f(X)(−2f(X)E[Y∣X]+f(X)2E[1∣X])=arg min⁡f(X)(−2f(X)E[Y∣X]+f(X)2) \begin{align*} & \argmin_{f(X)} \left( - 2E\left[ Yf(X) | X \right] + E\left[ f(X)^{2} | X \right] \right) \\ &\overset{[a]}{=} \argmin_{f(X)} \left( - 2f(X) E\left[ Y| X \right] + f(X)^{2}E\left[ 1 | X \right] \right) \\ &= \argmin_{f(X)} \left( - 2f(X) E\left[ Y| X \right] + f(X)^{2} \right) \\ \end{align*}

Differentiating to find the minimum value of f(X)f(X), we get

d(−2f(X)E[Y∣X]+f(X)2)df(X)=−2E[Y∣X]+2f(X)=0 \dfrac{d (- 2f(X) E\left[ Y| X \right] + f(X)^{2})}{d f(X)} = -2E\left[ Y | X \right] + 2f(X) = 0   ⟹  f(X)=E[Y∣X] \implies f(X) = E[Y | X]

■

(2)1

Subtract and add E[Y∣X]E\left[ Y | X \right], then wrap it front and back.

arg min⁡f(X)E[(Y−f(X))2]=arg min⁡f(X)E[(Y−E[Y∣X]+E[Y∣X]−f(X))2]=arg min⁡f(X)E[((Y−E[Y∣X])+(E[Y∣X]−f(X)))2] \begin{align*} \argmin_{f(X)} E\left[ (Y - f(X))^{2} \right] &= \argmin_{f(X)} E\left[ (Y - E\left[ Y | X \right] + E\left[ Y | X \right] - f(X))^{2} \right] \\ &= \argmin_{f(X)} E\left[ \Big( (Y - E\left[ Y | X \right]) + (E\left[ Y | X \right] - f(X)) \Big)^{2} \right] \end{align*}

Since this forms a perfect square identical to (a+b)2=a2+b2+2ab(a + b)^{2} = a^{2} + b^{2} + 2ab, by the linearity of expectation, we get

arg min⁡f(X)E[(Y−f(X))2]=arg min⁡f(X)E[((Y−E[Y∣X])+(E[Y∣X]−f(X)))2]=arg min⁡f(X)(E[(Y−E[Y∣X])2]+E[(E[Y∣X]−f(X))2]+2E[(Y−E[Y∣X])(E[Y∣X]−f(X))]) \begin{align*} & \argmin_{f(X)} E\left[ (Y - f(X))^{2} \right] \\ &= \argmin_{f(X)} E\left[ \Big( (Y - E\left[ Y | X \right]) + (E\left[ Y | X \right] - f(X)) \Big)^{2} \right] \\ &= \argmin_{f(X)} \Big( E\left[ (Y - E\left[ Y | X \right])^{2} \right] + E\left[ (E\left[ Y | X \right] - f(X))^{2} \right] \\ &\qquad\qquad\qquad + 2 E\Big[ (Y - E\left[ Y | X \right])(E\left[ Y | X \right] - f(X)) \Big] \Big) \end{align*}

The first term is independent of f(X)f(X), so it can be rearranged as follows.

arg min⁡f(X)E[(Y−f(X))2]=arg min⁡f(X)(E[(E[Y∣X]−f(X))2]−2E[(Y−E[Y∣X])(E[Y∣X]−f(X))]) \begin{align} & \argmin_{f(X)} E\left[ (Y - f(X))^{2} \right] \nonumber \\ &= \argmin_{f(X)} \Big( E\left[ (E\left[ Y | X \right] - f(X))^{2} \right] - 2 E\Big[ (Y - E\left[ Y | X \right])(E\left[ Y | X \right] - f(X)) \Big] \Big) \end{align}

Let’s compute the second term.

The smoothing property of conditional expectation (Pulling out known factors)(\text{Pulling out known factors})

[a] E[f(X)Y∣X]=f(X)E[Y∣X]E[f(X)Y | X] = f(X) E[Y | X]

The property of conditional expectation (Law of total expectation)(\text{Law of total expectation})

[b] EX[EY[Y∣X=x]]=EY[Y]E_{X}\left[ E_{Y}\left[ Y | X = x \right] \right] = E_{Y}\left[ Y \right]

Applying the above properties to the second term of (3)(3), we get

E[(Y−E[Y∣X])(E[Y∣X]−f(X))]=[b]E[E[(Y−E[Y∣X])(E[Y∣X]−f(X))∣X]]=[a]E[(E[Y∣X]−f(X))E[(Y−E[Y∣X])∣X]]=linearityE[(E[Y∣X]−f(X))(E[Y∣X]−E[E[Y∣X]∣X])] \begin{align*} & E\Big[ (Y - E\left[ Y | X \right])(E\left[ Y | X \right] - f(X)) \Big] \\ &\overset{[b]}{=} E \left[ E\Big[ (Y - E\left[ Y | X \right])(E\left[ Y | X \right] - f(X)) \Big| X \Big] \right] \\ &\overset{[a]}{=} E \left[ (E\left[ Y | X \right] - f(X)) E\Big[ (Y - E\left[ Y | X \right]) \Big| X \Big] \right] \\ &\overset{\text{linearity}}{=} E \left[ (E\left[ Y | X \right] - f(X)) \left( E[ Y | X ] - E \left[ E\left[ Y | X \right] \Big| X \right] \right) \right] \end{align*}

From the formula above, the last term is calculated as E[E[Y∣X]∣X]=[a]E[Y∣X]E[1∣X]=E[Y∣X]E \left[ E\left[ Y | X \right] \Big| X \right] \overset{[a]}{=} E\left[ Y | X \right] E \left[ 1 | X \right] = E\left[ Y | X \right] ,

E[(Y−E[Y∣X])(E[Y∣X]−f(X))]=E[(E[Y∣X]−f(X))(E[Y∣X]−E[Y∣X])]=E[(E[Y∣X]−f(X))⋅0]=0 \begin{align*} & E\Big[ (Y - E\left[ Y | X \right])(E\left[ Y | X \right] - f(X)) \Big] \\ &= E \Big[ (E\left[ Y | X \right] - f(X)) \left( E[ Y | X ] - E\left[ Y | X \right] \right) \Big] \\ &= E \Big[ (E\left[ Y | X \right] - f(X)) \cdot 0 \Big] \\ &= 0 \end{align*}

Therefore, the second term of (3)(3) is 00 and it is rearranged as follows.

arg min⁡f(X)E[(E[Y∣X]−f(X))2] \argmin_{f(X)} E\Big[ (E\left[ Y | X \right] - f(X))^{2} \Big]

Therefore,

arg min⁡f(X)E[(Y−f(X))2]=arg min⁡f(X)E[(E[Y∣X]−f(X))2]=E[Y∣X] \argmin_{f(X)} E\left[ (Y - f(X))^{2} \right] = \argmin_{f(X)} E\Big[ (E\left[ Y | X \right] - f(X))^{2} \Big] = E\left[ Y | X \right]

■

  1. https://gregorygundersen.com/blog/2019/10/04/expectation-median-opt/ ↩︎