Conditional Mean and Variance of the Multivariate Normal Distribution
📂Probability Distribution Conditional Mean and Variance of the Multivariate Normal Distribution Bivariate Normal Distribution ( X , Y ) ∼ N 2 ( [ μ 1 μ n ] , [ σ X 2 ρ σ X σ Y ρ σ X σ Y σ Y 2 ] )
\left( X, Y \right) \sim N_{2} \left( \begin{bmatrix} \mu_{1} \\ \mu_{n} \end{bmatrix} , \begin{bmatrix} \sigma_{X}^{2} & \rho \sigma_{X} \sigma_{Y} \\ \rho \sigma_{X} \sigma_{Y} & \sigma_{Y}^{2} \end{bmatrix} \right)
( X , Y ) ∼ N 2 ( [ μ 1 μ n ] , [ σ X 2 ρ σ X σ Y ρ σ X σ Y σ Y 2 ] )
When a random vector ( X , Y ) \left( X,Y \right) ( X , Y ) bivariate normal distribution as above, X ∣ Y X | Y X ∣ Y univariate normal distribution and the conditional mean and variance are as follows.
E ( X ∣ Y ) = μ X + ρ σ X σ Y ( Y − μ Y ) Var ( X ∣ Y ) = ( 1 − ρ 2 ) σ X 2
\begin{align*}
E \left( X | Y \right) =& \mu_{X} + \rho {{ \sigma_{X} } \over { \sigma_{Y} }} \left( Y - \mu_{Y} \right)
\\ \Var \left( X | Y \right) =& \left( 1 - \rho^{2} \right) \sigma_{X}^{2}
\end{align*}
E ( X ∣ Y ) = Var ( X ∣ Y ) = μ X + ρ σ Y σ X ( Y − μ Y ) ( 1 − ρ 2 ) σ X 2
Multivariate Normal Distribution X = [ X 1 X 2 ] : Ω → R n μ = [ μ 1 μ 2 ] ∈ R n Σ = [ Σ 11 Σ 12 Σ 21 Σ 22 ] ∈ R n × n
\begin{align*}
\mathbf{X} =& \begin{bmatrix} \mathbf{X}_{1} \\ \mathbf{X}_{2} \end{bmatrix} & : \Omega \to \mathbb{R}^{n}
\\ \mu =& \begin{bmatrix} \mu_{1} \\ \mu_{2} \end{bmatrix} & \in \mathbb{R}^{n}
\\ \Sigma =& \begin{bmatrix} \Sigma_{11} & \Sigma_{12} \\ \Sigma_{21} & \Sigma_{22} \end{bmatrix} & \in \mathbb{R}^{n \times n}
\end{align*}
X = μ = Σ = [ X 1 X 2 ] [ μ 1 μ 2 ] [ Σ 11 Σ 21 Σ 12 Σ 22 ] : Ω → R n ∈ R n ∈ R n × n random vector X ∼ N n ( μ , Σ ) \mathbf{X} \sim N_{n} \left( \mu , \Sigma \right) X ∼ N n ( μ , Σ ) multivariate normal distribution represented in Jordan block form as X \mathbf{X} X μ \mu μ Σ \Sigma Σ conditional probability vector X 1 ∣ X 2 : Ω → R m \mathbf{X}_{1} | \mathbf{X}_{2} : \Omega \to \mathbb{R}^{m} X 1 ∣ X 2 : Ω → R m mean vector and covariance matrix .
X 1 ∣ X 2 ∼ N m ( μ 1 + Σ 12 Σ 22 − 1 ( X 2 − μ 2 ) , Σ 11 − Σ 12 Σ 22 − 1 Σ 21 )
\mathbf{X}_{1} | \mathbf{X}_{2} \sim N_{m} \left( \mu_{1} + \Sigma_{12} \Sigma_{22}^{-1} \left( \mathbf{X}_{2} - \mu_{2} \right) , \Sigma_{11} - \Sigma_{12} \Sigma_{22}^{-1} \Sigma_{21} \right)
X 1 ∣ X 2 ∼ N m ( μ 1 + Σ 12 Σ 22 − 1 ( X 2 − μ 2 ) , Σ 11 − Σ 12 Σ 22 − 1 Σ 21 )
Derivations Simple, Complicated, and Special Derivations Children not familiar with matrix algebra might find even the bivariate normal distribution challenging. The following derivation process is simple enough to be followed by high school graduates, yet the development itself is complicated, stopping at the mean and variance of the bivariate normal distribution.
f ( x , y ) = 1 2 π σ X σ Y 1 − ρ 2 exp [ − 1 2 ( 1 − ρ 2 ) [ ( x − μ X σ X ) 2 + ( y − μ Y σ Y ) 2 − 2 ρ ( x − μ X ) ( y − μ Y ) σ X σ Y ] ]
f(x,y) = {{ 1 } \over { 2 \pi \sigma_{X} \sigma_{Y} \sqrt{1 - \rho^{2}} }} \exp \left[ - {{ 1 } \over { 2 \left( 1 - \rho^{2} \right) }} \left[ \left( {{ x - \mu_{X} } \over { \sigma_{X} }} \right)^{2} + \left( {{ y - \mu_{Y} } \over { \sigma_{Y} }} \right)^{2}- 2 \rho {{ \left( x - \mu_{X} \right) \left( y - \mu_{Y} \right) } \over { \sigma_{X} \sigma_{Y} }} \right] \right]
f ( x , y ) = 2 π σ X σ Y 1 − ρ 2 1 exp [ − 2 ( 1 − ρ 2 ) 1 [ ( σ X x − μ X ) 2 + ( σ Y y − μ Y ) 2 − 2 ρ σ X σ Y ( x − μ X ) ( y − μ Y ) ] ]
f ( x ∣ y ) = f ( x , y ) f Y ( y ) = 1 2 π σ X σ Y 1 − ρ 2 e [ − 1 2 ( 1 − ρ 2 ) [ ( x − μ X σ X ) 2 + ( y − μ Y σ Y ) 2 − 2 ρ ( x − μ X ) ( y − μ Y ) σ X σ Y ] ] 1 2 π σ Y e [ − 1 2 ( y − μ Y σ Y ) 2 ] = 1 2 π σ X 1 − ρ 2 e [ − 1 2 ( 1 − ρ 2 ) [ ( x − μ X σ X ) 2 + ( y − μ Y σ Y ) 2 − 2 ρ ( x − μ X ) ( y − μ Y ) σ X σ Y ] ] e [ − 1 − ρ 2 2 ( 1 − ρ 2 ) ( y − μ Y σ Y ) 2 ] = 1 2 π σ X 1 − ρ 2 exp [ − 1 2 ( 1 − ρ 2 ) [ ( x − μ X σ X ) 2 + ρ 2 ( y − μ Y σ Y ) 2 − 2 ρ ( x − μ X ) ( y − μ Y ) σ X σ Y ] ] = 1 2 π σ X 1 − ρ 2 exp [ − 1 2 ( 1 − ρ 2 ) [ ( x − μ X σ X ) − ρ ( y − μ Y σ Y ) ] 2 ] = 1 2 π σ X 1 − ρ 2 exp [ − 1 2 σ X 2 ( 1 − ρ 2 ) [ x − μ X − ρ σ X ( y − μ Y σ Y ) ] 2 ] = 1 2 π σ X 1 − ρ 2 exp [ − 1 2 σ X 2 ( 1 − ρ 2 ) [ x − μ X − ρ σ X σ Y ( y − μ Y ) ] 2 ]
\begin{align*}
& f \left( x | y \right)
\\ =& {{ f \left( x , y \right) } \over { f_{Y}(y) }}
\\ =& {{ {{ 1 } \over { 2 \pi \sigma_{X} \sigma_{Y} \sqrt{1 - \rho^{2}} }} e^{\left[ - {{ 1 } \over { 2 \left( 1 - \rho^{2} \right) }} \left[ \left( {{ x - \mu_{X} } \over { \sigma_{X} }} \right)^{2} + \left( {{ y - \mu_{Y} } \over { \sigma_{Y} }} \right)^{2}- 2 \rho {{ \left( x - \mu_{X} \right) \left( y - \mu_{Y} \right) } \over { \sigma_{X} \sigma_{Y} }} \right] \right]} } \over { {{ 1 } \over { \sqrt{2 \pi} \sigma_{Y} }} e^{ \left[ - {{ 1 } \over { 2 }} \left( {{ y - \mu_{Y} } \over { \sigma_{Y} }} \right)^{2} \right] } }}
\\ =& {{ 1 } \over { \sqrt{2 \pi} \sigma_{X} \sqrt{1 - \rho^{2}} }} {{ e^{\left[ - {{ 1 } \over { 2 \left( 1 - \rho^{2} \right) }} \left[ \left( {{ x - \mu_{X} } \over { \sigma_{X} }} \right)^{2} + \left( {{ y - \mu_{Y} } \over { \sigma_{Y} }} \right)^{2}- 2 \rho {{ \left( x - \mu_{X} \right) \left( y - \mu_{Y} \right) } \over { \sigma_{X} \sigma_{Y} }} \right] \right]} } \over { e^{\left[ - {{ 1 - \rho^{2} } \over { 2 \left( 1 - \rho^{2} \right) }} \left( {{ y - \mu_{Y} } \over { \sigma_{Y} }} \right)^{2} \right]} }}
\\ =& {{ 1 } \over { \sqrt{2 \pi} \sigma_{X} \sqrt{1 - \rho^{2}} }} \exp \left[ - {{ 1 } \over { 2 \left( 1 - \rho^{2} \right) }} \left[ \left( {{ x - \mu_{X} } \over { \sigma_{X} }} \right)^{2} + \rho^{2} \left( {{ y - \mu_{Y} } \over { \sigma_{Y} }} \right)^{2}- 2 \rho {{ \left( x - \mu_{X} \right) \left( y - \mu_{Y} \right) } \over { \sigma_{X} \sigma_{Y} }} \right] \right]
\\ =& {{ 1 } \over { \sqrt{2 \pi} \sigma_{X} \sqrt{1 - \rho^{2}} }} \exp \left[ - {{ 1 } \over { 2 \left( 1 - \rho^{2} \right) }} \left[ \left( {{ x - \mu_{X} } \over { \sigma_{X} }} \right) - \rho \left( {{ y - \mu_{Y} } \over { \sigma_{Y} }} \right) \right]^{2} \right]
\\ =& {{ 1 } \over { \sqrt{2 \pi} \sigma_{X} \sqrt{1 - \rho^{2}} }} \exp \left[ - {{ 1 } \over { 2 \sigma_{X}^{2} \left( 1 - \rho^{2} \right) }} \left[ x - \mu_{X} - \rho \sigma_{X} \left( {{ y - \mu_{Y} } \over { \sigma_{Y} }} \right) \right]^{2} \right]
\\ =& {{ 1 } \over { \sqrt{2 \pi} \sigma_{X} \sqrt{1 - \rho^{2}} }} \exp \left[ - {{ 1 } \over { 2 \sigma_{X}^{2} \left( 1 - \rho^{2} \right) }} \left[ x - \mu_{X} - \rho {{ \sigma_{X} } \over { \sigma_{Y} }} \left( y - \mu_{Y} \right) \right]^{2} \right]
\end{align*}
= = = = = = = f ( x ∣ y ) f Y ( y ) f ( x , y ) 2 π σ Y 1 e [ − 2 1 ( σ Y y − μ Y ) 2 ] 2 π σ X σ Y 1 − ρ 2 1 e [ − 2 ( 1 − ρ 2 ) 1 [ ( σ X x − μ X ) 2 + ( σ Y y − μ Y ) 2 − 2 ρ σ X σ Y ( x − μ X ) ( y − μ Y ) ] ] 2 π σ X 1 − ρ 2 1 e [ − 2 ( 1 − ρ 2 ) 1 − ρ 2 ( σ Y y − μ Y ) 2 ] e [ − 2 ( 1 − ρ 2 ) 1 [ ( σ X x − μ X ) 2 + ( σ Y y − μ Y ) 2 − 2 ρ σ X σ Y ( x − μ X ) ( y − μ Y ) ] ] 2 π σ X 1 − ρ 2 1 exp [ − 2 ( 1 − ρ 2 ) 1 [ ( σ X x − μ X ) 2 + ρ 2 ( σ Y y − μ Y ) 2 − 2 ρ σ X σ Y ( x − μ X ) ( y − μ Y ) ] ] 2 π σ X 1 − ρ 2 1 exp [ − 2 ( 1 − ρ 2 ) 1 [ ( σ X x − μ X ) − ρ ( σ Y y − μ Y ) ] 2 ] 2 π σ X 1 − ρ 2 1 exp [ − 2 σ X 2 ( 1 − ρ 2 ) 1 [ x − μ X − ρ σ X ( σ Y y − μ Y ) ] 2 ] 2 π σ X 1 − ρ 2 1 exp [ − 2 σ X 2 ( 1 − ρ 2 ) 1 [ x − μ X − ρ σ Y σ X ( y − μ Y ) ] 2 ]
This is the same as the probability density function of the univariate normal distribution with mean μ X + ρ σ X σ Y ( Y − μ Y ) \mu_{X} + \rho {{ \sigma_{X} } \over { \sigma_{Y} }} \left( Y - \mu_{Y} \right) μ X + ρ σ Y σ X ( Y − μ Y ) ( 1 − ρ 2 ) σ X 2 \left( 1 - \rho^{2} \right) \sigma_{X}^{2} ( 1 − ρ 2 ) σ X 2
Difficult, Simple, and General Derivations Let W : = X 1 − Σ 12 Σ 22 − 1 X 2 \mathbf{W} := \mathbf{X}_{1} - \Sigma_{12} \Sigma_{22}^{-1} \mathbf{X}_{2} W := X 1 − Σ 12 Σ 22 − 1 X 2 p : = ( m − n ) ∈ N p := (m - n) \in \mathbb{N} p := ( m − n ) ∈ N identity matrix I k ∈ R k × k I_{k} \in \mathbb{R}^{k \times k} I k ∈ R k × k zero matrix O ∈ R p × m O \in \mathbb{R}^{p \times m} O ∈ R p × m [ W X 2 ] = [ I m − Σ 12 Σ 22 − 1 O I p ] [ X 1 X 2 ]
\begin{bmatrix} \mathbf{W} \\ \mathbf{X}_{2} \end{bmatrix} = \begin{bmatrix}
I_{m} & - \Sigma_{12} \Sigma_{22}^{-1}
\\ O & I_{p}
\end{bmatrix} \begin{bmatrix} \mathbf{X}_{1} \\ \mathbf{X}_{2} \end{bmatrix}
[ W X 2 ] = [ I m O − Σ 12 Σ 22 − 1 I p ] [ X 1 X 2 ]
Normality of Linear Transformations : For a matrix A ∈ R m × n A \in \mathbb{R}^{m \times n} A ∈ R m × n vector b ∈ R m \mathbf{b} \in \mathbb{R}^{m} b ∈ R m linear transformation Y = A X + b \mathbf{Y} = A \mathbf{X} + \mathbf{b} Y = A X + b random vector X ∼ N n ( μ , Σ ) \mathbf{X} \sim N_{n} \left( \mu , \Sigma \right) X ∼ N n ( μ , Σ ) multivariate normal distribution still follows a multivariate normal distribution N m ( A μ + b , A Σ A T ) N_{m} \left( A \mu + \mathbf{b} , A \Sigma A^{T} \right) N m ( A μ + b , A Σ A T )
W \mathbf{W} W covariance matrix with mean vector
E W = μ 1 − Σ 12 Σ 22 − 1 μ 2
E \mathbf{W} = \mu_{1} - \Sigma_{12} \Sigma_{22}^{-1} \mu_{2}
E W = μ 1 − Σ 12 Σ 22 − 1 μ 2 A = Σ 12 Σ 22 − 1 A = \Sigma_{12} \Sigma_{22}^{-1} A = Σ 12 Σ 22 − 1 Cov X = A Σ A T = [ I m − Σ 12 Σ 22 − 1 O I p ] [ Σ 11 Σ 12 Σ 21 Σ 22 ] [ I m O T − Σ 22 − 1 Σ 21 I p ] = [ Σ 11 − Σ 12 Σ 22 − 1 Σ 21 O T O Σ 22 ] ⟹ Cov W = Σ 11 − Σ 12 Σ 22 − 1 Σ 21
\begin{align*}
\operatorname{Cov} \mathbf{X} =& A \Sigma A^{T}
\\ =& \begin{bmatrix}
I_{m} & - \Sigma_{12} \Sigma_{22}^{-1}
\\ O & I_{p}
\end{bmatrix} \begin{bmatrix}
\Sigma_{11} & \Sigma_{12}
\\ \Sigma_{21} & \Sigma_{22}
\end{bmatrix} \begin{bmatrix}
I_{m} & O^{T}
\\ - \Sigma_{22}^{-1} \Sigma_{21} & I_{p}
\end{bmatrix}
\\ =& \begin{bmatrix}
\Sigma_{11} - \Sigma_{12} \Sigma_{22}^{-1} \Sigma_{21} & O^{T}
\\ O & \Sigma_{22}
\end{bmatrix}
\\ \implies \operatorname{Cov} \mathbf{W} =& \Sigma_{11} - \Sigma_{12} \Sigma_{22}^{-1} \Sigma_{21}
\end{align*}
Cov X = = = ⟹ Cov W = A Σ A T [ I m O − Σ 12 Σ 22 − 1 I p ] [ Σ 11 Σ 21 Σ 12 Σ 22 ] [ I m − Σ 22 − 1 Σ 21 O T I p ] [ Σ 11 − Σ 12 Σ 22 − 1 Σ 21 O O T Σ 22 ] Σ 11 − Σ 12 Σ 22 − 1 Σ 21 A T A^{T} A T transpose of A A A
Equivalence of Independence and Zero Covariance : Let’s assume that ( X 1 , X 2 ) ∼ N n ( μ , Σ ) \left( \mathbf{X}_{1} , \mathbf{X}_{2} \right) \sim N_{n} \left( \mu , \Sigma \right) ( X 1 , X 2 ) ∼ N n ( μ , Σ ) X 1 ⊥ X 2 ⟺ Σ 12 = Σ 21 = O
\mathbf{X}_{1} \perp \mathbf{X}_{2} \iff \Sigma_{12} = \Sigma_{21} = O
X 1 ⊥ X 2 ⟺ Σ 12 = Σ 21 = O
Since X \mathbf{X} X Cov ( W , X 2 ) = O \operatorname{Cov} \left( \mathbf{W} , \mathbf{X}_{2} \right) = O Cov ( W , X 2 ) = O W \mathbf{W} W X 2 \mathbf{X}_{2} X 2 W ∣ X 2 \mathbf{W} | \mathbf{X}_{2} W ∣ X 2 W \mathbf{W} W W ∣ X 2 ∼ N m ( μ 1 − Σ 12 Σ 22 − 1 μ 2 , Σ 11 − Σ 12 Σ 22 − 1 Σ 21 )
\mathbf{W} | \mathbf{X}_{2} \sim N_{m} \left( \mu_{1} - \Sigma_{12} \Sigma_{22}^{-1} \mu_{2} , \Sigma_{11} - \Sigma_{12} \Sigma_{22}^{-1} \Sigma_{21} \right)
W ∣ X 2 ∼ N m ( μ 1 − Σ 12 Σ 22 − 1 μ 2 , Σ 11 − Σ 12 Σ 22 − 1 Σ 21 ) W \mathbf{W} W X 1 = W + Σ 12 Σ 22 − 1 X 2 \mathbf{X}_{1} = \mathbf{W} + \Sigma_{12} \Sigma_{22}^{-1} \mathbf{X}_{2} X 1 = W + Σ 12 Σ 22 − 1 X 2 X 1 ∣ X 2 ∼ N m ( μ 1 + Σ 12 Σ 22 − 1 ( X 2 − μ 2 ) , Σ 11 − Σ 12 Σ 22 − 1 Σ 21 )
\mathbf{X}_{1} | \mathbf{X}_{2} \sim N_{m} \left( \mu_{1} + \Sigma_{12} \Sigma_{22}^{-1} \left( \mathbf{X}_{2} - \mu_{2} \right) , \Sigma_{11} - \Sigma_{12} \Sigma_{22}^{-1} \Sigma_{21} \right)
X 1 ∣ X 2 ∼ N m ( μ 1 + Σ 12 Σ 22 − 1 ( X 2 − μ 2 ) , Σ 11 − Σ 12 Σ 22 − 1 Σ 21 )
■
