Various Properties of Covariance
Definitions and Properties
The covariance of probability variables $X$ and $Y$, whose means are $\mu_{X}$ and $\mu_{Y}$ respectively, is defined as $\operatorname{Cov} (X ,Y) : = E \left[ ( X - \mu_{X} ) ( Y - \mu_{Y} ) \right]$. Covariance has the following properties:
- [1]: $\operatorname{Var} (X) = \operatorname{Cov} (X,X)$
- [2]: $\operatorname{Cov} (X,Y) = \operatorname{Cov} (Y, X)$
- [3]: $\operatorname{Var} (X + Y) = \operatorname{Var} (X) + \operatorname{Var} (Y) + 2 \operatorname{Cov} (X,Y)$
- [4]: $\operatorname{Cov} (X + Y , Z ) = \operatorname{Cov}(X,Z) + \operatorname{Cov}(Y,Z)$
- [5]: $\operatorname{Cov} (aX + b , cY + d ) = ac \operatorname{Cov}(X,Y)$
Explanation
Covariance indicates the linear correlation between two variables and, unlike variance, can also be negative as well as $0$.
Proof
[1]
$$ \begin{align*} \operatorname{Cov} (X ,X) =& E[ ( X - \mu_{X} ) ( X - \mu_{X} ) ] \\ =& E[ ( X - \mu_{X} )^2 ] \\ =& \operatorname{Var} (X) \end{align*} $$
■
[2]
$$ \begin{align*} \operatorname{Cov} (X ,Y) =& E[ ( X - \mu_{X} ) ( Y - \mu_{Y} ) ] \\ =& E[ ( Y - \mu_{Y} ) ( X - \mu_{X} ) ] \\ =& \operatorname{Cov} (X ,Y) \end{align*} $$
■
[3]
$$ \begin{align*} \operatorname{Var} (X + Y) =& E [ ( X + Y - \mu_{X} - \mu_{Y} )^2 ] \\ =& E \left[ \left\{ ( X - \mu_{X} ) + (Y - \mu_{Y} ) \right\} ^2 \right] \\ =& E \left[ ( X - \mu_{X} )^2 + 2 ( X - \mu_{X} ) (Y - \mu_{Y} )+ (Y - \mu_{Y} )^2 \right] \\ =& E[ ( X - \mu_{X} )^2] + 2 E [ ( X - \mu_{X} ) (Y - \mu_{Y} ) ] + E [ (Y - \mu_{Y} )^2 ] \\ =& \operatorname{Var} (X) + 2 \operatorname{Cov} (X,Y) + \operatorname{Var} (Y) \end{align*} $$
■
[4]
$$ \begin{align*} \operatorname{Cov} (X + Y , Z ) =& E \left[ ( X + Y - \mu_{X} - \mu_{Y} ) ( Z - \mu_{Z} ) \right] \\ =& E \left[ \left\{ ( X - \mu_{X} ) + ( Y - \mu_{Y} ) \right\} ( Z - \mu_{Z} ) \right] \\ =& E \left[ ( X - \mu_{X} ) ( Z - \mu_{Z} ) \right] + E \left[ ( Y - \mu_{Y} ) ( Z - \mu_{Z} ) \right] \\ =& \operatorname{Cov}(X,Z) + \operatorname{Cov}(Y,Z) \end{align*} $$
■
[5]
$$ \begin{align*} \operatorname{Cov} (aX + b , cY + d ) =& E \left[ ( aX + b - a \mu_{X} - b ) ( cY + d - c \mu_{Y} - d ) \right] \\ =& E \left[ ( aX - a \mu_{X} ) ( cY - c \mu_{Y} ) \right] \\ =& E \left[ a c ( X - \mu_{X} ) ( Y - \mu_{Y} ) \right] \\ =& ac E \left[( X - \mu_{X} ) ( Y - \mu_{Y} ) \right] \\ =& ac \operatorname{Cov}(X,Y) \end{align*} $$
■