logo

Multivariate Normal Distribution 📂Probability Distribution

Multivariate Normal Distribution

Definition

The multivariate normal distribution Np(μ,Σ)N_{p} \left( \mu , \Sigma \right) has a probability density function based on the mean vector μ∈Rp\mathbf{\mu} \in \mathbb{R}^{p} and the covariance matrix Σ∈Rp×p\Sigma \in \mathbb{R}^{p \times p} as follows:

f(x)=((2π)pdet⁡Σ)−1/2exp⁡[−12(x−μ)TΣ−1(x−μ)],x∈Rp f (\textbf{x}) = \left( (2\pi)^{p} \det \Sigma \right)^{-1/2} \exp \left[ - {{ 1 } \over { 2 }} \left( \textbf{x} - \mathbf{\mu} \right)^{T} \Sigma^{-1} \left( \textbf{x} - \mathbf{\mu} \right) \right] \qquad , \textbf{x} \in \mathbb{R}^{p}

  • xT\mathbf{x}^{T} denotes the transpose of x\mathbf{x}.

Theorems

X=[X1X2]:Ω→Rnμ=[μ1μ2]∈RnΣ=[Σ11Σ12Σ21Σ22]∈Rn×n \begin{align*} \mathbf{X} =& \begin{bmatrix} \mathbf{X}_{1} \\ \mathbf{X}_{2} \end{bmatrix} & : \Omega \to \mathbb{R}^{n} \\ \mu =& \begin{bmatrix} \mu_{1} \\ \mu_{2} \end{bmatrix} & \in \mathbb{R}^{n} \\ \Sigma =& \begin{bmatrix} \Sigma_{11} & \Sigma_{12} \\ \Sigma_{21} & \Sigma_{22} \end{bmatrix} & \in \mathbb{R}^{n \times n} \end{align*} In the statements of the theorems below, unless otherwise specified, X\mathbf{X}, μ\mu, and Σ\Sigma refer to the same block matrix.

Linear transformation of multivariate normal distribution

The linear transformation Y=AX+b\mathbf{Y} = A \mathbf{X} + \mathbf{b} of a random vector X∼Nn(μ,Σ)\mathbf{X} \sim N_{n} \left( \mu , \Sigma \right) that follows a multivariate normal distribution, given matrix A∈Rm×nA \in \mathbb{R}^{m \times n} and vector b∈Rm\mathbf{b} \in \mathbb{R}^{m}, still follows a multivariate normal distribution Nm(Aμ+b,AΣAT)N_{m} \left( A \mu + \mathbf{b} , A \Sigma A^{T} \right).

Independence and zero correlation are equivalent in multivariate normal distributions

Let there be a random vector X∼Nn(μ,Σ)\mathbf{X} \sim N_{n} \left( \mu , \Sigma \right) that follows a multivariate normal distribution. Then, the following holds: X1⊥X2  ⟺  Σ12=Σ21=O \mathbf{X}_{1} \perp \mathbf{X}_{2} \iff \Sigma_{12} = \Sigma_{21} = O

Conditional mean and variance of multivariate normal distribution

Given a random vector X∼Nn(μ,Σ)\mathbf{X} \sim N_{n} \left( \mu , \Sigma \right) that follows a multivariate normal distribution, the conditional probability vector X1∣X2:Ω→Rm\mathbf{X}_{1} | \mathbf{X}_{2} : \Omega \to \mathbb{R}^{m} still follows a multivariate normal distribution, specifically having the following mean vector and covariance matrix: X1∣X2∼Nm(μ1+Σ12Σ22−1(X2−μ2),Σ11−Σ12Σ22−1Σ21) \mathbf{X}_{1} | \mathbf{X}_{2} \sim N_{m} \left( \mu_{1} + \Sigma_{12} \Sigma_{22}^{-1} \left( \mathbf{X}_{2} - \mu_{2} \right) , \Sigma_{11} - \Sigma_{12} \Sigma_{22}^{-1} \Sigma_{21} \right)

Multivariate normality of regression coefficient vectors

The estimators of regression coefficients β^\hat{\beta} follow a multivariate normal distribution as follows: β^∼N1+p(β,σ2(XTX)−1) \hat{\beta} \sim N_{1+p} \left( \beta , \sigma^{2} \left( X^{T} X \right)^{-1} \right)

Moment generating function

The moment generating function of X∼Np(μ,Σ)X \sim N_{p} \left( \mu , \Sigma \right) is as follows: MX(t)=exp⁡(tTμ+12tTΣt),t∈Rp M_{X} \left( \mathbf{t} \right) = \exp \left( \mathbf{t}^{T} \mu + {{ 1 } \over { 2 }} \mathbf{t}^{T} \Sigma \mathbf{t} \right) \qquad , \mathbf{t} \in \mathbb{R}^{p}

Entropy

The entropy of the multivariate normal distribution Np(μ,Σ)N_{p}(\mu, \Sigma) is as follows:

H=12ln⁡[(2πe)p∣Σ∣]=12ln⁡(det⁡(2πeΣ)) H = \dfrac{1}{2}\ln \left[ (2 \pi e)^{p} \left| \Sigma \right| \right] = \dfrac{1}{2}\ln (\det (2\pi e \Sigma))

∣Σ∣\left| \Sigma \right| is the determinant of the covariance matrix.

See Also

  • Univariate normal distribution: When p=1p = 1 followed by μ∈R1\mu \in \mathbb{R}^{1} and Σ∈R1×1\Sigma \in \mathbb{R}^{1 \times 1}, the above probability density function exactly becomes the probability density function of the univariate normal distribution.