Exponential Family of Probability Distributions 📂Mathematical Statistics

Exponential Family of Probability Distributions

Definition ¹ ²

If the probability mass function or probability density function of a probability distribution with parameter $\theta$ can be expressed in terms of some functions $p,K,H,q,h,c,w_{i},t_{i}$ as follows, it is said to belong to the Exponential Family or Exponential Class. $\begin{align*} f \left( x ; \theta \right) =& \exp \left( p (\theta) K (x) + H(x) + q(\theta) \right) \\ =& h(x) c (\theta) \exp \left( \sum_{i=1}^{k} w_{i} (\theta) t_{i} (x) \right) \end{align*}$

Description

It should be immediately apparent that the forms of the two equations in the definition are essentially the same.

Theorem

$T_{i} \left( X_{1} , \cdots , X_{n} \right) := \sum_{j=1}^{n} t_{i} \left( X_{j} \right)$ For a random sample $\left\{ X_{j} \right\}_{j=1}^{n}$ with distribution function $f (x;\theta)$ , if a statistic $T_{1} , \cdots , T_{k}$ is defined as above, then its joint probability density function can be expressed in terms of some function $H$ as follows. $f_{T} \left( u_{1} , \cdots , u_{k} ; \theta \right) = H \left( u_{1} , \cdots , u_{k} \right) \left[ c (\theta) \right]^{n} \exp \left( \sum_{i=1}^{k} w_{i} (\theta) u_{i} \right)$

Example

Considering a Bernoulli trial with probability $p \in (0,1)$ , $\begin{align*} p^{x} (1-p)^{1-x} =& \left( {{ p } \over { 1-p }} \right)^{x} (1-p) \\ =& (1-p) \exp \left( x \log {{ p } \over { 1-p }} \right) \end{align*}$ it can be expressed as above, so the Bernoulli distribution belongs to the exponential family. A distribution repeated $n$ times, namely the binomial distribution, can be expressed in terms of the identity function $t_{j}(x) = \text{id} (x) = x$ as $T_{1} = X_{1} + \cdots + X_{n} = \sum_{j=1}^{n} t_{j} \left( X_{j} \right)$ In fact, the binomial distribution can be expressed as follows. $\begin{align*} \binom{n}{x} p^{x} (1-p)^{n-x} =& \binom{n}{x} \left( {{ p } \over { 1-p }} \right)^{x} (1-p)^{n} \\ =& (1-p)^{n} \exp \left( x \log {{ p } \over { 1-p }} + \log \binom{n}{x} \right) \end{align*}$