logo

Sufficient Statistics and Maximum Likelihood Estimators of Exponential Distributions 📂Probability Distribution

Sufficient Statistics and Maximum Likelihood Estimators of Exponential Distributions

Theorem

Given a random sample X:=(X1,⋯ ,Xn)∼exp⁡(λ)\mathbf{X} := \left( X_{1} , \cdots , X_{n} \right) \sim \exp \left( \lambda \right) that follows an exponential distribution.

The sufficient statistic TT and maximum likelihood estimator λ^\hat{\lambda} for λ\lambda are as follows. T=∑k=1nXkλ^=n∑k=1nXk \begin{align*} T =& \sum_{k=1}^{n} X_{k} \\ \hat{\lambda} =& {{ n } \over { \sum_{k=1}^{n} X_{k} }} \end{align*}

Proof

Sufficient Statistic

f(x;λ)=∏k=1nf(xk;λ)=∏k=1nλe−λxk=λne−λ∑kxk=λne−λ∑kxk⋅1 \begin{align*} f \left( \mathbf{x} ; \lambda \right) =& \prod_{k=1}^{n} f \left( x_{k} ; \lambda \right) \\ =& \prod_{k=1}^{n} \lambda e^{-\lambda x_{k}} \\ =& \lambda^{n} e^{-\lambda \sum_{k} x_{k}} \\ =& \lambda^{n} e^{-\lambda \sum_{k} x_{k}} \cdot 1 \end{align*}

Neyman Factorization Theorem: Let a random sample X1,⋯ ,XnX_{1} , \cdots , X_{n} have the same probability mass/density function f(x;θ)f \left( x ; \theta \right) for a parameter θ∈Θ\theta \in \Theta. A statistic Y=u1(X1,⋯ ,Xn)Y = u_{1} \left( X_{1} , \cdots , X_{n} \right) is a sufficient statistic for θ\theta if there exist two non-negative functions k1,k2≥0k_{1} , k_{2} \ge 0 that satisfy the following. f(x1;θ)⋯f(xn;θ)=k1[u1(x1,⋯ ,xn);θ]k2(x1,⋯ ,xn) f \left( x_{1} ; \theta \right) \cdots f \left( x_{n} ; \theta \right) = k_{1} \left[ u_{1} \left( x_{1} , \cdots , x_{n} \right) ; \theta \right] k_{2} \left( x_{1} , \cdots , x_{n} \right) Note, k2k_{2} must not depend on θ\theta.

According to the Neyman Factorization Theorem, T:=∑kXkT := \sum_{k} X_{k} is the sufficient statistic for λ\lambda.

Maximum Likelihood Estimator

log⁡L(λ;x)=log⁡f(x;λ)=log⁡λne−λ∑kxk=nlog⁡λ−λ∑k=1nxk \begin{align*} \log L \left( \lambda ; \mathbf{x} \right) =& \log f \left( \mathbf{x} ; \lambda \right) \\ =& \log \lambda^{n} e^{-\lambda \sum_{k} x_{k}} \\ =& n \log \lambda - \lambda \sum_{k=1}^{n} x_{k} \end{align*}

The log-likelihood function of the random sample is as above, and for the likelihood function to be maximized, the partial derivative with respect to λ\lambda must be 00, thus 0=n1λ−∑k=1nxk  ⟹  λ=n∑k=1nxk \begin{align*} & 0 = n {{ 1 } \over { \lambda }} - \sum_{k=1}^{n} x_{k} \\ \implies & \lambda = {{ n } \over { \sum_{k=1}^{n} x_{k} }} \end{align*}

Therefore, the maximum likelihood estimator λ^\hat{\lambda} for λ\lambda is as follows. λ^=n∑k=1nXk \hat{\lambda} = {{ n } \over { \sum_{k=1}^{n} X_{k} }}

■