logo

Sufficient Statistics and Maximum Likelihood Estimators for the Binomial Distribution 📂Probability Distribution

Sufficient Statistics and Maximum Likelihood Estimators for the Binomial Distribution

Theorem

Let’s assume we have a random sample X:=(X1,⋯ ,Xn)∼U(0,θ)\mathbf{X} := \left( X_{1} , \cdots , X_{n} \right) \sim U \left( 0 , \theta \right) following a uniform distribution.

The sufficient statistic TT and maximum likelihood estimator θ^\hat{\theta} for θ\theta are as follows: T=max⁡k=1,⋯ ,nXkθ^=max⁡k=1,⋯ ,nXk \begin{align*} T =& \max_{k=1 , \cdots , n} X_{k} \\ \hat{\theta} =& \max_{k=1 , \cdots , n} X_{k} \end{align*}

Proof

Strategy: The sufficient statistic and maximum likelihood estimator of the uniform distribution are must-know statistics due to their practicality aside, especially for homework, midterms, and final exams. They can be derived directly by definition, but at first, this is not always straightforward.

Sufficient statistic and maximum likelihood estimator for location families: Suppose we have a random sample X1,⋯ ,Xn∼XX_{1} , \cdots , X_{n} \sim X from a location family with a probability density function of fX(x;θ)=fX(x−θ)f_{X} \left( x ; \theta \right) = f_{X} \left( x - \theta \right). The sufficient statistic and maximum likelihood estimator are

  • For a support of XX that is bounded above, use max⁡Xk\max X_{k}
  • For a support of XX that is bounded below, use min⁡Xk\min X_{k}

depending on the condition.

U(0,θ)U \left( 0 , \theta \right) is a location family, and while the sufficient statistic and maximum likelihood estimator for the location family can easily be guessed by corollaries, let’s derive them directly for an intuitive understanding.

Sufficient Statistic

Product of indicator functions: ∏i=1nI(−∞,θ](xi)=I(−∞,θ](max⁡i∈[n]xi) \prod_{i=1}^{n} I_{(-\infty, \theta]} \left( x_{i} \right) = I_{(-\infty, \theta]} \left( \max_{i \in [n]} x_{i} \right)

f(x;θ)=∏k=1nf(xk;θ)=∏k=1n1θI[0,θ](xk)=1θnI[0,θ](max⁡xk)=1θnI[0,θ](max⁡xk)⋅1 \begin{align*} f \left( \mathbf{x} ; \theta \right) =& \prod_{k=1}^{n} f \left( x_{k} ; \theta \right) \\ =& \prod_{k=1}^{n} {{ 1 } \over { \theta }} I_{[0,\theta]} \left( x_{k} \right) \\ =& {{ 1 } \over { \theta^{n} }} I_{[0,\theta]} \left( \max x_{k} \right) \\ =& {{ 1 } \over { \theta^{n} }} I_{[0,\theta]} \left( \max x_{k} \right) \cdot 1 \end{align*}

Neyman Factorization Theorem: Consider a random sample X1,⋯ ,XnX_{1} , \cdots , X_{n} with the same probability mass/density function f(x;θ)f \left( x ; \theta \right) for parameter θ∈Θ\theta \in \Theta. A statistic Y=u1(X1,⋯ ,Xn)Y = u_{1} \left( X_{1} , \cdots , X_{n} \right) is a sufficient statistic for θ\theta if it satisfies the existence of two non-negative functions k1,k2≥0k_{1} , k_{2} \ge 0 such that: f(x1;θ)⋯f(xn;θ)=k1[u1(x1,⋯ ,xn);θ]k2(x1,⋯ ,xn) f \left( x_{1} ; \theta \right) \cdots f \left( x_{n} ; \theta \right) = k_{1} \left[ u_{1} \left( x_{1} , \cdots , x_{n} \right) ; \theta \right] k_{2} \left( x_{1} , \cdots , x_{n} \right) However, k2k_{2} must not depend on θ\theta.

Based on the Neyman Factorization Theorem, T:=max⁡XkT := \max X_{k} is a sufficient statistic for θ\theta.

Maximum Likelihood Estimator

L(θ;x)=f(x;θ)=I[0,θ](max⁡xk) L \left( \theta ; \mathbf{x} \right) = f \left( \mathbf{x} ; \theta \right) = I_{[0,\theta]} \left( \max x_{k} \right) The likelihood function of the random sample can be determined as shown, eliminating the need to specifically use the partial derivative of the indicator function.

Definition of Maximum Likelihood Estimator: An estimator θ^:=θ^(X)\hat{\theta} := \hat{\theta} \left( \mathbf{X} \right) that satisfies the following is called Maximum Likelihood Estimator, or MLE for short: θ^=arg max⁡L(θ;X) \hat{\theta} = \argmax L \left( \theta ; \mathbf{X} \right)

Based on the Definition of Maximum Likelihood Estimator, instead of extensively worrying about the likelihood function, focusing solely on θ^≥max⁡Xk\hat{\theta} \ge \max X_{k} suffices, because if θ^<max⁡Xk\hat{\theta} < \max X_{k} then it results in L=0L = 0. As illustrated by this explanation and definition, the maximum likelihood estimator doesn’t have to be uniquely max⁡Xk\max X_{k}, nor does it need to be; hence, there’s no point in unnecessarily considering max⁡Xk+700\max X_{k} + 700 when max⁡Xk\max X_{k} is sufficient.

■