📂Mathematical Statistics

Proof of the Invariance Property of the Maximum Likelihood Estimator

Theorem

The Maximum Likelihood Estimator (MLE) is invariant with respect to transformation of function. In other words, if $\hat{\theta}$ is the MLE of the parameter $\theta$, then for any function $\tau$, $\tau \left( \hat{\theta} \right)$ is also the MLE of $\tau \left( \theta \right)$.

Proof ¹

Let $\eta := \tau \left( \theta \right)$ and define a new function $L^{\ast}$ for the likelihood function $L = L \left( \theta | \mathbf{x} \right)$ as $$L^{\ast} \left( \hat{\eta} | \mathbf{x} \right) = L^{\ast} \left( \tau^{-1} \left( \eta \right) | \mathbf{x} \right)$$.

If $\hat{\eta}$ maximizes the function value of the likelihood function $L^{\ast} \left( \eta | \mathbf{x} \right)$, $$L^{\ast} \left( \hat{\eta} | \mathbf{x} \right) = L^{\ast} \left( \tau \left( \hat{\theta} \right) | \mathbf{x} \right)$$ then it suffices to show that $$\begin{align*} L^{\ast} \left( \hat{\eta} | \mathbf{x} \right) =& \sup_{\eta} \sup_{\left\{ \theta : \tau (\theta) = \eta \right\}} L \left( \theta | \mathbf{x} \right) \\ =& \sup_{\theta} L \left( \theta | \mathbf{x} \right) \\ =& L \left( \hat{\theta} | \mathbf{x} \right) \\ =& \sup_{\left\{ \theta : \tau (\theta) = \tau \left( \hat{\theta} \right) \right\}} L \left( \theta | \mathbf{x} \right) \\ =& L^{\ast} \left( \tau \left( \hat{\theta} \right) | \mathbf{x} \right) \end{align*}$$.

Here, the reason for considering sets like $\left\{ \theta : \tau (\theta) = \eta \right\}$ is because there is no guarantee that $\tau$ is bijective. Of course, logically, one should not use expressions like $\tau^{-1}$ from the start, but it does not matter in the context of this theorem.

■