📂Mathematical Statistics

Jeffreys Prior Distribution

Definition ¹

For the distribution of the data $p( y | \theta)$, $\pi ( \theta ) \propto I^{1/2} ( \theta )$ is called the Jeffreys prior.

---

• $I$ denotes the Fisher information. $$ I ( \theta ) = E \left[ \left( \left. {{\partial \ln p (y | \theta) } \over {\partial \theta}} \right)^2 \right| \theta \right] = E \left[ \left. - {{\partial^2 \ln p (y | \theta) } \over { (\partial \theta )^2 }} \right| \theta \right] $$

Explanation

The Laplace prior $\pi (\theta) \propto 1$ was sufficient as a prior for the parameter $\theta$, but for a function of the parameter such as $\phi = \theta^2$, since $d \phi = 2 \theta d \theta$ we get $\displaystyle \pi (\phi ) \propto {{1} \over {\sqrt{\phi } }}$, so it is no longer the same prior as that of $\theta$. The Jeffreys prior is a prior that overcomes this lack of invariance, and is essentially an upward-compatible version of the Laplace prior.

Example

For example, suppose the data follow the exponential distribution $\displaystyle \exp \left( {{1} \over {\theta}} \right)$; then the Laplace prior $\displaystyle \pi ( \theta ) \propto c$ had the problem of yielding an improper posterior distribution.

First, computing the Jeffreys prior, since $\displaystyle p( y | \theta ) = {{1} \over { \theta }} \exp \left( - {{ y } \over { \theta }} \right)$, we have $$ {{\partial \ln p (y | \theta) } \over {\partial \theta}} = - {{1 } \over { \theta }} + {{ y} \over { \theta^2 }} $$ Taking the partial derivative with respect to $\theta$ once more, since $\displaystyle p( y | \theta ) = {{1} \over { \theta }} \exp \left( - {{ y } \over { \theta }} \right)$, we have $$ {{\partial^2 \ln p (y | \theta) } \over { (\partial \theta )^2 }} = {{1 } \over { \theta ^2}} - {{ 2 y} \over { \theta^3 }} $$ Therefore $$ E \left[ \left. - {{\partial^2 \ln p (y | \theta) } \over { (\partial \theta )^2 }} \right| \theta \right] = {{ 2 \theta } \over { \theta^3 }} - {{1 } \over { \theta ^2}} = {{1 } \over { \theta ^2}} $$ and we obtain the Jeffreys prior $\displaystyle \pi ( \theta ) = {{1 } \over { \theta }}$.

To check whether this posterior distribution is proper, setting $\displaystyle \theta = {{1} \over {z}}$ and computing the definite integral gives $$ \int_{0 }^{\infty} p ( \theta | y ) d \theta \propto \int_{0}^{\infty} z^2 \exp ( - y z ) {{1} \over {z^2}} dx = {{1} \over {y}} < \infty $$ Thus, in this case, we can confirm that the Jeffreys prior induced a proper posterior distribution.