📂Mathematical Statistics

Laplace Prior Distribution

Buildup

If there is almost no information about a parameter, there is no reason to bother thinking of a complicated prior distribution:

• Example 1: When asked to guess the sex ratio of next year’s incoming students in a certain university’s statistics department, someone who knows the department to some degree can make a reasonable guess by looking at the ratios from previous years. However, when someone with no relation to or interest in it hears this question, they will, unless there is a special reason, guess 50:50.
• Example 2: When told only that a certain bag contains red, blue, green, and yellow marbles, if there is no other information, one would simply guess that the probability of drawing a marble of each color is 25:25:25:25.

Definition ¹

A prior distribution used in a situation with such an extreme lack of information is called a noninformative prior. Among these, in particular, a prior distribution that does not assume any particular distribution and instead fairly leaves all possibilities open is called a laplace prior.

Explanation

Improper Prior Distribution

If a parameter $\theta$ belongs to some interval $(a,b)$, its prior distribution will be expressed as a uniform distribution, such as $\displaystyle \pi (\theta) = {{1} \over {b-a}} , a < \theta < b$. The problem is the case where the parameter is not bounded, such as $ -\infty \le \theta \le \infty$. In this case, if we set $\pi (\theta) $ to a uniform distribution, it is computed as $\displaystyle \int_{-\infty}^{\infty} \pi ( \theta ) d \theta = \infty$, so it is unfit for use as a distribution function. A prior distribution in such a case is called an improper prior. Since such an improper prior can lead to an improper posterior distribution, careful attention is required when using the Laplace prior.

Problems with the Improper Prior Distribution

For example, when the data follow an exponential distribution $\displaystyle \exp \left( {{1} \over {\theta}} \right)$, one can consider $\displaystyle \pi ( \theta ) \propto c$ as the Laplace prior.

In this case, the posterior distribution of $\theta$ is $$ p ( \theta | y ) \propto {{1} \over {\theta }} \exp \left( - {{y} \over {\theta }} \right) $$ To check whether this posterior distribution is proper, setting $\displaystyle \theta = {{1} \over {z}}$ and computing the definite integral gives $$ \int_{0 }^{\infty} p ( \theta | y ) d \theta \propto \int_{0}^{\infty} z \exp ( - y z ) {{1} \over {z^2}} dz = \infty $$ Therefore, the posterior distribution is not proper as a probability distribution function, so a different prior distribution must be considered.

The Improper Prior Distribution Is Not Always a Problem

However, an improper prior does not necessarily lead to an improper posterior distribution. For example, if the data follow a normal distribution $N ( \theta , \sigma^2 )$, one can consider $\displaystyle \pi ( \theta ) \propto c$ as the Laplace prior. In this case, the posterior distribution of $\theta$ is $$ p ( \theta | y_{1} , \cdots y_{n} ) \propto \exp \left( - {{1} \over {2 \sigma^2}} \sum_{i=1}^{n} (y_{i} - \theta )^2 \right) $$ With a little more computation, $$ p ( \theta | y_{1} , \cdots y_{n} ) \propto \exp \left( - {{n} \over {2 \sigma^2}} (\theta - \overline{y} )^2 \right) $$ so a proper posterior distribution $N ( \overline{y} , \sigma^2 / n )$ can be obtained.