📂Mathematical Statistics

Laplace Prior Distribution

Buildup

If there’s almost no information about the parameter, there’s no reason to consider a complex prior distribution:

• Example 1: If someone with a fair understanding of statistics is asked to guess the gender ratio of incoming freshmen for a certain university’s statistics department next year, they might make a reasonable guess based on the gender ratios in previous years. However, if someone with no relation or interest in the subject hears this question, they’ll likely guess 50:50 without a special reason.
• Example 2: If it’s said that there’s a bag with red, blue, green, and yellow marbles, and no other information is provided, then one would guess that the probability of drawing each color of marble is just 25:25:25:25.

Definition ¹

In such situations of extreme lack of information, the prior distribution used is called a Noninformative Prior. Among these, the prior distribution that does not assume any particular distribution and keeps all options open fairly is called the Laplace Prior.

Explanation

Improper Prior

If the parameter $\theta$ belongs to a certain interval $(a,b)$, then its prior distribution would be represented as a $\displaystyle \pi (\theta) = {{1} \over {b-a}} , a < \theta < b$, or uniform distribution. The problem arises when parameters like $-\infty \le \theta \le \infty$ are not bounded. In such cases, if $\pi (\theta)$ is put as a uniform distribution, then it would be calculated as $\displaystyle \int_{-\infty}^{\infty} \pi ( \theta ) d \theta = \infty$, hence cannot be used as a distribution function. Such prior distributions are called Improper Priors. Special care is needed when using the Laplace prior as such improper priors can lead to improper posteriors.

Problems with Improper Priors

For instance, if the data is assumed to follow an exponential distribution $\displaystyle \exp \left( {{1} \over {\theta}} \right)$, then one could consider $\displaystyle \pi ( \theta ) \propto c$ as the Laplace prior.

In this case, the posterior distribution of $\theta$ would be $$p ( \theta | y ) \propto {{1} \over {\theta }} \exp \left( - {{y} \over {\theta }} \right)$$ To check if this posterior distribution is appropriate, we set $\displaystyle \theta = {{1} \over {z}}$ and find its definite integral $$\int_{0 }^{\infty} p ( \theta | y ) d \theta \propto \int_{0}^{\infty} z \exp ( - y z ) {{1} \over {z^2}} dz = \infty$$ Therefore, the posterior distribution is not appropriate as a probability distribution function, thus, another prior distribution should be considered.

Improper Priors Are Not Always Problematic

However, not all improper priors lead to improper posteriors. For example, if the data is assumed to follow a normal distribution $N ( \theta , \sigma^2 )$, then one could consider $\displaystyle \pi ( \theta ) \propto c$ as the Laplace prior. In this case, the posterior distribution of $\theta$ would be $$p ( \theta | y_{1} , \cdots y_{n} ) \propto \exp \left( - {{1} \over {2 \sigma^2}} \sum_{i=1}^{n} (y_{i} - \theta )^2 \right)$$ With a bit more calculation, $$p ( \theta | y_{1} , \cdots y_{n} ) \propto \exp \left( - {{n} \over {2 \sigma^2}} (\theta - \overline{y} )^2 \right)$$ hence, an appropriate posterior distribution $N ( \overline{y} , \sigma^2 / n )$ can be obtained.