📂Mathematical Statistics

Conjugate Prior Distribution

Definition ¹

If the prior distribution and the posterior distribution belong to the same family of distributions, the prior distribution is called a conjugate prior.

Explanation

Although Bayesian methods are originally about finding the parameters through updating regardless of what the prior distribution is, if we have some knowledge about the model, using an appropriate prior distribution can simplify the mathematical computations and make the results easier to understand.

• (1) The conjugate prior of $\text{Bin} (n , \theta)$ is $\theta \sim \text{Beta} (\alpha, \beta)$
• (2) The conjugate prior of $\text{Poi} ( \theta)$ is $\theta \sim \text{Gamma} (a, b)$
• (3) When the population variance $\sigma^2$ is known, the conjugate prior of $N ( \mu , \sigma^2 )$ is $\mu \sim N ( \mu_{0} , \tau_{0}^2 )$
• (4) When the population mean $\mu$ is known, the conjugate prior of $N ( \mu , \sigma^2 )$ is $\tau \sim \text{Gamma} (a, b)$ with respect to $\displaystyle \tau = {{1} \over {\sigma^2 }}$
• (5) When $\lambda$ is known, the conjugate prior of $\text{Gamma} (a, \lambda)$ is $\lambda \sim \text{Gamma} (a, b)$

Example

As an example, let us consider the model $p ( y | \theta ) = \binom{ n}{ y } \theta^{y} (1-\theta)^{n-y}$ for binomial data:

Considering its conjugate prior $\theta \sim \text{Beta} (\alpha, \beta)$, we have $$ \pi (\theta) \propto \theta^{\alpha - 1} ( 1 - \theta )^{\beta -1 } $$ Then the posterior distribution of $\theta$ is $$ p ( \theta | y ) \propto \theta^{y + \alpha -1 } (1-\theta)^{n-y + \beta -1} $$ which means the beta distribution $ \theta \sim \text{Beta} (y + \alpha, n - y + \beta)$. Therefore, the posterior mean of $\theta$ is $$ E ( \theta | y) = {{y + \alpha} \over {n + \alpha + \beta}} $$ which can also be viewed in the form of a weighted average of the sample mean $\displaystyle {{y} \over {n}}$ and the prior mean $\displaystyle {{\alpha } \over {\alpha + \beta}}$, namely $$ E ( \theta | y) = {{n} \over {n + \alpha + \beta}} {{y} \over {n}} + {{\alpha + \beta } \over {n + \alpha + \beta}} {{\alpha} \over {\alpha + \beta}} $$

Therefore, we can infer that as the sample size $n$ grows larger, the influence of the prior mean diminishes and the result gets closer to frequentist inference.

Revisiting the discussion above with a focus on the concept of the conjugate prior, since the prior distribution was written as a beta distribution and the posterior distribution also turned out to be a beta distribution, we could confirm that sequential analysis is very convenient. Because the pmf of the binomial distribution and the pdf of the beta distribution look similar, there was no significant difficulty in the formulas, and everything went smoothly.