Proof of Bayes' Theorem, Prior Distribution, and Posterior Distribution 📂Mathematical Statistics

Proof of Bayes' Theorem, Prior Distribution, and Posterior Distribution

Theorem ¹

For a sample space $S$, an event $A$, and a probability $P$, if $\left\{ S_1, S_2, \cdots ,S_n \right\}$ is a partition of $S$, then the following holds. $$ P(S_k|A)=\frac { P(S_k)P(A|S_k) }{ \sum _{ k=1 }^{ n }{ P(S_k)P(A|S_k) } } $$

Definition

In Bayes’ theorem, $P \left( S_{k} \right)$ on the right-hand side is called the prior probability, and $P \left( S_{k} | A \right)$ on the left-hand side is called the posterior probability. The probability distributions formed by these probabilities are called the prior distribution and the posterior distribution, respectively. $A$ (or the probability $P(A) = \sum P(S_k)P(A|S_k)$ or its corresponding probability density function) is called the evidence or the marginal likelihood.

Explanation

This theorem, also known as Bayes’ rule, can be proved easily using only two laws, yet its applications are enormous. The so-called Bayesian paradigm is a way of thinking that divides statistics itself in two, and its importance cannot be overemphasized no matter how many times it is stressed.

What we want to know is the left-hand side of the equation above. What we already know are the probabilities of the event $A$ and of each of the parts $S_k$ of the partition of the sample space $S$, as well as the probability that $A$ occurs given that each of them has occurred. In short, we start from a state in which we know everything about the $S_k$ and the influence they have on $A$. Bayes’ theorem is, conversely, a theorem that lets us know what influence $A$ has on each of them. If this sounds difficult, it is enough to simply remember that what we want to find is the left-hand side.

Proof

By the law of total probability and the multiplication rule of probability, we obtain the following equation. $$ \begin{align*} P(A)=&P(A\cap S_1)+P(A\cap S_2)+…+P(A\cap S_n) \\ =&P(S_1)P(A|S_1)+P(S_2)P(A|S_2)+…+P(S_n)P(A|S_n) \\ =& \sum _{ k=1 }^{ n }{ P(S_k)P(A|S_k) } \end{align*} $$ Taking the reciprocal of both sides here gives $$ \begin{align*} & \frac { 1 }{ \sum _{ k=1 }^{ n }{ P(S_k)P(A|S_k) } }=\frac { 1 }{ P(A) } \\ \implies& \frac { P(A\cap S_k) }{ \sum _{ k=1 }^{ n }{ P(S_k)P(A|S_k) } }=\frac { P(A\cap S_k) }{ P(A) } \\ \implies& \frac { P(S_k)P(A|S_k) }{ \sum _{ k=1 }^{ n }{ P(S_k)P(A|S_k) } }=P(S_k|A) \end{align*} $$

■