📂Mathematical Statistics

Laplace's Rule of Succession

Theorem ¹

Suppose that the prior distribution of the binomial model $\displaystyle p(y | \theta) = \binom{ n }{ y} \theta^{y} (1- \theta)^{n-y}$ follows the uniform distribution $U (0,1)$ and its posterior distribution follows the beta distribution $\beta (y+1 , n-y+1)$, so that $p( \theta | y ) \sim \theta^{y} (1- \theta)^{n-y}$. Then, for the data $y$ obtained so far, the probability that a new $\tilde{y}$ equals $1$ is $$ p(\tilde{y} = 1| y) = {{y+1} \over {n+2}} $$

Explanation

From the frequentist’s perspective, the probability that $\tilde{y} = 1$ will be close to the sample proportion $\displaystyle {{y} \over {n}}$. Fundamentally, however, as $n$ grows larger, we have $\displaystyle {{y+1} \over {n+2}} \simeq {{y} \over {n}}$, so whether one is a frequentist or a Bayesian, both eventually arrive at similar estimates as the sample increases. On the other hand, if we consider the state in which no trial has been conducted at all, that is the case $n=0$, we obtain $\displaystyle p(\tilde{y} = 1| y) = {{1} \over {2}}$, which matches well with the uniform distribution that serves as the prior. This is also a mathematical demonstration that our inference started from $\displaystyle \theta = {{1} \over {2}}$.

Proof

$$ \begin{align*} p (\tilde{y} = 1| y) =& \int_{0}^{1} p(\tilde{y} = 1| \theta , y) p (\theta | y) d \theta \\ =& \int_{0}^{1} p(\tilde{y} = 1| \theta ) p (\theta | y) d \theta \\ =& \int_{0}^{1} \theta p (\theta | y) d \theta \\ =& E( \theta | y) \end{align*} $$

Since $\theta p (\theta | y)$ follows the beta distribution $\beta (y+1 , n-y+1)$,

$$ E( \theta | y) = {{y+1} \over {(y+1) + (n-y+1)}} = {{y+1} \over {n+2}} $$

■