📂Mathematical Statistics

Laplace's Succession Rule

Theorem ¹

Let’s say the prior distribution of the binomial model $\displaystyle p(y | \theta) = \binom{ n }{ y} \theta^{y} (1- \theta)^{n-y}$ follows a uniform distribution $U (0,1)$ and the posterior distribution follows a beta distribution $\beta (y+1 , n-y+1)$, hence $p( \theta | y ) \sim \theta^{y} (1- \theta)^{n-y}$. Then, for the data obtained so far $y$, the probability of observing a new $\tilde{y}$ being $1$ is $$p(\tilde{y} = 1| y) = {{y+1} \over {n+2}}$$

Explanation

From a frequentist’s perspective, the probability of $\tilde{y} = 1$ will be close to the sample rate $\displaystyle {{y} \over {n}}$. However, fundamentally as $n$ increases, since $\displaystyle {{y+1} \over {n+2}} \simeq {{y} \over {n}}$, both frequentists and Bayesians will eventually make similar estimates as the sample size increases. On the other hand, if we consider the case of having conducted no trials, that is $n=0$, it matches well with the uniform distribution as the prior, which is $\displaystyle p(\tilde{y} = 1| y) = {{1} \over {2}}$. This effectively demonstrates, through formulas, that our inference started from $\displaystyle \theta = {{1} \over {2}}$.

Proof

$$\begin{align*} p (\tilde{y} = 1| y) =& \int_{0}^{1} p(\tilde{y} = 1| \theta , y) p (\theta | y) d \theta \\ =& \int_{0}^{1} p(\tilde{y} = 1| \theta ) p (\theta | y) d \theta \\ =& \int_{0}^{1} \theta p (\theta | y) d \theta \\ =& E( \theta | y) \end{align*}$$

Since $\theta p (\theta | y)$ follows the beta distribution $\beta (y+1 , n-y+1)$,

$$E( \theta | y) = {{y+1} \over {(y+1) + (n-y+1)}} = {{y+1} \over {n+2}}$$

■