Difference Between Credible Intervals and Confidence Intervals
Summary
The difference between credible intervals and confidence intervals can truly be seen as the difference between Bayesians and frequentists.
- Confidence interval (frequentist): the parameter is a fixed constant, and the confidence interval is obtained at random.
- Credible interval (Bayesian): the parameter is also a variable with a distribution, and the credible interval is likewise obtained from the posterior distribution.
Confidence Interval
In classical statistics, what a $95 \% $ confidence interval $[a , b]$ for a parameter $\mu$ means is that if there are $100$ confidence intervals constructed by the same method, about $95$ of them satisfy $\mu \in [a,b]$. Expressed simply as a formula, this is $p ( \mu \in [a,b] ) = 95 \% $, but contrary to common sense, its interpretation is remarkably subtle. It is certainly the same formula, yet it can be read with the following different nuances.
- The probability that $\mu$ is contained in $[a,b]$ is $95 \% $. (X)
- The probability that $[a,b]$ contains $\mu$ is $95 \% $. (O)
Of course, the two sentences above say the same thing, but the reader must be able to sense the difference in nuance rather than the words themselves. In other words, it is not that the probability of $\mu$ lying in the confidence interval $[a , b]$ is $95 \% $; rather, according to the procedure by which confidence intervals are constructed, among $n$ $95 \% $ confidence intervals $[a , b]$, those containing $\mu$ make up about $95 \% $ of the total.
The parameter $\mu$ that we want to know is itself a constant whose distribution is unknown, whereas $a,b$, the upper and lower bounds of the confidence interval, do have known distributions, which is why we can consider the case where $a < \mu$ and $ \mu < b$. Because of this subtle difference, confidence intervals do not exactly match our intuition. That is why even explaining such a commonsense, easy concept comes with the messy explanation that if you make several of them, some of them contain it and some do not.
To begin with, if we take $\mu$ as the reference point, no distribution has ever been defined for $\mu$, so the very expression $p ( \mu \in [a,b] )$ makes no sense. If someone abruptly asked you the value of $P \left( 0 \le X \le 2 \right)$ without even telling you the distribution of $X$, you would be quite bewildered, and the wrong interpretation of confidence intervals is exactly like this.
From the frequentist point of view, the parameter $\mu$ exists as a fixed constant, and what varies with the sample is the confidence interval itself. Just as one assumes that the current sample will resemble the population, one constructs a single confidence interval and regards it as likely to be similar to the remaining confidence intervals not yet constructed.
Credible Interval
When a subset $C \subset \Theta$ of the parameter space $\Theta$ satisfies $P ( \theta \in C | y ) \ge 1 - \alpha$ for a significance level $\alpha$, $C$ is called a $100(1 - \alpha) \% $ credible interval for $\theta$ given the data $y$.
In contrast, a credible interval is, from its very definition, expressed as a probability about $\theta$. This is possible because a posterior distribution $p ( \theta | y)$ is properly assumed for the parameter $\theta$. This matches intuitively the definition of confidence/credible intervals that students of statistics have conceptually accepted.
Bayesians do not care about samples not yet obtained. They simply give the best answer based on the assumption about the posterior distribution and the samples obtained so far. Therefore, if one has constructed $100$ credible intervals, there is no need for the tedious explanation that about $95 \% $ of them have the parameter $\theta$ belonging to $C$.