📂Mathematical Statistics

Bayesian Paradigm

Buildup

Statistics can be described as ’the discipline that studies methods for grasping parameters’. It would be ideal if we could estimate parameters accurately through formulas or laws, just as we measure some physical quantity, but since that is practically impossible, we merely use assumptions and samples to find ‘what is presumed to be the parameter’. If we are interested in the height $X$ of Korean men, we assume $X \sim N ( \theta , \sigma^2 )$, find $\displaystyle \hat{\theta} = \overline{x} = {{1} \over {n}} \sum_{k = 1}^{ n } x_{k}$, and grasp it as $\theta = \hat{\theta}$. This kind of estimation is based on a fairly easy and simple concept.

Frequentist

The sample we have is obtained randomly from the population, and whatever sample we obtained, as long as the method was fair, we make no particular distinction among samples of the same size. Of course, in reality they are different samples, but whether a sample represents the population well is purely a matter of luck. The only certain thing is that a larger sample is better than a smaller one. Naturally, even for observations we did not obtain, we do not think they would differ greatly from the current sample. We simply failed to obtain them; if those observations were not similar to the current sample either, there would be no point in doing statistical analysis. Such inference starts from the expectation that the sample and the population are not greatly different, and the larger the sample, the closer that expectation comes to certainty. In this way, inference that considers not only the data obtained so far but also the data to be obtained in the future, or not yet obtained, is called frequentist inference. From the viewpoint that accuracy increases as the sample size (Frequency) grows larger, this naming can be said to be justified.

Bayesian

On the other hand, Bayesian inference considers only the sample obtained up to the present. Through Bayes’ theorem, the prior distribution is merely changed into the posterior distribution. It does think that the parameter has some distribution, but it does not particularly assume that this is accurate. Before beginning the analysis, one may posit any prior distribution based on an expert’s opinion or subjective experience. Even if that distribution changes upon obtaining a new sample, it does not mind. Even if the data to be obtained in the future differs greatly from the thinking so far, it simply updates accordingly. The only certain thing is that the posterior distribution at the completion of the analysis is the result obtained by reflecting the sample onto the prior distribution.

What is the Bayesian Paradigm? ¹

The Bayesian paradigm is composed as follows:

• (1): Determining the prior distribution of the parameter
• (2): Calculation through Bayes’ theorem
• (3): Estimating the parameter using the posterior distribution

If the prior distribution of the parameter $\theta$ is $\pi (\theta)$ and the observation is $y$, then by Bayes’ theorem, $$ p ( \theta | y ) = {{ p(y | \theta ) \pi (\theta ) } \over { p(y) }} $$ holds. Here, the probability distribution $p ( \theta | y )$ of the parameter with the data reflected is called the posterior distribution.

Example

Let us think of one simple example. Suppose there is a friend Adam who is frequently late for appointments.

If the time by which Adam breaks appointments follows a normal distribution $N ( 10 , 5^2 )$ with a mean of 10 minutes and a standard deviation of 5 minutes, then given the same data, when Adam is late for an appointment, the frequentist and the Bayesian would say the following:

• Frequentist: “Adam is a guy who is always 10 minutes late.”
• Bayesian: “From what I’ve seen, Adam tends to be about 10 minutes late.”

The frequentist inferred that the time Adam is late is 10 minutes on average, and since that is Adam’s essence, he expects that Adam has been about 10 minutes late so far and will be about 10 minutes late in the future as well. The Bayesian, thinking that based on what has been seen so far the probability of Adam being 10 minutes late is the highest, expects that the time he is late this time will also be 10 minutes.

At first glance, they seem to say the same thing. That is understandable, because the frequentist and the Bayesian merely have different viewpoints, and there is no difference in that they draw statistical inferences. The difference arises the next time, when Adam arrives exactly on time for the appointment:

• Frequentist: “Adam arriving on time is a rare case, one whose probability is less than 3%.”
• Bayesian: “So Adam comes early sometimes. Will he come at this time next time too?”

And at the next appointment, if we ask whether Adam will be able to show up on time, the two people’s answers will surely differ:

• Frequentist: “It’s hard to say Adam has changed. His showing up early this time was something that could sufficiently happen.”
• Bayesian: “The probability that Adam shows up late is still high, but it is also true that the probability of him showing up on time has risen.”

The frequentist merely checks whether the newly obtained observation conforms to the conclusion already drawn, but the Bayesian can be seen as having obtained a new posterior distribution by immediately updating the previously held conclusion. The ease of such sequential analysis is the greatest characteristic distinguishing it from the frequentist and, at the same time, an inherent advantage of Bayesian inference.