What is Information Content in Classical Information Theory?

Definition¹

For a discrete random variable $X$, the information (content) $I$ of the event $X=x$ is defined as follows.

$$ \begin{equation} I(x) = -\log_{2} p(x) \end{equation} $$

$p$ is the probability mass function of $X$.

Explanation

The person who presented a quantitative definition of the abstract concept of information is Claude Shannon, the founder of digital logic circuit theory and information theory. When you first see information content defined as ‘the negative logarithm of probability’, it may not make sense, but once you hear the explanation, you will come to think that nothing could be more natural.

The value of information is greater the more unlikely the event is—that is, the rarer the probability of its occurrence. For example, the sentence “Tomorrow the head of the physics department will come to the physics building” can be considered to carry almost no information. This is because the department head will obviously come to work tomorrow. On the other hand, the sentence “Tomorrow IVE will come to the physics building” is truly top-class information. Since the probability that IVE suddenly appears in the physics building is next to none, such information can be said to be of very high value. As another example, “Tomorrow Samsung Electronics’ stock will rise by within $1 \%$ point” would be information of almost no value, but “Tomorrow Samsung Electronics’ stock will hit its upper limit” is tremendous information. Therefore, an event with a low probability of occurrence can be regarded as carrying a lot of information.

Since the function value of probability satisfies $0 \le p \le 1$, taking the negative logarithm makes the function value of information larger as $p$ becomes smaller. Thus we can naturally define information as in $(1)$.

Since the range of $-\log_{2}(x)$ is $[0, \infty)$, an event with probability $1$—that is, something that necessarily happens—has information content $0$. Also, the lower the probability of occurrence, the more the value of information keeps increasing.

The information content of the random variable $X$ itself is called entropy.

See Also

• Information defined in probabilistic information theory