📂Probability Distribution

Categorical Distribution

Definition¹

Given a sample space with $k (\ge 2)$ categories, $\Omega = \left\{ 1, 2, \dots, k \right\}$, and a probability vector $\mathbf{p} = (p_{1}, \dots, p_{k})$, the discrete probability distribution with the following probability mass function is called the Categorical distribution.

$$p(x = i) = p_{i}, \qquad x \in \left\{ 1, 2, \dots, k \right\}$$

Description

The probability of each of the $k$ categories occurring is represented by $\mathbf{p} = (p_{1}, \dots, p_{k})$. Therefore, $\mathbf{p}$ must satisfy the following condition.

$$\sum_{i=1}^{k} p_{i} = 1, \qquad p_{i} \ge 0$$

If the Bernoulli distribution is compared to “flipping a coin once,” the Categorical distribution can be compared to “rolling a die once.”

$\Omega = \Big\{$ $\includegraphics[height=2em]{https://img1.daumcdn.net/thumb/R1280x0/?scode=mtistory2&fname=https\includegraphics[height=2em]{https://img1.daumcdn.net/thumb/R1280x0/?scode=mtistory2&fname=https\includegraphics[height=2em]{https://img1.daumcdn.net/thumb/R1280x0/?scode=mtistory2&fname=https\includegraphics[height=2em]{https://img1.daumcdn.net/thumb/R1280x0/?scode=mtistory2&fname=https\includegraphics[height=2em]{https://img1.daumcdn.net/thumb/R1280x0/?scode=mtistory2&fname=https\includegraphics[height=2em]{https://img1.daumcdn.net/thumb/R1280x0/?scode=mtistory2&fname=https$ $\Big\}$

$$\mathbf{p} = \left( \dfrac{1}{6}, \dfrac{1}{6}, \dfrac{1}{6}, \dfrac{1}{6}, \dfrac{1}{6}, \dfrac{1}{6} \right)$$

The following notation is used.

$$\operatorname{Cat}(k; p_{1}, \dots, p_{k}) = \operatorname{Cat}(k; \mathbf{p})$$

The Categorical distribution can be considered a generalization of categories from the Bernoulli distribution to $k$ categories. Further generalizing to $n$ trials leads to the Multinomial distribution.

Category Trials	$1$ times	$n$ times
$2$ items	Bernoulli distribution	Binomial distribution
$k$ items	Categorical distribution	Multinomial distribution

The probability mass function can also be expressed as follows.

$$p(j) = \prod\limits_{i=1}^{k} p_{i}^{\delta_{ji}} = \sum\limits_{i=1}^{k} \delta_{ji} p_{i}, \qquad j \in \left\{ 1, 2, \dots, k \right\}$$

$\delta_{ji}$ refers to the Kronecker delta.

Meanwhile, the sample space can be viewed as the standard basis of Euclidean space, and each realization can be considered as a one-hot vector. In this case, with a random vector $\mathbf{x} = (x_{1}, \dots, x_{k})$ satisfying the probability mass function, the Categorical distribution can be expressed as $\operatorname{Cat}(\mathbf{x}; \mathbf{p})$.

$$x_{i} \in \left\{ 0, 1 \right\}, \qquad \sum_{i=1}^{k} x_{i} = 1$$

$$p(\mathbf{x}) = p(x_{1}, \dots, x_{k}) = \prod\limits_{i=1}^{k} p_{i}^{x_{i}}$$