The Softmax Function in Deep Learning
Definition
Let $\mathbf{x} = (x_{1} , \cdots , x_{n}) \in \mathbb{R}^{n}$. For $\sigma_{j} ( \mathbf{x} ) = \dfrac{ e^{x_{j}} }{ {\sum_{i=1}^{n} e^{x_{i}} }}$, the function $\sigma : \mathbb{R}^{n} \to (0,1)^{n}$ defined by $\sigma ( \mathbf{x} ) := \left( \sigma_{1} (\mathbf{x}) , \cdots , \sigma_{n} (\mathbf{x} ) \right)$ is called the softmax.
Explanation
The softmax function is a kind of activation function, and it has the distinctive feature that its domain is $\mathbb{R}^{n}$. This is because it is used to take a vector as input and normalize its values. For any $\mathbf{x} \in \mathbb{R}$, every component of $\sigma ( \mathbf{x} )$ is a value between $0$ and $1$, and summing them all gives exactly $1$. This property resembles probability, and indeed it is used in the output layer when implementing an artificial neural network for classification problems.
The figure below shows how the input vector $\mathbf{x} = (2, -0.5, 0.5, -1, 3)$ passes through the softmax and is normalized into a probability distribution whose sum is $1$.
