Layer Normalization
Definition1
Let the output of some layer of a neural network be $\mathbf{a} = \begin{bmatrix} a_{1} & \cdots & a_{H} \end{bmatrix}^{\mathsf{T}} \in \mathbb{R}^{H}$. The mean $\mu$ and the standard deviation $\sigma$ of the components of $\mathbf{a}$ are as follows.
$$ \mu := \frac{1}{H} \sum_{i=1}^{H} a_{i} , \qquad \sigma := \sqrt{ \frac{1}{H} \sum_{i=1}^{H} \left( a_{i} - \mu \right)^{2} } $$
For a gain $\mathbf{g} \in \mathbb{R}^{H}$ and a bias $\mathbf{b} \in \mathbb{R}^{H}$, the standardization function $\operatorname{LN} : \mathbb{R}^{H} \to \mathbb{R}^{H}$ defined as follows is called layer normalization.
$$ \operatorname{LN}(\mathbf{a}) := \mathbf{g} \odot \dfrac{ \mathbf{a} - \mu \mathbf{1} }{\sigma} + \mathbf{b} $$
Here, $\mathbf{1} \in \mathbb{R}^{H}$ is the vector whose components are all $1$, and $\odot$ is the Hadamard product. Written componentwise, $\left[ \operatorname{LN}(\mathbf{a}) \right]_{i} = \dfrac{g_{i}}{\sigma} \left( a_{i} - \mu \right) + b_{i}$.
Explanation
A transformation such as $Z = \dfrac{X - \mu}{\sigma}$, using the mean and standard deviation of $X$, is called standardization in statistics, but in machine learning it is called normalization. There is a strong tendency to call any transformation applied to data a normalization, without distinction.
In the paper that proposed it, the gain and bias are written as $\mathbf{g}$ and $\mathbf{b}$, but the notation $\boldsymbol{\gamma}$, $\boldsymbol{\beta}$ is also commonly used. Meanwhile, in actual implementations, a small $\epsilon > 0$ is added to the denominator to prevent division by $0$ when $\sigma = 0$, written as follows.
$$ \operatorname{LN}(\mathbf{a}) := \mathbf{g} \odot \dfrac{ \mathbf{a} - \mu \mathbf{1} }{\sqrt{\sigma^{2} + \epsilon}} + \mathbf{b} $$
While 🔒(26/09/02)batch normalization groups the input data into particular batches and normalizes per batch, layer normalization normalizes with respect to a single input datum. In other words, in batch normalization all data in a mini-batch share the same $\mu$ and $\sigma$ for the same feature, whereas in layer normalization each datum has its own $\mu$ and $\sigma$. Layer normalization can be applied as is even in cases where normalizing along the batch direction is difficult, such as recurrent neural networks where the sequence length differs from datum to datum, and this is also why the Transformer uses layer normalization rather than batch normalization.
A visual comparison of how it works relative to batch normalization is shown below. Let $\mathbf{a}^{k}$ be the output for the $k$-th datum.
See Also
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization. arXiv preprint arXiv:1607.06450 (2016). https://arxiv.org/abs/1607.06450 ↩︎