Sinusoidal Positional Encoding
Definition
For a constant $C > 0$ and a data sequence $\mathbf{X} = \begin{bmatrix} \mathbf{x}_{1} & \cdots & \mathbf{x}_{n} \end{bmatrix}$ $(\mathbf{x}_{\text{pos}} \in \mathbb{R}^{d}$, $d$ is even$)$, the function $\operatorname{pe} : \mathbb{N} \to \mathbb{R}^{d}$ defined as below, or the mapping $\mathbf{x}_{\text{pos}} \mapsto \mathbf{x}_{\text{pos}} + \operatorname{pe}(\text{pos})$, is called the sinusoidal positional encoding. The indices of the components are counted from $1$, and $i$ ranges over $1 \le i \le d/2$.
$$ \begin{align*} \left[ \operatorname{pe}(\text{pos}) \right]_{2i-1} &:= \sin \left( \text{pos} / C^{2(i-1)/d} \right) \\ \left[ \operatorname{pe}(\text{pos}) \right]_{2i} &:= \cos \left( \text{pos} / C^{2(i-1)/d} \right) \end{align*} $$
Explanation
It became famous for being used as the positional encoding of the Transformer. In 🔒(26/08/27)Attention Is All You Need1, the paper that proposed the Transformer, $C = 10000$ and $d = d_{\text{model}} = 512$ were used. Writing out $\operatorname{pe}(\text{pos})$ as a vector gives the following.
$$ \operatorname{pe}(\text{pos}) = \begin{bmatrix} \sin \left( \text{pos} \right) \\ \cos \left( \text{pos} \right) \\ \sin \left( \text{pos} / C^{2/d} \right) \\ \cos \left( \text{pos} / C^{2/d} \right) \\ \vdots \\ \sin \left( \text{pos} / C^{(d-2)/d} \right) \\ \cos \left( \text{pos} / C^{(d-2)/d} \right) \end{bmatrix} $$
The odd-numbered components are sines and the even-numbered components are cosines, so that each pair of components $(2i-1, 2i)$ corresponds to one sinusoid whose wavelength is $2\pi C^{2(i-1)/d}$. The wavelengths form a geometric sequence from $2\pi$ up to $2\pi C^{(d-2)/d} \approx 2\pi C$, so the earlier the component is, the faster it oscillates, and the later it is, the slower it oscillates. The figure below plots the odd-numbered component $\left[ \operatorname{pe}(\text{pos}) \right]_{2i-1} = \sin \left( \text{pos} / 10000^{2(i-1)/512} \right)$ as a function of the position $\text{pos}$ for each of $i = 1, 10, 50, 100$, when $C = 10000$ and $d = 512$. One can see that the wavelength gets longer as $i$ grows.
The following is a heatmap obtained by lining up $\operatorname{pe}(1), \dots, \operatorname{pe}(100)$ as columns when $C = 10000$ and $d = 128$. The lower part, where the indices are small, oscillates fast whenever the position changes, while the upper part changes slowly.
In the Transformer it was used in order to include information about where each word is located within a sentence. If $\mathbf{x}_{\text{pos}}$ is a vector holding the meaning of a word, then $\operatorname{pe}(\text{pos})$ can be understood as a perturbation or noise carrying the information of which position the word appears at, and it allows the same word to be distinguished as different vectors depending on the position where it appears.
The role of the sinusoidal positional encoding is not only the above. For instance, for a natural number $k$, placing the following mapping at the very front of a neural network naturally implements a periodic function. Since every component is a function with period $2\pi$, no matter which neural network $f : \mathbb{R}^{2k} \to \mathbb{R}$ is composed after this mapping, the whole automatically becomes a function with period $2\pi$. NeRF, which represents three-dimensional scenes with a neural network, is a representative case that made use of this2.
$$ x \mapsto \begin{bmatrix} \sin x \\ \cos x \\ \sin 2x \\ \cos 2x \\ \vdots \\ \sin kx \\ \cos kx \end{bmatrix} $$