📂Machine Learning

Recurrent Neural Network

Definition¹

Let $\overline{\sigma}$ be function that applies the activation function $\sigma$ element-wise. That is, for a scalar function $\sigma : \mathbb{R} \to \mathbb{R}$, it is the vector function defined as follows.

$$ \overline{\sigma}(\mathbf{z}) = \begin{bmatrix} \sigma(z_{1}) \\ \sigma(z_{2}) \\ \vdots \\ \sigma(z_{m}) \end{bmatrix} \qquad \text{where } \mathbf{z} = \begin{bmatrix} z_{1} \\ z_{2} \\ \vdots \\ z_{m} \end{bmatrix} $$

Suppose a sequence of input vectors $(\mathbf{x}_{1}, \mathbf{x}_{2}, \dots, \mathbf{x}_{T})$ $(\mathbf{x}_{t} \in \mathbb{R}^{n})$ is given. Let $\mathbf{W} \in M^{m \times m}$, $\mathbf{U} \in M^{m \times n}$ be weights, and $\mathbf{b} \in \mathbb{R}^{m}$ a bias. A neural network that defines the hidden state $\mathbf{h}_{t} \in \mathbb{R}^{m}$ by the following recurrence relation is called a recurrent neural network (RNN).

$$ \mathbf{h}_{t} = \overline{\sigma}\left( \mathbf{W} \mathbf{h}_{t-1} + \mathbf{U} \mathbf{x}_{t} + \mathbf{b} \right), \qquad \mathbf{h}_{0} = \mathbf{0} $$

The output $\mathbf{y}_{t} \in \mathbb{R}^{p}$ at each time step $t$ is obtained by composing a fully connected layer with the hidden state. Letting $\mathbf{V} \in M^{p \times m}$ be a weight and $\mathbf{c} \in \mathbb{R}^{p}$ a bias,

$$ \mathbf{y}_{t} = \mathbf{V} \mathbf{h}_{t} + \mathbf{c} $$

Explanation

While a multilayer perceptron (MLP) was a function that takes one vector as input and produces one vector, a recurrent neural network is a function that handles a sequence of vectors. It was devised to process data in which order carries meaning, such as sentences in natural language, time series, and speech signals.

A neural network in which information flows in only one direction, from input to output, with no connections coming back—like an MLP or a convolutional neural network—is called a feedforward neural network. This term is used especially often in contrast to recurrent neural networks, which have a recurrent structure in which the hidden state feeds back into itself. In the early days it referred practically only to MLPs, but nowadays it can be taken to refer collectively to any neural network that is not a recurrent neural network.

The essence of a recurrent neural network is that the hidden state $\mathbf{h}_{t}$ depends simultaneously on the current input $\mathbf{x}_{t}$ and the previous hidden state $\mathbf{h}_{t-1}$. Since $\mathbf{h}_{t-1}$ in turn depends on $\mathbf{h}_{t-2}$, the hidden state serves as a kind of memory that summarizes and holds the past inputs. Unrolling the recurrence relation reveals that $\mathbf{h}_{t}$ is a function of all inputs up to time $t$.

$$ \mathbf{h}_{t} = \overline{\sigma}\left( \mathbf{W} , \overline{\sigma}\left( \mathbf{W} , \overline{\sigma}\left( \mathbf{W} , \overline{\sigma}( \cdots ) + \mathbf{U} \mathbf{x}_{t-2} + \mathbf{b} \right) + \mathbf{U} \mathbf{x}_{t-1} + \mathbf{b} \right) + \mathbf{U} \mathbf{x}_{t} + \mathbf{b} \right) $$

Weight Sharing

What deserves attention is that the weights $\mathbf{W}, \mathbf{U}$ and the bias $\mathbf{b}$ appearing in the equation above are identical at every time step. That is, rather than using a different layer at each time step, a single layer is applied repeatedly. Just as a convolutional layer shares one kernel across the entire space, a recurrent neural network shares one layer along the time axis.

Thanks to this, a recurrent neural network has only a fixed number of parameters regardless of the length $T$ of the input sequence. This is also why inputs of different lengths can be processed by the same neural network.

Vanishing Gradient

In the process of computing the hidden state in a recurrent neural network, the same weight $\mathbf{W}$ is multiplied repeatedly. Consequently, during backpropagation the gradient contains a term related to $\mathbf{W}$ multiplied roughly $T$ times over. Depending on the magnitude of $\mathbf{W}$, this value shrinks exponentially to $0$, causing the vanishing gradient, or diverges. As a result, the basic recurrent neural network has difficulty learning relationships between inputs that are far apart in time. LSTM and GRU alleviate this long-term dependency problem with a gate structure, and nowadays attention-based Transformers, rather than recurrent structures, are predominantly used for sequential data.

See Also

• What Is an Artificial Neural Network?
• Fully Connected Layer (Linear Layer, Dense Layer)
• Convolutional Layer