📂Machine Learning

seq2seq Model

Definition

The composition of an encoder, which maps a sequence of input vectors $\left\{ \mathbf{x}_{1}, \mathbf{x}_{2}, \dots, \mathbf{x}_{T_{x}} \right\}$ $(\mathbf{x}_{t} \in \mathbb{R}^{n})$ to a context vector $\mathbf{c} \in \mathbb{R}^{d_{c}}$, and a decoder, which maps the context vector to a sequence of output vectors $\left\{ \mathbf{y}_{1}, \mathbf{y}_{2}, \dots, \mathbf{y}_{T_{y}} \right\}$ $(\mathbf{y}_{t} \in \mathbb{R}^{p})$, is called a sequence-to-sequence model, or seq2seq for short. Expressing the encoder and decoder each as functions gives the following.

$$ \begin{align*} \operatorname{Encoder} : \left( \mathbb{R}^{n} \right)^{T_{x}} &\to \mathbb{R}^{d_{c}} \\ \left\{ \mathbf{x}_{t} \right\}_{t=1}^{T_{x}} &\mapsto \mathbf{c} \end{align*} $$

$$ \begin{align*} \operatorname{Decoder} : \mathbb{R}^{d_{c}} &\to \left( \mathbb{R}^{p} \right)^{T_{y}} \\ \mathbf{c} &\mapsto \left\{ \mathbf{y}_{t} \right\}_{t=1}^{T_{y}} \end{align*} $$

The seq2seq model is the composition of these two.

$$ \operatorname{seq2seq} := \operatorname{Decoder} \circ \operatorname{Encoder} : \left\{ \mathbf{x}_{1}, \mathbf{x}_{2}, \dots, \mathbf{x}_{T_{x}} \right\} \mapsto \left\{ \mathbf{y}_{1}, \mathbf{y}_{2}, \dots, \mathbf{y}_{T_{y}} \right\} $$

• Here $T_{x}, T_{y} \in \mathbb{N}$ are the lengths of the input sequence and the output sequence, respectively, and they need not be equal.

Explanation

The seq2seq model is a neural network model with an encoder-decoder structure, proposed for 🔒(26/07/28)machine translation in the paper “Sequence to Sequence Learning with Neural Networks” presented at NIPS 2014¹.

Recurrent neural networks (RNNs) are typically used as the encoder and decoder; the original paper specifically used LSTMs. The encoder reads the input sequence in temporal order, updating its hidden state $\mathbf{h}_{t}$. In general, the context vector can be defined as a function of the hidden states, $\mathbf{c} = q(\mathbf{h}_{1}, \dots, \mathbf{h}_{T_{x}})$; the original paper makes the simplest choice among these and uses the hidden state at the last time step directly as the context vector ($\mathbf{c} = \mathbf{h}_{T_{x}}$). The decoder takes $\mathbf{c}$ as its initial hidden state and generates output vectors one time step at a time, where the output of the previous time step is fed in as the input of the next time step. In practice, the output length $T_{y}$ is not fixed in advance; rather, the decoder keeps generating until it outputs a special 🔒(26/07/30)token that signifies the end of the sequence.

A characteristic of the seq2seq model, as mentioned in the definition, is that the input sequence and the output sequence need not have the same length ($T_{x} \ne T_{y}$). Considering machine translation, its representative application, this is an obviously required property: if we split the source text and its translation into tokens and 🔒(26/08/01)embed them, as when translating “나는 학생이다” into “I am a student”, the lengths of the two sequences are generally different.

Meanwhile, the only information passed between the encoder and the decoder is the single context vector $\mathbf{c}$ of fixed dimension $d_{c}$. No matter how long the input becomes, its content must be squeezed into a single vector, so the longer the input, the greater the information loss—a bottleneck that can be seen as a structural limitation of the seq2seq model. To solve this problem, 🔒(26/07/22)attention lets the decoder refer back to all of the encoder’s hidden states at every time step², and going further, the 🔒(26/07/24)transformer³ discards the recurrent structure entirely and processes sequences with attention alone.

See Also

• Recurrent Neural Network
• Encoder and Decoder
• 🔒(26/07/22)Attention
• 🔒(26/07/24)Transformer