📂Machine Learning

Luong Attention

Introduction

Suppose we are given three matrices $\mathbf{Q} \in M^{d_{k} \times n}$, $\mathbf{K} \in M^{d_{k} \times m}$, $\mathbf{V} \in M^{d_{v} \times m}$, called the query, the key, and the value respectively, together with a score function $f : M^{d_{k} \times n} \times M^{d_{k} \times m} \to M^{m \times n}$ for the queries and the keys. Attention refers to the function defined as follows.

$$ \begin{align*} \operatorname{Attention}: M^{d_{k} \times n} \times M^{d_{k} \times m} \times M^{d_{v} \times m} &\to M^{d_{v} \times n} \\ (\mathbf{Q},\mathbf{K},\mathbf{V}) &\mapsto \mathbf{V} \operatorname{Softmax}\left(f(\mathbf{Q},\mathbf{K})\right) \end{align*} $$

Here $\operatorname{Softmax}$ denotes the function that, for a given matrix $\mathbf{X} = \begin{bmatrix} \mathbf{x}_{1} & \cdots & \mathbf{x}_{N}\end{bmatrix}$, applies the softmax $\operatorname{softmax}$ to each column vector so that each column sums to $1$.

$$ \operatorname{Softmax}(\mathbf{X}) := \begin{bmatrix} \underset{\vert}{\overset{\vert}{\operatorname{softmax}(\mathbf{x}_{1})}} & \cdots & \underset{\vert}{\overset{\vert}{\operatorname{softmax}(\mathbf{x}_{N})}} \end{bmatrix} $$

The kinds of attention are distinguished by how $f$ is defined, and the forms of attention proposed by Luong et al. in the paper "Effective approaches to attention-based neural machine translation"¹ are as follows.

Definitions

• Dot-product attention:

  An attention function of the following form is called dot-product attention.

  $$ \operatorname{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V}) = \mathbf{V} \operatorname{Softmax} (\mathbf{K}^{\mathsf{T}} \mathbf{Q}) $$

• General attention:

  An attention function of the following form is called Luong general attention.

  $$ \operatorname{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V}) = \mathbf{V} \operatorname{Softmax} (\mathbf{K}^{\mathsf{T}} \mathbf{W} \mathbf{Q}) $$

Explanation

Luong’s paper distinguishes these under the names 'dot' and 'general', respectively. The form introduced there as 'concat' is a name given to the attention introduced by Bahdanau et al.

Dot-Product Attention

This takes the score function of a query vector $\mathbf{q}$ and a key vector $\mathbf{k}$ to be the inner product.

$$ \operatorname{score}(\mathbf{q}, \mathbf{k}) = \mathbf{k}^{\mathsf{T}} \mathbf{q} $$

$$ f(\mathbf{Q}, \mathbf{K}) = \mathbf{K}^{\mathsf{T}} \mathbf{Q} =\begin{bmatrix} \mathbf{k}_{1} \cdot \mathbf{q}_{1} & \mathbf{k}_{1} \cdot \mathbf{q}_{2} & \cdots & \mathbf{k}_{1} \cdot \mathbf{q}_{n} \\ \mathbf{k}_{2} \cdot \mathbf{q}_{1} & \mathbf{k}_{2} \cdot \mathbf{q}_{2} & \cdots & \mathbf{k}_{2} \cdot \mathbf{q}_{n} \\ \vdots & \vdots & \ddots & \vdots \\ \mathbf{k}_{m} \cdot \mathbf{q}_{1} & \mathbf{k}_{m} \cdot \mathbf{q}_{2} & \cdots & \mathbf{k}_{m} \cdot \mathbf{q}_{n} \end{bmatrix} $$

The scaled dot-product attention introduced in the Transformer paper multiplies this by the scaling factor $1/\sqrt{d_{k}}$ as below.

$$ \operatorname{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V}) = \mathbf{V} \operatorname{Softmax} \left( \dfrac{\mathbf{K}^{\mathsf{T}} \mathbf{Q}}{\sqrt{d_{k}}} \right) $$

General Attention

This takes the score function of a query vector $\mathbf{q}$ and a key vector $\mathbf{k}$ to be a bilinear transformation.

$$ \operatorname{score}(\mathbf{q}, \mathbf{k}) = \mathbf{k}^{\mathsf{T}} \mathbf{W} \mathbf{q} $$

$$ f(\mathbf{Q}, \mathbf{K}) = \mathbf{K}^{\mathsf{T}} \mathbf{W} \mathbf{Q} =\begin{bmatrix} \mathbf{k}_{1}^{\mathsf{T}} \mathbf{W} \mathbf{q}_{1} & \mathbf{k}_{1}^{\mathsf{T}} \mathbf{W} \mathbf{q}_{2} & \cdots & \mathbf{k}_{1}^{\mathsf{T}} \mathbf{W} \mathbf{q}_{n} \\ \mathbf{k}_{2}^{\mathsf{T}} \mathbf{W} \mathbf{q}_{1} & \mathbf{k}_{2}^{\mathsf{T}} \mathbf{W} \mathbf{q}_{2} & \cdots & \mathbf{k}_{2}^{\mathsf{T}} \mathbf{W} \mathbf{q}_{n} \\ \vdots & \vdots & \ddots & \vdots \\ \mathbf{k}_{m}^{\mathsf{T}} \mathbf{W} \mathbf{q}_{1} & \mathbf{k}_{m}^{\mathsf{T}} \mathbf{W} \mathbf{q}_{2} & \cdots & \mathbf{k}_{m}^{\mathsf{T}} \mathbf{W} \mathbf{q}_{n} \end{bmatrix} $$