Embedding in Machine Learning
Definition
In machine learning, an embedding means a mapping from a dataset $X$ into a vector space $\mathbb{R}^{n}$, or the act of performing such a mapping itself.
$$ \operatorname{Embedding}: X \to \mathbb{R}^{n} $$
Explanation
Embedding is originally a mathematical term. Although its definition differs slightly from field to field, it commonly refers to an 'injective map that places one object inside another while preserving structure'.
- Embedding in functional analysis refers to the situation where a normed space $X$ is a subspace of a normed space $Y$ and the identity operator $I : X \to Y$ is continuous.
- Embedding in topology is a continuous function that becomes a homeomorphism onto its own image.
- The embedding theorem for $L^{p}$ spaces states that on a region of finite volume, when $p \le q$, $L^{q}$ becomes a subset of $L^{p}$.
- A graph embedding is a function that sends a graph, its nodes, or its edges into Euclidean space, and it is quite close to the machine learning usage.
The machine learning notion of embedding inherits this idea but is far looser. Once we map data to a vector made of numbers, whatever the data originally was, it becomes an object we can compute with. This means we can measure the magnitude of that vector, or measure the distance between two vectors. For this, the embedding function must embed the data well, and in deep learning the embedding function itself becomes an object to be learned. For example, we can put a requirement such as 'send similar data to nearby vectors and different data to far-apart vectors' into the loss function and train on it.
Which embedding to use is an important issue, and this is well revealed by the limitations of one-hot encoding in natural language processing. The simplest way to turn categorical data into vectors is one-hot encoding, which assigns each item a single standard basis vector $\mathbf{e}_{i}$. However, in natural language processing, where the size of the vocabulary set $V$ ranges from tens of thousands to hundreds of thousands, this means a $\left| V \right|$-dimensional sparse vector in which only one component is $1$ and all the rest are $0$. A more serious problem is that distinct one-hot vectors are always orthogonal, that is, if $i \ne j$ then $\mathbf{e}_{i} \cdot \mathbf{e}_{j} = 0$. 'Puppy' and 'dog' are as unrelated to each other as 'puppy' and 'determinant', so the representation itself carries no similarity information at all. That is why the commonly used method is an embedding that converts this into a dense vector of much lower dimension $n \ll \left| V \right|$, giving meaning to the distance and direction between vectors.
Natural Language Processing
In 🔒(26/07/30)natural language processing, embedding is also called text vectorization and comes in various forms. Below, we take the vocabulary dictionary $V = \{$ puppy, dog, cat $\}$ (size $3$, with the component order following this order) as a common example.
One-hot encoding: The most basic method, which assigns an index to each word in a pre-built vocabulary dictionary and maps it to a standard basis vector. It is essentially the same as assigning integer indices, and if there are many words to use, it is practically hard to use due to the sparsity/orthogonality problems seen above.
Word One-hot vector Integer puppy $(1, 0, 0)$ $1$ dog $(0, 1, 0)$ $2$ cat $(0, 0, 1)$ $3$ count vectorization: A method that represents a document by the number of occurrences of each vocabulary word, and its representative form is bag-of-words. For example, the sentence "dog cat dog" is embedded as $(0, 2, 1)$ by counting the components, in order, as the frequencies of puppy, dog, and cat.
word embedding: This refers to representing a word as a low-dimensional dense vector, with methods such as Word2Vec, fastText, and GloVe. The values below are arbitrary examples for illustration only and are not actual learned values.
Word Embedding (example) puppy $(0.8, 0.1)$ dog $(0.7, 0.2)$ cat $(-0.6, 0.5)$ In this way, 'puppy' and 'dog', which have similar meanings, end up with vectors close to each other and far from 'cat', so that unlike one-hot encoding, the meaning of a word is contained in the distance between vectors.