📂Machine Learning

Tokens in Natural Language Processing

Definition

In natural language processing, a corpus refers to a set of natural language text data collected/refined for a specific purpose.

Let the smallest units that make up text—such as Korean syllables, Roman letters, digits, punctuation marks, and spaces—be called characters. Let $\Sigma$ be the finite set collecting all the characters we deal with, called the character set. That is, a single character is $a \in \Sigma$. A finite sequence, or tuple, of characters arranged in order is called a string. A string $s$ is written as a sequence of characters with each $a_{i} \in \Sigma$ as follows.

$$ s = (a_{1}, a_{2}, \dots, a_{N}), \qquad a_{i} \in \Sigma $$

Here $N$ is called the length of the string and is denoted by $|s| = N$. A function that divides a string into consecutive pieces from the front according to a specific criterion, sending it to a finite sequence of strings, is called a tokenizer. A tokenizer $\tau$ maps a string $s$ as follows.

$$ \tau : s = (a_{1}, \dots, a_{N}) \mapsto (\mathbf{t}_{1}, \mathbf{t}_{2}, \dots, \mathbf{t}_{M}) $$

Here each $\mathbf{t}_{j}$ is a consecutive substring of $s$, and concatenating them in order yields $s$ again. That is, there exist cut points $0 = k_{0} < k_{1} < \cdots < k_{M} = N$ satisfying the following.

$$ \mathbf{t}_{j} = (a_{k_{j-1}+1}, \dots, a_{k_{j}}), \qquad \sum_{j=1}^{M} |\mathbf{t}_{j}| = N $$

Each $\mathbf{t}_{j}$ of the token sequence obtained this way is called a token. And the act of applying a tokenizer to a string to obtain the token sequence $(\mathbf{t}_{1}, \dots, \mathbf{t}_{M})$ is called tokenization.

Explanation¹

Tokenization is a very important data preprocessing step in natural language processing. The definition above is merely a formal definition for expressing tokenization mathematically; it can simply be understood as splitting a string.

Let us look at several kinds of tokenization using the Korean sentence "그 학생은 도서관에서 책을 빌렸다." and the English sentence "The student borrowed a book from the library." as examples.

Word Tokenization

Splitting a string most simply by whitespace is called word tokenization. Splitting by whitespace, the two example sentences above are tokenized as follows. In this simplest form of word tokenization, punctuation marks are not separated out, so the period stays attached to the preceding word.

• Korean: 그 학생은 도서관에서 책을 빌렸다.
• English: The student borrowed a book from the library.

In Korean, particles are attached to words, so 학생은, 학생이, 학생을, and so on are recognized as distinct tokens. Likewise, 빌렸다., 빌렸다?, 빌렸다!, and so on are recognized as distinct tokens. If there is a sentence in which 도서관 is misspelled as 도사관, these two are also recognized as distinct tokens. Thus word tokenization is vulnerable to Korean affixes, typos, spacing errors, and the like.

Character Tokenization

Tokenizing at the level of individual characters is called character tokenization. Splitting by characters, the two example sentences above are tokenized as follows. Whitespace and punctuation marks such as the period are also single tokens.

• Korean: 그 　 학 생 은 　 도 서 관 에 서 　 책 을 　 빌 렸 다 .
• English: T h e 　 s t u d e n t 　 … (rest omitted)

In English, each letter is a token, so the token sequence becomes very long. Thus character tokenization has a very small character set—that is, vocabulary—making it robust to out-of-vocabulary words and typos, but it has the drawback that a single token cannot carry word-level meaning and the sequence becomes long. For Korean, one could go a step further and subdivide syllables into consonants and vowels, that is, jamo.

Morpheme Tokenization

Splitting by morphemes, the smallest units carrying meaning, is called morpheme tokenization. Splitting by morphemes, the two example sentences above are tokenized as follows.

• Korean: 그 학생 은 도서관 에서 책 을 빌리 었 다 .
• English: The student borrow ed a book from the library .

Since particles and endings come off separately, the problem pointed out for word tokenization—namely that 학생은, 학생이, and 학생을 all became different tokens—is alleviated, as they are split into the single morpheme 학생 and a particle. This approach suits Korean, an agglutinative language with well-developed particles and endings, particularly well. However, morpheme tokenization requires a language-specific morphological analyzer. Meanwhile, morpheme analysis of Korean can yield different boundaries depending on the analyzer.

Special Tokens

In natural language processing, the lengths of input token sequences vary, so special tokens are used to distinguish where the data begins and where it ends. Tokens that are artificially appended not as part of the text itself but to aid processing—marking the boundaries of a sentence, replacing out-of-vocabulary words, or matching the lengths of several sentences—are called special tokens. Frequently used special tokens are as follows.

Type	Description
<SOS>	Short for start of sentence, it indicates the beginning of an input or output token sequence. It is also written as <BOS>(beginning of sentence).
<EOS>	Short for end of sentence, it indicates the end of an input or output token sequence.
<UNK>	Short for unknown, it is a token that replaces a word not in the predetermined vocabulary.
<PAD>	Short for padding, it is a token filled in at the end of the shorter sequence to make the lengths of several token sequences equal.

Although the names contain the word sentence, what <SOS> and <EOS> actually mark is not a single sentence but the beginning and the end of the entire input/output token sequence going into and coming out of the model. Even if one input or output consists of several sentences—say, three sentences for the input and four for the output—they are attached only once each, at the very front and the very back of each token sequence.

If the two example sentences above are tokenized by word, <SOS> and <EOS> are attached to the front and back of each token sequence, and the shorter one is filled with <PAD> to match the lengths of the two token sequences, the result is as follows.

• Korean: <SOS> 그 학생은 도서관에서 책을 빌렸다. <EOS> <PAD> <PAD> <PAD>
• English: <SOS> The student borrowed a book from the library. <EOS>