📂Machine Learning

Encoder and Decoder in Machine Learning

Definition

Suppose a dataset $X \subset \mathbb{R}^{n}$ is given. In the context of machine learning, an encoder is defined as a function for an appropriate set $Z \subset \mathbb{R}^{m}$ ($m \le n$) such as the following:

$$\begin{align*} f : X &\to Z \\ \mathbf{x} &\mapsto \mathbf{z} = f(\mathbf{x}) \end{align*}$$

The following $g$ is called a decoder:

$$\begin{align*} g : Z &\to X \\ \mathbf{z} &\mapsto \mathbf{x} = g(\mathbf{z}) \end{align*}$$

• Here, $Z$ is called a latent space, and $\mathbf{z}$ is called a latent variable.

Explanation

In the context of information theory, the role of an encoder focuses on compressing information. The purpose of compression may be to create encryption for the security of communication contents or to use storage space efficiently.

In the context of machine learning, encoders still serve the role of compressing information, as reducing the dimensions of data saves memory and decreases computation time. Unlike in information theory, encoders in machine learning have another important role, which is feature extraction. An encoder is a function that outputs some latent features from the given input data. Hence, the output of an encoder is also referred to as a latent variable or feature vector, and the process of obtaining the encoder’s output from input data is called feature extraction. It is important that the encoder not only reduces the dimensionality of data but also preserves and extracts the characteristics of the data effectively. Conversely, the decoder’s role is to reconstruct the original data from the latent variables output by the encoder.

The reason for emphasizing meaningful outputs and feature extraction by the encoder in machine learning is that artificial neural networks function as black-box algorithms. Unlike traditional encryption or compression technologies, where the structure or principles of an encoder are explicit and understandable, thus making the nature of the output irrelevant, in deep learning, neural networks are only optimized in the direction that reduces the loss function. This optimization doesn’t allow us to clearly understand or explain the compression principles. Therefore, by visualizing or clustering the extracted feature vectors, we verify that their meanings are well-preserved.