📂Machine Learning

Autoregressive Model

Definition

For $N \in \mathbb{N}$, suppose we are given a set of ordered data $X = \left\{ x_{1}, \dots, x_{N} \right\}$ and its power set $P(X)$. A function $f : P(X) \to X$ of the following form is called an autoregressive model.

$$ f : \left\{ x_{1}, \dots, x_{t-1} \right\} \mapsto x_{t} $$

Explanation¹

Intuitively, it is a function that predicts the next value $x_{t}$ given the values so far $x_{1}, \dots, x_{t-1}$. It is used as a core component in time series forecasting and generative models, among others. In particular, by appending the predicted value $x_{t}$ back to the end of the input and predicting $x_{t+1}$, and repeating this process, one can generate sequences of arbitrary length; a representative example is how large language models generate sentences by appending tokens one at a time.

Narrow Sense

Traditionally, an autoregressive model refers to the autoregressive process of time series analysis. Given white noise $\left\{ e_{t} \right\}_{t \in \mathbb{N}}$ and coefficients $\phi_{t}$, the process $\left\{ Y_{t} \right\}_{t \in \mathbb{N}}$ defined as follows is called the autoregressive process of order $p$, denoted $AR(p)$.

$$ Y_{t} = \phi_{1}Y_{t-1} + \phi_{2}Y_{t-2} + \cdots + \phi_{p}Y_{t-p} + e_{t} $$

While $AR(p)$ explains the next value as a linear combination of past values, an autoregressive model in the broad sense need not be expressed linearly like this.

Definition as a Conditional Distribution

Unlike the narrow-sense $AR(p)$, which specifies one concrete (linear) stochastic process that produces the next value, in modern generative models and the like, an autoregressive model is often regarded as the following conditional probability density function itself.

$$ p \left( x_{t} \mid x_{1}, \dots, x_{t-1} \right) $$

Viewed this way, the joint probability density function of the entire data decomposes into a product of conditional probability density functions.

$$ p \left( x_{1}, \dots, x_{t} \right) = \prod_{n=1}^{t} p \left( x_{n} \mid x_{1}, \dots, x_{n-1} \right) $$

In deep learning, this conditional distribution is usually modeled by a nonlinear artificial neural network architecture, with the recurrent neural network, which reads the data in order, being a representative example.