📂Statistical Analysis

Next-Generation Reservoir Computing NG-RC

Definition ^{1 2}

As a formulation of non-linear vector autoregression, the method that replaces the hyperparameters that introduced randomness in reservoir computing is called Next-Generation Reservoir Computing.

Explanation

Reservoir computing is not exactly what one would call deep learning, but it is a black-box technique that uses the least squares method, employing a strategy of constructing a reservoir system to handle data nonlinearity.

The problem is that realizations of reservoir computers are affected by random numbers, leading to poor consistency, and they are so sensitive to hyperparameters that hyperparameter optimization becomes excessively costly.

NG-RC therefore simply adds delay vectors, constructs a design matrix from their polynomial functions, and proceeds directly to optimization. More simply, it doubles the variables by including delay terms and stops at adding squared terms.

Seeing is believing: given a dataset as $\left\{ \mathbf{u}_{t} = \left( x_{t}, y_{t}, z_{t} \right) \right\}_{t=1}^{n}$, let’s directly examine how the setup of the model $\mathbf{u}_{t+1} = f \left( \mathbf{u}_{t} \right) W$—which takes the $\left( x_{t}, y_{t} \right)$ vector at time $t$ and predicts the $z_{t}$ at time $t+1$—is defined. $$ \begin{align*} U_{t+1} =& f \left( U_{t} \right) W \\ \implies \begin{bmatrix} z_{1} \\ z_{2} \\ \vdots \\ z_{n} \\ z_{n+1} \end{bmatrix} =& \begin{bmatrix} 1 & x_{0} & y_{0} && x_{0}^{2} & x_{0} x_{1} & \cdots \\ 1 & x_{1} & y_{1} && x_{1}^{2} & x_{1} x_{2} & \cdots \\ \vdots & \vdots & \vdots && \vdots & \vdots & \ddots \\ 1 & x_{n-1} & y_{n-1} && x_{n-1}^{2} & x_{n-1} x_{n} & \cdots \\ 1 & x_{n} & y_{n} && x_{n}^{2} & x_{n} x_{n+1} & \cdots \end{bmatrix} W \end{align*} $$

Here, choices can differ such as how many backshifts the delay vector takes, up to what degree the polynomial functions are considered, whether one predicts time $t+1$ from the vector at time $t$, and how many targets are predicted from how many observers. The least-squares problem described in the main text is slightly different in form from that presented in the paper that introduced NG-RC, but fundamentally it is fine.

• In the paper, instead of mapping an input of dimension $N$ to an output of the same dimension $N$, the setting predicts time $t+1$ from time $t$. This is intended to demonstrate predictive performance on its own, without intervention by observers.