What is a Layer in Deep Learning?
Definition
In deep learning, vector functions $L^{mn} : \mathbb{R}^{n} \to \mathbb{R}^{m}$ are called layers.
Explanation
On the other hand, scalar functions are called activation functions.
The reason why these are called layers is that when visualized after being compounded multiple times, they appear to be stacked layers upon layers.
$$ L^{n_{m}n_{m-1}} \circ \cdots \circ L^{n_{2}n_{1}} : \begin{bmatrix} x_{1} \\ \vdots \\ x_{n_{1}}\end{bmatrix} \mapsto \begin{bmatrix} y_{1} \\ \vdots \\ \vdots \\ y_{n_{2}}\end{bmatrix} \mapsto \begin{bmatrix} z_{1} \\ \vdots \\ \vdots \\ \vdots \\ z_{n_{3}}\end{bmatrix} \mapsto \cdots \mapsto \begin{bmatrix} v_{1} \\ \vdots \\ \vdots \\ v_{n_{m-1}}\end{bmatrix} \mapsto \begin{bmatrix} w_{1} \\ \vdots \\ w_{n_{m}}\end{bmatrix} $$
The combination of a layer and an activation function is called an artificial neural network. Specifically, the combination of a linear function and a step function is called a (single-layer) perceptron. The term [deep neural network] refers to the result of “many” compounding, and the process of approximating functions using deep neural networks is called deep learning.
Types
- Linear layers
- Convolutional layers
- Pooling layers