What Are Online Learning and Batch Learning in Machine Learning?
Overview
This post explains online learning, batch learning, and minibatch learning. The names and differences among them are not actually that important, and the point here is that they are no big deal. In fact, one can consider that recent deep learning uses only minibatch learning.
Definition
Suppose we are given true values $\mathbf{y} = \left\{ y_{i} \right\}_{i=1}^{N}$ and their corresponding estimates $\hat{\mathbf{y}} = \left\{ \hat{y}_{i} \right\}_{i=1}^{N}$. Let the loss function with respect to the parameter $\theta$ be $L(y, \hat{y}) = L(y, \hat{y} ;\theta)$. When the method that optimizes the parameter is denoted $f(L, \theta)$, the approach of updating the parameter all at once over every $i$ is called batch learning.
$$ \theta \leftarrow f(L(\mathbf{y}, \hat{\mathbf{y}}), \theta) $$
The approach of updating the parameter individually for each $i$ is called online learning.
$$ \begin{align*} &\text{For $i=1$ to $N$} \\ &\qquad \theta \leftarrow f(L(y_{i}, \hat{y}_{i}), \theta) \end{align*} $$
Explanation
Taking gradient descent as an example, computing the gradient over all the data at once and updating with it is batch learning.
$$ \text{Batch learning: } \theta \leftarrow \theta - \alpha \nabla_{\theta}L(\mathbf{y}, \hat{\mathbf{y}}) $$
Repeatedly computing the gradient for each data point and updating the parameter is online learning.
$$ \text{Online learning: } \text{For $i=1$ to $N$,}\quad \theta \leftarrow \theta - \alpha \nabla_{\theta}L(y_{i}, \hat{y}_{i}) $$
One of the reasons that made the outstanding performance of deep learning possible is huge datasets, and applying batch learning to them is practically impossible in terms of hardware. Also, applying online learning has the drawback of taking a long time. Therefore, in deep learning, batch learning and online learning are in effect not used, and only minibatch learning, which learns in units of subsets of the dataset, is used. As a result, the term minibatch learning is not really used in practice either. When one simply says learning, it naturally means minibatch learning, and the term batch learning can also be regarded as meaning (mini)batch learning.
Minibatch Learning
Minibatch learning refers to, when the size of the dataset is $N = n \times m$, grouping the data into batches of $m$ to form $n$ subsets and taking each of them as a single unit of learning. If $\mathbf{y} = \mathbf{b}_{1} \cup \cdots \mathbf{b}_{n}$ $(\left| \mathbf{b}_{i} \right| = m)$, then
$$ \text{Minibatch learning: } \text{For $i=1$ to $n$,}\quad \theta \leftarrow \theta - \alpha \nabla_{\theta}L(\mathbf{b}_{i}, \hat{\mathbf{b}}_{i}) $$
Here, $m$ is called the batch size, and while it is common to set it to a power of $2$ such as $16, 32, 64$, it does not necessarily have to be so.
In particular, applying gradient descent in minibatches is called stochastic gradient descent.