Fall 2024 Omakase: Dual Numbers and Automatic Differentiation
Introduction
Hello, esteemed guests. For those of you disappointed that the magical moments of Cheoseo haven’t arrived, we have prepared a special omakase menu just for you. This course will introduce a variety of writings under the theme of “Dual Numbers and Automatic Differentiation.” Just like enjoying an omakase meal, savor each one in order to deepen your mathematical thinking.
Menu
Dual Numbers
First, as a starter for this omakase, let’s learn about dual numbers. Although they are not $$ \epsilon \ne 0, \qquad \epsilon^{2} = 0 $$, let’s call a number whose square is $$ \epsilon \ne 0, \qquad \epsilon^{2} = 0 $$ as $$ F(a+b\epsilon) = f(a) + f^{\prime}(a)b\epsilon $$.
$$ \epsilon \ne 0, \qquad \epsilon^{2} = 0 $$
Intuitively, you may think this doesn’t make sense, but if you consider numbers whose square is ▶eq4◀, which are also far from intuitive, it might be easier to accept. Now, we shall define a dual number for two real numbers ▶eq5◀ in the following form.
$$ a + b\epsilon $$
At first glance, it looks similar to a complex number ▶eq6◀, doesn’t it? For more on the meaning and specific properties of dual numbers, please refer to the following article:
Functions Defined on Dual Numbers
Let’s say a certain differentiable function ▶eq7◀ is given. Can we define this function on dual numbers? By considering the following function ▶eq9◀ for the given ▶eq8◀, it can be naturally extended.
$$ F(a+b\epsilon) = f(a) + f^{\prime}(a)b\epsilon $$
Defining it this way, you can discover some very interesting properties. For more details, please refer to the following article:
Automatic Differentiation
Have you heard of the term automatic differentiation? Automatic differentiation is one of the methods for computing derivatives on a computer, and it can be applied to functions composed of sums, products, and compositions of functions whose derivatives are already known. It may sound complicated, but it’s essentially just the chain rule.
Automatic Differentiation and Dual Numbers
Did the mention of differentiation catch you off guard while discussing dual numbers? In fact, dual numbers and automatic differentiation are closely related. By using dual numbers, you can easily implement automatic differentiation. It’s fascinating how dual numbers, which have a very pure mathematical definition, are connected to a method for implementing differentiation on a computer (and the derivatives obtained this way are even used in deep learning!).
Implementing Automatic Differentiation with Dual Numbers in Julia
For the final dessert, we have prepared content that demonstrates how to actually implement automatic differentiation using the concepts introduced earlier. The process of implementing automatic differentiation is simpler than you might think, and here, we use Julia. Through this process, you can see how theory transforms into practical code.