logo

Finite Difference Method 📂Numerical Analysis

Finite Difference Method

Definition1 2

The finite difference method is a numerical method for computing derivatives, approximating the derivative as the average rate of change over a short interval.

Explanation

The key to deriving the formula is the Taylor expansion.

f(x+h)=f(x)+f′(x)h+f′′(x)2!h2+f′′′3!h3+⋯(1) f(x+h) = f(x) + f^{\prime}(x)h + \dfrac{f^{\prime \prime}(x)}{2!}h^{2} + \dfrac{f^{\prime \prime \prime}}{3!}h^{3} + \cdots \tag{1}

Rearranging to have only the derivative on the left side,

f′(x)=f(x+h)−f(x)h+f′′(x)2!h+⋯=f(x+h)−f(x)h+O(h)≈f(x+h)−f(x)h \begin{align*} f^{\prime}(x) &= \dfrac{f(x+h) - f(x)}{h} + \dfrac{f^{\prime \prime}(x)}{2!}h + \cdots \\ &= \dfrac{f(x+h) - f(x)}{h} + \mathcal{O}(h) \\ &\approx \dfrac{f(x+h) - f(x)}{h} \end{align*}

Here, O(h)\mathcal{O}(h) is asymptotic notation. If hh is sufficiently small, the right side will approximate the left side quite well. In fact, derivatives or acceleration are often explained as the limit of the average rate of change when first learning.

Derivatives

Forward Difference

The derivative at xx is approximated by the average rate of change between xx and the next timestep x+hx+h, which is called the forward difference. The error term is O(h)\mathcal{O}(h).

f′(x)≈f(x+h)−f(x)h f^{\prime}(x) \approx \dfrac{f(x+h) - f(x)}{h}

Backward Difference

The derivative at xx is approximated by the average rate of change between xx and the previous timestep x−hx-h, which is called the backward difference. The error term is O(h)\mathcal{O}(h).

f′(x)≈f(x)−f(x−h)h f^{\prime}(x) \approx \dfrac{f(x) - f(x-h)}{h}

Central Difference

The central difference, unlike the previous two methods, has an error term of O(h2)\mathcal{O}(h^{2}). By substituting ±h2\pm\frac{h}{2} for hh in (1)(1),

f(x+h/2)=f(x)+f′(x)h2+f′′(x)2!(h2)2+f′′′(x)3!(h2)3+⋯ f(x+h/2) = f(x) + f^{\prime}(x)\frac{h}{2} + \dfrac{f^{\prime \prime}(x)}{2!}\left(\frac{h}{2}\right)^{2} + \dfrac{f^{\prime \prime \prime}(x)}{3!}\left(\frac{h}{2}\right)^{3} + \cdots

f(x−h/2)=f(x)−f′(x)h2+f′′(x)2!(h2)2−f′′′(x)3!(h2)3+⋯ f(x-h/2) = f(x) - f^{\prime}(x)\frac{h}{2} + \dfrac{f^{\prime \prime}(x)}{2!}\left(\frac{h}{2}\right)^{2} - \dfrac{f^{\prime \prime \prime}(x)}{3!}\left(\frac{h}{2}\right)^{3} + \cdots

Subtracting the equation below from the above,

f(x+h/2)−f(x−h/2)=f′(x)h+2f′′′(x)3!(h2)3+⋯ f(x+h/2) - f(x-h/2) = f^{\prime}(x)h + 2\dfrac{f^{\prime \prime \prime}(x)}{3!}\left(\frac{h}{2}\right)^{3} + \cdots

Rearranging for f′(x)f^{\prime}(x),

f′(x)=f(x+h/2)−f(x−h/2)h+f′′′(x)3!(h2)2+⋯=f(x+h/2)−f(x−h/2)h+O(h2)≈f(x+h/2)−f(x−h/2)h \begin{align*} f^{\prime}(x) &= \frac{f(x+h/2) - f(x-h/2)}{h} + \dfrac{f^{\prime \prime \prime}(x)}{3!}\left(\frac{h}{2}\right)^{2} + \cdots \\ &= \frac{f(x+h/2) - f(x-h/2)}{h} + \mathcal{O}(h^{2}) \\ &\approx \frac{f(x+h/2) - f(x-h/2)}{h} \end{align*}

Second-order Derivatives

Adding the two equations of (1)(1) and substituting −h-h for hh yields the following.

f(x+h)=f(x)+f′(x)h+f′′(x)2!h2+f′′′3!h3+⋯+f(x−h)=f(x)−f′(x)h+f′′(x)2!h2−f′′′3!h3+⋯f(x+h)+f(x−h)=2f(x)+f′′(x)h2+2f′′′′4!h4+⋯ \begin{align*} && f(x+h) &= f(x) + f^{\prime}(x)h + \dfrac{f^{\prime \prime}(x)}{2!}h^{2} + \dfrac{f^{\prime \prime \prime}}{3!}h^{3} + \cdots \\ {+} && f(x-h) &= f(x) - f^{\prime}(x)h + \dfrac{f^{\prime \prime}(x)}{2!}h^{2} - \dfrac{f^{\prime \prime \prime}}{3!}h^{3} + \cdots \\ \hline && f(x+h) + f(x-h) &= 2f(x) + f^{\prime \prime}(x)h^{2} + 2\dfrac{f^{\prime \prime \prime \prime}}{4!}h^{4} + \cdots \end{align*}

Rearranging for f′′f^{\prime \prime},

f′′(x)=f(x+h)−2f(x)+f(x−h)h2+2f′′′′4!h2+⋯=f(x+h)−2f(x)+f(x−h)h2+O(h2)≈f(x+h)−2f(x)+f(x−h)h2 \begin{align*} f^{\prime \prime}(x) &= \frac{f(x+h) - 2f(x) + f(x-h)}{h^{2}} + 2\dfrac{f^{\prime \prime \prime \prime}}{4!}h^{2} + \cdots \\ &= \frac{f(x+h) - 2f(x) + f(x-h)}{h^{2}} + \mathcal{O}(h^{2}) \\ &\approx \frac{f(x+h) - 2f(x) + f(x-h)}{h^{2}} \end{align*}

Ignoring the error term, this equation can also be derived by applying the forward difference method and then the backward difference method in turn to f′f^{\prime}.

f′′(x)≈f′(x+h)−f′(x)h≈f(x+h)−f(x)h−f(x)−f(x−h)hh=f(x+h)−2f(x)+f(x−h)h2 \begin{align*} f^{\prime \prime}(x) &\approx \dfrac{f^{\prime}(x+h) - f^{\prime}(x)}{h} \\ &\approx \dfrac{\dfrac{f(x+h)-f(x)}{h} - \dfrac{f(x)-f(x-h)}{h}}{h} \\ &= \dfrac{f(x+h) - 2f(x) + f(x-h)}{h^{2}} \end{align*}

See Also

  1. A. Iserles, A First Course in the Numerical Analysis of Differential Equations (2nd, 2009), p139-141 ↩︎

  2. Mykel J. Kochenderfer, Algorithms for Optimization (2019), p23-26 ↩︎