logo

Hyperbolic Tangent 📂Functions

Hyperbolic Tangent

Definition

Among the hyperbolic functions, the function defined below as the ratio of the hyperbolic sine to the hyperbolic cosine is called the hyperbolic tangent, commonly abbreviated as $\tanh$.

$$ \tanh x := \frac{\sinh x}{\cosh x} = \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}} = \frac{e^{2x} - 1}{e^{2x} + 1} $$

Properties

  • Range: The range is $(-1, 1)$, and as $x \to \pm\infty$, $\tanh x \to \pm 1$ asymptotically.

  • Odd function: $\tanh(-x) = -\tanh x$. In particular, since $\tanh 0 = 0$, the graph passes through the origin.

  • Derivative: The following holds. $$ (\tanh x)^{\prime} = 1 - \tanh^{2} x = \operatorname{sech}^{2} x $$ The derivative attains its maximum value $1$ at $x = 0$, and converges to $0$ as $|x|$ grows.

Relationship with the Sigmoid

It has the following relationship with the logistic sigmoid function $\sigma(x) = \dfrac{1}{1 + e^{-x}}$.

$$ \tanh x = 2\sigma(2x) - 1 $$

In other words, the hyperbolic tangent is the sigmoid stretched vertically so that its range moves from $(0, 1)$ to $(-1, 1)$, and translated so that it becomes symmetric about the origin.

$\tanh$ as an Activation Function

The hyperbolic tangent is one of the nonlinear activation functions widely used in artificial neural networks. Compared to the sigmoid, it has the following properties.

  • Zero-centered: Since the sigmoid’s range $(0, 1)$ is always positive, the mean of its output deviates from $0$; but $\tanh$ has the range $(-1, 1)$, symmetric about the origin, so its output is distributed around $0$. This stabilizes the learning of the next layer.
  • Larger gradient: As seen above, the derivative of $\tanh$ is at most $1$, whereas the derivative of the sigmoid cannot exceed $1/4$. Therefore $\tanh$ is relatively less vulnerable to gradient vanishing than the sigmoid.

However, since $\tanh$ also has the saturation property that its derivative approaches $0$ as $|x|$ grows, gradient vanishing still appears when layers are stacked very deeply. For this reason, the saturation-free $\operatorname{ReLU}$ family is mainly used in neural networks today, but $\tanh$ is still used when the network is not that deep or when one wants to restrict the range.

See Also