What is a Sigmoid Function?
Definition 1
A Sigmoid Function is defined as a bounded, differentiable scalar function $\sigma : \mathbb{R} \to \mathbb{R}$ that is defined for all $x \in \mathbb{R}$, is $\sigma ' (x) \ge 0$, and has exactly one inflection point.
- Sigmoidal functions are defined differently.
Types
Examples of sigmoid functions include:
- Logistic function: $\displaystyle f(x) := {{ 1 } \over { 1 + e^{-x} }}$
- Hyperbolic tangent: $\tanh x$
- Arctangent: $\arctan x$
Description
Sigmoid functions, by definition, possess the properties stated and tend to converge to either their maximum or minimum values past a certain point of input, making them popular as activation functions in fields such as deep learning. It’s phrased in the past tense because, as of 2020, they have fallen out of favor due to the widely recognized issue of gradient vanishing, prompting a shift towards adopting activation functions that overcome this problem.
Nevertheless, the reason for still explaining sigmoid functions is because someone must make it clear that these are not anything special. Despite the passé trend, those studying the topic inevitably come across older papers, and since the sigmoid function may seem daunting by name yet are of less significance compared to the currently popular functions, it’s okay to lightly move past them.