Leaky ReLU

Definition¹

In machine learning, the following function is called the leaky rectified linear unit (Leaky ReLU).

$$ \operatorname{LeakyReLU}(x) := \begin{cases} x & x \gt 0 \\ \alpha x & x \le 0 \end{cases} $$

Here $\alpha$ is a small positive constant, usually taken to be $0.01$.

In the figure, $\alpha = 0.1$ is used so that the shape is clearly visible.

Explanation

The Leaky ReLU is a variant of the $\operatorname{ReLU}$, proposed by Maas, Hannun, and Ng in 2013 in their work on acoustic models for speech recognition¹. The name ’leaky’ comes from the fact that, unlike the $\operatorname{ReLU}$, which completely blocks negative inputs and maps them to $0$, it lets negative inputs ’leak through’ little by little via a small slope $\alpha$.

Since the $\operatorname{ReLU}$ has both output and gradient equal to $0$ in the negative region, if the input to some neuron always ends up being negative during training, no gradient flows through that neuron at all and it is never updated again. This is called the dying $\operatorname{ReLU}$ problem, and the Leaky ReLU avoids it by leaving a gradient $\alpha \gt 0$ in the negative region as well. The original paper used $\alpha = 0.01$.

Meanwhile, a variant in which the slope $\alpha$ is treated not as a fixed constant but as a learnable parameter, the $\operatorname{PReLU}$ (parametric ReLU), was subsequently proposed as well².

Properties

Various Representations³

When $0 \lt \alpha \lt 1$, it can be expressed in various ways as follows.

$$ \begin{align*} \operatorname{LeakyReLU}(x) &:= \begin{cases} x & x \gt 0 \\ \alpha x & x \le 0 \end{cases} \\[1em] &= \max \left\{ \alpha x, x \right\} \\[1em] &= \operatorname{ReLU}(x) - \alpha \operatorname{ReLU}(-x) \\[1em] &= \alpha x + (1 - \alpha) \operatorname{ReLU}(x) \end{align*} $$

In particular, according to the last expression, the Leaky ReLU is a convex combination of the identity function $x$ and the $\operatorname{ReLU}$, that is, the ramp function.

Derivative

The derivative is as follows.

$$ \operatorname{LeakyReLU}^{\prime}(x) = \begin{cases} 1 & x \gt 0 \\ \alpha & x \lt 0 \end{cases} $$

If $\alpha \ne 1$, it is not differentiable at $x = 0$, but as with the $\operatorname{ReLU}$, this is not a problem in practice. In implementations, the slope at that point may simply be set to $\alpha$ or $1$.

Other Properties

For $c \ge 0$, $\operatorname{LeakyReLU}(cx) = c \operatorname{LeakyReLU}(x)$ holds.
Invertibility: If $\alpha \gt 0$, it is an increasing function and hence a bijection, and its inverse is a Leaky ReLU whose slope in the negative region is $1/\alpha$. Writing $\operatorname{LeakyReLU}_{\alpha}$ with the slope of the negative region made explicit, we have the following.

$$ \left( \operatorname{LeakyReLU}_{\alpha} \right)^{-1} = \operatorname{LeakyReLU}_{1/\alpha} $$