📂Functions

Hard Thresholding and Soft Thresholding as Functions

Definition ¹

Let a threshold $\lambda \in \mathbb{R}$ be given.

Hard thresholding

The $\eta _{H} \left( x ; \lambda \right) : \mathbb{R} \to \mathbb{R}$ defined as follows is called hard thresholding. $$ \begin{align*} \eta _{H} \left( x ; \lambda \right) =& x \cdot \mathbf{1}_{\left\{ \left| x \right| \ge \lambda \right\}} (x) \\ =& \begin{cases} x & , \text{if } x \in [-\lambda, \lambda] \\ 0 & , \text{if } x \notin [-\lambda, \lambda] \end{cases} \end{align*} $$ Here, $\mathbf{1}_{ A }$ is the indicator function that equals $1$ when the set $x \in A$ holds, and equals $0$ otherwise.

Soft thresholding

The $\eta _{S} \left( x ; \lambda \right) : \mathbb{R} \to \mathbb{R}$ defined as follows is called soft thresholding. $$ \begin{align*} \eta _{S} \left( x ; \lambda \right) =& \operatorname{sign} (x) \cdot \operatorname{ReLU} \left( \left| x \right| - \lambda \right) \\ =& \begin{cases} \lambda - \left| x \right| & , \text{if } x < - \lambda \\ 0 & , \text{if } x \in [-\lambda, \lambda] \\ \left| x \right| - \lambda & , \text{if } x > \lambda \end{cases} \end{align*} $$ Here, ▷eq08◯ is the sign, and ▷eq09◯ is the ReLU.

Remarks

The shapes of the functions introduced are as follows².

From an algorithmic perspective, thresholding means denoising by removing relatively small values.

There are various differences between hard and soft, but the most striking one mathematically is the continuity at $\pm \lambda$; otherwise they are almost similar — for example, in $\left[ - \lambda , \lambda \right]$ being $0$, or having the same subdifferential. Typically one chooses whichever is appropriate for the purpose, so it is rare to use both; notationally one often omits ▷eq14◯ or ▷eq15◯ and writes the threshold $\lambda$ in the subscript, as in $\eta_{\lambda} (x)$.