스칼라 함수의 그래디언트 표
스칼라 함수의 그래디언트
스칼라 함수 $f : \mathbb{R}^{n} \to \mathbb{R}$의 그래디언트는 다음과 같다. $$ \frac{ \partial f(\mathbf{x})}{ \partial \mathbf{x} } := \nabla f(\mathbf{x}) = \begin{bmatrix} \dfrac{ \partial f(\mathbf{x})}{ \partial x_{1} } & \dfrac{ \partial f(\mathbf{x})}{ \partial x_{2} } & \cdots & \dfrac{ \partial f(\mathbf{x})}{ \partial x_{n} } \end{bmatrix}^{\mathsf{T}} $$
여기서 $\dfrac{ \partial f(\mathbf{x})}{ \partial x_{i} }$는 $f$의 $x_{i}$에 대한 편도함수이다. 여러가지 스칼라 함수에 대한 그래디언트는 아래와 같다.
| 이름 | 함수 $f(\mathbf{x})$ | 조건 | 그래디언트 $\nabla f(\mathbf{x})$ |
|---|---|---|---|
| 내적 | $f(\mathbf{x}) = \left\langle \mathbf{w}, \mathbf{x} \right\rangle = \mathbf{w}^{\mathsf{T}}\mathbf{x}$ | $\mathbf{w}$ | |
| 놈 | $f(\mathbf{x}) = \left\| \mathbf{x} \right\|^{2} = \mathbf{x}^{\mathsf{T}} \mathbf{x}$ | $2\mathbf{x}$ | |
| 이차형식 | $f(\mathbf{x}) = \mathbf{x}^{\mathsf{T}}A\mathbf{x}$ | $(A + A^{\mathsf{T}})\mathbf{x}$ | |
| 이차형식 | $f(\mathbf{x}) = \mathbf{x}^{\mathsf{T}}A\mathbf{x}$ | $A$가 대칭 | $2A\mathbf{x}$ |
증명
내적
$$ \begin{align*} \frac{ \partial f(\mathbf{x})}{ \partial \mathbf{x} } =\frac{ \partial (\mathbf{w}^{\mathsf{T}}\mathbf{x})}{ \partial \mathbf{x} } &=\begin{bmatrix} \dfrac{ \partial \left( \sum _{i=1} ^{n} w_{i}x_{i}\right)}{ \partial x_{1} } & \dfrac{ \partial \left( \sum _{i=1} ^{n} w_{i}x_{i}\right)}{ \partial x_{2} } & \cdots & \dfrac{ \partial \left( \sum _{i=1} ^{n} w_{i}x_{i}\right)}{ \partial x_{n} } \end{bmatrix}^{\mathsf{T}} \\ &= \begin{bmatrix} w_{1} & w_{2} & \cdots & w_{n} \end{bmatrix}^{\mathsf{T}} \\ &= \mathbf{w} \end{align*} $$
또한 $\mathbf{w}^{\mathsf{T}}\mathbf{x} = \mathbf{x}^{\mathsf{T}}\mathbf{w}$이므로,
$$ \frac{ \partial \mathbf{x}^{\mathsf{T}}\mathbf{w}}{ \partial \mathbf{x} } = \mathbf{w} $$
■
놈
내적에 대한 증명과 마찬가지로,
$$ \begin{align*} \frac{ \partial f(\mathbf{x})}{ \partial \mathbf{x} } = \frac{ \partial (\mathbf{x}^{\mathsf{T}}\mathbf{x})}{ \partial \mathbf{x} } &=\begin{bmatrix} \dfrac{ \partial \left( \sum _{i=1} ^{n} x_{i}x_{i}\right)}{ \partial x_{1} } & \dfrac{ \partial \left( \sum _{i=1} ^{n} x_{i}x_{i}\right)}{ \partial x_{2} } & \cdots & \dfrac{ \partial \left( \sum _{i=1} ^{n} x_{i}x_{i}\right)}{ \partial x_{n} } \end{bmatrix}^{\mathsf{T}} \\ &=\begin{bmatrix} \dfrac{ \partial \left( \sum _{i=1} ^{n} x_{i}^{2}\right)}{ \partial x_{1} } & \dfrac{ \partial \left( \sum _{i=1} ^{n} x_{i}^{2}\right)}{ \partial x_{2} } & \cdots & \dfrac{ \partial \left( \sum _{i=1} ^{n} x_{i}^{2}\right)}{ \partial x_{n} } \end{bmatrix}^{\mathsf{T}} \\ &= \begin{bmatrix} 2x_{1} & 2x_{2} & \cdots & 2x_{n} \end{bmatrix}^{\mathsf{T}} \\ &= 2\mathbf{x} \end{align*} $$
■
이차형식
미분 계산이 쉽도록 먼저 계산을 다음과 같이 해준다. 임의의 $k\in\left\{1,\dots,n\right\}$에 대해서 다음의 식을 얻는다.
$$ \begin{align*} f(\mathbf{x}) &= \mathbf{x}^{\mathsf{T}}A\mathbf{x} =\sum \limits _{j=1} ^{n} x_{j} \sum \limits _{i=1} ^{n}r_{ji}x_{i} \\ & = x_{k} \sum \limits _{i=1} ^{n}r_{ki}x_{i} +\sum \limits _{j\ne k}x_{j} \sum \limits _{i=1} ^{n}r_{ji}x_{i} \\ &= x_{k}\left(r_{kk}x_{k} + \sum \limits _{i\ne k} r_{ki}x_{i}\right) +\sum \limits _{j\ne k}x_{j} \left(r_{jk}x_{k}+ \sum \limits _{i \ne k} r_{ji}x_{i} \right) \\ &= x_{k}^{2}r_{kk} + x_{k}\sum \limits _{i\ne k} r_{ki}x_{i} + \sum \limits _{j\ne k}x_{j}r_{jk}x_{k}+ \sum \limits _{j\ne k}\sum \limits _{i \ne k}x_{j} r_{ji}x_{i} \end{align*} $$
$\dfrac{ \partial f(\mathbf{x})}{ \partial x_{k}}$를 계산하면 다음과 같다.
$$ \begin{align*} \frac{ \partial f(\mathbf{x})}{ \partial x_{k} }&=\frac{ \partial }{ \partial x_{k} } \left( x_{k}^{2}r_{kk} + x_{k}\sum \limits _{i\ne k} r_{ki}x_{i} + \sum \limits _{j\ne k}x_{j}r_{jk}x_{k}+ \sum \limits _{j\ne k}\sum \limits _{i \ne k}x_{j} r_{ji}x_{i} \right) \\ &=2w_{k}r_{kk} + \sum \limits _{i\ne k} r_{ki}x_{i} + \sum \limits _{j\ne k}x_{j}r_{jk} \\ &=\sum \limits _{i=1}^{n} r_{ki}x_{i} + \sum \limits _{j=1}^{n}r_{jk}x_{j} \end{align*} $$
따라서 $f$의 그래디언트를 계산하면 다음과 같다.
$$ \begin{align*} \frac{ \partial f(\mathbf{x})}{ \partial \mathbf{x} } = \dfrac{\partial \left( \mathbf{x}^{\mathsf{T}}A\mathbf{x} \right)}{\partial \mathbf{x}} &= \begin{bmatrix} \dfrac{ \partial f (\mathbf{x})}{ \partial x_{1} } & \dfrac{ \partial f (\mathbf{x})}{ \partial x_{2} } & \dots & \dfrac{ \partial f (\mathbf{x})}{ \partial x_{n} } \end{bmatrix}^{\mathsf{T}} \\[1em] &= \begin{bmatrix} \sum _{i=1} ^{n} r_{1i}x_{i} + \sum _{j=1}^{n} r_{j1}x_{j} \\[0.5em] \sum _{i=1} ^{n} r_{2i}x_{i} + \sum _{j=1}^{n} r_{j2}x_{j} \\[0.5em] \vdots \\[0.5em] \sum _{i=1} ^{n} r_{ni}x_{i} + \sum _{j=1}^{n} r_{jn}x_{j} \end{bmatrix} \\[3.5em] &= \begin{bmatrix} \sum _{i=1} ^{n} r_{1i}x_{i} \\[0.5em] \sum _{i=1} ^{n} r_{2i}x_{i} \\[0.5em] \vdots \\[0.5em] \sum _{i=1} ^{n} r_{ni}x_{i} \end{bmatrix} + \begin{bmatrix} \sum _{j=1}^{n} r_{j1}x_{j} \\[0.5em] \sum _{j=1}^{n} r_{j2}x_{j} \\[0.5em] \vdots \\[0.5em] \sum _{j=1}^{n} r_{jn}x_{j} \end{bmatrix} \\[3.5em] &=A \mathbf{x} + A^{\mathsf{T}}\mathbf{x}=\left( A+A^{\mathsf{T}} \right)\mathbf{x} \end{align*} $$
$\mathbf{X} \in M_{n\times n}$, $\mathbf{y} \in M_{n \times 1}$에 대해서,
$$ \mathbf{X}\mathbf{y} =\begin{bmatrix} \sum _{i=1} ^{n}x_{1i}y_{i} \\ \sum _{i=1} ^{n}x_{2i}y_{i} \\ \vdots \\ \sum _{i=1} ^{n}x_{ni}y_{i} \end{bmatrix},\qquad \mathbf{X}^{\mathsf{T}}\mathbf{y} =\begin{bmatrix} \sum _{i=1} ^{n}x_{i1}y_{i} \\ \sum _{i=1} ^{n}x_{i2}y_{i} \\ \vdots \\ \sum _{i=1} ^{n}x_{in}y_{i} \end{bmatrix} $$
만약 $A$이 대칭행렬이면,
$$ \frac{ \partial }{ \partial \mathbf{x} }\left( \mathbf{x}^{\mathsf{T}}A\mathbf{x} \right)=2A\mathbf{x} $$