Generalizing Differentiation: Gradient Matrices and Matrix Calculus
📂Vector Analysis Generalizing Differentiation: Gradient Matrices and Matrix Calculus Definition We define the gradient matrix ∇ X f \nabla_{\mathbf{X}} f ∇ X f scalar function f : R n × n → R f : \mathbb{R}^{n \times n} \to \mathbb{R} f : R n × n → R matrix X = [ x i j ] ∈ R n × n \mathbf{X} = [x_{ij}] \in \mathbb{R}^{n \times n} X = [ x ij ] ∈ R n × n
[ ∇ X f ] i j = ∂ f ∂ x i j ( i , j = 1 , … , n )
[\nabla_{\mathbf{X}} f]_{ij} = \dfrac{\partial f}{\partial x_{ij}} \quad (i,j=1,\dots,n)
[ ∇ X f ] ij = ∂ x ij ∂ f ( i , j = 1 , … , n )
∇ X f = ∂ f ∂ X = [ ∂ f ∂ x 11 ⋯ ∂ f ∂ x 1 n ⋮ ⋱ ⋮ ∂ f ∂ x n 1 ⋯ ∂ f ∂ x n n ]
\nabla_{\mathbf{X}} f = \dfrac{\partial f}{\partial \mathbf{X}} = \begin{bmatrix}
\dfrac{\partial f}{\partial x_{11}} & \cdots & \dfrac{\partial f}{\partial x_{1n}} \\ \vdots & \ddots & \vdots \\ \dfrac{\partial f}{\partial x_{n1}} & \cdots & \dfrac{\partial f}{\partial x_{nn}} \end{bmatrix}
∇ X f = ∂ X ∂ f = ∂ x 11 ∂ f ⋮ ∂ x n 1 ∂ f ⋯ ⋱ ⋯ ∂ x 1 n ∂ f ⋮ ∂ x nn ∂ f
Explanation If you think of a matrix as a reshaped form of a vector , it is naturally generalizable from the concept of a gradient . Originally, a gradient is nothing more than a vector created by differentiating with respect to each component of a vector, rather than differentiating the vector itself. In the same way, the concept of differentiating with respect to a matrix is not truly differentiating with respect to a matrix but creating a matrix by differentiating with respect to each component of the matrix. Remember, this is essentially no different from differentiating a multivariable function. Specifically, differentiating "with respect to" a matrix is correct, but differentiating "a" matrix refers to differentiating a matrix function , which is different.
In fields dealing with data, including machine learning and deep learning, vector and matrix calculations and the calculations of their derivatives appear frequently. Thus, it is appropriate to call the operator ∇ X = ∂ ∂ X \nabla_{\mathbf{X}} = \dfrac{\partial}{\partial \mathbf{X}} ∇ X = ∂ X ∂ ∇ X f \nabla_{\mathbf{X}}f ∇ X f
See Also Generalization By considering the explanation above, ultimately, it is about whether the value being differentiated and the variable of differentiation are scalars or multivariables; thus, it need not be limited only to matrices. It can be defined for various cases as shown in the table below.
Variable \ Value Scalar y y y Vector y \mathbf{y} y Matrix Y \mathbf{Y} Y Scalar ∂ y ∂ x \dfrac{\partial y}{\partial x} ∂ x ∂ y ∂ y ∂ x \dfrac{\partial \mathbf{y}}{\partial x} ∂ x ∂ y ∂ Y ∂ x \dfrac{\partial \mathbf{Y}}{\partial x} ∂ x ∂ Y Vector ∂ y ∂ x \dfrac{\partial y}{\partial \mathbf{x}} ∂ x ∂ y ∂ y ∂ x \dfrac{\partial \mathbf{y}}{\partial \mathbf{x}} ∂ x ∂ y Matrix ∂ y ∂ X \dfrac{\partial y}{\partial \mathbf{X}} ∂ X ∂ y
However, what has actually been done is unifying existing concepts into a single, elegant notation.
Considering Jacobian or the chain rule , it is natural to treat the function value y \mathbf{y} y column vector and the variable x \mathbf{x} x row vector .
y = [ y 1 y 2 ⋮ y m ] , x = [ x 1 x 2 ⋯ x n ]
\mathbf{y} = \begin{bmatrix} y_{1} \\ y_{2} \\ \vdots \\ y_{m} \end{bmatrix}, \quad
\mathbf{x} = \begin{bmatrix} x_{1} & x_{2} & \cdots & x_{n} \end{bmatrix}
y = y 1 y 2 ⋮ y m , x = [ x 1 x 2 ⋯ x n ]
Hence, ∂ y ∂ x \dfrac{\partial y}{\partial \mathbf{x}} ∂ x ∂ y ∂ y ∂ x \dfrac{\partial \mathbf{y}}{\partial x} ∂ x ∂ y ∂ y ∂ x \dfrac{\partial y}{\partial x} ∂ x ∂ y y = y ( x ) y = y(\mathbf{x}) y = y ( x ) x = x ( t ) \mathbf{x} = \mathbf{x}(t) x = x ( t )
∂ y ∂ t = ∂ y ∂ x d x d t = [ ∂ y ∂ x 1 ⋯ ∂ y ∂ x n ] [ d x 1 d t ⋮ d x n d t ] = ∑ i = 1 n ∂ y ∂ x i d x i d t
\dfrac{\partial y}{\partial t} = \dfrac{\partial y}{\partial \mathbf{x}} \dfrac{d \mathbf{x}}{d t} = \begin{bmatrix} \dfrac{\partial y}{\partial x_{1}} & \cdots & \dfrac{\partial y}{\partial x_{n}} \end{bmatrix} \begin{bmatrix} \dfrac{d x_{1}}{d t} \\ \vdots \\ \dfrac{d x_{n}}{d t} \end{bmatrix} = \sum\limits_{i=1}^{n} \dfrac{\partial y}{\partial x_{i}} \dfrac{d x_{i}}{d t}
∂ t ∂ y = ∂ x ∂ y d t d x = [ ∂ x 1 ∂ y ⋯ ∂ x n ∂ y ] d t d x 1 ⋮ d t d x n = i = 1 ∑ n ∂ x i ∂ y d t d x i
Even if the notation is unified, it is easy to confuse whether it is a gradient or a vector-valued function differentiation. In these instances, consider ∂ ∂ x \dfrac{\partial }{\partial x} ∂ x ∂ ∂ ∂ x \dfrac{\partial }{\partial \mathbf{x}} ∂ x ∂ ∂ ∂ X \dfrac{\partial }{\partial \mathbf{X}} ∂ X ∂ Kronecker product ⊗ \otimes ⊗
Scalar By Differentiated by a Scalar Differentiating a scalar by a scalar is analogous to the ordinary differentiation .
∂ y ∂ x = lim h → 0 y ( x + h ) − y ( x ) h
\dfrac{\partial y}{\partial x} = \lim\limits_{h \to 0} \dfrac{y(x+h) - y(x)}{h}
∂ x ∂ y = h → 0 lim h y ( x + h ) − y ( x )
In terms of the Kronecker product, it can be expressed as,
∂ y ∂ x = ∂ ∂ x ⊗ y = ∂ y ∂ x
\dfrac{\partial y}{\partial x} = \dfrac{\partial }{\partial x} \otimes y = \dfrac{\partial y}{\partial x}
∂ x ∂ y = ∂ x ∂ ⊗ y = ∂ x ∂ y
Differentiated by a Vector This is analogous to the gradient . As the value is a scalar and the variable is a row vector, the gradient is also naturally regarded as a row vector.
∂ y ∂ x = ∇ x y = [ ∂ y ∂ x 1 ⋯ ∂ y ∂ x n ]
\dfrac{\partial y}{\partial \mathbf{x}} =
\nabla_{\mathbf{x}} y =
\begin{bmatrix} \dfrac{\partial y}{\partial x_{1}} & \cdots & \dfrac{\partial y}{\partial x_{n}} \end{bmatrix}
∂ x ∂ y = ∇ x y = [ ∂ x 1 ∂ y ⋯ ∂ x n ∂ y ]
In terms of the Kronecker product, it can be expressed as,
∂ y ∂ x = ∂ ∂ x ⊗ y = [ ∂ ∂ x 1 ⋯ ∂ ∂ x n ] ⊗ [ y ] = [ ∂ y ∂ x 1 ⋯ ∂ y ∂ x n ]
\dfrac{\partial y}{\partial \mathbf{x}} = \dfrac{\partial }{\partial \mathbf{x}} \otimes y = \begin{bmatrix} \dfrac{\partial }{\partial x_{1}} & \cdots & \dfrac{\partial }{\partial x_{n}} \end{bmatrix} \otimes \begin{bmatrix} y \end{bmatrix} =
\begin{bmatrix} \dfrac{\partial y}{\partial x_{1}} & \cdots & \dfrac{\partial y}{\partial x_{n}} \end{bmatrix}
∂ x ∂ y = ∂ x ∂ ⊗ y = [ ∂ x 1 ∂ ⋯ ∂ x n ∂ ] ⊗ [ y ] = [ ∂ x 1 ∂ y ⋯ ∂ x n ∂ y ]
The Laplacian Δ y = ∇ 2 y \Delta y = \nabla^{2} y Δ y = ∇ 2 y
Δ y = ∇ 2 y = ∑ i = 1 n ∂ 2 y ∂ x i 2 = ( ∂ y ∂ x ) ( ∂ y ∂ x ) T = Tr ( ( ∂ y ∂ x ) T ( ∂ y ∂ x ) )
\Delta y = \nabla^{2} y = \sum\limits_{i=1}^{n} \dfrac{\partial^{2} y}{\partial x_{i}^{2}} = \left( \dfrac{\partial y}{\partial \mathbf{x}} \right) \left( \dfrac{\partial y}{\partial \mathbf{x}} \right)^{\mathsf{T}} = \Tr \left( \left( \dfrac{\partial y}{\partial \mathbf{x}} \right)^{\mathsf{T}} \left( \dfrac{\partial y}{\partial \mathbf{x}} \right) \right)
Δ y = ∇ 2 y = i = 1 ∑ n ∂ x i 2 ∂ 2 y = ( ∂ x ∂ y ) ( ∂ x ∂ y ) T = Tr ( ( ∂ x ∂ y ) T ( ∂ x ∂ y ) )
( ∂ y ∂ x ) T ( ∂ y ∂ x ) \left( \dfrac{\partial y}{\partial \mathbf{x}} \right)^{\mathsf{T}} \left( \dfrac{\partial y}{\partial \mathbf{x}} \right) ( ∂ x ∂ y ) T ( ∂ x ∂ y ) Hessian .
H ( y ) = ( ∂ y ∂ x ) T ( ∂ y ∂ x ) = [ ∂ 2 y ∂ x 1 2 ⋯ ∂ 2 y ∂ x 1 ∂ x n ⋮ ⋱ ⋮ ∂ 2 y ∂ x n ∂ x 1 ⋯ ∂ 2 y ∂ x n 2 ]
H(y) = \left( \dfrac{\partial y}{\partial \mathbf{x}} \right)^{\mathsf{T}} \left( \dfrac{\partial y}{\partial \mathbf{x}} \right) = \begin{bmatrix}
\dfrac{\partial^{2} y}{\partial x_{1}^{2}} & \cdots & \dfrac{\partial^{2} y}{\partial x_{1} \partial x_{n}} \\ \vdots & \ddots & \vdots \\ \dfrac{\partial^{2} y}{\partial x_{n} \partial x_{1}} & \cdots & \dfrac{\partial^{2} y}{\partial x_{n}^{2}} \end{bmatrix}
H ( y ) = ( ∂ x ∂ y ) T ( ∂ x ∂ y ) = ∂ x 1 2 ∂ 2 y ⋮ ∂ x n ∂ x 1 ∂ 2 y ⋯ ⋱ ⋯ ∂ x 1 ∂ x n ∂ 2 y ⋮ ∂ x n 2 ∂ 2 y
Differentiated by a Matrix This refers to the matrix gradient defined above.
∇ X y = ∂ y ∂ X = [ ∂ y ∂ x 11 ⋯ ∂ y ∂ x 1 n ⋮ ⋱ ⋮ ∂ y ∂ x m 1 ⋯ ∂ y ∂ x m n ]
\nabla_{\mathbf{X}} y = \dfrac{\partial y}{\partial \mathbf{X}} = \begin{bmatrix}
\dfrac{\partial y}{\partial x_{11}} & \cdots & \dfrac{\partial y}{\partial x_{1n}} \\ \vdots & \ddots & \vdots \\ \dfrac{\partial y}{\partial x_{m1}} & \cdots & \dfrac{\partial y}{\partial x_{mn}} \end{bmatrix}
∇ X y = ∂ X ∂ y = ∂ x 11 ∂ y ⋮ ∂ x m 1 ∂ y ⋯ ⋱ ⋯ ∂ x 1 n ∂ y ⋮ ∂ x mn ∂ y
In terms of the Kronecker product, it can be expressed as,
∂ y ∂ X = ∂ ∂ X ⊗ y = [ ∂ ∂ x 11 ⋯ ∂ ∂ x 1 n ⋮ ⋱ ⋮ ∂ ∂ x m 1 ⋯ ∂ ∂ x m n ] ⊗ [ y ] = [ ∂ y ∂ x 11 ⋯ ∂ y ∂ x 1 n ⋮ ⋱ ⋮ ∂ y ∂ x m 1 ⋯ ∂ y ∂ x m n ]
\dfrac{\partial y}{\partial \mathbf{X}} = \dfrac{\partial }{\partial \mathbf{X}} \otimes y = \begin{bmatrix} \dfrac{\partial }{\partial x_{11}} & \cdots & \dfrac{\partial }{\partial x_{1n}} \\ \vdots & \ddots & \vdots \\ \dfrac{\partial }{\partial x_{m1}} & \cdots & \dfrac{\partial }{\partial x_{mn}} \end{bmatrix} \otimes \begin{bmatrix} y \end{bmatrix} =
\begin{bmatrix}
\dfrac{\partial y}{\partial x_{11}} & \cdots & \dfrac{\partial y}{\partial x_{1n}} \\ \vdots & \ddots & \vdots \\ \dfrac{\partial y}{\partial x_{m1}} & \cdots & \dfrac{\partial y}{\partial x_{mn}} \end{bmatrix}
∂ X ∂ y = ∂ X ∂ ⊗ y = ∂ x 11 ∂ ⋮ ∂ x m 1 ∂ ⋯ ⋱ ⋯ ∂ x 1 n ∂ ⋮ ∂ x mn ∂ ⊗ [ y ] = ∂ x 11 ∂ y ⋮ ∂ x m 1 ∂ y ⋯ ⋱ ⋯ ∂ x 1 n ∂ y ⋮ ∂ x mn ∂ y
Vector By Differentiated by a Scalar This is analogous to the vector-valued function differentiation .
∂ y ∂ x = y ′ ( x ) = lim h → 0 y ( x + h ) − y ( x ) h
\dfrac{\partial \mathbf{y}}{\partial x} = \mathbf{y}^{\prime}(x) = \lim\limits_{h \to 0} \dfrac{\mathbf{y}(x+h) - \mathbf{y}(x)}{h}
∂ x ∂ y = y ′ ( x ) = h → 0 lim h y ( x + h ) − y ( x )
As the function value y \mathbf{y} y x x x
∂ y ∂ x = ∂ ∂ x ⊗ y = [ ∂ ∂ x ] ⊗ [ y 1 ⋮ y m ] = [ ∂ y 1 ∂ x ⋮ ∂ y m ∂ x ]
\dfrac{\partial \mathbf{y}}{\partial x} = \dfrac{\partial }{\partial x} \otimes \mathbf{y} = \begin{bmatrix} \dfrac{\partial }{\partial x} \end{bmatrix} \otimes \begin{bmatrix} y_{1} \\ \vdots \\ y_{m} \end{bmatrix} =
\begin{bmatrix} \dfrac{\partial y_{1}}{\partial x} \\ \vdots \\ \dfrac{\partial y_{m}}{\partial x} \end{bmatrix}
∂ x ∂ y = ∂ x ∂ ⊗ y = [ ∂ x ∂ ] ⊗ y 1 ⋮ y m = ∂ x ∂ y 1 ⋮ ∂ x ∂ y m
Differentiated by a Vector Differentiating a vector by a vector is equivalent to the Jacobian . Considering the definition of the Jacobian,
∂ y ∂ x = ∂ ( y 1 , y 2 , … , y m ) ∂ ( x 1 , x 2 , … , x n ) = [ ∂ y 1 ∂ x 1 ⋯ ∂ y 1 ∂ x n ⋮ ⋱ ⋮ ∂ y m ∂ x 1 ⋯ ∂ y m ∂ x n ]
\dfrac{\partial \mathbf{y}}{\partial \mathbf{x}} = \dfrac{\partial (y_{1}, y_{2}, \dots, y_{m})}{\partial (x_{1}, x_{2}, \dots, x_{n})} =
\begin{bmatrix}
\dfrac{\partial y_{1}}{\partial x_{1}} & \cdots & \dfrac{\partial y_{1}}{\partial x_{n}} \\ \vdots & \ddots & \vdots \\ \dfrac{\partial y_{m}}{\partial x_{1}} & \cdots & \dfrac{\partial y_{m}}{\partial x_{n}} \end{bmatrix}
∂ x ∂ y = ∂ ( x 1 , x 2 , … , x n ) ∂ ( y 1 , y 2 , … , y m ) = ∂ x 1 ∂ y 1 ⋮ ∂ x 1 ∂ y m ⋯ ⋱ ⋯ ∂ x n ∂ y 1 ⋮ ∂ x n ∂ y m
It produces the same outcome as when expressed with the Kronecker product.
∂ y ∂ x = ∂ ∂ x ⊗ y = [ ∂ ∂ x 1 ⋯ ∂ ∂ x n ] ⊗ [ y 1 ⋮ y m ] = [ ∂ y 1 ∂ x 1 ⋯ ∂ y 1 ∂ x n ⋮ ⋱ ⋮ ∂ y m ∂ x 1 ⋯ ∂ y m ∂ x n ]
\dfrac{\partial \mathbf{y}}{\partial \mathbf{x}} = \dfrac{\partial }{\partial \mathbf{x}} \otimes \mathbf{y} = \begin{bmatrix} \dfrac{\partial }{\partial x_{1}} & \cdots & \dfrac{\partial }{\partial x_{n}} \end{bmatrix} \otimes \begin{bmatrix} y_{1} \\ \vdots \\ y_{m} \end{bmatrix} =
\begin{bmatrix}
\dfrac{\partial y_{1}}{\partial x_{1}} & \cdots & \dfrac{\partial y_{1}}{\partial x_{n}} \\ \vdots & \ddots & \vdots \\ \dfrac{\partial y_{m}}{\partial x_{1}} & \cdots & \dfrac{\partial y_{m}}{\partial x_{n}} \end{bmatrix}
∂ x ∂ y = ∂ x ∂ ⊗ y = [ ∂ x 1 ∂ ⋯ ∂ x n ∂ ] ⊗ y 1 ⋮ y m = ∂ x 1 ∂ y 1 ⋮ ∂ x 1 ∂ y m ⋯ ⋱ ⋯ ∂ x n ∂ y 1 ⋮ ∂ x n ∂ y m
Matrix By Differentiated by a Scalar This is analogous to the differentiation of a matrix function . In terms of the Kronecker product, it can be expressed as,
∂ Y ∂ x = ∂ ∂ x ⊗ Y = [ ∂ ∂ x ] ⊗ [ y 11 y 12 ⋯ y 1 n y 21 y 22 ⋯ y 2 n ⋮ ⋮ ⋱ ⋮ y m 1 y m 2 ⋯ y m n ] = [ ∂ y 11 ∂ x ⋯ ∂ y 1 n ∂ x ⋮ ⋱ ⋮ ∂ y m 1 ∂ x ⋯ ∂ y m n ∂ x ]
\dfrac{\partial \mathbf{Y}}{\partial x} = \dfrac{\partial }{\partial x} \otimes \mathbf{Y} = \begin{bmatrix} \dfrac{\partial }{\partial x} \end{bmatrix} \otimes \begin{bmatrix} y_{11} & y_{12} & \cdots & y_{1n} \\ y_{21} & y_{22} & \cdots & y_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ y_{m1} & y_{m2} & \cdots & y_{mn} \end{bmatrix}
= \begin{bmatrix} \dfrac{\partial y_{11}}{\partial x} & \cdots & \dfrac{\partial y_{1n}}{\partial x} \\ \vdots & \ddots & \vdots \\ \dfrac{\partial y_{m1}}{\partial x} & \cdots & \dfrac{\partial y_{mn}}{\partial x} \end{bmatrix}
∂ x ∂ Y = ∂ x ∂ ⊗ Y = [ ∂ x ∂ ] ⊗ y 11 y 21 ⋮ y m 1 y 12 y 22 ⋮ y m 2 ⋯ ⋯ ⋱ ⋯ y 1 n y 2 n ⋮ y mn = ∂ x ∂ y 11 ⋮ ∂ x ∂ y m 1 ⋯ ⋱ ⋯ ∂ x ∂ y 1 n ⋮ ∂ x ∂ y mn
