logo

Matrix Calculus of Trace 📂Vector Analysis

Matrix Calculus of Trace

Formula

  • Let X\mathbf{X} be n×nn \times n matrix. Define X=X\dfrac{\partial }{\partial \mathbf{X}} = \nabla_{\mathbf{X}} as the matrix gradient. Then, the following formula holds:

    Tr(X)X=I,Tr(aX)X=aI(1) \dfrac{\partial \Tr(\mathbf{X})}{\partial \mathbf{X}} = I, \qquad \dfrac{\partial \Tr(a\mathbf{X})}{\partial \mathbf{X}} = aI \tag{1}

    Here, aRa \in \mathbb{R} is a constant (scalar), and II is an identity matrix.

  • Suppose ARn×p\mathbf{A} \in \mathbb{R}^{n \times p} and XRp×n\mathbf{X} \in \mathbb{R}^{p \times n}. Then, the following holds: Tr(AX)X=Tr(XA)X=AT(2) \dfrac{\partial \Tr(\mathbf{A}\mathbf{X})}{\partial \mathbf{X}} = \dfrac{\partial \Tr( \mathbf{X}\mathbf{A})}{\partial \mathbf{X}} = \mathbf{A}^{\mathsf{T}} \tag{2} Tr(AXT)X=Tr(XTA)X=A \dfrac{\partial \Tr(\mathbf{A}\mathbf{X}^{\mathsf{T}})}{\partial \mathbf{X}} = \dfrac{\partial \Tr( \mathbf{X}^{\mathsf{T}}\mathbf{A})}{\partial \mathbf{X}} = \mathbf{A}

    • As a corollary, the following holds for ARn×p\mathbf{A} \in \mathbb{R}^{n \times p}, XRp×q\mathbf{X} \in \mathbb{R}^{p \times q}, BRq×n\mathbf{B} \in \mathbb{R}^{q \times n}: Tr(AXB)X=ATBT(3) \dfrac{\partial \Tr(\mathbf{A}\mathbf{X}\mathbf{B})}{\partial \mathbf{X}} = \mathbf{A}^{\mathsf{T}}\mathbf{B}^{\mathsf{T}} \tag{3}
  • Let ARn×n\mathbf{A} \in \mathbb{R}^{n \times n} and XRm×n\mathbf{X} \in \mathbb{R}^{m \times n}. Then, the following holds: Tr(AXTX)X=Tr(XTXA)X=Tr(XAXT)X=X(AT+A)(4) \dfrac{\partial \Tr(\mathbf{A}\mathbf{X}^{\mathsf{T}}\mathbf{X})}{\partial \mathbf{X}} = \dfrac{\partial \Tr(\mathbf{X}^{\mathsf{T}}\mathbf{X}\mathbf{A})}{\partial \mathbf{X}} = \dfrac{\partial \Tr(\mathbf{X}\mathbf{A}\mathbf{X}^{\mathsf{T}})}{\partial \mathbf{X}} = \mathbf{X}(\mathbf{A}^{\mathsf{T}} + \mathbf{A}) \tag{4} For A,XRn×n\mathbf{A}, \mathbf{X} \in \mathbb{R}^{n \times n}, the following holds: Tr(AXX)X=XTAT+ATXT(5) \dfrac{\partial \Tr(\mathbf{A} \mathbf{X}\mathbf{X})}{\partial \mathbf{X}} = \mathbf{X}^{\mathsf{T}}\mathbf{A}^{\mathsf{T}} + \mathbf{A}^{\mathsf{T}}\mathbf{X}^{\mathsf{T}} \tag{5} For A,B,XRn×n\mathbf{A}, \mathbf{B}, \mathbf{X} \in \mathbb{R}^{n \times n}, the following holds: XTr(AXTBX)=Tr(AXTBX)X=BXA+BTXAT(6) \nabla_{\mathbf{X}} \Tr (\mathbf{A} \mathbf{X}^{\mathsf{T}} \mathbf{B} \mathbf{X}) = \dfrac{\partial \Tr (\mathbf{A} \mathbf{X}^{\mathsf{T}} \mathbf{B} \mathbf{X})}{\partial \mathbf{X}} = \mathbf{B} \mathbf{X} \mathbf{A} + \mathbf{B}^{\mathsf{T}} \mathbf{X} \mathbf{A}^{\mathsf{T}} \tag{6}

  • For XRn×n\mathbf{X} \in \mathbb{R}^{n \times n} and natural number nn, the following holds: Tr(Xn)X=n(Xn1)T(7) \dfrac{\partial \Tr(\mathbf{X}^{n})}{\partial \mathbf{X}} = n(\mathbf{X}^{n-1})^{\mathsf{T}} \tag{7}

Explanation

The trace TrA\Tr A of a matrix AA is the sum of all its diagonal elements. While it can be simply viewed as a value, it can also be considered a function that converts a matrix into a scalar. In this manner, the trace is a function defined as follows:

Tr:Rn×nR \Tr : \mathbb{R}^{n \times n} \to \mathbb{R}

When discussing a function, it is essential to talk about differentiation. Reflecting on the function’s value, Tr(A)=i=1naii\Tr(A) = \sum\limits_{i=1}^{n} a_{ii} indicates that it is merely a linear function, implying there is no concern about differentiability. However, the fact that the variable is a matrix might not be intuitive. For more details, refer to the gradient matrix article.

Examining the results above reveals similarities with scalar differentiation, leading one to accept the trace as a tool that facilitates intuitive handling of matrix differentiation. Particularly when all matrices such as A,B,X\mathbf{A}, \mathbf{B}, \mathbf{X} are symmetric, which means the specific nature of matrices doesn’t need to be considered, the results align closely with polynomial function differentiation.

  • (1)(1): The identity matrix II acts as an identity element in matrix multiplication, corresponding to scalar differentiation dxdx=1\dfrac{d x}{d x} = 1.
  • (2)(2), (3)(3): Provide intuitive results corresponding to linear function differentiation.
  • (4)(4), (5)(5): Correspond with quadratic function differentiation.
  • (7)(7): Correspond with polynomial function differentiation.

While the proof below shows direct calculations, using the method known as trace trick simplifies the calculations. For any arbitrary form concerning X\mathbf{X} or expressions containing multiple instances of X\mathbf{X}, direct calculation is practically unfeasible, necessitating the use of the trace trick.

More formulas can be found in Table of Scalar Function Matrix Differentiation.

Proof

(1)(1)

Since Tr(X)=i=1nxii\Tr (\mathbf{X}) = \sum\limits_{i=1}^{n} x_{ii}, the following holds:

Tr(X)xij={1,i=j0,ij \dfrac{\partial \Tr (\mathbf{X})}{\partial x_{ij}} = \begin{cases} 1 &, i=j \\ 0 &, i \neq j \end{cases}

Thus, the following result is obtained:

Tr(X)X=[Tr(X)x11Tr(X)x1nTr(X)xn1Tr(X)xnn]=[100010001]=I \dfrac{\partial \Tr (\mathbf{X})}{\partial \mathbf{X}} = \begin{bmatrix} \dfrac{\partial \Tr (\mathbf{X})}{\partial x_{11}} & \cdots & \dfrac{\partial \Tr (\mathbf{X})}{\partial x_{1n}} \\ \vdots & \ddots & \vdots \\ \dfrac{\partial \Tr (\mathbf{X})}{\partial x_{n1}} & \cdots & \dfrac{\partial \Tr (\mathbf{X})}{\partial x_{nn}} \end{bmatrix} = \begin{bmatrix} 1 & 0 & \cdots & 0 \\ 0 & 1 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & 1 \end{bmatrix} = I

(2)(2)

Let’s assume ARn×p\mathbf{A} \in \mathbb{R}^{n \times p} and XRp×n\mathbf{X} \in \mathbb{R}^{p \times n}. Initially, by the cyclic property of trace, we have Tr(AX)=Tr(XA)\Tr (\mathbf{A}\mathbf{X}) = \Tr(\mathbf{X}\mathbf{A}).

Cyclic property of trace

Tr(AB)=Tr(BA) \Tr(AB) = \Tr(BA)

The element ijij of matrix AX\mathbf{A} \mathbf{X} is k=1paikxkj\sum\limits_{k=1}^{p} a_{ik} x_{kj}; therefore, it is Tr(AX)=i=1nk=1paikxki\Tr (\mathbf{A} \mathbf{X}) = \sum\limits_{i=1}^{n}\sum\limits_{k=1}^{p} a_{ik} x_{ki}. Thus, it is Tr(AX)xij=aji\dfrac{\partial \Tr (\mathbf{A}\mathbf{X})}{\partial x_{ij}} = a_{ji}, and the following holds:

Tr(AX)X=[Tr(AX)x11Tr(AX)x1nTr(AX)xp1Tr(AX)xpn]=[a11a21an1a12a22an2a1pa2panp]=AT \begin{align*} \dfrac{\partial \Tr (\mathbf{A}\mathbf{X})}{\partial \mathbf{X}} &=\begin{bmatrix} \dfrac{\partial \Tr (\mathbf{A} \mathbf{X})}{\partial x_{11}} & \cdots & \dfrac{\partial \Tr (\mathbf{A} \mathbf{X})}{\partial x_{1n}} \\ \vdots & \ddots & \vdots \\ \dfrac{\partial \Tr (\mathbf{A} \mathbf{X})}{\partial x_{p1}} & \cdots & \dfrac{\partial \Tr (\mathbf{A} \mathbf{X})}{\partial x_{pn}} \end{bmatrix} \\ &= \begin{bmatrix} a_{11} & a_{21} & \cdots & a_{n1} \\ a_{12} & a_{22} & \cdots & a_{n2} \\ \vdots & \vdots & \ddots & \vdots \\ a_{1p} & a_{2p} & \cdots & a_{np} \end{bmatrix} \\ &= \mathbf{A}^{\mathsf{T}} \end{align*}

By using this result and the cyclic property of trace, (3)(3) is obtained immediately.

(4)(4)

According to the Matrix Power Formula, the ijij component, trace, and partial derivative of AXTX\mathbf{A}\mathbf{X}^{\mathsf{T}}\mathbf{X} are as follows:

[AXTX]ij=k=1n=1maikxkxj,Tr(AXTX)=s=1nk=1n=1maskxkxs [\mathbf{A}\mathbf{X}^{\mathsf{T}}\mathbf{X}]_{ij} = \sum\limits_{k=1}^{n}\sum\limits_{\ell=1}^{m} a_{ik} x_{\ell k}x_{\ell j}, \quad \Tr (\mathbf{A}\mathbf{X}^{\mathsf{T}}\mathbf{X}) = \sum\limits_{s=1}^{n}\sum\limits_{k=1}^{n}\sum\limits_{\ell=1}^{m} a_{sk} x_{\ell k}x_{\ell s}

Tr(AXTX)xij=k=1najkxik+s=1nasjxis=k=1nxikajk+k=1nxikakj=[XAT]ij+[XA]ij \begin{align*} \dfrac{\partial \Tr (\mathbf{A}\mathbf{X}^{\mathsf{T}}\mathbf{X})}{\partial x_{ij}} &= \sum\limits_{k=1}^{n}a_{jk}x_{ik} + \sum\limits_{s=1}^{n}a_{sj}x_{is} \\ &= \sum\limits_{k=1}^{n}x_{ik}a_{jk} + \sum\limits_{k=1}^{n}x_{ik}a_{kj} &= [\mathbf{X}\mathbf{A}^{\mathsf{T}}]_{ij} + [\mathbf{X}\mathbf{A}]_{ij} \end{align*}

Hence,

Tr(AXTX)X=XAT+XA=X(AT+A) \dfrac{\partial \Tr (\mathbf{A}\mathbf{X}^{\mathsf{T}}\mathbf{X})}{\partial \mathbf{X}} = \mathbf{X}\mathbf{A}^{\mathsf{T}} + \mathbf{X}\mathbf{A} = \mathbf{X}(\mathbf{A}^{\mathsf{T}} + \mathbf{A})

(5)(5)

Since [AXX]ij=k,s=1naikxksxsj[\mathbf{A} \mathbf{X} \mathbf{X}]_{ij} = \sum\limits_{k, s=1}^{n} a_{ik}x_{ks}x_{sj},

Tr(AXX)==1nk,s=1nakxksxs \Tr (\mathbf{A}\mathbf{X} \mathbf{X}) = \sum\limits_{\ell= 1}^{n} \sum\limits_{k, s=1}^{n} a_{\ell k}x_{ks}x_{s\ell}

Therefore, the partial derivative is as follows:

Tr(AXX)xij==1naixj+k=1najkxki \dfrac{\partial \Tr (\mathbf{A}\mathbf{X} \mathbf{X})}{\partial x_{ij}} = \sum\limits_{\ell=1}^{n} a_{\ell i}x_{j\ell} + \sum\limits_{k=1}^{n} a_{jk}x_{ki}

Thus, the following is obtained:

[Tr(AXX)X]ij==1naixj+k=1najkxki=[ATXT]ij+[XTAT] \left[ \dfrac{\partial \Tr (\mathbf{A}\mathbf{X} \mathbf{X})}{\partial \mathbf{X}} \right]_{ij} = \sum\limits_{\ell=1}^{n} a_{\ell i}x_{j\ell} + \sum\limits_{k=1}^{n} a_{jk}x_{ki} = [\mathbf{A}^{\mathsf{T}}\mathbf{X}^{\mathsf{T}}]_{ij} + [\mathbf{X}^{\mathsf{T}}\mathbf{A}^{\mathsf{T}}]

    Tr(AXX)X=ATXT+XTAT \implies \dfrac{\partial \Tr (\mathbf{A}\mathbf{X} \mathbf{X})}{\partial \mathbf{X}} = \mathbf{A}^{\mathsf{T}}\mathbf{X}^{\mathsf{T}} + \mathbf{X}^{\mathsf{T}}\mathbf{A}^{\mathsf{T}}

(6)(6)

Assuming we have followed the proof process above without difficulty, we shall briefly describe:

[AXTBX]ij=k,s,=1naikxskbsxj \left[ \mathbf{A} \mathbf{X}^{\mathsf{T}} \mathbf{B} \mathbf{X} \right]_{ij} = \sum_{k,s,\ell = 1}^{n} a_{ik} x_{sk} b_{s\ell} x_{\ell j}

    Tr(AXTBX)=r=1nk,s,=1narkxskbsxr \implies \Tr (\mathbf{A} \mathbf{X}^{\mathsf{T}} \mathbf{B} \mathbf{X}) = \sum\limits_{r=1}^{n}\sum_{k,s,\ell = 1}^{n} a_{rk} x_{sk} b_{s\ell} x_{\ell r}

    [XTr(AXTBX)]ij=r,arjbixr+k,sajkxskbsi=r,bixrarj+k,sbsixskajk=[BXA]ij+[BTXAT]ij \begin{align*} \implies \left[ \nabla_{\mathbf{X}} \Tr (\mathbf{A} \mathbf{X}^{\mathsf{T}} \mathbf{B} \mathbf{X}) \right]_{ij} &= \sum_{r,\ell} a_{rj}b_{i\ell}x_{\ell r} + \sum_{k,s} a_{jk}x_{sk}b_{si} \\ &= \sum_{r,\ell} b_{i\ell}x_{\ell r}a_{rj} + \sum_{k,s} b_{si}x_{sk}a_{jk} \\ &= [\mathbf{B} \mathbf{X} \mathbf{A}]_{ij} + [\mathbf{B}^{\mathsf{T}} \mathbf{X} \mathbf{A}^{\mathsf{T}}]_{ij} \end{align*}

    XTr(AXTBX)=BXA+BTXAT \implies \nabla_{\mathbf{X}} \Tr (\mathbf{A} \mathbf{X}^{\mathsf{T}} \mathbf{B} \mathbf{X}) = \mathbf{B} \mathbf{X} \mathbf{A} + \mathbf{B}^{\mathsf{T}} \mathbf{X} \mathbf{A}^{\mathsf{T}}

(7)(7)

[Xn]ij=k(2),,k(n)=1nxik(2)xk(2)k(3)xk(n)j [\mathbf{X}^{n}]_{ij} = \sum\limits_{k_{(2)}, \dots, k_{(n)}=1}^{n} x_{ik_{(2)}} x_{k_{(2)}k_{(3)}} \cdots x_{k_{(n)}j}

    Tr(Xn)==1nk(2),,k(n)=1nxk(2)xk(2)k(3)xk(n) \implies \Tr(\mathbf{X}^{n}) = \sum\limits_{\ell=1}^{n} \sum\limits_{k_{(2)}, \dots, k_{(n)}=1}^{n} x_{\ell k_{(2)}} x_{k_{(2)}k_{(3)}} \cdots x_{k_{(n)}\ell}

    [XTr(Xn)]ij=nk(3),,k(n)=1nxjk(3)xk(3)k(4)xk(n)i=n[(Xn1)T]ij \implies [\nabla_{\mathbf{X}} \Tr(\mathbf{X}^{n})]_{ij} = n \sum\limits_{k_{(3)}, \dots, k_{(n)}=1}^{n} x_{j k_{(3)}} x_{k_{(3)}k_{(4)}} \cdots x_{k_{(n)}i} = n[(\mathbf{X}^{n-1})^{\mathsf{T}}]_{ij}

    XTr(Xn)=n(Xn1)T \implies \nabla_{\mathbf{X}} \Tr(\mathbf{X}^{n}) = n(\mathbf{X}^{n-1})^{\mathsf{T}}