📂Matrix Algebra

Hadamard Product of Matrices

Definition

The Hadamard product $A \odot B$ of two matrices $A, B \in M_{m \times n}$ is defined as follows.

$$A \odot B = \begin{bmatrix} a_{11} & \cdots & a_{1n} \\ \vdots & \ddots & \vdots \\ a_{m1} & \cdots & a_{mn} \end{bmatrix} \odot\begin{bmatrix} b_{11} & \cdots & b_{1n} \\ \vdots & \ddots & \vdots \\ b_{m1} & \cdots & b_{mn} \end{bmatrix} := \begin{bmatrix} a_{11}b_{11} & \cdots & a_{1n}b_{1n} \\ \vdots & \ddots & \vdots \\ a_{m1}b_{m1} & \cdots & a_{mn}b_{mn} \end{bmatrix}$$

$$[A \odot B]_{ij} := [A]_{ij} [B]_{ij}$$

Description

The code for $\odot$’s $\TeX$ is \odot.

It is also commonly called the elementwise product. Unlike matrix multiplication, it is only defined for matrices of the same size and the commutative law applies.

• $A \odot B = B \odot A$
• $(A \odot B) \odot C = A \odot (B \odot C)$
• $A \odot (B + C) = A \odot B + A \odot C$
• $k(A \odot B) = (kA) \odot B = A \odot (kB)$

Hadamard Product of Vectors

The Hadamard product of two vectors $\mathbf{x}$ and $\mathbf{y} \in \mathbb{R}^{n}$ is defined as follows.

$$\mathbf{x} \odot \mathbf{y} = \begin{bmatrix} x_{1}y_{1} \\ \vdots \\ x_{n}y_{n} \end{bmatrix}$$

The definition itself is a special case for matrices regarding the Hadamard product of matrices with $n \times 1$. The following equation holds for diagonal matrices.

$$\mathbf{x} \odot \mathbf{y} = \diag(\mathbf{x})\mathbf{y} = \diag(\mathbf{y})\mathbf{x}$$

Hadamard Product of a Vector and a Matrix

The Hadamard product between a vector $\mathbf{x} \in \mathbb{R}^{n}$ and a matrix $\mathbf{Y} = \begin{bmatrix} \vert & & \vert \\ \mathbf{y}_{1} & \cdots & \mathbf{y}_{n} \\ \vert & & \vert \end{bmatrix}$ is defined as follows.

$$\mathbf{x} \odot \mathbf{Y} = \begin{bmatrix} \!\!\vert & & \!\!\vert \\ \mathbf{x} \odot \mathbf{y}_{1} & \cdots & \mathbf{x} \odot \mathbf{y}_{n} \\ \!\!\vert & & \!\!\vert \end{bmatrix}$$

Simply put, it takes the Hadamard product of the vector with each column of the matrix. The definition immediately gives the equation below.

$$\mathbf{x} \odot \mathbf{Y} = \diag(\mathbf{x}) \mathbf{Y}$$

Looking at the two definitions, one can see that for vector $\mathbf{x}$, defining $\mathbf{x} \odot$ itself as a matrix of $\mathbf{x} \odot := \diag(\mathbf{x})$ is reasonable.

In Programming Languages

Such pointwise operations are implemented by adding a dot . to the existing operation symbols. This notation is quite intuitive; for example, if multiplication is *, then elementwise multiplication is .*.

Julia

julia> A = [1 2 3; 4 5 6]
2×3 Matrix{Int64}:
 1  2  3
 4  5  6

julia> B = [2 2 2; 2 2 2]
2×3 Matrix{Int64}:
 2  2  2
 2  2  2

julia> A.*B
2×3 Matrix{Int64}:
 2   4   6
 8  10  12

MATLAB

>> A = [1 2 3; 4 5 6]
A =
     1     2     3
     4     5     6

>> B = [2 2 2; 2 2 2]
B =
     2     2     2
     2     2     2

 
>> A.*B
ans =
     2     4     6
     8    10    12

However, in the case of Python, since it is not a language for scientific computing like Julia or MATLAB, it is not implemented in the same way. The multiplication symbol * stands for elementwise multiplication, and matrix multiplication is denoted by @.

>>> import numpy
>>> A = np.array([[1, 2, 3], [4, 5, 6]])
>>> B = np.array([[2, 2, 2], [2, 2, 2]])
>>> A*B
array([[ 2,  4,  6],
       [ 8, 10, 12]])

>>> import torch
>>> A = torch.tensor([[1, 2, 3],[4, 5, 6]])
>>> B = torch.tensor([[2, 2, 2],[2, 2, 2]])
>>> A*B
tensor([[ 2,  4,  6],
        [ 8, 10, 12]])