Tensor Product of Product Vectors
Buildup
- For convenience, we will develop the concept in the complex number space $\mathbb{C}$, but $\mathbb{R}$ or any vector space is also applicable.
Let’s denote the set of functions from the finite set $\Gamma$ to the complex number space as indicated by $\mathbb{C}^{\Gamma}$.
$$ \mathbb{C}^{\Gamma} = \left\{ f : \Gamma \to \mathbb{C} \right\} $$
When $\Gamma = \mathbf{n} = \left\{ 1, \dots, n \right\}$, it essentially becomes $\mathbb{C}^{\mathbf{n}} = \mathbb{C}^{n}$, and the tensor product of vector spaces is defined as follows.
$$ \mathbb{C}^{\Gamma_{1}} \otimes \mathbb{C}^{\Gamma_{2}} := \mathbb{C}^{\Gamma_{1} \times \Gamma_{2}} $$
Assuming $v_{i} \in \mathbb{C}^{\Gamma_{i}}$ and $n_{i} = \left| \Gamma_{i} \right|$, let the standard basis corresponding to $\mathbb{C}^{\Gamma_{i}}$ be denoted as $\left\{ e_{j_{i}} \right\}_{j_{i} \in \Gamma_{i}}$ respectively. Then, $v_{i}$ can be represented as follows.
$$ \begin{align*} v_{1} &: \left\{ 1, \dots, n_{1} \right\} \to \mathbb{C} &&& v_{2} &: \left\{ 1, \dots, n_{2} \right\} \to \mathbb{C} \\ v_{1} &= (v_{1}(1), \dots, v_{1}(n_{1})) \in \mathbb{C}^{n_{1}} &&& v_{2} &= (v_{2}(1), \dots, v_{2}(n_{2})) \in \mathbb{C}^{n_{2}} \\ & = \sum \limits_{j_{1} = 1}^{n_{1}}v_{1}(j_{1}) e_{j_{1}} &&&& = \sum \limits_{j_{2} = 1}^{n_{2}}v_{2}(j_{2}) e_{j_{2}} \end{align*} $$
The elements of the tensor product $\mathbb{C}^{\Gamma_{1}} \otimes \mathbb{C}^{\Gamma_{2}}$ that can be represented as $v_{1} \otimes v_{2}$ are called the product vector of $v_{1}$ and $v_{2}$.
Definition
The product vector $v_{1} \otimes v_{2}$ of $v_{1}$ and $v_{2}$ is defined as follows.
$$ \begin{align*} v_{1} \otimes v_{2} &= \left( \sum \limits_{j_{1} \in \Gamma_{1}}v_{1}(j_{1}) e_{j_{1}} \right) \otimes \left( \sum \limits_{j_{2} \in \Gamma_{2}}v_{2}(j_{2}) e_{j_{2}} \right) \\ &:= \sum\limits_{(j_{1}, j_{2}) \in \Gamma_{1} \times \Gamma_{2}} \left( \prod\limits_{i=1}^{2} v_{i}(j_{i}) \right) e_{j_{1}} \otimes e_{j_{2}} \end{align*} $$
In this case, $v_{1} \otimes v_{2}$ becomes an element of $\mathbb{C}^{\Gamma_{1}} \otimes \mathbb{C}^{\Gamma_{2}}$ by the definition of the tensor product.
$$ v_{1} \otimes v_{2} := \sum\limits_{(j_{1}, j_{2}) \in \Gamma_{1} \times \Gamma_{2}} \left( \prod\limits_{i=1}^{2} v_{i}(j_{i}) \right) e_{j_{1}} \otimes e_{j_{2}} \in \mathbb{C}^{\Gamma_{1}} \otimes \mathbb{C}^{\Gamma_{2}} $$
Explanation
Not all elements of $\mathbb{C}^{\Gamma_{1}} \otimes \mathbb{C}^{\Gamma_{2}}$ can be represented as the product vector form of $v_{1} \otimes v_{2}$. For example, the following vector can be expressed as the product of two vectors, but $(e_{1} \otimes e_{1}) + (e_{2} \otimes e_{2})$ cannot.
$$ e_{1} \otimes e_{1} - e_{1} \otimes e_{2} + e_{2} \otimes e_{1} - e_{2} \otimes e_{2} = (e_{1} + e_{2}) \otimes (e_{1} - e_{2}) $$
As a simple example, let’s further unpack the definition above. Let’s say $\Gamma_{1} = \left\{ 1, 2 \right\}$, $\Gamma_{2} = \left\{ 1, 2, 3 \right\}$. Assume $v_{i} \in \mathbb{C}^{\Gamma_{i}}$. Let the standard basis corresponding to $\mathbb{C}^{\Gamma_{1}} = \mathbb{C}^{2}$ be $\left\{ e_{j_{1}} \right\}_{j_{1} \in \Gamma_{1}}$, and the standard basis corresponding to $\mathbb{C}^{\Gamma_{2}} = \mathbb{C}^{3}$ be $\left\{ e_{j_{2}} \right\}_{j_{2} \in \Gamma_{2}}$. Then, $v_{1}$, $v_{2}$ are as follows.
$$ \begin{align*} v_{1} &: \left\{ 1, 2 \right\} \to \mathbb{C} &&& v_{2} &: \left\{ 1, 2, 3 \right\} \to \mathbb{C} \\ v_{1} &= (v_{1}(1), v_{1}(2)) \in \mathbb{C}^{2} &&& v_{2} &= (v_{2}(1), v_{2}(2), v_{2}(3)) \in \mathbb{C}^{3} \\ & = \sum \limits_{j_{1} = 1}^{2}v_{1}(j_{1}) e_{j_{1}} &&&& = \sum \limits_{j_{2} = 1}^{3}v_{2}(j_{2}) e_{j_{2}} \end{align*} $$
Then, the product vector of $v_{1}$ and $v_{2}$ is as follows.
$$ \begin{align*} v_{1} \otimes v_{2} &= (v_{1}(1), v_{1}(2)) \otimes (v_{2}(1), v_{2}(2), v_{2}(3)) \\ &= \left( \sum \limits_{j_{1} = 1}^{2} v_{1}(j_{1}) e_{j_{1}} \right) \otimes \left( \sum \limits_{j_{2} = 1}^{3} v_{2}(j_{2}) e_{j_{2}} \right) \\ &:= \sum\limits_{(j_{1}, j_{2}) \in \Gamma_{1} \times \Gamma_{2}} \left( \prod\limits_{i=1}^{2} v_{i}(j_{i}) \right) e_{j_{1}} \otimes e_{j_{2}} \in \mathbb{C}^{\Gamma_{1}} \otimes \mathbb{C}^{\Gamma_{2}} \\ &= v_{1}(1)v_{2}(1)e_{1} \otimes e_{1} + v_{1}(1)v_{2}(2)e_{1} \otimes e_{2} + v_{1}(1)v_{2}(3)e_{1} \otimes e_{3} \\ &\quad + v_{1}(2)v_{2}(1)e_{1} \otimes e_{1} + v_{1}(2)v_{2}(2)e_{1} \otimes e_{2} + v_{1}(2)v_{2}(3)e_{1} \otimes e_{3} \\ &= \left( v_{1}(1)v_{2}(1), v_{1}(1)v_{2}(2), v_{1}(1)v_{2}(3), v_{1}(2)v_{2}(1), v_{1}(2)v_{2}(2), v_{1}(2)v_{2}(3) \right) \\ &\in \mathbb{C}^{6} \cong \mathbb{C}^{\Gamma_{1}} \otimes \mathbb{C}^{\Gamma_{2}} \end{align*} $$
By carefully observing the components of $v_{1} \otimes v_{2}$, one could guess its relation to matrices.
Coordinate Matrix
Consider the matrix space $M_{m \times n}(\mathbb{C})$. If $E_{ij}$ has components $(i,j)$ as $1$ and the rest as $0$, it is called an $m \times n$ matrix, and $\left\{ E_{ij} \right\}$ becomes the basis of $M_{m\times n}(\mathbb{C})$. Let $\phi$ be a linear transformation that maps the basis vector $e_{i} \otimes e_{j}$ of the tensor product $\mathbb{C}^{m} \otimes \mathbb{C}^{n}$ to $E_{ij}$.
$$ \begin{align*} \phi : \mathbb{C}^{m} \otimes \mathbb{C}^{n} &\to M_{m \times n} (\mathbb{C}) \\ e_{i} \otimes e_{j} &\mapsto E_{ij} \end{align*} $$
Since it maps a basis to a basis, it becomes an isomorphism. If two vectors $v \in \mathbb{C}^{m}$, $w \in \mathbb{C}^{n}$ are as follows,
$$ v = \sum_{i} \alpha_{i}e_{i} = \begin{bmatrix} \alpha_{1} \\ \vdots \\ \alpha_{m} \end{bmatrix} \qquad w = \sum_{j} \beta_{j}e_{j} = \begin{bmatrix} \beta_{1} \\ \vdots \\ \beta_{n} \end{bmatrix} $$
Sending the product vector $v, w$ through $\phi$ results in the following.
$$ \begin{align*} \phi ( v \otimes w ) &= \phi \left( \sum\limits_{i,j} \alpha_{i}\beta_{j} e_{i} \otimes e_{j} \right) \\ &= \sum\limits_{i,j} \alpha_{i}\beta_{j} \phi \left( e_{i} \otimes e_{j} \right) \\ &= \sum\limits_{i,j} \alpha_{i}\beta_{j} E_{ij} \\ &= \begin{bmatrix} \alpha_{1}\beta_{1} & \cdots & \alpha_{1}\beta_{n} \\ \vdots & \ddots & \vdots \\ \alpha_{m}\beta_{1} & \cdots & \alpha_{m}\beta_{n} \\ \end{bmatrix} \\ &= \begin{bmatrix} \alpha_{1} \\ \vdots \\ \alpha_{m} \end{bmatrix} \begin{bmatrix} \beta_{1} & \cdots & \beta_{n} \end{bmatrix} \\ &= vw^{T} \end{align*} $$
This corresponds to a matrix whose elements are $\alpha_{i}\beta_{j}$. Therefore, by $\phi$, the product vector $v \otimes w$ corresponds to a single $m \times n$. The matrix $\phi (v \otimes w) = vw^{T}$ is called the coordinate matrix of $v \otimes w$ with respect to the standard basis. This concept can be seen as analogous to a vector’s coordinate vector.
Generalization
For finite sets $\Gamma_{i} (1 \le i \le r)$, $\Gamma = \Gamma_{1} \times \cdots \times \Gamma_{r}$, $v_{i} \in \mathbb{C}^{\Gamma_{i}}$, the product vectors of $v_{i}$ are defined as follows.
$$ \begin{align*} v_{1} \otimes \cdots \otimes v_{r} &= \left( \sum \limits_{j_{1} \in \Gamma_{1}}v_{1}(j_{1}) e_{j_{1}} \right) \otimes \cdots \otimes \left( \sum \limits_{j_{r} \in \Gamma_{r}}v_{r}(j_{r}) e_{j_{r}} \right) \\ &:= \sum\limits_{(j_{1}, \dots, j_{r}) \in \Gamma} \left( \prod\limits_{i=1}^{r} v_{i}(j_{i}) \right) e_{j_{1}} \otimes \cdots \otimes e_{j_{r}} \\ &= \in \mathbb{C}^{\Gamma_{1}} \otimes \cdots \otimes \mathbb{C}^{\Gamma_{r}} \end{align*} $$