Generalized Orthogonal Group
Introduction
The orthogonal group is the set of orthogonal matrices which forms a group under matrix multiplication. Since orthogonal matrices are matrices that preserve the inner product, the orthogonal group can also be viewed as the group of matrices that preserve the inner product.
$$ \begin{align*} \operatorname{O}(n) &= \left\{ Q \in M_{n \times n}(\mathbb{R}) : Q^{\mathsf{T}}Q = I \right\} \\ &= \left\{ Q \in M_{n \times n}(\mathbb{R}) : \braket{Q\mathbf{x}, Q\mathbf{y}} = \braket{\mathbf{x}, \mathbf{y}} \quad \forall \mathbf{x}, \mathbf{y} \in \mathbb{R}^{n} \right\} \end{align*} $$
The standard inner product on a $n$-dimensional vector space is given as follows.
$$ \braket{\mathbf{x}, \mathbf{y}} = \mathbf{x}^{\mathsf{T}} \mathbf{y} = x_{1} y_{1} + \cdots + x_{n} y_{n} $$
This is a kind of symmetric bilinear form, and the operation $[\cdot, \cdot]$ defined on the vector space $\mathbb{R}^{n+k}$ below is also a symmetric bilinear form.
$$ [\mathbf{x}, \mathbf{y}]_{n,k} = x_{1}y_{1} + \cdots + x_{n}y_{n} - x_{n+1}y_{n+1} - \cdots - x_{n+k}y_{n+k} $$
Then, as a generalization of the orthogonal group, one can consider the set of matrices that preserve $[\cdot, \cdot]_{n,k}$, and this is the generalized orthogonal group.
Definition1
Following the notation above, define the generalized orthogonal group $\operatorname{O}(n, k)$ as follows.
$$ \operatorname{O}(n, k) = \left\{ Q \in M_{(n+k) \times (n+k)}(\mathbb{R}) : [Q \mathbf{x}, Q \mathbf{y}]_{n,k} = [\mathbf{x}, \mathbf{y}]_{n,k} \quad \forall \mathbf{x}, \mathbf{y} \in \mathbb{R}^{n+k} \right\} $$
Explanation
If you have studied physics, the above $[\cdot, \cdot]$ will likely feel familiar. Particularly of interest in physics is $\operatorname{O}(1, 3)$, which is called the Lorentz group. There are two definitions used for the Lorentz group, and recently there seems to be a growing tendency to prefer $\operatorname{O}(1, 3)$. However, $\operatorname{O}(3; 1)$ is still widely used, and in older literature it is often denoted as $\operatorname{O}(3, 1)$. Of course, essentially they are the same, so choosing either definition is a matter of taste.
Let $\Lambda$ be the diagonal matrix which has $+1$ in components up to the $n$-th, and $-1$ in components from the $n+1$-th through the $n+k$-th.
$$ \Lambda = \begin{bmatrix} I_{n\times n} & O \\ O & -I_{k\times k} \end{bmatrix} \tag{1} $$
Then the following holds.
$$ [\mathbf{x}, \mathbf{y}]_{n,k} = \mathbf{x}^{\mathsf{T}} \Lambda \mathbf{y} = \braket{\mathbf{x}, \Lambda \mathbf{y}} \tag{2} $$
Moreover, from the theorem below, one can see that the above definition is equivalent to the following.
$$ \operatorname{O}(n, k) = \left\{ Q \in M_{(n+k) \times (n+k)}(\mathbb{R}) : Q^{\mathsf{T}} \Lambda Q = \Lambda \right\} $$
Subgroups
The generalized orthogonal group is a subgroup of the general linear group $\operatorname{GL}(n+k, \mathbb{R})$. By the subgroup test, it suffices to show that $A, B \in \operatorname{O}(n, k) \implies AB^{-1} \in \operatorname{O}(n, k)$. Since $A$ preserves $[\cdot, \cdot]_{n,k}$,
$$ [AB^{-1}\mathbf{x}, AB^{-1}\mathbf{y}]_{n,k} = [A(B^{-1}\mathbf{x}), A(B^{-1}\mathbf{y})]_{n,k} = [B^{-1}\mathbf{x}, B^{-1}\mathbf{y}]_{n,k} $$
Since $B$ preserves $[\cdot, \cdot]_{n,k}$,
$$ [B^{-1}\mathbf{x}, B^{-1}\mathbf{y}]_{n,k} = [B(B^{-1}\mathbf{x}), B(B^{-1}\mathbf{y})]_{n,k} = [\mathbf{x}, \mathbf{y}]_{n,k} $$
Therefore $AB^{-1} \in \operatorname{O}(n, k)$, hence it is a subgroup of $\operatorname{GL}(n+k, \mathbb{R})$.
$$ \operatorname{O}(n, k) \le \operatorname{GL}(n+k, \mathbb{R}) $$
Matrix Lie group
$\operatorname{O}(n, k)$ is a closed subgroup of $\operatorname{GL}(n+k, \mathbb{R})$, hence it becomes a matrix Lie group. Define the function $f : \operatorname{GL}(n+k, \mathbb{R}) \to M_{(n+k)\times(n+k)}(\mathbb{R})$ as follows.
$$ f(Q) = Q^{\mathsf{T}} \Lambda Q $$
Then $f$ is a continuous function. Recall that the preimage of a closed set under a continuous function is closed (preimage of closed sets under continuous functions). The necessary and sufficient condition for $Q \in \operatorname{O}(n, k)$ is $Q^{\mathsf{T}} \Lambda Q = \Lambda$, and since $f^{-1}(\left\{ \Lambda \right\}) = \operatorname{O}(n, k)$ is the preimage of the closed set $\left\{ \Lambda \right\}$, $\operatorname{O}(n, k)$ is a closed subgroup of $\operatorname{GL}(n+k, \mathbb{R})$ and hence a matrix Lie group.
Properties
Let $Q = \begin{bmatrix} q_{1} & \cdots & q_{n+k} \end{bmatrix}$. Let $e_{i}$ denote the standard basis vectors.
(a) If $Q \in \operatorname{O}(n, k)$, then $[q_{i}, q_{j}]_{n,k} = [e_{i}, e_{j}]_{n,k}$ and its value is as follows.
$$ \begin{align*} [q_{i}, q_{j}]_{n,k} &= 0 && i \ne j \\ [q_{i}, q_{i}]_{n,k} &= 1 && 1 \le i \le n \\ [q_{i}, q_{i}]_{n,k} &=-1 && n+1 \le i \le n+k \\ \end{align*} $$
The converse also holds.
(b) For $Q \in \operatorname{O}(n, k)$, one has $\det (Q) = \pm 1$.
Proof
(a)
This follows from the proof of the theorem below.
■
(b)
From the result of the theorem below, $$ \det (Q^{\mathsf{T}} \Lambda Q) = \det (\Lambda) \implies (\det(Q))^{2} \det(\Lambda) = \det(\Lambda) \implies \det(Q) = \pm 1 $$
■
Theorem
Let $\Lambda$ be equal to $(1)$. A necessary and sufficient condition for $Q \in \operatorname{O}(n, k)$ is that $Q^{\mathsf{T}}\Lambda Q = \Lambda$ holds.
$$ [Q \mathbf{x}, Q \mathbf{y}]_{n,k} = [\mathbf{x}, \mathbf{y}]_{n,k} \iff Q^{\mathsf{T}}\Lambda Q = \Lambda $$
Proof
$(\implies)$
Let $Q = \begin{bmatrix} q_{1} & \cdots & q_{n+k} \end{bmatrix} \in \operatorname{O}(n, k)$. Then by $(2)$ the following holds.
$$ \begin{align*} Q^{\mathsf{T}} q Q &= \begin{bmatrix} –q_{1}^{\mathsf{T}}– \\ \vdots \\ –q_{n+k}^{\mathsf{T}}– \end{bmatrix} \Lambda \begin{bmatrix} \underset{\vert}{\overset{\vert}{q_{1}}} & \cdots & \underset{\vert}{\overset{\vert}{q_{n+k}}} \end{bmatrix} \\ &= \begin{bmatrix} \braket{q_{1}, \Lambda q_{1}} & \braket{q_{1}, \Lambda q_{2}} & \cdots & \braket{q_{1}, \Lambda q_{n+k}} \\ \braket{q_{2}, \Lambda q_{1}} & \braket{q_{2}, \Lambda q_{2}} & \cdots & \braket{q_{2}, \Lambda q_{n+k}} \\ \vdots & \vdots & \ddots & \vdots \\ \braket{q_{n+k}, \Lambda q_{1}} & \braket{q_{n+k}, \Lambda q_{2}} & \cdots & \braket{q_{n+k}, \Lambda q_{n+k}} \end{bmatrix} \\ &= \begin{bmatrix} [q_{1}, q_{1}]_{{n,k}} & [q_{1}, q_{2}]_{{n,k}} & \cdots & [q_{1}, q_{n+k}]_{{n,k}} \\ [q_{2}, q_{1}]_{{n,k}} & [q_{2}, q_{2}]_{{n,k}} & \cdots & [q_{2}, q_{n+k}]_{{n,k}} \\ \vdots & \vdots & \ddots & \vdots \\ [q_{n+k}, q_{1}]_{{n,k}} & [q_{n+k}, q_{2}]_{{n,k}} & \cdots & [q_{n+k}, q_{n+k}]_{{n,k}} \end{bmatrix} \\ &= \begin{bmatrix} 1 & 0 & \cdots & 0 \\ 0 & 1 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & & \cdots & -1 \end{bmatrix} \\ &= \Lambda \end{align*} $$
■
$(\impliedby)$
Since $[\cdot, \cdot]$ is bilinear, it suffices to show that $[Qe_{i}, Qe_{j}] = [e_{i}, e_{j}]$ holds for the standard basis vectors $e_{i}$. Let $Q^{\mathsf{T}} g Q = g$. From the calculation above, this is equal to the following.
$$ \begin{bmatrix} [q_{1}, q_{1}]_{{n,k}} & [q_{1}, q_{2}]_{{n,k}} & \cdots & [q_{1}, q_{n+k}]_{{n,k}} \\ [q_{2}, q_{1}]_{{n,k}} & [q_{2}, q_{2}]_{{n,k}} & \cdots & [q_{2}, q_{n+k}]_{{n,k}} \\ \vdots & \vdots & \ddots & \vdots \\ [q_{n+k}, q_{1}]_{{n,k}} & [q_{n+k}, q_{2}]_{{n,k}} & \cdots & [q_{n+k}, q_{n+k}]_{{n,k}} \end{bmatrix} = g = \begin{bmatrix} 1 & 0 & \cdots & 0 \\ 0 & 1 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & & \cdots & -1 \end{bmatrix} $$
That is, the following holds.
$$ \begin{align*} [q_{i}, q_{j}]_{n,k} &= 0 && i \ne j \\ [q_{i}, q_{i}]_{n,k} &= 1 && 1 \le i \le n \\ [q_{i}, q_{i}]_{n,k} &=-1 && n+1 \le i \le n+k \\ \end{align*} $$
This is equal to the value of $[e_{i}, e_{j}]_{n,k}$.
$$ \begin{align*} [q_{i}, q_{j}]_{n,k} &= 0 = [e_{i}, e_{j}]_{n,k} && i \ne j \\ [q_{i}, q_{i}]_{n,k} &= 1 = [e_{i}, e_{i}]_{n,k} && 1 \le i \le n \\ [q_{i}, q_{i}]_{n,k} &=-1 = [e_{i}, e_{i}]_{n,k} && n+1 \le i \le n+k \\ \end{align*} $$
And since $Qe_{i} = q_{i}$ holds, we obtain the following.
$$ [Qe_{i}, Qe_{j}]_{n,k} = [q_{i}, q_{j}]_{n,k} = [e_{i}, e_{j}]_{n,k} $$
Therefore $Q \in \operatorname{O}(n, k)$.
■
Brian C. Hall. Lie Groups, Lie Algebras, and Representations (2nd), p8-9 ↩︎
