Orthogonal Transformation
Definition
A linear transformation $T : V \to V$ on a finite-dimensional real inner product space $V$ is called an orthogonal transformation if it satisfies the following for all $\mathbf{x}, \mathbf{y} \in V$.
$$ \braket{T\mathbf{x}, T\mathbf{y}} = \braket{\mathbf{x}, \mathbf{y}} $$
Explanation
In a word, an orthogonal transformation is a linear transformation that preserves the inner product. Note that no matrix appears in the definition. An orthogonal transformation is characterized solely by the property that it moves vectors while leaving the inner product structure of $V$ intact, regardless of which coordinate system it is computed in or how.
The reason the name contains 'orthogonal' is that this condition preserves orthogonality. Two vectors being orthogonal means $\braket{\mathbf{u}, \mathbf{v}} = 0$, so if the inner product is preserved, the following holds.
$$ \braket{\mathbf{u}, \mathbf{v}} = 0 \implies \braket{T\mathbf{u}, T\mathbf{v}} = \braket{\mathbf{u}, \mathbf{v}} = 0 $$
In other words, what is perpendicular stays perpendicular. Going one step further, considering an orthonormal basis $\left\{ \mathbf{e}_{1}, \cdots, \mathbf{e}_{n} \right\}$, we obtain the following.
$$ \braket{T\mathbf{e}_{i}, T\mathbf{e}_{j}} = \braket{\mathbf{e}_{i}, \mathbf{e}_{j}} = \delta_{ij} $$
That is, an orthogonal transformation maps an orthonormal basis to an orthonormal basis. It moves coordinate axes made of mutually perpendicular unit vectors, as a whole, to coordinate axes that are still made of mutually perpendicular unit vectors. In the theorem below, this fact leads to the equivalence with orthogonal matrices.
On the other hand, be careful: 'preserving only orthogonality' and 'being an orthogonal transformation' are not the same thing. For example, consider $T = 2I$. When $\braket{\mathbf{u}, \mathbf{v}} = 0$, we have $\braket{2\mathbf{u}, 2\mathbf{v}} = 4\braket{\mathbf{u}, \mathbf{v}} = 0$, so every right angle is sent to a right angle. However, this is not an orthogonal transformation.
$$ \braket{2\mathbf{x}, 2\mathbf{y}} = 4 \braket{\mathbf{x}, \mathbf{y}} \ne \braket{\mathbf{x}, \mathbf{y}} $$
This is because every length is doubled. To be an orthogonal transformation, not only angles but also lengths must be preserved, and the condition of preserving the inner product demands exactly both at once. Indeed, substituting $\mathbf{y} = \mathbf{x}$ immediately shows that the norm is preserved.
$$ \left\| T\mathbf{x} \right\|^{2} = \braket{T\mathbf{x}, T\mathbf{x}} = \braket{\mathbf{x}, \mathbf{x}} = \left\| \mathbf{x} \right\|^{2} $$
Conversely, preserving only the norm automatically preserves the inner product, which is dealt with in the theorem below.
The phrase 'any' orthonormal basis in (d) of the theorem is also worth noticing. It is not that one must choose a particular orthonormal basis well for the matrix representation to be an orthogonal matrix; rather, whichever orthonormal basis is chosen, the matrix representation is an orthogonal matrix. This means that orthogonality is not a property attached to a coordinate system but a property of the transformation itself, and this is the benefit of defining it via a transformation rather than a matrix. Of course, in actual computation, it suffices to take the standard basis of $\mathbb{R}^{n}$ and check $A^{\mathsf{T}}A = I$.
In addition, there is no need to assume even linearity in the definition. This is because an isometry that fixes the origin is automatically linear and thus an orthogonal transformation.
Theorem
Let $V$ be an $n$-dimensional real inner product space and $T : V \to V$ a linear transformation. Then the following statements are all equivalent.
(a) $T$ is an orthogonal transformation. That is, $\braket{T\mathbf{x}, T\mathbf{y}} = \braket{\mathbf{x}, \mathbf{y}}$ for all $\mathbf{x}, \mathbf{y} \in V$.
(b) $T$ preserves the norm. That is, $\left\| T\mathbf{x} \right\| = \left\| \mathbf{x} \right\|$ for all $\mathbf{x} \in V$.
(c) $T$ maps orthonormal bases to orthonormal bases. That is, if $\beta$ is an orthonormal basis of $V$, then $T(\beta)$ is also an orthonormal basis of $V$.
(d) For any orthonormal basis $\beta$, the matrix representation $A = [T]_{\beta}$ of $T$ is an orthogonal matrix. That is, $A^{\mathsf{T}}A = I$.
Proof
We show (a) $\implies$ (b) $\implies$ (a) and (a) $\implies$ (c) $\implies$ (d) $\implies$ (a).
First, let us point out a fact that will be used repeatedly in the latter two links. If $\beta = \left\{ \mathbf{e}_{1}, \cdots, \mathbf{e}_{n} \right\}$ is an orthonormal basis and $\mathbf{u} = \sum_{i} a_{i} \mathbf{e}_{i}$, $\mathbf{v} = \sum_{j} b_{j} \mathbf{e}_{j}$, then for the coordinate vectors $[\mathbf{u}]_{\beta} = (a_{1}, \cdots, a_{n})^{\mathsf{T}}$, $[\mathbf{v}]_{\beta} = (b_{1}, \cdots, b_{n})^{\mathsf{T}}$, the following holds.
$$ \braket{\mathbf{u}, \mathbf{v}} = \sum_{i=1}^{n} \sum_{j=1}^{n} a_{i} b_{j} \braket{\mathbf{e}_{i}, \mathbf{e}_{j}} = \sum_{i=1}^{n} a_{i}b_{i} = [\mathbf{u}]_{\beta}^{\mathsf{T}} [\mathbf{v}]_{\beta} \tag{1} $$
That is, taking coordinate vectors with respect to an orthonormal basis carries the inner product of $V$ over exactly to the inner product of $\mathbb{R}^{n}$.
(a) $\implies$ (b)
Substituting $\mathbf{y} = \mathbf{x}$ gives the following.
$$ \left\| T\mathbf{x} \right\|^{2} = \braket{T\mathbf{x}, T\mathbf{x}} = \braket{\mathbf{x}, \mathbf{x}} = \left\| \mathbf{x} \right\|^{2} $$
Since norms are nonnegative, $\left\| T\mathbf{x} \right\| = \left\| \mathbf{x} \right\|$.
■
(b) $\implies$ (a)
Since $T$ is linear, $T\mathbf{x} \pm T\mathbf{y} = T(\mathbf{x} \pm \mathbf{y})$. Applying the polarization identity for real inner product spaces to this, we obtain the following.
$$ \begin{align*} \braket{T\mathbf{x}, T\mathbf{y}} &= \frac{1}{4} \left( \left\| T\mathbf{x} + T\mathbf{y} \right\|^{2} - \left\| T\mathbf{x} - T\mathbf{y} \right\|^{2} \right) \\ &= \frac{1}{4} \left( \left\| T(\mathbf{x} + \mathbf{y}) \right\|^{2} - \left\| T(\mathbf{x} - \mathbf{y}) \right\|^{2} \right) \\ &= \frac{1}{4} \left( \left\| \mathbf{x} + \mathbf{y} \right\|^{2} - \left\| \mathbf{x} - \mathbf{y} \right\|^{2} \right) \\ &= \braket{\mathbf{x}, \mathbf{y}} \end{align*} $$
■
(a) $\implies$ (c)
Let $\beta = \left\{ \mathbf{e}_{1}, \cdots, \mathbf{e}_{n} \right\}$ be an orthonormal basis of $V$. Since the inner product is preserved, the following holds.
$$ \braket{T\mathbf{e}_{i}, T\mathbf{e}_{j}} = \braket{\mathbf{e}_{i}, \mathbf{e}_{j}} = \delta_{ij} $$
Therefore, $T(\beta) = \left\{ T\mathbf{e}_{1}, \cdots, T\mathbf{e}_{n} \right\}$ is an orthonormal set. An orthonormal set is an orthogonal set not containing the zero vector, so it is linearly independent, and $n$ linearly independent vectors form a basis of $V$, an $n$-dimensional space. Hence $T(\beta)$ is an orthonormal basis of $V$.
■
(c) $\implies$ (d)
Let $\beta = \left\{ \mathbf{e}_{1}, \cdots, \mathbf{e}_{n} \right\}$ be an arbitrary orthonormal basis of $V$ and let $A = [T]_{\beta}$. By the definition of the matrix representation, the $j$-th column of $A$ is the coordinate vector $[T\mathbf{e}_{j}]_{\beta}$. Writing this as $\mathbf{a}_{j}$, by $(1)$ and assumption (c) we obtain the following.
$$ \mathbf{a}_{i}^{\mathsf{T}} \mathbf{a}_{j} = [T\mathbf{e}_{i}]_{\beta}^{\mathsf{T}} [T\mathbf{e}_{j}]_{\beta} = \braket{T\mathbf{e}_{i}, T\mathbf{e}_{j}} = \delta_{ij} $$
Since the $(i, j)$ entry of $A^{\mathsf{T}}A$ is exactly $\mathbf{a}_{i}^{\mathsf{T}} \mathbf{a}_{j}$, we have $A^{\mathsf{T}}A = I$, and $A$ is an orthogonal matrix.
■
(d) $\implies$ (a)
Take an orthonormal basis $\beta$ of $V$ and let $A = [T]_{\beta}$. By assumption, $A^{\mathsf{T}}A = I$, and by the definition of the matrix representation, $[T\mathbf{x}]_{\beta} = A[\mathbf{x}]_{\beta}$ for all $\mathbf{x} \in V$. Therefore, by $(1)$, the following holds.
$$ \begin{align*} \braket{T\mathbf{x}, T\mathbf{y}} &= [T\mathbf{x}]_{\beta}^{\mathsf{T}} [T\mathbf{y}]_{\beta} \\ &= \left( A[\mathbf{x}]_{\beta} \right)^{\mathsf{T}} \left( A[\mathbf{y}]_{\beta} \right) \\ &= [\mathbf{x}]_{\beta}^{\mathsf{T}} A^{\mathsf{T}} A [\mathbf{y}]_{\beta} \\ &= [\mathbf{x}]_{\beta}^{\mathsf{T}} [\mathbf{y}]_{\beta} \\ &= \braket{\mathbf{x}, \mathbf{y}} \end{align*} $$
■