Principal Component Analysis in Statistics
Overview
Suppose we perform a multiple regression analysis $Y \gets X_{1} , \cdots, X_{p}$. principal component analysis, abbreviated as PCA in English, is, put simply, a method of analysis that ‘reconstructs’ quantitative variables so that they become properly independent. From the perspective of multivariate data analysis, it has significance as ‘dimensionality reduction’ for explaining a phenomenon with fewer variables.
To properly understand the theoretical derivation of principal component analysis, one needs knowledge of linear algebra, and if possible even numerical linear algebra. If it just does not sink in, let us try reading and understanding Steps 3 and 4 as well. If you are reasonably confident in mathematical statistics, it is also good to read the principal component analysis in mathematical statistics post.
Derivation 1
Step 1. Standardization — data with $p$ independent variables and $n$ samples
$$ \begin{bmatrix} y_{1} \\ y_{2} \\ \vdots \\ y_{n} \end{bmatrix} = \begin{bmatrix} 1 & x_{11} & \cdots & x_{1p} \\ 1 & x_{21} & \cdots & x_{2p} \\ \vdots & \vdots & \ddots & \vdots \\ 1 & x_{n1} & \cdots & x_{np} \end{bmatrix} \begin{bmatrix} \beta_{0} \\ \beta_{1} \\ \vdots \\ \beta_{p} \end{bmatrix} $$
can be expressed as $Y = X \beta + \varepsilon$ using the design matrix. Standardizing this $X$ into a matrix $Z$ means the matrix whose $(i,j)$-component is
$$ \left( Z \right)_{ij}: = {{ x_{ij} } - \overline{ x_{j} } \over { s_{ X_{j} } }} $$
with respect to the sample mean $\overline{ x_{j} }$ and the sample standard deviation $s_{ X_{j} }$ of the $j$th independent variable $X_{j}$. Then, for a new regression coefficient
$$ \Theta := \begin{bmatrix} \theta_{1} \\ \theta_{2} \\ \vdots \\ \theta_{p} \end{bmatrix} $$
we can obtain the design matrix equation $Y = Z \Theta + \varepsilon$ of a regression analysis without a constant term. This $Z = \begin{bmatrix} Z_{1} & \cdots & Z_{p} \end{bmatrix}$ is a $( n \times p )$ matrix composed of $Z_{1} , \cdots , Z_{p}$, which are the standardizations of the vectors $X_{1} , \cdots , X_{p}$.
Step 2.
Regarding the spectral decomposition, $Z^{T} Z$ is a $( p \times p )$ symmetric matrix, and thinking about its definition, $\displaystyle {{1} \over {n-1}} Z^{T} Z$ becomes the covariance matrix for $Z_{1} , \cdots , Z_{p}$. In particular, since $Z$ is a standardized matrix, it simultaneously becomes the correlation coefficient matrix as well. According to spectral theory, there exist an orthogonal matrix
$$ Q = \begin{bmatrix} q_{11} & q_{12} & \cdots & q_{1p} \\ q_{21} & q_{22} & \cdots & q_{2p} \\ \vdots & \vdots & \ddots & \vdots \\ q_{p1} & q_{p2} & \cdots & q_{pp} \end{bmatrix} $$
and a diagonal matrix consisting of the eigenvalues of $Z^{T} Z$
$$ \Lambda = \text{diag} ( \lambda_{1} , \lambda_{2} , \cdots , \lambda_{p} ) = \begin{bmatrix} \lambda_{1} & 0 & \cdots & 0 \\ 0 & \lambda_{2} & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & \lambda_{p} \end{bmatrix} $$
satisfying
$$ \begin{cases} Z^{T} Z = Q \Lambda Q^{T} \\ Q^{T} Q = Q Q^{T} = I \end{cases} $$
Now, for convenience, let us set them so that $\lambda_{1} \ge \lambda_{2} \ge \cdots \ge \lambda_{p}$, and think of $Z$ as a matrix rearranged correspondingly as well.
Step 3. Constructing the principal components
Since $I = QQ^{T}$, $$ Y = Z \Theta + \varepsilon = Z Q Q^{T} \Theta + \varepsilon $$ Here, setting $C := ZQ $ and $\alpha := Q^{T} \Theta$, $$ Y = C \alpha + \varepsilon $$ For this $C = \begin{bmatrix} C_{1} & \cdots & C_{p} \end{bmatrix}$, $C_{1} , \cdots , C_{p}$ are called the principal components of $Z_{1}, \cdots , Z_{p}$. The shape of the $j$th principal component is
$$ C_{j} = q_{1j} Z_{1} + \cdots + q_{pj} Z_{p} = \sum_{i=1}^{p} q_{ij} Z_{j} $$
a reconstruction of the existing independent variables as a linear combination.
Step 4.
Independence of the principal components — the independence among these principal components can also be verified through the following calculation. $$ \begin{align*} & Z^{T} Z = Q \Lambda Q^{T} \\ \implies& Q^{T} Z^{T} Z Q = \Lambda \\ \implies& \left( Z Q \right) ^{T} \left( Z Q \right) = \Lambda \\ \implies& C^{T} C = \Lambda \end{align*} $$ In other words, since $$ C_{j}^{T} C_{j} = \begin{cases} \lambda_{j} & , i=j \\ 0 & , i \ne j \end{cases} $$ the principal components are necessarily independent of one another, and saying that an eigenvalue $\lambda_{j}$ is small enough to be close to $0$ means that $\displaystyle C_{j} = \sum_{i=1}^{p} q_{ij} Z_{j}$ is close to the zero vector, so we can regard $Z_{1} , \cdots , Z_{p}$ as having multicollinearity.
■
Limitations
Principal component regression $Y \gets C_{1} , \cdots , C_{p}$ avoids the problem of multicollinearity by removing variables with problematic eigenvalues. Also, compared with the original regression analysis, it uses far fewer variables, so we can say the dimensionality has been reduced.
At first glance, principal component analysis seems all-powerful, but that is not necessarily so. First, standardizing to create $Z$ means there are many aspects that are hard to handle regarding qualitative variables or transformations, and in the course of this ‘reconstruction’ the analysis itself becomes hard to understand.
Considering that statistics is also needed by people who do not understand statistics, this point is quite fatal. For example, if principal component analysis is used in an analysis of the Korean economy, it ends up being expressed not with easily understandable figures such as the unemployment rate $X_{2}$ or the average starting salary $X_{7}$, but with strange terms such as a ‘composite employment index’ $C_{4}$. There can be a catastrophe in which even the analyst, though they may have produced a usable regression equation, fails to grasp its true meaning. (In computer science, since prediction and classification matter more than understanding the data, they do not worry much about this drawback.)
Also, if we use $Y \gets C_{1} , \cdots , C_{p}$ as is without dropping any principal component, it is no different from the original $Y \gets X_{1} , \cdots , X_{p}$; and dropping some principal components here itself means giving up data that was originally there. If it is still needed, one should use it, but if it is not needed there is no particular reason to use it. Even when using it, one should use it while clearly knowing what drawbacks and limitations it has.
Condition Number 1
Meanwhile, through the eigenvalues obtained during the derivation process, we can compute the condition number, a numerical index that diagnoses multicollinearity. $$ \kappa := \sqrt{ {{ \lambda_{1} } \over { \lambda_{p} }} } $$ This has an important meaning in numerical matrix algebra and can explain the instability of the analysis; empirically, it is said that when $\kappa > 15$ one can surmise that the original data has multicollinearity, but it is not actually used much in a statistical context.