📂Statistical Analysis

Multiple Regression Analysis

Overview

Regression analysis is a method for finding relationships between variables, and it is especially useful for identifying linear relationships. Multiple linear regression refers to a regression analysis that examines the effect of multiple independent variables (explanatory variables) on a single dependent variable (response variable).

Model¹

$$Y = \beta_{0} + \beta_{1} X_{1} + \cdots + \beta_{p} X_{p} + \varepsilon $$

We are interested in whether the variables have a linear relationship as above. Each variable is assumed to be independent of the others, and likewise the regression coefficient represents the rate of change per unit of the corresponding variable when the other variables are held fixed. Expressed with the design matrix, it becomes $$ \begin{bmatrix} y_{1} \\ y_{2} \\ \vdots \\ y_{n} \end{bmatrix} = \begin{bmatrix} 1 & x_{11} & \cdots & x_{p1} \\ 1 & x_{12} & \cdots & x_{p2} \\ \vdots & \vdots & \ddots & \vdots \\ 1 & x_{1n} & \cdots & x_{pn} \end{bmatrix} \begin{bmatrix} \beta_{0} \\ \beta_{1} \\ \vdots \\ \beta_{p} \end{bmatrix} + \begin{bmatrix} \varepsilon_{1} \\ \varepsilon_{2} \\ \vdots \\ \varepsilon_{n} \end{bmatrix} $$ and rearranging gives $Y = X \beta + \varepsilon$.

The computation itself uses the method of least squares just as in simple regression analysis, and fortunately the method of least squares does not care much about the dimension $p$. However, unlike simple regression analysis, since it is generalized to $p$ dimensions, it is also hard to check with a graph when $p \ge 3$.

Since one cannot tell whether the analysis was done properly just by looking at it, the analyst must justify the results through various diagnostics. Even if the results pass such diagnostics, problems such as interaction and multicollinearity remain, and which variables to select is also an important issue.

See Also

• Results of Multiple Regression Analysis in R
• F-test of Regression Coefficients
• Derivation of the Estimator of the Multiple Regression Coefficient Vector