Regression Coefficient's t-test 📂Statistical Test

Regression Coefficient's t-test

Hypothesis Testing

$\begin{bmatrix} y_{1} \\ y_{2} \\ \vdots \\ y_{n} \end{bmatrix} = \begin{bmatrix} 1 & x_{11} & \cdots & x_{p1} \\ 1 & x_{12} & \cdots & x_{p2} \\ \vdots & \vdots & \ddots & \vdots \\ 1 & x_{1n} & \cdots & x_{pn} \end{bmatrix} \begin{bmatrix} \beta_{0} \\ \beta_{1} \\ \vdots \\ \beta_{p} \end{bmatrix} + \begin{bmatrix} \varepsilon_{1} \\ \varepsilon_{2} \\ \vdots \\ \varepsilon_{n} \end{bmatrix}$ When independent variables of $p$ and $n$ data are given, the linear multiple regression model can be represented using the design matrix as shown above, and let’s denote it simply as $Y = X \beta + \varepsilon$ . Assuming that the residuals satisfy linearity, homoscedasticity, independence, and normality in the model diagnostics, the hypothesis testing for each regression coefficient in multiple regression analysis is as follows:

$H_{0}$ : $\beta_{j} = 0$ , that is, the $j$ th independent variable has no correlation with the dependent variable.
$H_{1}$ : $\beta_{j} \ne 0$ , that is, the regression coefficient for the $j$ th independent variable is significant.

Derivation ¹

Normality of Regression Coefficients: $\hat{\beta} \sim N_{1+p} \left( \beta , \sigma^{2} \left( X^{T} X \right)^{-1} \right)$
Unbiased Estimator of Residual Sum of Squares and Standard Error of Regression Coefficients: $E \widehat{\sigma^{2}} = E \left[ {{ 1 } \over { n-p-1 }} \sum_{i=1}^{n} \left( y_{i} - \hat{y}_{i} \right)^{2} \right] = \sigma^{2}$ $\text{s.e.} \left( \hat{\beta}_{k} \right) = \hat{\sigma} \sqrt{ \left[ \left( X^{T} X \right)^{-1} \right]_{kk} }$

Let’s define $t_{j}$ for the estimate of regression coefficient $\hat{ \beta_{j} }$ and standard error $\text{se} \left( \hat{ \beta_{j} } \right)$ as follows: $t_{j} := {{\hat{ \beta_{j} }} \over {\text{se} \left( \hat{ \beta_{j} } \right)}}$

Sum of Random Variables following Chi-Squared Distribution: Let’s assume that the random variables $X_{1} , \cdots , X_{n}$ are mutually independent. If $X_i \sim \chi^2 ( r_{i} )$ , then $\sum_{i=1}^{n} X_{i} \sim \chi ^2 \left( \sum_{i=1}^{n} r_{i} \right)$

The residual sum of squares $\sum_{i=1}^{n} \left( y_{i} - \hat{y}_{i} \right)^{2} / \sigma^{2}$ uses the sample mean for $n$ data points, $p$ independent variables, and $1$ constant terms—thus, since only $(n-p-1)$ independent random variables are used, it follows a chi-squared distribution with degrees of freedom $(n-p-1)$ under the null hypothesis—assuming the null hypothesis is true, then $\beta_{j} = 0$ results in $\hat{\beta}_{j} \sim N \left( 0 , \sigma^{2} \left( X^{T} X \right)^{-1}_{jj} \right)$ , thus obtaining: $\begin{align*} t_{j} =& {{\hat{ \beta_{j} }} \over {\text{se} \left( \hat{ \beta_{j} } \right)}} \\ =& {{\hat{ \beta_{j}} - 0 } \over { \hat{\sigma} \sqrt{ \left[ \left( X^{T} X \right)^{-1} \right]_{kk} } }} \\ =& {{\hat{ \beta_{j}} - 0 } \over { \sqrt{ {{ \sum_{i=1}^{n} \left( y_{i} - \hat{y}_{i} \right)^{2} } \over { n-p-1 }} \left[ \left( X^{T} X \right)^{-1} \right]_{kk} } }} \\ =& {{ {{ \hat{ \beta_{j}} - 0 } \over { \sqrt{ \left[ \left( X^{T} X \right)^{-1} \right]_{kk} } }} } \over { \sqrt{ {{ \sum_{i=1}^{n} \left( y_{i} - \hat{y}_{i} \right)^{2} } \over { n-p-1 }} } }} \\ =& {{ {{ \hat{ \beta_{j} } - 0 } \over { \sigma \sqrt{ \left[ \left( X^{T} X \right)^{-1} \right]_{kk} } }} } \over { \sqrt{ {{ \sum_{i=1}^{n} \left( y_{i} - \hat{y}_{i} \right)^{2} } \over { \sigma^{2} }} / (n-p-1) } }} \\ \sim & {{ N (0,1) } \over { \sqrt{\chi^{2} (n-p-1) / n-p-1} }} \end{align*}$

Derivation of t-Distribution: If two random variables $W,V$ are independent and $W \sim N(0,1)$ , $V \sim \chi^{2} (r)$ , then $T = { {W} \over {\sqrt{V/r} } } \sim t(r)$

In conclusion, $t_{j}$ follows a t-distribution with degrees of freedom $(n-p-1)$ . Representing it in a formula again gives: $t_{j} = {{\hat{ \beta_{j} }} \over {\text{se} \left( \hat{ \beta_{j} } \right)}} \sim t(n-p-1)$ which is used for hypothesis testing. If $\left| t_j \right| \ge t_{(n-p-1 , {{\alpha} \over {2}})}$ then the null hypothesis is rejected. That $\left| t_j \right|$ is of such magnitude means $\hat{ \beta_{j} }$ is too large to believe that the null hypothesis is true.

■

Explanation

$j$ th variable and significance level $\alpha$ can also calculate the confidence interval $\hat{ \beta_{j} } \pm t_{(n-p-1 , {{\alpha} \over {2}})} \text{se} ( \hat{ \beta_{j}} )$ .

While the derivation process is written calmly, it could be quite difficult for undergraduates learning regression analysis to understand it. Following $t_{j}$ to a t-distribution is not very challenging, but it requires understanding the distribution of regression coefficients as a corollary, and having a firm grasp on the concepts of model diagnostics.