📂Statistical Analysis

Variable Selection Criteria in Statistical Analysis

Overview

The problem of selecting variables inevitably involves the analyst’s subjectivity, but numerical indicators were needed to help draw conclusions as objectively as possible. If such values can be computed, they provide a clear answer to when to stop the variable selection procedure. However, there are several kinds of such criteria, and applying a different criterion can lead to different results.

Indicators¹

R-Squared $R^2$

The R-squared is calculated as $\displaystyle R^2 = 1 - {{ \text{ SSE } } \over { \text{ SST} }}$, and the closer it is to $1$, the better the model can be interpreted as explaining the data.

However, as a variable selection criterion it is not very appropriate, since it becomes meaningless when the models being compared have different numbers of independent variables.

Adjusted R-Squared $R_{a}^2$

In regression analysis, an increase in the number of variables means that much more data becomes available, and in that case the R-squared $R^{2}$ always increases. In contrast, the adjusted R-squared differs in that it is calculated as $\displaystyle R^{2}_{a} = 1 - {{ \text{ SSE } / (n - p - 1) } \over { \text{ SST} / (n - 1) }}$, so that the number of variables is reflected.

The adjusted R-squared $R^{2}_{a}$ overcomes the drawback of the R-squared, which cannot be used when the numbers of variables differ, by applying a penalty for the number of variables. Moreover, unlike other variable selection criteria, which are relative indicators and thus only meaningful when comparing one model against another, the adjusted R-squared by itself also tells how well the model explains the data, so it is an indicator one cannot help but consult no matter which criterion is used. Although it is not popular as a variable selection criterion, it can be usefully employed if what one wants is not the optimal model but the model with the highest explanatory power.

Akaike Information Criteria $\text{AIC}_{p}$

For $p$ independent variables, the Akaike information criteria is calculated as $\displaystyle \text{AIC}_{p} := n \ln \left( {{ \text{SSE}_{p} } \over {n}} \right) + 2(p+1) $. AIC is the most favored measure in actual analysis; when comparing two models, the one with the smaller AIC is judged to be better.

The second term in the formula can be seen as imposing a penalty as $p$ grows, that is, as the number of variables increases. The drawback of AIC is that the comparison becomes inaccurate when the sample sizes $n$ differ. One might wonder how $n$ could possibly differ when analyzing the same data with only the variables changed, but this can become a fatal problem when a particular variable has many missing values.

Bayes Information Criteria $\text{ BIC }_{p}$

For $p$ independent variables, the Bayes information criteria is calculated as $\displaystyle \text{BIC}_{p} := n \ln \left( {{ \text{SSE}_{p} } \over {n}} \right) + ( p +1 ) \ln n $. It is similar to AIC but complements AIC by modifying the last term, and like AIC, the smaller one is judged to be better.

Mallows $C_{p}$

For $p$ independent variables, Mallows $C_{p}$ is calculated as $\displaystyle C_{p} := {{ \text{SSE}_{p} } \over { \hat{\sigma}^2 }} + ( 2p - n )$.

$C_{p}$ selects variables in the direction of less bias, and the closer it is to the number of variables, the better it is judged to be. In other words, the closer it gets to $C_{p} \approx p$, the more it can be regarded as unbiased. It would be usable if there is an analysis where bias must be taken into account, and it is nice since things also become mathematically clean, but these days techniques that fit precisely by drastically reducing variance, even at the cost of some bias, are in the spotlight, so it is not popular.