11.3 Recursive Feature Elimination

“This technique begins by building a model on the entire set of predictors and computing an importance score for each predictor. The least important predictor(s) are then removed, the model is re-built, and importance scores are computed again.”

Not all models can be paired with the RFE method, and some models benefit more from RFE than others. Because RFE requires that the initial model uses the full predictor set, then some models cannot be used when the number of predictors exceeds the number of samples. As noted in previous chapters, these models include multiple linear regression, logistic regression, and linear discriminant analysis. In addition, some models benefit more from the use of RFE than others.

Random Forests can handle multicollinearity, but the selection of important variables that are the same can vary based upon the sample splits, so one split might have one variable and another another. This can cause issues when you want to slim down variables because an RF takes all variables as input.