Backward Stepwise Subset Selection (BsSS)

  • backward stepwise selection provides an efficient alternative to best subset selection.

  • It’s begins with the full least squares model containing all \(p\) predictors, and then iteratively removes the least useful predictor, one-at a-time

  1. Make sure that \(n > p\)
  2. Let \(\mathcal{M}_p\) denote the full model with all p predictors
  3. For \(k = p, p - 1, ..., 1\):
  • Consider all \(k\) models that result in dropping a single predictor from \(\mathcal{M}_k\) (thus containing \(k - 1\) predictors)
  • Choose the best among these \(k\) models, and christen it \(\mathcal{M}_{k-1}\)
  1. Select the model among \(\mathcal{M}_0, ..., \mathcal{M}_k\) that minimizes validation error (or some estimate of it)