Forward Stepwise Subset Selection (FsSS)

1. Let $\mathcal{M}_0$ denote the null model (no predictors)
2. For $k = 1, ..., p$:

• Fit all $p - (k - 1)$ predictors not in model $\mathcal{M}_{k - 1}$
• Select the predictor that raises $R^2$ the most and add it to model $\mathcal{M}_{k - 1}$ to create model $\mathcal{M}_k$

3. Select the model among $\mathcal{M}_0, ..., \mathcal{M}_k$ that minimizes validation error (or some estimate of it)

• When $p = 20$, best subset selection requires fitting 1,048,576 models, whereas forward stepwise selection requires fitting only 211 models.