8.16 Bagging (continued)

• Cue the bootstrap, i.e., take repeated samples from the single training set

• Generate \(B\) different bootstrapped training data sets

• Then train our method on the \(b\)th bootstrapped training set to get \(\hat{f}^{*b}\), the prediction at a point x

• Average all the predictions to obtain \[\hat{f}_{bag}(x) = \frac{1}{B}\sum_{b=1}^B\hat{f}^{*b}(x)\]

• In the case of classification trees:

  • for each test observation:

    • record the class predicted by each of the \(B\) trees

    • take a majority vote: the overall prediction is the most commonly occurring class among the \(B\) predictions

NOTE: the number of trees \(B\) is not a critical parameter with bagging - a large \(B\) will not lead to overfitting