8.23 Boosting algorithm

where:

\(\hat{f}(x)\) is the decision tree (model)

\(r\) = residuals

\(d\) = number of splits in each tree (controls the complexity of the boosted ensemble)

\(\lambda\) = shrinkage parameter (a small positive number that controls the rate at which boosting learns; typically 0.01 or 0.001 but right choice can depend on the problem)

• Each of the trees can be small, with just a few terminal nodes (determined by \(d\))

• By fitting small trees to the residuals, we slowly improve our model (\(\hat{f}\)) in areas where it doesn’t perform well

• The shrinkage parameter \(\lambda\) slows the process down further, allowing more and different shaped trees to ‘attack’ the residuals

• Unlike bagging and random forests, boosting can OVERFIT if \(B\) is too large. \(B\) is selected via cross-validation