8.15 Bagging

• Also known as bootstrap aggregation is a general-purpose procedure for reducing the variance of a statistical learning method

• It’s useful and frequently used in the context of decision trees

• Recall that given a set of \(n\) independent observations \(Z_1,..., Z_n\), each with variance \(\sigma^2\), the variance of the mean \(\bar{Z}\) of the observations is given by \(\sigma^2/n\)

• So, averaging a set of observations reduces variance

• But, this is not practical because we generally do not have access to multiple training sets!

• What to do?!