6.4 Considerations in High Dimensions
Figure 6.22
- Data sets containing more features \(p\) than observations \(n\) are often referred to as high-dimensional.
- Modern data can have a huge number of predictors (eg: 500k SNPs, every word ever entered in a search)
- When \(n <= p\), linear regression memorizes the training data, but can suck on test data.
Figure 6.23 - simulated data set with n = 20 training observations, all unrelated to outcome.
Lasso (etc) vs Dimensionality
- Reducing flexibility (all the stuff in this chapter) can help.
- It’s important to choose good tuning parameters for whatever method you use.
- Features that aren’t associated with \(Y\) increase test error (“curse
of dimensionality”).
- Fit to noise in training, noise in test is different.
- When \(p > n\), never use train MSE, p-values, \(R^2\), etc, as evidence of goodness of fit because they’re likely to be wildly different from test values.
Figure 6.24