10.4 Estimating performance

To recap, the resampling methods above estimate overall model performance using the predictions from the assessment sets. The {tune} package (included in tidymodels package) contains a function called fit_resamples (which is akin to fit()) that computes a set of performance metrics across resamples (or just one, as is the case with a validation set). The call requires either a parsnip model specification or a workflows::workflow, and rset object (as created with rsample::vfold_cv for example). You can also specify the performance metrics you want with the metrics argument or stick with the defaults. The control argument can be used to view/retain/save outputs if further tuning is desired. Your call might look like:

rf_res <-
  rf_wflow %>% 
  fit_resamples(resamples = ames_folds, control = keep_pred)

The output (not viewed here because it’s thicc) can be manipulated in a number of ways to view just what you need. You can for example run tune::collect_metrics(rf_res) to see just the performance metrics.

For more on how the outputs can be used for diagnostics and further model evaluation refer to section 10.3 in the book.