16.6 Shrinkage & bias_variance trade-off
set.seed(84735)
predictions_hierarchical <- posterior_predict(spotify_hierarchical,
newdata = artist_means)
# Posterior predictive plots
ppc_intervals(artist_means$popularity, yrep = predictions_hierarchical,
prob_outer = 0.80) +
ggplot2::scale_x_continuous(labels = artist_means$artist,
breaks = 1:nrow(artist_means)) +
xaxis_text(angle = 90, hjust = 1) +
geom_hline(yintercept = 58.4, linetype = "dashed")
Quizz What is shringage in this example ?
Shrinkage refers to the phenomenon in which the group-specific local trends in a hierarchical model are pulled or shrunk toward the global trends.
Shrinkage increases as the number of observations on group j, nj, decreases. That is, we rely more and more on global trends to understand a group for which we have little data.
Shrinkage increases when the variability within groups, σy, is large in comparison to the variability between groups, σμ. That is, we rely more and more on global trends to understand a group when there is little distinction in the patterns from one group to the next
The artists that shrunk the most are those with smaller sample sizes nj and popularity levels at the extremes of the spectrum.