12.1 What is a Tuning Parameter?

An unknown structural or other kind of value that has significant impact on the model but cannot be directly estimated from these data

12.1.1 Examples

• Machine Learning (hyperparameters)
  • Boosting: number of boosting iterations
  • ANN: number of hidden units and type of activation function
  • Modern Gradient Descent: Learning rates, momentus, and iterations
  • Random Forest: number of predictors, number of trees, number of data points
• Preprocessing (tuning parameters)
  • PCA: number of extracted components
  • Imputation (uses KNN): number of neighbors
• Statistical Models (structural parameters)
  • Binary Regression (logistic regression): probit, logit link
  • Longitudinal Models: correlation and covariance structure of the data