Explanatory Model Analysis Book Club
Welcome
Book club meetings
Pace
Introductions
Before reading this book
Before reading this book
What will we learn?
What will we learn?
1
Introduction
Purpose of Predictive Models
First Stage
Starting the Big Data Era
The Dangers of Black Box Models
A New Modeling Process
We need more tools
Ethics requirements for any predictive model
Terminology
Terminology
Glass-box
Glass-box benefits
Model-specific
Model-specific Limitations
Model-specific Limitations
Model-agnostic techniques
Meeting Videos
Cohort 1
2
Model Development
Modeling Approaches
Predictive Modeling Examples
Importance of Model Development Process
CRISP-DM:
Cross-industry Standard Process for Data Mining
Lifecycle of a Predictive Model
MDP:
Model-development Process
MDP:
Model-development Process
Notation
Notation
Notation
Notation
Notation
Model description
Data understanding
Understanding dependent variable
Understanding explanatory variables
Model assembly (fitting)
The expected squared-error of prediction
Difference between explanatory and predictive modelling
Model audit
Meeting Videos
Cohort 1
3
Do-it-yourself
SLIDE 1
Meeting Videos
Cohort 1
4
Datasets and Models
SLIDE 1
Meeting Videos
Cohort 1
5
Introduction to Instance-level Exploration
5.1
Instance-level exploration methods
5.2
Break-down plots (variable attributions)
5.3
Local Interpretable Model-agnostic Explanations (LIME)
5.4
Ceteris-paribus profiles and methods
Meeting Videos
Cohort 1
6
Break-down Plots for Additive Attributions
6.1
Introduction
6.2
Intuition
6.3
Example: Titanic dataset
6.4
Pros and cons
6.5
R code examples
Meeting Videos
Cohort 1
7
Break-down Plots for Interactions
7.1
Intuition and Simplified Example
7.2
Method to Calculate Net Interaction Effects
7.3
Realistic Example Using Titanic Dataset
7.4
Pros and Cons
7.5
R Code Examples
Meeting Videos
Cohort 1
8
Shapley Additive Explanations (SHAP) for Average Attributions
Applying random order to BD plots
fare and class summary
Mean value of the attributions
SHAP description
SHAP description
Calculating Shapley Values
Important properties
Example: Johnny D
Example: Johnny D
Example: Johnny D
Pros and cons
Code snippets
Code snippets
Code snippets
Code snippets
Meeting Videos
Cohort 1
9
Local Interpretable Model-agnostic Explanations (LIME)
Overview
9.1
Intuition
9.2
Method
9.3
Titanic Data Example
9.4
Pros and Cons
9.5
Code Examples in R
Meeting Videos
Cohort 1
10
Ceteris-paribus Profiles
10.1
Introduction
10.2
Intuition
10.3
Example: Titanic imputed dataset
10.4
Pros and cons
10.5
Additional references
Meeting Videos
Cohort 1
11
Ceteris-paribus Oscillations
Basic idea
Graphical representation
Method
Method challenge
Local vs global importance
Example: Henry - random forest model
Example: Henry - random forest model
Example: Henry - random forest model
Pros and cons
Loading the data and model
Creating the explainer
Creating oscillations uniform
Plotting uniform results
Plotting empirical results
Applying a custom grid
Plotting custom grid
Meeting Videos
Cohort 1
12
Local-diagnostics Plots
Intuition
Local-fidelity plot example
CP profile example
Local-stability example
Method: Nearest neighbours
Method: Gower similarity measure
Pros and cons
Getting data and models
Creating model explainer
predict_diagnostics function
predict_diagnostics results
Local-fidelity plot
Running local-stability
Local-stability age plot
Local-stability class plot
Meeting Videos
Cohort 1
13
Summary of Instance-level Exploration
SLIDE 1
Meeting Videos
Cohort 1
14
Introduction to Dataset-level Exploration
14.1
Part III - Contents
Meeting Videos
Cohort 1
15
Model-performance Measures
15.1
Introduction
15.2
Method
15.3
Example: Apartment prices
15.4
Example: Titanic data
15.5
Pros and cons
15.6
R code snippetts
Cohort 1
16
Variable-importance Measures
SLIDE 1
Meeting Videos
Cohort 1
17
Partial-dependence Profiles
Overview
Intuition
Method
Basic Equations
Clustered partial-dependence profiles
Grouped partial-dependence profiles
Contrastive partial-dependence profiles
Feature Importance
Example: apartment-prices data
Partial-dependence-profiles
Clustered partial-dependence profiles
Grouped partial-dependence profiles
Constrastive partial-dependence profiles
Pros and Cons
Code Snippets in R
Partial-dependence profiles
Clustered partial-dependence profiles
Contrastive partial-dependence profiles
Meeting Videos
Cohort 1
18
Local-dependence and Accumulated-local Profiles
Partial-dependence (PD) profiles
Describing the problem - Linear Example
Describing the problem - Linear Example
Describing the problem - Linear Example
Describing the problem - Linear Example
Describing the problem - Linear Example
Describing the problem - Linear Example
Describing the problem - Linear Example
Describing the problem - Linear Example
Describing the problem - Linear Example
Describing the problem - Linear Example
Describing the problem - Linear Example
Describing the problem - Tree Example
Describing the problem - Tree Example
Describing the problem - Tree Example
Describing the problem - Tree Example
Local-dependence profile
Smooth boundaries between
\(N_j\)
subsets
LDP correlation problem
Accumulated-local profile
Selecting a c
Approximating an AL profile
Approximating an AL profile
Approximating an AL profile
Example with interactions
Ceteris-paribus (CP) profiles
\(X^1\)
Partial-dependence (PD) profile for
\(X^1\)
Defining local intervals for
\(X^1\)
Local-dependence (PD) profile for
\(X^1\)
Accumulated-local (AL) profile for
\(X^1\)
Example: apartment-prices data
Interpretation tips
DALEX
as wrapper of
ingredients
model_profile function
Creating LD plot
Creating AL plot
Creating all profiles plot
Meeting Videos
Cohort 1
19
Residual-Diagnostics Plots
Quality of predictions
Characteristics of a good model
Graphical methods to verify proporties
Standardized
(Pearson)
residuals
Exploring residuals for classification models
Exploring residuals for classification models
Exploring residuals for classical linear-regression models
Residuals
\(r_i\)
in function of predicted values
Square root of standardized residuals
\(\sqrt{\tilde{r}_i}\)
in function of predicted values
Standardized residuals
\(\tilde{r}_i\)
in function of leverage
\(l_i\)
Standardized residuals
\(\tilde{r}_i\)
in function of leverage
\(l_i\)
Standardized residuals
\(\tilde{r}_i\)
in function of values expected from the standard normal distribution
Apartment-prices: Model Performance
Apartment-prices: Residual distribution
Apartment-prices: Residual distribution
Apartment-prices: Residual distribution
Random Forest: Residuals
\(r_i\)
in function of observed values
Random Forest: Predicted in function of observed values
Random Forest: Residuals
\(r_i\)
in function of an (arbitrary) identifier of the observation
Random Forest: Residuals
\(r_i\)
in function of predicted value
Random Forest: Absolute value of residuals in function of the predicted
Pros and cons
Meeting Videos
Cohort 1
20
Summary of Dataset-level Exploration
SLIDE 1
Meeting Videos
Cohort 1
21
FIFA 19
SLIDE 1
Meeting Videos
Cohort 1
22
Reproducibility
SLIDE 1
Meeting Videos
Cohort 1
Published with bookdown
Explanatory Model Analysis Book Club
Chapter 15
Model-performance Measures
Learning objectives:
Model performance measures and evaluation
Goodness-of-fit (GoF)
Goodness-of-prediction (GoP)