Recall the Advertising data from Chapter 2. Here are a few important questions that we might seek to address:
Simple linear regression: Very straightforward approach to predicting response \(Y\) on predictor \(X\).
\[Y \approx \beta_{0} + \beta_{1}X\]
\[\hat{y} = \hat{\beta}_{0} + \hat{\beta}_{1}x\]
\(\hat{\beta}_{0}\) = our approximation of intercept
\(\hat{\beta}_{1}\) = our approximation of slope
\(x\) = sample of \(X\)
\(\hat{y}\) = our prediction of \(Y\) from \(x\)
hat symbol denotes “estimated value”
Linear regression is a simple approach to supervised learning
Advertising data, the least squares fit for the regression of sales onto TV is shown. The fit is found by minimizing the residual sum of squares. Each grey line segment represents a residual. In this case a linear fit captures the essence of the relationship, although it overestimates the trend in the left of the plot.\[\mathrm{RSS} = e^{2}_{1} + e^{2}_{2} + \ldots + e^{2}_{n}\]
\[\mathrm{RSS} = (y_{1} - \hat{\beta}_{0} - \hat{\beta}_{1}x_{1})^{2} + (y_{2} - \hat{\beta}_{0} - \hat{\beta}_{1}x_{2})^{2} + \ldots + (y_{n} - \hat{\beta}_{0} - \hat{\beta}_{1}x_{n})^{2}\]
\[\hat{\beta}_{1} = \frac{\sum_{i=1}^{n}{(x_{i}-\bar{x})(y_{i}-\bar{y})}}{\sum_{i=1}^{n}{(x_{i}-\bar{x})^{2}}}\]
\[\hat{\beta}_{0} = \bar{y} - \hat{\beta}_{1}\bar{x}\]
Advertising data, using sales as the response and TV as the predictor. The red dots correspond to the least squares estimates \(\\hat\\beta_0\) and \(\\hat\\beta_1\), given by (3.4).Learning Objectives:
\[Y = \beta_{0} + \beta_{1}X + \epsilon\]
\[\mathrm{RSE} = \sqrt{\frac{\mathrm{RSS}}{n - 2}}\]
\[\mathrm{SE}(\hat\beta_0)^2 = \sigma^2 \left[\frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2}\right],\ \ \mathrm{SE}(\hat\beta_1)^2 = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}\]
\[\hat\beta_1 \pm 2\ \cdot \ \mathrm{SE}(\hat\beta_1)\]
\[\hat\beta_0 \pm 2\ \cdot \ \mathrm{SE}(\hat\beta_0)\]
Learning Objectives:
\[R^2 = 1 - \frac{RSS}{TSS}\]
where TSS is the total sum of squarse:
\[TSS = \Sigma (y_i - \bar{y})^2\]
Quiz: Can \(R^2\) be negative?
Multiple linear regression extends simple linear regression for p predictors:
\[Y = \beta_{0} + \beta_{1}X_1 + \beta_{2}X_2 + ... +\beta_{p}X_p + \epsilon_i\]
\(\beta_{j}\) is the average effect on \(Y\) from \(X_{j}\) holding all other predictors fixed.
Fit is once again choosing the \(\beta_{j}\) that minimizes the RSS.
Example in book shows that although fitting sales against newspaper alone indicated a significant slope (0.055 +- 0.017), when you include radio in a multiple regression, newspaper no longer has any significant effect. (-0.001 +- 0.006)
Is at least one of the predictors \(X_1\), \(X_2\), … , \(X_p\) useful in predicting the response?
F statistic close to 1 when there is no relationship, otherwise greater then 1.
\[F = \frac{(TSS-RSS)/p}{RSS/(n-p-1)}\]
Do all the predictors help to explain \(Y\) , or is only a subset of the predictors useful?
p-values can help identify important predictors, but it is possible to be mislead by this especially with large number of predictors. Variable selection methods include Forward selection, backward selection and mixed. Topic is continued in Chapter 6.
How well does the model fit the data?
\(R^2\) still gives proportion of the variance explained, so look for values “close” to 1. Can also look at RSE which is generalized for multiple regression as:
\[RSE = \sqrt{\frac{1}{n-p-1}RSS}\]
Given a set of predictor values, what response value should we predict, and how accurate is our prediction?
Three sets of uncertainty in predictions:
\[y_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \epsilon_i\]
| Qualitative Predicitor | \(x_{i1}\) | \(x_{i2}\) |
|---|---|---|
| level 0 (baseline) | 0 | 0 |
| level 1 | 1 | 0 |
| level 2 | 0 | 1 |
Coefficients are interpreted the average effect relative to the baseline.
Alternative is to use index variables, a different coefficient for each level:
\[y_i = \beta_{0 1} + \beta_{0 2} +\beta_{0 3} + \epsilon_i\]
Interaction / Synergy effects
Include a product term to account for synergy where one changes in one variable changes the association of the Y with another:
\[Y = \beta_{0} + \beta_{1}X_1 + \beta_{2}X_2 + \beta_{3}X_1 X_2 + \epsilon_i\]
\[Y = \beta_{0} + \beta_{1}X + \beta_{2}X^2 + ... \beta_{n}X^n + \epsilon_i\]
Non-linear relationships
Residual plots are useful tool to see if any remaining trends exist. If so consider fitting transformation of the data.
Correlation of Error Terms
Linear regression assumes that the error terms \(\epsilon_i\) are uncorrelated. Residuals may indicate that this is not correct (obvious tracking in the data). One could also look at the autocorrelation of the residuals. What to do about it?
Non-constant variance of error terms
Again this can be revealed by examining the residuals. Consider transformation of the predictors to remove non-constant variance. The figure below shows residuals demonstrating non-constant variance, and shows this being mitigated to a great extent by log transforming the data.
Outliers
High Leverage Points
Collinearity
\[VIF(\hat{\beta_j}) = \frac{1}{1-R^2_{X_j|X_{-j}}}\]
where \(R^2_{X_j|X_{-j}}\) is the \(R^2\) from a regression of \(X_j\) onto all the other predictors.
Is there a relationship between advertising budget and sales?
Tool: Multiple regression, look at F-statistic.
How strong is the relationship between advertising budget and sales?
Tool: \(R^2\) and RSE
Which media are associated with sales?
Tool: p-values for each predictor’s t-statistic. Explored further in chapter 6.
How large is the association between each medium and sales?
Tool: Confidence intervals on \(\hat{\beta_j}\)
How accurately can we predict future sales?
Tool:: Prediction intervals for individual response, confidence intervals for average response.
Is the relationship linear?
Tool: Residual Plots
Is there synergy among the advertising media?
Tool: Interaction terms and associated p-vales.