4.4 Generative Models for Classification

Why Logistic Regression is not ideal?

• When there is substantial separation between the two classes, the parameter estimates for the logistic regression model are surprisingly unstable.

• If the distribution of the predictors X is approximately normal in each of the classes and the sample size is small, then the generative modelling may be more accurate than logistic regression.

• Generative modelling can be naturally extended to the case of more than two response classes.

Common notations:
- K \(\Longrightarrow\) response class

• \(π_k \Longrightarrow\) overall or prior probability that a randomly chosen observation comes from the prior kth class; can be obtained from the random sample from the population

• \(f_k(X) ≡ Pr(X|Y = k)^1 \Longrightarrow\) the density function of X density for an observation that comes from the kth class; requires some underlying assumption to estimate

Bayes’ theorem states that

\[Pr(Y = k|X = x) = \frac {π_k f_k(x)}{\sum_{l =1}^{k} π_lf_l(x)}\]

• \(p_k(x) = Pr(Y = k|X = x) \Longrightarrow\) posterior probability that an observation posterior X = x belongs to the kth class; computed from \(f_k(X)\)