Bayes and Logistic Regression

The class posterior distribution for a Naive Bayes classification model has the same form as multinomial logistic regression:

$$p(y = c|\vec{x}, \vec{\theta}) = \displaystyle\frac{e^{\beta_{c}^{T}\vec{x} + \gamma_{c}}}{\displaystyle\sum_{c'=1}^{C} e^{\beta_{c}^{T}\vec{x} + \gamma_{c}}}$$

Naive Bayes

$$f(y | x_{1}, x_{2}, ..., x_{p}) = \frac{f(y) \cdot L(y | x_{1}, x_{2}, ..., x_{p})}{\sum_{y'} f(y') \cdot L(y' | x_{1}, x_{2}, ..., x_{p})}$$

• conditionally independent $\rightarrow$ computationally efficient
• generalizes to more than two categories
• assumptions violated commonly in practice
• optimizes joint likelihood $\displaystyle\prod_{n} p(y_{n},\vec{x}_{n}|\vec{\theta})$

Logistic Regression

$$\log\left(\frac{\pi}{1-\pi}\right) = \beta_{0} + \beta_{1}X_{1} + \cdots + \beta_{k}X_{p}$$

• binary classification
• coefficients $\rightarrow$ illumination of the relationships among these variables
• optimizes conditional likelihood $\displaystyle\prod_{n} p(y_{n}|\vec{x}_{n},\vec{\theta})$