9.6 Mathematics of the SVC

• The SVC classifies a test observation based on which side of the hyperplane it lies.

$$\text{max}_{\beta_{0}...\beta_{p}, \epsilon_{1}...\epsilon_{n}, M} \space M$$ $$\text{subject to } \sum_{j=1}^{p}\beta_{j}^2 = 1$$ $$y_{i}(\beta_{0} + \beta_{1}X_{i1} + \beta_{2}X_{i2} ... + \beta_{p}X_{ip}) \geq M(1 - \epsilon_{i})$$ $$\epsilon_{i} \geq 0, \quad \sum_{i=1}^{n}\epsilon_{i} \leq C$$

• $C$ is a nonnegative tuning parameter, typically chosen through cross-validation, and can be thought of as the budget for margin violation by the observations.

• The $\epsilon_{i}$ are slack variables that allow individual observations to be on the wrong side of the margin or hyperplane. The $\epsilon_{i}$ indicates where the $i^{\text{th}}$ observation is located with regards to the margin and hyperplane.

  • If $\epsilon_{i} = 0$, the observation is on the correct side of the margin.
  • If $\epsilon_{i} > 0$, the observation is on the wrong side of margin
  • If $\epsilon_{i} > 1$, the observation is on the wrong side of the hyperplane.

• Since $C$ constrains the sum of the $\epsilon_{i}$, it determines the number and magnitude of violations to the margin. If $C=0$, there is no margin for violation, thus all the $\epsilon_{1},...,\epsilon_{n} = 0$.

• Note that if $C>0$, no more than $C$ observations can be on wrong side of hyperplane, since in these cases $\epsilon_{i} > 1$.