Support Vector Machines: Hype or Hallelujah?

1
Support Vector MachinesHype or Hallelujah?

• Kristin Bennett
• Math Sciences Dept
• Rensselaer Polytechnic Inst.
• http//www.rpi.edu/bennek

2
Outline

• Support Vector Machines for Classification
• Linear Discrimination
• Nonlinear Discrimination
• Extensions
• Hallelujah
• Hype

3
Binary Classification

• Example Medical Diagnosis
• Is it benign or malignant?

4
Linear Classification Model

• Given training data
• Linear model - find
• Such that

5
Best Linear Separator?
6
Best Linear Separator?
7
Best Linear Separator?
8
Best Linear Separator?
9
Best Linear Separator?
10
Find Closest Points in Convex Hulls
d
c
11
Plane Bisect Closest Points
d
c
12
Find using quadratic program
Many existing and new solvers.
13
Best Linear SeparatorSupporting Plane Method
Maximize distance Between two parallel
supporting planes
Distance Margin
14
Maximize margin using quadratic program
15
Dual of Closest Points Method is Support Plane
Method
Solution only depends on support vectors
16
Support Vector Machines (SVM)
A methodology for inference based on Vapniks
Statistical Learning Theory.

• Key Ideas
• Maximize Margins
• Do the Dual
• Construct Kernels

17
Statistical Learning Theory

• Misclassification error and the function
  complexity bound generalization error.
• Maximizing margins minimizes complexity.
• Eliminates overfitting.
• Solution depends only on Support Vectors not
  number of attributes.

18
Margins and Complexity
Skinny margin is more flexible thus more complex.
19
Margins and Complexity
Fat margin is less complex.
20
Linearly Inseparable Case
 
                                         
Convex Hulls Intersect! Same argument
wont work.
21
Reduced Convex Hulls Dont Intersect
Reduce by adding upper bound D
22
Find Closest Points Then Bisect
No change except for D. D determines number of
Support Vectors.
23
Linearly Inseparable CaseSupporting Plane Method
Just add non-negative error vector z.
24
Dual of Closest Points Method is Support Plane
Method
Solution only depends on support vectors
25
Nonlinear Classification
26
Nonlinear Classification Map to higher
dimensional space
IDEA Map each point to higher dimensional
feature space and construct linear discriminant
in the higher dimensional space.
Dual SVM becomes
27
Generalized Inner Product
By Hilbert-Schmidt Kernels (Courant and Hilbert
1953)
for certain ? and K, e.g.
28
Final Classification via Kernels
The Dual SVM becomes
29

30
Final SVM Algorithm

• Solve Dual SVM QP
• Recover primal variable b
• Classify new x

Solution only depends on support vectors
31
Support Vector Machines (SVM)

• Key Formulation Ideas
• Maximize Margins
• Do the Dual
• Construct Kernels
• Generalization Error Bounds
• Practical Algorithms

32
Hallelujah!

• Generalization theory and practice meet
• General methodology for many types of problems
• Same Program New Kernel New method
• No problems with local minima
• Few model parameters. Selects capacity.
• Robust optimization methods.
• Successful Applications

BUT
33
HYPE?

• Will SVMs beat my best hand-tuned method Z for X?
• Do SVM scale to massive datasets?
• How to chose C and Kernel?
• What is the effect of attribute scaling?
• How to handle categorical variables?
• How to incorporate domain knowledge?
• How to interpret results?

34
Support Vector Machine Resources

• http//www.support-vector.net/
• http//www.kernel-machines.org/
• Links off my web page
• http//www.rpi.edu/bennek