Mathematical Programming for Support Vector Machines - PowerPoint PPT Presentation

About This Presentation
Title:

Mathematical Programming for Support Vector Machines

Description:

Typically nonlinear in the input space. Linear in a higher dimensional space ... allows complex classification (No positive definiteness 'Mercer' condition! ... – PowerPoint PPT presentation

Number of Views:133
Avg rating:3.0/5.0
Slides: 44
Provided by: olvilman9
Learn more at: https://ftp.cs.wisc.edu
Category:

less

Transcript and Presenter's Notes

Title: Mathematical Programming for Support Vector Machines


1
Mathematical Programmingfor Support Vector
Machines
  • Olvi L. Mangasarian
  • University of Wisconsin - Madison

INRIA Rocquencourt 17 July 2001
2
What is a Support Vector Machine?
  • An optimally defined surface
  • Typically nonlinear in the input space
  • Linear in a higher dimensional space
  • Implicitly defined by a kernel function

3
What are Support Vector Machines Used For?
  • Classification
  • Regression Data Fitting
  • Supervised Unsupervised Learning

(Will concentrate on classification)
4
Example of Nonlinear ClassifierCheckerboard
Classifier
5
Outline of Talk
  • Generalized support vector machines (SVMs)
  • Completely general kernel allows complex
    classification (No positive definiteness
    Mercer condition!)
  • Smooth support vector machines
  • Smooth solve SVM by a fast global Newton
    method
  • Reduced support vector machines
  • Handle large datasets with nonlinear rectangular
    kernels
  • Nonlinear classifier depends on 1 to 10 of
    data points
  • Proximal support vector machines
  • Proximal planes replace halfspaces
  • Solve linear equations instead of QP or LP
  • Extremely fast simple

6
Generalized Support Vector Machines2-Category
Linearly Separable Case
A
A-
7
Generalized Support Vector MachinesAlgebra of
2-Category Linearly Separable Case
8
Generalized Support Vector MachinesMaximizing
the Margin between Bounding Planes
A
A-
9
Generalized Support Vector MachinesThe Linear
Support Vector Machine Formulation
10
Breast Cancer Diagnosis Application97 Tenfold
Cross Validation Correctness780 Samples494
Benign, 286 Malignant
11
Another Application Disputed Federalist
PapersBosch Smith 199856 Hamilton, 50
Madison, 12 Disputed
12
SVM as an Unconstrained Minimization Problem
13
Smoothing the Plus Function Integrate the
Sigmoid Function
14
SSVM The Smooth Support Vector Machine
Smoothing the Plus Function
15
Newton Minimize a sequence of quadratic
approximations to the strongly convex objective
function, i.e. solve a sequence of linear
equations in n1 variables. (Small dimensional
input space.)
Armijo Shorten distance between successive
iterates so as to generate sufficient decrease in
objective function. (In computational reality,
not needed!)
Global Quadratic Convergence Starting from any
point, the iterates guaranteed to converge to
the unique solution at a quadratic rate, i.e.
errors get squared. (Typically, 6 to 8
iterations without an Armijo.)
16
Nonlinear SSVM Formulation(Prior to Smoothing)
17
The Nonlinear Classifier
  • Where K is a nonlinear kernel, e.g.

18
(No Transcript)
19
Checkerboard Polynomial Kernel ClassifierBest
Previous Result Kaufman 1998
20
(No Transcript)
21
Difficulties with Nonlinear SVM for Large
Problems
  • Nonlinear separator depends on almost entire
    dataset
  • Have to store the entire dataset after solve the
    problem

22
Reduced Support Vector Machines (RSVM) Large
Nonlinear Kernel Classification Problems
  • RSVM can solve very large problems

23
Checkerboard 50-by-50 Square Kernel Using 50
Random Points Out of 1000
24
RSVM Result on Checkerboard Using SAME 50 Random
Points Out of 1000
25
RSVM on Large UCI Adult DatasetStandard
Deviation over 50 Runs 0.001
26
CPU Times on UCI Adult DatasetRSVM, SMO and
PCGC with a Gaussian Kernel
27
CPU Time Comparison on UCI DatasetRSVM, SMO and
PCGC with a Gaussian Kernel
Time( CPU sec. )
Training Set Size
28
PSVM Proximal Support Vector Machines
  • Fast new support vector machine classifier
  • Proximal planes replace halfspaces
  • Order(s) of magnitude faster than standard
    classifiers
  • Extremely simple to implement
  • 4 lines of MATLAB code
  • NO optimization packages (LP,QP) needed

29
Proximal Support Vector MachineUse 2 Proximal
Planes Instead of 2 Halfspaces
A
A-
30
PSVM Formulation
We have the SSVM formulation
This simple, but critical modification, changes
the nature of the optimization problem
significantly!
31
Advantages of New Formulation
  • Objective function remains strongly convex
  • An explicit exact solution can be written in
    terms of the problem data
  • PSVM classifier is obtained by solving a single
    system of linear equations in the usually small
    dimensional input space
  • Exact leave-one-out-correctness can be obtained
    in terms of problem data

32
Linear PSVM
  • Setting the gradient equal to zero, gives a
    nonsingular system of linear equations.
  • Solution of the system gives the desired PSVM
    classifier

33
Linear PSVM Solution
34
Linear Proximal SVM Algorithm
35
Nonlinear PSVM Formulation
36
Nonlinear PSVM
However, reduced kernel technique (RSVM) can be
used to reduce dimensionality.
37
Linear Proximal SVM Algorithm
Non

Solve
38
PSVM MATLAB Code
function w, gamma psvm(A,d,nu) PSVM linear
and nonlinear classification INPUT A,
ddiag(D), nu. OUTPUT w, gamma w, gamma
pvm(A,d,nu) m,nsize(A)eones(m,1)HA
-e v(dH) vHDe
r(speye(n1)/nuHH)\v solve (I/nuHH)rv
wr(1n)gammar(n1) getting w,gamma from
r
39
Linear PSVM Comparisons with Other SVMsMuch
Faster, Comparable Correctness
40
Gaussian Kernel PSVM Classifier Spiral Dataset
94 Red Dots 94 White Dots
41
Conclusion
  • Mathematical Programming plays an essential role
    in SVMs
  • Theory
  • New formulations
  • Generalized proximal SVMs
  • New algorithm-enhancement concepts
  • Smoothing (SSVM)
  • Data reduction (RSVM)
  • Algorithms
  • Fast SSVM, PSVM
  • Massive RSVM

42
Future Research
  • Theory
  • Concave minimization
  • Concurrent feature data reduction
  • Multiple-instance learning
  • SVMs as complementarity problems
  • Kernel methods in nonlinear programming
  • Algorithms
  • Multicategory classification algorithms
  • Incremental algorithms

43
Talk Papers Available on Web
  • www.cs.wisc.edu/olvi
Write a Comment
User Comments (0)
About PowerShow.com