View by Category

Loading...

PPT – Mathematical Programming for Support Vector Machines PowerPoint presentation | free to download - id: 10ff94-ZjgyZ

The Adobe Flash plugin is needed to view this content

About This Presentation

Write a Comment

User Comments (0)

Transcript and Presenter's Notes

Mathematical Programmingfor Support Vector

Machines

- Olvi L. Mangasarian
- University of Wisconsin - Madison

INRIA Rocquencourt 17 July 2001

What is a Support Vector Machine?

- An optimally defined surface
- Typically nonlinear in the input space
- Linear in a higher dimensional space
- Implicitly defined by a kernel function

What are Support Vector Machines Used For?

- Classification
- Regression Data Fitting
- Supervised Unsupervised Learning

(Will concentrate on classification)

Example of Nonlinear ClassifierCheckerboard

Classifier

Outline of Talk

- Generalized support vector machines (SVMs)
- Completely general kernel allows complex

classification (No positive definiteness

Mercer condition!) - Smooth support vector machines
- Smooth solve SVM by a fast global Newton

method - Reduced support vector machines
- Handle large datasets with nonlinear rectangular

kernels - Nonlinear classifier depends on 1 to 10 of

data points - Proximal support vector machines
- Proximal planes replace halfspaces
- Solve linear equations instead of QP or LP
- Extremely fast simple

Generalized Support Vector Machines2-Category

Linearly Separable Case

A

A-

Generalized Support Vector MachinesAlgebra of

2-Category Linearly Separable Case

Generalized Support Vector MachinesMaximizing

the Margin between Bounding Planes

A

A-

Generalized Support Vector MachinesThe Linear

Support Vector Machine Formulation

Breast Cancer Diagnosis Application97 Tenfold

Cross Validation Correctness780 Samples494

Benign, 286 Malignant

Another Application Disputed Federalist

PapersBosch Smith 199856 Hamilton, 50

Madison, 12 Disputed

SVM as an Unconstrained Minimization Problem

Smoothing the Plus Function Integrate the

Sigmoid Function

SSVM The Smooth Support Vector Machine

Smoothing the Plus Function

Newton Minimize a sequence of quadratic

approximations to the strongly convex objective

function, i.e. solve a sequence of linear

equations in n1 variables. (Small dimensional

input space.)

Armijo Shorten distance between successive

iterates so as to generate sufficient decrease in

objective function. (In computational reality,

not needed!)

Global Quadratic Convergence Starting from any

point, the iterates guaranteed to converge to

the unique solution at a quadratic rate, i.e.

errors get squared. (Typically, 6 to 8

iterations without an Armijo.)

Nonlinear SSVM Formulation(Prior to Smoothing)

The Nonlinear Classifier

- Where K is a nonlinear kernel, e.g.

(No Transcript)

Checkerboard Polynomial Kernel ClassifierBest

Previous Result Kaufman 1998

(No Transcript)

Difficulties with Nonlinear SVM for Large

Problems

- Nonlinear separator depends on almost entire

dataset

- Have to store the entire dataset after solve the

problem

Reduced Support Vector Machines (RSVM) Large

Nonlinear Kernel Classification Problems

- RSVM can solve very large problems

Checkerboard 50-by-50 Square Kernel Using 50

Random Points Out of 1000

RSVM Result on Checkerboard Using SAME 50 Random

Points Out of 1000

RSVM on Large UCI Adult DatasetStandard

Deviation over 50 Runs 0.001

CPU Times on UCI Adult DatasetRSVM, SMO and

PCGC with a Gaussian Kernel

CPU Time Comparison on UCI DatasetRSVM, SMO and

PCGC with a Gaussian Kernel

Time( CPU sec. )

Training Set Size

PSVM Proximal Support Vector Machines

- Fast new support vector machine classifier
- Proximal planes replace halfspaces
- Order(s) of magnitude faster than standard

classifiers - Extremely simple to implement
- 4 lines of MATLAB code
- NO optimization packages (LP,QP) needed

Proximal Support Vector MachineUse 2 Proximal

Planes Instead of 2 Halfspaces

A

A-

PSVM Formulation

We have the SSVM formulation

This simple, but critical modification, changes

the nature of the optimization problem

significantly!

Advantages of New Formulation

- Objective function remains strongly convex
- An explicit exact solution can be written in

terms of the problem data - PSVM classifier is obtained by solving a single

system of linear equations in the usually small

dimensional input space - Exact leave-one-out-correctness can be obtained

in terms of problem data

Linear PSVM

- Setting the gradient equal to zero, gives a

nonsingular system of linear equations. - Solution of the system gives the desired PSVM

classifier

Linear PSVM Solution

Linear Proximal SVM Algorithm

Nonlinear PSVM Formulation

Nonlinear PSVM

However, reduced kernel technique (RSVM) can be

used to reduce dimensionality.

Linear Proximal SVM Algorithm

Non

Solve

PSVM MATLAB Code

function w, gamma psvm(A,d,nu) PSVM linear

and nonlinear classification INPUT A,

ddiag(D), nu. OUTPUT w, gamma w, gamma

pvm(A,d,nu) m,nsize(A)eones(m,1)HA

-e v(dH) vHDe

r(speye(n1)/nuHH)\v solve (I/nuHH)rv

wr(1n)gammar(n1) getting w,gamma from

r

Linear PSVM Comparisons with Other SVMsMuch

Faster, Comparable Correctness

Gaussian Kernel PSVM Classifier Spiral Dataset

94 Red Dots 94 White Dots

Conclusion

- Mathematical Programming plays an essential role

in SVMs

- Theory

- New formulations
- Generalized proximal SVMs

- New algorithm-enhancement concepts
- Smoothing (SSVM)
- Data reduction (RSVM)

- Algorithms

- Fast SSVM, PSVM

- Massive RSVM

Future Research

- Theory

- Concave minimization
- Concurrent feature data reduction
- Multiple-instance learning

- SVMs as complementarity problems

- Kernel methods in nonlinear programming

- Algorithms

- Multicategory classification algorithms
- Incremental algorithms

Talk Papers Available on Web

- www.cs.wisc.edu/olvi

About PowerShow.com

PowerShow.com is a leading presentation/slideshow sharing website. Whether your application is business, how-to, education, medicine, school, church, sales, marketing, online training or just for fun, PowerShow.com is a great resource. And, best of all, most of its cool features are free and easy to use.

You can use PowerShow.com to find and download example online PowerPoint ppt presentations on just about any topic you can imagine so you can learn how to improve your own slides and presentations for free. Or use it to find and download high-quality how-to PowerPoint ppt presentations with illustrated or animated slides that will teach you how to do something new, also for free. Or use it to upload your own PowerPoint slides so you can share them with your teachers, class, students, bosses, employees, customers, potential investors or the world. Or use it to create really cool photo slideshows - with 2D and 3D transitions, animation, and your choice of music - that you can share with your Facebook friends or Google+ circles. That's all free as well!

For a small fee you can get the industry's best online privacy or publicly promote your presentations and slide shows with top rankings. But aside from that it's free. We'll even convert your presentations and slide shows into the universal Flash format with all their original multimedia glory, including animation, 2D and 3D transition effects, embedded music or other audio, or even video embedded in slides. All for free. Most of the presentations and slideshows on PowerShow.com are free to view, many are even free to download. (You can choose whether to allow people to download your original PowerPoint presentations and photo slideshows for a fee or free or not at all.) Check out PowerShow.com today - for FREE. There is truly something for everyone!

You can use PowerShow.com to find and download example online PowerPoint ppt presentations on just about any topic you can imagine so you can learn how to improve your own slides and presentations for free. Or use it to find and download high-quality how-to PowerPoint ppt presentations with illustrated or animated slides that will teach you how to do something new, also for free. Or use it to upload your own PowerPoint slides so you can share them with your teachers, class, students, bosses, employees, customers, potential investors or the world. Or use it to create really cool photo slideshows - with 2D and 3D transitions, animation, and your choice of music - that you can share with your Facebook friends or Google+ circles. That's all free as well!

For a small fee you can get the industry's best online privacy or publicly promote your presentations and slide shows with top rankings. But aside from that it's free. We'll even convert your presentations and slide shows into the universal Flash format with all their original multimedia glory, including animation, 2D and 3D transition effects, embedded music or other audio, or even video embedded in slides. All for free. Most of the presentations and slideshows on PowerShow.com are free to view, many are even free to download. (You can choose whether to allow people to download your original PowerPoint presentations and photo slideshows for a fee or free or not at all.) Check out PowerShow.com today - for FREE. There is truly something for everyone!

presentations for free. Or use it to find and download high-quality how-to PowerPoint ppt presentations with illustrated or animated slides that will teach you how to do something new, also for free. Or use it to upload your own PowerPoint slides so you can share them with your teachers, class, students, bosses, employees, customers, potential investors or the world. Or use it to create really cool photo slideshows - with 2D and 3D transitions, animation, and your choice of music - that you can share with your Facebook friends or Google+ circles. That's all free as well!

For a small fee you can get the industry's best online privacy or publicly promote your presentations and slide shows with top rankings. But aside from that it's free. We'll even convert your presentations and slide shows into the universal Flash format with all their original multimedia glory, including animation, 2D and 3D transition effects, embedded music or other audio, or even video embedded in slides. All for free. Most of the presentations and slideshows on PowerShow.com are free to view, many are even free to download. (You can choose whether to allow people to download your original PowerPoint presentations and photo slideshows for a fee or free or not at all.) Check out PowerShow.com today - for FREE. There is truly something for everyone!

For a small fee you can get the industry's best online privacy or publicly promote your presentations and slide shows with top rankings. But aside from that it's free. We'll even convert your presentations and slide shows into the universal Flash format with all their original multimedia glory, including animation, 2D and 3D transition effects, embedded music or other audio, or even video embedded in slides. All for free. Most of the presentations and slideshows on PowerShow.com are free to view, many are even free to download. (You can choose whether to allow people to download your original PowerPoint presentations and photo slideshows for a fee or free or not at all.) Check out PowerShow.com today - for FREE. There is truly something for everyone!

Recommended

«

/ »

Page of

«

/ »

Promoted Presentations

Related Presentations

Page of

Home About Us Terms and Conditions Privacy Policy Contact Us Send Us Feedback

Copyright 2018 CrystalGraphics, Inc. — All rights Reserved. PowerShow.com is a trademark of CrystalGraphics, Inc.

Copyright 2018 CrystalGraphics, Inc. — All rights Reserved. PowerShow.com is a trademark of CrystalGraphics, Inc.

The PowerPoint PPT presentation: "Mathematical Programming for Support Vector Machines" is the property of its rightful owner.

Do you have PowerPoint slides to share? If so, share your PPT presentation slides online with PowerShow.com. It's FREE!

Committed to assisting Wisc University and other schools with their online training by sharing educational presentations for free