View by Category

Loading...

PPT – Support Vector Machines: Linear Case PowerPoint presentation | free to download - id: 1d805d-MDRiN

The Adobe Flash plugin is needed to view this content

About This Presentation

Write a Comment

User Comments (0)

Transcript and Presenter's Notes

Support Vector Machines Linear Case

- Jieping Ye
- Department of Computer Science and Engineering
- Arizona State University
- http//www.public.asu.edu/jye02

Source Andrews tutorials on SVM

History of SVM

- SVM is inspired from statistical learning theory

3. - SVM was first introduced in 1992 1.
- SVM becomes popular because of its success in

handwritten digit recognition 2. - SVM is now regarded as an important example of

kernel methods, arguably the hottest area in

machine learning. http//www.kernel-machines.org/

1 B.E. Boser et al. A Training Algorithm for

Optimal Margin Classifiers. Proceedings of the

Fifth Annual Workshop on Computational Learning

Theory 5 144-152, Pittsburgh, 1992. 2 L.

Bottou et al. Comparison of classifier methods

a case study in handwritten digit recognition.

Proceedings of the 12th IAPR International

Conference on Pattern Recognition, vol. 2, pp.

77-82. 3 V. Vapnik. The Nature of Statistical

Learning Theory. 1nd edition, Springer, 1996.

Outline of lecture

- Linear classifier
- Maximum margin classifier
- Estimate the margin
- SVM for separable data
- SVM for non-separable data

Linear Classifiers

a

x

f

y

f(x,w,b) sign(w. x b)

denotes 1 denotes -1

How would you classify this data?

Linear Classifiers

a

x

f

y

f(x,w,b) sign(w. x b)

denotes 1 denotes -1

How would you classify this data?

Linear Classifiers

a

x

f

y

f(x,w,b) sign(w. x b)

denotes 1 denotes -1

How would you classify this data?

Linear Classifiers

a

x

f

y

f(x,w,b) sign(w. x b)

denotes 1 denotes -1

Any of these would be fine.. ..but which is best?

Classifier Margin

a

x

f

y

f(x,w,b) sign(w. x b)

denotes 1 denotes -1

Define the margin of a linear classifier as the

width that the boundary could be increased by

before hitting a datapoint.

Maximum Margin

a

x

f

y

f(x,w,b) sign(w. x b)

denotes 1 denotes -1

The maximum margin linear classifier is the

linear classifier with the maximum margin. This

is the simplest kind of SVM (Called an LSVM)

Linear SVM

Maximum Margin

a

x

f

y

f(x,w,b) sign(w. x b)

denotes 1 denotes -1

The maximum margin linear classifier is the

linear classifier with the maximum margin. This

is the simplest kind of SVM (Called an LSVM)

Support Vectors are those data points that the

margin pushes up against

Linear SVM

Why Maximum Margin?

- Intuitively this feels safest.
- If weve made a small error in the location of

the boundary this gives us least chance of

causing a misclassification. - The model is immune to removal of any

non-support-vector datapoints. - Theres some theory (using VC dimension) that is

related to (but not the same as) the proposition

that this is a good thing. - Empirically it works very very well.

f(x,w,b) sign(w. x - b)

denotes 1 denotes -1

The maximum margin linear classifier is the

linear classifier with the, um, maximum

margin. This is the simplest kind of SVM (Called

an LSVM)

Support Vectors are those datapoints that the

margin pushes up against

Estimate the Margin

wx b 0

x

- What is the distance expression for a point x to

a line wxb 0?

Estimate the Margin

wx b 0

distance

y

x

Estimate the Margin

wx b 0

Margin

- What is the expression for margin?

Maximize Margin

wx b 0

Margin

Maximize Margin

wx b 0

Margin

- Min-max problem

Maximize Margin

wx b 0

Margin

- Strategy

Maximum Margin Linear Classifier

- How to solve it?

Learning via Quadratic Programming

- QP is a well-studied class of optimization

algorithms to maximize a quadratic function of

some real-valued variables subject to linear

constraints.

Quadratic Programming

Quadratic criterion

Find

Subject to

n additional linear inequality constraints

And subject to

e additional linear equality constraints

Quadratic Programming

Non-separable

This is going to be a problem! What should we do?

This is going to be a problem! What should we

do? Idea 1 Find minimum w.w, while minimizing

number of training set errors. Problemette Two

things to minimize makes for an ill-define

optimization

Non-separable

This is going to be a problem! What should we

do? Idea 1.1 Minimize w.w C (train

errors) Theres a serious practical problem

thats about to make us reject this approach. Can

you guess what it is?

Non-separable

Tradeoff parameter

Non-separable

This is going to be a problem! What should we

do? Idea 1.1 Minimize w.w C (train

errors) Theres a serious practical problem

thats about to make us reject this approach. Can

you guess what it is?

Tradeoff parameter

Cant be expressed as a Quadratic Programming

problem. Solving it may be too slow. (Also,

doesnt distinguish between disastrous errors and

near misses)

Non-separable

This is going to be a problem! What should we

do? Idea 2.0 Minimize w.w C (distance of

error points to their

correct place)

Support Vector Machine for Noisy Data

- Balance the trade off between margin and

classification errors

Support Vector Machine for Noisy Data

Support Vector Machine for Noisy Data

- How do we determine the appropriate value for c ?
- Cross-validation

Support Vector Machine for Noisy Data

General optimization problem

Define the Lagrangian

Lagrangian dual problem

Weak duality theorem

Duality gap

Let

be the minimum of the Lagrangian with respect to

w, and let

be the maximum of the lagrangian dual with

respect to

If the constrains g are linear functions of w,

then the duality gap is 0.

Online book on optimization http//www.stanford.e

du/boyd/cvxbook/

Support Vector Machine for Noisy Data

Karush-Kuhn-Tucker Conditions

Complementarity condition

Feasibility condition

Support Vector Machine for Noisy Data

Use the Lagrangian formulation for the

optimization problem. Introduce a positive

Lagrangian multiplier for each inequality

constraint.

Lagrangian multipliers

Get the following Lagrangian

Support Vector Machine for Noisy Data

The Dual Form of QP

Maximize

where

Subject to these constraints

Then define

The Dual Form of QP

Maximize

where

Subject to these constraints

Then define

Then classify with f(x,w,b) sign(w. x b)

An Equivalent QP

Maximize

where

Subject to these constraints

Then define

Datapoints with ak gt 0 will be the support vectors

..so this sum only needs to be over the support

vectors.

Support Vectors

The Dual Form of QP

Maximize

where

Subject to these constraints

Then define

Then classify with f(x,w,b) sign(w. x b)

How to determine b ?

An Equivalent QP Determine b

Fix w

- A linear programming problem !

Another approach based on support vectors

SVM Applications

- Character recognition
- Tumor classification
- Document categorization
- Image classification

SVM Applications

- Protein second structure prediction
- Protein Fold and Remote Homology Detection

Next class

- Topics
- Introduction to kernels
- Nonlinear SVM
- Readings
- A Tutorial on Support Vector Machines for Pattern

Recognition - http//www.public.asu.edu/jye02/CLASSES/Fall-200

5/PAPERS/SVM-tutorial.pdf - An introduction to kernel-based learning

algorithms - http//ieeexplore.ieee.org/iel5/72/19749/00914517.

pdf?arnumber914517

About PowerShow.com

PowerShow.com is a leading presentation/slideshow sharing website. Whether your application is business, how-to, education, medicine, school, church, sales, marketing, online training or just for fun, PowerShow.com is a great resource. And, best of all, most of its cool features are free and easy to use.

You can use PowerShow.com to find and download example online PowerPoint ppt presentations on just about any topic you can imagine so you can learn how to improve your own slides and presentations for free. Or use it to find and download high-quality how-to PowerPoint ppt presentations with illustrated or animated slides that will teach you how to do something new, also for free. Or use it to upload your own PowerPoint slides so you can share them with your teachers, class, students, bosses, employees, customers, potential investors or the world. Or use it to create really cool photo slideshows - with 2D and 3D transitions, animation, and your choice of music - that you can share with your Facebook friends or Google+ circles. That's all free as well!

For a small fee you can get the industry's best online privacy or publicly promote your presentations and slide shows with top rankings. But aside from that it's free. We'll even convert your presentations and slide shows into the universal Flash format with all their original multimedia glory, including animation, 2D and 3D transition effects, embedded music or other audio, or even video embedded in slides. All for free. Most of the presentations and slideshows on PowerShow.com are free to view, many are even free to download. (You can choose whether to allow people to download your original PowerPoint presentations and photo slideshows for a fee or free or not at all.) Check out PowerShow.com today - for FREE. There is truly something for everyone!

You can use PowerShow.com to find and download example online PowerPoint ppt presentations on just about any topic you can imagine so you can learn how to improve your own slides and presentations for free. Or use it to find and download high-quality how-to PowerPoint ppt presentations with illustrated or animated slides that will teach you how to do something new, also for free. Or use it to upload your own PowerPoint slides so you can share them with your teachers, class, students, bosses, employees, customers, potential investors or the world. Or use it to create really cool photo slideshows - with 2D and 3D transitions, animation, and your choice of music - that you can share with your Facebook friends or Google+ circles. That's all free as well!

For a small fee you can get the industry's best online privacy or publicly promote your presentations and slide shows with top rankings. But aside from that it's free. We'll even convert your presentations and slide shows into the universal Flash format with all their original multimedia glory, including animation, 2D and 3D transition effects, embedded music or other audio, or even video embedded in slides. All for free. Most of the presentations and slideshows on PowerShow.com are free to view, many are even free to download. (You can choose whether to allow people to download your original PowerPoint presentations and photo slideshows for a fee or free or not at all.) Check out PowerShow.com today - for FREE. There is truly something for everyone!

presentations for free. Or use it to find and download high-quality how-to PowerPoint ppt presentations with illustrated or animated slides that will teach you how to do something new, also for free. Or use it to upload your own PowerPoint slides so you can share them with your teachers, class, students, bosses, employees, customers, potential investors or the world. Or use it to create really cool photo slideshows - with 2D and 3D transitions, animation, and your choice of music - that you can share with your Facebook friends or Google+ circles. That's all free as well!

For a small fee you can get the industry's best online privacy or publicly promote your presentations and slide shows with top rankings. But aside from that it's free. We'll even convert your presentations and slide shows into the universal Flash format with all their original multimedia glory, including animation, 2D and 3D transition effects, embedded music or other audio, or even video embedded in slides. All for free. Most of the presentations and slideshows on PowerShow.com are free to view, many are even free to download. (You can choose whether to allow people to download your original PowerPoint presentations and photo slideshows for a fee or free or not at all.) Check out PowerShow.com today - for FREE. There is truly something for everyone!

For a small fee you can get the industry's best online privacy or publicly promote your presentations and slide shows with top rankings. But aside from that it's free. We'll even convert your presentations and slide shows into the universal Flash format with all their original multimedia glory, including animation, 2D and 3D transition effects, embedded music or other audio, or even video embedded in slides. All for free. Most of the presentations and slideshows on PowerShow.com are free to view, many are even free to download. (You can choose whether to allow people to download your original PowerPoint presentations and photo slideshows for a fee or free or not at all.) Check out PowerShow.com today - for FREE. There is truly something for everyone!

Recommended

«

/ »

Page of

«

/ »

Promoted Presentations

Related Presentations

Page of

Home About Us Terms and Conditions Privacy Policy Contact Us Send Us Feedback

Copyright 2018 CrystalGraphics, Inc. — All rights Reserved. PowerShow.com is a trademark of CrystalGraphics, Inc.

Copyright 2018 CrystalGraphics, Inc. — All rights Reserved. PowerShow.com is a trademark of CrystalGraphics, Inc.

The PowerPoint PPT presentation: "Support Vector Machines: Linear Case" is the property of its rightful owner.

Do you have PowerPoint slides to share? If so, share your PPT presentation slides online with PowerShow.com. It's FREE!

Committed to assisting Asu University and other schools with their online training by sharing educational presentations for free