An NPoint SMO Implementation for the Support Vector Machine

1
An N-Point SMO Implementation for the Support
Vector Machine Christopher Sentelle, Michael
Georgiopoulos, Georgios Anagnostopoulos, Cynthia
Young
Summary and Conclusions
In this research, we derived an extension to the
SMO algorithm for an N-point sub-problem (N 2).
The N-point update is described by a system of
linear equations having a closed-form solution.
In addition, constraints can be incrementally
added to the problem as the binding inequality
constraints are discovered. In conclusion, we
have maintained the ability to analytically solve
the QP sub-problem, while extending the size of
the sub-problem. This facilitates algorithm
implementation by those less familiar with QP
techniques while offering decreased training
times. The performance results, where we tested a
4-point SMO, demonstrate that we have extended
the sub-problem size without significantly
degrading the performance when compared to the
original 2-point sub-problem. In addition,
similar to the 2-point SMO algorithm, the 4-point
algorithm significantly outperforms the SVMLight
algorithm (one of the benchmark SVM algorithms),
in terms of training time. Finally, for the
four-class dataset, we observe an advantage in
working with more than 2 points per sub-problem
(faster training time is attained with the
4-point SMO). This advantage might be observed
for other datasets too (future research focus).
Support Vector Machines
The Algorithm
The Support Vector Machine (SVM), introduced by
Vapnik in 1982, is a modern, state-of-the-art
machine learning approach with strong theoretical
underpinnings. Support vector machines exhibit
good generalization (i.e., correctly recognize
unseen data) and resist the tendency to
over-train, the phenomenon where training
examples are memorized at the expense of good
generalization on unseen data. The SVM
classifier gets its name from the fact that it
identifies key training examples crucial to
classification. If the system were retrained with
only the support vectors, identical results would
be obtained.
SVM example from the UCI four class dataset. In
this instance, the data was converted to two
classes. The three sets of blue lines represent
the separating hyperplane and margins when
non-linearly mapped to the problem space. The
yellow markers represent data points identified
as support vectors. These contain all of the
information necessary for generating the
separating surface.
Identify violated constraints for global minimum
Theil-Van de Panne Procedure
Exit
Box constraints satisfied
NViolSize1
For each violator set of NViolSize
Convert violator set to equality constraints and
solve
Lagrange Multipliers 0
Exit
Box constraints satisfied
More sets of NViolSize
Create a set of NViolSize1 from each violated
constraint
NViolSize NViolSize1
Lagrange multiplier
Separating hyperplane
Each equality constraint can be incrementally
added to the N-Point update
Primal Problem
Penalty term
Applications
Slack variable

• The update equation does not deal with the
  inequality constraints (box constraints)
• The solution can be simply clipped for the
  2-point case
• For N-points, clipping becomes exponentially
  complex
• The Theil-Van de Panne procedure allows smart
  determination of which inequality constraints
  become equality constraints
• Note that a closed-form inverse exists, which can
  be incrementally computed as equality
  constraints are added

Future Work
Dual Problem

• Testing beyond a 4-Point implementation
• Research alternatives to the Theil-Van de Panne
  procedure
• Investigate interior point method simplifications
  with box constraint simplifications
• Further investigate interior point methods
• Include SVM assumptions in interior point method
  such as box constraints
• Investigate parallel interior point algorithms
  for SVM

• Bioinformatics
• Predicting Gene Sequences
• Predicting Protein Structures
• Speech recognition
• Face recognition
• Facial expression recognition
• Network intrusion detection
• Handwritten recognition

Support vector
Kernel
Margin

• The support vector machine solves a quadratic
  program with a global minimum
• The dual form allows introduction of a non-linear
  mapping kernel function
• Slack variables allow some data points to violate
  the margin
• The support vectors are within or on the margin

Dual Problem Matrix Formulation
2-Point Update
References
Motivation

• J. C. G. Boot, Quadratic Programming, Vol. 2,
  Rand McNally Company, pp. 95-124, 1964.
• R.-E. Fan, P.-H. Chen, and C.-J. Lin. Working
  set selection using the second order information
  for training SVM. Journal of Machine Learning
  Research, vol. 6, 1889-1918, 2005.
• T. Joachims, Making large-Scale SVM Learning
  Practical. Advances in Kernel Methods - Support
  Vector Learning, B. Schölkopf and C. Burges and
  A. Smola (ed.), MIT-Press, 1999.
• J. C. Platt, Fast Training of Support Vector
  Machines Using Sequential Minimal Optimization -
  Support Vector Learning, B. Schölkopf and C.
  Burges and A. Smola (ed.), MIT-Press, 1999.
• V. Vapnik, Estimation of Dependences Based on
  Empirical Data, Springer-Verlag, (1982).

The Support Vector Machine approach has been
successfully applied to a number of fields, and
for problems where the dataset size and the
number of features per dataset is large. While
the SVM algorithm performs well, training times
are still comparatively slow for large datasets.
The difficulty arises in solving the quadratic
problem (QP) associated with the support vector
machine approach, which is intractable for most
commercial QP solvers. The Sequential Minimum
Optimization (SMO) algorithm, introduced by Platt
in 1999, decomposes the QP problem into a
sub-problem containing 2 data points.
Sub-problems are repeatedly chosen and solved
until the larger problem is solved. Platt shows
that the sub-problem can be solved analytically,
avoiding QP techniques. In this work, we show how
SMO can be extended to more than 2 data points,
per sub-problem, while still providing an
analytical solution and fast training times
commensurate with the 2 point technique.
N-Point Update

• The Sequential Minimal Optimization (SMO)
  algorithm converts the QP problem to an
  analytical one by analyzing two points at a
  time
• In this research, we derive a more general
  N-Point update

• The SMO method for selecting 2-points is simply
  extended. This example shows the extension to 4
  points.

Results

• Testing performed with 4-Point SMO
• Compared w/ SVMLight, LIBSVM implementations
• Four-class (RBF Kernel)
• 2x faster, 4x fewer iterations
• Sonar (Linear Kernel)
• 2x fewer iterations, equivalent speed
• German
• 2x faster for linear kernel only
• Time per iteration 2x that of 2-Point SMO
• Note that the 4-Point SMO is consistently faster
  than SVMLight

Acknowledgments
Chris Sentelle gratefully acknowledges the
support of the College of Engineering Computer
Science and the I2Lab at the University of
Central Florida.