The 2005 UK Workshop on Computational Intelligence

1
The 2005 UK Workshop on Computational
Intelligence 5-7 September 2005, London
L2-SVM Based Fuzzy Classifier with Automatic
Model Selection and Fuzzy Rule Ranking Shang-Min
g Zhou and John Q. Gan Department of Computer
Science, University of Essex, UK
2
Background and Objectives(1/4)

• Advantage of SVM
• Parsimonious solutions based on quadratic
  programming

• The challenges
• To apply SVM techniques to parsimonious fuzzy
  system modelling for regression and
  classification.
• Difficult to link the kernel functions in SVM to
  basis functions in fuzzy system.

3
Background and Objectives(2/4)

• Chen and Wangs work Chen and Wang 2003
• Established this sort of relation for fuzzy
  classification based on L1-SVM techniques.
• Parameters kernel parameters and regularization
  parameter not updated optimally from data for
  fuzzy rule induction.

• One objective
• To apply L2-SVM techniques to fuzzy system
  modelling to optimally learn the parameters from
  data in terms of radius-margin bound J
• Radius-margin bound not hold in L1-SVM.

4
Background and Objectives(3/4)

• Rule ranking, rule selection
• Rule base structure Setnes and Babuska 2001
• SVD-QR with column pivoting algorithm and pivoted
  QR decomposition method Yen and Wang 1998,1999,
  Setnes and Babuska 2001
• Contribution of fuzzy rule consequents
• More effective Setnes and Babuska 2001
• OLS Chen et al 1991
• Both rule base structure and contribution of
  fuzzy rule consequents
• Highly desired Setnes and Babuska 2001
• Not reported yet in literature.

5
Background and Objectives(4/4)

• Another objective
• -values of fuzzy rules
• Contribution of rule consequents
• -values of fuzzy rules
• Rule base structure and contribution of rule
  consequents.

6
L2-SVM based Fuzzy Classifier Construction (1/10)

• Fuzzy Classifier

7
L2-SVM based Fuzzy Classifier Construction (2/10)

• Conditions of Applying SVM to Fuzzy Classifier
  Construction
• are Mercer kernel
• If are generated from a reference
  function through location shift, then
  are Mercer kernel Chen and Wang 2003
• leading to
  Gaussian MFs
• Kernel parameters manually selected in Cheng and
  Wang 2003.

8
L2-SVM based Fuzzy Classifier Construction (3/10)

• L2-SVM based Fuzzy Classifier
• Parameters optimally updated in terms of
  radius-margin bound
• The number of rules L, prototypes , weights
  , bias , and scaling parameters .

9
L2-SVM based Fuzzy Classifier Construction (4/10)

• Two quadratic programming problems
• 1)
• st
• where
  are Lagrangian multipliers,

10
L2-SVM based Fuzzy Classifier Construction (5/10)

• 2)
• st
• Radius-margin bound

11
L2-SVM based Fuzzy Classifier Construction (6/10)

• Automatic Model Selection Algorithm

12
L2-SVM based Fuzzy Classifier Construction (7/10)

• Extraction Fuzzy Rules from L2-SVM Learning
  Results
• The number of fuzzy rules L is the number of
  support vectors
• The premise parts of fuzzy rules
• where is the jth element of the ith
  support vector .
• The consequent parts of fuzzy rules
• where are the non-zero Lagrangian
  multipliers.

13
L2-SVM based Fuzzy Classifier Construction (8/10)

• Fuzzy rule ranking based on L2-SVM learning
• R-values of fuzzy rules Setnes and
  Babuska 2001
• Absolute values of the diagonal elements of
  matrix R in the QR decomposition of firing
  strength matrix
• -values of fuzzy rules
• Determining the depth of the effect of the rule
  consequent.
• -values of fuzzy rules
• Considering both rule base structure and effect
  of the rule consequent.

14
L2-SVM based Fuzzy Classifier Construction (9/10)

• Fuzzy rule selection procedure
• Evaluate the misclassification rates (MRs) of
  on the validation data set V and the test
  data set T separately
• and
• Select the most influential fuzzy rules
• where is the threshold.
• Construct a fuzzy classifier by using
  the influential fuzzy rules selected.

15
L2-SVM based Fuzzy Classifier Construction (10/10)

• Fuzzy rule selection procedure (cont.)
• Apply to the validation data set V and
  the test data set T to obtain new MRs
  and
• If gt , stop selection
  otherwise, assign a higher threshold value and go
  to step 2.

16
Experimental Results(1/6)

• Benchmark problem-ringnorm
• 2 classes
• 7400 samples
• 20 attributes
• Theoretically expected MR 1.3 Breiman 1998
• 400 training samples 5000 testing samples 2000
  validation samples.
• Initial conditions
• C1
• Learning rates for updating C and 0.0001
  and 0.01 separately
• Threshold for updating the radius-margin bound

17
Experimental Results(2/6)

• L2-SVM Induced Fuzzy Classifier
• 249 fuzzy rules generated
• MR 1.32 on test data set
• Comparison with the well-known methods on
  generalization performance

Algorithms LDA QDA OLS-RBF with Gausian BFs OLS-RBF with Cauchy BFs MLP The proposed
MRs 24.54 2.6 2.52 3.12 13.0 1.32
18
Experimental Results(3/6)

• Fuzzy rule ranking results

19
Experimental Results(4/6)
20
Experimental Results(5/6)
21
Experimental Results(6/6)

• Fuzzy rule selection results

Using R-value index Using R-value index Using R-value index Using R-value index Using -value index Using -value index Using -value index Using -value index Using -value index Using -value index Using -value index Using -value index
No. of rules selected No. of rules selected No. of rules selected
0 249 1.45 1.32 0 249 1.45 1.32 0 249 1.45 1.32
0.001 242 1.45 1.32 0.001 90 1.45 1.32 0.0001 90 1.45 1.32
0.002 214 1.45 1.32 0.002 89 1.50 1.32 0.0006 89 1.45 1.32
0.003 193 1.80 1.5 0.005 88 1.55 1.38 0.0008 88 1.50 1.34
22
Conclusions and Discussions(1/1)

• To have applied L2-SVM to fuzzy rule induction
  for classification
• Fuzzy rules optimally generated in term of
  radius-margin bound.
• Efficient way of avoiding the curse of
  dimensionality in high dimensional space.
• Two novel indices for fuzzy rule ranking
• Experimentally proved to be very effective in
  producing parsimonious fuzzy classifiers.