Integrating User Feedback Log into Relevance Feedback by Coupled SVM for Content-Based Image Retrieval

1
Integrating User Feedback Log into Relevance
Feedback by Coupled SVM for Content-Based Image
Retrieval

• 9-April, 2005
• Steven C. H. Hoi , Michael R. Lyu , Rong Jin
  
• Department of Computer Science Engineering
• The Chinese University of Hong Kong
• Shatin, N.T., Hong Kong SAR
• Department of Computer Science and Engineering
• Michigan State University
• East Lansing, MI 48824, USA
• The 1st IEEE EMMA Workshop
• in conjunction with 21st IEEE ICDE, Japan, April,
  2005.

2
Outline

• Introduction
• Background
• Log-based Relevance Feedback
• Coupled Support Vector Machine
• Support Vector Machine
• Formulation
• Alternating Optimization
• A Practical Algorithm
• Experimental Results
• Conclusion

3
Introduction

• Content-based Image Retrieval (CBIR)
• An important component in visual information
  retrieval
• QBE query-by-example based on low-level visual
  features
• Semantic gap low-level features, high-level
  concepts

QBE
4
Introduction

• Relevance Feedback (RF)
• A powerful tool to attack the semantic gap
  problem
• Interactive mechanism to solicit users feedbacks
• Boost the retrieval performance of CBIR greatly
• Many existing techniques already
• Problems
• Regular relevance feedback needs too many rounds
  of interactions for achieving satisfactory
  results.

5
Introduction

• Motivation

Relevance Feedback
?
User Feedback Log
Can user feedback log be used to improve the
regular relevance feedback?
Problem
6
Background

• Log-based Relevance Feedback (LRF)
• Relevance Matrix R
• RF round / Log session Nl images are marked
• Elements relevant (1), irrelevant (-1), unknown
  (0)

Image samples
-1
-1
1
1
1
-1
-1
0
1
-1
1
-1
1
-1
-1
-1
-1
-1
0
-1
-1
1
Log Sessions
7
Background

• Learning Problem for LRF
• Low-level image content
• User feedback log
• Multi-Modal Learning Problem

8
Coupled Support Vector Machine

• Motivation
• How to attack the learning problem on the two
  modalities?
• Low-level Image content X
• User relevance feedback log R
• Support Vector Machines superior classification
  performance
• A Straightforward Solution
• Learn an SVM classifier on each modality
  respectively
• For image content X, we learn an optimal
  weighting vector w
• For log content R, we learn an optimal weighting
  vector u
• Combine their results together linearly

9
Coupled Support Vector Machine

• A Straightforward Solution
• For the image content modality wTx
• For the user feedback log modality uTr

10
Coupled Support Vector Machine

• Disadvantages of the straightforward solution
• Linear combination
• Modality Consistence
• Our better solution Coupled SVM
• Learn the two modalities in a unified formulation
• Enforce the prediction on the two types of
  information to be consistent.

11
Coupled Support Vector Machine

• Formulation Coupled SVM

12
Coupled Support Vector Machine

• Optimization of Coupled SVM
• Hard to be solved directly
• Alternating Optimization (AO)
• AO two-step optimization
• Fix Y, try to find (u, b_u), and (w, b_w)
• Fix (u, b_u) and (w, b_w), try to find Y

13
Coupled Support Vector Machine

• Alternating Optimization
• Fix Y, the primal optimization is equivalent to
  solving the two optimization subproblems

14
Coupled Support Vector Machine

• Alternating Optimization (AO)
• By introducing non-negative Lagrange multipliers,
  the above two subproblems can be solved

15
Coupled Support Vector Machine

• Alternating Optimization (AO)
• After solving (u, b_u) and (w, b_w), fixing them,
  the optimal Y can be found to fit the data as
  follows

16
Coupled Support Vector Machine

• Summary of AO procedure
• 1) Beginning with a small value of
• 2) Performing the two-step AO procedure
• 3) Repeating 2) by increasing until it
  achieves the setting threshold
• Comments on the Coupled SVM
• Can be a general approach for multi-modal
  learning problems
• Need to investigate the convergence issue of
  Alternating Optimization
• Need to study better methods for solving the
  optimization problem
• Require to take some practical considerations
  when fitting for specific problems.

17
Coupled Support Vector Machine

• A Practical Algorithm
• Practical considerations
• Cannot engage all unlabeled samples due to
  response requirement for relevance feedback
• Strategy for choosing unlabeled samples
• Closest to the decision boundary of SVM most
  informative according to active learning
• Closest to the labeled samples to avoid too much
  effort in learning the label information
• Introducing a parameter to control the
  error for label correction to avoid overlarge
  change in the labeled set

18
Coupled Support Vector Machine

• A Practical Algorithm (contd)

19
Coupled Support Vector Machine

• A Practical Algorithm (contd)

20
Experimental Results

• Dataset
• Images selected from COREL image CDs
• Two ground-truth datasets
• 20-Category each category contains 100 images,
  totally 2,000
• 50-Category each category contains 100 images,
  totally 5,000

21
Experimental Results (contd)

• Low-level Image Representation
• Color Moment
• 9-dimension
• Edge Direction Histogram
• 18-dimension
• Canny detector, 18 bins of 20 degrees each
• Wavelet-based texture
• 9-dimension
• Daubechies-4 wavelet, 3-level DWT
• Entropies of 9 subimages are generated for the
  texture feature

22
Experimental Results (contd)

• Collection of User Log Data
• Log format
• A log session (LS) corresponds a relevance
  feedback round
• Each log session contains 20 images labeled by
  users
• Log data
• On 20-Category 161 log sessions
• On 50-Category 184 log sessions

23
Experimental Results (contd)

• CBIR GUI for collecting feedback data

24
Experimental Results (contd)

• Performance Evaluation
• Measurement Metric
• Average Precision relevant images /
  returned images
• Experimental Setting
• 100 queries
• 20 initially labeled images
• SVM RBF kernel, parameters set via training data
• Comparison Schemes
• RF-SVM
• traditional relevance feedback by SVM
• LRF-2SVM
• log-based relevance feedback by learning two SVMs
  respectively
• LRF-CSVM
• log-based relevance feedback by Coupled SVM

25
Experimental Results (contd)

• Performance Evaluation on 20-Category Dataset

26
Experimental Results (contd)

• Performance Evaluation on 50-Category Dataset

27
Experimental Results (contd)
28
Experimental Results (contd)
29
Conclusion

• A log-based relevance feedback scheme was studied
  by integrating user feedback log into the content
  learning of low-level visual features in
  content-based image retrieval.
• A general multimodal learning technique, i.e.
  Coupled Support Vector Machine, was proposed for
  studying the data with multiple modalities.
• A practical algorithm by Coupled SVM was
  presented to attack the log-based relevance
  feedback problem in CBIR.
• Experimental results show our proposed scheme is
  effective for the log-based relevance feedback
  problem.

30
QA
31
References

• Chu-Hong Hoi and Michael R. Lyu, A Novel
  Log-based Relevance Feedback Technique in
  Content-based Image Retrieval, in Proc. ACM
  Multimedia, New York, USA, 10-16 October, pp.
  24-31, 2004
• S. Tong and E. Chang. Support vector machine
  active learning for image retrieval. In Proc. ACM
  Multimedia, pages 107--118, 2001.