Graph Based Multi-Modality Learning

1
Graph Based Multi-Modality Learning

• Hanghang Tong Jingrui He
• Carnegie Mellon University
• Mingjing Li
• Microsoft Research Asia

2
Outline

• Motivation
• Graph-based Semi-supervised learning
• Methods
• The Linear Fusion Scheme
• The Sequential Fusion Scheme
• Justifications
• Similarity Propagation
• Bayesian Interpretation
• Graph-based un-supervised learning
• Experimental Results
• Conclusion

3
Motivation

• Multi-Modality in Multimedia
• Video
• Digital Image color vs.
• Web Image content vs. surrounding text
• Traditional methods
• Linear combination super-kernel
• Co-Training
• Multi-view version of EM, DBSCAN

All Vector Model based !
4
Motivation (conts)

• Two Key issues
• How to learning within each modality
• How to combine
• A more recent hot topic
• Graph-based learning
• Spectral Cluster Eigen Map
• Manifold Ranking
• Explore graph-based method in the context of
  multi-modality!

5
Notation

• n data points
• c classes,
• Two modalities a and b
• One Affinity Matrix for each modality
• for modality a (nxn)
• for modality b (nxn)
• Labeling Matrix (nxc)
• Vectorial Function (nxc)
• Learning task
• s

6
Basic Idea

• What is a good vectorial function F?
• As consistent as possible with
• Info from modality a
• Info from modality b
• Info from Labeled points Y
• How to?
• Take into account the various constraints by
  optimization

7
Linear Fusion Scheme
Constrains. from modality b
Constrains. from modality a

• Optimization strategy
• Optimization Solution
• Iterative form
• Closed form

Constrains. from Labels Y
Converge
8
Sequential Fusion Scheme
Constrains. from modality a and Y

• Optimization strategy
• Optimization Solution
• Iterative form
• Closed form

Constrains. from modality b and
Converge
9
Similarity Propagation

• Taylor expansion (linear fusion)
• Similar result for sequential form

Initial Label
gt Further propagate similarity by a and b gt Fuse
the result by weighted sum
gt Propagate Y by a and b gt Fuse the result by
weighted sum
10
Bayesian Interpretation

• Optimal F by MAP (linear form)
• Assuming
• Conditional pdf
• Prior by modality a
• Prior by modality b
• Fuse prior by
• The above setting leads to

11
Extension to Un-Supervised Case

• Compare
• For one modality
• For two modalities (linear form)
• Graph Laplacian Learning
• Linear Form
• Sequential From
• Feed it the spectral cluster or embedding

Quite similar !
Independent on Y !
12
Experimental Results Coral Image
Sequential Form
Linear Form
Treat as one modality
Color Hist
Texture
13
Experimental Results Web Image
Linear Form
Sequential Form
Treat as one modality
Content Fea
Surrounding Text
14
Conclusion

• Study multi-modality learning by graph based
  method
• Propose two schemes for semi-supervised learning
• Extend them to un-supervised learning

15
QA

• The End
• Thanks