Image Segmentation Jianbo Shi Robotics Institute Carnegie Mellon University

1
Image SegmentationJianbo ShiRobotics
InstituteCarnegie Mellon University
Cuts, Random Walks, and Phase-Space Embedding
Joint work with Malik,Malia,Yu
2
Taxonomy of Vision Problems

• Reconstruction
• estimate parameters of external 3D world.
• Visual Control
• visually guided locomotion and manipulation.
• Segmentation
• partition I(x,y,t) into subsets of separate
  objects.
• Recognition
• classes face vs. non-face,
• activities gesture, expression.

3
(No Transcript)
4
We see Objects
5
Outline

• Problem formulation
• Normalized Cut criterion algorithm
• The Markov random walks view of Normalized Cut
• Combining pair-wise attraction repulsion
• Conclusions

6
Edge-based image segmentation

• Edge detection by gradient operators
• Linking by dynamic programming, voting,
  relaxation,
• Montanari 71, ParentZucker 89, GuyMedioni 96,
  ShaashuaUllman 88
• WilliamsJacobs 95, GeigerKumaran 96,
  Heitgervon der Heydt 93
• - Natural for encoding curvilinear grouping
• - Hard decisions often made prematurely

7
Region-based image segmentation

• Region growing, split-and-merge, etc...
• - Regions provide closure for free, however,
• approaches are ad-hoc.
• Global criterion for image segmentation

• Markov Random Fields e.g. GemanGeman 84
• Variational approaches e.g. MumfordShah 89
• Expectation-Maximization e.g. AyerSawhney 95,
  Weiss 97

- Global method, but computational complexity
precludes exact MAP estimation - Problems due to
local minima
8
Bottom line It is hard, nothing worked well,
use edge detection, or just avoid it.
9
Global good, local bad.

• Global decision good, local bad
• Formulate as hierarchical graph partitioning
• Efficient computation
• Draw on ideas from spectral graph theory to
  define an eigenvalue problem which can be solved
  for finding segmentation.
• Develop suitable encoding of visual cues in terms
  of graph weights.

ShiMalik,97
10
Image segmentation by pairwise similarities

• Image pixels
• Segmentation partition of image into segments
• Similarity between pixels i and j
• Sij Sji 0

Sij

• Objective similar pixels should be in the same
  segment, dissimilar pixels should be in different
  segments

11
Segmentation as weighted graph partitioning

• Pixels i I vertices of graph G
• Edges ij pixel pairs with Sij gt 0
• Similarity matrix S Sij
• is generalized adjacency matrix
• di Sj Sij degree of i
• vol A Si A di volume of A I

i
Sij
j
12
Cuts in a graph

• (edge) cut set of edges whose removal makes a
  graph disconnected
• weight of a cut
• cut( A, B ) Si A,j B Sij
• the normalized cut

NCut( A,B ) cut( A,B )( )
1 . vol A
1 . vol B
13
Normalized Cut and Normalized Association

• Minimizing similarity between the groups, and
  maximizing similarity within the groups can be
  achieved simultaneously.

14
The Normalized Cut (NCut) criterion

• Criterion
• min NCut( A,A )
• Small cut between subsets of balanced grouping

NP-Hard!
15
Some definitions
16
Normalized Cut As Generalized Eigenvalue problem

• Rewriting Normalized Cut in matrix form

17
More math
18
Normalized Cut As Generalized Eigenvalue problem

• after simplification, we get

19
Interpretation as a Dynamical System
20
Interpretation as a Dynamical System
21
Brightness Image Segmentation
22
brightness image segmentation
23
Results on color segmentation
24
Malik,Belongie,Shi,Leung,99
25
(No Transcript)
26
Motion Segmentation with Normalized Cuts

• Networks of spatial-temporal connections

27
Motion Segmentation with Normalized Cuts

• Motion proto-volume in space-time
• Group correspondence

28
Results
29
Results

• ShiMalik,98

30
Results
31
Results
32
Stereoscopic data
33
Conclusion I

• Global is good, local is bad
• Formulated Ncut grouping criterion
• Efficient Ncut algorithm using generalized
  eigen-system
• Local pair-wise allows easy encoding and
  combining of Gestalt grouping cues

34
FAQs

• Why is this segmentation criterion better?
• 30
• Is there a good way of picking W(i,j)
• 40
• How do we integrate prior information?
• how is higher level information encoded?
• - 20

35
Learning Segmentation

• Learning Segmentation with Random Walk

Maila Shi, NIPS 00
36
Goals of this work

• Better understand why spectral segmentation works
• random walks view for NCut algorithm
• complete characterization of the ideal case
• ideal case is more realistic/general than
  previously thought
• Learning feature combination/object shape model
• Max cross-entropy method for learning

MaliaShi,00
37
The random walks view

• Construct the matrix
• P D-1S
• D S
• P is stochastic matrix Sj Pij 1
• P is transition matrix of Markov chain with state
  space I
• p d1 d2 . . . dn T is stationary
  distribution

S11 S12 S1n S21 S22 S2n . . . Sn1 Sn2
Snn
d1 d2 . . .
dn
1 . vol I
38
Reinterpreting the NCut criterion

• NCut( A, A ) PAA PAA
• PAB Pr A --gt B A under P, p
• NCut looks for sets that trap the random walk
• Related to Cheeger constant, conductivity in
  Markov chains

39
Reinterpreting the NCut algorithm
Px lx l11 l2 . . . ln x1
x2 . . . xn

• (D-W)y mDy
• m10 m2 . . . mn
• y1 y2 . . . Yn

mk 1 - lk yk xk
The NCut algorithm segments based on the second
largest eigenvector of P
40
So far...

• We showed that the NCut criterion its
  approximation the NCut algorithm have simple
  interpretations in the Markov chain framework
• criterion - finds almost ergodic sets
• algorithm - uses x2 to segment
• Now
• Will use Markov chain view to show when the NCut
  algorithm is exact, i.e. when P has K piecewise
  constant eigenvectors

41
Piecewise constant eigenvectors Examples

• Block diagonal P (and S)

Eigenvectors
Eigenvalues
S

• Equal rows in each block

Eigenvectors
P
Eigenvalues
42
Piecewise constant eigenvectors general case

• TheoremMeilaShi Let P D-1S with D
  non-singular and let D be a partition of I.
  Then P has K piecewise constant eigenvectors
  w.r.t D iff P is block stochastic w.r.t D and P
  non-singular.

Eigenvectors
P
Eigenvalues
43
Block stochastic matrices

• D ( A1, A2, . . . AK ) partition of I
• P is block stochastic w.r.t D
• Sj As Pij Sj As Pij for i,i in same
  segment As
• Intuitively Markov chain can be aggregated so
  that random walk over D is Markov
• P transition matrix of Markov chain over D

44
Learning image segmentation

• Targets Pij for i in
  segment A
• Model Sij exp( Sq lqfqij )

45
The objective function

• J - Si I Sj I Pij log Pij
• J KL( P P )
• where p0 and Pi j p0 Pij the flow i
  j
• Max entropy formulation maxPij H( j i )
• s.t.
  ltfijqgtp0Pij ltfijqgtp0Pij for all q
• Convex problem with convex constraints at
  most one optimum
• The gradient ltfijqgt p0Pij -
  ltfijqgt p0Pij

1 . I
1234567890Qwertyuiop Asdfghjkl zxcvbnm,./ -\

1 . I
46
Experiments - the features
i

• IC - intervening contour
• fijIC max Edge( k )
• k (i,j)
• Edge( k ) output of edge filter for pixel k
• Discourages transitions across edges
• CL - colinearity/cocircularity
• fijCL
  
• Encourages transitions along flow lines
• Random features

k
j
Edgeness
2-cos(2ai)-cos(2aj) 1 - cos(al)
2-cos(2ai aj) 1 - cos(a0)
ai
j
i
aj
orientation
47
Training examples
lCL
lIC
48
Test examples

• Original Image Edge Map
  Segmentation

49
Conclusion II

• Showed that the NCut segmentation method has a
  probabilistic interpretation
• Introduced
• a principled method for combining features in
  image segmentation
• supervised learning and synthetic data to find
  the optimal combination of features

Graph Cuts
Generalized Eigen-system
Markov Random Walks
50
(No Transcript)
51
(No Transcript)
52
Summary

• Grouping is a global process
• Normalized Cuts formalism provides efficient
  algorithm based on spectral graph theory
• Grouping is from multiple cues
• Learning Segmentation with Random Walks