Learning Jigsaws for clustering appearance and shape

1
Learning Jigsawsfor clustering appearance and
shape

• John Winn, Anitha Kannan and Carsten Rother

NIPS 2006
2
Learning jigsaws
Aim Cluster regions in images with similar
appearance and shape.
3
Road map

• Clustering image patches
• The Jigsaw model
• Results on toy and real images
• Learning jigsaw pieces
• Discussion and conclusions

4
Clustering image patches
5
Clustering image patches
Cluster?
6
Clustering image patches
Cluster?
7
Clustering image patches
Cluster?
8
Clustering image patches
Cluster?
Need to adapt the patch shape depending on the
image.
9
Road map

• Clustering image patches
• The Jigsaw model
• Results on toy and real images
• Learning jigsaw pieces
• Discussion and conclusions

10
Aims of jigsaw model

• Learn clusters (jigsaw pieces) so that
• Clustered patches have similar shape and
  appearance
• Patches are as large as possible
• Every image pixel belongs to exactly one patch
  (i.e. the images are segmented into patches)

11
The Jigsaw model
Jigsaw J
...
12
The Jigsaw model
Jigsaw J
Appearance model
Offset map prior (Potts model)
13
Road map

• Clustering image patches
• The Jigsaw model
• Results on toy and real images
• Learning jigsaw pieces
• Discussion and conclusions

14
Toy example
Image
with segmentation
Learned by iteratively maximising joint
probability w.r.t. jigsaw and offset maps (see
paper for details)
15
Comparison Mixture of Gaussians
Cluster centres

• fixed patch shape

16
Comparison Epitome
Epitome

• fixed patch shape
• translation invariant

Jojic et al., ICCV 2003
17
Comparison Jigsaw
Jigsaw

• learned patch shape
• translation invariant
• non-overlapping patches

18
Comparison all methods
MoG
Original
Error 0.103
Jigsaw
Epitome
Error 0.054
Error 0.071
19
Faces example
Face images
with segmentations
Source Olivetti face database
20
Road map

• Clustering image patches
• The Jigsaw model
• Results on toy and real images
• Learning jigsaw pieces
• Discussion and conclusions

21
Learning the jigsaw pieces
Jigsaw J
...
22
Learning the jigsaw pieces
Jigsaw J
...
23
Learning the jigsaw pieces
Jigsaw J
...
24
Shape clustering on faces
Jigsaw showing pieces
25
Road map

• Clustering image patches
• The Jigsaw model
• Results on toy and real images
• Learning jigsaw pieces
• Discussion and conclusions

26
Jigsaw applications

• Can be used as plug-and-play replacement for
  fixed-shape patch model in existing systems.

• Applications include
• Object recognition/detection
• Object segmentation
• Stereo matching
• Texture synthesis
• Super-resolution
• Motion segmentation
• Image/video compression

27
Future work

• Allow rotation/scaling/deformationof the
  patches.
• Incorporate shape clustering into the
  probabilistic model
• Incorporate additional invariances e.g. to
  illumination
• Apply to other domains audio, biology

28
Conclusions

• Jigsaw model allows learning the shape and
  appearance of recurring regions in images.
• Jigsaw performs unsupervised discovery of object
  parts.

29
Thank you
http//johnwinn.org
30
(No Transcript)
31
Comparison Epitome
Epitome

• fixed patch shape
• translation invariant
• overlapping patches

Jojic et al., ICCV 2003
32
Patch averaging
Epitome
MoG
Error 0.071
Error 0.054