Simultaneous Point Matching and Recovery of Rigid and Nonrigid Shapes

1
Simultaneous Point Matching andRecovery of Rigid
and Nonrigid Shapes
THESIS pROPOSAL
Thesis director Francesc Moreno
Noguer Tutor Alberto Sanfeliu Cortés
2
Objective
Simultaneously solve the correspondence problem
and recover rigid and nonrigid shapes.
Given two point clouds extracted from different
views of the same object, the objective is to
simultaneously solve for point correspondence and
recover the mapping between the two rigid or
nonrigid model representations.
3
Motivation

• Given um from a model point set and vt from a
  target point set, find

model point set
target point set
4
Motivation

• Given um from a model point set and vt from a
  target point set, find

CORRESPONDENCES
um
vt
5
Motivation

• Given um from a model point set and vt from a
  target point set, find

CORRESPONDENCES
um
vt
TRANSFORM ESTIMATION
Hest
6
Motivation

• Given um from a model point set and vt from a
  target point set, find

CORRESPONDENCES
um
vt
TRANSFORM ESTIMATION
Hest
7
Motivation

• Common problems

1. Outliers
2. Occlusions
3. Clutteredbackground
4. Observationnoise
5. Repetitivepatterns
6. Rigid / Nonrigidmodel
7. 2D/3D or2D-2D / 3D-3D

um
vt
correspondences that do not fit the model
8
Motivation

• Common problems

1. Outliers
2. Occlusions
3. Clutteredbackground
4. Observationnoise
5. Repetitivepatterns
6. Rigid / Nonrigidmodel
7. 2D/3D or2D-2D / 3D-3D

um
vt
partial matching parts of the scene are occluded
9
Motivation

• Common problems

1. Outliers
2. Occlusions
3. Clutteredbackground
4. Observationnoise
5. Repetitivepatterns
6. Rigid / Nonrigidmodel
7. 2D/3D or2D-2D / 3D-3D

um
vt
points that do not belong to the model and
hinder the recognition
10
Motivation

• Common problems

1. Outliers
2. Occlusions
3. Clutteredbackground
4. Observationnoise
5. Repetitivepatterns
6. Rigid / Nonrigidmodel
7. 2D/3D or2D-2D / 3D-3D

um
vt
error in the position of points
11
Motivation

• Common problems

1. Outliers
2. Occlusions
3. Clutteredbackground
4. Observationnoise
5. Repetitivepatterns
6. Rigid / Nonrigidmodel
7. 2D/3D or2D-2D / 3D-3D

um
vt
regular structures are indistinguishable
algorithms fall into local minima
12
Motivation

• Common problems

1. Outliers
2. Occlusions
3. Clutteredbackground
4. Observationnoise
5. Repetitivepatterns
6. Rigid / Nonrigidmodel
7. 2D/3D or2D-2D / 3D-3D

um
vt
model shapes can undergo rigid or nonrigid
deformations
13
Motivation

• Common problems

1. Outliers
2. Occlusions
3. Clutteredbackground
4. Observationnoise
5. Repetitivepatterns
6. Rigid / Nonrigidmodel
7. 2D/3D or2D-2D / 3D-3D

um
vt
model shapes can undergo rigid or nonrigid
deformations
14
Motivation

• Common problems

1. Outliers
2. Occlusions
3. Clutteredbackground
4. Observationnoise
5. Repetitivepatterns
6. Rigid / Nonrigidmodel
7. 2D/3D or2D-2D / 3D-3D

um
vt
Coordinate System 1
Coordinate System 2
the transform can be embedded in 2D or 3D or
projective as in monocular view case (2D-3D)
15
Motivation

• Common problems

1. Outliers
2. Occlusions
3. Clutteredbackground
4. Observationnoise
5. Repetitivepatterns
6. Rigid / Nonrigidmodel
7. 2D/3D or2D-2D / 3D-3D

16
Motivation

• Common problems

1. Outliers
2. Occlusions
3. Clutteredbackground
4. Observationnoise
5. Repetitivepatterns
6. Rigid / Nonrigidmodel
7. 2D/3D or2D-2D / 3D-3D

Camera Coordinate System
World Coordinate System
the transform can be embedded in 2D or 3D or
projective as in monocular view case (2D-3D)
17
Motivation

• Common problems

1. Outliers
2. Occlusions
3. Clutteredbackground
4. Observationnoise
5. Repetitivepatterns
6. Rigid / Nonrigidmodel
7. 2D/3D or2D-2D / 3D-3D

Camera Coordinate System
World Coordinate System
xcam A R t xworld
18
State-of-Art
2D-3D
2D-2D or 3D-3D
Rigid Model
Rigid 2D-2D / 3D-3D
Rigid 2D-3D
Nonrigid Model
Nonrigid 2D-3D
Nonrigid 2D-2D / 3D-3D
19
State-of-Art
2D-3D
2D-2D or 3D-3D
Rigid Model
Nonrigid Model
20
State-of-Art
RIGID 2D-2D / 3D-3D MATCHING
1981 Fischler Bolles. Random Sample
Consensus A Paradigm for Model Fitting with
Applications to Image Analysis and Automated
Cartography
RANSAC (global solution / high complexity)
21
State-of-Art
RIGID 2D-2D / 3D-3D MATCHING
1981 Fischler Bolles. Random Sample
Consensus A Paradigm for Model Fitting with
Applications to Image Analysis and Automated
Cartography
1992 Besl McKay, A Method for Registration of
3D Shapes
ICP Iterative Closest Point (requires good
initialization)
22
State-of-Art
RIGID 2D-2D / 3D-3D MATCHING
1981 Fischler Bolles. Random Sample
Consensus A Paradigm for Model Fitting with
Applications to Image Analysis and Automated
Cartography
1992 Besl McKay, A Method for Registration of
3D Shapes
2005 Chum Matas. Matching With PROSAC -
Progressive Sample Consensus
PROSAC RANSAC Appearance
23
State-of-Art
RIGID 2D-2D / 3D-3D MATCHING

• Unsolved when
• Weak detection,
• outliers,
• occlusions,
• image noise,
• repetitive patterns,
• highly textured scenes,
• oblique angles
• PROSAC complexity similar to RANSAC (too high)

1981 Fischler Bolles. Random Sample
Consensus A Paradigm for Model Fitting With
Applications to Image Analysis and Automated
Cartography
1992 Besl McKay, A Method for Registration of
3D Shapes
2005 Chum Matas. Matching With PROSAC -
Progressive Sample Consensus
24
State-of-Art
2D-3D
2D-2D or 3D-3D
Rigid Model
Nonrigid Model
25
State-of-Art
RIGID 2D-3D MATCHING
1970
1980
1990
Known correspondences Perspective-n-Point
(PnP) (old problem)
2000
2010
2009 Moreno-Noguer et al, EPnP An Accurate
O(n) Solution to the PnP Problem
26
State-of-Art
RIGID 2D-3D MATCHING
1970
1981 Fischler Bolles. Random Sample
Consensus A Paradigm for Model Fitting With
Applications to Image Analysis and Automated
Cartography
1980
RANSAC Random Sampled Consensus DLT Direct
Linear Transform (high complexity)
1990
2000
2010
2009 Moreno-Noguer et al, EPnP An Accurate
O(n) Solution to the PnP Problem
27
State-of-Art
RIGID 2D-3D MATCHING
1970
1981 Fischler Bolles. Random Sample
Consensus A Paradigm for Model Fitting With
Applications to Image Analysis and Automated
Cartography
1980
1990
SoftPOSIT Unknown correspondences
2002 David et al,SoftPOSIT Simultaneous Pose
and Correspondence Determination
2000
2010
2009 Moreno-Noguer et al, EPnP An Accurate
O(n) Solution to the PnP Problem
28
State-of-Art
RIGID 2D-3D MATCHING
1970
1981 Fischler Bolles. Random Sample
Consensus A Paradigm for Model Fitting With
Applications to Image Analysis and Automated
Cartography
1980
Blind PnP Unknown correspondences Pose Priors
(geometrical consistency) Kalman Filter to
propagate pose uncertainty
1990
2002 David et al,SoftPOSIT Simultaneous Pose
and Correspondence Determination
2000
2008 Moreno-Noguer et al, Pose Priors for
Simultaneously Solving Alignment and
Correspondence.
2010
2009 Moreno-Noguer et al, EPnP An Accurate
O(n) Solution to the PnP Problem
29
State-of-Art
RIGID 2D-3D MATCHING
1970

• Modeling the uncertainty
• Kalman Filter linearizes the uncertainty model
• Work with Bayesian non-parametric models
  (Gaussian Processes)

1981 Fischler Bolles. Random Sample
Consensus A Paradigm for Model Fitting With
Applications to Image Analysis and Automated
Cartography
1980
1990
2002 David et al,SoftPOSIT Simultaneous Pose
and Correspondence Determination
2000
2008 Moreno-Noguer et al, Pose Priors for
Simultaneously Solving Alignment and
Correspondence.
2010
2009 Moreno-Noguer et al, EPnP An Accurate
O(n) Solution to the PnP Problem
30
State-of-Art
2D-3D
2D-2D or 3D-3D
Rigid Model
Nonrigid Model
31
State-of-Art
NONRIGID 2D-2D / 3D-3D MATCHING
Soft-assign (requires good initialization)
1970
1980
1998 Gold et al.. New Algorithms for 2D and 3D
Point Matching Point Estimation and
Correspondence
1990
2003 Chui Rangarajan. A New Point Matching
Algorithm for Non-Rigid Registration
2000
Soft-assign Thin-plate splines (requires good
initialization) (smooth deformations)
2010
32
State-of-Art
NONRIGID 2D-2D / 3D-3D MATCHING
1970
1980
1998 Gold et al.. New Algorithms for 2D and 3D
Point Matching Point Estimation and
Correspondence
1990
2003 Hannel et al.. An Extension of the ICP
Algorithm for Modeling Nonrigid Objects with
Mobile Robots
2003 Chui Rangarajan. A New Point Matching
Algorithm for Non-Rigid Registration
2000
2008 Li et al, Global Correspondence
Optimization for Non-Rigid Registration of
Depth Scans
2010
Nonrigid ICP Variants (Require a good
initialization)
33
State-of-Art
NONRIGID 2D-2D / 3D-3D MATCHING
Graph Matching Thin-plate splines (Smooth
deformations)
1970
Shape appearance Thin-plate splines (Smooth
deformations)
1980
1998 Gold et al.. New Algorithms for 2D and 3D
Point Matching Point Estimation and
Correspondence
2002 Belongui, Shape matching and Object
Recognition Using Shape Contexts
1990
2003 Hannel et al.. An Extension of the ICP
Algorithm for Modeling Nonrigid Objects with
Mobile Robots
2003 Chui Rangarajan. A New Point Matching
Algorithm for Non-Rigid Registration
2000
2008 Li et al, Global Correspondence
Optimization for Non-Rigid Registration of
Depth Scans
2010
2010 Deng et al, Retinal Fundus Image
Registration via Vascular Structure Graph Matching
34
State-of-Art
NONRIGID 2D-2D / 3D-3D MATCHING
1970

• Unsolved problems
• Harsh deformations (Gaussian Processes vs TPS)
• Avoid local minima (Global Search vs ICP et al.)

1980
1998 Gold et al.. New Algorithms for 2D and 3D
Point Matching Point Estimation and
Correspondence
2002 Belongui, Shape matching and Object
Recognition Using Shape Contexts
1990
2003 Hannel et al.. An Extension of the ICP
Algorithm for Modeling Nonrigid Objects with
Mobile Robots
2003 Chui Rangarajan. A New Point Matching
Algorithm for Non-Rigid Registration
2000
2008 Li et al, Global Correspondence
Optimization for Non-Rigid Registration of
Depth Scans
2010 Myronenko Song, Point-Set Registration
Coherent Point Drift
2010
2010 Deng et al, Retinal Fundus Image
Registration via Vascular Structure Graph Matching
35
State-of-Art
2D-3D
2D-2D or 3D-3D
Rigid Model
Nonrigid Model
36
State-of-Art
NONRIGID 2D-3D MATCHING
Deformable surfaces
Articulated structures
1970
1970
1980
1980
1990
1990
2000
2000
2010
2010
37
State-of-Art
NONRIGID 2D-3D MATCHING
Deformable surfaces
Articulated structures
1970
1970
2003 Shakhnarovich et al., Fast Pose Estimation
with Parameter Sensitive Hashing
2006 Sigal Black, Humaneva Synchronized
Video and Motion Capture Dataset for Evaluation
of Articulated Human Motion
1980
1980
1990
1990
Discriminative methods Database learning
Nearest Neigbour Selection
2000
2000
2010
2010
38
State-of-Art
NONRIGID 2D-3D MATCHING
Deformable surfaces
Articulated structures
1970
1970
2003 Shakhnarovich et al., Fast Pose Estimation
with Parameter Sensitive Hashing
2006 Sigal Black, Humaneva Synchronized
Video and Motion Capture Dataset for Evaluation
of Articulated Human Motion
1980
1980
2007 Salzmann et al., Surface
Deformation Models for Non-Rigid 3D Shape Recovery
1990
1990
2000
2000
2010 Sanchez et al., Simultaneous Pose,
Correspondence and Non-Rigid Shape
Generative methods PCA model of the surface
2010
2010
39
State-of-Art
NONRIGID 2D-3D MATCHING
Deformable surfaces
Articulated structures
1970
1970
2003 Shakhnarovich et al., Fast Pose Estimation
with Parameter Sensitive Hashing
2006 Sigal Black, Humaneva Synchronized
Video and Motion Capture Dataset for Evaluation
of Articulated Human Motion
1980
1980
2007 Salzmann et al., Surface
Deformation Models for Non-Rigid 3D Shape Recovery
1990
1990
Thin-plate Splines (Medical Imaging!)
2000
2000
2010 Sanchez et al., Simultaneous Pose,
Correspondence and Non-Rigid Shape
2010
2010
2009 Groher, Deformable 2D-3D Registration of
Vascular Structures in a One View Scenario
40
State-of-Art
NONRIGID 2D-3D MATCHING
Deformable surfaces
Articulated structures
1970
1970
2003 Shakhnarovich et al., Fast Pose Estimation
with Parameter Sensitive Hashing
2006 Sigal Black, Humaneva Synchronized
Video and Motion Capture Dataset for Evaluation
of Articulated Human Motion
1980
1980
2007 Salzmann et al., Surface
Deformation Models for Non-Rigid 3D Shape Recovery
1990
1990
Combining Discriminative Generative methods
2000
2000
2010 Sanchez et al., Simultaneous Pose,
Correspondence and Non-Rigid Shape
2010 Salzmann Urtasun, Combining
Discriminative and Generative Methods for 3D
Deformable Surface and Articulated Pose
Reconstruction
2010
2010
2009 Groher, Deformable 2D-3D Registration of
Vascular Structures in a One View Scenario
41
State-of-Art
NONRIGID 2D-3D MATCHING
Deformable surfaces
Articulated structures
1970
1970
2003 Shakhnarovich et al., Fast Pose Estimation
with Parameter Sensitive Hashing
2006 Sigal Black, Humaneva Synchronized
Video and Motion Capture Dataset for Evaluation
of Articulated Human Motion

• Potential improvements
• Better parameterization for articulated
  structures
• Harsh deformations (Gaussian Processes vs TPS)

1980
1980
2007 Salzmann et al., Surface
Deformation Models for Non-Rigid 3D Shape Recovery
1990
1990
2000
2000
2010 Sanchez et al., Simultaneous Pose,
Correspondence and Non-Rigid Shape
2010 Salzmann Urtasun, Combining
Discriminative and Generative Methods for 3D
Deformable Surface and Articulated Pose
Reconstruction
2010
2010
2009 Groher, Deformable 2D-3D Registration of
Vascular Structures in a One View Scenario
42
Contributions
2D-3D
2D-2D or 3D-3D
Rigid Model
Nonrigid Model
43
Contributions

• Rigid registration

Homography (2D-to-2D) estimation
44
Contributions

• Rigid registration
• Feature Point Detectors assigns multiple
  correspondences
• PROSAC picks just the one with better similarity
  score !!!

Homography (2D-to-2D) estimation
45
Contributions

• Rigid registration
• Feature Point Detectors assigns multiple
  correspondences
• Using Kalman Filter we can propagate geometry
  priors,thus constraining the search regions for
  each feature point

Homography (2D-to-2D) estimation
46
Contributions

• Rigid registration
• Feature Point Detectors assigns multiple
  correspondences
• Using Kalman Filter we can propagate geometry
  priors,thus constraining the search regions for
  each feature point
• Iterative approach

Homography (2D-to-2D) estimation
47
Contributions

• Rigid registration
• Feature Point Detectors assigns multiple
  correspondences
• Using Kalman Filter we can propagate geometry
  priors,thus constraining the search regions for
  each feature point
• Iterative approach

Homography (2D-to-2D) estimation
48
Contributions

• Rigid registration
• Feature Point Detectors assigns multiple
  correspondences
• Using Kalman Filter we can propagate geometry
  priors,thus constraining the search regions for
  each feature point
• Iterative approach

Homography (2D-to-2D) estimation
49
Contributions

• Rigid registration
• Feature Point Detectors assigns multiple
  correspondences
• Using Kalman Filter we can propagate geometry
  priors,thus constraining the search regions for
  each feature point
• Iterative approach backtracking when necessary

Homography (2D-to-2D) estimation
50
Contributions

• Rigid registration
• Some results

PROSAC
Blind Homography
51
Contributions

• Rigid registration
• E. Serradell, M. Ozuysal, V. Lepetit, P. Fua and
  F. Moreno-Noguer Combining Geometric and
  Appearance Priors for Robust Homography
  Estimation . In ECCV 2010

Homography (2D-to-2D) estimation
52
Contributions
2D-3D
2D-2D or 3D-3D
Rigid Model
Nonrigid Model
53
Contributions

• Nonrigid registration
• Nonrigid 2D-to-3D registration
• Project in collaboration with
• New parameterization for articulated models

2D X-ray Image
CT 3D Volume
54
Contributions

• Nonrigid registration
• Recursive parameterization of the nodes of the
  articulated structure

Camera (known)
CT 3D Volume
2D X-ray Image
55
Contributions

• Nonrigid registration
• Recursive parameterization of the nodes of the
  articulated structure
• Generative model Probabilistic PCA

Camera (known)
synthetic samples
2D features
56
Contributions

• Nonrigid registration
• Recursive parameterization of the nodes of the
  articulated structure
• Generative model Probabilistic PCA

Camera (known)
generative model
2D features
57
Contributions

• Nonrigid registration
• Recursive parameterization of the nodes of the
  articulated structure
• Generative model Probabilistic PCA
• Iterative update
• 1.- Kalman Filter model projection
• 2.- Assign correspondences

Camera (known)
generative model
2D features
58
Contributions

• Nonrigid registration
• E. Serradell, A. Romero, R. Leta, C. Gatta and F.
  Moreno-Noguer Simultaneous Correspondence and
  Non-Rigid 3D Reconstruction of the Coronary Tree
  from Single X-ray Images. In ICCV 2011

shape prior
recoveredmodel
2D X-ray Image
CT 3D Volume
59
Contributions
2D-3D
2D-2D or 3D-3D
Rigid Model
Nonrigid Model
60
Contributions

• Nonrigid registration
• Nonrigid 2D-to-2D or 3D-to-3D registration

Partial matching
10-6 m
10-7 m
Optical Microscope Image Stack
Electron Microscope Image Stack
61
Contributions

• Nonrigid registration
• Nonrigid 2D-to-2D or 3D-to-3D registration
• Extract neuronal tree ? Graph Matching

Optical Microscope Image Stack
Electron Microscope Image Stack
62
Contributions

• Nonrigid registration
• Nonrigid 2D-to-2D or 3D-to-3D registration
• Extract neuronal tree ? Graph Matching
• Two step process
• 1.- Affine transform (Kalman Filter approach)

y A x b
Optical Microscope Image Stack
Electron Microscope Image Stack
63
Contributions

• Nonrigid registration
• Nonrigid 2D-to-2D or 3D-to-3D registration
• Extract neuronal tree ? Graph Matching
• Two step process
• 1.- Affine transform (Kalman Filter approach)
• 2.- Nonrigid transform (Gaussian Processes for
  regression)

y A x b f(x)
Optical Microscope Image Stack
Electron Microscope Image Stack
64
Contributions

• Nonrigid registration
• Some results

original graphs
affinetransform
nonlinear refining
65
Contributions

• Nonrigid registration
• Some results

original graphs
affinetransform
nonlinear refining
affinetransform
nonlinear refining
66
Contributions

• Nonrigid registration
• E. Serradell, J. Kybic, F. Moreno-Noguer and P.
  Fua Robust Elastic 2D/3D Geometric Graph
  Matching, submitted to SPIE Medical Imaging

Optical Microscope Image Stack
Electron Microscope Image Stack
67
Contributions
2D-3D
2D-2D or 3D-3D
Rigid Model
Rigid 2D-2D / 3D-3D
Rigid 2D-3D
Nonrigid Model
Nonrigid 2D-3D
Nonrigid 2D-2D / 3D-3D
68
Contributions

• Global solution to point registration
• Valid for 2D-2D,3D-3D,2D-3D / rigid and nonrigid
  models
• Using Gaussian Processes some preliminary
  results

initial shape
recovered shape
69
Task Planning

1. Master ARV

1. Simultaneous Correspondence Robust Estimation

1. Nonrigid Model Reconstruction from Single Images

1. Elastic Graph Matching

1. Write Thesis

70
Task Planning
Done
On-going
To Do

1. Master ARV

1. Simultaneous Correspondence Robust Estimation

1. Nonrigid Model Reconstruction from Single Images

1. Elastic Graph Matching

1. Write Thesis

71
Task Planning
Done
On-going
To Do

1. Master ARV

1. Simultaneous Correspondence Robust Estimation

1. Nonrigid Model Reconstruction from Single Images

1. Elastic Graph Matching

1. Write Thesis

Research Stay at CVLAB (EPFL)
72
Task Planning
Done
On-going
To Do

1. Master ARV

1. Simultaneous Correspondence Robust Estimation

1. Nonrigid Model Reconstruction from Single Images

1. Elastic Graph Matching

1. Write Thesis

Project in collaboration with CVC (UAB), MAiA
(UB) and Hospital Sant Pau
73
Task Planning
Done
On-going
To Do

1. Master ARV

1. Simultaneous Correspondence Robust Estimation

1. Nonrigid Model Reconstruction from Single Images

1. Elastic Graph Matching

1. Write Thesis

Research Stay at CVLAB (EPFL)
74
Task Planning
Done
On-going
To Do

1. Master ARV

1. Simultaneous Correspondence Robust Estimation

1. Nonrigid Model Reconstruction from Single Images

1. Elastic Graph Matching

1. Write Thesis

Research Stay at CVLAB (EPFL)
75
Task Planning
Done
On-going
To Do

1. Master ARV

1. Simultaneous Correspondence Robust Estimation

1. Nonrigid Model Reconstruction from Single Images

1. Elastic Graph Matching

1. Write Thesis

76
Summary of achievements

• Published papers
• E. Serradell, M. Ozuysal, V. Lepetit, P. Fua and
  F. Moreno-Noguer Combining Geometric and
  Appearance Priors for Robust Homography
  Estimation . In ECCV 2010
• E. Serradell, A. Romero, R. Leta, C. Gatta and F.
  Moreno-Noguer Simultaneous Correspondence and
  Non-Rigid 3D Reconstruction of the Coronary Tree
  from Single X-ray Images. In ICCV 2011
• ICCV, ECCV acceptance rate lt 30
• Submitted papers
• E. Serradell, J. Kybic, F. Moreno-Noguer and P.
  Fua Robust Elastic 2D/3D Geometric Graph
  Matching, submitted to SPIE Medical Imaging

77
THANKS!