Example Based 3D Shape Completion

1
Example Based 3D Shape Completion
Mark Pauly1,2, Niloy J. Mitra1, Joachim Giesen2,
Markus Gross2, Leonidas J. Guibas1
1 Stanford University
2 ETH, Zurich
2
Shape Completion

• Incomplete raw scans
• Imperfect range scanned data
• Complex objects with occluded regions
• Misalignment of multiple-views depth image scans
• Specular highlights

Ill-posed problem
Use prior knowledge !!
3
Diffusion (Davis, Marschner, Garr, Levoy, 2002)
4
Context Based Solution (Sharf, Alexa, Cohen-or,
2004)
5
Template based Solution (Allen, Curless,
Popovic, 2003 Kraevoy and Sheffer, 2005)
6
Our Solution

• Use 3D model database to provide geometric priors
  for shape completion
• Apply non-rigid transforms on the models
• More deformation ? less likely completion
• Consistently combine geometric information from
  multiple context models
• Final result comes with confidence values

7
Shape Completion Pipeline
8
Shape Completion Pipeline
9
Data Classification

• Local analysis
• quality of fit
• uniformity of sample distribution

High

• Scored Point Cloud
• confidence value assigned to each point

Low
10
Shape Completion Pipeline
11
Database Retrieval
12
Shape Completion Pipeline
13
Non-rigid Alignment

• Similar to the approaches proposed by
• Allen, Curless and Popovic, 2003.
• Sumner and Popovic, 2004.

14
Non-rigid Alignment

• Deformation Model
• Piecewise linear.Each vertex of the mesh
  assigned an independent displacement vector.
• Optimize for smallest Shape Matching Penalty
• Distortion Measure
• Geometric Error
• Feature Correspondence

Derived in the continuous setting to allow
consistent comparison between different context
models.
15
Warped Models
Low
Context Model
Warped Model
Matching Penalty
High
16
Non-rigid Transform
17
Shape Completion Pipeline
18
Initial Segmentation
19
Patch Growing
20
Video
21
Shape Completion Pipeline
22
Blending Final Segments
23
Result
24
Shape Completion Pipeline
25
Shape Completion Pipeline
26
Giraffe Example
27
Giraffe Example
28
Evaluation
Input Data
Context Model
Final Model
Evaluation
29
Enriching the Database
30
Additional Constraints
31
Galleria dell Accademia
32
Galleria dell Accademia
33
Galleria dell Accademia
34
Galleria dell Accademia
35
Future Directions

• Improve the retrieval stage.
• Automatic feature point detection.
• Use of more semantic information.
• Apply learning techniques to shape completion.
• Completion of additional attributes like surface
  texture, color.

36
Acknowledgements

• NSF grants CARGO-0138456, ITR-0205671,
  FRG-0454543, ARO grant DAAD19-03-1-033.
• Stanford Graduate fellowship.
• Mario Botsch, David Koller, Doo Young Kwon, Marc
  Levoy, Filip Sadlo, Vin da Silva, and Bob
  Sumner.

37
Questions
38
(No Transcript)
39
Correspondence
Invalid
Valid
40
Video
41
Related Works
Drori et al. 03

• Example-based image completion
• Drori et al. 2003 Jia and tang 2003 Sun et
  al. 2003
• Texture synthesis
• Efros and Leung 1999 Efros and Freeman 2001
• Wei and Levoy 2000 Ying et al. 2001
• Smooth surface completion
• Curless and Levoy 1996 Davis et al. 2002
• Ilic and Fua 2003 Verdera et al. 2003 Liepa
  2003
• Model-based surface reconstruction Savchenko et
  al. 2002
• Curve analogies Hertzmann et al. 2002

Wei and Levoy 00
Liepa 03
Hertzmann et al. 02