3D reconstruction Class 16

1
3D reconstruction Class 16
2
3D photography course schedule
3
Papers
http//www.unc.edu/courses/2004fall/comp/290b/089/
papers/
4
Ideas for a project?
5
3D modeling

• Aligning range images
• Pairwise
• Globally
• Surface reconstruction
• Single range image
• Merged

(some slides from S. Rusinkiewicz, J. Ponce,)
6
Aligning 3D Data

• If correct correspondences are known,it is
  possible to find correct relative
  rotation/translation

7
Intermezzo quaternions
q is a quaternion, a 2 R is its real part, and ?
2 R3 is its imaginary part.
q a ?
Operations on quaternions

• Sum of quaternions

• Multiplication by a scalar

• Quaternion product

• Conjugate )

Note
8
Intermezzo quaternions and rotations
Let R denote the rotation of angle ? about the
unit vector u. Define Then for any vector ?,

Reciprocally, if q a ( b, c, d )T is a
unit quaternion, the corresponding rotation
matrix is
9
Estimate rigid transformation
Problem Find the rotation matrix R and the
vector t that minimize
Or E
10
Aligning 3D Data

• How to find corresponding points?
• Previous systems based on user input,feature
  matching, surface signatures, etc.

11
Aligning 3D Data

• Alternative assume closest points correspond to
  each other, compute the best transform

12
Aligning 3D Data

• and iterate to find alignment
• Iterated Closest Points (ICP) Besl McKay 92
• Converges if starting position close enough

13
ICP Variants

• Classic ICP algorithm not real-time
• To improve speed examine stages of ICP and
  evaluate proposed variants
• Rusinkiewicz Levoy, 3DIM 2001

• Selecting source points (from one or both meshes)
• Matching to points in the other mesh
• Weighting the correspondences
• Rejecting certain (outlier) point pairs
• Assigning an error metric to the current
  transform
• Minimizing the error metric

14
ICP Variant Point-to-Plane Error Metric

• Using point-to-plane distance instead of
  point-to-point lets flat regions slide along each
  other more easily Chen Medioni 91

15
Finding Corresponding Points

• Finding closest point is most expensive stage of
  ICP
• Brute force search O(n)
• Spatial data structure (e.g., k-d tree) O(log
  n)
• Voxel grid O(1), but large constant, slow
  preprocessing

16
Finding Corresponding Points

• For range images, simply project point Blais 95
• Constant-time, fast
• Does not require precomputing a spatial data
  structure

17
High-Speed ICP Algorithm

• ICP algorithm with projection-based
  correspondences, point-to-plane matchingcan
  align meshes in a few tens of ms.(cf. over 1
  sec. with closest-point)

Rusinkiewicz Levoy, 3DIM 2001
18
3D Global Registration

19
3D Global registration

• The problem
• Given n scans around an object
• Goal align them all
• First attempt ICP each scan to one other

20
3D Global registration

• Want method for distributing accumulated error
  among all scans

21
Approach 1 Avoid the Problem

• In some cases have 1 scan that covers large part
  of surface (e.g., cylindrical scan)
• Align all other scans to this anchor
• Disadvantage not always practical to obtain
  anchor scan

22
Approach 2 The Greedy Solution

• Align each new scan to all previous scans
• Disadvantages
• Order dependent
• Doesnt spread out error

23
Approach 3 Brute-Force Solution

• While not converged
• For each scan
• For each point
• For every other scan
• Find closest point
• Minimize error w.r.t. transforms of all scans
• Disadvantage
• Solve (np)?(np) matrix equation, where n is
  number of scans and p is number of points per scan

24
Approach 3a Slightly Less Brute-Force

• While not converged
• For each scan
• For each point
• For every other scan
• Find closest point
• Minimize error w.r.t. transform of this scan
• Faster than previous method (matrices are p?p)

25
Graph Methods

• Many globalreg algorithms create a graph of
  pairwise alignments between scans

Scan 3
Scan 5
Scan 1
Scan 4
Scan 2
Scan 6
26
Pullis Algorithm

• Perform pairwise ICPs, record sample (e.g. 200)
  of corresponding points
• For each scan, starting w. most connected
• Align scan to existing set
• While (change in error) gt threshold
• Align each scan to others
• All alignments during globalreg phase use
  precomputed corresponding points

27
Sharp et al. Algorithm

• Perform pairwise ICPs, record only optimal
  rotation/translation for each
• Decompose alignment graph into cycles
• While (change in error) gt tolerance
• For each cycle
• Spread out error equally among all scans in the
  cycle
• For each scan belonging to more than 1 cycle
• Assign average transform to scan

28
Lu and Milios Algorithm

• Perform pairwise ICPs, record optimal
  rotation/translation and covariance for each
• Least squares simultaneous minimization of all
  errors (covariance-weighted)
• Requires linearization of rotations
• Worse than the ICP case, since dont converge to
  (incremental rotation) 0

29
Open Questions in Global Registration

• Best way to do correctly-weighted globalreg
  without linearizing rotations?
• How to prevent bias (if many scans in one area,
  few scans in another)?
• Robust outlier detection
• For graph methods, pairwise ICP often automated
• One bad ICP can throw off the entire model

30
Bad ICP in Globalreg

• One bad ICP can throw off the entire model

Correct Globalreg
Globalreg Including Bad ICP
31
Hubers Automatic Modeling System

• Start with unaligned cans
• Generate candidate pairwise alignments using spin
  images
• Incrementally add scans, checking consistency
• ICP error in overlapping regions
• Free space consistency
• Occupied space consistency

32
Spin Images

• Johnson and Hebert
• Signature that captures local shape
• Similar shapes ? similar spin images

33
Computing Spin Images

• Start with a point on a 3D model
• Find (averaged) surface normal at that point
• Define coordinate system centered at this point,
  oriented according to surface normal and two
  (arbitrary) tangents
• Express other points (within some distance) in
  terms of the new coordinates

34
Computing Spin Images

• Compute histogram of locations of other points,
  in new coordinate system, ignoring rotation
  around normal

35
Computing Spin Images
36
Spin Image Parameters

• Size of neighborhood
• Determines whether local or global shapeis
  captured
• Big neighborhood more discriminatory power
• Small neighborhood resistance to clutter
• Size of bins in histogram
• Big bins less sensitive to noise
• Small bins captures more detail, less storage

37
Using Spin Images for Alignment

• Not robust

38
Verification Overlap Distance
Views 1 and 2 48, 2.1 mm
Views 1 and 9 15, 3.1 mm
39
Verification Visibility Consistency
surfaces are close
S2 blocks S1
S1 is not observed
S1
S1
S2
S2
S1
S2
C1
C1
C2
C2
C1
C2
consistentsurfaces
free spaceviolation
occupied spaceviolation
40
Automatically Modeling the Angel
connectivity afterlocal verification
initial model(bad matches in red)
41
Inferring New Pose Relations
30 overlap
1 overlap
42
The Final Angel Model
43
Problems With Reconstruction from Point Clouds
44
Surface Reconstruction from Range Images

• Often an easier problem than reconstruction from
  arbitrary point clouds
• Implicit information about adjacency,
  connectivity
• Roughly uniform spacing

45
Surface Reconstruction From Range Images

• First, construct surface from each range image
• Then, merge resulting surfaces
• Obtain average surface in overlapping regions
• Control point density

46
Range Image Tesselation

• Given a range image, connect up the neighbors

47
3D surface model
Depth image
Texture image
Triangle mesh
Textured 3D Wireframe model
48
Range Image Tesselation

• Caveat 1 cant be too aggressive
• Introduce distance threshold for tesselation

49
Range Image Tesselation

• Caveat 2 Which way to triangulate?
• Possible heuristics
• Shorter diagonal
• Dihedral angle closer to 180?
• Maximize smallest angle in both triangles
• Always the same way (best triangle strips)

50
Scan Merging Using Zippering

• Turk Levoy, 1994
• Erode geometry in overlapping areas
• Stitch scans together along seam
• Re-introduce all data
• Weighted average

51
Zippering
52
Point Weighting

• Higher weights to points facing the camera
• Favor higher sampling rates

53
Point Weighting

• Lower weights(tapering to 0)near boundaries
• Smooth blendsbetween views

54
Volumetric Reconstruction

• Implicit function defined volumetrically
• Usually stored sampled on a 3D grid
• Can be compressed (e.g., using RLE)
• Another possibility hierarchical data structures
• Can extract isosurface (i.e., subset of space
  where implicit function some constant)

55
Volumetric Reconstruction Overview

• Generate signed distance function (or something
  close to it) for each scan
• Compute average (possibly weighted)
• Extract isosurface

56
Volumetric integration
(Curless and Levoy, Siggraph96)
range surfaces
signed distance to surface
volume
weight (accuracy)
distance
depth
sensor
surface1

• use voxel space
• new surface as zero-crossing
• (find using marching cubes)
• least-squares estimate
• (zero derivativeminimum)

surface2
combined estimate
57
Volumetric Reconstruction Benefits

• Always generates a manifold surface
• Can control sampling density
• Averaging of signed distance functions
  corresponds to averaging the surfaces

58
Volumetric Reconstruction Drawbacks

• Represent a 3D entity rather than 2D
• Running time
• Storage
• Resampling step bandlimits the function
• Generates consistent topology, but not always the
  topology you wanted
• Problems with very thin surfaces

59
From volume to meshMarching Cubes

• First 2D, Marching Squares

Marching Cubes A High Resolution 3D Surface
Construction Algorithm,William E. Lorensen and
Harvey E. Cline,Computer Graphics (Proceedings
of SIGGRAPH '87), Vol. 21, No. 4, pp. 163-169.
60
From volume to meshMarching Cubes
Marching Cubes A High Resolution 3D Surface
Construction Algorithm,William E. Lorensen and
Harvey E. Cline,Computer Graphics (Proceedings
of SIGGRAPH '87), Vol. 21, No. 4, pp. 163-169.
61
From volume to meshMarching Cubes

• Improvement

Marching Cubes A High Resolution 3D Surface
Construction Algorithm,William E. Lorensen and
Harvey E. Cline,Computer Graphics (Proceedings
of SIGGRAPH '87), Vol. 21, No. 4, pp. 163-169.
62
Volumetric 3D integration
Multiple depth images
Volumetric integration
63
Next class appearance modeling