Generic Imaging: Calibration and 3D Reconstruction

1
Generic Imaging Calibration and 3D Reconstruction
Advancement to Candidacy by Srikumar
Ramalingam Department of Computer
Science University of California, Santa
Cruz srikumar_at_cse.ucsc.edu
2
Overview

• Problem Statement
• Introduction and Motivation
• Previous Work
• Current Work
• - Generic Calibration
• - Generic Structure-from-Motion Framework
• Proposed Work
• - Solving the degeneracy cases Axial cameras
• - Towards complete generic calibration
• - Generic self-calibration
• - Generic epipolar curves
• Conclusion

3
Problem Statement
1. Black box Camera Calibration
Some exotic camera
Can we calibrate any camera using a black box
calibration algorithm? - transformation from the
3D scene to its corresponding 2D image without
knowing anything about the internals of the
cameras
4
Problem Statement
pinhole
Reconstruction
omni
Can we do 3D Reconstruction from any two cameras?
Input Images from any two cameras (pinhole,
stereo, omni, non-central, etc.)
Goals 3D reconstruction of the scene
5
Problem Statement
x2
x1

• We propose the concept of generic imaging model
• Calibration algorithm for the generic imaging
  model
• 3D reconstruction algorithms for generic imaging
  model
• ..

X2
X1
general imaging model
6
Motivation Pinhole model
Physical setup for perspective projection by
Durer 1575
Pinhole camera model

• central model
• closer to human perception

7
Motivation Camera is a set of unconstrained rays
Thus we can obtain novel cameras by giving
more degrees of freedom to the rays
8
Motivation
Calibration and recovery of the shape of the
cornea
Modifying the resolution characteristics of a
scene
9
Motivation Summary
Pinhole spatial resolution, texture Stereo
absolute scale, 3D reconstruction of nonrigid
scenes Omnidirectional - motion estimation, large
field of view, better navigation Non-central -
absolute scale, motion estimation Catadioptric
Modify the resolution characteristics of a scene,
shape of the eye Hybrid scenarios have
advantages Example Scenarios 1) Video
conferencing / Surveillance systems 2) Large
scale modeling
pinhole
Omndirectional (with fisheye lens)
10
Introduction
General Camera Model
A pixel watches along one viewing ray
A camera is a set of pixels, with the associated
rays
Examples
Pinhole
11
Introduction
mirror (hyperboloid)
General Camera Model
A pixel watches along one viewing ray
A camera is a set of pixels, with the associated
rays
Examples
Pinhole
Central catadioptric
12
Introduction
General Camera Model
A pixel watches along one viewing ray
A camera is a set of pixels, with the associated
rays
Examples
Pinhole
Central catadioptric
Non-central catadioptric
13
Introduction
General Camera Model
A pixel watches along one viewing ray
A camera is a set of pixels, with the associated
rays
Examples
Pinhole
Non-central mosaic
Central catadioptric
Non-central catadioptric
14
Introduction
General Camera Model
A pixel watches along one viewing ray
A camera is a set of pixels the associated rays
Examples
Pinhole
Non-central mosaic
Multi- stereo
Central catadioptric
Non-central catadioptric
Etc. e.g. pinhole with radial or any
other distortion
15
Introduction Practical systems
16
Previous Work
Generic calibration M.D. Grossberg and S.K.
Nayar, A general imaging model and a method for
finding its parameters, ICCV-2001. P. Sturm and
S.Ramalingam, A generic method for camera
calibration, ECCV-2004. S. Ramalingam, S.K. Lodha
and P. Sturm, A generic cross-camera
structure-from-motion analysis,
Omnivis-2004. Cross-camera scenarios C. Geyer and
K. Daniilidis, A unifying theory of central
panoramic systems and practical implications,
ECCV-2000. P.Sturm, Mixing catadioptric and
perspective cameras, OMNIVIS-2002. Jingyi Yu and L
eonard McMillan, General Linear Cameras,
ECCV-2004. Structure-from-Motion R.I.Hartley and
A.Zisserman, Multiple view geometry in computer
vision, 2000. B.Micusik and T.Pajdla, 3D metric
reconstruction from uncalibrated omnidirectional
images, ACCV-2004. H. Bakstein and T. Pajdla, An
overview of non-central cameras, Research Report
CTU-CMP-2000-14. Robert Pless. Using many
cameras as one, CVPR-2003. Bundle adjustment B.
Triggs, P. McLauchlan, R.I. Hartley, and A.
Fitzgibbon, Bundle Adjustment- A modern
synthesis, Workshop on vision algorithms Theory
and Practice, 2000.
17
Current Work Calibration in the case of known
motion
18
Current Work Calibration in the case of unknown
motion
Estimate motions that make points collinear
camera
19
Current Work Calibration in the case of unknown
motion
Q
Q
Q
Estimate motions that make points collinear
R, t
R, t
camera
rank lt 3
20
Current Work Trifocal tensor
det 0
a trifocal tensor
21
Current Work Trifocal tensor coefficients and
coupled variables
3D general camera with nonplanar calibration
grids
22
Current Work Summary for generic calibration
(1) Take images of object in different poses
(2) 2D-3D matching (pixels to points on object)
(3) Estimation of tensors
(4) Extraction of motion parameters from tensors
(5) Computation of projection rays
(6) Bundle adjustment
Variants of the calibration method
3D vs. planar calibration object
general (non-central) vs. central camera model
23
Current Work Experiments for generic calibration
Central model applied on pinhole camera with
slight radial distortion
24
Current Work Experiments for generic calibration
Homemade non-central cameras
25
Current Work Experiments for generic calibration
Central model applied on fisheye
Distortion correction
26
Current Work Generic Structure-from-Motion
pipeline
27
Current Work Motion estimation for generic
cameras
28
Current Work Structure recovery
29
Current Work Bundle adjustment

• Ray-point (relatively straightforward)
• We minimize the distance between the 3D point and
  the intersecting rays.
• Easily extends to non-parametrically calibrated
  cameras and non-central cameras.
• Reprojection based bundle adjustment
• No parametric association between the pixel and
  the ray.
• Optimization functions like Levenberg Marquardt
  iteration uses the derivatives of reprojection
  errors.
• Difficult to compute in a non-parametric scenario.

30
Current Work Any camera is a cluster of central
rays
Divide all the rays into k central clusters of
rays Eg k1 for pinhole camera, k2 for stereo
camera, kn for an oblique camera, etc.
We intersect a plane to each of the central
cluster
On each central cluster, reproj based bundle
adjustment is applied.
Issues Choice of the plane, clustering issues.
31
Current Work Experiments for generic SfM
3D reconstruction after generic calibration for
pinhole images
32
Current Work Experiments for generic SfM
algorithm
Cross camera SfM for stereo camera and
omnidirectional camera with fisheye lens
33
Current Work Experiments for generic SfM
algorithm
Cross camera SfM for pinhole camera and
omnidirectional camera with fisheye lens
34
Current Work Experiments for generic SfM
algorithm
Cross camera SfM for pinhole camera and
omnidirectional camera with fisheye lens
35
Current Work Experiments for generic SfM
algorithm
Percentage errors in
the pinhole and omni scenario (house)
Percentage errors in the
stereo and omni scenario (objects)
Difference measures in the
pinhole and omni scenario (stevensons appts, UCSC)
36
Proposed Work Solving the degeneracy cases-Axial
cameras
Axial cameras All projection rays intersect a
single ray in space
Examples stereo camera, multicamera setups
with centers along a single line, pushbroom
cameras, cross-slit cameras, catadioptric
cameras with perpective lens and spherical
mirrors, etc
37
Proposed Work Solving the degeneracy cases-Axial
cameras
Status of variants Nature of solutions
(unique/degenerate, no of grids)
Axial camera parallel planes
38
Proposed Work Towards complete generic
calibration

• Testing of our algorithm on more exotic systems
• Catadioptric cameras with hyperbolic mirror and
  perspective lens
• Catadioptric cameras with spherical mirror and
  perspective lens
• Catadioptric camera based on eye
• Multicamera setups
• Pushbroom and other cross-slit cameras

39
Proposed Work Towards complete generic
calibration

• Calibration using multiple boards
• Pose estimation of additional grids
• Bundle adjustment to refine the pose of all the
  boards
• Model selection problem for approximately central
  cameras

40
Proposed Work Generic self-calibration
Can we calibrate our generic imaging model
without using calibration grids (without any
knowledge of the scene)
Coplanarity constraints in the case of pure
translation
41
Proposed Work Generic self-calibration
Pure rotation The ray corresponding to p1
becomes the ray corresponding to p2 . We also
have coplanarity constraints.

• Further goals
• For rotations about axes that do not go through
  the optical center
• Finally for completely general motions

42
Proposed Work Generic epipolar curves

• Pinhole
• Epipolar curve is a line
• One epipole per image

• Hyperbolic mirror with perspective lens
• Epipolar curve can be line, circle, parabola,
  conic, etc
• Two epipole per image

- Parametric algorithms become very complex for
novel cameras
43
Proposed Work Generic epipolar curves

• Epipolar curve computation problem simplifies
  itself to a simple ray intersection problem after
  the non-parametric generic calibration
• Goal Compute f(x1,y1,x2,y2) where
• (x1,y1) and (x2,y2) are corresponding image
  points
• f can be linear, quadratic or cubic depending on
  the nature of the cameras

44
Impact

• Simplifies the existing theories for calibration
  and 3D reconstruction
• Allows us to simultaneously avail the advantages
  of several novel cameras
• Solves challenging problems like cross-camera
  structure-from-motion analysis

Thanks!