Geometry and Algebra of Multiple Views

1
Geometry and Algebra of Multiple Views

• René Vidal
• Center for Imaging Science
• BME, Johns Hopkins University
• http//www.eecs.berkeley.edu/rvidal
• Adapted for use in CS 294-6 by Sastry and Yang
• Lecture 13. October 11th, 2006

2
Two View Geometry

• From two views
• Can recover motion 8-point algorithm
• Can recover structure triangulation

• Why multiple views?
• Get more robust estimate of structure more data
• No direct improvement for motion more data
  unknowns

3
Why Multiple Views?

• Cases that cannot be solved with two views
• Uncalibrated case need at least three views
• Line features need at least three views
• Some practical problems with using two views
• Small baseline good tracking, poor motion
  estimates
• Wide baseline good motion estimates, poor
  correspondences
• With multiple views one can
• Track at high frame rate tracking is easier
• Estimate motion at low frame rate throw away data

4
Problem Formulation
Input Corresponding images (of features) in
multiple images. Output Camera motion, camera
calibration, object structure.
5
Multiframe SFM as an Optimization Problem

• Can we minimize the re-projection error?
• Number of unknowns 3n 6 (m-1) 1
• Number of equations 2nm
• Very likely to converge to a local minima
• Need to have a good initial estimate

6
Motivating Examples
Image courtesy of Paul Debevec
7
Motivating Examples
Image courtesy of Kun Huang
8
Using Geometry to Tackle the Optimization Problem

• What are the basic relations among multiple
  images of a point/line?
• Geometry and algebra
• How can I use all the images to reconstruct
  camera pose and scene structure?
• Algorithm
• Examples
• Synthetic data
• Vision based landing of unmanned aerial vehicles

9
Projection Point Features
Homogeneous coordinates of a 3-D point
Homogeneous coordinates of its 2-D image
Projection of a 3-D point to an image plane
10
Multiple View Matrix for Point Features

• WLOG choose frame 1 as reference
• Rank deficiency of Multiple View Matrix

(generic)
(degenerate)
11
Geometric Interpretation of Multiple View Matrix

• Entries of Mp are normals to epipolar planes
• Rank constraint says normals must be parallel

12
Multiple View Matrix for Point Features
Mp encodes exactly the 3-D information missing in
one image.
13
Rank Conditions vs. Multifocal Tensors

• Other relationships among four or more views,
  e.g. quadrilinear constraints, are algebraically
  dependent!

Two views epipolar constraint
Three views trilinear constraints
14
Reconstruction Algorithm for Point Features

• Given m images of n points

15
Reconstruction Algorithm for Point Features
Given m images of n (gt6) points
For the jth point
For the ith image
SVD
16
Reconstruction Algorithm for Point Features

• Initialization
• Set k0
• Compute using the 8-point algorithm
• Compute and normalize so that
• Compute as the null space of
• Compute new as the null space of
  Normalize so that
• If stop, else kk1 and goto 2.

17
Reconstruction Algorithm for Point Features
18
Reconstruction Algorithm for Point Features
19
Multiple View Matrix for Line Features
Homogeneous representation of a 3-D line
Homogeneous representation of its 2-D image
Projection of a 3-D line to an image plane
20
Multiple View Matrix for Line Features

• Point Features

• Line Features

21
Multiple View Matrix for Line Features
each is an image of a (different) line in 3-D
Rank 3 any lines
Rank 2 intersecting lines
Rank 1 same line
.
.
.
.
.
.
.
.
.
22
Multiple View Matrix for Line Features
23
Reconstruction Algorithm for Line Features

• Initialization
• Set k0
• Compute and using linear algorithm
• Compute and normalize so that
• Compute as the null space of
• Compute new as the null space of
  Normalize so that
• If stop, else kk1 and goto 2.

24
Reconstruction Algorithm for Line Features

• Initialization
• Use linear algorithm from 3 views to obtain
  initial estimate for and
• Given motion parameters compute the structure
  (equation of each line in 3-D)
• Given structure compute motion
• Stop if error is small stop, else goto 2.

25
Universal Rank Constraint

• What if I have both point and line features?
• Traditionally points and lines are treated
  separately
• Therefore, joint incidence relations not
  exploited
• Can we express joint incidence relations for
• Points passing through lines?
• Families of intersecting lines?

26
Universal Rank Constraint

• The Universal Rank Condition for images of a
  point on a line

• Multi-nonlinear constraints
• among 3, 4-wise images.

• Multi-linear constraints
• among 2, 3-wise images.

27
Universal Rank Constraint points and lines
28
Universal Rank Constraint family of intersecting
lines
each can randomly take the image of any of the
lines
Nonlinear constraints among up to four views
.
.
.
29
Universal Rank Constraint multiple images of a
cube
Three edges intersect at each vertex.
.
.
.
30
Universal Rank Constraint multiple images of a
cube
31
Universal Rank Constraint multiple images of a
cube
32
Universal Rank Constraint multiple images of a
cube
33
Multiple View Matrix for Coplanar Point Features
Homogeneous representation of a 3-D plane
Corollary Coplanar Features
Rank conditions on the new extended remain
exactly the same!
34
Multiple View Matrix for Coplanar Point Features
Given that a point and line features lie on a
plane in 3-D space
In addition to previous constraints, it
simultaneously gives homography
35
Example Vision-based Landing of a Helicopter
36
Example Vision-based Landing of a Helicopter
Tracking two meter waves
Landing on the ground
37
Example Vision-based Landing of a Helicopter
38
General Rank Constraint for Dynamic Scenes
For a fixed camera, assume the point moves with
constant acceleration
39
General Rank Constraint for Dynamic Scenes
40
General Rank Constraint for Dynamic Scenes
Projection from n-dimensional space to
k-dimensional space
We define the multiple view matrix as
where s and s are images and
coimages of hyperplanes.
41
General Rank Constraint for Dynamic Scenes
Projection from to
Single hyperplane
Projection from to
42
Summary

• Incidence relations rank conditions
• Rank conditions multiple-view factorization
• Rank conditions imply all multi-focal constraints
• Rank conditions for points, lines, planes, and
  (symmetric) structures.

43
Incidence Relations among Features
Pre-images are all incident at the
corresponding features
.
.
.
44
Traditional Multifocal or Multilinear Constraints

• Given m corresponding images of n points
• This set of equations is equivalent to
• Multilinear constraints among 2, 3, 4 views

45
Multiview Formulation The Multiple View Matrix
Point Features
Line Features