Structure from motion

1
Structure from motion

• Marc Pollefeys
• COMP 256

Some slides and illustrations from J. Ponce, A.
Zisserman, R. Hartley, Luc Van Gool,
2
Last time Optical Flow
Ixu
Ix
u
Ixu- It
It
Aperture problem

• two solutions
• - regularize (smoothness prior)
• constant over window
• (i.e. Lucas-Kanade)

Coarse-to-fine, parametric models, etc
3
Tentative class schedule
Aug 26/28 - Introduction
Sep 2/4 Cameras Radiometry
Sep 9/11 Sources Shadows Color
Sep 16/18 Linear filters edges (Isabel hurricane)
Sep 23/25 Pyramids Texture Multi-View Geometry
Sep30/Oct2 Stereo Project proposals
Oct 7/9 Tracking (Welch) Optical flow
Oct 14/16 - -
Oct 21/23 Silhouettes/carving (Fall break)
Oct 28/30 - Structure from motion
Nov 4/6 Project update Camera calibration
Nov 11/13 Segmentation Fitting
Nov 18/20 Prob. segm.fit. Matching templates
Nov 25/27 Matching relations (Thanksgiving)
Dec 2/4 Range data Final project
4
Todays menu

• Affine structure from motion
• Geometric construction
• Factorization
• Projective structure from motion
• Factorization
• Sequential

5
Affine Structure from Motion
Reprinted with permission from Affine Structure
from Motion, by J.J. (Koenderink and A.J.Van
Doorn, Journal of the Optical Society of America
A, 8377-385 (1990). ? 1990 Optical Society of
America.

• Given m pictures of n points, can we recover
• the three-dimensional configuration of these
  points?
• the camera configurations?

(structure) (motion)
6
Orthographic Projection
Parallel Projection
7
Weak-Perspective Projection
Paraperspective Projection
8
The Affine Structure-from-Motion Problem
Given m images of n fixed points P we can write
j
2mn equations in 8m3n unknowns
Overconstrained problem, that can be solved using
(non-linear) least squares!
9
The Affine Ambiguity of Affine SFM
When the intrinsic and extrinsic parameters are
unknown
So are M and P where
i
j
and
Q is an affine transformation.
10
Affine Spaces (Semi-Formal) Definition
11
2
Example R as an Affine Space
12
In General
The notation
is justified by the fact that choosing some
origin O in X allows us to identify the point P
with the vector OP.
Warning Pu and Q-P are defined independently
of O!!
13
Barycentric Combinations

• Can we add points? RPQ

NO!

• But, when

we can define

• Note

14
Affine Subspaces
15
Affine Coordinates

• Coordinate system for U

• Coordinate system for YOU

• Affine coordinates

• Coordinate system for Y

• Barycentric
• coordinates

16
When do m1 points define a p-dimensional
subspace Y of an n-dimensional affine space X
equipped with some coordinate frame basis?
Rank ( D ) p1, where
Writing that all minors of size (p2)x(p2) of D
are equal to zero gives the equations of Y.
17
Affine Transformations

• Bijections from X to Y that
• map m-dimensional subspaces of X onto
  m-dimensional
• subspaces of Y
• map parallel subspaces onto parallel subspaces
  and
• preserve affine (or barycentric) coordinates.

• Bijections from X to Y that
• map lines of X onto lines of Y and
• preserve the ratios of signed lengths of
• line segments.

3
In E they are combinations of rigid
transformations, non-uniform scalings and shears.
18
Affine Transformations II

• Given two affine spaces X and Y of dimension m,
  and two
• coordinate frames (A) and (B) for these spaces,
  there exists
• a unique affine transformation mapping (A) onto
  (B).

• Given an affine transformation from X to Y, one
  can always write

• When coordinate frames have been chosen for X
  and Y,
• this translates into

19
Affine projections induce affine transformations
from planes onto their images.
20
Affine Shape
Two point sets S and S in some affine space X
are affinely equivalent when there exists an
affine transformation y X X such that X
y ( X ).
Affine structure from motion affine shape
recovery.
recovery of the corresponding motion
equivalence classes.
21
Geometric affine scene reconstruction from two
images (Koenderink and Van Doorn, 1991).
22
Affine Structure from Motion
Reprinted with permission from Affine Structure
from Motion, by J.J. (Koenderink and A.J.Van
Doorn, Journal of the Optical Society of America
A, 8377-385 (1990). ? 1990 Optical Society of
America.
(Koenderink and Van Doorn, 1991)
23
The Affine Epipolar Constraint
Note the epipolar lines are parallel.
24
Affine Epipolar Geometry
25
The Affine Fundamental Matrix
where
26
An Affine Trick..
Algebraic Scene Reconstruction
27
The Affine Structure of Affine Images
Suppose we observe a scene with m fixed cameras..
The set of all images of a fixed scene is a 3D
affine space!
28
has rank 4!
29
From Affine to Vectorial Structure
Idea pick one of the points (or their center of
mass) as the origin.
30
What if we could factorize D? (Tomasi and
Kanade, 1992)
Affine SFM is solved!
Singular Value Decomposition
We can take
31
From uncalibrated to calibrated cameras
Weak-perspective camera
Calibrated camera
Problem what is Q ?
Note Absolute scale cannot be recovered. The
Euclidean shape (defined up to an arbitrary
similitude) is recovered.
32
Reconstruction Results (Tomasi and Kanade, 1992)
Reprinted from Factoring Image Sequences into
Shape and Motion, by C. Tomasi and T. Kanade,
Proc. IEEE Workshop on Visual Motion (1991). ?
1991 IEEE.
33
More examples
Tomasi Kanade92, Poelman Kanade94
34
More examples
Tomasi Kanade92, Poelman Kanade94
35
More examples
Tomasi Kanade92, Poelman Kanade94
36
Further Factorization work

• Factorization with uncertainty
• Factorization for indep. moving objects
• Factorization for dynamic objects
• Perspective factorization (next week)
• Factorization with outliers and missing pts.

(Irani Anandan, IJCV02)
(Costeira and Kanade 94)
(Bregler et al. 2000, Brand 2001)
(Sturm Triggs 1996, )
(Jacobs 1997 (affine), Martinek and Pajdla
2001, Aanaes 2002 (perspective))
37
Multiple indep. moving objects
38
Multiple indep. moving objects
39
Dynamic structure from motion
(Bregler et al 00 Brand 01)

• Extend factorization approaches to deal with
  dynamic shapes

40
Representing dynamic shapes
(fig. M.Brand)
represent dynamic shape as varying linear
combination of basis shapes
41
Projecting dynamic shapes
(figs. M.Brand)
Rewrite
42
Dynamic image sequences
One image
(figs. M.Brand)
Multiple images
43
Dynamic SfM factorization?
Problem find J so that M has proper structure
44
Dynamic SfM factorization
(Bregler et al 00)
Assumption SVD preserves order and orientation
of basis shape components
45
Results
(Bregler et al 00)
46
Dynamic SfM factorization
(Brand 01)
constraints to be satisfied for M
constraints to be satisfied for M, use to compute
J
hard!
(different methods are possible, not so simple
and also not optimal)
47
Non-rigid 3D subspace flow
(Brand 01)

• Same is also possible using optical flow in stead
  of features, also takes uncertainty into account

48
Results
(Brand 01)
49
Results
(Brand 01)
50
Results
(Bregler et al 01)
51
Next class Projective structure from motion