Affine structure from motion

- Marc Pollefeys
- COMP 256

Some slides and illustrations from J. Ponce, A.

Zisserman, R. Hartley, Luc Van Gool,

Last time Optical Flow

Ixu

Ix

u

Ixu- It

It

Aperture problem

- two solutions
- - regularize (smoothness prior)
- constant over window
- (i.e. Lucas-Kanade)

Coarse-to-fine, parametric models, etc

Tentative class schedule

Jan 16/18 - Introduction

Jan 23/25 Cameras Radiometry

Jan 30/Feb1 Sources Shadows Color

Feb 6/8 Linear filters edges Texture

Feb 13/15 Multi-View Geometry Stereo

Feb 20/22 Optical flow Project proposals

Feb27/Mar1 Affine SfM Projective SfM

Mar 6/8 Camera Calibration Silhouettes and Photoconsistency

Mar 13/15 Springbreak Springbreak

Mar 20/22 Segmentation Fitting

Mar 27/29 Prob. Segmentation Project Update

Apr 3/5 Tracking Tracking

Apr 10/12 Object Recognition Object Recognition

Apr 17/19 Range data Range data

Apr 24/26 Final project Final project

AFFINE STRUCTURE FROM MOTION

- The Affine Structure from Motion Problem
- Elements of Affine Geometry
- Affine Structure from Motion from two Views
- A Geometric Approach
- Affine Epipolar Geometry
- An Algebraic Approach
- Affine Structure from Motion from Multiple Views
- From Affine to Euclidean Images
- Structure from motion of multiple and deforming

object

Reading Chapter 12.

Affine Structure from Motion

Reprinted with permission from Affine Structure

from Motion, by J.J. (Koenderink and A.J.Van

Doorn, Journal of the Optical Society of America

A, 8377-385 (1990). ? 1990 Optical Society of

America.

- Given m pictures of n points, can we recover
- the three-dimensional configuration of these

points? - the camera configurations?

(structure) (motion)

Orthographic Projection

Parallel Projection

Weak-Perspective Projection

Paraperspective Projection

The Affine Structure-from-Motion Problem

Given m images of n fixed points P we can write

j

2mn equations in 8m3n unknowns

Overconstrained problem, that can be solved using

(non-linear) least squares!

The Affine Ambiguity of Affine SFM

When the intrinsic and extrinsic parameters are

unknown

So are M and P where

i

j

and

Q is an affine transformation.

Affine Spaces (Semi-Formal) Definition

2

Example R as an Affine Space

In General

The notation

is justified by the fact that choosing some

origin O in X allows us to identify the point P

with the vector OP.

Warning Pu and Q-P are defined independently

of O!!

Barycentric Combinations

- Can we add points? RPQ

NO!

- But, when

we can define

- Note

Affine Subspaces

Affine Coordinates

- Coordinate system for U

- Coordinate system for YOU

- Affine coordinates

- Coordinate system for Y

- Barycentric
- coordinates

When do m1 points define a p-dimensional

subspace Y of an n-dimensional affine space X

equipped with some coordinate frame basis?

Rank ( D ) p1, where

Writing that all minors of size (p2)x(p2) of D

are equal to zero gives the equations of Y.

Affine Transformations

- Bijections from X to Y that
- map m-dimensional subspaces of X onto

m-dimensional - subspaces of Y
- map parallel subspaces onto parallel subspaces

and - preserve affine (or barycentric) coordinates.

- Bijections from X to Y that
- map lines of X onto lines of Y and
- preserve the ratios of signed lengths of
- line segments.

3

In E they are combinations of rigid

transformations, non-uniform scalings and shears.

Affine Transformations II

- Given two affine spaces X and Y of dimension m,

and two - coordinate frames (A) and (B) for these spaces,

there exists - a unique affine transformation mapping (A) onto

(B).

- Given an affine transformation from X to Y, one

can always write

- When coordinate frames have been chosen for X

and Y, - this translates into

Affine projections induce affine transformations

from planes onto their images.

Affine Shape

Two point sets S and S in some affine space X

are affinely equivalent when there exists an

affine transformation y X X such that X

y ( X ).

Affine structure from motion affine shape

recovery.

recovery of the corresponding motion

equivalence classes.

Geometric affine scene reconstruction from two

images (Koenderink and Van Doorn, 1991).

Affine Structure from Motion

Reprinted with permission from Affine Structure

from Motion, by J.J. (Koenderink and A.J.Van

Doorn, Journal of the Optical Society of America

A, 8377-385 (1990). ? 1990 Optical Society of

America.

(Koenderink and Van Doorn, 1991)

The Affine Epipolar Constraint

Note the epipolar lines are parallel.

Affine Epipolar Geometry

The Affine Fundamental Matrix

where

An Affine Trick..

Algebraic Scene Reconstruction

The Affine Structure of Affine Images

Suppose we observe a scene with m fixed cameras..

The set of all images of a fixed scene is a 3D

affine space!

has rank 4!

From Affine to Vectorial Structure

Idea pick one of the points (or their center of

mass) as the origin.

What if we could factorize D? (Tomasi and

Kanade, 1992)

Affine SFM is solved!

Singular Value Decomposition

We can take

From uncalibrated to calibrated cameras

Weak-perspective camera

Calibrated camera

Problem what is Q ?

Note Absolute scale cannot be recovered. The

Euclidean shape (defined up to an arbitrary

similitude) is recovered.

Reconstruction Results (Tomasi and Kanade, 1992)

Reprinted from Factoring Image Sequences into

Shape and Motion, by C. Tomasi and T. Kanade,

Proc. IEEE Workshop on Visual Motion (1991). ?

1991 IEEE.

More examples

Tomasi Kanade92, Poelman Kanade94

More examples

Tomasi Kanade92, Poelman Kanade94

More examples

Tomasi Kanade92, Poelman Kanade94

Further Factorization work

- Factorization with uncertainty
- Factorization for indep. moving objects (now)
- Factorization for articulated objects (now)
- Factorization for dynamic objects (now)
- Perspective factorization (next week)
- Factorization with outliers and missing pts.

(Irani Anandan, IJCV02)

(Costeira and Kanade 94)

(Yan and Pollefeys 05)

(Bregler et al. 2000, Brand 2001)

(Sturm Triggs 1996, )

(Jacobs 97 (affine), Martinek Pajdla01

Aanaes02 (perspective))

Structure from motion of multiple moving objects

Structure from motion of multiple moving objects

Shape interaction matrix

- Shape interaction matrix for articulated objects

looses block diagonal structure

Costeira and Kanades approach is not usable for

articulated bodies (assumes independent motions)

Articulated motion subspaces

Motion subspaces for articulated bodies intersect

(Yan and Pollefeys, CVPR05) (Tresadern and Reid,

CVPR05)

Joint (1D intersection)

(jointorigin)

(rank8-1)

Hinge (2D intersection)

(hingez-axis)

(rank8-2)

Exploit rank constraint to obtain better estimate

Also for non-rigid parts if

(Yan Pollefeys, 06?)

Results

- Toy truck
- Segmentation
- Intersection

- Student
- Segmentation
- Intersection

Articulated shape and motion factorization

(Yan and Pollefeys, 2006?)

- Automated kinematic chain building for

articulated non-rigid obj. - Estimate principal angles between subspaces
- Compute affinities based on principal angles
- Compute minimum spanning tree

Structure from motion of deforming objects

(Bregler et al 00 Brand 01)

- Extend factorization approaches to deal with

dynamic shapes

Representing dynamic shapes

(fig. M.Brand)

represent dynamic shape as varying linear

combination of basis shapes

Projecting dynamic shapes

(figs. M.Brand)

Rewrite

Dynamic image sequences

One image

(figs. M.Brand)

Multiple images

Dynamic SfM factorization?

Problem find J so that M has proper structure

Dynamic SfM factorization

(Bregler et al 00)

Assumption SVD preserves order and orientation

of basis shape components

Results

(Bregler et al 00)

Dynamic SfM factorization

(Brand 01)

constraints to be satisfied for M

constraints to be satisfied for M, use to compute

J

hard!

(different methods are possible, not so simple

and also not optimal)

Non-rigid 3D subspace flow

(Brand 01)

- Same is also possible using optical flow in stead

of features, also takes uncertainty into account

Results

(Brand 01)

Results

(Brand 01)

Results

(Bregler et al 01)

Next class Projective structure from motion

Reading Chapter 13