Incorporating constraints and prior knowledge into factorization algorithms an algorithm with an app

1
Incorporating constraints and prior knowledge
into factorization algorithms an algorithm with
an application to 3D recovery

• Amit Gruber and Yair Weiss
• The Hebrew University of Jerusalem

2
Outline

• Structure From Motion (SFM)
• Problem description and formulation
• Challenges
• Our approach
• Priors and Constraints
• An EM algorithm for matrix factorization
• Results

3
Structure from Motion

• Reconstruct 3D structure of a scene from multiple
  views.

4
Affine ModelTomasi-Kanade 92
projection of P features in F images
W measurement
M motion
S shape
5
Missing Data and Directional Uncertainty

• In any realistic situation, the measurements
  matrix will have missing entries due to
  occlusions, tracking failure or changes in field
  of view.
• Directional uncertainty correlated non-uniform
  Gaussian noise in u and v coordinates.

Aperture problem
6
Multiple Motions(Motion Segmentation)

• In the case of multiple motions, if we group
  together points according to motion, the
  factorization can be written as Costeira-
  Kanade, 95

7
Previous Work

• Linear fitting with missing data, Jacobs, CVPR
  1997.
• Iterative methods Shum, Ikeuchi, Reddy 1995,
  Morris-Kanade, ICCV 1999.
• Incremental SVD, Brand, ECCV 2002.
• Factorization with uncertainty, Irani, Anandan,
  ECCV 2000.
• Motion segmentation algorithms Costeira-Kanade,
  1995, Gear, 1998, Kanatani, 2002, Vidal, 2004.
• Non-rigid 3D shape Torresani, Hertzmann, Bregler

8
Our approach

• Incorporate meaningful priors.
• Impose constraints on the resulting factors.
• Both are done by formulating the factorization
  problem as a problem of factor analysis.

9
Factor Analysis

• y(t) Ax(t)
• y(t) noisy observations
• x(t) factors (unknown)
• A linear coefficients (constant in time,
  unknown).
• Minimize
• with respect to A, x(t).

10
SFM as factor analysis

• is known from 2D motion analysis.
• For missing observations, set to
  zeros.

11
Temporal Coherence

• In a video sequence, the camera location and
  orientation at time t1 will probably be similar
  to its location and orientation at time t.

12
Graphical Model
xt are latent variables describing camera
parameters at time t. yt are the observations at
time t 2D image locations.
13
EM for SFMBased on EM for Factor Analysis, D.
Rubin and D. Thayer. 1982
14
Multiple Motions Imposing constraints

• Each column of the structure matrix, S, consists
  of (at most) 4 non zero entries.
• Modified M-step (find segmentation)
• For each point, find structure according to
  each of the K different motions Choose the
  motion that maximizes likelihood.
• No assumptions regarding the rank of the motion
  matrix are needed.

15
Input sequence
16
Results
17
Boujou 2d3 Results
No apparent structure is visible
18
Input sequence
19
Results
20
Performance evaluation
Influence of noise Influence of
missing data
21
Motion Segmentation - Results
Object A Object
B
22
Performance evaluation
Influence of noise
Influence of missing data
23
Summary

• Factor Analysis provides a simple way for
  factorization with missing data and directional
  uncertainty.
• By incorporating meaningful prior, factorization
  can be performed even when the noise level is
  high and large portion of the measurements is
  missing.
• Imposing constraints on the resulting factors
  improves robustness to noise by eliminating the
  need for a combinatorial search.