Real-time Combined 2D 3D Active Appearance Models Jing Xiao, Simon Baker,Iain Matthew, and Takeo Kanade CVPR 2004

1
Real-time Combined 2D3D Active Appearance
ModelsJing Xiao, Simon Baker,Iain Matthew, and
Takeo KanadeCVPR 2004

• Presented by Pat Chan
• 23/11/2004

2
Outline

• Introduction
• Active Appearance Models AAMs
• 3D Morphable Models 3DMMs
• Representational Power of AAM
• Combined 2D3D AAMs
• Conclusion

3
Introduction

• Active Appearance Models are generative models
  commonly used to model faces
• Another closely related type of face models are
  3D Morphable Models
• In this paper, it tries to model 3D phenomena by
  using the 2D AAM
• Constrain the AAM with the 3D models to achieve a
  real-time algorithm for fitting the AAM

4
Active Appearance Models (AAMs)

• 2D linear shape is defined by 2D triangulated
  mesh and in particular the vertex locations of
  the mesh.
• Shape s can be expressed as a base shape s0.
• pi are the shape parameter.
• s0 is the mean shape and the matrices si are the
  eigenvectors corresponding to the m largest
  eigenvalues

5
Active Appearance Models (AAMs)

• The appearance of an independent AAM is defined
  within the base mesh s0. A(u) defined over the
  pixels u ? s0
• A(u) can be expressed as a base appearance A0(u)
  plus a linear combination of l appearance
• Coefficients ?i are the appearance parameters.

6
Active Appearance Models (AAMs)

• The AAM model instance with shape parameters p
  and appearance parameters ? is then created by
  warping the appearance A from the base mesh s0 to
  the model shape s.

Piecewise affine warp W(u p) (1) for any pixel
u in s0 find out which triangle it lies in, (2)
warp u with the affine warp for that triangle.
M(W(up))
7
Fitting AAMs

• Minimize the error between I (u) and M(W(u p))
  A(u).
• If u is a pixel in s0, then the corresponding
  pixel in the input image I is W(u p).
• At pixel u the AAM has the appearance
• At pixel W(u p), the input image has the
  intensity I (W(u p)).
• Minimize the sum of squares of the difference
  between these two quantities

8
DEMO Video 2D AAMs
9
DEMO Video 2D AAMs
10
3D Morphable Models (3DMMS)

• 3D shape of 3DMM is defined by 3D triangulated
  mesh and in particular the vertex location of the
  mesh.
• The s can be expressed as a based shape s 0 plus
  a linear combination of m shape matrices s i

11
3D Morphable Models (3DMMS)

• The appearance of a 3DMM is defined within a 2D
  triangulated mesh that has the same topology as
  the base mesh s0.
• The appearance Â(u) can be expressed as a based
  appearance Â0(u) plus a linear combination of l
  appearance images Âi(u).

12
3D Morphable Models (3DMMS)

• To generate a 3DMM model instance, an image
  formation model is used to convert the 3D shape s
  in to 2D mesh.
• The result of the imaging 3D point x (x, y, z)T
  is
• i, j are the projection axes, o is the offset of
  the origin
• Given shape parameters pi ? compute 3D shape ?
  map to 2D ? compute appearance ? warp onto 2D
  mesh (defined by mapping from 2D vertices in s0
  to 2D vertices for 3D s.)

13
Representational Power of AAM

• Can 2D shape models represent 3D?
• The 2D shape variation of the 3D model
• The projection matrix can be expressed as
• Therefore 3D model can be represented as
  combination of
• The variation of the 3D model can therefore be
  represented by an appropriate set of 2D shape
  vectors, such as

6(m1) 2D shape vectors needed to represent a
3D model
14
Representational Power of AAM

• Experiments
• Use 3D-cube s s0 p1s1
• Generate 60 sets p1 and P randomly
• Synthesize 2D shapes of 60 3D model instances
• Compute 2D shape model by performing PCA on 60 2D
  shapes
• Result 12 shapes vectors for each 2D shape mode
• Confirm 6(m1) 2D vector is required
• However, 2D models generate invalid cases.
• Constrains is need to add on the model

15
Combined 2D 3D AAMs

• At time t, we have
• 2D AAM shape vector in all N images into a
  matrix
• Represent as a 3D linear shape modes W MB

16
Compute the 3D Model

• Perform singular value decomposition (SVD) on W
  and factorize it into
• The scaled projection matrix M and the shape
  vector matrix B are given by
• G is the corrective matrix.
• Additional rotational and basis constrain to
  compute G ? M and B can be
  determined
• Thus, the 3D shape modes can be computed from the
  2D AMM shape modes and the 2D AAM tracking
  results.

17
Calculate the Corrective Matrix

• Rotational constraints and basis constraints are
  used.
• Rotational constraints (denote GGT by Q)
• where M2i-12i represents the ith two-row of
  M
• c is the coefficient and R is rotation matrices
• Due to orthogonormality of rotation matrices and
  Q is symmetric,

18
Calculate the Corrective Matrix

• Basis constraints
• We find K frames including independent shapes and
  treat those shapes as a set of bases, the bases
  are determined uniquely, we have

19
Compute the 3D Model
AAM shapes
AAM appearance
First three 3D shapes modes
20
Constraining an AAM with 3D Shape

• Constraints on the 2D AAM shape parameters p
  (p1, , pm) that force the AAM to only move in a
  way that is consistent with the 3D shape modes
• and the 2D shape variation of the 3D shape modes
  over all imaging condition is
• Legitimate values of P and p such that the 2D
  projected 3D shape equals the 2D shape of AAM.
  The constraint is written as

21
Fitting with 3D Shape Constraints

• AAM fitting is to minimize
• I.e the error between the appearance and the
  original image
• Impose the constrains of 2D projected 3D shape
  equals the 2D shape of AAM as soft constrains on
  the above equation with a large K

22
Fitting with 3D Shape Constraints

• Optimize for the AAM shape p, q, and the
  appearance ? parameters
• Calculate the square difference between the
  appearance and the original image and project the
  difference into orthogonal complement of the
  linear subspace spanned by the vectors A1, , Al.
• It is optimized by using inverse compositional
  algorithm, I.e. iteratively minimizing
• Then, solve the appearance parameters using the
  linear closed form solution

23
Experimental Results
Estimated 3D shape
Estimates of the 3D Pose extracted from the
current estimate of the camera matrix P
Initialization
2D AAM
After 30 Iterations
Converged
24
Experimental Results

• Results of using the algorithm to track a face in
  180
• frame video sequence by fitting the model in each
  frame

25
DEMO Video -- 2D3D AAMs
26
2D3D AAM Model Reconstruction
Input Image
Tracked result (2D3D fit result)
Shows two new view reconstruction
2D3D model reconstruction
27
Compare the fitting speed with 2D AAMs

• Frames per second of 2D3D gt 2D AAM
• Iteration per second of 2D gt 2D3D, but 2D need
  more iteration for convergence

28
Conclusion

• 2D AAMs can represent any phenomena that 3DMMs
  can.
• Showed how to compute the equivalent 3D shape
  models from a 2D AAM with basis constrains,
  rotational constrains.
• Improve the fitting speed of the 2D AAMs with 3D
  shapes constrains
• 2D 3D AAM is the ability to render the 3D model
  from novel viewpoint.

29
Q A