Recovery of NonRigid Facial Shape and Motion

1
Recovery of Non-Rigid Facial Shape and Motion

• Jingyu Yan
• Spring 2003

2
Summary

• Recover the shape and motion of a non-rigid human
  face based on a morphable model from video
  sequence
• C S C is the morphing coefficient and S is
  the shape basis
• Animate the 3D facial model

Figure source 1
3
Steps

• I. Motion Estimation
• II. Recovery of Shape and Motion
• III. Surface reconstruction and texture mapping

4
I. Motion Estimation

• Motions from optical flows
• Y XF (Y temporal image gradients X spacial
  image gradients F optical flows)

5
I. Motion Estimation

• Hierarchical motion estimation
• build a pyramid of imagesThe motion data from
  the coarser level provides estimates for the
  finer level.

6
I. Motion Estimation

• Impose rank constraints on motion data
• P R(C S) T rank(S)lt 3K gt rank(R(C
  S))lt3K
• P-T R(C S) rank(P-T) lt3K

7
I. Motion Estimation

• Rank constraints on motion data
• Weak features can be corrected?
• Not necessarily true the non-rigid part gives
  room for drifting
• But it works well for rigid motion

8
I. Motion Estimation

• Textureless area
• Relax the Lambertian surface requirement
• Further weight down weak features by its
  dissimilarity before enforcing rank constraints
• Recover weak features separately using rigid
  motion constraints

9
II. Recovery of Shape and Motion

• Overview
• Factorize P-T into M S
• Factorize M into R C
• The difficulty
• R has special rotation matrix structure
• M M J-1 , S J S
• Direct solution requires solving F( F number of
  frames) equations both quartic and quadratic of
  (3K)2 variables

10
II. Recovery of Shape and Motion

• Present approaches
• Breglers approach iterate between R,C,S
• Brands approach no iteration minimize
  deformations
• Using Brands shape regularization during each
  iteration gives better results

11
II. Recovery of Shape and Motion

• Factorization with uncertainty
• Some motion data (P-T) is more reliable than
  others.
• Certainty-warp the motion data before
  factorization and unwarp the data afterwards.
  Thus, uncertainty can be integrated out.
• A certainty warp is Q VD1/2 where the covariant
  matrix X VDVT.

12
III. Surface reconstruction and Texture-mapping

• The topology of the 3D points is not defined
• Recovered rotations help to properly reconstruct
  the surface
• Texture-mapping

13
Demo

• Test video 250 frames, 20 features
• Recovered facial shapes
• Sample 1
• Sample 2
• Deformation 1
• Deformation 2

14
Demo

• Animate the 3D facial models
• View 1 with features
• View 2 without features
• View 3 a different viewing angle
• View 4 comparison

15
Further improvements

• Refining shape bases by tracking
• Dealing with lighting change
• Full perspective model

16
References

• M. Brand. Morphable 3D models from video.
• Trancking and Modelling non-rigid Objects with
  Rank Constraints Lorenzo Torresani, Danny Yang,
  Gene Alexander, Christoph BreglerProc. IEEE CVPR
  2001
• M. Irani. Multi-frame optical flow estimation
  using subspace constraints. In Proc. ICCV, 1999.
• C. Tomasi and T. Kanade. Shape and motion from
  image streams under orthography A factorization
  method. International Journal of Computer Vision,
  9(2)137154, 1992.