An isometric model for facial animation

1
An isometric model for facial animation and beyond
Michael M. Bronstein
Department of Computer Science Technion Israel
Institute of Technology

2
Co-authors
Ron Kimmel
Alex Bronstein
3
Agenda
Single texture mapping onto an animated face
Morphing
Expression interpolation and extrapolation
Beyond
4
Isometric model of facial expressions

• Face deformable Riemannian surface with
  geodesic distances
• Facial expression approximate isometry

B2K, IJCV 2005
5
Virtual makeup
Map a single texture image onto a 3D video
sequence of animated face in an
expression-invariant manner
TEXTURE
3 D V I D E O S E Q U E N C E
6
Approach I Common parametrization

• Parametrize and over a common
  parametrization domain
• by the maps and
• Draw the texture in the
  parametrization domain
• Map the texture to and using the maps
  and

7
How to find a parametrization?
Embed and into the plane by a
minimum-distortion map
G. Zigelman et al., IEEE TVCG, 2002
8
Multidimensional scaling
Given a sampling
the minimum-distortion embedding is found by
optimizing over the images
and not on itself
Alternative, more robust formulation

• Approximately common parametrization
• Requires alignment (usually manual, according to
  some fiducial points)
• Difficult to handle different or complicated
  topologies

A. Elad, R. Kimmel, CVPR 2001
9
Approach II Correspondence problem

• Assume that the texture is
  drawn on
• Find correspondence between
  and
• Transfer the texture by the map
• In case of common parametrization,

10
How to find the correspondence?

• Fiducial points-based methods usually give sparse
  correspondence and
• require manual assistance
• Optical flow between texture images (Blanz et
  al.) is not applicable when
• only geometric information is given

Embed into by a minimum-distortion map
11
Generalized multidimensional scaling (I)
G
MDS
MDS

• are computed once using
• fast marching
• have to be computed at each
  iteration
• Note that are not restricted to
• the mesh vertices

B2K, PNAS 2006
12
Generalized multidimensional scaling (II)
A weighted least-squares version of the problem

• More robust in practice
• Weights allow to handle different
• topologies (e.g. open mouth) and
• missing data (scanner artifacts)
• Multiresolution / multigrid schemes
• to prevent local convergence

B2K, PNAS 2006
13
Reference
Transferred texture
14
Calculus of faces (I)
Interpolation
Extrapolation

• Abstract manifold of facial articulations
• Face animation trajectory
• Minimum-distortion correspondence allows creating
  a (locally) linear
• space, in which faces are represented as vectors

15
Calculus of faces (II)
CORRESPONDENCE
Extrinsic geometry
Texture

• Extrinsic coordinates and texture interpolation

16
Interpolation
I N T E R P O L A T E D F R A M E S
0
1
0.5
0.25
0.75

• Temporal super-resolution increase frame rate of
  3D video by
• adding interpolated frames
• Interpolation of geometry and texture

17
Extrapolation
NEUTRAL
EXPRESSION
EXAGGERATED EXPRESSION
0
1.5
1

• Expression exaggeration synthesize new
  expressions using a
• non-convex combination
• Interpolation of geometry and texture

18
Bronstein2 Kimmel An isometric model for
facial animation and beyond
19
Morphing
SOURCE
TARGET
0
1
0.5
0.25
0.75

• Convex combination between two different faces
• Morphing of geometry and texture

20
Bronstein2 Kimmel An isometric model for
facial animation and beyond
21
Virtual body art
Texture mapping on articulated human body,
similarly to body art
22
22
Bronstein2 Kimmel An isometric model for
facial animation and beyond
Reference
Transferred texture
23
23
Bronstein2 Kimmel An isometric model for
facial animation and beyond
Reference
Transferred texture
24
Summary

• Isometric model of facial expressions
• Automatic dense correspondence based on the
  minimum-distortion
• mapping
• Possibility to find correspondence between
  partially missing or partially
• overlapping surfaces (COME TO THE SECOND TALK AT
  1530)
• Texture mapping, expression synthesis, morphing,
  etc
• GMDS - a generic tool that can be applied to
  different problems