Learning the Appearance of Faces for an automated Image Analysis'

1
Learning the Appearance of Faces for an automated
Image Analysis.
Thomas Vetter
University of Basel
Switzerland
http//informatik.unibas.ch
2
Change Your Image ...
3
Analysis by Synthesis
Computer Graphics can help to solve Computer
Vision!
4
Analysis by Synthesis
Image
3D World
Image Description
5
(No Transcript)
6
(No Transcript)
7
Synthesis of Faces
Modeler
Result
Input Image
8
Database of 3D Faces
9
Approach Example based modeling of faces
2D Image 2D Face
Examples
w1 w2
w3 w4
. . .
10
Cylindrical Coordinates
red(h,f) green(h,f) blue(h,f)
radius(h,f)
11
Morphing 3D Faces
1
__


2
12
Shape and Texture Vectors
Example i
Reference Head
13
Correspondence A two step process!

• Correspondence between
• two examples ( Optical Flow like algorithms).
• many examples ( Morphable Model )

14
Bootstrapping the Morphable Model
Reference Scan
15
Vector space of 3D faces.

• A Morphable Model can generate new faces.

a1 a2 a3
a4 . . .

b1 b2 b3
b4 . . .
16
Manipulation of Faces
17
Continuous Modeling in Face Space
18
Modelling the Appearance of Faces
A face is represented as a point in face space.

• Which directions code for specific attributes ?

19
Learning from Labeled Example Faces
Fitting a regression function
20
Facial Attributes
Gender
Original
21
3D Shape from Images
Input Image
3D Head
22
Matching a Morphable 3D-Face-Model

• R Rendering Function
• Parameters for Pose, Illumination, ...
• 
  
• Find optimal a, b, r !

23
Automated Parameter Estimation
150 shape coefficients ai 150 texture
coefficients bi

• Face Parameters

head position head orientation focal length

• 3D Geometry

• Ambient intensity, color
• Parallel intensity, color, direction
• Color contrast, gains, offsets

• Light and Color

24
Error Function
Minimize Image Differences
Maximize prior probability
25
Image Formation at each Vertex k
bi
ai
26
3D Shape from an Image
Input Image
27
Texture
Texture Extraction
28
Phong Illumination

• Ambient light ir, amb ig, amb ib, amb
• Parallel light ir, dir ig, dir ib, dir
  from direction l
• Lambertian shading and specular reflections
• Reflectivity at vertex i Ri Gi Bi ,
  shininess s
• surface normal ni , reflected ray ri
• vi normalized difference of position of camera
  and vertex

29
Illumination-Corrected Texture Extraction
Vertex i
Reflectance Ri Gi Bi
30
(No Transcript)
31
Recognition from Images
32
3D Computer Graphics
33
Optimization steps
Original
7 Feature PointsManually
34
Optimization steps
Original
StartAutomated Process
35
Optimization steps
Original
Fit to Featuresautomatically
36
Optimization steps
Fit Illuminationautomatically
Original
37
Reconstruction
Fitting the 4 individual face segments
automatically
Original
4.5 min on 2GHz Pentium 4
38
CMU-PIE database
39
Identification from Model Coefficients
Gallery

40
Correct Identification 1 out of 68 ()
gallery
profile
side
front
probe
99.8
front
99.9
side
98.3
profile
total
CMU-PIE database 4488 images of 68 individuals
3 poses x 22
illuminations 66 images per individual
41
Image Preprocessing for FRVT 2002
42
Image Preprocessing for FRVT 2002
43
Image Preprocessing for FRVT 2002
44
Segmenting hair a general requirement !
45
Try New Hairstyles
3D Shape and Texture
46
More Hairstyles
3D Shape and Texture
3D Angle, Position Illumination, Foreground,
Background
47
Novel Hairstyles
Customer
Hairstyle
48
V. Blanz, C. Basso, T. Poggio T.
Vetter Reanimating Faces in images and Video
Proc. of Eurographics 2003
49
Facial Attributes I Gender
male
female
50
Experiment I Hypotheses
Not only the gender but also the facial features
of a person affect gender-stereotypic
attributions.
H1 - Subjects rate the leadership aptitude of...
a) a man higher than of a woman. b) a
masculine person higher than of a feminine person.
H2 - Subjects rate the social competence
of... a) a woman higher than of a man. b) a
feminine person higher than of a masculine person.
51
Experiment I Results
feminine
masculine
Mean SC Mean LA
male
4.66 4.48
female
4.7 4.09
Mean SC 4.77 4.58Mean LA 4.25 4.32
52
References
http//informatik.unibas.ch
A Morphable Model for the Synthesis of 3D Faces.
Blanz, V. and Vetter, T. SIGGRAPH'99
Conference Proceedings, pp. 187-194, 1999. A
bootstrapping algorithm for learning linear
models of object classes. T. Vetter, M.J. Jones
and T. Poggio, IEEE Conference on Computer
Vision and Pattern Recognition -- CVPR'97, pp
40-46, Puerto Rico, USA, 1997. A Morphable
Model for face Recognition Volker Blanz and
Thomas Vetter IEEE PAMI (9), 2003
53
Acknowledgement
Volker Blanz Sami Romdhani Curzio Basso