Realtime Facial Expression Recognition using Active Appearance Model and Multilayer Perceptron

1
Realtime Facial Expression Recognition using
Active Appearance Model and Multilayer Perceptron

• Hyun-Chul Choi
• Intelligent Robotics Laboratory
• POSTECH
• The 6th POSTECH-KYUTECH Joint Workshop

2
Outline

• Face System Overview
• AdaBoost Face Detection
• AAM with Second Order Minimization
• 2D 3D AAM Face fitting
• Face Recognition using MLP
• Conclusion

3
Face System
Identity, Expression
Facial components
Position, angle, scale
Position, angle, scale
Detection
Tracking
Fitting
Recognition
Active Appearance Model
MLP
AdaBoost
2D position, Scale, Rotation (all-axis)
Exact shape and position of all facial components
in 3D
Easy to train but needs so many samples
2D position, angle, scale
4
AdaBoost Face Detection
subtract
add
classifier

face
Search All positions, scales and angles !
5
Active Appearance Model

• Appearance Shape Texture

Shape PCA
1. Shape analysis
Shape model
texture PCA
2. Texture analysis
Warping to mean shape
Texture model
Appearance
6
Second Order Minimization

• AAM gradient descent
• Previous AAM use first order derivative
• - risk of local minimum, divergence and slow
  convergence
• Second order minimization
• Ezio Malis (I.N.R.I.A., France), Improving
  vision-based Control using Efficient second-order
  minimization techniques, ICRA2004.
• - more accurate and fast convergence

7
Second Order Minimization
First-order minimization
second-order minimization
8
3D shape model from 2D
Iain Mattews, Jing Xiao, and Simon Baker, On the
Dimensionality of Deformable Face Models,
Technical Report CMU-RI-TR-06-12, Carnegie Mellon
University, Robotics Institute, 2006.
decompose
Find G
2D shape samples
Projection matrix ? 3D shape model
9
3D shape model from 2D
3D mean shape constructed by using about 2000 2D
shape samples
10
2D3D AAM Face fitting
2D model
Use the Steepest descent method
3D shape model
Find out the parameters to minimize the cost
Cost function
11
Face Database

• Face images of 10 persons obtained in office
  environment
• Training image
• - 5 poses for each person (front, left, right,
  up, down)
• - changes in facial components (eye and mouth
  open close)
• - 15 images 15 ground truth of 66 feature
  points / person
• Test image
• - an image sequence of 200 frame for each person

12
Face DB ??
13
Face DB ??
14
Face Recognition Identity
Training use all the training image in DB Input
texture shape
STD PCA
2?66
Normalized shape
60?60
Normalized texture
Neural Network
AAM fitting
of hidden layer 15
15
Face Recognition Identity
out
in
Recognition performance 99.75 (error 0.25)
16
Face Recognition Identity
Training training images of 3 front pose in
DB ? Need pose transformation! Input texture
shape
STD PCA
Neural network1 pose detection of hidden
layer 15
2?66
Normalized shape
Pose i
STD PCA
60?60
Wixbi
Normalized texture
AAM fitting
Neural network2 identity recognition of hidden
layer 15
Pose transformation
17
Face Recognition Identity
Pose transformation ith pose to front
Front to front
W1xb1


down to front
Mean front images of 10 person
W5xb5
Find Wi and bi by least squares method
18
Face Recognition Identity
Examples of Pose transformation
front to front
right to front
left to front
up to front
Posed image
front image
transformed image
down to front
19
Face Recognition Identity
The case of training with front images without
pose transformation
The case of training with front images with pose
transformation
out
out
in
in
Recognition performance 84.45
Recognition performance 95.1
20
Face Recognition Expression
Expression
PCASTD
10000 neutral 01000 happy 00100 sad 00010
angry 00001 surprise
MLP7-5-5
Neural network for expression recognition
Size 32 by 32
400 samples for 5 expressions of a person using
AAM fitting Training data randomly selected 100
samples Test data the other 200 samples
21
Face Recognition Expression

• Result for test samples

Missed samples
22
Conclusion

• Second Order Minimization technique can improve
  AAM convergence rate at the cost of computational
  efficiency ? no overburden
• MLP could be a Easy Recognizer for face and
  facial expression with well normalized inputs
  obtained by warping the texture tracked by AAM
  tracker
• This approach could be a Unified Face System with
  other tracking technique such as mean-shift
  algorithm which can reduce computational cost by
  substitute AAM when our attention is not need.