A Study on the 3D Facial Feature Extraction and Motion


1
A Study on the 3D Facial Feature
Extraction and Motion Recovery
using Multi-modal Information
Sang Hoon Kim Hankyong National Univ. Korea
south wind1104_at_hanmail.net http//auto.hankyong.a
c.kr/kimsh
2
Experience
1989-1994 Researcher Institute of LG
Semiconductor co.Ltd 1994-1999 Research
Assistant Dept. of Electronics Korea
University 1999-2001 Senior Research Scientist
Imaging and Media Research Center
KIST(Korea Institute of Science and
Technology) 1999- present Assistant professor in
Dept. of Information and Control
Hankyong National University
Kyonggi-do Korea
Research Interests
Face detection and tracking in real time Pose
estimation Multi-modal fusion for object
detection and recognition
3
Problems are
face detection

• How to detect facial area exactly from complex
  background
• in variable conditions.
• How to define and extract facial features
  location(and how many
• features are proper to do next step)
• How to recover 3D facial features and global
  motion from
• the 2D information.
• How to combine the whole results to synchronize a
  synthetic
• face model with a real face.

global motion recovery and application
Evaluation method is important !!
to prove the accuracy of 3D
recovered information
4

• Categorization of methods for face detection
• Knowledge-based top-down methods
• Bottom-up feature based methods (this work
  classified here from the survey)
• Template matching
• Appearance-based methods

Detecting Faces in Images A Survey IEEE
Trans. on PAMI vol.24 no.1 Jan 2002 Ming-Hsuan
Yang David J. Kriegman Narendra Ahuja
5
Overall Blockdiagram
6
Generalized Skin Color Distribution(GSCD)
The skin color distribution in the
normalized (rg) domain can be
approximated by the 2D Gaussian distribution
GSCD in (rg) domain
7
GSCD color transform
1. Color Transform defined as
Z(xy) GSCD ( r(xy) g(xy)
) 2. Result intensity value of
enhanced facial regions suggests
the possibility of being facial color
GSCD
input color image Ic(xy)
gray level image Z(xy)
8
LoG and MPC Disparity Map
Left camera
Image grab
Enhanced image
Disparity map
Right camera
LoG
MPC stereo matching
Disparity D(xy) is
Vth predefined threshold value
9
MPC similarity measure
disparity map results with Random-dot stereo
image (a) left image (b) right image (c) SAD (d)
NCC (e) MPC similarity measures
matching ratio comparison when noises are
increased ( 0 --- 20 )
10
MPC similarity measure
11
2Disparity Histogram(DH)
1. Definition occurrence frequency of each
disparity

value obtained from MPC disparity map 2.
Meaning the locations and number of objects

12
Range Segmentation
1. The outline of the DH curve is smoothed
with average filtering 2. Region having
the smallest disparity value is defined as
background (Although it has skin-colored
components they can be regarded as noises
or useless data) 3. Regions having frequency
values less than a specified threshold are
assumed to have no object and merged to
background region 4. Regions having continuous
disparity values greater than a specified
threshold are defined as objects with same
label
13
Moving Color Enhancement 1
This idea based on the fact that regions
detected as faces only using skin color are
unstable and have variable skin color space due
to the lighting condition so adding the motion
information to the skin color enhanced region
would be effective so as to increase the
probability of faces So given an GSCD
transformed image region that has a low gray
value which means low probability of face
should be detected as a face only when it has
large motion while region that has high gray
level which means high probability of face can
be detected as a face even when it has small
motion.
14
Moving Color Enhancement 2
The SWUPC(Sigmoid function Weighted Unmatched
Pixel Count) operation emphasizes only the region
with a motion in the skin-color enhanced region.
where
15
Sigmoid Function
16
Moving color enhanced results
17
Face Detection Results using multi-modal fusion
18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
Test image 6
face region detection result using rangeskin
color and motion information (a)input color
image(t1)(b)input color image(t2) (c)MPC
disparity map(d)range segmented image(g)skin
color transformed image (h)swupc transform image
(i)final face area
24
FDP of MPEG4 snhc vs facial feature set for
the work
23
74
25
Automatic Facial Features Extraction - eyes
eyebrows
26
Automatic Facial Features Extraction - eyes
eyebrows
27
Paraperspective Camera model
why use This model can describe the change
of objects depth information
Refer to A Paraperspective factorization
method for shape and motion recovery Conrad
J.Poelman and Takeo Kanade
28
Sequential SVD Process
(Matrix time series)
29
Pyramid type 3D test object
30
Multi-view Synthetic Pyramid test images
31
Multi-view Synthetic Pyramid test images
32
Shape Recovery Results using synthetic 3D shape
33
Motion Recovery Results using synthetic 3D shape
34
Motion and Shape Recovery Results using
synthetic face image synchronized with real face
35
Real facial information -- Face Animation
Parameter
36
Conclusions
1. This technique works well when detecting faces
using GSCD(Generalized Skin Color
Distribution) combined with the
Range Segmentation using MPC(Matching
Pixel Count) DH(Disparity Histogram)
and Objects moving
color information(SWUPC) 2. Automatic facial
features extraction(95) using moving color
information(SWUPC BWCD) and morphological
process. 3. Motion and 3D facial features
recovery(80 208 of 210 frames) using
multi-view paraperspective model and
factorization(SVD) method
4. The results can be used for various
applications