A Study on the 3D Facial Feature Extraction and Motion


1
A Study on the 3D Facial Feature
Extraction and Motion Recovery
using Multimodal Information
Sang Hoon Kim Hankyong National Univ. Korea
south [e-mail] http//auto.hankyong.a
c.kr/kimsh
2
Experience
19891994 Researcher Institute of LG
Semiconductor co.Ltd 19941999 Research
Assistant Dept. of Electronics Korea
University 19992001 Senior Research Scientist
Imaging and Media Research Center
KISTKorea Institute of Science and
Technology 1999 present Assistant professor in
Dept. of Information and Control
Hankyong National University
Kyonggido Korea
Research Interests
Face detection and tracking in real time Pose
estimation Multimodal fusion for object
detection and recognition
3
Problems are
face detection

• How to detect facial area exactly from complex
  background
• in variable conditions.
• How to define and extract facial features
  locationand how many
• features are proper to do next step
• How to recover 3D facial features and global
  motion from
• the 2D information.
• How to combine the whole results to synchronize a
  synthetic
• face model with a real face.

global motion recovery and application
Evaluation method is important
to prove the accuracy of 3D
recovered information
4

• Categorization of methods for face detection
• Knowledgebased topdown methods
• Bottomup feature based methods this work
  classified here from the survey
• Template matching
• Appearancebased methods

Detecting Faces in Images A Survey IEEE
Trans. on PAMI vol.24 no.1 Jan 2002 MingHsuan
Yang David J. Kriegman Narendra Ahuja
5
Overall Blockdiagram
6
Generalized Skin Color DistributionGSCD
The skin color distribution in the
normalized rg domain can be
approximated by the 2D Gaussian distribution
GSCD in rg domain
7
GSCD color transform
1. Color Transform defined as
Zxy GSCD rxy gxy
2. Result intensity value of
enhanced facial regions suggests
the possibility of being facial color
GSCD
input color image Icxy
gray level image Zxy
8
LoG and MPC Disparity Map
Left camera
Image grab
Enhanced image
Disparity map
Right camera
LoG
MPC stereo matching
Disparity Dxy is
Vth predefined threshold value
9
MPC similarity measure
disparity map results with Randomdot stereo
image a left image b right image c SAD d
NCC e MPC similarity measures
matching ratio comparison when noises are
increased 0 20
10
MPC similarity measure
11
2Disparity HistogramDH
1. Definition occurrence frequency of each
disparity

value obtained from MPC disparity map 2.
Meaning the locations and number of objects

12
Range Segmentation
1. The outline of the DH curve is smoothed
with average filtering 2. Region having
the smallest disparity value is defined as
background Although it has skincolored
components they can be regarded as noises
or useless data 3. Regions having frequency
values less than a specified threshold are
assumed to have no object and merged to
background region 4. Regions having continuous
disparity values greater than a specified
threshold are defined as objects with same
label
13
Moving Color Enhancement 1
This idea based on the fact that regions
detected as faces only using skin color are
unstable and have variable skin color space due
to the lighting condition so adding the motion
information to the skin color enhanced region
would be effective so as to increase the
probability of faces So given an GSCD
transformed image region that has a low gray
value which means low probability of face
should be detected as a face only when it has
large motion while region that has high gray
level which means high probability of face can
be detected as a face even when it has small
motion.
14
Moving Color Enhancement 2
The SWUPCSigmoid function Weighted Unmatched
Pixel Count operation emphasizes only the region
with a motion in the skincolor enhanced region.
where
15
Sigmoid Function
16
Moving color enhanced results
17
Face Detection Results using multimodal fusion
18
No Transcript
19
No Transcript
20
No Transcript
21
No Transcript
22
No Transcript
23
Test image 6
face region detection result using rangeskin
color and motion information ainput color
imaget1binput color imaget2 cMPC
disparity mapdrange segmented imagegskin
color transformed image hswupc transform image
ifinal face area
24
FDP of MPEG4 snhc vs facial feature set for
the work
23
74
25
Automatic Facial Features Extraction eyes
eyebrows
26
Automatic Facial Features Extraction eyes
eyebrows
27
Paraperspective Camera model
? why use? This model can describe the change
of objects depth information
Refer to A Paraperspective factorization
method for shape and motion recovery Conrad
J.Poelman and Takeo Kanade
28
Sequential SVD Process
Matrix time series
29
Pyramid type 3D test object
30
Multiview Synthetic Pyramid test images
31
Multiview Synthetic Pyramid test images
32
Shape Recovery Results using synthetic 3D shape
33
Motion Recovery Results using synthetic 3D shape
34
Motion and Shape Recovery Results using
synthetic face image synchronized with real face
35
Real facial information Face Animation
Parameter
36
Conclusions
1. This technique works well when detecting faces
using GSCDGeneralized Skin Color
Distribution combined with the
Range Segmentation using MPCMatching
Pixel Count DHDisparity Histogram
and Objects moving
color informationSWUPC 2. Automatic facial
features extraction95 using moving color
informationSWUPC BWCD and morphological
process. 3. Motion and 3D facial features
recovery80 208 of 210 frames using
multiview paraperspective model and
factorizationSVD method
4. The results can be used for various
applications