LYU0603

1
LYU0603

• A Generic Real-Time Facial Expression Modelling
  System

Supervisor Prof. Michael R. Lyu Group
Member Cheung Ka Shun (05521661) Wong Chi Kin
(05524554)
2
Outline

• Previous Work
• Objectives
• Work in Semester Two
• Review of implementation tool
• Implementation
• Virtual Camera
• 3D Face Generator
• Face Animation
• Conclusion
• QA

3
Previous Work

• Face analysis
• Detect the facial expression
• Draw corresponding model

4
Objectives

• Enrich the functionality of web-cam
• Make net-meeting more interesting
• Users are not required to pay extra cost on
  specific hardware
• Extract human face and approximate face shape

5
Work in Semester Two

• Virtual Camera
• Make Facial Expression Modelling to be available
  in net-meeting software
• Face Generator
• Approximate face shape
• Generate a 3D face texture
• Face Animation
• Animate the generated 3D face
• Convert into standard file format

6
Review - DirectShow

• Filter graph
• Source ? Transform ? Renderer

7
Review - Direct3D

• Efficiently process and render 3-D scenes to a
  display, taking advantage of available hardware
• Fully Compatible with DirectShow

8
Virtual Camera

• Focus on MSN Messenger

9
Virtual Camera

• Two components
• 3D model as output
• Face Mesh Preview

10
Virtual Camera

• Actually it is a source filter

11
Virtual Camera

• Inner filter graph in virtual camera

12
Virtual Camera
13
Demonstration

• We are going to play a movie clip which
  demonstrate Virtual Camera

14
(No Transcript)
15
3D Face Generator

• Aims To approximate the human face and shape
• Comprise two parts

16
(No Transcript)
17
FaceLab

• Adopted from the face analysis project of Zhu
  Jian Ke, CUHK CSE Ph.D. Student
• The analysis is decomposed into training and
  building part
• The whole training phase is made up of three
  steps

18
FaceLab Data Acquisition

• To acquire human face structure data

• However, thousands of point are demanded to
  describe the complex structure of human face
• It can be acquired either by 3D scanner or
  computer vision algorithm

19
FaceLab Data Registration

• To normalize the 3D data into same scale with
  correspondences

• Problem
• The most accurate way is to compute 3D optical
  flow
• Commercial 3D scanners and 3D registration are
  computed with specific hardware

20
FaceLab

• To simplify the process, it is decided to use
  software to generate the human face data.
• Each has a set of 752 3D vertex data to describe
  the shape of face

21
FaceLab Shape Model Building

• A shape is defined as a geometry data by removing
  the translational, rotational and scaling
  components.
• The object containing N vertex data is
  represented as a matrix below

22
FaceLab Shape Model Building

• The set of P shapes will form a point cloud in
  3N-dimensional space which is a huge domain.

• A conventional principle component analysis (PCA)
  is performed.

23
FaceLab Shape Model Building PCA
Implementation

• It performs an orthogonal linear transform
• A new coordinate system which points to the
  directions of maximum variation of the point
  cloud.
• In this Implementation, the covariance method is
  used.

24
FaceLab Shape Model Building PCA
Implementation

• Step 1 Compute the empirical mean
• which is the mean shape along each dimension

25
FaceLab Shape Model Building PCA
Implementation

• Step 2 Calculate the covariance matrix C
• The axes of the point cloud are collected from
  the eigenvectors of the covariance matrix.

26
FaceLab Shape Model Building PCA
Implementation

• Step 3 Compute the matrix of eigenvectors V
• where D is the eigenvalue matrix of C
• The eigenvalue represents the distribution of the
  objects datas energy

27
FaceLab Shape Model Building PCA
Implementation

• Final Step Represent the resulted shape model as
• where ms are the shape parameters
• Adjusting the value of the shape parameters can
  generate a new face model by computing

28
FaceLab Shape Model Building PCA
Implementation

• An extra step Select a subset of the
  eigenvectors
• The eigenvalue represents the variation of the
  corresponding axis
• The first seven columns are used in the system
  and achieve a majority of the total variance.

29
FaceLab Render the face model

• The resulted data set is a 3D face mesh data
• Use OpenGL to render it

30
(No Transcript)
31
System Overview of Face Texture Generator
Facial Expression Modelling
Face Texture Generator
32
Face Texture Generator

• Face texture extraction
• Three Approaches
• Largest area triangle aggregation
• Human-defined triangles aggregation
• Single photo on effect face

33
Largest area triangle aggregation
Right face
Left face
Front face
34
Largest area triangle aggregation
35
Largest area triangle aggregation

• Result

36
Human-defined triangles aggregation

• Divide the face mesh into three parts
• Define the particular photo to be sampled in
  triangles in each region
• Reduce fragmentation

37
Human-defined triangles aggregation

• Redefine the face mesh Effect Face

38
Human-defined triangles aggregation

• Result

39
Single photo on effect face

• Similar to Human-defined triangles aggregation
• Use a single photo for pixel sampling
• Use Effect Face as outline

40
Single photo on effect face

• Result

41
Face Generator Filter
42
Dynamic Texture Generation

• To get back the rendered data from the video
  display card

43
Dynamic Texture Generation

• Lock the video display buffer

44
Dynamic Texture Generation

• Common buffer content is changed
• Update the texture buffer to reflect the changes
  immediately

45
Dynamic Texture Generation

• From 2D face mesh to 3D face mesh

46
Completed 3D Face Generator
47
Demonstration

• We are going to play a movie clip which
  demonstrates Face Generator

48
(No Transcript)
49
Face Viewer
50
Generate simple animation

• Looking at the mouse cursor

• Feature points provide sufficient information to
  locate the eye
• The two eyes will form a triangle planar with the
  mouse cursor

51
Generate simple animation

• Eye blinking

• One of the natural movements on a human face
• Adjust the vertex geometry on the eyelids
• The eyebrow is also needed to move backward

52
Generate simple animation
Smiling

• A lot of muscle needed to be modified
• The cheeks are being pushed up and wide
• The chin is being pulled down
• To change the shape of the lip

53
Convert into standard file format

• Save the mesh
• in .x file format
• Current selected
• face mesh data
• would be saved
• The Microsoft DirectX .x file format is not
  specific
• It can be used by any other application

54
Demonstration

• We are going to play a movie clip which
  demonstrate Face Animation

55
(No Transcript)
56
Conclusion

• We have achieved our goals
• Enrich functionality of webcam
• Make net-meeting become more interesting
• Extract human face and approximate face shape
• Discover the potential of face recognition
  technology

57
End

• Thank you!
• QA