Geometric Algorithms in 3D Real-Time Rendering and Facial Expression Modeling (Parts II and III)

1
Geometric Algorithms in 3D Real-Time Rendering
and Facial Expression Modeling (Parts II and III)

• Prof. Marina L. Gavrilova
• University of Calgary

2
Where we are

• Brief Overview of the Area
• Geometric Approaches for mesh reconstruction and
  image morphing
• Real-Time Optimally Adapting Mesh Algorithms
• Non Photo-realistic Facial Expression Modeling
  and Morphing Algorithms
• Swarm Real-time Visualization System

3
Part II

• NPR Face Rendering

4
Terminology

• DT Delaunay triangulation
• VD Voronoi Diagram
• FT Feature Transform
• DT Distance Transform

5
Preliminaries

• Transformation from a photograph into an artistic
  work

Li and Huang 03
6
Preliminaries

• Transformation from an Excel into Artistic Work ?

7
NPR vs. PR philosophy

• PHOTOREALISTIC
• GOALS imitate as realistically as possible
  facial expression and details
• Methodology
• complex geometry fitting,
• generating high density meshes,
• recovering camera positions,
• utilizing data interpolation,
• simulating light conditions,
• applying texturing and other techniques
• Outcome
• Very costly processing, highly realistic
  visualization

• NON-PHOTOREALISTIC
• GOALS imitate emotions as closely as possible,
  while at the same time allowing the model its own
  identity and artistic representation
• Methodology
• Utilize artistic drawings
• Feature line extraction
• Expression morphing
• Model partitioning onto regions
• Outcome
• Expression and emotion preservation, efficient
  storage, fast processing, artistic visualization

8
Animation Trivia

• Face models used in Pixars Toy Story had
  approximately how many control points each?
• 100
• 1000
• 10000
• None they were made by hand out of clay!

9
Animation Trivia

• TOY STORY" used how many SUN workstations to
  render images?
• 1
• 100
• 1000
• None they were all IBM workstations!

10
Animation Trivia

• Pixar have received how many Academy Awards
  from the Academy of Motion Picture Arts and
  Sciences, including an Oscar for Best Short
  Animated Film for 'Tin Toy' and a Technical
  Achievement Award for Pixar's RenderMan
  software?
• 2
• 6
• 12
• One for every animated character created!

11
NPR vs. PR

• PHOTOREALISTIC - Final Fantasy (from Final
  Fantasy website)

NON-PHOTOREALISTIC - Shrek 10 to 1 preference
Crawford (Shrek movie web site)
12
Objectives of our Research

• To develop efficient and realistic portrait
  generation method from 2D photographs
• To develop fast and easy 2D morphing algorithm
• To create appropriate 3D mesh representation for
  model morphing

13
Developed Methodology

• Proposed to use NPR modeling by example for
  portrait generation
• Applied binarization method to produce black and
  white images from photograph
• Developed segmentation technique for detailed
  feature extraction
• Used combined edge eextraction strategy
• Used application-dependent style for rendering
• Developed Euclidean distance transform based
  algorithm for morphing

14
Facial Modeling Research

• Focus on Human Faces

Lin and Hong 02
Li and Kobatake 97
15
Facial Modeling Research

• Other examples

Lin and Hong 02
Gooch 04
Hong 04
16
Motivation

• Shortcoming of some previous works
• Facial creases, wrinkles and freckles are hard to
  represent
• Detail feature lines of different expressions and
  which lines?
• Efficient morphing

17
Contribution

• Task Generate a smooth line-drawing animation
  from two input photographs
• Input/Output two human face images as input and
  output of a NPR facial animation
• Methodology
• Create NPR style portrait from a frontal facial
  photograph to generate a digital artwork.
• Use segmentation and tracking methods to map
  expressive lines representing expressive
  wrinkles and creases into the image.
• Utilize morphing algorithm based on Euclidean
  distance transform that does not require control
  points or control lines, and finds pixel
  correspondence automatically.
• Use the developed system as a prototype for 3D
  morphing based on controllable mesh deformations
  utilizing Delaunay triangulation.

18
System FlowChart
19
Methodology Use artists rendering as a template
Six Primary Facial Expressions drawn by
artistsanger, disgust, fear, happiness, sadness
and surprise
20
Methodology Portrait generator

• Producing black-and-white images using
  binarization

21
Methodology Details Extraction
Enlarge certain areas
Edge Detection
Delete normal contours
Detail feature lines tracking
Save feature Lines information
22
Methodology region subdivision

• Five interested sub-regions are used to extract
  details

23
Methodology edge detection

• Edge Detection Process Sobel, Prewitt, Roberts,
  LoG, Canny operators (from (b) to (f)), g
  combination of e and f

24
Methodology details extraction

• Close-up Details Extraction feature

25
Methodology feature line tracking

• Feature Lines Tracking

26
Methodology feature line tracking

• Tracking Strategy

27
Methodology feature line tracking

• Data Structure to store the feature lines

28
Methodology Distance transform for Morphing
Image
Template
29
Distance Transform
Given an n x m binary image I of white and black
pixels, the distance transform of I is a map that
assigns to each pixel the distance to the nearest
black pixel (a feature).
30
Feature Transform
The feature transform of I is a map that assigns
to each pixel the feature that is nearest to it.
31
A Fast Algorithm for Computing the Euclidean
Distance Transform

• Works in Euclidean Metric
• Optimal linear in the number of pixels
• Proceed in two passes over the image, top down
  and bottom up
• For each row
• Maintain a polygonal chain - a set of pixels in
  which the nearest feature may lies
• Updates the chain for next row by pruning - it
  becomes the minimal set of pixels
• Prune Endpoints and Internal Vertices

32
Chain Definition

• One Chain for the row
• Contains at most one pixel per column
• Contains the lowest pixel in the column
• After pruning, it contains only those points that
  will be nearest features to some points in the
  row
• Dynamically updated from row to row

33
Example of a a Chain
34
Pruning Endpoints

• Identifying superfluous endpoints

35
Pruning Internal Points

• Point q is removed

36
Pruning Internal Points

• Point q remains

37
Results on NPR Rendering
38
Experimental results for distance transform
algorithm
Algm 1 with pruning was tested Saturation levels
5, 15, 30 Results Linear Running Time,
Regardless of Saturation ( of black pixels in
image)
39
Morphing Direction
40
Results of utilizing Distance Transform for
Morphing
Starting Frame
Ending Frame
41
Results of feature lines extraction
42

• Line Simplification and Rendering Results

43

• Line Simplification and Rendering

44
Animation
45
3D Facial Model Synthesis
46
Brief Preliminaries

• Over the last 30 years, thousands of papers were
  devoted to 3D face animation and modeling.
• Different classification and registration
  techniques have been developed, including those
  based on 2D feature matching, multiple point of
  view registration (using calibrated cameras),
  nearest-neighbor feature comparison,
  segmentation-based feature extraction, ICP
  (Iterative Closest Point-based Matching), TPS
  (Thin Plates Spline) algorithms and others.

47
3D Facial System Classification(by Akarun,
Gokberk, Salah)
48
3D Facial System Classification(by Akarun,
Gokberk, Salah) cont.
49
3D Facial System Classification(by Akarun,
Gokberk, Salah) cont.
50
3D Facial System Classification(by Akarun,
Gokberk, Salah) cont.
51
Voronoi Diagram for 3D face modeling

• Voronoi Diagram is a partitioning of the space
  into convex regions such that any point within
  the region is closer to the site that originates
  this region than to any oethr site in the space.
• Delaunay Triangulation is the dual structure that
  connects those sites whose Voronoi Regions share
  an edge (in 2D) or a face (in 3D).

52
Methodology

• Utilize geometric concept Voronoi diagram in
  application to FFD (free-form deformation) model.
  
• Develop a method for 3D facial model synthesis
  that combines the traditional free-form
  deformation (FFD) model with techniques of data
  interpolation based on Voronoi diagrams.
• With 18 feature points extracted from 2D facial
  images in two orthogonal views, Dirichlet Free
  Form Deformation (DFFD) is utilized for modifying
  a generic 3D face to produce the individual face.
  
• The main advantages of this approach over former
  extensions of FFD is in removing the constraints
  on control lattice and control points location.
• By assigning different weights to those control
  points, we make the method more adaptable to the
  facial deformation. The reconstructed 3D faces
  can be used to generate different facial
  animations.

53
Methodology

• Build DFFD model
• Compute Sibson coordinates using space
  partitioning introduced by Voronoi diagram
• Compute deformation in 3D using weighted DFFD
  relationship based on computed Sibson coordinates
• Adapt approach for face synthesis using Delaunay
  triangulation

54
Illustration

• 28 zones (below).

• Control points (above).

55
Illustration

• FaceGen generic mesh and texturing result

56
Illustration

• Original face (left) and generated face (right)

57
Extensions

• The developed methodology is currently being
  utilized for 3D face morphing and expression
  synthesis.

58
Part III Strategic Modeling
59
Problem Definition

• Given a set of unsophisticated agents and their
  behavior.
• Examples ants, bees, fish, animals, robots
• Complex behavior can emerge as a result of
  collective intelligence

60
Our Methodology

• Stigmergy (approach where agents interact with
  environment) is usually realized through
  neighborhood cohesion (Boids flocking) or through
  chemical traces modeling.
• We introduce a novel behavioral Genotype encoding
  with novel agent Classification Genetic Algorithm
  and more complex set of rules determining missile
  as well as target behavior.
• We also test it in a complex 3D environment.

61
Illustration Missile Basic Maneuvers
XF
F
U
N
LU
A
62
Rendering and Physical Engine

• Regular physics engine will not suffice
• Approximation aggravates trajectory computation
• Construct original visualization system
• Advanced look-ahead estimation based physics
  engine
• Robust Rendering engine
• Anisotropic Texture filtering
• Multiple LOD based geometry rendering
• Particle engine
• Highly optimized exclusive adaptive pattern
  identification
• Flexible and robust system

63
The Simulation Engine

• Robust design Separation of Rendering modules
  from the simulation
• Implement Command Console
• Runtime performance is highly efficient
• For 50 missiles
• Full quality rendering at 50FPS (better than most
  oethr results reported)
• Simulation runs up to 50 times faster (FPS2200)
  if rendering is turned off (for evolutionary
  algorithm)
• Excellent Rendering quality (anisotropic texture
  mapping, particle engine are utilized)

64
More Results on Strategic Modeling

• See Animation Demo

65
Conclusions

• Presentation discussed two systems developed for
  real-time massive 3D terrain rendering and for
  NPR facial expression modeling. In both cases,
  efficient and robust results are achieved through
  the utilization of computational geometry
  algorithms.
• It is utilized in other practical 3D modeling
  applications
• collision detection optimization project in
  collaboration with Prof. Ivana Kolingerova,
  Computational Geometry Group (University of West
  Bohemia, Czech Republic),
• protein modeling project carried out with Prof.
  Sergei Bereg (University of Texas at Dallas,
  USA),
• material structure modeling project together with
  Prof. Nikolai Medvedev, Novosibirsk State
  University, Russia.

66
Acknowledgements NSERCGranting Agency,
CanadaGEOIDENetwork of Excellence,
CanadaCanadian Foundation for Innovation All
students of SPARC LAB and BT Lab, University of
Calgary, in particularly Russel Apu and Luo Yuan
CollaboratorsAudience