Artificial Intelligence for Computer Animation

1
Artificial Intelligence for Computer Animation
2
Outline

• Virtual humans
• Body motion control
• Facial animation
• Body animation
• Behavioral animation

3
Acknowledgments

• References
• Handbook of virtual humans, Magnenat-Thalmann,
  N., and Thalmann, D., John Wiley Sons, 2004.

4
What are virtual humans?

• Virtual humans
• Human simulations for virtual environments
• Virtual environments comprise 3D graphics, 3D
  sounds, images, and videos in a passive mode.
• In interactive virtual environments, there is a
  need for virtual embodiments of the participant
  and the people.
• Virtual humans represent the virtual presenters,
  actors, players evolving in these virtual
  environments.

5
What are virtual humans?

• Virtual humans require
• High-level behavior (decision-making,
  intelligence, motivation, social behavior)
• Perception (virtual sensors in virtual worlds,
  real sensors in real worlds)
• Animation (flexible motion control)
• Graphics (realistic modeling and rendering of
  skin, hair, clothes, ).

6
What are virtual humans?

• Some important milestones and projects
• Early human modeling in the 50s (Boeings
  Landing Signal Officer), the 60s, and the 70s
  (Boeings First Man, Fourth Man and Woman) were
  developed for ergonomic studies.
• Short films and movies in the 80s (3D scan of
  Peter Fondas head and Adam the Juggler by Triple
  I, Dream Flight by Nadine Magnenat-Thalmann,
  Daniel Thalmann, and Philippe Bergeron, Tony de
  Petrie movie, ). Tin Toy (1988) won the first
  Best Animated Short Film Oscar award.Geris Game
  (1990) by Pixar won the Academy Award for
  Animated Short.
• Evolution towards real-time (Jack software from
  University of Pennsylvania).In the 90s
  evolution toward real-time animation and
  interaction in virtual worlds (Cybertennis with
  Marilyn as a referee, 1997, Nadine
  Magnenat-Thalmann, Daniel Thalmann).
• Now hundreds of researchers work in this area.

7
What are virtual humans?

• Applications of virtual humans are endless
• Virtual people for simulation-based learning and
  training (Example Just, virtual teachers, ).
• Virtual users for ergonomic analysis in work
  environments and vehicles, human factor analysis.
• Virtual patients for surgery, orthopedics,
  prosthetics, ....
• Virtual presenters for TV and the Web (Example
  SigningAvatar in schools).
• Virtual individuals in crowd simulations for
  emergency training.
• Virtual mannequins for clothing design (Example
  MyVirtualModel).
• Virtual actors for movies (Star Wars, Jurassic
  Park, Terminator 2, Forrest Gump, Titanic, Lord
  of the Rings ).
• Virtual patients for phobia treatment, virtual
  psychotherapies.
• Virtual inhabitants for city design and
  landscaping.
• Virtual ancient people for reconstructed cities,
  worlds, and cultural heritage (Example Namaz
  prayer).
• Virtual characters for computer games and
  casinos, interactive drama titles.
• Virtual athletes for sport simulation and
  teaching.
• Virtual soldiers for military simulations,
  training, and studies.
• Virtual participants in virtual conferences.
• Virtual employees for equipment design and
  maintenance, workers for industrial applications
  (Example lab training).

8
What are virtual humans?

• Challenges in virtual humans
• Representation of faces and bodies
• Reconstruction from 2-D photos
• Reconstruction from video sequences
• Construction base don laser technology
• Flexible motion control
• High-level behavior
• Emotional behavior
• Realistic appearance
• Interaction with the virtual world
• Interaction with the real world.

9
What are virtual humans?

• Example The Sims (Will Wright, Maxis,
  2000)Basically I want the games I program to
  surprise me, and The Sims does that on a daily
  basis. I see these hilarious things happen that
  were not explicitly programmed, but they were
  just a function of these simple rules
  interacting. And that to me is like the core of
  artificial life. The really interesting part
  about it is the emergent behavior.

10
What are virtual humans?

• From The Sims, Maxis

11
Outline

• Virtual humans
• Body motion control
• Facial animation
• Body animation
• Behavioral animation

12
Body motion control

• Body motion control has for goal to convey a
  sense of realism through the coordinated
  evolution of the character body.
• Realism means that the body motion is
  subjectively believable by the audience
  although not experimented through experimental
  validation.
• Other criteria user-friendliness, productivity,
  and performance.

13
Body motion control

• State of the art in 3D character animation
• The levels of abstraction of the musculo-skeletal
  system
• Two levels of motion control
• Primary motion is animation of the internal
  motion through a Multi-Body System (MBS)
• Secondary motion is animation of the external
  envelope and cloth
• Integrate the activity of all the muscles through
  their action lines.
• Integrate the motion control problems with soft
  tissue deformation.
• Solve the problem of bone surfaces contacts.
• Model all the tissues, including the bones, with
  finite elements too computationally intensive
  for most applications.

14
Body motion control

• Animation techniques and control of the
  Multi-Body system at the joint level
• Joint model
• Simplified joint models for motion control on the
  whole body.
• Motion is allowed by joints through a degree
  between the two body parts it connects, called
  the range of motion.
• Ideal kinematic models on the body provide useful
  data.
• Strength model
• Model the forces that the muscles apply on the
  bones.
• Measure of torque due to the group of muscles
  acting on a joint.
• Add the joint strength as well (velocity,
  position, adjacent joints).
• Fatigue model
• Model how strength decreases over time.
• Partial models only exist.

15
Body motion control

• Animation techniques and control of the
  Multi-Body system at the posture level
• Relates postures to the forces that can be
  exerted by an end-effector.

16
Body motion control

• Motion
• Motion is the change in the position of an object
  with respect to a reference.
• Mechanics is the part of physics that studies the
  motion of objects.
• Kinematics studies the geometry of motion
  regardless of its physical realization position,
  velocity, and acceleration.
• Dynamics studies the relation between the causes
  of motion (forces) and the acceleration of a body.

17
Body motion control

• Motion
• Two manipulation techniques
• Kinematic manipulation techniques
• Similar to manipulating am articulated doll
• Adjust the joints angles to get the posture we
  want
• Inverse kinematics problem deals with how to
  configure the joints in order to achieve the
  desired task (example to take an object).
• Kinematic animation techniques
• Generate believable motion, generally by skilled
  animators key frames for key postures, the others
  being interpolated.
• Another technique is the motion capture
  technique. Gives natural-looking motions, and is
  used a lot in the entertainment industry.

18
Body motion control

• Cybertennis (Magnenat-Thalmann Thalmann)

19
Body motion control

• Motion
• Balance control
• Not well understood how humans keep their
  balance.
• For static balance, the center of mass position
  is constrained by the angular values of the
  ankle, knee, and hip joints.
• Other approaches

20
Outline

• Virtual humans
• Body motion control
• Facial animation
• Body animation
• Behavioral animation

21
Facial deformation

• Facial animation relies on deformation models.
• Deformation model needs to give realistic and
  natural-looking faces.
• Models are based on the anatomy of the face.

22
Facial deformation

• Anatomy of the face
• Skin comprises several layers (epidermis on the
  surface, with underneath dermis and then
  hypodermis made of connective tissue) can move
  freely on bones.Motion made possible by collagen
  and elastin fibers.
• Muscles between the bone and skin, sometimes
  between skin only or bones only.Muscular
  contraction causes facial movement.All facial
  muscles have isotonic contractions, meaning that
  they cause shortening.Thin muscles, voluntary.
• Bones determine the shape of the face and head.

23
Facial deformation

• Face muscles from Sobotta, Atlas of Human Anatomy

24
Facial deformation

• Control parametrization
• Three main methods for parametrization
• Interpolationfinding an intermediate value
  between given end values extreme values being
  the expressions of the face.
• Facial action coding system (FACS)system
  describing all the possible visually
  distinguishable facial movements (Ekman), made of
  46 action units.
• Facial animation parameters (FAP)model to encode
  animation of faces reproducing expressions,
  emotions, and speech pronunciation, made of 68
  parameters.Facial animation table (FAT)
  describes which vertices are affected by a
  particular FAP and how.

25
Facial deformation

• Facial deformation models
• Shape interpolationrunning an interpolation
  function between the vertices corresponding to
  extreme poses or expressions of
  faces.limitation range of expressions,
  data-intensive.

26
Facial deformation

• Facial deformation models
• Parametric modelappeared later as an improvement
  over interpolation method.a set of parameters
  permits to morph a set of polygons.small set of
  parameters for large range of faces.model
  dependent on facial topology.limitation set of
  parameters limited, not available for all facial
  expressions.

27
Facial deformation

• Facial deformation models
• Muscle-based modelsmodels based on simplified
  structures of bones, muscles, skin and connective
  tissue. simulate the characteristics of facial
  muscles.two main methods
• Physics-based muscle modeling deformation
  mechanics.
• Pseudo-muscle modeling visual muscle simulation.

28
Facial deformation

• FAT-based deformationFAPs are defined as
  motions of feature points. Points are grouped
  in transform groups, to which geometrical
  transformations are applied.(from Ostermann,
  1998)

29
Facial deformation

• MPEG-4 high-level expressions (from Ostermann,
  1998)

30
Outline

• Virtual humans
• Body motion control
• Facial animation
• Body animation
• Behavioral animation

31
Body animation

• Body animation can be achieved by three types of
  models
• Surface models, which refer to models relying on
  two layers only a skeleton and the skin
• Volumetric models, which refer to models composed
  of different geometric primitivves
• Multi-layered models, which refer to models
  relying on more than two layers a skeleton,
  muscles, and the skin at minimum

32
Body animation

• Surface models
• Rely on two layers only
• An articulated structure or skeleton
• An external envelope or skin.
• External envelope follows deformation of the
  skeleton
• Skin model can be modeled as a set of independent
  polygonal meshes added on top of the skeleton and
  anchored to it so that joint motions also move
  the skin.
• Surface deformation is achieved through several
  types of local surface operators.

33
Body animation

• Surface models
• Local surface operators mathematical continuous
  function that links the skin deformation to the
  joint values
• Example Marilyn, Magnenat-Thalmann et al., 1988
• Limitations
• Each joint type needs a specific function
• Mathematical functions cannot reproduce all the
  variability found in natural anatomy
• Not easy to control by the animator.

34
Body animation
(from Marilyn, Magnenat-Thalmann et al., 1988)
35
Body animation

• Surface models
• Other surface models
• Skinning an algorithm that is capable of
  adapting to all kinds of joints. Called Smooth
  binding in Maya.
• Contour deformation approximate the different
  parts of the human body by cylinders, and deform
  the cylinders.
• Deformation by example deform by blending
  predefined examples called key-shapes, which
  consist of triangle meshes in various skeletal
  poses.

36
Body animation

• Volumetric models
• Composed using different primitive shapes such as
  ellipsoids and spheres
• Implicit surfaces have been used for organic
  and body modeling for a long time.An implicit
  surface is a surface defined by a function such
  as f(x, y, z) constant.Are advantageous for
  body modeling because they blend smoothly into
  one another, and because they require little
  storage on the computer. Can be mixed with
  surface deformation models.Limitations
• Unwanted blending.

37
Body animation

• Volumetric models
• Collision models allow realistic handling of
  collisions between different models or their
  parts.

38
Body animation

• Multi-layered models
• Most widely used approach to body deformation
• Composed of several layers
• Skeleton layer
• Muscle layer
• Fat layer
• Skin layer.

39
Body animation

• Multi-layered models
• Skeleton layer - an articulated structure
  composed of a hierarchy of segments or
  parallepipedes sometimes covered by bones
  represented by geometric primitives or triangle
  meshes.
• Muscle layer modeled first through Free Form
  Deformations (FFD) , or implicit surfaces.
  Fusiform muscles are well represented by
  volume-preserving ellipsoids.More complicated
  muscles are represented by cylinders and their
  deformations, sets of centroid curves and
  vertex-based shapes, or B-spline volumes.

40
Body animation

• Multi-layered models
• Fat layer most often not represented or blended
  in the muscle layer. Some increase the thickness
  of the skin.
• Skin layer has been modeled by many types of
  surfaces polygonal, parametric (B-spline
  piecewise parametric polynomial curve-, Bezier
  patches, ), and implicit.

41
Body animation

• Multi-layered models
• Three ways of deforming the skin
• Surface deformation models deformation
  propagated from the skin to the other layers.
• Skin deformation by a mechanical model and skin
  stays at a distance from the underlying layers.
• Skin is the surface of a volume finite element
  model of the body.
• Examplesarm bend (deforming metaballs)leg bend
  (deforming metaballs)muscle building
  (anatomically-based muscle deformation)muscle
  scaling (semi-automatic muscle scaling)

42
Body animation
(from muscle building, Aubel, 2002, each muscle
is parametrized and deformed by a set of
centroid curves)
43
Outline

• Virtual humans
• Body motion control
• Facial animation
• Body animation
• Behavioral animation

44
Behavioral animation

• Behavioral animation refers to modeling the
  behavior of characters in animations
• Animation is a bottleneck in production of movies
  and video games.
• Virtual humans are able to perceive their
  environment and able to react and take decisions.
• Typical computer animation models only the shape
  and physical properties of the characters,
  whereas behavioral or character-based animation
  seeks to model the behavior pf the character
  (Reynolds, 1987)

45
Behavioral animation

• Examples
• Games
• Game Creatures used a neural network to teach
  the fantasy creature how it should behave.
• Game The Sims designed a simulation of virtual
  humans in their everyday life.
• Game Black and white.
• Animation platforms such as Improv system, Motion
  Factorys Motivate, Virtools NeMo.
• Virtual reality
• STEVE (Rickel and Johnson, 1997) an immersive VR
  application for peace-keeping in Bosnia.
• Virtual storytelling
• OZ project (Bates, 1994, Carnegie Mellon
  University).

46
Behavioral animation

• Behavior refers to how an individual responds to
  the stimulations from the environment.
• State-of-the-art
• Perception
• Synthetic vision (Renault et al. 1990) provides a
  technique for the virtual human to perceive
  objects. Objects are color-coded with a table of
  correspondence between objects and colors, for
  fast recognition in images, and memorization by
  the agent of the objects and the environment.
  Requires a lot of memory, not suitable for
  real-time animations.
• Geometric vision (Bordeux et al. 1999) is less
  complex, and often used in real-time
  applications. A perception filter receives
  information from the objects in the scene and
  decides to let it in or not (considerations of
  distance for example). Several other filters
  screen different aspects of the input.

47
Behavioral animation

• State-of-the-art
• Defining behaviors
• An agent goes through the steps of perception of
  the environment, selection of the next action,
  and reaction. Different action selection models
  have been proposed.
• Planning - an agent possesses generic plans with
  preconditions to trigger them, and
  conclusions.Pattern-matching refers to
  determining which of these generic plans or rules
  are applicable in the current situation of the
  agent.
• Machine learning an agent can learn new
  knowledge or information with machine learning
  (decision trees, artificial neural networks,
  genetic algorithms, ..).

48
Behavioral animation

• State-of-the-art
• Defining behaviors
• Motivated action selection mechanism - agent has
  motivations (eat, drink, work, ), and chooses
  the best action for his/her survival in the
  environment. Apply different rules to select
  which action is best in the current situation,
  for example Prefer to choose physiological
  actions over locomotion actions (Tyrrel, 1992).
• Hybrid approaches
• Interactions represent non-verbal
  communication, such as walking sadly, so that the
  other agents perceive others feelings. Represent
  also verbal communication.
• Animation represent the capability of the
  virtual agents to handle several tasks at the
  same time.

49
Behavioral animation
(from Agents Common Environment, each agent is
loaded in a thread.)