Segmentation and tracking of the upper body model from range data with applications in hand gesture

1
Segmentation and tracking of the upper body
model from range data with applications in hand
gesture recognition

• Navin Goel
• Intel Corporation
• Department of Computer Science, University of
  Nevada, Reno

2
Overview

• Introduction
• Overall System
• Upper Body Model
• Segmentation Problem
• Tracking
• Color Based Segmentation
• Results
• Conclusion and Future Work
  

3
Introduction

• Applications 3D editing system/ HCI
  systems,
• American
  Sign Language Recognition,
• 
  Entertainment,
• Industrial
  Control,
• Video
  coding, teleconferencing
• Requirements Background and illumination
  independent,
• Occlusions
  and self occlusions of the body components,
• Robust hand
  free initialization,
• Robust
  tracking.

4
Overall System
Stereo (RGBZ) video sequence
Invalid Track
Initial Segmentation
Tracking
Valid Track
Train Reco
Upper Body Model
Color video sequence
Hue Moments Calculation
Color-based segmentation
5
Upper Body Model
LHa
LHe
LU
LHa
LT
LF
LF
L
LU
Wl
El
Sl
N
Sr
Er
Wr
J
He
T
Hal
Fl
Ul
Ur
Fr
Har
C
O
Oij
6
Upper Body Model
Head Normal component model
SizeHead
Neck
Neck
Planar component model
WidthTorso
Elbow
Linear component models
Wrist
7
Upper Body Model
Linear PDF Parameters
are the spherical coordinates of Jc with the
origin in Jp
Where,
The conditional probability of a joint Jc given
its parent joint Jp and the anthropological
measure L is given by
represent the
Where, KJc is a normalization constant,
minimum and maximum values of parameters
8
The Segmentation Problem
Looking for all possible joint configuration is
computationally impractical. Therefore,
segmentation takes place in two stages.
Stage I
Stage II
Simplifying assumptions

• Only one user is visible and his/hers torso is
  the largest body component,
• The torso plane is perpendicular to the camera
  and,
• Head is in vertical position.

Notations
state assignments and joint for the arm and body
(head torso) regions.
9
The Upper Body Segmentation. Stage I

• Step 1 Estimate the torso plane parameters from
  all data using EM. Estimate the torso and head
  bounding box, and the plane that includes N.
• Step 2 Estimate the head blob parameters from
  all data using EM.

• Step 3 Compute

• Step 4 Estimate the joints

• Step 5 Repeat steps 3-4 until convergence of

10
The Upper Body Segmentation. Stage II
Given the fix positions of Sl and Sr, we sub
sample the joint space to get NE18 possible
positions for each of the joints El and Er. Given
each position of the elbow joints we search for
NW 16 possible positions for each of the joints
Wl , Wr.

• Step 1. For each possible arm parameters estimate
  the mean of the linear pdfs corresponding to the
  upper and fore arms, and the mean of the normal
  pdf for the hands,
• Step 2. For each joint configuration JA
• a) compute the best state assignment of the
  observation vectors given the joint
  configuration,
• b) compute the observation likelihood given the
  joint configuration.
• Step 3. Find the max likelihood over all joint
  configuration and determine the best set of
  joints and the corresponding best state
  assignment.

11
Arm Tracking

• for each joint Jp we build a set of Jc1, Jc2,
  Jc3, Jc4, Jc5 five possible child joint
  positions such that each of them lies on the
  surface of the sphere with parent joint as the
  center.

Z
Jc2
Jc1 (r,F,?) joint center from last frame
Jc3
F
Jc1
Jc3 (r,F,???)
Jc2 (r,F-?F,?)
Jc5
Jc5 (r,F?F,?)
Jc4 (r,F,?-??)
Y
Jc4
?
X

• Step 1 estimate the mean of the linear pdfs
  corresponding to the upper and fore arms, and the
  mean of the normal pdf for the hands

• Step 2 for each joint configuration we determine
  the best state assignment of the observations

determines the best joint configuration.

• Step 3 the max log likelihood

12
Color Based Segmentation
Depth Segmentation
Pixels with no depth information cannot be
assigned to body components by the previous
segmentation algorithm. Need to estimate the
depth of all pixels and perform global
segmentation.
13
Color Based Segmentation
In practice
Suppose, k left forearm, then l all the
body components except left forearm, and if Zk
a then Zl zmin zmax gt a.
Color Segmentation
14
Upper Body Segmentation and Tracking. Results
15
Conclusion and Future Work

• Contributions
• Articulated upper body model from dense disparity
  maps,
• Linear pdf for the fore arms and upper arms,
• Hand free initialization of the system from the
  optimal joint configuration,
• Upper body tracking, seen as a particular case of
  the initialization.
• Future work
• Improvements to the background segmentation,
• Learn the anthropological measures,
• Integration with other HCI systems (gesture reco,
  face reco, speech reco, speaker identification
  etc.)
•