Segmentation and Tracking of Interacting Human Body Parts under Occlusion and Shadowing

1
Segmentation and Tracking of Interacting Human
Body Partsunder Occlusion and Shadowing

• Sangho Park
• Computer Vision and Robotics Research Lab
• Electrical and Computer Engineering
• University of California, San Diego

2
Overview

• Objectives
• Related studies
• Pixel classification
• Blob formation
• Tracking multiple blobs
• Human body parts
• Results
• Conclusion

3
Objective

• Objective
• Recognition of human interaction in detailed
  level
• Segmentation and tracking of multiple body
  parts.
• Robustness to occlusion and shadows
• Body part labeling and pose recovery

4
How to Segment and Track Body Parts?
pushing sequence
Problems involved ? segmenting multiple body
parts ? tracking body parts ? treatment of
occlusion and shadows between body parts ?
recognizing human interaction type
5
Related Works

• Tracking human motion using multiple cameras
• - Q. Cai J.K. Aggarwal, 1996
• Ghost a human body part labeling system using
  silhouettes
• I. Haritaoglu L. Davis, 1998
• Recognition of human interaction using multiple
  features in grayscale images
• S. Park J.K. Aggarwal, 2000
• Tracking people in presence of occlusion
• S. Khan M. Shah, 2000
• Inferring body pose without tracking
• Rosales and S. Sclaroff
• Recognizing tracking two-person interactions in
  outdoor image sequences
• - K. Sato J.K. Aggarwal, 2001
• Object detection and tracking in an open and
  dynamic world
• T. Ellis M. Xu, 2001

6
Overview

• Objective
• Related studies
• Pixel classification
• Blob formation
• Tracking multiple blobs
• Human body parts
• Results
• Conclusion

7
Gaussian Mixture for Pixel Color
8
Gaussian Mixture Model for Pixel color

• Foreground pixel color
• a multivariate Gaussian distribution in HSV color
  space.
• Mixture of C Gaussians P(x)
• color distribution of the foreground pixel x.

P(wi) prior probability of class wi, the i-th
Gaussian. d the dimension of color space c
the number of Gaussians
9
Pixel Color Classification

• Expectation-Maximization (EM)
• for learning the Gaussian parameters
• Maximum a Posteriori (MAP) classification
• for pixel-color labeling

10
Multivariate Gaussian Distributions
Value
Saturation
Hue
iso-surface of multiple Gaussians
11
Pixel Classification Results
foreground image
12
Skin Blob Detection

• Skin information is very useful in recognizing
  body parts.
• A simple threshold model
• using the chromaticity channels H and S in the
  HSV color space.

Thresholds TH1, TH2, TS1, TS2 obtained from
training data
13
Skin vs. Nonskin Detection
Pixel-color classification
14
Overview

• Objective
• Related studies
• Pixel classification
• Blob formation
• Tracking multiple blobs
• Human body parts
• Results
• Conclusion

15
Blob Formation

• We need image features such as contours and
  regions, which are more descriptive than pixels
  for high-level image understanding.
• Such features are not only described by the
  properties of the features themselves but are
  also related to one another by relationships
  between them.
• We apply a blob-based approach.

16
Blob Formation
17
Attribute Relational Graph
18
Modeling the Inter-blob Relations

• Relaxation Labeling in Attribute Relational Graph
  (ARG) framework
• The relational structure R

• node set S ? the set of blobs
• neighborhood system N ? the adjacency list for
  the blobs
• degree of relationship D ? unary, binary, and
  tertiary relations

19
Blob Merging Procedure

• Region growing
• to merge the over-segmented blobs.
• Search space
• local consistency constraint in the MRF
  framework.
• S ? Ni for i-th blob
• Blob-merging criteria
• blobs Ai and Aj are merged only if the following
  criteria are satisfied
• Adjacency criterion
• Border-ratio criterion
• Color similarity criterion
• Small blob criterion
• Skin blob criterion

20
Attribute Relational Graph
21
Relaxation Labeling
(b) Relational graph
22
Relaxation Labeling
(b) Relational graph
23
Relaxation Labeling
(b) Relational graph
24
Merge Graph
25
Blob Merging Results
Pixel-color classification
26
Overview

• Objective
• Related studies
• Pixel classification
• Blob formation
• Tracking multiple blobs
• Human body parts
• Results
• Conclusion

27
Multi-blob Tracking
28
Multi-target, Multi-assignment Tracking (MMT)
Established tracks at t-1
Blobs at t
1 ?
1 ?
2 ?
2 ?
3 ?
3 ?
4 ?
4 ?
5 ?
6 ?
5 ?
Multi-target, multi-assignment tracking framework
29
Blob Association in MMT

• One-to-one association
• Iterated multi-association
• Terminology
• track an already tracked blob at frame t-1
• blob a new blob at the current frame t
• Similarity between the tracks and the blobs
• features unary features
• blob size, mean color, centroid

30
Similarity Measure in MMT

• Feature vectors for blob association

• Mahalanobis distance for blob dissimilarity

? blob size, ? mean color, I,J centroid,
?covariance matrix of m
31
One-to-one Association in MMT

• Weighted bipartite matching
• A bipartite graph Gb (U, V, E)
• U track set, V blob set, E association

32
Iterated Multi-association in MMT

• Matching in the one-to-one association
• well-established reference tracks T1(t-1)
• well-established reference blobs B1(t).
• Unmatched residuals
• unmatched tracks T0(t-1)
• unmatched blobs B0(t)
• Additional associations (MMT)
• 1-to-1 associations between T1(t-1) and B0(t)
• 1-to-1 associations between T0(t-1) and B1(t)

33
Iteration Process
34
Overview

• Objective
• Related studies
• Pixel classification
• Blob formation
• Tracking multiple blobs
• Human body parts
• Results
• Conclusion

35
Human Body Parts

• Human body model
• domain knowledge
• group the blobs into body parts.
• build semantically meaningful body parts.
• The initial assignment
• blobs to body parts
• performed when people are isolated before
  occlusion.

36
Human Body Modeling
37
Tracking Body Parts

• Blob level vs. Object level
• the inter-weaved mechanism between the blob-level
  and the object-level tracking.

38
Tracking Body Parts

• Mutual constraints
• bottom-up (blob to object) top-down (object to
  blob) processes.
• The human body model
• maintains the list of proper blobs for each body
  part over time.
• Segmenting and tracking body parts
• amounts to the proper update of the blob lists
  across the image sequence.

39
Results Tracking Body Parts
Blob merging
Body part segmentation tracking
40
Overview

• Objective
• Related studies
• Pixel classification
• Blob formation
• Tracking multiple blobs
• Human body parts
• Results
• Conclusion

41
Results Example Sequences

• Pointing

42
Results Example Sequences

• Shaking hands

43
Results Example Sequences

• Pushing

44
Results Example Sequences

• Hugging

45
Conclusion

• Simultaneous segmentation and tracking of
  multiple human body parts
• Integration of relaxation labeling in ARG and
  multi-target multi-assignment tracking
• Integration of minimal domain knowledge (human
  body model)
• Handling of the occlusion and shadowing artifacts
  at the object level

46
Thank You