Detection, Segmentation, and Pose Recognition of Hands in Images

1
Detection, Segmentation, and Pose Recognition
ofHands in Images

• by
• Christopher Schwarz

Thesis Chair Dr. Niels da Vitoria Lobo
2
Outline

• Introduction
• Detection and Segmentation
• Line Finding
• Curve Finding
• Detection
• Grouping
• Results
• Pose Recognition
• Preprocessing
• Matching
• Results
• Discussions and Conclusions

3
Introduction

• Hands present an exciting challenge for Computer
  Vision researchers.
• Foils traditional object detection due to
  nonrigidity and 21 DoF
• Uses
• Surveillance applications
• Gang signs, obscene gestures, drawing of a weapon
• Human-Computer Interaction
• Alternative input devices, motion capture,
  augmented reality.

4
Terminology

• Detection Find presence of target
• Segmentation Separate known target from
  background
• Pose Recognition Determine what pose or posture
  a hand is in.

5
Related Work

• Huang 2000
• Athitsos and Sclaroff 2003
• Kölsch and Turk 2004
• Baris Caglar 2005

6
Detection and Segmentation Outline
Part 1 Detection and Segmentation
Input Image

• High-resolution images
• Monochromatic images
• Straight fingers
• Open fingers

Generate Line Sketch
Find Curves
Find Candidate Fingers
Group and Revisit
7
Line Sketch Image
Part 1 Detection and Segmentation

• Use a Customized Line Finder
• Modified Burns
• Replace line combination with iterative method
• Add a cost of fit measure per line
• Union results of running Line Finder over 5
  varying inputs to obtain Line Sketch
• 4 varying scale
• 1 Double Canny input
• Large-gaussian Canny over output of
  small-gaussian canny to divide textured regions
  from untextured regions

8
Line Finder
Part 1 Detection and Segmentation

• Iterative Joining of Lines
• Find line segments
• Find nearby, almost-parallel line pairs
• If pair meets thresholds, combine them
• Rejoins lines split from angle thresholds or gaps
  in the edge input.

9
Line Finder
Part 1 Detection and Segmentation

• Cost of Fit Measure output with each line
• Cost of fitting line model to underlying data

These lines will have a higher Cost of Fit
10
Line Sketch
Part 1 Detection and Segmentation
Unioned lines of length gt 15
Input image
Unioned Components
Blur 0
Blur 1
Blur 2
Half-Size
Double Canny
11
Line Sketch Examples
Part 1 Detection and Segmentation
12
Line Sketch Examples
Part 1 Detection and Segmentation
13
Curve Finder
Part 1 Detection and Segmentation

• Second input to algorithm
• Discovers curves that may represent fingertips
• See Jan Prokajs thesis Scale Space Based
  Grammar for Hand Detection
• Model

14
Curve Finder Examples
Part 1 Detection and Segmentation
15
FingerFinder Pseudocode
Part 1 Detection and Segmentation

• For each pair of lines
• if pair meets criteria
• for all curves nearby curves if curve meets
  criteria
• add fingerCandidate

16
Finger Candidate Criteria
Part 1 Detection and Segmentation

• Finger Score based on empirically found
  thresholds
• Criteria
• Geometric
• Other

17
Geometric Criteria
Part 1 Detection and Segmentation

• 11 tests measuring how well a line pair and a
  curve approximates target configuration

18
Non-Geometric Criteria
Part 1 Detection and Segmentation

• Line Inaccuracy Measure of line curvature found
  during line finding
• Canny Density Amount of edge pixels detected in
  area.

Variance in Canny Density Sparse finger regions
against cluttered background
19
Results
Part 1 Detection and Segmentation
First row Input images Second row Detected
candidates
20
Grouping Candidate Fingers
Part 1 Detection and Segmentation

• Find finger groups possibly within the same hand
  using
• Locations, using Euclidian distance
• Region intensities, comparing median values
• Revisit weaker candidates to reinstate if
  supported by neighbors

21
Results
Part 1 Detection and Segmentation
First row Input images Second row "Strong"
candidates before grouping Third row Detected
fingers, including those re-added during grouping
22
Grouping Result Breakdown
Part 1 Detection and Segmentation

• Results show detections from all groups
• Often, individual groups divide false from true
  positives

23
Grouping Result Breakdown
Part 1 Detection and Segmentation
24
Pose Recognition Goals
Part 2 Pose Recognition
Segmentation-based method using a database and an
input contour

• Assumes
• High-resolution
• Open fingers

25
Flowchart of Our Method
Part 2 Pose Recognition
26
Preprocessing
Part 2 Pose Recognition
Preprocessing is identical for the test and every
database image.
Input contour silhouette

1. Erode
2. Dilate
3. Compare with the original to find protrusions.

27
Preprocessing
Part 2 Pose Recognition

• Ignore tiny protrusions as palm
• Remove palm
• Use K-Means clustering to find center of palm
  from wrist-palm segment
• Count finger segments and find average
  direction

28
Preprocessing Examples
Part 2 Pose Recognition

• Matching takes test and set of database images
  processed in this way

29
Matching Phase Overview
Part 2 Pose Recognition
Matching via sum of two distance measures

• Chamfer Distance
• Segment-Based Matching

30
Chamfer Distance
Part 2 Pose Recognition

• Numerical similarity between edge images
• For each point in X, find nearest point in Y
• The average is the chamfer distance

31
Chamfer Distance Direction
Part 2 Pose Recognition

• c(X,Y) ! c(Y,X)

c(X,Y) lt c(Y,X)
Undirected Chamfer c(X,Y) c(Y,X)
32
Segment Based MatchingOverview
Part 2 Pose Recognition

• Generate CODAA Vector for every pair of test
  segment and model segment.
• Vector contains five segment comparators
• Rank comparator vectors
• Rank database images with sum of comparator
  rankings

33
Segment Based MatchingCODAA Vectors
Part 2 Pose Recognition
C Chamfer distance between contours
O Difference between orientations
D Distance between centers
A Difference in size
A Difference in angle relative to palm center
34
Segment Based Matching
Part 2 Pose Recognition

1. Score each CODAA vector via progressive
   thresholds of the five values.
2. Rank vectors according to scores
3. For each model image segment, find match in test
   image with highest score
4. For each segment in test image, find match in
   model image with highest score
5. Sum forward and reverse measures
6. Divide by number of fingers
7. Rank model images by score

35
Combination
Part 2 Pose Recognition

• Combine results of Chamfer Distance and SBM by
  summing the Log (base 2) of a models rank in
  each measure.
• Rank models by this combined score
• Filter known-incorrect models
• Incorrect finger count
• Incorrect average finger angle

36
Video Test Results
Part 2 Pose Recognition
Use video frames as a "database," to find ones
matching an input pose
37
Still-Image Test Results
Part 2 Pose Recognition
Use a standard database
38
Publications

• Segment-Based Hand Pose Estimation. In IEEE CRV
  2005.
• Hand Detection and Segmentation for Pose
  Recognition in Monochromatic Images. In progress.
• Line Sketch. To be written.

39
Future Work

• Develop and test bridge between segmentation and
  recognition algorithms
• Feasible to convert finger candidate regions into
  framework of SBM
• Results improved if palm center can be reliably
  located

40
Acknowledgements

• Thesis Committee
• Dr. Niels da Vitoria Lobo
• Dr. Charles Hughes
• Dr. Mubarak Shah
• Dr. Huaxin You
• Support
• NSF REU Program