Shape-Representation

1
Shape-Representation
and
Shape Similarity
Part 1 Shapes
Dr. Rolf Lakaemper
2
May I introduce myself

• Rolf Lakaemper
• PhD (Doctorate Degree) 2000
• Hamburg University, Germany
• Currently Assist. Professor at Department
• of Computer and Information Sciences,
• Temple University, Philadelphia, USA
• Main Research Area Computer Vision

3
Research Goal
Teaching robots to recognize the world they see
using SHAPE
4
Motivation
WHY SHAPE ?
5
Motivation
These objects are recognized by
6
Motivation
These objects are recognized by
Texture Color Context Shape
X X
X X
X
X
X
X X
7
Why Shape ?
Several applications in computer vision use shape
processing Object recognition Image
retrieval Processing of pictorial information
Video compression (eg. MPEG-7) This
presentation focuses on object recognition and
image retrieval.
8
Motivation
Typical Application Multimedia Image Database
Query by Shape / Texture / (Color / Keyword)
9
ISS Database
Example ISS-Database http//knight.cis.temple.e
du/shape
10
The Interface (JAVA Applet)
11
The Sketchpad Query by Shape
12
The First Guess Different Shape - Classes
13
Selected shape defines query by shape class
14
Result
15
ISS Database
ISS Query by Shape / Texture
Sketch of Shape Query by Shape
only Result Satisfying ?
16
ISS Database
SHAPE recognition seems to be possible and
leads to satisfying results !
17
ISS Database
Well talk about the ISS Database a bit
later, so stay alert !
18
Overview

• Part 1
• General thoughts about shape recognition
• Feature based approaches
• Part 2
• Part based, direct approaches
• The ISS database
• Applications

19
Data Retrieval

• The most obvious sensor to gain the data for
  shape recognition is a camera. But shape is not
  only perceived by visual means
• tactical sensors can also provide shape
  information that are processed in a similar way.
• robots range sensor provide shape information,
  too.
• Hence shape is a general, widely applicable
  object descriptor!

20
Shape

• Typical problems
• How to describe shape ?
• What is the matching transformation?
• No one-to-one correspondence
• Occlusion
• Noise

21
Shape

• Partial match only part of query appears in part
  of database shape

22
What is Shape ?
lets start with some properties easy to agree
on Shape describes a spatial region Shape is
a (the ?) specific part of spatial cognition
Typically addresses 2D space
23
What is Shape ?
Shape or Not ? Continuous transformation
from shape to two shapes Is there a point when
it stops being a single shape?
24
What is Shape ?
But theres no doubt that a single, connected
region is a shape. Right ?
25
What is Shape ?
A single, connected region. But a shape
? A question of scale !
26
What is Shape ?

• Theres no easy, single definition of shape
• In difference to geometry, arbitrary shape is not
  covered by an axiomatic system
• Different applications in object recognition
  focus on different shape related features
• Special shapes can be handled
• Typically, applications in object recognition
  employ a similarity measure to determine a
  plausibility that two shapes correspond to each
  other

27
Similarity
So the new question is What is Shape
Similarity ? or How to Define a Similarity
Measure
28
Similarity
Again its not so simple (sorry). Theres
nothing like THE similarity measure
29
Similarity Measure

• Requirements to a similarity measure
• Should not incorporate context knowledge (no AI),
  thus computes generic shape similarity

30
Similarity Measure

• Requirements to a similarity measure
• Must be able to deal with noise
• Must be invariant with respect to basic
  transformations

Next Strategy
Scaling (or resolution)
Rotation
Rigid / non-rigid deformation
31
Similarity Measure

• Requirements to a similarity measure
• Must be able to deal with noise
• Must be invariant with respect to basic
  transformations
• Must be in accord with human perception

32
Similarity Measure

• Desired Properties of a Similarity Function C
• (Basri et al. 1998)
• C should be a metric
• C should be continuous
• C should be invariant (to)

33
Properties
Metric Properties S set of patterns Metric d
S S R satisfying 1. Self-identity " xÎS,
d(x,x)0 2. Positivity " x ¹yÎS, d(x,y)gt0 3.
Symmetry " x, yÎS, d(x,y) d(y,x) 4. Triangle
inequality " x, y, zÎS, d(x,z)d(x,y)d(y,z)
Semi-metric 1, 2, 3 Pseudo-metric 1, 3, 4 S
with fixed metric d is called metric space
34
Properties

• Self-identity " xÎS, d(x,x)0
• Positivity " x ¹yÎS, d(x,y)gt0
• surely makes sense

35
Properties

36
Properties

37
Properties

• In general
• a similarity measure in accordance with human
  perception is NOT a metric. This leads to deep
  problems in further processing, e.g. clustering,
  since most of these algorithms need metric spaces
  !

38
Similarity Measures Overview

• Similarity Measure depends on
• Shape Representation
• Boundary
• Area (discrete point set)
• Structural (e.g. Skeleton)
• Comparison Model
• feature vector
• direct

39
Similarity Measures
direct feature based
Boundary Spring model, Cum. Angular Function, Chaincode, Arc Decomposition (ASR-Algorithm) Central Dist. Fourier Distance histogram
Area (point set) Hausdorff Moments Zernike Moments
Structure Skeleton ---

40
Feature Based Coding
Feature Based Coding This category defines all
approaches that determine a feature-vector for a
given shape. Two operations need to be defined
a mapping of shape into the feature space and a
similarity of feature vectors.

Representation
Feature Extraction
Vector Comparison
41
Feature Based Coding
Again TWO operations need to be defined We
hence have TWO TIMES an information reduction of
the basic representation, which by itself is
already a mapping of the reality.

Representation
Feature Extraction
Vector Comparison
42
Vector Comparison

• Example
• Vector of Elementary Descriptors
• Shape A,B given as
• Area (continous) or
• Point Sets (discrete)

43
Vector Comparison

44
Vector Comparison

Similarity (scalar value)
45
Vector Comparison

• All Feature Vector approaches have similar
  properties
• Provide a compact representation
• this is especially interesting for database
  indexing !
• Works for any shape
• Requires complete shapes (global comparison)
• Sensible to noise (except Zernike moments which
  are computationally demanding)
• Map dissimilar shapes to similar feature
  vectors (!)
• They can be used as a prefilter for database
  applications !
• Make the choice of a similarity function
  difficult

46
Direct Comparison

End of Feature Based Coding ! Next Direct
Comparison
47

• Part II
• Behind The Scenes of the ISS - Database
• Modern Techniques of Shape
• Recognition and Database Retrieval

48
Overview

• Topics
• The Shape Recognition Algorithm Implemented in
  ISS
• Possible Applications in Different Areas of
  Computer Vision

49
Results first

• Image Database providing query by
• Keyword
• Texture
• Shape
• Shape is given by user-sketch, a mouse-drawn
  outline

50
ISS - GUI
51
The Sketchpad Query by Shape
52
The First Guess Different Shape - Classes
53
Selected shape defines query by shape class
54
Result
55
Key Steps
Retrieval by Vantage Objects
Retrieval by Direct Shape Comparison
56
Requirements
Robust automatic recognition of arbitrary shaped
objects which is in accord with human visual
perception
Wide range of applications...
... recognition of complex and arbitrary patterns
... invariance to basic transformations
... results which are in accord with human
perception
... applicable to three main tasks of recognition
... parameter-free operation
Industrial requirements...
... robustness
... low processing time
57
Requirements
Next Strategy
Scaling (or resolution)
Rotation
Robust automatic recognition of arbitrary shaped
objects which is in accord with human visual
perception
Rigid / non-rigid deformation
Wide range of applications...
... recognition of complex and arbitrary patterns
... invariance to basic transformations
... results which are in accord with human
perception
... applicable to three main tasks of recognition
... parameter-free operation
Industrial requirements...
... robustness
... low processing time
58
Requirements
Simple Recognition (yes / no)
... robustness
... low processing time
Robust automatic recognition of arbitrary shaped
objects which is in accord with human visual
perception
Common Rating (best of ...)
Wide range of applications...
... recognition of complex and arbitrary patterns
Analytical Rating (best of, but...)
... results which are in accord with human
perception
... invariance to basic transformations
... applicable to three main tasks of recognition
... parameter-free operation
Industrial requirements...
... robustness
... low processing time
59
The 2nd Step First Shape Comparison
ISS implements the ASR (Advanced Shape
Recognition) Algorithm
Developed by Dr. Latecki / Dr. Lakaemper in
cooperation with Siemens AG, Munich, for
industrial applications in...
... robotics ... multimedia (MPEG 7)
60
MPEG 7
MPEG-7 ASR outperformes classical approaches !
Similarity test (70 basic shapes, 20 different
deformations)
ASR Hamburg Univ./Siemens AG 76.45
Curvature Scale Space Mitsubishi ITE-VIL 75.44

Multilayer Eigenvector Hyundai 70.33
Zernicke Moments Hanyang University 70.22
Wavelet Contour Heinrich Hertz Institute
Berlin 67.67
DAG Ordered Trees Mitsubishi/Princeton
University 60.00
(Capitulation -) IBM --.--
61

• The shape similarity algorithm behind the
  ISS-database is a direct, part based similarity
  measure.

62
Motivation
WHY PARTS ?
63
Motivation
64
Motivation

• Global similarity measures fail at
• Occlusion
• Global Deformation
• Partial Match
• (actually everything that occurs under
• real conditions)

65
Requirements for a Part Based Shape Representation
Principal approach Hoffman/Richards
(85) Part decomposition should precede part
description gt No primitives, but general
principles
66
Parts
No primitives, but general principals
When two arbitrarily shaped surfaces are made to
interpenetrate they always meet in a contour of
concave discontinuity of their tangent planes
(transversality principle)
67
Parts

• How should parts be defined ?
• Some approaches
• Decomposition of interior
• Skeletons
• Maximally convex parts
• Best combination of primitives
• Boundary Based
• High Curvature Points
• Constant Curvature Segments

68
Visual Parts
Motivated by psychological experiments
(Hoffmann/Richards)
split bounding-curve into convex / concave arcs
69
ASR Strategy
ASR Strategy
Source 2D - Image
Object - Segmentation
Contour Extraction
Evolution
Contour Segmentation
Arc Matching
70
Curve Evolution
Target reduce data by elimination of irrelevant
features, preserve relevant features
... noise reduction
... shape simplification
71
Curve Evolution Tangent Space
Transformation from image-space to tangent-space
bild s.22
72
Tangent Space Properties
In tangent space...
... the height of a step shows the turn-angle
... monotonic increasing intervals represent
convex arcs
... height-shifting corresponds to rotation
... the resulting curve can be interpreted as 1
dimensional signal gt idea filter signal
in tangent space (demo 'fishapplet')
73
Curve Evolution Step Compensation
New nonlinear filter merging of 2 steps with
area difference F given by
(a-b)pq p q
F
q
a g b
F
F
p
74
Curve Evolution Step Compensation
Interpretation in image space
... Polygon linearization
... removal of visual irrelevant vertices
q
p
removed vertex
75
Curve Evolution Step Compensation
next Iterative SC
Interpretation in image space
... Polygon linearization
... removal of visual irrelevant vertices
76
Curve Evolution Iterative Step Compensation

Keep it simple repeated step compensation !
Remark there are of course some traps ...
77
Curve Evolution Properties
The evolution...
... reduces the shape-complexity
... is robust to noise
... is invariant to translation, scaling and
rotation
... preserves the position of important vertices
... extracts line segments
... is in accord with visual perception
... offers noise-reduction and shape abstraction
... is parameter free
... is translatable to higher dimensions
78
Curve Evolution Properties
Robustness (demo noiseApplet)
79
Curve Evolution Properties
Preservation of position, no blurring !
80
Curve Evolution Properties
Strong relation to digital lines and segments
81
Curve Evolution Properties
Noise reduction as well as shape abstraction
82
Curve Evolution Properties
Parameter free
83
Curve Evolution Properties
Extendable to higher dimensions
84
Curve Evolution Properties
Extendable to higher dimensions
85
Curve Evolution Properties
Extendable to higher dimensions
86
Curve Evolution Properties
Extendable to higher dimensions
87
Shape Comparison Measure
Tangent space offers an intuitive measure
88
Shape Comparison Measure
Drawback not adaptive to unequally
distributed noise
Solution partition bounding curve
89
Shape Comparison Contour Segmentation
Solution partition bounding curve
90
Shape Comparison Contour Segmentation
Motivated by psychological experiments
(Hoffmann/Richards)
split bounding-curve into convex / concave arcs
91
Shape Comparison Correspondence
Optimal arc-correspondence
find one to many (many to one) correspondence,
that
minimizes the arc-measure !
92
Graph of Correspondence
arc
a0
a3
a2
a0 a1 a2 a3
a1
b0
b0 b1 b2 b3
b3
b2
correspondence
b1
Graph
... edge represents correspondence
... node represents matched arcs
93
Shape Comparison Correspondence
Example
a0 a1 a2 a3
a0
a3
a2
a1
b0 b1 b2 b3
b0
b3
b2
b1
94
Shape Comparison Correspondence
Result Optimal correspondence is given by
cheapest way
95
Correspondence Results
96
(Movie Deer.avi)
97
Correspondence Results
Correspondence and arc-measure allow...
... the identification of visual parts as well as
... the identification of the entire object
... a robust recognition of defective parts
... a shape matching which is in accord with
human perception
98
ASR Applications in Computer Vision

• Robotics Shape Screening
• (Movie Robot2.avi)
• Straightforward Training Phase
• Recognition of Rough Differences
• Recognition of Differences in Detail
• Recognition of Parts

99
ASR Applications in Computer Vision
Application 2 View Invariant Human Activity
Recognition (Dr. Cen Rao and Mubarak Shah,
School of Electrical Engineering and Computer
Science, University of Central Florida)
100
Application Human Activity Recognition

• Human Action Defined by Trajectory
• Action Recognition by Comparison of Trajectories
• (Movie Trajectories)
• Rao / Shah
• Extraction of Dynamic Instants by Analysis of
  Spatiotemporal Curvature
• Comparison of Dynamic Instants (Sets of
  unconnected points !)
• ASR
• Simplification of Trajectories by Curve Evolution
• Comparison of Trajectories

101
Application Human Activity Recognition
Simplification
Trajectory
102
Activity Recognition Typical Set of Trajectories
103
Trajectories in Tangent Space
104
Trajectory Comparison by ASR Results
105
Recognition of 3D Objects by Projection
Background MPEG 7 uses fixed view
angles Improvement Automatic Detection of Key
Views
106
Automatic Detection of Key Views

• (Pairwise) Comparison of Adjacent Views
• Detects Appearance of Hidden Parts

107
Automatic Detection of Key Views
Expected Result (work in progress)
108
Conclusion Research in Shape Similarity has a
lot of challenges, some solutions, and for sure
is fun ! Thats it, Thanks !