Shape-Representation

1
Shape-Representation
and
Shape Similarity PART 2
Dr. Rolf Lakaemper
2
Motivation
WHY PARTS ?
3
Motivation
4
Motivation
5
Motivation
6
Motivation

• Global similarity measures fail at
• Occlusion
• Global Deformation
• Partial Match
• (actually everything that occurs under
• real conditions)

7
Parts

• Requirements for a Part Based Shape
  Representation
• (Siddiqi / Kimia 96 Parts of Visual Form
  Computational Aspects)

8
Parts

• How should parts be defined / computed ?
• Some approaches
• Decomposition of interior
• Skeletons
• Maximally convex parts
• Best combination of primitives
• Boundary Based
• High Curvature Points
• Constant Curvature Segments

9
Parts
Principal approach Hoffman/Richards
(85) Part decomposition should precede part
description gt No primitives, but general
principles
10
Parts
No primitives, but general principals
When two arbitrarily shaped surfaces are made to
interpenetrate they always meet in a contour of
concave discontinuity of their tangent planes
(transversality principle)
11
Parts
When two arbitrarily shaped surfaces are made to
interpenetrate they always meet in a contour of
concave discontinuity of their tangent planes
(transversality principle)
Divide a plane curve into parts at negative
minima of curvature
12
Parts

• Different notions of parts
• Parts object is composed of rigid parts
• Protrusions object arises from object by
  deformation due to a (growth) process
  (morphology)
• Bends Parts are result of bending the base object

13
Parts
The Shape Triangle
14
Parts
This lecture focuses on parts, i.e. on
partitioning a shape
15
Framework

• A Framework for a Partitioning Scheme
• Scheme must be invariant to 2 classes of changes
• Global changes translations, rotations
  scaling of 2D shape, viewpoint,
• Local changes occlusions, movement of parts
  (rigid/non-rigid deformation)

16
Framework
A general decomposition of a shape should be
based on the interaction between two parts rather
than on their shapes. -gt Partitioning by Part
Lines
17
Framework
Definition 1 A part line is a curve whose end
points rest on the boundary of the shape, which
is entirely embedded in it, and which divides it
into two connected components.
18
Framework
Definition 2 A partitioning scheme is a
mapping of a connected region in the image to a
finite set of connected regions separated by
part-lines.
19
Framework
Definition 3 A partitioning scheme is
invariant if the part lines of a shape that is
transformed by a combination of translations,
rotations and scalings are transformed in exactly
the same manner.
20
Framework
Definition 4 A partitioning scheme is robust
if for any two shapes A and B, which are exactly
the same in some neighborhood N, the part lines
contained in N for A and B are exactly equivalent.
21
Framework
Definition 5 A partitioning scheme is stable
if slight deformations of the boundary of a shape
cause only slight changes in its part lines
22
Framework
Definition 6 A partitioning scheme is
scale-tuned if when moving from coarse to fine
scale, part lines are only added, not removed,
leading to a hierarchy of parts.
23
Framework
A general purpose partitioning scheme that is
consistent with these requirements is the
partitioning by limbs and necks
24
Limbs and Necks
Definition A limb is a part-line going through
a pair of negative curvature minima with
co-circular boundary tangents on (at least) one
side of the part-line
25
Limbs and Necks
Motivation co-circularity
The decomposition of the right figure is no
longer intuitive absence of good continuation
26
Limbs and Necks

• Smooth continuation an example for
• form from function
• Shape of object is given by natural function
• Different parts having different functions show
  sharp changes in the 3d surface of the connection
• Projection to 2d yields high curvature points

27
Limbs and Necks
Examples of limb based parts
28
Limbs and Necks
Definition A neck is a part-line which is also
a local minimum of the diameter of an inscribed
circle
29
Limbs and Necks

• Motivation for necks
• form from function
• Natural requirements (e.g. space for articulation
  and economy of mass at the connection) lead to a
  narrowing of the joint between two parts

30
Limbs and Necks

• The Limb and Neck partitioning scheme is
  consistent with the previously defined
  requirements
• Invariance
• Robustness
• Stability
• Scale tuning

31
Limbs and Necks
Examples
32
Limbs and Necks
The scheme presented does NOT include a
similarity measure !
33
Algorithms
Part Respecting Similarity Measures
34
CSS
Curvature Scale Space (Mokhtarian/Abbasi/Kittler)
A similarity measure implicitely respecting
parts
35
CSS
Creation of reflection-point based
feature-vector which implicitely contains part
information
36
CSS

• Properties
• Boundary Based
• Continous Model (!)
• Computes Feature Vector
• compact representation of shape
• Performs well !

37
CSS

• The idea
• Smooth (continous) boundary curve using
  convolution with an increasing gaussian kernel
• Use the runlength position of curvature
  zero-crossings on the boundary as index set for
  each kernel size, thereby creating the Curvature
  Scale Space
• The maxima of the CSS are used for shape
  representation
• Similarity of shape is defined by difference
  between the maxima of the CSS representation

38
CSS

• Smooth (continous) boundary curve using
  convolution with an increasing gaussian kernel
• Boundary curve S
• S(x(u),y(u) u ? 0,1
• Each coordinate of S is convolved with a 1D
  Gaussion kernel of width d
• The resulting curve S(d) is smoother than S

39
CSS
Inflection points (curvature zero crossings)
40
CSS

• Use the runlength position of curvature
  zero-crossings on the boundary as index set for
  each kernel size, thereby creating the
• Curvature Scale Space

41
CSS

• The maxima of the CSS are used for shape
  representation

42
CSS

• Similarity of shape is defined by difference
  between the maxima of the CSS representation

43
CSS

• Similarity of shape is defined by difference
  between the maxima of the CSS representation

44
CSS
Some results (Database 450 marine animals)
45
CSS
Some results (Database 450 marine animals)
46
CSS

• Problems of CSS
• Convex shapes dont have inflection points
• Different shapes can have identical CSS !

47
CSS
The main problem CSS is continous, the computer
vision world is discrete. How to measure
curvature in discrete boundaries ?
48
Dominant Points
Local curvature average curvature in region of
support To define regions of support,
dominant points are needed !
49
Dominant Points
Dominant Points (Things should be expressed as
simple as possible, but not simpler, A.
Einstein) Idea given a discrete boundary S
compute polygonal boundary S with minimum number
of vertices which is visually similar to S.
50
Dominant Points

• Example Algorithms
• ( 3 of billions)
• Ramer
• Line Fitting
• Discrete Curve Evolution

51
DCE
Discrete Curve Evolution (Latecki / Lakaemper
99) Idea Detect subset of visually
significant points
52
Curve Evolution
Target reduce data by elimination of irrelevant
features, preserve relevant features
... noise reduction
... shape simplification
53
Curve Evolution Tangent Space
Transformation from image-space to tangent-space
54
Tangent Space Properties
In tangent space...
... the height of a step shows the turn-angle
... monotonic increasing intervals represent
convex arcs
... height-shifting corresponds to rotation
... the resulting curve can be interpreted as 1
dimensional signal gt idea filter signal
in tangent space (demo 'fishapplet')
55
Curve Evolution Step Compensation
(Nonlinear) filter merging of 2 steps with area
difference F given by
(a-b)pq p q
F
q
a g b
F
F
p
56
Curve Evolution Step Compensation
Interpretation in image space
... Polygon linearization
... removal of visual irrelevant vertices
q
p
removed vertex
57
Curve Evolution Step Compensation
next Iterative SC
Interpretation in image space
... Polygon linearization
... removal of visual irrelevant vertices
58
Curve Evolution Iterative Step Compensation

Keep it simple repeated step compensation !
(demo EvoApplet)
Remark there are of course some traps ...
59
Curve Evolution Iterative Step Compensation

• Remark there are of course some traps
• Self intersection / Topology preservation
• Stop parameter
• Edge movement

60
Curve Evolution Properties
The evolution...
... reduces the shape-complexity
... is robust to noise
... is invariant to translation, scaling and
rotation
... preserves the position of important vertices
... extracts line segments
... is in accord with visual perception
... offers noise-reduction and shape abstraction
... is parameter free
... is translatable to higher dimensions
61
Curve Evolution Properties
Robustness (demo noiseApplet)
62
Curve Evolution Properties
Preservation of position, no blurring !
63
Curve Evolution Properties
Strong relation to digital lines and segments
64
Curve Evolution Properties
Noise reduction as well as shape abstraction
65
Curve Evolution Properties
Parameter free (?)
66
Curve Evolution Properties
Extendable to higher dimensions
67
Curve Evolution Properties
Extendable to higher dimensions
68
Curve Evolution Properties
Extendable to higher dimensions
69
Curve Evolution Properties
Extendable to higher dimensions
70
Curve Evolution Properties

• Result
• The DCE creates a polygonal shape representation
  in different levels of granularity Scale Space
• Curvature can be defined as the turning angle at
  the vertices
• Regions of support are defined by vertices
• Easy traceable Scale Space is created, since no
  points are relocated

71
Scale Space
Ordered set of representations on different
information levels
72
Polygonal Representation
The polygonal representation achieved by the DCE
has a huge advantage It allows easy boundary
partitioning using convex / concave parts
(remember the limbs !) (MATLAB Demo MatchingDemo)
73
DCE
Some results of part line decomposition
74
ASR
The ASR (Advanced Shape Recognition) Algorithm
uses the boundary parts achieved by the polygonal
representation for a part based similarity
measure ! (Note this is NOT the area
partitioning shown in the previous slide)
75
ASR / ISS
The ASR is used in the ISS Database
76
The Interface (JAVA Applet)
77
The Sketchpad Query by Shape
78
The First Guess Different Shape - Classes
79
Selected shape defines query by shape class
80
Result
81
Specification of different shape in shape class
82
Result
83
Let's go for another shape...
84
...first guess...
85
...and final result
86
Query by Shape, Texture and Keyword
87
Result
88
How does it work ?

• Behind The Scenes of the ISS - Database
• Modern Techniques of Shape
• Recognition and Database Retrieval

89
The 2nd Step First Shape Comparison
ISS implements the ASR (Advanced Shape
Recognition) Algorithm
Developed by Hamburg University in cooperation
with Siemens AG, Munich, for industrial
applications in...
... robotics ... multimedia (MPEG 7)
90
Reticent Proudness
MPEG-7 ASR outperformes classical approaches !
Similarity test (70 basic shapes, 20 different
deformations)
ASR Hamburg Univ./Siemens AG 76.45
Curvature Scale Space Mitsubishi ITE-VIL 75.44

Multilayer Eigenvector Hyundai 70.33
Zernicke Moments Hanyang University 70.22
Wavelet Contour Heinrich Hertz Institute
Berlin 67.67
DAG Ordered Trees Mitsubishi/Princeton
University 60.00
(Capitulation -) IBM --.--
91
Requirements
Robust automatic recognition of arbitrary shaped
objects which is in accord with human visual
perception
Wide range of applications...
... recognition of complex and arbitrary patterns
... invariance to basic transformations
... results which are in accord with human
perception
... applicable to three main tasks of recognition
... parameter-free operation
Industrial requirements...
... robustness
... low processing time
92
Requirements
Next Strategy
Scaling (or resolution)
Rotation
Robust automatic recognition of arbitrary shaped
objects which is in accord with human visual
perception
Rigid / non-rigid deformation
Wide range of applications...
... recognition of complex and arbitrary patterns
... invariance to basic transformations
... results which are in accord with human
perception
... applicable to three main tasks of recognition
... parameter-free operation
Industrial requirements...
... robustness
... low processing time
93
Requirements
Simple Recognition (yes / no)
... robustness
... low processing time
Robust automatic recognition of arbitrary shaped
objects which is in accord with human visual
perception
Common Rating (best of ...)
Wide range of applications...
... recognition of complex and arbitrary patterns
Analytical Rating (best of, but...)
... results which are in accord with human
perception
... invariance to basic transformations
... applicable to three main tasks of recognition
... parameter-free operation
Industrial requirements...
... robustness
... low processing time
94
Different Approaches
Pattern Matching...
... Correlation
Geometrical description...
... Hough Transformation
Feature Vectors...
... (Zernicke - ) Moments
Based on Visual Parts...
... Mokhtarian
... ASR
95
Curvature Scale Space (Mokhtarian, Mitsubishi)
Creation of reflection-point based
feature-vector which implicitely contains part
information
96
ASR Strategy
ASR Strategy
Source 2D - Image
Object - Segmentation
Contour Extraction
Evolution
Contour Segmentation
Arc Matching
97
ASR Strategy
ASR Strategy
DCE
Contour Segmentation
Arc Matching
98
Contour Segmentation
Correspondence ?
Similarity of parts ?
99
Part Similarity
Similarity of parts ? Boundary Similarity
Measure Similarity of polygons
100
Tangent Space
Transformation from image-space to tangent-space
101
Shape Comparison Measure
Tangent space offers an intuitive measure
102
Shape Comparison Measure
Drawback not adaptive to unequally
distributed noise if used globally !
but works for single parts
103
Shape Comparison Contour Segmentation
104
Shape Comparison Correspondence
next Corr. -example
Optimal arc-correspondence
find one to many (many to one) correspondence,
that
minimizes the arc-measure !
105
Graph of Correspondence
arc
a0
a3
a2
a0 a1 a2 a3
a1
b0
b0 b1 b2 b3
b3
b2
correspondence
b1
Graph
... edge represents correspondence
... node represents matched arcs
106
Shape Comparison Correspondence
Example
a0 a1 a2 a3
a0
a3
a2
a1
b0 b1 b2 b3
b0
b3
b2
b1
107
Shape Comparison Correspondence
next Corr. - Results
Result Optimal correspondence is given by
cheapest way
108
Correspondence Results
109
(MATLAB Demo)
110
Correspondence Results
Correspondence and arc-measure allow...
... the identification of visual parts as well as
... the identification of the entire object
... a robust recognition of defective parts
... a shape matching which is in accord with
human perception
111
ASR Results

• Correspondence and arc-measure meet the
  requirements stated by Kimia et al.
• Discrete
• Easy computable

112
ASR Applications in Computer Vision

• Robotics Shape Screening
• (Movie Robot2.avi)
• Straightforward Training Phase
• Recognition of Rough Differences
• Recognition of Differences in Detail
• Recognition of Parts

113
ASR Applications in Computer Vision
Application 2 View Invariant Human Activity
Recognition (Dr. Cen Rao and Mubarak Shah,
School of Electrical Engineering and Computer
Science, University of Central Florida)
114
Application Human Activity Recognition

• Human Action Defined by Trajectory
• Action Recognition by Comparison of Trajectories
• (Movie Trajectories)
• Rao / Shah
• Extraction of Dynamic Instants by Analysis of
  Spatiotemporal Curvature
• Comparison of Dynamic Instants (Sets of
  unconnected points !)
• ASR
• Simplification of Trajectories by Curve Evolution
• Comparison of Trajectories

115
Application Human Activity Recognition
Simplification
Trajectory
116
Activity Recognition Typical Set of Trajectories
117
Trajectories in Tangent Space
118
Trajectory Comparison by ASR Results
119
Recognition of 3D Objects by Projection
Background MPEG 7 uses fixed view
angles Improvement Automatic Detection of Key
Views
120
Automatic Detection of Key Views

• (Pairwise) Comparison of Adjacent Views
• Detects Appearance of Hidden Parts

121
Automatic Detection of Key Views
Result (work in progress)
122
The Main Application Back to ISS
Task Create Image Database Problem Response
Time Comparison of 2 Shapes 23ms on
Pentium1Ghz ISS contains 15,000
images Response Time about 6 min. Clustering
not possible ASR failed on measuring
dissimilarities !
123
Vantage Objects
Solution Full search on entire database using
a simpler comparison Vantage Objects
(Vleugels / Veltkamp, 2000) provide a simple
comparison of n- dimensional vectors (n
typically lt 100)
124
Vantage Objects
The Idea Compare the query-shape q to a
predefined subset S of the shapes in the
database D The result is an n-dimensional
Vantage Vector V, n S
s1
v1
s2
v2
q
s3
v3

sn
vn
125
Vantage Objects

• - Each shape can be represented by a single
  Vantage Vector
• - The computation of the Vantage Vector calls
  the ASR comparison only n times
• - ISS uses 54 Vantage Objects, reducing the
  comparison time (needed to create the Vantage
  Vector) to lt 1.5s
• - How to compare the query object to the database
  ?

126
Vantage Objects

• - Create the Vantage Vector vi for every shape
  di in the database D
• - Create the Vantage Vector vq for the
  query-shape q
• - compute the (euclidean) distance between vq and
  vi
• - best response is minimum distance
• Note computing the Vantage Vectors for the
  database objects is an offline process !

127
Vantage Objects

• How to define the set S of Vantage Objects ?

128
Vantage Objects

• Algorithm 1 (Vleugels / Veltkamp 2000)
• Predefine the number n of Vantage Objects
• S0
• Iteratively add shapes di ? D\Si-1 to Si-1 such
  that
• Si Si-1 ? di
• and
• ?k1..i-1 ?e(di , sk) maximal. (?e eucl. dist.)
• Stop if i n.

129
Vantage Objects

• Result
• Did not work for ISS.

130
Vantage Objects

• Algorithm 2 (Latecki / Henning / Lakaemper)
• Def.
• A(s1,s2) ASR distance of shapes s1,s2
• q query shape
• Vantage Query determining the result r by
  minimizing e(vq , vi ) vi Vantage Vector to si
• ASR Query determining the result r by
  minimizing A(q,di )
• Vantage Query has certain loss of retrieval
  quality compared to ASR query.
• Define a loss function l to model the extent of
  retrieval performance

131
Vantage Objects

• Given a Database D and a set V of Vantage
  Vectors, the loss of retrieval performance for a
  single query by shape q is given by
• lV,D (q) A(q,r),
• Where r denotes the resulting shape of the
  vantage query to D using q.
• Property
• lV,D (q) is minimal if r is the result of the
  ASR-Query.

132
Vantage Objects

• Now define retrieval error function L(S) of set
• Ss1 ,, sn ? D of Vantage Vectors of Database
  D
• L(S) 1/n ? lS,D\si (si)
• Task
• Find subset S ? D such that L(S) is minimal.

133
Vantage Objects
Algorithm V0 iteratively determine sj
in D\Sj-1 such that Sj Sj-1 ? sj and L(Vj)
minimal. Stop if improvement is low
134
Vantage Objects
Result Worked fine for ISS, though handpicked
objects stil performed better.
Handpicked Algorithm 2
L(S)
Number of Vantage Objects
135
Vantage Objects
some of the Vantage Objects used in ISS