Shape Distinction for 3D Object Retrieval

1
Shape Distinction for 3D Object Retrieval

• Philip Shilane

2
Shape Matching
Shape matching is important for numerous projects
including constructing 3D models, object
recognition, and protein analysis.
Molecular Biology Protein DatabankH.M. Berman et
al. 2000.
3
Shape Matching
Shape matching is important for numerous projects
including constructing 3D models, object
recognition, and protein analysis.
Molecular Biology Protein DatabankH.M. Berman et
al. 2000.
Mechanical CAD National Design RepositoryW.
Regli et al. 2001. Engineering Shape BenchmarkN.
Iyer et al. 2005.
4
Shape Matching
Shape matching is important for numerous projects
including constructing 3D models, object
recognition, and protein analysis.
Molecular Biology Protein DatabankH.M. Berman et
al. 2000.
Tracking E3D
Mechanical CAD National Design RepositoryW.
Regli et al. 2001. Engineering Shape BenchmarkN.
Iyer et al. 2005.
5
Shape Matching
Shape matching is important for numerous projects
including constructing 3D models, object
recognition, and protein analysis.
Molecular Biology Protein DatabankH.M. Berman et
al. 2000.
Tracking E3D
Mechanical CAD National Design RepositoryW.
Regli et al. 2001. Engineering Shape BenchmarkN.
Iyer et al. 2005.
Computer Graphics Princeton Shape BenchmarkP.
Shilane et al. 2004.
6
Min et al.
7
Min et al.
8
Shape Retrieval
Find similar shapes in a database using a query
shape.
3D Model
BestMatches
Model Database
9
Shape Descriptors
Shape descriptors are feature vector
representations for shapes
3D Model
BestMatches
Model Database
10
Example Shape Descriptors

• Extended Gaussian Image (EGI)
• Complex Extended Gaussian Image (CEGI)
• Shape Histograms (Shells)
• Shape Histograms (Sectors)
• Shape Histograms (SecShells)
• D2 Shape Distributions
• Spherical Extent Function (EXT)
• Radialized Spherical Extent Function (REXT)
• Voxel
• Gaussian Euclidean Distance Transform (GEDT)
• Harmonic Shape Descriptor (HSD)
• Fourier Shape Descriptor (FSD)
• Light Field Descriptor (LFD)
• Depth Buffer Descriptor (DBD)

Surface Normals
Surface Distribution
Morphing Distance
Image-Based
11
Properties of Shape Descriptors
12
Shape Retrieval with Descriptors

• Which shape descriptors are the best?
• How do we evaluate retrieval success?

Princeton Shape BenchmarkPhilip Shilane, Patrick
Min, Michael Kazhdan, and Thomas Funkhouser
13
Princeton Shape Benchmark

• Large shape database
• 1,814 classified models, 161 classes
• Separate training and test sets
• Standardized suite of tests
• Multiple classifications
• Targeted sets of queries
• Standardized evaluation tools
• Visualization software
• Quantitative metrics

14
Princeton Shape Benchmark (PSB)
15
Princeton Shape Benchmark
16
Evaluation Tools
Base Classification (92)

• Visualization tools
• Precision/recall plot
• Best matches
• Distance image
• Tier image
• Quantitative metrics
• Nearest neighbor
• First and Second tier
• E-Measure
• Discounted Cumulative Gain (DCG)
• Multiple Classifications
• Granularity
• Model Properties

17
Evaluation Tools

• Visualization tools
• Precision/recall plot
• Best matches
• Distance image
• Tier image
• Quantitative metrics
• Nearest neighbor
• First and Second tier
• E-Measure
• Discounted Cumulative Gain (DCG)
• Multiple Classifications
• Granularity
• Model Properties

18
Evaluation Tools

• Visualization tools
• Precision/recall plot
• Best matches
• Distance image
• Tier image
• Quantitative metrics
• Nearest neighbor
• First and Second tier
• E-Measure
• Discounted Cumulative Gain (DCG)
• Multiple Classifications
• Granularity
• Model Properties

19
PSB Contributions

• Methodology to compare shape descriptors
• Vary classifications
• Query lists targeted at specific properties
• Unexpected results
• EGI good at discriminating man-made vs. natural
• objects, though poor at fine-grained
  distinctions
• DBD good overall performance across tests
• Freely available Princeton Shape Benchmark
• classified polygonal models
• Source code for evaluation tools
• Downloaded 8,700 times since 2003
• http//shape.cs.princeton.edu/benchmark

20
Shape Retrieval
Matching the whole shape has drawbacks.
3D Model
BestMatches
Model Database
21
Local Matches for Retrieval
Matching local regions of shapes to improve
retrieval.
3D Model
BestMatches
Model Database
22
Local Matches for Retrieval
Focusing on matching local regions of shapes can
improve retrieval results.
Using many local descriptors is slow.
3D Model
BestMatches
Model Database
23
Local Matches for Retrieval
Focusing on matching local regions of shapes can
improve retrieval results.
Using many local descriptors is slow. Many
descriptors do not represent distinguishing parts.
3D Model
BestMatches
Model Database
24
Local Matches for Retrieval
Focusing on matching local regions of shapes can
improve retrieval results.
Focusing on the distinctive regions improves
retrieval time and accuracy.
3D Model
BestMatches
Model Database
25
Outline

• Introduction
• Shape Descriptors for Retrieval
• Related Work
• System Overview
• Evaluating Distinction
• Computational Improvements
• Applications
• Conclusion

26
Related Work

• Random Selection
• Perceptual Criteria
• Eye Tracking
• Saliency
• Shape Matching
• Likelihood
• Stability

27
Related Work

• Random Selection
• Perceptual Criteria
• Eye Tracking
• Saliency
• Shape Matching
• Likelihood
• Stability

Mori 2001Frome 2004
28
Related Work

• Random Selection
• Perceptual Criteria
• Eye Tracking
• Saliency
• Shape Matching
• Likelihood
• Stability

Howlett et al.
29
Related Work

• Random Selection
• Perceptual Criteria
• Eye Tracking
• Saliency
• Shape Matching
• Likelihood
• Stability

Other ProjectsLi et al.Novotni et al.Frintrop
et al.Watanabe et al.Hoffman et al.
30
Related Work

• Random Selection
• Perceptual Criteria
• Eye Tracking
• Saliency
• Shape Matching
• Likelihood
• Stability

Johnson et al.Chua et al.
31
Related Work

• Random Selection
• Perceptual Criteria
• Eye Tracking
• Saliency
• Shape Matching
• Likelihood
• Stability

Shan et al.
32
Distinct Regions
Distinctive regions distinguish an object from
other types.
1
Distinction
0
Mesh
Mesh Distinction
33
Key Idea
Determining which regions are important is based
on the other shapes under consideration.
MeshDistinction
Shape DB
Mesh
34
Key Idea
A classified shape database provides a ground
truth to assess which regions of a shape are
distinctive.
MeshDistinction
Classified Shape DB
Mesh
35
Outline

• Introduction
• Shape Descriptors for Retrieval
• Related Work
• System Overview
• Evaluating Distinction
• Computational Improvements
• Applications
• Conclusion

36
System Overview
ShapeDescriptors
Vertex Distinction
Distinct Regions
Mesh
Regions
Distinctive Regions of 3D SurfacesPhilip Shilane
and Thomas FunkhouserACM Transactions on Graphics
37
Constructing Regions
Consistent segmentation is difficult and
unnecessary. Regions could be disjoint, overlap,
or exist at multiple scales.
Segmentation
Katz et al.
38
Constructing Regions
Randomly selecting points on the surface is a
simple solution.
39
Constructing Regions
Regions are created at multiple scales at each
position.
Size of Regions
0.5
1.0
0.25
0.5
1.0
2.0
40
Describing Shapes

• Quick to compute
• Compact to store
• Fast to compare
• Robust to mesh problems
• Discriminating of similar models

41
Describing Shapes

• Quick to compute
• Compact to store
• Fast to compare
• Robust to mesh problems
• Discriminating of similar models

• Experimented with several descriptors
• Shells Shape Histogram
• Harmonic Shape Descriptor
• Fourier Shape Descriptor

42
Measuring Distinction
We would like to calculate distinction scores for
each region.
Mesh
Distinction
43
Measuring Distinction
Our approach is to analyze each region relative
to a database to determine which regions match
shapes that are similar.
Distinction
Mesh
Classified Shape DB
44
Measuring Distinction
We would like to know the distinction of all
combinations of regions.
Distinction
Mesh
Classified Shape DB
45
Measuring Distinction
We make an independence assumption to calculate
distinction for each region.
Distinction
Mesh
Classified Shape DB
46
Distinction Retrieval Performance
The distinction of each local descriptor is based
on how well it retrieves shapes of the correct
class.
QueryDescriptors
Retrieval Results
47
Distinction Retrieval Performance
The distinct descriptors that distinguish between
classes are classification dependent.
QueryDescriptors
Retrieval Results
48
Mapping to Vertices
For visualization, mesh simplification, and icon
generation, it is useful to have distinction
scores for each vertex.
49
Mapping to Vertices
We map distinction scores to each vertex using a
Gaussian weighted average.
50
Outline

• Introduction
• Shape Descriptors for Retrieval
• Related Work
• System Overview
• Evaluating Distinction
• Computational Improvements
• Applications
• Conclusion

51
Evaluating Distinction

• What regions are distinctive?
• Does distinction improve retrieval?

52
Analysis
Distinctive regions of meshes correspond to
important regions that define a class of objects.
53
Analysis
Compared to a database of flying objects, the
helicopter blades are most distinctive.
Mesh Saliency
54
Analysis
For human models in this pose, the elbow area is
most distinctive.
55
Analysis
The distinctive regions of a mesh change with the
database under consideration.
Princeton Shape Benchmark
Vehicle DB
Plane DB
Mesh Saliency
56
Evaluating Distinction

• What regions are distinctive?
• Does distinction improve retrieval?

Partial Matching of 3D Shapes with
Priority-Driven Search Thomas Funkhouser and
Philip ShilaneSymposium on Geometry Processing,
Sardinia, Italy, July 2006
57
Distinction within a Database
Classification
Preprocess
Shape DB
Descriptor DB
DistinctionFunction
Retrieval Evaluation
Local Descriptors
Query
Local Descriptors
Match
Shape
RetrievalList
58
Matching Local Shapes

• 1. Generate local shape features
• 2. Find correspondences minimizing distance
  function

Surface
A
B
59
Matching Local Shapes

• 1. Generate local shape features
• 2. Find correspondences minimizing distance
  function

Points
Features
A
B
60
Matching Local Shapes

• 1. Generate local shape features
• 2. Find correspondences minimizing distance
  function

FeatureCorrespondences
Features
A
B
61
Matching Local Shapes

• 1. Generate local shape features
• 2. Find correspondences minimizing distance
  function

FeatureCorrespondences
Spatial Consistency (Deformation)
Features
A
B
62
Priority-Driven Search

• Use a priority queue to perform best-first search
  for the optimal set of K correspondences in the
  database.

B2
A1
C1
A5
B3
A6
C2
C5
B5
A2
B1
C3
C4
A4
A3
B4
Query Object
Target Objects
Feature Correspondences_______________D(match)
Priority Queue
Low Cost
High Cost
63
Matching Local Shapes
Filter the database to the most distinctive
regions during preprocessing
Mesh
Descriptors
DistinctionScores
4 SelectedDescriptors
64
Alternative Selection Techniques
The database can be filtered according to various
selection techniques.
Distinction
Likelihood
Saliency
Random
65
Shape Matching Results
Using the most distinctive descriptors improves
matching versus other techniques.
66
Shape Matching Results
Using multiple descriptors improves over using a
single descriptor.
67
Shape Matching Results
Using a single descriptor on the surface is
better than using a global descriptor.
68
Shape Matching Results
The oracle case shows an upper bound on matching
performance, suggesting research areas for
improvement.
69
Shape Matching Results
Priority-driven search with distinctive regions
has better retrieval performance than previous
global techniques.
70
Outline

• Introduction
• Shape Descriptors for Retrieval
• Related Work
• System Overview
• Evaluating Distinction
• Computational Improvements
• Applications
• Conclusion

71
Computational Bottlenecks

• Computing distinction requires a full retrieval
  list 2.4 minutes
• Computing 128 descriptors at 4 scales 3
  minutes
• Times are per model

72
Computational Bottlenecks

• Computing distinction requires a full retrieval
  list 2.4 minutes
• Computing 128 descriptors at 4 scales 3
  minutes
• Times are per model

73
Calculating Distinction
Discounted cumulative gain requires a full
retrieval list of n results.
74
Calculating Distinction
Approximate distinction with a short retrieval
list of length k lt n.
75
Calculating Distinction
Cover tree index is a spatial structure for
efficiently finding neighbors in high dimensional
space.
Query descriptor
Image adapted from Qin Lv
76
Calculating Distinction
64 neighbors can be found in a fraction of the
time of searching the whole database, with
similar retrieval accuracy.
77
Updating Distinction
A cover tree index can also be used for updating
distinction

• Build cover tree for DB descriptors
• Record R distances in red-black tree
• Find K neighbors for new descriptor, K gt R
• Update distinction for K neighbors and DB
  descriptors with R distance gt K distance

Kth NeighborK gt R
Rth Neighbor
78
Updating Distinction
A fraction of the DB must be updated depending on
K neighbor search and R-distinction
79
Computational Bottlenecks

• Computing distinction requires a full retrieval
  list 2.4 minutes
• Computing 128 descriptors at 4 scales 3
  minutes
• Times are per model

80
Predicting Distinction
We want a predicted distinction score for each
position on the model.
Descriptors
Distinction
Predict
Selecting Distinctive 3D Shape Descriptors for
Similarity RetrievalPhilip Shilane and Thomas
FunkhouserShape Modeling International,
Matsushima, Japan, June 2006
81
Predicting Distinction
We map descriptors into a compact space where we
learn distinction from a training set.
Distinction
Distinction
Descriptors
Descriptor Domain
82
Prediction Overview
Training
Shape DB
Likelihood
Local Descriptors
Descriptor DB
DistinctionFunction
Retrieval Evaluation
Classification
Query
Likelihood
Evaluate Distinction
Local Descriptors
Match
SelectDescriptors
Shape
RetrievalList
83
Likelihood of Descriptors
Multi-dimensional normal density Johnson 2000
84
Build Predicted Distinction Function
Measure likelihood and retrieval performance of
each descriptor.
85
Build Predicted Distinction Function
Measure likelihood and retrieval performance of
each descriptor.
86
Build Predicted Distinction Function
Measure likelihood and retrieval performance of
each descriptor.
87
Build Predicted Distinction Function
Retrieval performance is averaged within each
likelihood bin.
88
Predicted Distinction Function
A likelihood mapping separates descriptors with
different retrieval performance.
Less Likely
More Likely
89
Predicted Distinction Function
The most common features are the worst for
retrieval.
Less Likely
More Likely
90
Mapping Descriptors to Distinction
During the query phase, we predict distinction as
we generate descriptors
Descriptors
Distinction
Likelihood Mapping
91
Local Matches for Retrieval
3D Model
BestMatches
Model Database
Cost Function
92
Matching with Distinctive Descriptors
3D Model
BestMatches
Model Database
Cost Function
93
Alternative Prediction Methods
Distinction improves retrieval more than other
techniques.
94
Outline

• Introduction
• Shape Descriptors for Retrieval
• Related Work
• System Overview
• Evaluating Distinction
• Computational Improvements
• Applications
• Conclusion

95
Applications
Importance scores on the surface of a mesh are
useful for numerous graphics and geometric
processing applications

• Shape Matching
• Mesh Simplification
• Icon Generation
• Alignment
• Rendering
• Morphing
• Segmentation

96
Mesh Simplification
Distinct Regions
Mesh Saliency
Garland
1,700 tri
300 tri
97
Mesh Simplification
97K tri
2K tri
2K tri zoom
Mesh Saliency
Distinct Regions
98
Icon Generation
99
Icon Generation
Focusing on the most distinctive regions often
(but not always) highlights important features.
100
Outline

• Introduction
• Shape Descriptors for Retrieval
• Related Work
• System Overview
• Evaluating Distinction
• Computational Improvements
• Applications
• Conclusion

101
Conclusion

• PSB data set and tools for evaluating retrieval
  methods
• Defined distinctive regions based on retrieval
  performance
• Analyzed properties of distinction
• Focused shape-matching on distinctive regions
• Calculate/update distinction efficiently
• Predict distinctive regions from training set
• Apps of distinction icons and mesh
  simplification
• Focus on generic methods, independent of shape
  descriptor

102
Future Work

• Efficiency of shape retrieval
• Distinction for an unclassified database
• Scalability of updating distinction
• Predict distinction with improved likelihood
  model or other mapping
• Apply distinction analysis to image retrieval and
  other applications

103
Acknowledgements

• Committee Thomas Funkhouser, Szymon
  Rusinkiewicz, Adam Finkelstein, David Dobkin,
  Andrea LaPaugh, and Kai Li
• Graphics Group and entire CS Department
• Family and friends
• Funding Sources Princeton University
  Departmental Award National Science Foundation
  Grants IIS-0612231, CCR-0093343, CNS 0406415,
  and 11S-0121446Air Force Research Laboratory
  Grant FA8650-04-1-1718Google Research Grant

104
(No Transcript)
105
Extra Slides
106
Constructing Regions
Selecting the vertices biases the sampling to the
underlying polygon representation.
Vertices
107
Applications Shape Matching

• General strategy
• Sample set of points on surface of object
• Build shape descriptorcentered at every pointat
  multiple scales
• Find matches withhigh descriptor similarity
  low geometric deformation

108
Mesh Simplification
Simplifying a complex mesh is important for
improving rendering time.
Fewer polygons(quadric error shown)
Many polygons
Garland et al.
109
Icon Generation
Recent work selects the viewpoint that maximizes
the amount of salient surfaces or that minimizes
the symmetric surfaces.
110
Alternative Prediction Methods
Distinction improves retrieval more than other
techniques.
111
Typical Shape Databases
112
(No Transcript)
113
Typical Shape Databases
114
Typical Shape Databases
115
Evaluation Tools

• Visualization tools
• Precision/recall plot
• Best matches
• Distance image
• Tier image
• Quantitative metrics
• Nearest neighbor
• First and Second tier
• E-Measure
• Discounted Cumulative Gain (DCG)
• Multiple Classifications
• Granularity
• Model Properties

116
Evaluation Tools

• Visualization tools
• Precision/recall plot
• Best matches
• Distance image
• Tier image
• Quantitative metrics
• Nearest neighbor
• First and Second tier
• E-Measure
• Discounted Cumulative Gain (DCG)
• Multiple Classifications
• Granularity
• Model Properties

Dining Chair Desk Chair
117
Related Work

• Random Selection
• Perceptual Criteria
• Eye Tracking
• Saliency
• Shape Matching
• Likelihood
• Stability

Other ProjectsLi et al.Novotni et al.Frintrop
et al.Watanabe et al.Hoffman et al.
118
Analysis
We investigated whether alternative, faster
techniques for calculating importance are
correlated with distinction.

• Distance from center of mass
• Surface Area percentage of mesh enclosed within
  a region
• Likelihood consider each shape descriptor as a
  feature vector
• Saliency change in curvature as calculated by
  saliency.exe

119
Analysis
We investigated whether alternative, faster
techniques for calculating importance are
correlated with distinction.

• Distance from center of mass
• Surface Area percentage of mesh enclosed within
  a region
• Likelihood consider each shape descriptor as a
  feature vector
• Saliency change in curvature as calculated by
  saliency.exe

Correlation Coefficient
120
Analysis
We investigated whether alternative, faster
techniques for calculating importance are
correlated with distinction.

• Distance from center of mass
• Surface Area percentage of mesh enclosed within
  a region
• Likelihood consider each shape descriptor as a
  feature vector
• Saliency change in curvature as calculated by
  saliency.exe

• r -0.04
• r 0.07
• r 0.04
• r 0.03

121
Shape Matching Results
Priority-driven search with distinctive regions
has better retrieval performance than previous
global techniques.
122
Mesh Simplification
When simplifying a mesh, important regions should
be preserved while less important regions are
simplified.
69K tri
1K tri
Garland et al.
123
Mesh Simplification
Quadric Error is related to the distance each
vertex has moved during simplification.
Garland et al.
124
Mesh Simplification
We modified the standard error metric to include
distinction scores for each vertex.
We augment quadric error with distinction
scores. After an edge collapse, the remaining
vertex is given the maximum distinction score of
the two vertices involved.
Garland et al.
125
Icon Generation
Selecting the best view point is important for
creating icons that are quickly recognizable.
Blanz et al.
126
Icon Generation
Important surfaces should be visible when
creating icons for a catalog of shapes.
Good View Poor View
Good View Poor View
Lee et al.
Podolak et al.
127
Evaluation Tools

• Visualization tools
• Precision/recall plot
• Best matches
• Distance image
• Tier image
• Quantitative metrics
• Nearest neighbor
• First and Second tier
• E-Measure
• Discounted Cumulative Gain (DCG)
• Multiple Classifications
• Granularity
• Model Properties

Man-made vs. Natural (2)