Finding Glass

1
Finding Glass

• Kenton McHenry
• Jean Ponce
• David Forsyth

2
Background

• Layer Seperation
  (Szleski, Avidan, and Aniandan, CVPR'00),
  (Levin,
  Zomet, and Weiss, CVPR'04)

• 3D Structure
  (Hata, Saitoh, Kumamura and Kaida,
  ICPR'96)
  (Ben-Ezra and Nayar, ICCV'03)
  (Miyazaki,
  Kagesawa and Ikeuchi, ICCV'03)
  (Murase, ICCV'90)

• Recognition
  (Osadchy, Jacobs, and Ramamoorthi, ICCV'03)

• Segmentation
  (Singh and Huang, CVPR'03)

3
(Adelson and Anandan, AAAI'90)
I aIB e

• 0 lt a 1
• e 0

4
Classifying Junctions
Non-Reversing transparency, ambiguous depth
ordering
Single-Reversing transparency
Double-Reversing no transparency
5
(Singh and Huang, CVPR'03)
6
(Singh and Huang, CVPR'03)
7
Our Goal
8
The Background

• The appearance of a glass object changes with the
  background (i.e. the scene w/o any transparent
  objects)
• We have seen how knowledge of the background can
  be extremeley useful in reconstructing
  transparent surfaces
• Ideal situation know the background, use
  background subtraction

9
Glass Objects and their Edges
Why?

• Highlights
• Mirrors
• Hysteresis

10
Adelson et al Revisited

• Though they focus on junctions they are
  classifying edges
• The proposed rules are binary cues between a
  transparent object and its background

11
Proposed Method

• Break edges into small segments and classify them
  based on the information from the two sides
• Properties of glass transparency, refraction and
  reflection

12
Cues

• Transparency
• Color Similarity
• Overlay Consistency
• Refraction
• Texture Distortion
• Blurring
• Reflection
• Highlights

13
Color Similarity

• (HSV) Hue
• (HSV) Saturation

14
Overlay Consistency
15
Texture Distortion

• Filer Bank 2 scales, 6 orientations (0,p)

16
Blurring

• DCT
• Shift in mean in frequency space

17
Highlights

• Highlights on smooth shiny surfaces tend to have
  a profile with a sharp spike
  (Healey and Binford, '87),
  (Nayar, Ikeuchi and Kanade, '91)

18
Highlights

• Iteratively fit a line to perimeter (starting
  from threshold of 1.0)
• Plot line fit errors

19
Highlights
20
Single Classifier

• 5 cues provide 6 values
• SVM with Gaussian kernel
• Must be conservative with false positives
• Classifier can achieve high accuracy on training
  data
• Move hyperplane until true positives lt 30

21
Multiple Classifiers

• If we were to consider the 6 values as logical
  propositions we could write

glass ? similar_color ? high_alpha ?
(low_emmission ? highlight ?
smoother ? distortion)
22
Multiple Classifiers

• We can re-write the previous statement as four
  different statements of three propositions

glass ? similar_color ? high_alpha ?
low_emmission glass ? similar_color ? high_alpha
? highlight glass ? similar_color ? high_alpha ?
smoother glass ? similar_color ? high_alpha ?
distortion
23
Multiple Classifiers

• Each proposition is a seperatley trained
  classifier of lower dimension
• Combining the sub-classifiers
• Logical OR
• Weighted Sum
• Exponential Model

24
Global Integration

• Due to conservativeley built classifiers we will
  have few positives
• Hysteresis connect positves along a common edge
• Snakes
  (Kass, Witkin, Terzopoulos,
  '87)

25
Experiments

• Training Set 15 images, 6 with glass objects in
  front of various backgrounds, 9 with no glass
  objects
• 333 positive examples
• 4581 negative examples
• Test Set 50 images, 35 with glass objects, 15
  with no glass objects at all

26
Experiments
Precision 68.76 56.04 58.78 56.04 73.70
Single SVM Multiple SVM's OR Multiple SVM's
Weighted Sum Multiple SVM's Exponential
Model Multiple SVM's Weighted Sum (sampled)
27
Results
28
Results
29
Results
30
Results
31
Classifying Regions as Glass

• We need not restrict ourselves to regions around
  edges
• Given two regions we ask the question is one
  region a glass covered version of the other?

32
Over Segmentation

• We want regions of similar material
  (Felzenszwalb and Huttenlocher, '04)
• Can adjust size of super-pixels (degree of
  over-segmentation) with smaller k values
• Use color, texture, and edgels to set weights

33
Discrepency

• We use our previous classifier as a measure of
  how much two regions don't belong two the same
  material (i.e. glass and not glass)
• Use distance from seperating hyperplane (Platt,
  '00)
• Large values far on the postive glass side
• Small values (negative) far on the not glass
  side
• Reasonable if data takes a normal distribution
• Drop blur cue since DCT can't be done on
  non-rectangular regions.

34
Ambiguities

• Discrepency is high for a material and a glass
  covered version of that material, but also for
  two completley different materials
• Above example has two possible segmentations

35
Affinity
Aij 1 aij / p
36
Affinity

• Because of refraction most straight background
  edges that pass through the glass will appear
  broken
• Edges from glass contour ussually the longest
  smoothest edges in the area

37
Affinity
38
Certainty of Discrepency/Affinity

• High discrepency likely different materials
• Low discrepency cannot ascertain whether one
  regions is glass and the other is background
• High affinity likely same material
• Low affinity not very informative, edge path may
  just have been broken

39
Objective Function

• We wish to maximize our measures
• First term maximize discrepency between glass
  and other stuff
• Second term maximize affinity in the glass
• Third term minimize affinities between glass and
  other
• Combinatorial problem!

40
Relaxed Objective Function

• Relax region constraints
• Treat pixels as a sampling of an underlying
  continuous function

41
Geodesic Active Contours
42
Curve Evolution
43
Experiments
Precision 68.76 56.04 58.78 56.04 73.70 77.03

Single SVM Multiple SVM's OR Multiple SVM's
Weighted Sum Multiple SVM's Exponential
Model Multiple SVM's Weighted Sum
(sampled) Proposed Method
44
Results
45
Results
46
Results
47
Results
48
Results