Depth and Motion Discontinuities

1
Depth and Motion Discontinuities

• Stan Birchfield
• Ph.D. oral defense
• Stanford University
• January 1999

2
Discontinuities

• PHOTOMETRIC
• GEOMETRIC

intensity, color, texture, ...
camera
camera
surface
surface
Depth
Motion
3
Importance of Discontinuities

• Fundamental to image understanding
• Object boundaries(well-defined)
• Assumption of continuity
• Occlusion

4
Motivation
IMAGES
LOW-LEVEL FILTERS
HIGH-LEVEL TASKS
segmentation
tracking
detection
classification
camera control
video retrieval
5
Evidence for Discontinuities

• LOCAL
• T-junctionsParida et al. 1997, Ruzon Tomasi
  1999
• Occlusion Toh Forrest 1990, Wixson 1993
• Multi-modal depth or motionSpoerri Ullman
  1987, Little Gillett 1990
• Changes in intensity, color, texture, Canny
  1986, Wang Binford 1994
• GLOBAL
• Continuity of discontinuitiesMarr Poggio
  1979
• Changes in depth or motion fieldThompson et al.
  1982, Birchfield Tomasi 1998

6
Complications from Sampling

• Amount of discontinuity
• Curvature

?
7
Finding and Using Discontinuities
Depth discontinuities from stereo
Motion discontinuities from a sequence
Maximum flow for stereo and motion
Using discontinuities to track heads
8
Depth Discontinuities from a Stereo Pair
epipolar constraint
lamp
disparity
wall
pixel
Use dynamic programming to minimize
discontinuity penalty
dissimilarity
9
Discontinuity Penalty
u(3) lt 3u(1)
GOOD
u(3) 3u(1)
BAD
penalty
occlusion length
10
Stereo Algorithm

• Matches pixels directly with convex penalty
  function(no preprocessing or windows)
• Handles large untextured regions
• Solves sampling problem
• Handles slanted surfaces
• Is fast

11
Handling Untextured Regions
Assumption Depth discontinuities are
accompanied by intensity
edge
Naïve constraint Depth discontinuities lie near
intensity edge
Our constraint Depth discontinuities lie to a
particular side of
intensity edge
Discontinuities in left (right) scanline lie
to the
left (right) of intensity
edge
Intensity edges (x-derivative with low
threshold)
With naïve constraint
With our constraint
12
Discontinuity Lies to the Left of an Edge
far object
Physical setup
near object
left camera
right camera
depth discontinuity
Matches violate assumption
edge
Obvious matches
Matches are consistent with assumption
depth discontinuity
edge
13
The Problem of Image Sampling
Left scanline
Right scanline
Absolute difference
14
A Dissimilarity Measure That is Insensitive to
Image Sampling
d(xL,xR)
xL
xR
d(xL,xR) mind(xL,xR) ,d(xR,xL)
Our dissimilarity measure
15
Analysis of Dissimilarity Measure
intensity function
Concave/convex regions measure is guaranteed to
work
Inflection points measure works in practice
when lens is
defocused to remove aliasing
(inflection points _at_ linear convex, concave)
16
Analysis of Dissimilarity Measure
As a function of lens defocus
absolute difference
our measure

1
defocus
aliasing
Tc
fc
cutoff frequency
17
Speeding Computation
occlusion
RIGHT
Pruning bad nodes
Standard
Disparity map
time
LEFT
Ours
maximum disparity
depth discontinuity
18
Handling Slanted Surfaces
Independent scanlines no coherence
disparity
Postprocessing forbid propagation if disparity
changes by just one level
column
19
Results
20
More Results
Images from JISCT data set
21
Finding and Using Discontinuities
Depth discontinuities from stereo
Motion discontinuities from a sequence
Maximum flow for stereo and motion
Using discontinuities to track heads
22
Motion Discontinuities from a Sequence

• Similar to stereo(matching pixels to fit
  piecewise-smooth function)
• But different(2D search, non-rigid
  transformation, many images)
• How far can we get using sparse features?

23
Motion Video
24
Difficulties of Image Sequences

• Instantaneous velocityfrom sampled positions
• Accumulation of evidence
• Frame of reference, motion model

feature
position
?
time
feature 1
threshold
position
feature 2
background
time
feature
threshold
position
background
time
25
Image Strain
26
Motion Algorithm

• Select and track features
• Group
• Trace boundariesbetween groups

27
Finding and Using Discontinuities
Depth discontinuities from stereo
Motion discontinuities from a sequence
Maximum flow for stereo and motion
Using discontinuities to track heads
28
Why Maximum Flow?

• Our stereo algorithm (Dynamic programming)
• Good results on difficult images
• Fast
• But, suboptimal(processes rows, then columns
  independently)
• Newer algorithms (Maximum flow)
• Set up graph, find maximum flow ---minimum cut
  yields disparity or motion map
• Able to look at whole image

29
Minimizing a 2D Cost Function
Minimize
d
30
Maximum Flow for Stereo and Motion
Global algorithm Roy Cox 1998, Ishikawa
Geiger 1998
penalty
BAD
discontinuity amount
Local algorithm Boykov et al. 1998
GOOD
31
Challenges for Maximum Flow
and slant of table
With our intensity variation constraint and
large penalty
With small penalty
With large penalty
and slant of surfaces
32
Finding and Using Discontinuities
Depth discontinuities from stereo
Motion discontinuities from a sequence
Maximum flow for stereo and motion
Using discontinuities to track heads
33
Problem
ZOOM
TILT
PAN
CHALLENGES rotation multiple people
zoom
APPLICATIONS video conferencing
distance learning
34
Previous Methods
Moving people in background
3D Rotation
Template
N
Y
35
Head Tracking Algorithm
MODEL
36
Evaluation of Head Tracker
1. Tracks head in real time on standard
hardware 2. Insensitive to - full
360-degree out-of-plane rotation -
arbitrary camera movement (including zoom)
- multiple moving people - severe but
brief occlusion - hair/skin color, hair
length, facial hair, glasses
37
Headtracker Video
38
Finding and Using Discontinuities
Depth discontinuities from stereo
Motion discontinuities from a sequence
Maximum flow for stereo and motion
Using discontinuities to track heads