Binocular Stereo Vision

1
Binocular Stereo Vision

• Region-based stereo matching algorithms
• Properties of human stereo processing

2
Solving the stereo correspondence problem
3
Measuring goodness of match between patches
(1) sum of absolute differences
S pleft pright
Optional divide by n number of pixels in
patch
(1/n)
patch
(2) normalized correlation
(pleft pleft) (pright pright)
S
(1/n)
spleft spright
patch
4
Region-based stereo matching algorithm

• for each row r
• for each column c
• let pleft be a square patch centered
  on (r,c) in the left image
• initialize best match score mbest to 8
• initialize best disparity dbest
• for each disparity d from drange to drange
• let pright be a square patch centered on
  (r,cd) in the right image
• compute the match score m
  between pleft and pright
• (sum of absolute differences)
• if (m lt mbest), assign mbest m and dbest
  d
• record dbest in the disparity map at (r,c)

(normalized correlation)
How are the assumptions used??
5
The real world works against us sometimes
left
right
6
Example Region-based stereo matching, using
filtered images and sum-of-absolute differences
(from Carolyn Kim, 2013)
7
Properties of human stereo processing
Use features for stereo matching whose position
and disparity can be measured very precisely
Stereoacuity is only a few seconds of visual
angle difference in depth ? 0.01 cm at a viewing
distance of 30 cm
8
Properties of human stereo processing
Matching features must appear similar in the
left and right images
For example, we cant fuse a left stereo image
with a negative of the right image
9
Properties of human stereo processing
Only fuse objects within a limited range of
depth around the fixation distance
Vergence eye movements are needed to fuse objects
over larger range of depths
10
Properties of human stereo vision
We can only tolerate small amounts of vertical
disparity at a single eye position
Vertical eye movements are needed to handle
large vertical disparities
11
Properties of human stereo processing
In the early stages of visual processing, the
image is analyzed at multiple spatial scales
Stereo information at multiple scales can be
processed independently
12
Neural mechanisms for stereo processing
G. Poggio colleagues complex cells in area V1
of primate visual cortex are selective for stereo
disparity neurons that are selective for a larger
disparity range have larger receptive fields
zero disparity at fixation distance near in
front of point of fixation far
behind point of fixation
13
In summary, some key points

• Image features used for matching
• simple, precise locations, multiple scales,
  similar between left/right images
• At single fixation position, match features over
  a limited range of horizontal vertical
  disparity
• Eye movements used to match features over larger
  range of disparity
• Neural mechanisms selective for particular ranges
  of stereo disparity

14
Matching features for the MPG stereo algorithm
zero-crossings of convolutions with ?2G operators
of different size
L
rough disparities over large range
M
accurate disparities over small range
S
15
large w
left
large w
right
small w
left
small w
right
correct match outside search range at small scale
16
large w
left
right
vergence eye movements!
small w
left
right
correct match now inside search range at small
scale
17
Stereo images (Tsukuba, CMU)
18
Zero-crossings for stereo matching

-


19
Simplified MPG algorithm, Part 1

• To determine initial correspondence
• (1) Find zero-crossings using a ?2G operator with
  central positive width w
• (2) For each horizontal slice
• (2.1) Find the nearest neighbors in the right
  image for each zero-crossing fragment in the
  left image
• (2.2) Fine the nearest neighbors in the left
  image for each zero-crossing fragment in the
  right image
• (2.3) For each pair of zero-crossing fragments
  that are closest neighbors of one another, let
  the right fragment be separated by dinitial from
  the left. Determine whether dinitial is within
  the matching tolerance, m. If so, consider the
  zero-crossing fragments matched with disparity
  dinitial

m w/2
20
Simplified MPG algorithm, Part 2
To determine final correspondence (1) Find
zero-crossings using a ?2G operator with reduced
width w/2 (2) For each horizontal slice (2.1)
For each zero-crossing in the left
image (2.1.1) Determine the nearest
zero-crossing fragment in the left image that
matched when the ?2G operator width was
w (2.1.2) Offset the zero-crossing fragment by
a distance dinitial, the disparity of the
nearest matching zero-crossing fragment found at
the lower resolution with operator width w (2.2)
Find the nearest neighbors in the right image for
each zero-crossing fragment in the left
image (2.3) Fine the nearest neighbors in the
left image for each zero-crossing fragment in the
right image (2.4) For each pair of zero-crossing
fragments that are closest neighbors of one
another, let the right fragment be separated by
dnew from the left. Determine whether dnew is
within the reduced matching tolerance, m/2. If
so, consider the zero-crossing fragments matched
with disparity dfinal dnew dinitial
21
Coarse-scale zero-crossings
w 8 m 4
Use coarse-scale disparities to guide fine-scale
matching
w 4 m 2
Ignore coarse-scale disparities
w 4 m 2