Motion Segmentation over Image Sequences Using Multiway Cuts and Affine Transformations

1
Motion Segmentation over Image Sequences Using
Multiway Cuts and Affine Transformations

• Braga Natarajan

2
Organization
Motion Segmentation
Energy minimization, graph cuts, multiway cuts
Motion models, affine transformation
Combining multiway cuts and affine transformations
Algorithms and results
3
Motion segmentation

• Image segmentation by color, texture, shape,
  and motion
• Motion very important cue
• Divide image into regions exhibiting relatively
  different coherent motion
• Ideal motion
• segmentation

4
Why motion segmentation?

• Number of applications
• Robotics
• Video coding/compression
• Video indexing/retrieval
• Object tracking, surveillance
• Intermediate image processing task output given
  to high level computer vision problems

5
Motion representation

• Pixel motion represented by a 2D vectorEither
  dense (optical flow) or sparse (features)
• Models

translation 2 parameters
affine 6 parameters
6
Lucas-Kanade affine estimation

• Minimize nonlinear equation
• Iterative linear solution (Newtons method)

current image
2x2 matrix
2x1 vector
residue
pixel location
next image
unknown parameters (A, d), 6x1 vector
6x6 gradient matrix
6x1 error vector
Lucas Kanade, 1981 Shi Tomasi, 1994
7
(No Transcript)
8
Energy minimization

• Motion segmentation as energy minimization
• challenge Thousands of dimensions!
• Solution Graph cut methods Boykov, Veksler,
  and Zabih 1999
• accurate
• fast

Smoothness penalties
Data penalties
9
Image Correspondence

• Motion segmentation comes under the general
  category of image correspondence
• Goal of image correspondence assign labels to
  every pixel in the image
• Energy functions can be devised once labels are
  defined and listed
• What do labels mean?

10
Stereo Correspondence
11
Motion
12
Binary cut

• Maximum flow-minimum cut, Ford-Fulkerson 1956
• graphs constructed - a node per pixel
• source and sink terminals binary variables 0 and
  1
• t-link weight data penalty, n-link weight
  smoothness penalty
• Minimum s-t cut, pixels get label 0 or 1 based on
  what links are cut

t-link
n-link
cut n-link
13
Multiway cuts

• More than 2 labels typical for motion and stereo
  multiway cuts
• Repeated binary cuts by forming binary graphs for
  every pair of labels alpha-beta swap
• Repeated binary cuts by forming binary graphs for
  a particular label and the existing label, for
  all labels alpha expansion

14
Multiway cuts
15
Parent algorithm

• Multiway cut for stereo and motion with slanted
  surfaces, Birchfield and Tomasi 1999
• Combines multiway cuts and affine transformation
• Works iteratively by progressive refinement of
  displacement functions of labels
• Algorithm re-implemented, proposed algorithms are
  extensions to this paper

16
(No Transcript)
17
(No Transcript)
18
Motion segmentation over image sequences

• Parent algorithm when employed on sequence of
  images, does not produce consistent results
• Also computationally inefficient to exhaustively
  search over all translational displacement
  functions for every frame.

frame1, 5 segments
frame2, 6 segments
19
Changes to parent algorithm
result from previous frame pair
control number of loop iterations
do affine merge at the end
20
Algorithm
frame t
frame t1
parent algorithm, affine merge

• Run parent algorithm on first frame and get
  correct number of segments, parameters are fixed
• Set number of iteration loop for affine update of
  displacement functions to a constant
• Initial motion segment image for next frame is
  predicted by affine warping of current motion
  segments and re-estimation of displacement
  functions
• Final iterative affine merge step merges
  neighboring regions

predict label image
initialize parent algorithm for next frame
frame t1
frame t2
21

• Affine merging of regions if neighbor regions
  within threshold, then merge if number of
  segments is still more relax threshold and repeat
• This step similar to over segmentation step but
  does not involve energy computation, hence
  threshold dependent

22

• Results for frames 2, 10, 19 and 25 are shown.
• Number of motion segments maintained
• Algorithm took 71.18 seconds to run on 27 frames,
  parent algorithm took 97.04 seconds
• Boundaries between segments are not crisp due to
  occlusions and lack of texture

23

• Taxi sequence, algorithm works well for frames 1
  to 36. Frame 5, 18 are shown.
• Failure for 36 to 40 due to small motion of taxi
  and two components for right vehicle
• Frame 40 failure of affine merge shown
• Right vehicle segmentation poor due to occlusion
  by tree

24
Hard constraint points for stereo

• Another extension to stereo correspondence
• Cost functional not able to preserve small and
  thin long objects in depth maps
• Multiway cuts smoothes out small regions

25
Normalized correlation

• Normalized correlation performed, unambiguous
  disparity points are chosen and initialized as
  hard constraint points
• These points initialize multiway cuts, number of
  iterations of affine updating is controlled

sum of squared differences inside window
scan line
left image
right image
clear minimum
ambiguous minimum
26
(No Transcript)
27
Occlusion detection

• Errors of regions in between motion layers due to
  movement of foreground over background
• Selective occlusion detection is done using
  estimated affine parameters
• Assumption multiway cuts labels occluded areas
  with the label of the foreground

28

• Compute residues of region a and region b based
  on affine parameters of both region 1 and region
  2 and pick the worst.
• The occluded region has the worst residue because
  it has no matching region in the next frame.

29
(No Transcript)
30
Conclusions

• Studied, analyzed and implemented multiway cuts
  and affine transformation techniques
• All implementation in C from scratch, using
  Kolmogorov and Blepo
• Two extensions to the parent algorithm motion
  segmentation over image sequences and hard
  constraint points for stereo
• Simple occlusion detection presented
• Results are reasonable

31
Future work

• Spatiotemporal multiway cuts for segmenting
  object in the video volume
• Redesigning cost functionals to improve
  segmentation results
• Integrate occlusion detection with multiway cuts
  for getting cleaner borders.

32
Thank You