An Iterative Optimization Approach for Unified Image Segmentation and Matting

1
An Iterative Optimization Approach for Unified
Image Segmentation and Matting

• Jue Wang1 and Michael F. Cohen2
• University of Washington
• Microsoft Research

2
Introduction

• Image Matting

Observed Image
Alpha Matte
3
Introduction

• Trimap-based Matting

Background
Unknown
Foreground
Original
Trimap
Matte
4
Introduction

• Matting After Segmentation
• Trimap generation can be erroneous

Automatically Generated Trimap
Original Users input
Graph-cut Segmentation GrabCut, Rother et al.,
LazySnapping, Li et al., SIGGRAPH 2004
5
Introduction

• Our Approach Unified Segmentation and Matting
• No explicit trimap is required

Iterative Optimization
Initial Input
Final Matte
6
Related Work

• Blue Screen Matting
• Mishima et al. 93
• Smith et al. SIGGRAGPH96

Problem is simplified by photographing foreground
objects against a constant-colored background
7
Related Work

• Bayesian Matting, Chuang et al. CVPR2001
• Bayesian framework MAP solver
• Extended to video in Chuang et al. SIGGRAPH 2002

Bayesian Image Matting
Bayesian Video Matting
8
Related Work

• Poisson Matting, Sun et al. SIGGRAPH 2004
• Formulate matting as solving Poisson equations
• Assumption intensity change in the foreground
  and background is smooth
• User can interactively manipulate the matte

9
Limitations of A Trimap

• Key requirement for A trimap
• should be as accurate as possible
• Automatically generated trimap
• is not optimal
• can be erroneous
• User specified trimap
• can be very tedious
• to create

10
Our Approach

• Iterative Matte Optimization
• Solving a Conditional Random Field (CRF)

11
CRF vs. MRF
Observations Y
Hidden Field X

• In our system we solve CRF by iteratively solving
  MRFs

12
Iterative Solver

• Color Sampling The conditional part.

Background Samples
Foreground Samples
13
Iterative Solver

• Solve MRF

14
Iterative Solver
Step 1 Color Sampling
Step 2 Solve MRF by Belief Propagation
15
MRF Set Up

• Attributes of a Pixel/Node

Foreground samples
Background samples
Observed color
Observations
Hidden Node
Quantized alpha level
Likelihoods
Estimated foreground color
1. Limit the size of the MRF
Estimated background color
2. Guide foreground/background sampling
Uncertainty
3. Setting weights for nodes
16
Iterative Solver

• Matte is estimated in a front-propagation fashion

Region of Consideration Modeled as MRF
Definite Foreground
Definite Background
17
MRF Set Up

• Attributes of a Pixel/Node

Foreground samples
Background samples
Observed color
Observations
Hidden Node
Quantized alpha level
Likelihoods
Estimated foreground color
Estimated background color
Uncertainty
18
MRF Set Up
Local Sampling Local samples from low
uncertainty areas are given high weights.

• Color Sampling

• Global Sampling
• Train a GMM model on all foreground samples
• Assign each sample to a single Gaussian
• For a given node, choose the nearest Gaussian in
  color space, and collect samples belonging to
  this Gaussian
• Global samples are given lower weight.

19
MRF Set Up

• Data Costs

Foreground samples
Background samples
Observed color
Observations
Hidden Node
Quantized alpha level
Likelihoods
Estimated foreground color
Estimated background color
Uncertainty
20
MRF Set Up

• Data Cost

Foreground samples
Background samples
Observed color
Observations
Hidden Node
Quantized alpha level
Likelihoods
Estimated foreground color
Estimated background color
Uncertainty
21
MRF Set Up

• Neighborhood Cost

p
q
22
Solving MRF by Belief Propagation

• Minimize

T(p)
Msg(p, T(p))
Msg(T(p),p)
Msg(p, R(p))
Msg(L(p), p)
Msg(R(p),p)
Msg(p,L(p))
L(p)
p
R(p)
Msg(B(p),p)
Msg(p,B(p))
B(p)
23
Solving MRF by Belief Propagation

• Minimize

T(p)
Msg(p, R(p))
p
L(p)
R(p)
B(p)
24
Solving MRF by Belief Propagation

• Minimize

T(p)
p
L(p)
R(p)
B(p)
25
CRF Update
Foreground samples
Background samples
Observed color
Observations
Hidden Node
Quantized alpha level
Estimated foreground color
Estimated background color
Uncertainty
26
CRF Update

• Defining Uncertainties
• Each pixel is associated with an uncertainty
  value
• Pixels with low uncertainties will contribute
  more for solving the MRF

27
Results
0
3
6
9
14
28
Results
Original image with user input
Extracted Matte
29
Results

• Comparison with Bayesian Matting

User specified trimap
Extracted matte using Bayesian matting
30
Results
Original image with user input
Extracted matte
User specified trimap
Estimated matte by Bayesian Matting
31
Results
Original image with user input
Extracted matte
User specified trimap
Estimated matte by Bayesian Matting
32
Extension to Video
Automatic initialization on frame 2
Frame 1
Matte1
Matte 2
Matte 20
Matte 5
Matte 15
Matte 10
33
Extension to Video
Extracted Matte
Original Video
All User Inputs
34
Failure Mode
Original image with user input
Extracted matte
35
Solution
Extracted matte using our system
User specified rough trimap
Extracted matte using Bayesian Matting
36
Timings

• A brute force implementation is computational
  expensive
• 15-20 min for a 640480 input image
• Speedups
• Fast Belief Propagation ideas Felzenszwalb et
  al., CVPR 04
• Hierarchical methods
• Using gradient information
• Current system 20-30 seconds per image

37
Future Directions

• Combining all visual information
• Color, texture, shape

38
Summary

• Solve a CDF for unified segmentation and matting
• CDF solved iteratively
• each iteration solves an MRF using Belief
  Propagation
• Advantages
• No accurate trimap is required
• Efficient for large semi-transparent objects
• Extends to video
• Disadvantages
• Computational expensive
• No real-time interaction
• Based only on color information