Curvature%20Prior%20for%20MRF-based%20Segmentation%20and%20Shape%20Inpainting

1
Curvature Prior for MRF-based Segmentation and
Shape Inpainting
This work was supported bu EU projects
FP7-ICT-247870 NIFTi and FP7-ICT-247525 HUMAVIPS
and the Czech project 1M0567 CAK
DAGM-OAGM 2012
Alexander Shekhovtsov, Pushmeet Kohli and Carsten
Rother
TexPoint fonts used in EMF. Read the TexPoint
manual before you delete this box. AAAAAAAA
2
Motivation

• Would like to have a model tailored for the
  specific shape class

Looked at higher-order MRFs and Field of Experts
Experts
Pixels

• Focus on the curvature cost as a simple example
  of a shape model

3
Motivation

• How can we model shapes with higher-order models?

- nonlinear function of linear filters -
continuous variables
Black and Roth. (2009) Field of Experts
hard pattern
Komodakis and Paragios (2009) Pattern-based
Higher Order Potentials
Rother et al. (2009) Sparse Higher Order
Potentials
expert state
soft pattern
4
Curvature in Discrete Setting

• Most of the works go for explicit edge
  representation (discrete setting)

Brukstain (2001) approximation
Cell-complex
Schoenemann et al. (2009) Schoenemann, Kahl ,et
al. (2011) Schoenemann, Kuang, et al.
(2011) Strandmark and Kahl (2011)
straight on a large scale, but highly penalized

• Convex relaxations in the continuous setting
  Bredies et al. (2012), Goldluecke and Cremers
  (2011)

5
The Model

• Keep the segmentation pixel-wise but assess
  curvature from a local window

window of the higher-order model
think of the curve with the lowest possible
curvature consistent with discretization
lager windows have a better chance of a more
accurate estimate
You would never thought of this curve, unless you
know something
6
The Model
Rother et al. (2009)

• pixel-wise segmentation

densely, at every pixel location, there is a
higher-order term
restriction to the window
Energy
window locations
Higher-order term
for fixed y a modular (linear) functions of x
lower envelope of the modular functions of x
7
The Model

• What this model can do?

in the minimum
or
8
Minimization

• Good news minimization reduces to pairwise model

expands as
-join optimization in segmentation and latent
variables y
can combine with standard MRF models

• Bad news still hard to optimize ?

- BP-S/TRW-S (Kolmogorov, 2006) implementation
saving a factor of NP (number of patterns) memory
(lazy asymmetric message handling)
9
BP Schedule dependence
Solution by BP-S (max-product) (swep from left
to right, from top to bottom)
Input (inpaint the gray area)
10
Parallel (Synchronous) BP
Parallel BP
TRW-S
curvature (old model)
curvature length
curvature more length
11
Learning

• For the case of curvature model, we have a
  simpler learning problem we can learn the model
  locally.

Generate smooth curves
Discretize
true curvature cost (analytic)
Fit the lower envelope model K-means like
algorithm, needs good initialization
12
Learning
example (circle radius model cost)
cost function to learn
learned patterns
size 8x8 96 in total
predefined patterns assign 0 cost to
off-boundary locations
13
Learning

• Approximation Error

discrete approximation vs. exact contour integral
Testing shape samples (analytic)
(overestimating)
14
Shape Inpainting
area for inpainting
known segmentation
inpainted segmentation
15
Shape Inpainting
16
Shape Inpainting
17
Segmentation
Input with user seeds
(saturation)
curvature strength
18
Segmentation (skip)
Input with user seeds
Standard length regularization
regularization strength
19
Segmentation (more)

• Extending the model we added artificially an ear
  pattern.
• its cost was tuned manually

after
before
20
Curvature and Length Regularization
only curvature
curvature length
curvature more length
21
Towards Object Inpainting
area for completion
our shape inpainting
interactive segmentation
Texture added automatically thanks to Barnes et
al. (2009)