Image Completion using Global Optimization

1
Image Completion using Global Optimization

• Presented by Tingfan Wu

2
The Image Inpainting Problem
3
Outline

• Introduction
• History of Inpainting
• Camps Greedy Global Opt.
• Model and Algorithm
• Markov Random Fields (MRF) Inpainting
• Belief Propagation (BP)
• Priority BP
• Results
• Structural Propagation

4
Method Type
PriorityTexture Synth.
Need User Guidance
5
Exampled Based MethodJigsaw Puzzle
PatchesNot Available
6
Method Type
PriorityTexture Synth.
Need User Guidance
7
Greedy v.s Global Optmization
Greedy Method
Global Optimization
Refine Globally ?
Cannot go back ?
8
Outline

• Introduction
• History of Inpainting
• Camps Greedy Global Opt.
• Model and Algorithm
• Markov Random Fields (MRF) Inpainting
• Belief Propagation (BP)
• Priority BP
• Results
• Structural Propagation

9
Random Fields / Belief Network
Random Variable(Observation)
Good Project Writer?(High Project grade)
Smart Student?(High GPA)
Good Test Taker?(High test score)
Good Employee (No Observation yet)
Edge Dependency

• RFRandom Variables on Graph
• Node Random Var. (Hidden State)
• Belief from Neighbors, and Observation

10
Story about MRF
Hidden Markov Model (HMM)
Office Helper Wizard

• (Bayesian) Belief Network (DAG)
• Markov Random Fields (Undirected, Loopy)
• Special Case
• 1D - Hidden Markov Model (HMM)

11
Inpainting as MRF optimization

• Node Grid on target region, overlapped patches
• Edge A node depends only on its neighbors
• Optimal labeling (hidden state) that minimizing
  mismatch energy

12
MRF Potential Functions

• Mismatch (Energy) between ..
• Vp (Xp ) Source Image vs. New Label
• Vpq(Xp, Xq) Adjacent Labels
• Sum of Square Distances (SSD) in Overlapping
  Region

13
Global Optimizatoin
min
14
Outline

• Introduction
• History of Inpainting
• Camps Greedy Global Opt.
• Model and Algorithm
• Markov Random Fields (MRF) Inpainting
• Belief Propagation (BP)
• Priority BP
• Results
• Structural Propagation

15
Belief Propagation(1/3)
Good Project Writer?(High Project grade)
Smart Student?(High GPA)
Good Test Taker?(High test score)
Good Employee (No Observation yet)

• Undirected and Loopy
• Propagate forward and backward

16
Belief Propagation(2/3)

• Message Forwarding
• Iterative algorithm until converge

O(Candidate2)
Candidates at Node Q
Candidates at Node P
Neighbors (P)
17
Belief Propagation(3/3)
18
Priority BP

• BP too slow
• Huge candidates ? Timemsg O(Candidates2)
• Huge Pairs ?Cannot cache pairwise SSDs.
• Observations
• Non-Informative messages in unfilled regions
• Solution to some nodes is obvious (fewer
  candidates.)

19
Human Wisdom
Candidates
Start from non-ambiguous part And Search
for Brown feathergreen grass
Nobody start from here
20
Priority BP

• Observations
• needless messages in unfilled regions
• Solution to some nodes is obvious (fewer
  candidates.)
• Solution Enhanced BP
• Easy nodes goes first (priority message
  scheduling)
• Keep only highly possible candidates (maintain a
  Active Set)

21
Priority Pruning
Discard Blue Points
High Priorityprune a lot
Low Priority
Candidates sorted by relative belief
Pruning may miss correct label
22
Candidates after Pruning
Active Set (Darker means smaller)
Histogram of candidates
Similar candidates
23
A closer look at Priority BP

• Priority Calculation
• Priority 1/(significant candidate)
• Pruning (on the fly )
• Discard Low Confidence Candidates
• Similar patches ? One representative (by
  clustering)
• Result
• More Confident ?More Pruning
• Confident node helps increase neighbors
  confidence.
• Warning
• PBP and Pruning must be used together

24
Extensions (Optional)

• Adding constraints by modifying distance function
• Spatial Coherence fill target region with large
  chunks.
• ? Good for texture synthesis
• Patch blending with weights confidence
• Multi-scale inpainting.
• Create pseudo source image at fine scale
• Recover both coarse and fine texture
• Fast SSD by FFT.

25
Outline

• Introduction
• History of Inpainting
• Camps Greedy Global Opt.
• Model and Algorithm
• Markov Random Fields (MRF) Inpainting
• Belief Propagation (BP)
• Priority BP
• Results
• Conclusion
• Structural Propagation

26
Results-Inpainting(1/3)
Darker pixels ? higher priority Automatically
start from salient parts.
27
Results-Inpainting(2/3)
28
Results-Inpainting(3/3)

• Up to 2minutes / image (256x170) on P4-2.4G

29
More Texture Synthesis

• Interpolation as well as extrapolation

30
(No Transcript)
31
Conclusion

• Priority BP
• Confident node first candidate pruning
• Generic applicable to other MRF problems.
• Speed up
• MRF for Inpainting
• Global optimization
• avoid visually inconsistence by greedy
• Priority BP for Inpainting
• Automatically start from salient point.

32
Sometimes

• Image contains hard high-level structure
• Hard for computers
• Interactive completion guided by human.

33
Potential Func. For Structural Propagation

• User input a guideline by human region.
• Potential Function respect distance between lines

Jian Sun et al, SIGGRAPH 2005
34
Video

• LinkMicrosoft Research