Speeding Up MRF Optimization using Graph Cuts for Computer Vision - PowerPoint PPT Presentation

Loading...

PPT – Speeding Up MRF Optimization using Graph Cuts for Computer Vision PowerPoint presentation | free to download - id: 52aa59-NmNmM



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Speeding Up MRF Optimization using Graph Cuts for Computer Vision

Description:

Title: CVPR 09 Author: CVIT Last modified by: Vibhav Created Date: 6/13/2009 9:20:15 AM Document presentation format: On-screen Show Company: IIIT-Hyd – PowerPoint PPT presentation

Number of Views:4
Avg rating:3.0/5.0
Date added: 22 June 2020
Slides: 59
Provided by: CVIT
Learn more at: http://cvit.iiit.ac.in
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Speeding Up MRF Optimization using Graph Cuts for Computer Vision


1
Speeding Up MRF Optimization using Graph Cuts for
Computer Vision
  • Vibhav Vineet
  • Adviser Prof. P. J. Narayanan

2
Labelling Problem
Extracting Foreground Pixels
Disparity map calculation
Image Denoising
Extracting Foreground Object
Left Tsukuba Image
Disparity Map
Noisy House Image
Denoised Image
Flower Image
Pixel Labeling Assigning a label to each pixel
in image.
Image Denoising Involves assigning denoised
intensity value to each pixel in image.
3
Labelling Problem
  • - MAP Estimation
  • To find the best possible configuration.
  • But the complexity increases
  • With the number of variables/pixels
  • With the number of labels in the label set
  • Using joint probability or conditional
    probabilities to evaluate the best possible
    configuration
  • Very hard with the limited computation and
    memory power
  • Energy minimization method
  • MAP MRF equiivalence
  • Methods provide approximate solution at a
    moderate times
  • Generally, in computer vision an energy function
    involve unary cost and pairwise interactions
    between variables.

4
Labelling Problem
Image-Graph Equivalence
Graph G( V, E )
Unary Cost ( Per Vertex Cost)
Cost of Assignment for fg is low. Cost of
Assignment for bg is high.
5
MAP-MRF Formulation
MAP(X)
Min Energy(X)
MAP estimation of a configuration X is equivalent
to the minimum energy defined over the
configuration
Energy Data Term Smoothness Term
Graph Cuts in Computer Vision
Image
Energy Function
Graph Construction
st-MinCut
6
Graph Construction
Image-Graph Equivalence
Foreground Pixels
Background Pixels
Image
Graph G(V,E)
Add n-edges
  • Graph Constructed for vision problems
  • Grid graphs
  • Low connectivity
  • Connectivity is limited to 4, 8, or 27

7
The st-Mincut Problem
  • Given a Graph G(V,E,W) and two vertices s and t.
  • Partition G into two disjoint components
    containing s and t respectively such that sum of
    edge weights from s to t is minimum

s
Mincut
t
8
Computing the st-Mincut
  • Solve the dual Maximum Flow problem
  • Two approaches
  • Edmond Karps Augmenting path method
  • Goldbergs Push-Relabel method

st-Mincut
Max Flow
Dual
In every network, the maximum flow equals the
cost of st-mincut
9
Edmond Karp Method
  • Initialize flow in G to 0
  • Find a shortest path from s to t.
  • Augment the path with minimum possible flow
  • Repeat until there exists a path from s to t
  • Initialize flow in G to 0
  • Find a shortest path from s to t.
  • Augment the path with minimum possible flow
  • Repeat until there exists a path from s to t
  • Initialize flow in G to 0
  • Find a shortest path from s to t.
  • Augment the path with minimum possible flow
  • Repeat until there exists a path from s to t
  • Initialize flow in G to 0
  • Find a shortest path from s to t.
  • Augment the path with minimum possible flow
  • Repeat until there exists a path from s to t
  • Initialize flow in G to 0
  • Find a shortest path from s to t.
  • Augment the path with minimum possible flow
  • Repeat until there exists a path from s to t
  • Initialize flow in G to 0
  • Find a shortest path from s to t.
  • Augment the path with minimum possible flow
  • Repeat until there exists a path from s to t
  • Initialize flow in G to 0
  • Find a shortest path from s to t.
  • Augment the path with minimum possible flow
  • Repeat until there exists a path from s to t

Current Flow 0
Current Flow 0
Current Flow 3
Current Flow 3
Current Flow 10
Current Flow 10
Current Flow 16
  • Flow lt Edge Capacity
  • Edge Capcity must be positive

10
Edmond Karp Method
  • Initialize flow in G to 0
  • Find a shortest path from s to t.
  • Augment the path with minimum possible flow
  • Repeat until there exists a path from s to t

s
87
37
6
7
3
7
17
1
1
Current Flow 16
t
  • Flow lt Edge Capacity
  • Edge Capcity must be positive

11
Goldbergs Push-Relabel Algorithm
  • Initialize excess flow and heights in G
  • Perform an applicable Push or Relabel operation
  • Repeat until there exists an applicable push or
    relabel operation
  • Initialize excess flow and heights in G
  • Perform an applicable Push or Relabel operation
  • Repeat until there exists an applicable push or
    relabel operation
  • Initialize excess flow and heights in G
  • Perform an applicable Push or Relabel operation
  • Repeat until there exists an applicable push or
    relabel operation
  • Initialize excess flow and heights in G
  • Perform an applicable Push or Relabel operation
  • Repeat until there exists an applicable push or
    relabel operation
  • Initialize excess flow and heights in G
  • Perform an applicable Push or Relabel operation
  • Repeat until there exists an applicable push or
    relabel operation
  • Initialize excess flow and heights in G
  • Perform an applicable Push or Relabel operation
  • Repeat until there exists an applicable push or
    relabel operation
  • Initialize excess flow and heights in G
  • Perform an applicable Push or Relabel operation
  • Repeat until there exists an applicable push or
    relabel operation
  • Initialize excess flow and heights in G
  • Perform an applicable Push or Relabel operation
  • Repeat until there exists an applicable push or
    relabel operation
  • Initialize excess flow and heights in G
  • Perform an applicable Push or Relabel operation
  • Repeat until there exists an applicable push or
    relabel operation
  • Initialize excess flow and heights in G
  • Perform an applicable Push or Relabel operation
  • Repeat until there exists an applicable push or
    relabel operation
  • Initialize excess flow and heights in G
  • Perform an applicable Push or Relabel operation
  • Repeat until there exists an applicable push or
    relabel operation
  • Initialize excess flow and heights in G
  • Perform an applicable Push or Relabel operation
  • Repeat until there exists an applicable push or
    relabel operation
  • Initialize excess flow and heights in G
  • Perform an applicable Push or Relabel operation
  • Repeat until there exists an applicable push or
    relabel operation
  • Initialize excess flow and heights in G
  • Perform an applicable Push or Relabel operation
  • Repeat until there exists an applicable push or
    relabel operation
  • Initialize excess flow and heights in G
  • Perform an applicable Push or Relabel operation
  • Repeat until there exists an applicable push or
    relabel operation

Current Flow 0
Current Flow 0
Current Flow 0
Current Flow 0
Current Flow 0
Current Flow 0
Current Flow 0
Current Flow 9
Current Flow 9
Current Flow 9
Current Flow 9
Current Flow 9
Current Flow 9
Current Flow 9
Current Flow 16
  • Flow lt Edge Capacity
  • Edge Capcity must be positive

12
Goldbergs Push-Relabel Algorithm
  • Initialize excess flow and heights in G
  • Perform an applicable Push or Relabel operation
  • Repeat until there exists an applicable push or
    relabel operation

s
Height
0
6
37
87
0
0
7
3
17
t
1
0
1
6
Current Flow 16
  • Flow lt Edge Capacity
  • Edge Capcity must be positive

13
Motivation
  • Fast Computation Required
  • Robot navigation, surveillance, video processing
    etc
  • Video Processing at real time
  • You tube and other web-servers
  • Large images Processing
  • Even our offshelf cameras take high resolution
    images
  • Interactive tools

14
Mapping to CUDA
Image
Grid
Thread per pixel
Image
CUDA Grid
CUDA Block
15
Push-Relabel Algorithm on CUDA
  • Push is an local operation with each node sending
    flows to its neighbors.
  • Relabel is also a local operation, each vertex
    updates its own height.
  • Problems faced
  • Read After Write consistency
  • Synchronization of threads

16
Handling Problems using Atomics
  • Push operations can performed without any read
    after write inconsistencies
  • Relabel is a per vertex operation
  • Employing atomic Capabilities and combining the
    push and pull kernels
  • Push Kernel
  • Relabel Kernel
  • Lowers Global memory access, empirically faster
    convergence is observed.

17
The Push Kernel
  • Load heights from the global memory to the shared
    memory.
  • Synchronize threads ensuring the completion of
    load operation.
  • Push flows to eligible neighbors atomically.
  • Update the edge-weights atomically in the
    residual graph.
  • Update excess flow atomically in the residual
    graph.

18
The Relabel Kernel
  • Load height from the global memory to the shared
    memory.
  • Synchronize ensuring the completion of load
    operation.
  • Compute the minimum height of all neighbors and
    set own height to plus one of this.
  • Write the new height to global memory.

19
Using Shared Memory
  • Per CUDA block requires (Block_size2 X
    Block_size2) memory to be loaded into the
    shared memory
  • Corner pixels need heights from other blocks

Block size 2
Thread
Block size 2
Height needed per thread
Block
20
Heuristics on Push and Relabel
  • On grid graphs Global relabel (BFS based) is an
    expensive operation
  • Local relabel perform better empirically
  • Multiple pushes can be performed before applying
    a relabel step using
  • For most general graphs m3 and k7 are found to
    be optimal.

(mPush Relabel)k Global Relabel
21
Stochastic Cuts
  • MRF consists of simple and difficult pixels.
  • Simple pixels get their correct labels in few
    initial iterations
  • Difficult (few) pixels exchange flows with their
    neighbors in later iterations
  • Stochastic Cuts processes pixels based on their
    activity
  • Activity is based on change in flows from
    previous to current iteration. Low activity is
    observed for simple pixels
  • Heuristically process simple pixels after a fixed
    number of iterations

22
Experimental Results
23
Experimental Results
Image Size Time CPU (ms) Time Non Atomic (ms) Time Atomic (ms) Time Stochastic (ms)
Sponge 640x480 142 28 16 11
Flower 608x456 188 33 26 16
Person 608x456 140 31 27 20
Synthetic 1Kx1K 655 19 10 7
24
Graph Reparameterization
S
2
9
Graph Cuts
Graph Cuts
1
2
5
4
t
25
Graph Reparameterization
S
2
9
Graph Cuts
Graph Cuts
1
2
5
4
t
Graph Reparameterized
Graph Reparameterized
No change in cut
S
S
22
9
4
9
1
Graph Cuts
Graph Cuts
1
2
2
52
4
7
4
t
t
26
Dynamic Cuts
EA
SA
EB
SB
Problem Instance 1
Problem Instance 2
Problems instances where they differ slightly.
Solving each independently is computationally
expensive Example Continuous frames in a video
27
Dynamic Cuts Steps Involved
  • Edge capacities are updated and reparameterized
    using
  • Previous frame edge capacities
  • Previous frame residual flow
  • Current frame edge capacities

Previous Frame
ci
fi
Previous Frame after st-MinCut
ri
Current Frame
ci
Updation Step ri ri ci - ci
ri
Approximate cut using previous frame and its
st-MinCut
fi
Reparameterization Step rsi 0 rit cit -
fit fsi csi
Final st-MinCut of current frame
28
Dynamic Cuts are parallizable
  • Updation and Reparameterization are independent
    and parallizable operations, work locally at
    every vertex.
  • st-Mincut is performed using a parallel
    implementation of Push Relabel algorithm.

29
Dynamic Cuts Empirically
  • Running time depends on the percentage of weights
    that changed
  • On a low resolution video, the dynamic cuts
    takes about 2 ms compared to 7 ms on the same
    image for the st-MinCut

Consecutive frames of a video segmented using
dynamic cuts
30
The Multilabeling problem
  • Multi-way cut on any graph is an NP-Hard problem
    for L gt 2
  • Approximate solutions based on graph cuts
  • a-Expansion
  • aß-Swap

31
The a-Expansion
  • 1 Initialize the MRF with an arbitrary labeling
    X
  • 2 For each label alpha \in L do
  • 3 Construct the graph based on the current
    configuration
  • 4 Perform one a-Expansion step (st-cut)
  • 5 Update the configuration if energy decreases
  • 6 End For
  • 7 Repeat steps 2 to 6 till convergence.

Step 2-6 is a cycle and 3-5 is an iteration
32
Incremental a-Expansion
  • Reusability of flows, as in dynamic MRF
  • Better initializations for next graph cut
  • Incremental/Dynamic
  • Reuse the flows from label to label and and
    re-cycle flows from cycle to cycle.

Cycle 1
Label1
Label2
Input
Label3
Cycle 2
33
Incremental a-Expansion Results
Tsukuba
Teddy
Penguin
Panorama
34
Incremental a-Expansion Results
Total Timings on Different Datasets
35
Processing on High Detailed Scene
  • High Detailed Scene
  • High Resolution Image
  • High Dynamic Ranges of Colors
  • Wide View Angles.
  • Challenges
  • High Computation Cost
  • High Memory Requirement
  • Interaction with high resolution images
  • Statistics of image sizes available on Google
    images.
  • An overwhelming fraction of images are of size 2
    to 10 million pixels.
  • Only 0.6 of fewer images had more than 40 mega
    pixels.

36
Processing on High Detailed Scene
Define E(x) for coarsest image
Final Result for this level
Define E(x) for next finer level
Final Result at this level
Solve an optimization problem at the coarser
level to dynamically update the optimization
instance for the next level for better
initialization.
37
Pyramid Reparameterization
Pyramid Construction
Pyramid is Constructed. Input largest image at
the base of the pyramid. Each pixel coarser
level image is mean of 4 pixels at the previous
finer level
38
Pyramid Reparameterization
39
Pyramid Reparameterization
Computationally Expensive Graph Cuts
Graph at the current level
Final Residual Graph at the current level
40
Pyramid Reparameterization
Graph Cuts
Upsampled graph of previous level
Reuse of flows
Difference between two graphs
Cheaper solution
Computationally Expensive Graph Cuts
Graph at the current level
Final Residual Graph at the current level
41
Upsampling Rules
Graph Upsampling
Graph Cuts
Residual Graph Upsampling
42
Upsampling Rules
Graph Upsampling
Graph Cuts
Residual Graph Upsampling
43
Upsampling Rules
Graph Upsampling
Graph Cuts
Residual Graph Upsampling
44
Image Segmentation Results
Horse 3.3 MP (2048x1600)
45
Image Segmentation Results
46
Interactive Image Segmentation Tool
  • User Interaction Important in foreground/backgrou
    nd separation

User 2
User 1
  • Results of a user study on image size for
    comfortable manipulation for two display sizes.
  • Average subjective response for six image sizes.
  • Images that are larger than the display is
    disfavored users.

47
Pyramid Segmentation System
Display Window
  • User interacts at the display window of
    comfortable size

Actual Image
  • Actual Segmentation goes on in background on
    other levels
  • Quick Segment
  • Display the segmentation results on this display
    image
  • Provides perceptual response to start planning
    further interactions

48
User Study
Results of User Study on CPU and GPU version of
Pyramid Segmentation With GrabCut and Quick
Selection
Interaction Time
Response Time
Total Time
Subjective Response of Users
49
Multiresolution alpha-expansion
  • - Build pyramid of graphs.
  • Perform alpha-expansion at a lower resolution
    graph.
  • Save the initial and final residual graphs for
    all the labels.
  • Upsample and reparameterize the previous
    resolution initial and final graphs and current
    resolution initial graph.
  • Perform alpha-expansion at this level.
  • Repeat this for all the levels in the pyramid.

50
Stereo Correspondence
Image size 1328 x 1104 Number of Labels
(Disparity) 200 - 290
51
Stereo Correspondence
Optimization Time
Total Time (Optimization time graph
construction time energy function calculation
time)
Running time in seconds for stereo correpondence
using Pyramid Cuts on the GPU (G-PyCut), the CPU
(C-PyCut) and a single level Graph Cuts(GCuts).
A speed up of 5-6 times on the CPU is observed.
52
Image Denoising
Image size 1000 x 1000 Number of Labels
(Disparity) 256
53
Image Denoising
Total Time (Optimization time graph
construction time energy function calculation
time)
Optimization Time
Running time in seconds for stereo correpondence
using Pyramid Cuts on the GPU (G-PyCut), the CPU
(C-PyCut) and a single level Graph Cuts(GCuts).
A speed up of 5-6 times on the CPU is observed.
54
Future Work
  • Higher order Interactions of variables in MRF
  • Computationally more challenging
  • Modelling this on our hierarchical and
    multiresolution framework
  • Using multiple GPUs to parallelize the
    alpha-expansion
  • Better interactive tools
  • Both global and local interactions

55
Conclusion
  • Two methods to optimize basic graph cuts
    algorithm
  • Using facilities provided by parallel
    accelerators like GPU
  • Modelling graph cuts on hierarchical and dynamic
    framework for better initialization
  • Graph Cuts methods proved very instrumental
    solving many computationally challenging problems
  • Successes of graph cuts -gt Promising future in
    the realm of energy minimization methods

56
Related Publications
  • P. J. Narayanan, Vibhav Vineet and Timo Stitch.
    Fast Graph Cuts on the GPU. GPU Computing Gems
    (GCG), Volume 1 Dec. 2010 (Book Chapter).
  • Vibhav Vineet and P. J. Narayanan. Solving
    Multi-label MRFs using incremental
    alpha-expansion move on the GPUs. In Proceeding
    of Ninth Asian Conference on Computer Vision.
    (ACCV-2009), China, 2009.
  • Vibhav Vineet and P. J. Narayanan. CUDA Cuts
    Fast Graph Cuts on the GPU. In Proceeding of CVPR
    workshop on Visual Computer Vision on GPUs
    (CVGPU-2008), Alaska, USA, 2008.
  • Vibhav Vineet, Pawan Harish, Suryakant Patidar
    and P. J. Narayanan. Fast Minimum Spanning Tree
    for Large Graphs on the GPU. In Proceeding of ACM
    SIGGRAPH High Performance Graphics (HPG-2009),
    New Orleans, LA, USA, 2009.
  • Pawan Harish, Vibhav Vineet and P. J. Narayanan.
    Large Graph Algorithms for Massively
    Multithreaded Architectures. IIIT Tech Report,
    IIIT/TR/2009/74.
  • CUDA Cuts Fast Graph Cuts on the GPU.
    http//cvit.iiit.ac.in/index.php?pageresources.
    (Software).
  • Vibhav Vineet, Pawan Harish, Suryakant Patidar
    and P. J. Narayanan. Fast Minimum Spanning Tree
    for Large Graphs on the GPU. GPU Computing Gems
    (GCG). (Book Chapter).

57
Changes made to the thesis
  • Reviewer1 (Dr. Kishore)
  • Tables containing experimental results on more
    standard images (around 60) images added.
  • Sections on related and background works
    expanded.
  • Reviewer 2 (Dr. Srinivasan)
  • Missed references added to the related work
    section.
  • Formal results on speed up for dynamic graph
    cuts on video segmentation added to the result
    section.
  • Figure captions properly referenced with the
    paper of Kohli and Torr.
  • Other minor changes made as recommended.

58
Thank You
About PowerShow.com