Texture Optimization for Example-based Synthesis

1
Texture Optimization for Example-based Synthesis

• Vivek Kwatra1,2 Irfan Essa1Aaron Bobick1Nipun
  Kwatra1

1GVU Center/College of Computing, Georgia
Tech 2Computer Science Dept., UNC Chapel Hill
2
Motivation


InputTexture
InputFlow Field
OutputSequence
3
Example-based Texture Synthesis
4
Example-based Texture Synthesis

1. Dynamic texturing (animation/video)
2. Greater control external knobs, e.g., flow,
   shape, illumination
3. Explicit texture quality evaluation and refinement

5
Contributions I

• Global Optimization for texture synthesis
• Global texture quality metric
• Iterative multi-scale optimization

6
Contributions II

• Flow-guided texture animation
• Flow incorporated into optimization
• Applications fluid rendering, flow visualization

7
Related Work (a selected sample)
TextureOptimizationKwatra05
MRF AnnealingPaget98
Histogram MatchingHeeger95
Local vs. Global
Graphcut TexturesKwatra03
Non-parametricsamplingEfros99,Wei00
Image QuiltingEfros01
Pixel vs. Patch
8
Related Work (a selected sample)
Better Quality
TextureOptimizationKwatra05
MRF AnnealingPaget98
Histogram MatchingHeeger95
Local vs. Global
Graphcut TexturesKwatra03
Non-parametricsamplingEfros99,Wei00
Image QuiltingEfros01
Pixel vs. Patch
9
Related Work (a selected sample)
Better Quality
TextureOptimizationKwatra05
MRF AnnealingPaget98
Greater Controllability
Histogram MatchingHeeger95
Local vs. Global
Graphcut TexturesKwatra03
Non-parametricsamplingEfros99,Wei00
Image QuiltingEfros01
Pixel vs. Patch
10
Related Work

• Flow-guided Synthesis
• Advected Textures Neyret03
• Flow-based Video Editing Bhat04
• Optimization
• Image-based Priors Fitzgibbon03
• Space-time Video Completion Wexler04

11
Flow-guided Texture Animation

• Flow consistency
• Perceived motion similar to flow
• Texture similarity
• Shape, size, orientation of texture elements
  similar to source

Source Texture
Texture Similarity
FlowConsistency
Flowing Target
Target Flow
12
Naive Approach

• Ignore Texture Similarity
• Warp each frame via flow field

Source Texture
Warp
TargetFrames
X1
Xn
X0
Texture Similarity
FlowConsistency
Flowing Target
Target Flow
13
Naive Approach

• Texture structure not maintained

Source Texture
Texture Similarity
FlowConsistency
Flowing Target
Target Flow
14
Instead of

• Warp

TargetFrames
15
Instead of

• Warp

TargetFrames
16
Our Approach

• Warp Correct

TargetFrames
17
Correct Operation

• Link to original goals

TargetFrames
X
X
Correct
Warp
W
18
Correct Operation

• Link to original goals

TargetFrames
X
X
Correct
Flow Consistency
Warp
W
19
Correct Operation

• Link to original goals

Source
Texture Similarity
TargetFrames
X
X
Correct
Flow Consistency
Warp
W
20
Optimization of Energy / Cost

• Energy Flow Energy Texture Energy

Source
Texture Similarity
TargetFrames
X
X
Correct
Flow Consistency
Warp
W
21
Flow Energy
TargetFrames
X
X
Flow Energy
Warp
W
22
Texture Energy

• Tricky!
• Pixel-by-pixel comparison of source and target
  not possible
• Compare texture elements
• Local pixel neighborhoods
• Want each target neighborhood to be similar to
  some source neighborhood

23
Texture Energy
Z(source)
X(target frame)
24
Energy for Single Neighborhood
Z(source)
p (pixel)
X(target frame)
25
Energy for Single Neighborhood
Z(source)
Xp (neighborhood)
X(target frame)
26
Energy for Single Neighborhood
Z(source)
Xp (neighborhood)
X(target frame)
27
Energy for Single Neighborhood
(nearest neighbor) Zp
Z(source)
Xp (neighborhood)
X(target frame)
28
Energy for Single Neighborhood
(nearest neighbor) Zp
Z
Xp (neighborhood)
Texture Energy(single neighborhood)
X
29
Texture Energy for Entire Image
Z
X
30
Texture Energy for Entire Image
Z
X
? individual neighborhood energy
31
Optimization

• Optimize Total Energy of target frame
• Initialize X ? W(target frame ? warped frame)
• Iteratively improve target frame

32
Optimization Intuition
Zp
Texture Force
Xp
Source
Target Frame
Flow Force
Wp
Warped Frame
33
Optimization Intuition
Zp
Xp
Blend
Source
Target Frame
Wp
Warped Frame
34
Overlapping Neighborhoods
Zp
Xq
Zq
Blend
Source
Xp
Target Frame
Wp
Warped Frame
35
We are not done yet!
Nearest neighbor may change
Zp
Xp
Source
Target Frame
Wp
Warped Frame
36
Iterative Algorithm

• Step 1
• Find Nearest Source Neighborhoods
• Step 2
• Blend Source Warped Neighborhoods
• Repeat until convergence

37
Iterative Algorithm

• Step 1
• Find Nearest Source Neighborhoods
• Minimize E( X ) w.r.t Zp
• Approximate nearest-neighbors
• Hierarchical Clustering Johnson67
• Tree-Structured Vector Quantization Wei00

38
Iterative Algorithm

• Step 2
• Blend Source Warped Neighborhoods
• Minimize E( X ) w.r.t X
• Set ? Solve Linear System

39
Robust Optimization

• Quadratic Texture Energy Et sensitive to outlier
  neighborhoods
• Non-quadratic Et more robust
• Et( X ) ?p kXp Zpkr, 0 lt r lt 2

40
Robust Optimization

• Quadratic Texture Energy Et sensitive to outlier
  neighborhoods
• Non-quadratic Et more robust
• Et( X ) ?p kXp Zpkr, 0 lt r lt 2
• Solve weighted quadratic at each iteration
• Et( X ) ?p kXp Zpkr-2. kXp Zpk2
• ?p ?p . kXp
  Zpk2

41
Minimize Texture Energy Alone

• Novel Static Texture Synthesis Algorithm

Source
Texture Similarity
X
X
Correct
Flow Consistency
W
42
Texture Energy Graph

• Random Initialization
• Multiple
• Resolution Levels
• Neighborhood Sizes
• Progressively refined output

43
Texture Energy Graph

• Random Initialization
• Multiple
• Resolution Levels
• Neighborhood Sizes
• Progressively refined output

44
Results Texture Synthesis
45
(No Transcript)
46
Results Texture Synthesis
Input
Wei-LevoyWei00
Image QuiltingEfros01
Graph-cutsKwatra03
Texture OptimizationKwatra05
47
Results Texture Synthesis
Image QuiltingEfros01
Graph-cutsKwatra03
Texture OptimizationKwatra05
48
Results Flow-guided Synthesis
49
Results Timing

• Static texture synthesis
• 5-10 iterations per resolution level
• Flowing texture synthesis
• 20-60 seconds per frame

Resolution One iteration Total time
64x64 2.5 sec 20 sec
128x128 10 sec 2 min
256x256 25 sec 8.5 min
50
Summary

• Global Optimization for controllable texture
  synthesis
• Explicit evaluation of texture quality
• Iterative progressive refinement of texture
• Flow-guided texture animation within optimization
  framework

51
Limitations and Future Work

• Optimization finds local minima
• Blurry / misaligned texture elements
• Texture scale should be compatible with flow
  field
• Extensions
• Controllable video textures
• Other control criteria like shape, illumination

52
Acknowledgements

• Eugene Zhang for flow design code
• Greg Turk, Gabriel Brostow, Delphine Nain, Ravi
  Ruddarraju, Gaurav Chanda, Stephanie Brubaker,
  Mitch Parry, Pei Yen
• Computational Perception Lab, Geometry Group _at_
  Georgia Tech
• CMU Graphics Lab
• Web page
• http//www.cc.gatech.edu/cpl/projects/textureoptim
  ization/