Dynamic Texture Synthesis

1
Dynamic Texture Synthesis

• Vivek Kwatra

Google Inc.
2
Acknowledgements

• For slides
• Arno Schödl
• For making this course happen
• Li-Yi Wei
• Sylvain Lefebvre
• Greg Turk
• Other contributors
• Antonio Criminisi
• Alexei A. Efros
• Aaron Hertzmann
• Yanxi Liu

3
Still photos
4
Video Clips
5
Video Textures
6
Spatio-temporal Textures
7
Spatio-Temporal Textures
8
Controllable Texture Animation
9
3D Flow and Texture
10
This Talk

• Focus on the presented examples
• Video Textures
• Graphcut Textures
• Texture Optimization for Flow Guided Synthesis
• Fluid Texturing
• Relate to 2D texture synthesis along the way

11
Video Textures

• Schödl et al.00
• Synthesize infinitely long videos
• Find transitions by matching frames

time
time
12
Finding good transitions

• Compute L2 distance Di, j between all frames

frame i
frame j
Similar frames make good transitions
13
Transition costs

• Transition from i to j if successor of i is
  similar to j
• Cost function Ci?j Di1, j

14
Probabilistic Transitions

• Probability for transition Pi?j inversely related
  to cost
• Pi?j exp ( Ci?j / s2 )

high ?
low ?
15
Preserving dynamics
16
Preserving dynamics
17
Preserving dynamics

• Cost for transition i?j
• Ci?j wk Dik1, jk

18
Preserving dynamics effect

• Cost for transition i?j
• Ci?j wk Dik1, jk

19
Region-based analysis

• Divide video up into regions
• Generate a video texture for each region

20
Limitation

• Problematic (blurry) for chaotic motions

21
Graphcut Textures

• Seamless transitions for spatio-temporal
  extension
• (Kwatra et al. 2003)

22
Related work

• Image Quilting (Efros Freeman01)
• Seamless texture synthesis in 2D
• Find seams between image patches
• Dynamic Programming

23
Graph cuts for Video

• Find seams in video
• Seam is a surface in 3D space-time
• Dynamic programming intractable in 3D

24
Approach (Image case)

• Treat the whole input as a single patch
• Perceptual matching at two scales
• Patch Placement (coarse scale)
• Seam Optimization (fine scale)

25
Approach (Video case)

• Spatio-temporal offset for a video patch
• More flexibility in copying pixels
• Allows spatial extension of video

time
space
26
Seam Optimization

• Based on
• Energy Minimization via Graph Cuts(Boykov et
  al.99)
• Originally developed for stereo

27
Seam Optimization
B
Construct graph such thatGraph Mincut ? Best
Seam
1
4
7
1
4
7
2
5
8
2
5
8
3
6
9
3
6
9
A
28
Seam Optimization
B
Construct graph such thatGraph Mincut ? Best
Seam
A
29
Seam Optimization
B
Potential Seam Cost
Constraint
20
100
8
8
1
4
7
30
8
8
40
10
A
B
2
5
8
35
8
8
15
85
3
6
9
30
Seam Optimization
B
1
4
7
2
5
8
3
6
9
Seam
A
20
1
4
7
30
10
A
B
2
5
8
15
3
6
9
31
Results Image Textures
32
Results Natural images
33
Results Perspective
34
Video Results
35
Spatio-Temporal Extension
36
Spatio-Temporal Extension
37
Spatio-Temporal Extension
38
Results Interactive Merging
39
Results Interactive Merging
40
Results Interactive Merging
41
Back to Dynamic Textures

• Graph cuts preserve natural appearance
• BUT how do we control behavior ?

42
Fluid Texturing

• Combine
• Motion from Simulation
• Appearance from Texture

43
Fluid Texturing
44
Related Problem in 2D

• Flow-guided texture animation
• 2D Planar flow fields

Texture
Flow Field
Output
45
Flow-guided Texture Animation

• Flow consistency
• Perceived motion similar to flow
• Texture similarity
• Shape, size, orientation of texture elements
  similar to source

Source Texture
Texture Similarity
FlowConsistency
Flowing Target
Target Flow
46
Naive Approach

• Ignore Texture Similarity
• Warp each frame via flow field

Source Texture
Warp
TargetFrames
X1
Xn
X0
Texture Similarity
FlowConsistency
Flowing Target
Target Flow
47
Naive Approach

• Texture structure not maintained

Source Texture
Texture Similarity
FlowConsistency
Flowing Target
Target Flow
48
Instead of

• Warp

TargetFrames
49
Instead of

• Warp

TargetFrames
50
Our Approach

• Warp Correct

TargetFrames
51
Correct Operation

• Link to original goals

TargetFrames
X
X
Correct
Warp
W
52
Correct Operation

• Link to original goals

TargetFrames
X
X
Correct
Flow Consistency
Warp
W
53
Correct Operation

• Link to original goals

Source
Texture Similarity
TargetFrames
X
X
Correct
Flow Consistency
Warp
W
54
Optimization of Energy / Cost

• Energy Flow Energy Texture Energy

Source
Texture Similarity
TargetFrames
X
X
Correct
Flow Consistency
Warp
W
55
Flow Energy
TargetFrames
X
X
Flow Energy
Warp
W
56
Texture Energy

• Tricky!
• Pixel-by-pixel comparison of source and target
  not possible
• Compare texture elements
• Local pixel neighborhoods
• Want each target neighborhood to be similar to
  some source neighborhood

57
Texture Energy
Z(source)
X(target frame)
58
Energy for Single Neighborhood
Z(source)
p (pixel)
X(target frame)
59
Energy for Single Neighborhood
Z(source)
Xp (neighborhood)
X(target frame)
60
Energy for Single Neighborhood
Z(source)
Xp (neighborhood)
X(target frame)
61
Energy for Single Neighborhood
(nearest neighbor) Zp
Z(source)
Xp (neighborhood)
X(target frame)
62
Energy for Single Neighborhood
(nearest neighbor) Zp
Z
Xp (neighborhood)
Texture Energy(single neighborhood)
X
63
Texture Energy for Entire Image
Z
X
64
Texture Energy for Entire Image
Z
X
? individual neighborhood energy
65
Optimization

• Optimize Total Energy of target frame
• Initialize X ? W(target frame ? warped frame)
• Iteratively improve target frame

66
Optimization

• Two variables we need to optimize over

67
Optimization

• Two variables we need to optimize over

Target Frame
68
Optimization

• Two variables we need to optimize over

Source Neighborhoods
69
Optimization

• Alternate between the two

70
Iterative Algorithm

• Step 1
• Minimize E( X ) w.r.t Zp
• Find Nearest Source Neighborhoods

71
Iterative Algorithm

• Step 2
• Minimize E( X ) w.r.t X
• Set ? Solve Linear System

72
Iterative Algorithm

• Step 2
• Minimize E( X ) w.r.t X
• Set ? Diagonal Matrix in this
  case

73
Texture Energy Alone

• Texture synthesis in 2D
  

Source
Texture Similarity
X
X
Correct
Flow Consistency
W
74
Texture Energy Graph

• Random Initialization
• Multiple
• Resolution Levels
• Neighborhood Sizes
• Progressively refined output

75
Texture Energy Graph

• Random Initialization
• Multiple
• Resolution Levels
• Neighborhood Sizes
• Progressively refined output

76
Results Flow-guided Synthesis
77
3D Fluid Texturing

• Combine
• Motion from Simulation
• Appearance from Texture

78
3D Fluid Texturing

• Texture synthesis over dynamic fluid surface
• Governed by simulated velocity field

time
79
3D Fluid Texturing

• Texture information stored on mesh vertices
• Color
• Local texture orientation

time
80
Texture Representation

• Orientation vector field
• 2D coordinate system at each sample point

81
Recall Texture Energy
Nearest Neighbor
Xp
Zp
Texture Energy(single neighborhood)
82
Fluid Simulation

• Navier-Stokes Equations (incompressible)

Momentum Conservation
Mass Conservation
83
Fluid Simulation

• Navier-Stokes Equations (incompressible)

Momentum Conservation
Transport / Advection
84
Fluid Simulation

• Navier-Stokes Equations (incompressible)

Transport / Advection
Advection of Passive Scalar Quantities
85
Dynamic Fluid Surface

• Level Set of Advected Implicit function

Advection of Passive Scalar Quantities
86
Dynamic Fluid Surface

• Level Set of Advected Implicit function

No Correspondence between Surfaces!
Advection of Passive Scalar Quantities
87
Texture Transport

• Texture Information
• Color
• Orientation
• No correspondence between surfaces

88
Color Transport

• Back-track source of each vertex

Surfacen1
Surfacen
2D Illustration
89
Color Transport

• Back-track source of each vertex

Surfacen1
Surfacen
2D Illustration
90
Color Transport

• Back-tracking source
• Scalar advection of vertex coordinates
• ? Spatial coordinate u Velocity vector

91
Texture Transport

• Texture Information
• Color
• Orientation (needs special handling)

92
Orientation Vector Transport

• Orientation is a vector
• Scalar advection only translates the vector

Scalar Advection of Components
Original Orientation
Velocity Field
93
Orientation Vector Transport

• Orientation is a vector
• Need to take rotation into account

Vector Advection
Scalar Advection of Components
Original Orientation
Velocity Field
94
Orientation Vector Transport

• Novel Formulation
• d Orientation vectoru Velocity

95
Orientation Vector Transport

• Novel Formulation

Scalar advection term
d Orientation vectoru Velocity
96
Orientation Vector Transport

• Novel Formulation

Distortion term
d Orientation vectoru Velocity
97
Orientation Vector Transport

• Novel Formulation

Distortion term
u2
2
d
u1
1
d Orientation vectoru Velocity
98
Orientation Vector Transport

• Novel Formulation

Distortion term
u2
2
d (u2-u1)?t
d
u1
1
d Orientation vectoru Velocity
99
Orientation Vector Transport

• Novel Formulation

Projection term(keeps d a unit vector)
d Orientation vectoru Velocity
100
Results
101
Feature-guided Texturing

• Correlate texture with fluid features
• Curl, divergence, curvature

102
Results
103
Results
104
Results
105
Results
106
Limitations

• Linear Solve may cause blurriness
• Solution Discrete Optimization Han06
• Can get stuck in local minima
• Very slow (especially on surfaces)\
• Exploit parallel hardware
• Nearest neighbor queries can be made in parallel

107
Questions?