Using Photographs to Enhance Videos of a Static Scene

1
Using Photographs to Enhance Videos of a Static
Scene

• Pravin Bhat1, C. Lawrence Zitnick2, Noah
  Snavely1, Aseem Agarwala3, Maneesh Agrawala4,
  Michael Cohen1,2, Brian Curless1, Sing Bing
  Kang2

University of Washington1, Microsoft Research
Redmond2 University of California3, Adobe
Systems4
EGSR 2007
2
Motivation
3
Motivation
Consumer Photographs
4
Motivation
Consumer Photographs
5
Motivation
Consumer Photographs
6
Motivation
Consumer Photographs
7
Motivation
Consumer Photographs
8
(No Transcript)
9
(No Transcript)
10
Video Enhancements
11
Video Enhancements

• One framework. Variety of enhancements
• Transferring photographicqualities
• resolution
• lighting
• dynamic range
• Easy video editing
• object touch-up
• Image filters
• Object removal
• Camera shake removal

12
Video Enhancements

• One framework. Variety of enhancements
• Transferring photographicqualities
• resolution
• lighting
• dynamic range
• Easy video editing
• object touch-up
• Image filters
• Object removal
• Camera shake removal

13
Video Enhancements

• One framework. Variety of enhancements
• Transferring photographicqualities
• resolution
• lighting
• dynamic range
• Easy video editing
• object touch-up
• Image filters
• Object removal
• Camera shake removal

14
Video Enhancements

• One framework. Variety of enhancements
• Transferring photographicqualities
• resolution
• exposure
• dynamic range
• Easy video editing
• object touch-up
• Image filters
• Object removal
• Camera shake removal

15
Video Enhancements

• One framework. Variety of enhancements
• Transferring photographicqualities
• resolution
• exposure
• dynamic range
• Easy video editing
• object touch-up
• Image filters
• Object removal
• Camera shake removal

16
Video Enhancements

• One framework. Variety of enhancements
• Transferring photographicqualities
• resolution
• exposure
• dynamic range
• Easy video authoring
• object touch-up
• Image filters
• Object removal
• Camera shake removal

17
Video Enhancements

• One framework. Variety of enhancements
• Transferring photographicqualities
• resolution
• exposure
• dynamic range
• Easy video authoring
• object touch-up
• Image filters
• Object removal
• Camera shake removal

18
Video Enhancements

• One framework. Variety of enhancements
• Transferring photographicqualities
• resolution
• exposure
• dynamic range
• Easy video authoring
• object touch-up
• image filters
• Object removal
• Camera shake removal

19
Video Enhancements

• One framework. Variety of enhancements
• Transferring photographicqualities
• resolution
• exposure
• dynamic range
• Easy video authoring
• object touch-up
• image filters
• object removal
• Camera shake removal

20
Video Enhancements

• One framework. Variety of enhancements
• Transferring photographicqualities
• resolution
• exposure
• dynamic range
• Easy video authoring
• object touch-up
• image filters
• object removal
• camera shake removal

21
System Overview
22
System Overview
23
System Overview
24
System Overview
25
System Overview
26
Geometry Estimation
27
Multi-View Stereo
28
Multi-View Stereo

• Based on Zitnick et als work at SIGGRAPH 04
• Good for videos and noisy images

29
Multi-View Stereo

• Based on Zitnick et als work at SIGGRAPH 04

synchronized camera arrays
30
Multi-View Stereo

• Based on Zitnick et als work at SIGGRAPH 04

over-segmentation based approach
31
Multi-View Stereo

• Based on Zitnick et als work at SIGGRAPH 04

over-segmentation based approach
32
Multi-View Stereo

• Based on Zitnick et als work at SIGGRAPH 04

BP based optimization
33
Multi-View Stereo

• Based on Zitnick et als work at SIGGRAPH 04

depth consistency across views
34
Multi-View Stereo

• Our Contributions

35
Multi-View Stereo

• Our Contributions
• Augmented depth planes
• View-dependent planes
• Non-fronto parallel planes

36
Multi-View Stereo

• Our Contributions
• Augmented depth planes
• View-dependent planes
• Non-fronto parallel planes
• Automated viewpoint graph construction

37
Multi-View Stereo

• Our Contributions
• Augmented depth planes
• View-dependent planes
• Non-fronto parallel planes
• Automated viewpoint graph construction
• Using SFM point cloud as a soft prior

38
Geometry Estimation
39
System Overview
40
System Overview
41
System Overview
42
Novel View Interpolation
43
Novel View Interpolation
44
Novel View Interpolation

• Objective
• Reconstruct every video frame (orange cones)
  using nearby photos (blue cones)

45
Novel View Interpolation

• Objective
• Reconstruct every video frame (orange cones)
  using nearby photos (blue cones)

46
Novel View Interpolation

• Objective
• Reconstruct every video frame (orange cones)
  using nearby photos (blue cones)

47
Novel View Interpolation

• Objective
• Reconstruct every video frame (orange cones)
  using nearby photos (blue cones)

48
Novel View Interpolation

• Objective
• Reconstruct every video frame (orange cones)
  using nearby photos (blue cones)

49
Novel View Interpolation

• Objective
• Reconstruct every video frame (orange cones)
  using nearby photos (blue cones)

50
Novel View Interpolation
Video Frame
51
Novel View Interpolation
Photo-1
Video Frame
Photo-2
52
Novel View Interpolation
Photo-1 Depths
Video Frame
Photo-2 Depths
53
Novel View Interpolation
Projection-1
Video Frame
Projection-2
54
Novel View Interpolation

• Simple IBR Reconstruction
• Result average( Projection-1 , Projection-2 )

55
Novel View Interpolation

• Simple IBR Reconstruction
• Result average( Projection-1 , Projection-2 )
• Artifacts
• blurring and ghosting
• loss of dynamic lighting

56
Novel View Interpolation

• Simple IBR Reconstruction
• Result average( Projection-1 , Projection-2 )
• Artifacts
• blurring and ghosting
• loss of dynamic lighting
• Our Solution
• Patch based reconstruction
• Gradient domain compositing
• Exploit video data

57
System Overview
58
System Overview
59
Video Reconstruction

• Objective
• Reconstruct every video frame using projected
  images

60
Video Reconstruction

• Our Solution
• Reconstruction problem Labeling problem
• Each projection provides one label
• Graphcuts optimization to find a labeling that
  minimizes a well-designed cost function

61
Video Reconstruction
Projection-1
Video Frame
Projection-2
62
Video Reconstruction
Label-1
Video Frame
Label-2
63
Video Reconstruction
Label-1
Labeling
Label-2
64
Video Reconstruction

• Cost function

65
Video Reconstruction

• Cost function
• Data term encourages

66
Video Reconstruction

• Cost function
• Data term encourages
• depth matching

67
Video Reconstruction

• Cost function
• Data term encourages
• depth matching

Pixeldepth
Labeldepths
68
Video Reconstruction

• Cost function
• Data term encourages
• depth matching

Pixeldepth
Labeldepths
69
Video Reconstruction

• Cost function
• Data term encourages
• depth matching
• color matching

70
Video Reconstruction

• Cost function
• Data term encourages
• depth matching
• color matching

Pixelcolor
Labelcolors
71
Video Reconstruction

• Cost function
• Data term encourages
• depth matching
• color matching

Pixelcolor
Labelcolors
72
Video Reconstruction

• Cost function
• Data term encourages
• depth matching
• color matching
• no holes

73
Video Reconstruction

• Cost function
• Data term encourages
• depth matching
• color matching
• no holes

74
Video Reconstruction

• Cost function
• Data term encourages
• depth matching
• color matching
• no holes

x
?
75
Video Reconstruction

• Cost function
• Data term encourages
• depth matching
• color matching
• no holes
• Smoothness term encourages
• Seamless label transitions
• Seams to run

76
Video Reconstruction

• Cost function
• Data term encourages
• depth matching
• color matching
• no holes
• Smoothness term encourages
• Seamless label transitions
• Seams to run
• through regions of similar color

77
Video Reconstruction

• Cost function
• Data term encourages
• depth matching
• color matching
• no holes
• Smoothness term encourages
• Seamless label transitions
• Seams to run
• through regions of similar color

Label-1
Label-2
78
Video Reconstruction

• Cost function
• Data term encourages
• depth matching
• color matching
• no holes
• Smoothness term encourages
• Seamless label transitions
• Seams to run
• through regions of similar color

good transition
Label-1
Label-2
79
Video Reconstruction

• Cost function
• Data term encourages
• depth matching
• color matching
• no holes
• Smoothness term encourages
• Seamless label transitions
• Seams to run
• through regions of similar color
• along strong edges

80
Video Reconstruction

• Cost function
• Data term encourages
• depth matching
• color matching
• no holes
• Smoothness term encourages
• Seamless label transitions
• Seams to run
• through regions of similar color
• along strong edges

Label-1
Label-2
81
Video Reconstruction

• Cost function
• Data term encourages
• depth matching
• color matching
• no holes
• Smoothness term encourages
• Seamless label transitions
• Seams to run
• through regions of similar color
• along strong edges

good transition
Label-1
Label-2
82
Video Reconstruction
Label-1
Labeling
Label-2
83
Video Reconstruction
Label-1
Label-2
84
Video Reconstruction
Label-1
Reconstructed Video Frame
Label-2
85
Video Reconstruction
Label-1
Video Frame
Label-2
86
Video Reconstruction
Label-1
Reconstructed Video Frame
Label-2
87
(No Transcript)
88
(No Transcript)
89
System Overview
90
System Overview
91
Spacetime Fusion

• Objective To eliminate
• Holes
• Spatial seams
• Temporal incoherence
• Loss of lighting dynamics

92
Spacetime Fusion

• Objective To eliminate
• Holes
• Spatial seams
• Temporal incoherence
• Loss of lighting dynamics
• Solution
• Define artifact free gradient field
• Integrate gradient field

93
Gradient Field Integration
94
Gradient Field Integration
I
95
Gradient Field Integration
Gx
I
Gy
96
Gradient Field Integration
Gx
I
Gx(x, y) I(x, y) I(x 1, y) Gy(x, y) I(x,
y) I(x, y - 1)
Gy
97
Gradient Field Integration
Gx
Gy
98
Gradient Field Integration
?
Gx
I
Gy
99
Gradient Field Integration
?
Gx
I
Solve linear system Av b
Gy
100
Gradient Field Integration
?
Gx
I
Solve linear system Av b Constraints vx, y
vx - 1, y Gx(x, y) vx, y vx, y - 1 Gy(x, y)
Gy
101
Gradient Field Integration
Gx
I
Solve linear system Av b Constraints vx, y
vx - 1, y Gx(x, y) vx, y vx, y - 1 Gy(x, y)
Gy
102
Defining Spatial Gradients
Gx,y
103
Defining Spatial Gradients
Gx,y
104
Defining Spatial Gradients
Labeling
Gx,y
105
Defining Spatial Gradients
Labeling
Label-1
Gx,y
106
Defining Spatial Gradients
Labeling
Label-1
Label-2
Gx,y
107
Defining Spatial Gradients
Labeling
Label-1
Label-2
Gx,y
Videoframe
108
Defining Spatial Gradients
Labeling
Label-1
Label-2
Gx,y
Videoframe
Inside a patch
109
Defining Spatial Gradients
Labeling
Label-1
Label-2
Gx,y
Videoframe
Inside a patch
110
Defining Spatial Gradients
Labeling
Label-1
Label-2
Gx,y
Videoframe
Inside a patch
111
Defining Spatial Gradients
Labeling
Label-1
Label-2
Gx,y
Videoframe
Inside a patch
112
Defining Spatial Gradients
Labeling
Label-1
Label-2
Gx,y
Videoframe
Inside a patch
113
Defining Spatial Gradients
Labeling
Label-1
Label-2
Gx,y
Videoframe
Inside a patch
114
Defining Spatial Gradients
Labeling
Label-1
Label-2
Gx,y
Videoframe
Inside a patch
115
Defining Spatial Gradients
Labeling
Label-1
Label-2
Gx,y
Videoframe
Inside a hole
116
Defining Spatial Gradients
Labeling
Label-1
Label-2
Gx,y
Videoframe
Inside a hole
117
Defining Spatial Gradients
Labeling
Label-1
Label-2
Gx,y
Videoframe
118
Defining Spatial Gradients
Labeling
Label-1
Label-2
Gx,y
Videoframe
Across seams
119
Defining Spatial Gradients
Labeling
Label-1
Label-2
Gx,y
Videoframe
Across seams
120
Defining Spatial Gradients
Labeling
Label-1
Label-2
Gx,y
Videoframe
Across seams
121
Defining Spatial Gradients
Labeling
Label-1
Label-2
Gx,y
Videoframe
Across seams
122
Defining Spatial Gradients
Labeling
Label-1
Label-2
Gx,y
Videoframe
Across seams
123
Defining Spatial Gradients
Labeling
Label-1
Label-2
Gx,y
Videoframe
124
Defining Spatial Gradients
Labeling
Label-1
Label-2
Enhanced Video Frame
Videoframe
Integration result
125
Defining Spatial Gradients
Video reconstruction result
2D integration result
126
Spacetime Fusion

• 3D Gradient field of Enhanced Video

127
Spacetime Fusion

• 3D Gradient field of Enhanced Video
• Spatial gradients Gx and Gy
• as defined earlier

128
Spacetime Fusion

• 3D Gradient field of Enhanced Video
• Spatial gradients Gx and Gy
• as defined earlier
• Temporal gradients Gt

129
Spacetime Fusion

• 3D Gradient field of Enhanced Video
• Spatial gradients Gx and Gy
• as defined earlier
• Temporal gradients Gt
• Gt(x, y, t) V(x, y, t) - V(x, y, t - 1)

130
Spacetime Fusion
Video frame t - 1
Video frame t
131
Spacetime Fusion
Video frame t - 1
Video frame t
Reconstruction frame t
Reconstruction frame t - 1
132
Spacetime Fusion

• Temporal gradients Gt
• Gt(x, y, t) V(x, y, t) - V(x, y, t - 1)

Video frame t - 1
Video frame t
Reconstruction frame t
Reconstruction frame t - 1
133
Spacetime Fusion

• Temporal gradients Gt
• Gt(x, y, t) V(x, y, t) - V(x, y, t - 1)

Gt
Video frame t - 1
Video frame t
Reconstruction frame t
Reconstruction frame t - 1
134
Spacetime Fusion

• Temporal gradients Gt
• Gt(x, y, t) V(x, y, t) - V(x, y, t - 1)

Gt
Video frame t - 1
Video frame t
Gxy
Reconstruction frame t
Reconstruction frame t - 1
135
Spacetime Fusion

• Temporal gradients Gt
• Gt(x, y, t) V(x, y, t) - V(x, y, t - 1)
• Gt incompatible with Gx,y

Gt
Video frame t - 1
Video frame t
Reconstruction frame t
Reconstruction frame t - 1
136
Spacetime Fusion
Video frame t - 1
Video frame t
Reconstruction frame t
Reconstruction frame t - 1
137
Spacetime Fusion
Gt
Video frame t - 1
Video frame t
Reconstruction frame t
Reconstruction frame t - 1
138
Spacetime Fusion

• Temporal gradients Gt
• Gt(x, y, t) V(x, y, t) - V(x u, y v, t 1)

Gt
Video frame t - 1
Video frame t
Reconstruction frame t
Reconstruction frame t - 1
139
Spacetime Fusion

• Temporal gradients Gt
• Gt(x, y, t) V(x, y, t) - V(x u, y v, t 1)
• Gt compatible with Gx,y

Gt
Video frame t - 1
Video frame t
Reconstruction frame t
Reconstruction frame t - 1
140
Spacetime Fusion

• 3D Gradient field of Enhanced Video
• Spatial gradients Gx and Gy
• as defined earlier
• Temporal gradients Gt
• Gt(x, y, t) V(x, y, t) - V(x, y, t - 1)

141
Spacetime Fusion

• 3D Gradient field of Enhanced Video
• Spatial gradients Gx and Gy
• as defined earlier
• Temporal gradients Gt
• Gt(x, y, t) V(x, y, t) - V(x - u, y - v, t - 1)

Where (u,v) is the correspondence vector
142
Spacetime Fusion

• Solving for Enhanced Video

Solve linear system Av b
143
Spacetime Fusion

• Solving for Enhanced Video

Solve linear system Av b Constraints vx, y,
t vx-1, y, t Gx(x, y, t) vx, y, t vx,
y-1, t Gy(x, y, t) vx, y, t vx-u, y-v, t
Gt(x, y, t)
144
Spacetime Fusion
Spacetime fusion result
Video reconstruction result
145
Applications
146
Enhanced Exposure
147
Super-Resolution
148
Video Editing
149
Edit Propagation
150
Edit Propagation

• User Interaction
• User edits one or more images

151
Edit Propagation

• User Interaction
• User edits one or more images
• Solution
• Reconstruct video frames using edited pixels

152
Edit Propagation
153
Edit Propagation
Input Video
154
Edit Propagation
User Edits
155
Edit Propagation
User Edits
156
Edit Propagation
User Edits
157
Edit Propagation
User Edits
158
Edit Propagation
User Edits
159
Edit Propagation
User Edits
160
Edit Propagation
Input Video
161
Edit Propagation
162
Camera Shake Removal
163
Camera Shake Removal

• Solution
• Smooth out input-video camera path
• Re-render video from new camera path

164
Camera Shake Removal
Input Video
165
Camera Shake Removal
Stabilized Video
166
Object Removal
167
Object Removal

• User Interaction
• User draws object mask in one image

168
Object Removal

• User Interaction
• User draws object mask in one image
• Solution
• Transfer mask to all video frames
• Reconstruct masked portions using unmasked
  portions

169
Object Removal
170
Conclusion
171
Conclusion

• General framework
• Enhanced multi-view stereo
• Novel IBR
• Renders dynamic lighting
• Avoids ghosting
• Temporarily consistent
• IBR framework can be applied to dynamic scenes

172
Future Work

• Speed ups
• UI
• Dynamic scenes

173
Thanks

• Reviewers / EGSR organizers
• Advisors - Michael Cohen and Brian Curless
• NVIDIA Fellowship

174
Performance

• 5 Minutes per frame (853x480)
• 2 minutes on SFM
• 2 minutes on MVS
• 1 minute on IBR

175
Photo Count

• Grotto 11 photos
• Bust 7 photos
• Dessert table 12 photos
• Suzzalo 6 photos