Stylized Video Cubes

1
Stylized Video Cubes

• Allison W. Klein
• Peter-Pike J. Sloan
• Adam Finkelstein
• Michael F. Cohen

SCA 2002, the ACM SIGGRAPH Symposium on Computer
Animation.
2
Contents

• Introduction
• Related Works
• Processing video in 3D
• Rendering solids
• Stylized Rendering
• Performance notes
• Conclusion
• Future work

3
Introduction
4
New rendering style

• Video cube
• Abstract representation of the video
• 3D Space-time volume
• Set of rendering solid
• Rendering solid
• Partition of video cube
• Saving parameters for NPR primitives
• Rendered in real-time for output frame

5
Overall processing

• Step1 defining rendering solids
• Sample NPR primitives
• Define rendering style
• Modify parameters interactively
• Step 2 rendering of rendering solids
• Extract NPR parameters
• Composite of output frames

6
Related works
7
NPR related works

• Painterly hand-created look
• Simulation of art medium
• Impressionism, Pen and ink
• Watercolor, Engraving
• Applying to video sequence
• Some artifacts
• To overcome artifacts
• Use optical flow changes
• Litwinowicz97, Hertzmann2000

8
Video volume related works

• P. Rademacher et al. 98
• Two dimensional slice of video cube
• S. Fels and K. Mase 2000
• Video cube
• Cutting plane parallel to time axis
• Epipolar diagrams
• Leverage
• Applying these approaches to NPR of video

9
Video volume related works

• ? Cutting the video cube
• parallel to the X-Y plane

Arbitrary cutting ? the video cube
10
Aesthetics inspiration

• Cubist and futurist arts
• Early 20th century
• Mixing space and time with an image
• Leverage
• Mixing space and time within a video
• Slice the input video in non-standard way
• Creating many different styles

Houses after rain D.M. Ross
11
Processing video
12
Video representation

• In previous work
• A sequence of images
• In video cube
• Three-dimensional volume of pixel data
• Manipulate time in more unified way

13
Rendering solid

• Rendering solid..
• In time interval ,mapping time to
  -D vector
• Vector for expressing properties
• Color, position, variation, etc.
• A set of parameterized rendering solids in a
  video cube

14
Rendering solid

• Evaluation from rendering solid
• A plane passes through the cube
• At time t, provides parameters for NPR primitives
• Composites output frame
• Benefits
• Time-continuous video volume
• Temporal coherence

15
Defining rendering solids

• Video pre-processing
• Sampling information
• Color gradients, importance, etc.
• Input as rendering solids
• Types of rendering solids
• Shard-like rendering solid
• KD-tree-based rendering solid
• Worm-like rendering solid

16
Shard-like rendering solid

• How to generate..
• Two control points
• On key frames
• Center, direction
• Scrolling to other key frame
• Modify center or direction
• Interpolation between key frames
• Form cutting surfaces

17
Shard-like rendering solid

• How to generate..
• Union of cutting surfaces rendering
  solid
• ID to each solid

18
Shard-like rendering solid

• Flow definition using cutting surfaces

For any point P in frame , center of L, C 1.
Initialize flow vector (0,0) 2. Find cutting
lines forming polygonal cross-section of shard
enclosing P 3. For each line L of shard, with C
and P (a) DistancePs perpendicular distance
to L in frame (b) Weight(length of L in
)/(distance?) (c) Displacement velocity (C
in 1) - (C in ) (d) Rotation vector(Ls
direction in ) x (Ls direction in 1 )
19
Shard-like rendering solid

• Flow definition using cutting surfaces

(e) Angular velocityRotation vector x (vector
P to C in ) (f) Current velocityDisplaceme
nt velocity angular velocity (g) Flow
vector Current velocity Weight 4. Normalize
flow vector by sum of all weights
20
KD-tree based RS

• 3D summed area table
• Importance value
• Values for evaluating integralof importance over
  sub-region

21
KD-tree based RS

• KD-tree
• Subdivide the video volume
• Each sub-volume has approximately same amount of
  importance
• Position of lines changes through time
• Swept surface

22
Worm-like rendering solid

• A curved line in time dimension
• Point trajectory through time
• location of trajectoryguides the place
  of textured sprites
• Worm-like rendering solid
• The union of sprites for a single trajectory
• Parameters recording along trajectory

23
Worm-like rendering solid

• Trajectory types
• Constant value position
• Point interpolation
• Flow following trajectory

24
Flow following trajectory

• Adapting streamline placement algorithm Jobard
  97
• Divide volume to regular grid
• Growing curves following flow
• As advanced
• Checking neighborhood cells
• Stop growing when too closeto next curve or face
  edge

25
Stylized rendering
26
Stylized rendering

• Frame rendering
• Evaluate rendering solid at time, t
• Extract parameters
• Position, orientation, scale, etc.
• Parameter modification
• NPR effects
• Depend on parameter setting style
• Variety of visual representation

27
Strokes type rendering

• Worm-like rendering solid
• Colored sprites
• Texture atlas
• Input to stroke-based style
• Color addition from solid
• Frame construction
• A plane passes through the cube
• By compositing sprite for each solidintersecting
  the plane

28
Strokes type rendering

• A girl catching ball
• Standard painterly rendering

• An exploding fireball
• Rounded texture sprite

29
Voronoi cell rendering

• Previous ways problem
• Shower door effect
• Little temporal coherence
• In our work
• Worm-like solid
• Sampling position, velocity of trajectory
• Tangent line
• Drawing cone for each line

30
Voronoi cell rendering

• ? Seed movement effect

? use second texture
31
Shard and mosaic rendering

• Shard-like rendering solids
• Runtime modification
• Zooming shards (scaling)
• Cutting line moving
• Multiple texturing
• Mosaic representation
• Photo-mosaic style
• Spatial, temporal offset
• Shard rotation, zooming

32
Shard rendering results

• Simultaneous capture of two videos

• Photo mosaic style

33
Abstract tiling style

• KD-tree based rendering solid
• Colorful, rectangular composition
• Color of cells average
• Runtime-modification
• Color remapping, traversal depth, etc.

34
Performance notes

• System
• CPU P4 1.7Ghz
• RAM 512MB
• VGA GeForce3
• Performance notes
• NPR style
• 50 and 200 frames/sec
• Worm-like solid filling following flow
• 2 minutes for 640x480 size of 300 frames
• Preprocessing of sampling parameters
• 20 minutes for 300 frames
• Summed-area-table
• A few seconds

35
Conclusion and future work
36
Conclusion

• A new framework for stylized rendering of video
• Space-time volume of image data
• Rendering solid
• Three benefits
• Local choice
• Flexibility
• New visual style

37
Future work

• Better support for multiple simultaneous video
  cubes
• Applying rendering solids to data visualization
• Applying rendering solids to higher-D
  approximations of the plenoptic function