TimeVarying Surface Appearance:

1
Time-Varying Surface Appearance Acquisition,
Modeling and Rendering
Columbia University MERL
SIGGRAPH Conference July, 2006, Boston, USA
2
Time-Varying Surface Appearance
3
An Example Rusting Steel
4
Simulation-Based Approach

• Based on first principles, scene geometry,
  surface accessibility, etc..
• Cons
• Specific models for specific processes.
• Limited realism in rendering.

J. Dorsey et al., 99
Y. Chen et al., 05
J. Dorsey et al., 96
T. Wong et al., 97
5
Data-Driven Approach

• Create realistic appearance efficiently from
  captured data.

Assume appearance static over time.
6
Related Work for Time-Varying Appearance

• Capture and Reconstruction of Time-Varying
  Texture
• Matrix decomposition
• Not much controllability

M. Koudelka, 04

• Transfer of Material Drying History
• Specific only for drying
• Single lighting and view

J. Lu et al., 05

• Appearance Manifolds for Time-Variant Appearance
• Interactive editing tool

J. Wang, et.al., 06
7
Our Work

• Data-Driven Time-Varying Surface Appearance
• Acquisition of a time-varying appearance database
  including various phenomena
• Model for Space-Time Appearance Factorization
• Rendering applications of time-varying appearance

8
Texture 2D
Spatially-Varying BRDF 6D
9
Time
Light
View
Time-and-Space-Varying BRDF (TSV-BRDF) 7D
10
Our Work

• Data-Driven Time-Varying Surface Appearance
• Acquisition of a time-varying appearance database
  including various phenomena
• Model for Space-Time Appearance Factorization
• Rendering applications of time-varying appearance

11
Data Acquisition

• Challenges
• Capture spatially-varying appearance from
  multiple lighting/view.
• Fast acquisition with simultaneously changing
  lighting/view.

12
Samples of Database Light 80, View 07
13
TSV-BRDF Representation

• Nonparametric interpolation
• Tabulate the acquired raw data
• Barycentric interpolation for novel lighting and
  view

14
TSV-BRDF Representation

• Fitting parametric BRDF model

Lambertian
Torrance-Sparrow

• Reflectance of each point at each time frame

Surface roughness
Specular intensity
Diffuse color
15
Nonparametric vs. Parametric
Parametric Model
Barycentric Interpolation
Time 0.00 min
Time 0.00 min
16
TSV-BRDF Changing Light with Time
17
Database of TSV-BRDF
Corrosion
Burning
Drying (Smooth Surface)
Waffle Toasting
Charred Wood 2
Decaying
Drying (Rough Surface)
18
Database of TSV-BRDF

• High dynamic range
• 30 time frames
• Resolution 400x400
• For each sample,
• Raw data about 30 GB
• Fitted Params about 80 MB

Please send an email to staf_at_cs.columbia.edu for
a copy of the database.
19
Our Work

• Data-driven Time-Varying Surface Appearance
• Acquisition of a time-varying appearance database
  including various phenomena
• Model for Space-Time Appearance Factorization
• Rendering applications of time-varying appearance

20
Why Factorization?

• With factorization, we can easily do
• Control
• Synthesis
• Transfer

Spatial Variation
Time-Varying Appearance
Temporal Variation
21
A Closer Look At the Curves
Red Diffuse p(x,y,t)
Sample
Time
0 0.5 1
22
Key Assumption

• All points have one common overall temporal
  variation.
• Different points evolve at different rates and
  offsets.

?

• Two problems
• What is the overall temporal variation?
• How to define and calculate the rates and offsets?

23
STAF Space-Time Appearance Factorization
p(x,y,t)
f(t)
Time t
Effective Time t
24
STAF Space-Time Appearance Factorization
p(x,y,t) A(x,y) f(t) D(x,y)
t R(x,y) t - O(x,y)
p(x,y,t)
f(t)
Time Extrapolation
Time t
Effective Time t
25
p(x,y,t) A(x,y) f(t) D(x,y)
t R(x,y) t - O(x,y)

• STAF model estimation

Initialization A(x,y)R(x,y)1 D(x,y)O(x,y)0
Fix A/R/D/O, compute f(t).
Fix f(t), update A/R/D/O

• Typically 5 iterations are good enough.

26
STAF Model Estimation Result
27
More Results
Charred Wood
Rusting Steel
Decaying Apple
Drying Towel
Samples
Red Diffuse p(x,y,t)
Overall Temporal Curve f(t)
28
Reconstruction
Original
Reconstruction
29
Time Normalization
Original
Time Normalization R(x,y) 1, O(x,y) 0
30
Our Work

• Data-driven Time-Varying Surface Appearance
• Acquisition of a time-varying appearance database
  including various phenomena
• Model for Space-Time Appearance Factorization
• Rendering applications of time-varying appearance

31
Application I Texture Synthesis
Initial
Synthesized Initial
Drying Rock
Final
Synthesized Final
32
Application I Texture Synthesis
Original
Synthesized
33
Application II Time Extrapolation
Decaying Apple Slice
Interpolation
Extrapolation
Extrapolation
Time (min)
60.3
36.4
0.0
-20.2
34
Application II Time Extrapolation
Decaying Apple Slice
Interpolation
Extrapolation
Extrapolation
Time (min)
60.3
36.4
0.0
-20.2
35
Application III Control
36
Application III Control
37
Application IV Transfer Control
New Steel
Original Steel
Global Curvature
Local Curvature
38
Application IV Transfer Control
39
Application IV Transfer Control
40
SIGGRAPH On Fire
41
Summary
Rendering Applications with STAF Model
42
Conclusions

• It is time to bring time variation into
  data-driven appearance.
• Our work is a small step in this new area.
• More complex time-varying appearance
• BTF, Subsurface, Volumetric Scattering,
• Future work
• STAF for more complex time-varying appearance