Title: A Theory of Locally Low Dimensional Light Transport
1A Theory of Locally Low Dimensional Light
Transport
- Dhruv Mahajan
(Columbia University) - Ira Kemelmacher-Shlizerman
(Weizmann Institute) - Ravi Ramamoorthi
(Columbia University) - Peter Belhumeur
(Columbia University)
2Image Relighting
Ng et al 2003
3Relighting Linear Combination
Nimeroff et al 94
Hallinan 94
Dorsey 95
Lighting Intensities
Images lit by directional light sources
4Relighting Matrix Vector Multiply
B
T
L
Output Image
Input Lighting
Vector
(Unfolded Cubemap)
Transport Matrix
5Light Transport Computational Cost
- Light transport matrix dimensions
- 512 x 512 images
- 6 x 32 x 32 6144 cubemap lighting
- Multiplication / Relighting cost
- Approximately 1010 computations per frame
- Multiplication intractable in real time
- Need to compress the light transport
6Light Transport SVD
T
T
U
S
V
V
L
Lighting
Eigenvalues
Transport Matrix
Vector
Relit Image
Projection Weights
Hallinan 94
Basis Images
7Light Transport SVD
T
V
Eigenvalues
Transport Matrix
- Global Dimensionality
Large
8Global Dimensionality
-
- Computation still intractable
9Locally Low Dimensional Light Transport
p rows
SVD
Transport Matrix
p pixels
Dimensionality of the patch
Locally Low Dimensional Transport
Lighting Resolution
10Previous Work
- Blockwise PCA Nayar et al. 04
- Image divided in to fixed size square patches
- Each patch compressed using PCA
- Clustered PCA Sloan et al. 03
- Object divided in to fixed number of clusters
- Each cluster compressed using PCA
11Previous Work
- Surface light fields
- Nishino et al. 01
- Chen et al. 02
- General reflectance fields
- Matusik et al. 02
- Garg et al. 06
- Compression
- JPEG, MPEG
No Theoretical Analysis
Dimensionality vs Patch Size?
Dimensionality vs Material Properties?
Dimensionality vs Global Effects ?
12Local Light Transport Dimensionality
- Analysis of local light transport dimensionality
Dimensionality
P
1
13Local Light Transport Dimensionality
- Analysis of local light transport dimensionality
Dimensionality
2 x 2
14Local Light Transport Dimensionality
- Analysis of local light transport dimensionality
Dimensionality
15Local Light Transport Dimensionality
- Analysis of local light transport dimensionality
Dimensionality
16Local Light Transport Dimensionality
- Analysis of local light transport dimensionality
Dimensionality
17Local Light Transport Dimensionality
- Analysis of local light transport dimensionality
Dimensionality
18Rendering Cost
- Theoretical analysis of rendering cost
Overhead cost for rendering
Dimensionality
19Overhead Cost
Overhead Cost Projection Weights
Dimensionality cost number of bases
20Rendering Cost
- Theoretical analysis of rendering cost
Overhead cost for rendering
P
21Rendering Cost
- Theoretical analysis of rendering cost
Overhead cost for rendering
22Rendering Cost
- Theoretical analysis of rendering cost
Overhead cost for rendering
23Rendering Cost
- Theoretical analysis of rendering cost
Overhead cost for rendering
24Rendering Cost
- Theoretical analysis of rendering cost
Overhead cost for rendering
25Rendering Cost
- Theoretical analysis of rendering cost
Overhead cost for rendering
26Rendering Cost
- Theoretical analysis of rendering cost
Overhead cost for rendering
Rendering cost Dimensionality Overhead
Optimal
Patch Size
27Contributions
- Analysis of dimensionality of local light
transport - Change of dimensionality with size
- Diffuse and glossy reflections
- Shadows
- Analyzing rendering cost
- Analytical formula for optimal patch size
- Practical Applications
- Fine tuning parameters of existing methods
- Scale images to very high resolutions
- Develop adaptive clustering algorithm
28Local Light Transport Dimensionality
- Analysis of local light transport dimensionality
Dimensionality
29Dimensionality vs. Patch Size
slope 1
Diffuse/Specular BRDF
Large Area
linear relationship
slope - rate of change of dimensionality
Independent of material properties
Dimensionality Patch Area
pixels
dimensionality
30Dimensionality vs. Patch Size
slope lt 1
Diffuse/Specular BRDF
Small Area
sub - linear relationship
pixels
dimensionality
31Mathematical Tools for Analysis
- Convolution formula for glossy reflections and
shadows - Ramamoorthi and Hanrahan 01
- Basri and Jacobs 01
- Ramamoorthi et al 04
- Szegos Eigenvalue Distribution Theorem
- Eigenvalues of the light transport matrix of the
patch - Fourier Scale and Convolution Theorems
- Dimensionality as a function of patch size
32Central Result
Patch Dimensionality
Patch Dimensionality
Bandwidth of BRDF
Patch Area
Bandwidth of BRDF
Patch Area
Constant
Constant
33Central Result
Patch Dimensionality
Bandwidth of BRDF
Patch Area
Constant
low frequency
high frequency
99 Energy
Bandwidth
BRDF/ Material Properties
34Central Result
(
)
(
)
Patch Dimensionality
Bandwidth of BRDF
Patch Area
Bandwidth of BRDF
Patch Area
Constant
Constant
Large Area
Diffuse/Specular BRDF
35Large Area
(
)
(
)
Patch Dimensionality
Bandwidth of BRDF
Patch Area
Diffuse/Specular BRDF
36Large Area
(
)
(
)
Patch Dimensionality
Bandwidth of BRDF
Patch Area
)
(
)
(
(
)
Patch Dimensionality
Bandwidth of BRDF
Patch Area
Diffuse/Specular BRDF
37Large Area
linear relationship
)
(
)
(
(
)
Patch Dimensionality
Bandwidth of BRDF
Patch Area
slope 1
Diffuse/Specular BRDF
38Small Area
sublinear relationship
)
(
)
(
(
)
Patch Dimensionality
Bandwidth of BRDF
Patch Area
slope lt 1
Diffuse/Specular BRDF
39Contributions
- Analysis of dimensionality of local light
transport - Change of dimensionality with size
- Glossy reflections
- Shadows
- Analyzing rendering cost
- Analytical formula for optimal patch size
- Practical Applications
- Fine tuning parameters of existing methods
- Scale images to very high resolutions
- Develop adaptive clustering algorithm
40Visibility Function
Visibility Function 0
Lighting Directions
Blocker
Visibility Function 1
Visibility Function 1
P
41Shadows
Light Transport Visibility Function
slope 1
slope .5
Diffuse and Specular BRDF
Shadows
- Dimensionality changes slowly in presence of
shadows
42Shadows Step Blocker
Light Transport Visibility Function
Step Blocker
Lighting
Direction
Same Visibility Function
Different Visibility Function
Dimensionality changes only along one dimension
Dimensionality vPatch Area
log (Dimensionality) .5 log(Patch Area)
43Shadows Step Blocker
Light Transport Visibility Function
Step Blocker
Same Visibility Function
Different Visibility Function
Dimensionality changes only along one dimension
Dimensionality vPatch Area
log (Dimensionality) .5 log(Patch Area)
44Contributions
- Analysis of dimensionality of local light
transport - Change of dimensionality with size
- Glossy reflections
- Shadows
- Analyzing rendering cost
- Analytical formula for optimal patch size
- Practical Applications
- Fine tuning parameters of existing methods
- Scale images to very high resolutions
- Develop adaptive clustering algorithm
45Local Light Transport Dimensionality
- Analysis of dimensionality of local light
transport - Diffuse and Glossy reflections, dimensionality
area - Shadows, dimensionality varea
Bandwidth of BRDF
Patch Dimensionality
Patch Area
Constant
46Contributions
- Analysis of dimensionality of local light
transport - Change of dimensionality with size
- Glossy reflections
- Shadows
- Analyzing rendering cost
- Analytical formula for optimal patch size
- Practical Applications
- Fine tuning parameters of existing methods
- Scale images to very high resolutions
- Develop adaptive clustering algorithm
47Overhead Cost
Dimensionality
48Overhead Cost
Overhead
Dimensionality
P
49Overhead Cost
Overhead
Dimensionality
50Overhead Cost
Overhead
Dimensionality
51Overhead Cost
Overhead
Dimensionality
52Overhead Cost
Overhead
Dimensionality
53Overhead Cost
Overhead
Dimensionality
54Rendering Cost
Overhead
Rendering Cost
Dimensionality
55Rendering Cost vs. Patch Size
Large Patch size
gt
Rate of increase in dimensionality
Rate of decrease in overhead
Increasing patch size increases total cost
Overhead
Linear regime
Rendering Cost
Dimensionality
56Rendering Cost vs. Patch Size
Large Patch size
gt
Rate of increase in dimensionality
Rate of decrease in overhead
Increasing patch size increases total cost
Linear regime
Rendering Cost
Dimensionality
Overhead
Rendering cost Dimensionality Overhead
57Rendering Cost vs. Patch Size
Small Patch size
lt
Rate of increase in dimensionality
Rate of decrease in overhead
Increasing patch size decreases total cost
Overhead
Sublinear regime
Rendering Cost
Dimensionality
58Rendering Cost vs. Patch Size
Small Patch size
lt
Rate of increase in dimensionality
Rate of decrease in overhead
Increasing patch size decreases total cost
Sublinear regime
Rendering Cost
Dimensionality
Overhead
Rendering cost Dimensionality Overhead
59Rendering Cost vs. Patch Size
Intermediate size
Rate of increase in dimensionality
Rate of decrease in overhead
Total cost minimum
Overhead
Rendering Cost
Minimum
Dimensionality
60Rendering Cost vs. Patch Size
Intermediate size
Rate of increase in dimensionality
Rate of decrease in overhead
Total cost minimum
Rendering Cost
Minimum
Dimensionality
Overhead
Rendering cost Dimensionality Overhead
61Optimal Patch Size
Optimal Patch Size
- Global Dimensionality
62Optimal Patch Size
Optimal Patch Size
- Global Dimensionality
- Function of slope of dimensionality curve
- From our theoretical analysis
- Empirically from the given dataset
Dimensionality Curve
63Optimal Patch Size CPCA Example
- Global Dimensionality
Optimal Patch Size
- Function of slope of
dimensionality curve
Total cost
110 220 330 440
550
average cluster size
Face dataset across lighting
64Glossy Reflections
- Global Dimensionality
Optimal Patch Size
- Function of slope of
dimensionality curve
Independent of material properties
Number of pixels in the patch increases with
glossiness
65Contributions
- Analysis of dimensionality of local light
transport - Change of dimensionality with size
- Glossy reflections
- Shadows
- Analyzing rendering cost
- Analytical formula for optimal patch size
- Practical Applications
- Fine tuning parameters of existing methods
- Scale images to very high resolutions
- Develop adaptive clustering algorithm
66Setting Optimal Patch Size CPCA
67Setting Optimal Patch Size CPCA
cost per pixel
clusters
220
114.78
Estimated
114.78-130
130-600
large
310.7
11
24000 vertices
6 X 32 X 32
45.0 Hz.
Cube Map
68Contributions
- Analysis of dimensionality of local light
transport - Change of dimensionality with size
- Glossy reflections
- Shadows
- Analyzing rendering cost
- Analytical formula for optimal patch size
- Practical Applications
- Fine tuning parameters of existing methods
- Scale images to very high resolutions
- Develop adaptive clustering algorithm
69Scaling of Cost With Resolution
- Global Dimensionality
- Function of slope of dimensionality curve
new resolution
Independent of patch resolution
Optimal patch size same for both resolutions
Subdivide More
70Scaling of Cost With Resolution
Sub-linear increase in cost with resolution
Increase in resolution -
Increase in cost -
new resolution
1.85
71Scaling of Cost With Resolution
- Sublinear increase in cost with resolution
800 x 600
1024 1024
72Scaling of Cost With Resolution
73Summary
- Analysis of dimensionality of local light
transport - Diffuse and Glossy reflections, dimensionality
area - Shadows, dimensionality varea
- Analysis of rendering cost
- Optimal patch size
- Scaling of cost with resolution
- Practical Applications
- Setting optimal parameters in existing methods
- Adaptive clustering algorithms
74Summary
- Analysis of dimensionality of local light
transport - Diffuse and Glossy reflections, dimensionality
area - Shadows, dimensionality varea
- Analysis of rendering cost
- Optimal patch size
- Scaling of cost with resolution
- Practical Applications
- Setting optimal parameters in existing methods
- Adaptive clustering algorithms
Patch Dimensionality
Bandwidth of BRDF
Patch Area
Constant
75Summary
- Analysis of dimensionality of local light
transport - Change of dimensionality with size
- Glossy reflections, dimensionality area
- Shadows, dimensionality varea
- Analyzing rendering cost
- Derive optimal patch size
- Practical Applications
- Fine tuning parameters of existing methods
- Scale to very high resolutions
- Develop adaptive clustering algorithms
76Future Work
- More solid theoretical foundation
- High dimensional appearance compression
- Representation
- ECCV 2006, PAMI 2007
- Analysis of light transport in frequency domain
- TOG, Jan. 2007
- Analysis of light transport in gradient domain
- Siggraph 2007
- Analysis of general local light transport for
patches