Computational Fundamentals of Reflection

1
Computational Fundamentals of Reflection
COMS 6998-3, Lecture 7
2
Motivation

• Understand intrinsic computational structure of
  reflection and illumination
• Necessary for many applications in computer
  graphics (cannot solve by brute force!!)
• Real-time forward rendering
• IBR sampling rates, dimensionality explosion
• Inverse rendering and inverse problems in general
• Computer vision complex lighting, materials

3
Real-Time Rendering Demo

• Motivation Interactive rendering with
  complex natural illumination and realistic,
  measured BRDFs

4
Questions

• Images are view-dependent (4D quantity)
• Can we find low-dimensional structure to capture
  view-dependence?

5
Space of images as lighting varies

• Illuminate subject from many incident directions

6
Example Images
Images from Debevec et al. 00
7
Principal Component Analysis

• Try to approximate with low dimensional subspace
• Linear combination of few principal components

.5
.5

.7
.3

Principal component images
8
Lighting Variability

• Theory
• Infinite number of light directions, one
  coefficient/direction
• Space of images infinite dimensional Belhumeur
  98
• Empirical Hallinan 94, Epstein 95
• Diffuse objects 5D subspace suffices
• No satisfactory theoretical explanation of
  observations

9
Complex Light Transport

• Shadows high frequency
• Analysis possible?
• Low-dim. structure?
• Real-time complex lights?

Agrawala, Ramamoorthi, Heirich, Moll SIGGRAPH 00
10
Challenges

• Illumination complexity
• Material (BRDF)/view complexity
• Transport complexity (shadows, interreflection)
• Fundamental questions
• Theoretical analysis of intrinsic complexity
• Sampling rates and resolutions
• Efficient practical algorithms

11
Outline

• Lighting variability in appearance PAMI Oct,
  2002
• View variability real-time rendering SIGGRAPH
  02
• Visibility/shadows In Progress

12
Lighting variability analysis

• Frequency space analytic PCA construction
• Mathematical derivation of principal components
• Explain empirical results quantitatively
• Dimensionality of approximating subspace
• Forms of principal components
• Relative importance of principal components

13
Assumptions

• Single view of single object
• Lambertian
• Distant illumination
• Discount texture
• Discount concavities interreflection, cast
  shadows
• Consider attached shadows (backfacing normals)

14
Definitions
Lambertian half-cosine
15
Previous Theoretical Work

• Discount attached shadows Shashua 97,
• Resulting 3D subspace does not fully explain
  data
• Analytic PCA (without shadows) Zhao Yang 99

Lambertian half-cosine
16
Spherical Harmonics
0
1
2 . . .
-1
-2
0
1
2
17
Spherical Harmonic Expansion

• Expand lighting (L), irradiance (E) in basis
  functions

.67
.36

18
Lambertian BRDF Expansion
Lambertian coefficients
19
Analytic Irradiance Formula

• Lambertian surface acts like low-pass filter

0
0
1
2
Basri Jacobs 01 Ramamoorthi Hanrahan 01
20
9 Parameter Approximation
Order 2 9 terms
Exact image
0
RMS Error 1
1
For any illumination, average error lt 2 Basri
Jacobs 01
2
-1
-2
0
1
2
21
Open Questions

• Relationship between spherical harmonics, PCA
• 9D approximation gt 5D empirical subspace
• Key insight Consider approximations over visible
  normals (upper hemisphere), not entire sphere

22
Intuition Backwards Half-Cosine
23
Intuition dimensionality reduction

• Start with 9D space, remove dimensions
• Mean (constant term) subtracted
• Backwards half-cosine
• x, xz very similar
• y, yz very similar
• Left with 5D subspace

24
Results Image of a Sphere

• Principal components (eigenvectors) mix (linear
  combinations of) spherical harmonics
• Results agree with experiment Epstein 95
• We predict 3 eigenvectors 91 variance, 5
  give 96
• Empirical 3 eigenvectors 94 variance, 5
  give 98

2
2
43
24
24
VAF (eigenvalue)
25
Results Human Face

• Numerically compute orthogonality matrix
• Specific distribution of surface normals
  important
• Symmetries in sphere broken (faces are elongated)
• Principal components somewhat different from
  sphere

4
2
VAF
42
33
16
26
Results Human Face

• Prediction Principal components have specific
  forms
• Empirical Hallinan 94
• Frontal lighting, side, above/below, extreme
  side, corner

4
2
VAF
42
33
16
Extreme side
Corner
Frontal
Side
Above/Below
27
Results Human Face

• Prediction Space is close to 5D
• 3 principal components 91 variance, 5
  components 97
• Empirical Epstein 95
• 3 principal components 90 variance, 5
  components 94

4
2
VAF
42
33
16
Extreme side
Corner
Frontal
Side
Above/Below
28
Results Human Face

• Prediction groups of principal components
• Group 1 first two (frontal and side)
• Group 2 next three with above/below always 3rd
• Empirical Hallinan 94
• Two groups first two (frontal,side) and next
  three
• Within group, VAF close, may exchange places

4
2
VAF
42
33
16
Extreme side
Corner
Frontal
Side
Above/Below
29
Summary Lighting Analysis

• Analytic PCA construction with attached shadows
• Spherical harmonic analysis Orthogonality
  matrix
• Mathematically derive principal components
• Qualitative, quantitative agreement with
  experiment
• Extend 9D Lambertian model to single view case

30
Implications Lighting Analysis

• Attached shadows nearly free 5D subspace enough
• Mathematical derivation of principal components
• Basis functions for subspace methods for
  recognition,
• Graphics applications Image-Based, inverse
  rendering
• Complex illumination in computer vision

31
Outline

• Lighting variability in appearance PAMI Oct,
  2002
• View variability real-time rendering SIGGRAPH
  02
• Visibility/shadows In Progress

32
Reflection Equation
2D Environment Map
33
Reflection Equation
2D Environment Map
BRDF
34
Reflection Equation
4D Orientation Light Field
2D Environment Map
BRDF
Previous Work Blinn Newell 76, Miller
Hoffman 84, Greene 86, Kautz McCool 99, Cabral
et al. 99,
35
Goals

• Efficiently precompute and represent OLF
• Real-time rendering with OLF

36
Questions

• Parameterization and structure of OLF
• Structure leads to representation
• Computation and rendering of OLF

37
OLF Parameterization
38
OLF Parameterization
N
V
39
OLF Parameterization

• Captures structure of BRDF (and hence OLF) better
• Reflective BRDFs become low-dimensional

N
N
R
Reparameterize by reflection vector
V
V
40
OLF Structure
2D view array of reflection maps
2D image array of view maps
41
OLF Structure Phong
Phong Reflection Map (blurred environment map)
Environment Map
2D view array of reflection maps
2D image array of view maps
42
OLF Structure Phong
Viewy
Viewx
Same reflection map for all views
43
OLF Structure Phong
Viewy
Viewx
Same reflection map for all views
View maps constant for each R
44
OLF Structure Phong
Viewy
Reflectiony
Viewx
Reflectionx
Same reflection map for all views
View maps constant for each R
45
OLF Structure Lafortune
Viewy
Viewx
46
OLF Structure Lafortune
Viewy
Reflectiony
Reflectionx
Viewx
View maps vary slowly
47
A Simple Factorization
Viewy
Reflectiony
Viewx
Reflectionx
48
Spherical Harmonic Reflection Map

• View-dependent reflection (cube)map
• Encode view maps with low-order
  spherical harmonics

49
Prefiltering

• Directly compute SHRM from Lighting, BRDF
• Convolution easier to compute in frequency domain

Input Lighting and BRDF
Spherical Harmonic coeffs.
Convolution
SHRM
50
Prefiltering

• 3 to 4 orders of magnitude faster (lt 1 s compared
  to minutes or hours)
• Detailed analysis, algorithms, experiments in
  paper

Input Lighting and BRDF
Spherical Harmonic coeffs.
Convolution
SHRM
51
Number of terms CURET

• Analysis for all 61 samples full bar chart in
  paper
• For essentially all materials, 9-16 terms in SHRM
  suffice

52
Demo

53
Summary view variability

• Theoretical, empirical analysis of sampling rates
  and resolutions
• Frequency space analysis directly on lighting,
  BRDF
• Low order expansion suffices for essentially all
  BRDFs
• Spherical Harmonic Reflection Maps
• Hybrid angular-frequency space
• Compact, efficient, accurate
• Easy to analyze errors, determine number of terms
• Fast computation using convolution

54
Implications

• Frequency space methods for rendering
• Global illumination
• Fast computation of surface light fields
• Compression for optimal factored representations
• PCA on SHRMs
• Theoretical analysis of sampling rates,
  resolutions
• General framework for sampling in image-based
  rendering

55
Outline

• Lighting variability in appearance PAMI Oct,
  2002
• View variability real-time rendering SIGGRAPH
  02
• Visibility/shadows In Progress

56
Visibility complexity (high freq)
57
But Sparse (lt 4)
58
Questions on Visibility

• Theory
• Locally low-dimensional subspaces?
• Intrinsic complexity of binary function?
• Practical
• Real-time rendering with complex soft shadows,
  changing illumination for lighting design,
  simulation
• Efficient encoding/decoding (wavelets, PCA,
  dictionaries, hierarchical?)
• In progress.

59
Overall Summary

• Many applications in graphics cannot be solved by
  brute force
• Real-time rendering
• IBR sampling rates, dimensionality explosion
• Inverse rendering, inverse problems
• Computer vision complex lighting, materials
• Need fundamental understanding of nature of
  reflection/lighting
• Illumination complexity
• Material (view) complexity
• Transport complexity

60
Overall Summary

• Theoretical analysis tools
• Signal processing, sampling theory
• Low-dimensional subspaces
• Information theory, information-based complexity?
• Practical algorithms
• Real-time rendering with complex lights,view,
  transport?
• Lighting, Material design?
• Exploit theoretical analysis (sampling rates,
  forward/inverse duality, angular/frequency/sparsit
  y duality, subspace results, differential
  analysis, perception)