A SignalProcessing Framework for Forward and Inverse Rendering

1
A Signal-Processing Framework for Forward and
Inverse Rendering
Ph.D. Orals June 3, 2002
2
Illumination Illusion

• People perceive materials more easily under
  natural illumination than simplified
  illumination.

Images courtesy Ron Dror and Ted Adelson
3
Illumination Illusion

• People perceive materials more easily under
  natural illumination than simplified
  illumination.

Images courtesy Ron Dror and Ted Adelson
4
Material Recognition
Photographs of 4 spheres in 3 different lighting
conditions courtesy Dror and Adelson
5
Estimating BRDF and Lighting
Photographs
Geometric model
6
Estimating BRDF and Lighting
Forward RenderingAlgorithm
Photographs
BRDF
Rendering
Lighting
Geometric model
7
Estimating BRDF and Lighting
Forward RenderingAlgorithm
Photographs
BRDF
Novel lighting
Rendering
Geometric model
8
Inverse Problems Difficulties
Ill-posed (ambiguous)
9
Real-Time Rendering

• Interactive rendering with natural lighting,
  physical BRDFs

10
Contributions

• Formalize reflection as convolution
• Signal-processing framework
• Practical forward and inverse algorithms

11
Outline

• Motivation
• Reflection as Convolution
• Preliminaries, assumptions
• Reflection equation, Fourier analysis (2D)
• Spherical Harmonic Analysis (3D)
• Signal-Processing Framework
• Applications
• Summary and Implications

12
Assumptions

• Known geometry

13
Assumptions

• Known geometry
• Convex curved surfaces no shadows,
  interreflection

Complex geometry use surface normal
14
Assumptions

• Known geometry
• Convex curved surfaces no shadows,
  interreflection
• Distant illumination

Illumination Grace Cathedral courtesy Paul
Debevec
Photograph of mirror sphere
15
Assumptions

• Known geometry
• Convex curved surfaces no shadows,
  interreflection
• Distant illumination
• Homogeneous isotropic materials

Anisotropic
Isotropic
16
Assumptions

• Known geometry
• Convex curved surfaces no shadows,
  interreflection
• Distant illumination
• Homogeneous isotropic materials
• Later, practical algorithms relax some
  assumptions

17
Reflection
18
Reflection as Convolution (2D)
L
B
19
Reflection as Convolution (2D)
20
Reflection as Convolution (2D)
21
Reflection as Convolution (2D)
Fourier analysis
R. Ramamoorthi and P. Hanrahan Analysis of
Planar Light Fields from Homogeneous Convex
Curved Surfaces under Distant Illumination SPIE
Photonics West 2001 Human Vision and Electronic
Imaging VI pp 195-208
22
Related Work

• Qualitative observation of reflection as
  convolution Miller Hoffman 84, Greene
  86, Cabral et al. 87,99
• Reflection as frequency-space operator DZmura
  91
• Lambertian reflection is convolution Basri
  Jacobs 01
• Our Contributions
• Explicitly derive frequency-space convolution
  formula
• Formal quantitative analysis in general 3D case

23
Spherical Harmonics
0
1
2 . . .
-1
-2
0
1
2
24
Spherical Harmonic Analysis
2D
3D
25
Outline

• Motivation
• Reflection as Convolution
• Signal-Processing Framework
• Insights, examples
• Well-posedness of inverse problems
• Applications
• Summary and Implications

26
Insights Signal Processing

• Signal processing framework for reflection
• Light is the signal
• BRDF is the filter
• Reflection on a curved surface is convolution

27
Insights Signal Processing

• Signal processing framework for reflection
• Light is the signal
• BRDF is the filter
• Reflection on a curved surface is convolution

Filter is Delta function Output Signal
28
Insights Signal Processing

• Signal processing framework for reflection
• Light is the signal
• BRDF is the filter
• Reflection on a curved surface is convolution

Signal is Delta function Output Filter
29
Phong, Microfacet Models
Mirror
Illumination estimation ill-posed for rough
surfaces
Analytic formulae in R. Ramamoorthi and P.
Hanrahan A Signal-Processing Framework for
Inverse Rendering SIGGRAPH 2001 pp 117-128
30
Lambertian
Incident radiance (mirror sphere)
Irradiance (Lambertian)
31
Inverse Lighting
Given B,? find L

• Well-posed unless denominator vanishes
• BRDF should contain high frequencies Sharp
  highlights
• Diffuse reflectors low pass filters Inverse
  lighting ill-posed

32
Inverse BRDF
Given B,L find ?

• Well-posed unless Llm vanishes
• Lighting should have sharp features (point
  sources, edges)
• BRDF estimation ill-conditioned for soft lighting

Area source Same BRDF
Directional Source
33
Factoring the Light Field

• Light Field can be factored
• Up to global scale factor
• Assumes reciprocity of BRDF
• Can be ill-conditioned
• Analytic formula derived

Given B find L and ?
More knowns (4D) than unknowns (2D/3D)
34
Outline

• Motivation
• Reflection as Convolution
• Signal-Processing Framework
• Applications
• Forward rendering (convolution)
• Inverse rendering (deconvolution)
• Summary and Implications

35
Computing Irradiance

• Classically, hemispherical integral for each
  pixel
• Lambertian surface is like low pass filter
• Frequency-space analysis

Incident Radiance
Irradiance
36
9 Parameter Approximation
Order 0 1 term (constant)
Exact image
RMS error 25
37
9 Parameter Approximation
Order 1 4 terms (linear)
Exact image
RMS Error 8
38
9 Parameter Approximation
Order 2 9 terms (quadratic)
Exact image
RMS Error 1
For any illumination, average error lt 2 Basri
Jacobs 01
39
Comparison
Irradiance map Texture 256x256 Hemispherical Inte
gration 2Hrs
Irradiance map Texture 256x256 Spherical
Harmonic Coefficients 1sec
Incident illumination 300x300
40
Video
R. Ramamoorthi and P. Hanrahan An Efficient
Representation for Irradiance Environment Maps
SIGGRAPH 2001 pp 497-500 R. Ramamoorthi and P.
Hanrahan Frequency Space Environment Map
Rendering SIGGRAPH 2002
41
Video
42
Inverse Rendering Previous Work
Lighting
Unknown
Known
Known
BRDF
Unknown
Textures are a third axis
43
Contributions

• Complex illumination
• Factorization of BRDF, lighting (find both)
• New representations and algorithms
• Formal study of inverse problems (well-posed?)

44
Complications

• Incomplete sparse data (few photographs)
• Concavities Self Shadowing
• Spatially varying BRDFs

45
Complications

• Challenge Incomplete sparse data (few
  photographs) Difficult to compute
  frequency spectra
• Solution
• Use parametric BRDF model
• Dual angular and frequency space representation

46
Algorithm Validation
Photograph
True values
47
Algorithm Validation
Photograph
Renderings
Image RMS error 5
Known lighting
Unknown lighting
True values
48
Inverse BRDF Spheres
Photographs
Renderings (Recovered BRDF)
49
Complications

• Challenge Complex geometry with concavities
  Self shadowing
• Solution
• Use associativity of convolution
• Blur lighting, treat specular BRDF term as mirror
• Single ray for shadowing, easy in ray tracer

50
Complex Geometry

• 3 photographs of a sculpture
• Complex unknown illumination
• Geometry known
• Estimate microfacet BRDF and distant lighting

51
Comparison
52
New View, Lighting
Photograph
Rendering
53
Complications

• Challenge Spatially varying BRDFs
• Solution
• Use textures to modulate BRDF parameters

54
Textured Objects
Rendering
Photograph
55
Summary

• Reflection as convolution
• Frequency-space analysis gives many insights
• Practical forward and inverse algorithms
• Signal-Processing A useful paradigm for forward
  and inverse rendering in graphics and vision

56
Implications and Future Work

• Duality between forward and inverse problems
• Analyzing intrinsic structure of light field
• How many images in image-based rendering?
• How many principal components in PCA?
• Differential framework for reflection
• Complex illumination in computer vision

57
Papers

• R. Ramamoorthi and P. Hanrahan A
  Signal-Processing Framework for Inverse
  Rendering SIGGRAPH 2001 pp 117-128
• R. Ramamoorthi and P. Hanrahan An Efficient
  Representation for Irradiance Environment Maps
  SIGGRAPH 2001 pp 497-500
• R. Ramamoorthi and P. Hanrahan Frequency Space
  Environment Map Rendering SIGGRAPH 2002
• R. Ramamoorthi and P. Hanrahan On the
  Relationship between Radiance and Irradiance
  Determining the Illumination from images of a
  Convex Lambertian Object Journal of the Optical
  Society of America A 18(10) 2001 pp 2448-2459
• R. Ramamoorthi and P. Hanrahan Analysis of
  Planar Light Fields from Homogeneous Convex
  Curved Surfaces under Distant Illumination SPIE
  Photonics West 2001 Human Vision and Electronic
  Imaging VI 195-208

ravir_at_graphics.stanford.edu http//graphics.stanf
ord.edu/ravir
58
Acknowledgements

• Pat Hanrahan
• Committee Marc, Jitendra, Ron, Bernd
• Szymon Rusinkiewicz and Steve Marschner
• Bill Mark and Kekoa Proudfoot
• GLAB
• gerth, ada, heather, seander, maneesh,
  henrik, jedavis, olaf, humper, dk, renng,
  tpurcell, psen, vaibhav, zsh, munzner, lucasp,
  liyiwei,
• Hodgson-Reed Stanford Graduate Fellowship
• NSF ITR grant 0085864 Interacting with the
  Visual World

59
The End
60
Motivation

• Understand nature of reflection and illumination
• Applications in computer graphics
• Real-time forward rendering
• Inverse rendering

61
Photorealistic Rendering
62
Measuring Materials, Light
Measure BRDF (reflectance) Point light source
63
Interactive Forward Rendering

• Classically, rendering with natural
  illumination is very expensive compared to using
  simplified illumination

Directional Source
Natural Illumination
64
Lighting Invariant Recognition

• Theory Infinite number of light directions
  
  Space of images
  infinite-dimensional
• Empirical 5D subspace enough for diffuse objects

Images from Debevec et al. 00
65
Lighting Invariant Recognition

• Theory Space of images infinite-dimensional
  
  for Lambertian Belhumeur and Kriegman 98
  
  
• Empirical 5D subspace enough for diffuse
  objects
  Hallinan 94, Epstein et al. 95

66
Open Questions

• Relationship between spherical harmonics, PCA
• 9D approximation gt 5D empirical subspace
• Key insight Consider approximations over visible
  normals (upper hemisphere), not entire sphere

Ramamoorthi CVPR IOAVL 01
67
Light Field in 3D

• In flatland, 2D function
• In three dimensions, 4D function

Plenoptic Light Field
Surface Light Field
68
Dual Representation

• Diffuse BRDF Filter width small in frequency
  domain
• Specular Filter width small in spatial (angular)
  domain
• Practical Representation Dual angular,
  frequency-space

69
Related Work

• Precomputed (prefiltered) Irradiance maps
  Miller and Hoffman 84, Greene 86, Cabral
  et al 87
• Empirical observation Irradiance varies slowly
  with surface normal. Use low resolution
  irradiance maps
• Contributions
• Analytic Irradiance formula
• Fast computation
• Compact 9 parameter representation
• Procedural rendering with programmable shading
  hardware
• Our approach can be extended to general BRDFs

70
Comparison
Rendering (known L)
Photograph
Rendering (unknown L)