A SignalProcessing Framework for Forward and Inverse Rendering

1
A Signal-Processing Framework for Forward and
Inverse Rendering
Columbia University Feb 11, 2002
2
Illumination Illusion

• People perceive materials more easily under
  natural illumination than simplified
  illumination.

Images courtesy Ron Dror and Ted Adelson
3
Illumination Illusion

• People perceive materials more easily under
  natural illumination than simplified
  illumination.

Images courtesy Ron Dror and Ted Adelson
4
Material Recognition
Photographs of 4 spheres in 3 different lighting
conditions courtesy Dror and Adelson
5
Estimating BRDF and Lighting
Photographs
Geometric model
6
Estimating BRDF and Lighting
Forward RenderingAlgorithm
Photographs
BRDF
Rendering
Lighting
Geometric model
7
Estimating BRDF and Lighting
Forward RenderingAlgorithm
Photographs
BRDF
Novel lighting
Rendering
Geometric model
8
Inverse Problems Difficulties
Ill-posed (ambiguous)
9
Real-Time Rendering

• Interactive rendering with natural lighting,
  physical BRDFs

10
Motivation

• Understand nature of reflection and illumination
• Applications in computer graphics
• Real-time forward rendering
• Inverse rendering

11
Contributions of Thesis

• Formalize reflection as convolution
• Signal-processing framework
• Practical forward and inverse algorithms

12
Outline

• Motivation
• Reflection as Convolution
• Preliminaries, assumptions
• Reflection equation, Fourier analysis (2D)
• Spherical Harmonic Analysis (3D)
• Signal-Processing Framework
• Applications
• Summary and Implications

13
Assumptions

• Known geometry

14
Assumptions

• Known geometry
• Convex curved surfaces no shadows,
  interreflection

Complex geometry use surface normal
15
Assumptions

• Known geometry
• Convex curved surfaces no shadows,
  interreflection
• Distant illumination

Illumination Grace Cathedral courtesy Paul
Debevec
Photograph of mirror sphere
16
Assumptions

• Known geometry
• Convex curved surfaces no shadows,
  interreflection
• Distant illumination
• Homogeneous isotropic materials

Anisotropic
Isotropic
17
Assumptions

• Known geometry
• Convex curved surfaces no shadows,
  interreflection
• Distant illumination
• Homogeneous isotropic materials
• Later, practical algorithms relax some
  assumptions

18
Reflection
19
Reflection as Convolution (2D)
L
B
20
Reflection as Convolution (2D)
21
Reflection as Convolution (2D)
22
Convolution
u
Signal f(x)
Output h(u)
Filter g(x)
23
Convolution
u1
u
x
Signal f(x)
Output h(u)
Filter g(x)
24
Convolution
u2
u
x
Signal f(x)
Output h(u)
Filter g(x)
25
Convolution
u3
u
x
Signal f(x)
Output h(u)
Filter g(x)
26
Convolution
u
x
Signal f(x)
Output h(u)
Filter g(x)
27
Reflection as Convolution (2D)
Fourier analysis
R. Ramamoorthi and P. Hanrahan Analysis of
Planar Light Fields from Homogeneous Convex
Curved Surfaces under Distant Illumination SPIE
Photonics West 2001 Human Vision and Electronic
Imaging VI pp 195-208
28
Related Work

• Qualitative observation of reflection as
  convolution Miller Hoffman 84, Greene
  86, Cabral et al. 87,99
• Reflection as frequency-space operator DZmura
  91
• Lambertian reflection is convolution Basri
  Jacobs 01
• Our Contributions
• Explicitly derive frequency-space convolution
  formula
• Formal quantitative analysis in general 3D case

29
Spherical Harmonics
0
1
2 . . .
-1
-2
0
1
2
30
Spherical Harmonic Analysis
2D
3D
31
Outline

• Motivation
• Reflection as Convolution
• Signal-Processing Framework
• Insights, examples
• Well-posedness of inverse problems
• Applications
• Summary and Implications

32
Insights Signal Processing

• Signal processing framework for reflection
• Light is the signal
• BRDF is the filter
• Reflection on a curved surface is convolution

33
Insights Signal Processing

• Signal processing framework for reflection
• Light is the signal
• BRDF is the filter
• Reflection on a curved surface is convolution

Filter is Delta function Output Signal
34
Insights Signal Processing

• Signal processing framework for reflection
• Light is the signal
• BRDF is the filter
• Reflection on a curved surface is convolution

Signal is Delta function Output Filter
35
Phong, Microfacet Models
Mirror
Illumination estimation ill-posed for rough
surfaces
Analytic formulae in R. Ramamoorthi and P.
Hanrahan A Signal-Processing Framework for
Inverse Rendering SIGGRAPH 2001 pp 117-128
36
Lambertian
Incident radiance (mirror sphere)
Irradiance (Lambertian)
37
Inverse Lighting
Given B,? find L

• Well-posed unless denominator vanishes
• BRDF should contain high frequencies Sharp
  highlights
• Diffuse reflectors low pass filters Inverse
  lighting ill-posed

38
Inverse BRDF
Given B,L find ?

• Well-posed unless Llm vanishes
• Lighting should have sharp features (point
  sources, edges)
• BRDF estimation ill-conditioned for soft lighting

Area source Same BRDF
Directional Source
39
Factoring the Light Field

• Light Field can be factored
• Up to global scale factor
• Assumes reciprocity of BRDF
• Can be ill-conditioned
• Analytic formula derived

Given B find L and ?
More knowns (4D) than unknowns (2D/3D)
40
Outline

• Motivation
• Reflection as Convolution
• Signal-Processing Framework
• Applications
• Forward rendering (convolution)
• Inverse rendering (deconvolution)
• Summary and Implications

41
Computing Irradiance

• Classically, hemispherical integral for each
  pixel
• Lambertian surface is like low pass filter
• Frequency-space analysis

Incident Radiance
Irradiance
42
9 Parameter Approximation
Order 0 1 term (constant)
Exact image
RMS error 25
43
9 Parameter Approximation
Order 1 4 terms (linear)
Exact image
RMS Error 8
44
9 Parameter Approximation
Order 2 9 terms (quadratic)
Exact image
RMS Error 1
For any illumination, average error Jacobs 01
45
Comparison
Irradiance map Texture 256x256 Hemispherical Inte
gration 2Hrs
Irradiance map Texture 256x256 Spherical
Harmonic Coefficients 1sec
Incident illumination 300x300
46
Video
R. Ramamoorthi and P. Hanrahan An Efficient
Representation for Irradiance Environment Maps
SIGGRAPH 2001 pp 497-500 R. Ramamoorthi and P.
Hanrahan Frequency Space Environment Map
Rendering submitted
47
Video
48
Inverse Rendering
Lighting
Unknown
Known
Known
BRDF
Unknown
Textures are a third axis
49
Contributions

• Complex illumination
• Factorization of BRDF, lighting (find both)
• New representations and algorithms
• Formal study of inverse problems (well-posed?)

50
Complications

• Incomplete sparse data (few photographs)
• Concavities Self Shadowing
• Spatially varying BRDFs

51
Complications

• Challenge Incomplete sparse data (few
  photographs) Difficult to compute
  frequency spectra
• Solution
• Use parametric BRDF model
• Dual angular and frequency space representation

52
Algorithm Validation
Photograph
True values
53
Algorithm Validation
Photograph
Renderings
Image RMS error 5
Known lighting
Unknown lighting
True values
54
Inverse BRDF Spheres
Photographs
Renderings (Recovered BRDF)
55
Complications

• Challenge Complex geometry with concavities
  Self shadowing
• Solution
• Use associativity of convolution
• Blur lighting, treat specular BRDF term as mirror
• Single ray for shadowing, easy in ray tracer

56
Complex Geometry

• 3 photographs of a sculpture
• Complex unknown illumination
• Geometry known
• Estimate microfacet BRDF and distant lighting

57
Comparison
58
New View, Lighting
Photograph
Rendering
59
Complications

• Challenge Spatially varying BRDFs
• Solution
• Use textures to modulate BRDF parameters

60
Textured Objects
Rendering
Photograph
61
Summary

• Reflection as convolution
• Frequency-space analysis gives many insights
• Practical forward and inverse algorithms
• Signal-Processing A useful paradigm for forward
  and inverse rendering in graphics and vision

62
Implications and Future Work

• Duality between forward and inverse problems
  Ill-posed inverse problem ? fast forward
  algorithm
• Example Inverse lighting from Lambertian
  surface ill-posed ? computing irradiance is fast

63
Implications and Future Work

• Differential framework for reflection
• Analysis on object surface
• Complex illumination and BRDF

64
Implications and Future Work

• Analyzing intrinsic structure, complexity of
  light field Sampling theory based on
  signal-processing
• How many images in image-based rendering?
• How many principal components in PCA?

65
Lighting Invariant Recognition

• Theory Space of images infinite-dimensional
  
  for Lambertian Belhumeur and Kriegman 98
  
  
• Empirical 5D subspace enough for diffuse
  objects
  Hallinan 94, Epstein et al. 95, BK98,

Images from Yale face database
66
Implications and Future Work

• Complex illumination in computer vision
• Generally assume simple lighting (point source)
  without considering visibility (attached shadows)
• Signal processing can be used to reduce effects
  of complex illumination (with shadows) to
  low-dimensional subspace
• Many applications stereo, photometric stereo,
  shape from shading, lighting invariant
  recognition etc.

67
Acknowledgements

• Pat Hanrahan
• Marc Levoy
• Szymon Rusinkiewicz
• Steve Marschner
• Stanford graphics group
• Hodgson-Reed Stanford Graduate Fellowship
• NSF ITR grant 0085864 Interacting with the
  Visual World

68
Papers

• R. Ramamoorthi and P. Hanrahan A
  Signal-Processing Framework for Inverse
  Rendering SIGGRAPH 2001 pp 117-128
• R. Ramamoorthi and P. Hanrahan An Efficient
  Representation for Irradiance Environment Maps
  SIGGRAPH 2001 pp 497-500
• R. Ramamoorthi and P. Hanrahan Frequency Space
  Environment Map Rendering submitted
• R. Ramamoorthi and P. Hanrahan On the
  Relationship between Radiance and Irradiance
  Determining the Illumination from images of a
  Convex Lambertian Object Journal of the Optical
  Society of America A 18(10) 2001 pp 2448-2459
• R. Ramamoorthi and P. Hanrahan Analysis of
  Planar Light Fields from Homogeneous Convex
  Curved Surfaces under Distant Illumination SPIE
  Photonics West 2001 Human Vision and Electronic
  Imaging VI pp 195-208
• R. Ramamoorthi Analytic PCA Construction for
  Theoretical Analysis of Lighting Variability,
  Including Attached Shadows, in a Single Image of
  a Convex Lambertian Object CVPR 2001 workshop on
  Identifying Objects across Lighting Variations pp
  48-55

ravir_at_graphics.stanford.edu http//graphics.stanf
ord.edu/ravir
69
The End
70
Photorealistic Rendering
71
Measuring Materials, Light
Measure BRDF (reflectance) Point light source
72
Interactive Forward Rendering

• Classically, rendering with natural
  illumination is very expensive compared to using
  simplified illumination

Directional Source
Natural Illumination
73
Lighting Invariant Recognition

• Theory Infinite number of light directions
  
  Space of images
  infinite-dimensional
• Empirical 5D subspace enough for diffuse objects

Images from Debevec et al. 00
74
Lighting Invariant Recognition

• Theory Space of images infinite-dimensional
  
  for Lambertian Belhumeur and Kriegman 98
  
  
• Empirical 5D subspace enough for diffuse
  objects
  Hallinan 94, Epstein et al. 95, BK98,

75
Open Questions

• Relationship between spherical harmonics, PCA
• 9D approximation 5D empirical subspace
• Key insight Consider approximations over visible
  normals (upper hemisphere), not entire sphere

Ramamoorthi CVPR IOAVL 01
76
Light Field in 3D

• In flatland, 2D function
• In three dimensions, 4D function

Plenoptic Light Field
Surface Light Field
77
Dual Representation

• Diffuse BRDF Filter width small in frequency
  domain
• Specular Filter width small in spatial (angular)
  domain
• Practical Representation Dual angular,
  frequency-space

78
Related Work

• Precomputed (prefiltered) Irradiance maps
  Miller and Hoffman 84, Greene 86, Cabral
  et al 87
• Empirical observation Irradiance varies slowly
  with surface normal. Use low resolution
  irradiance maps
• Contributions
• Analytic Irradiance formula
• Fast computation
• Compact 9 parameter representation
• Procedural rendering with programmable shading
  hardware
• Our approach can be extended to general BRDFs

79
Comparison
Rendering (known L)
Photograph
Rendering (unknown L)