Spherical Convolution in Computer Graphics and Vision

1
Spherical Convolution in Computer Graphics and
Vision
Ravi Ramamoorthi
Columbia Vision and Graphics Center Columbia
University
SIAM Imaging Science Conference May 17, 2006
2
Outline

• Motivation and Practical Problems
• Spherical Convolution and Applications
• A Theory of Spherical Harmonic Identities
• Signal Processing for Visual Appearance

3
Real-Time Rendering

• Motivation Interactive rendering with natural
  illumination and realistic, measured materials

4
Inverse Rendering
Photographs
Geometric model
5
Inverse Rendering
Forward RenderingAlgorithm
Photographs
BRDF
Novel lighting
Rendering
Geometric model
6
Direct Object Relighting

Unknown Lighting
Unknown BRDF
?
7
Lighting-Insensitive Recognition

• Illuminate subject from many incident
  directions Space of images as lighting is varied

8
Checking Image Consistency

• Easy to tamper / splice images
• Image processing software widely available
• In news reporting and other applications
• Need to detect tampering or photomontage
• Verify image consistency
• Try to check consistency of lighting, shading

9
Checking Image Consistency
10
Checking Image Consistency

• Without knowing illumination and reflectance
• How do we know if two objects / people in a
  photograph are lit consistently?
• How do we detect inconsistencies or tampering?

11
Outline

• Motivation and Practical Problems
• Spherical Convolution and Applications
• A Theory of Spherical Harmonic Identities
• Signal Processing for Visual Appearance

12
Environment Maps
Miller and Hoffman, 1984
Later, Greene 86, Cabral 87, 99,
13
Irradiance Environment Maps
Incident Radiance (Illumination Environment Map)
Irradiance Environment Map
14
Computing Irradiance

• Classically, hemispherical integral for each
  pixel
• Lambertian surface is like a low pass filter
• Frequency-space analysis (spherical harmonics)

Incident Radiance
Irradiance
15
Spherical Harmonics
0
1
2 . . .
-1
-2
0
1
2
16
Analytic Irradiance Formula

• Lambertian surface is low-pass filter

0
0
1
2
Basri Jacobs 01 Ramamoorthi Hanrahan 01a
17
9 Parameter Approximation
Order 2 9 terms
Exact image
0
RMS Error 1
1
For any illumination, average error lt 2
Basri, Jacobs 01
2
-1
-2
0
1
2
Ramamoorthi and Hanrahan 01b
18
Real-Time Rendering

• Simple procedural rendering method (no textures)
• Requires only matrix-vector multiply and
  dot-product
• In software or NVIDIA vertex programming hardware
• Widely used in Games (AMPED for Microsoft Xbox),
  Movies (Pixar, Framestore CFC, )

19
Computer Vision Complex Illumination

• Low Dimensional Subspace
• Lighting Insensitive Recognition (Basri and
  Jacobs 01, Lee et al. 01, Ramamoorthi 02, )
• Photometric stereo, shape acquisition

20
Convolution for General Materials

Spherical Harmonics
Ramamoorthi and Hanrahan 01
21
Related Theoretical Work

• Qualitative observation of reflection as
  convolution Miller Hoffman 84, Greene
  86, Cabral et al. 87,99
• Reflection as frequency-space operator DZmura
  91
• Lambertian reflection is convolution Basri
  Jacobs 01
• Our Contributions
• Explicitly derive frequency-space convolution
  formula
• Formal quantitative analysis in general 3D case
• Apply to real-time, inverse rendering, computer
  vision

22
Natural Lighting, Realistic Materials

Ramamoorthi and Hanrahan 02
23
Inverse Rendering

• 3 photographs of cat sculpture
• Complex unknown illumination
• Geometry known
• Estimate microfacet BRDF and distant lighting

24
New View, Lighting
Photograph
Rendering
Ramamoorthi and Hanrahan, 01c
25
Outline

• Motivation and Practical Problems
• Spherical Convolution and Applications
• A Theory of Spherical Harmonic Identities
• Signal Processing for Visual Appearance

Mahajan, Ramamoorthi, Curless ECCV 06
26
Two Objects Two Lightings
Material 1
Material 2
Lighting 1
Lighting 2
27
Two Objects Two Lightings
Material 1
Material 2
Lighting 1
Lighting 2
28
Two Objects Two Lightings
Material 1
Material 2
Lighting 1
Lighting 2
29
Two Objects Two Lightings
Material 1
Material 2
Lighting 1
Independent of Lighting and BRDF
Lighting 2
30
Image Estimation Framework
Material 1
Material 2
Lighting 1
?
Lighting 2
31
Image Estimation Framework
Material 1
Material 2
Lighting 1
?
Lighting 2
32
Image Estimation Framework
Material 1
Material 2
Lighting 1
?
Lighting 2
33
Image Estimation BRDF Transfer
Material 1
Material 2
Lighting 1
BRDF Transfer Function
?
Lighting 2
34
Image Estimation Lighting Transfer
Material 1
Material 2
Lighting 1
?
Lighting 2
35
Image Estimation Lighting Transfer
Material 1
Material 2
Lighting 1
?
Lighting 2
36
Image Estimation Lighting Transfer
Material 1
Material 2
Lighting 1
Light Transfer
Function
?
Lighting 2
37
Image Estimation

Lighting 1
Object 1 Object 2
?
Lighting 2
Our Method
Actual
No BRDF and lighting known or estimated
38
Image Consistency Checking
Spliced Image
39
Image Consistency Checking
Spliced Image
40
Image Consistency Checking
Tampered Cat
Untampered Cat
Tampered Cat
Two Lightings Same Reflectance Identity
Single Image Identity diffuse specular case
Two Materials Two Lightings identity
41
Outline

• Motivation and Practical Problems
• Spherical Convolution and Applications
• A Theory of Spherical Harmonic Identities
• Signal Processing for Visual Appearance

42
Signal Processing for Appearance

• Signal Processing widely applicable visual
  appearance
• Convolution relation for cast shadows
  Soler and Sillion 98, Ramamoorthi
  et al. 04
• Convolution with glows for participating media
  (mist, fog, haze) Sun et al. 05
• Signal-Processing analysis of light field and
  reflectance Chai et al. 00, Zickler et al. 06
• Triple Product Integrals Ng et al. 04
• First Order Analysis Ramamoorthi et al. 06

43
Real Time Rendering of Scattering
Video clip 1
44
Reflectance Sharing
Video clip 1
45
Acknowledgements

• Collaborators
• Dhruv Mahajan
• Brian Curless
• Pat Hanrahan
• Sameer Agarwal (helpful discussions)
• Funding NSF and Sloan Foundation
• http//www.cs.columbia.edu/ravir

46
Questions