Recovering Photometric Properties of Architectural Scenes from Photographs

1
Recovering Photometric Properties of
Architectural Scenes from Photographs
Yizhou Yu Jitendra Malik
Computer Science Division University of
California at Berkeley
July 1998
2
Context

• IBMR re-renders from novel viewpoints.
• Façade, Plenoptic modeling,
• Lumigraph, Light field,
• Panoramic mosaics
• But, unlike traditional rendering,
  lighting cannot be changed.

3
The Problem

• Texture Maps are not Reflectance Maps !
• Need to factorize images into
  lighting and reflectance maps

Illumination
Radiance
Reflectance
4
Objective

• Start from photographs
• Recover parametric models for lighting and
  reflectance
• Re-render the scene under novel lighting
  conditions

5
Some Photographs...
6
Camera Radiance Response Curve

• Pixel brightness value is a nonlinear function of
  radiance.
• Debevec MalikSiggraph97 give a method to
  recover this nonlinear mapping.

Intensity
Saturation
Radiance
Radiance
7
Previous Work

• BRDF measurement and recovery
• Ward 92,Dana et al. 97
• Sato Ikeuchi 96, Sato et al. 97
• Rendering outdoor scenes under skylight
• Nishita and Nakamae 86, Tadamura et al. 93

8
Basic Approach

• Recover geometric model
• Measure and recover illumination
• Recover reflectance
• Predict illumination at novel times of day
• Render

Illumination
Radiance
Reflectance
9
Technical Challenges

• Nonlinear mapping between input radiance and
  digital output .
• Photographs cannot easily recover full spectral
  BRDF.
• Re-rendering the scene at novel times of day
  requires predicting lighting conditions.

10
Basic Approach

• Measure and recover illumination
• Recover reflectance
• Predict illumination at novel times of day
• Render

Illumination
Radiance
Reflectance
11
Modeling the Illumination

• The sun
• Its diameter extends 31.8 seen from the earth.
• The sky
• A hemispherical area light source.
• The surrounding environment
• Modeled as a set of oriented Lambertian facets.

12
A Sky Radiance Model----based on Perez 93
zenith
Sky element
sun

• Recover a set of parameters
  for each color channel
• Take photographs for parts of the sky
• Use Levenberg-Marquardt algorithm to fit data

Lvz, a, b, c, d, e, f
13
A Recovered Sky Radiance Model
R,G,B channels
14
Coarse-grain Environment Radiance Maps

• Partition the lower hemisphere
  into small regions
• Take photographs at several
  times of day
• Project pixels into regions and
  obtain the average radiance
• Use photometric stereo to
  recover a facet model for each region

15
Basic Approach

• Measure and recover illumination
• Recover reflectance
• Predict illumination at novel times of day
• Render

16
Recovering Reflectance

• Parametric model Lafortune et al.
• Triangulate the surfaces
• Set a grid on each triangle to
  capture spatial variations
• Use one-bounce reflection to approximate
  self-interreflections

17
Pseudo-BRDF

• R, G, B color channels perform integration.
  Define pseudo-BRDF
• In general, the pseudo-BRDF varies with the
  spectral distribution of the light source.
• Recover two sets of surface pseudo-BRDFs
• One gt spectral distribution of the
  sun
• The other gt the sky and environment

18
Diffuse Term

• For each side, at least two photographs for
  diffuse albedo recovery.
• From the photograph not lit by the sun
• From the photograph lit by the sun
• Solve for

19
Specular Term

• Use an empirical specular reflection model
  proposed in Lafortune et al. 97.
• Recover the parameters using least
  squares and robust statistics.

20
Basic Approach

• Measure and recover illumination
• Recover reflectance
• Predict illumination at novel times of day
• Render

21
Simulating Novel Lighting for the Sun and Sky

• Interpolation with solar position alignment to
  obtain novel sky radiance distributions
• Use to model solar radiance
  during sunrise and sunset
• This is similar to the absorption term used in
  scattering theory.

22
A Local Facet Model for the Environment

• Recover a distinct model for each environment
  region
• Obtain environment radiance maps.
• Set up over-determined systems as in
  photometric stereo and ignore inter-reflections.
• Solve for

lsun
nenv
23
Recovered Environment Radiance Models
Synthetic
Real
24
Relative Importance of the Components

• On shaded sides, the irradiance from the
  landscape is larger than that from the sky.
• On sunlit sides, the sun dominates the
  illumination.
• The specular component is very small compared to
  the diffuse component.

25
Video
26
Basic Approach

• Measure and recover illumination
• Recover reflectance
• Predict illumination at novel times of day
• Render

27
Comparison with Real Photographs
Synthetic
Real
28
High Resolution Re-rendering

• Low resolution
  and High resolution
• and are given.
• since the illumination
  has small variations in high frequencies.

29
High Resolution Re-rendering
Real reference image
High resolution synthetic image
Low resolution synthetic image
30
Video
31
Summary

• An approach to render real architectural scenes
  under novel lighting conditions
• The pseudo-BRDF concept
• Methods for modeling lighting at novel
  times of day
• A simple method for high resolution
  re-rendering

32
Acknowledgments

• George Borshukov
• Paul Debevec
• David Forsyth
• Greg Ward Larson
• Carlo Sequin
• MURI 3DDI
  California MICRO Program
  Philips Corporation