High Dynamic Range from Multiple Images: Which Exposures to Combine?

1
High Dynamic Range from Multiple Images Which
Exposures to Combine?

• Michael Grossberg and Shree Nayar
• CAVE Lab, Columbia University
• ICCV Workshop on CPMCV
• October, 2003, Nice, France
• Partially funded by NSF ITR Award, DARPA/ONR MURI

2
Cameras Have Limited Dynamic Range
3
Combining Different Exposures
Low Dynamic Range Exposures
Combination Yields High Dynamic Range
Ginosar and Zeevi, 88, Madden, 93, Mann and
Picard, 95, Debevec and Malik, 97, Mitsunaga and
Nayar, 99
4
Combining Different Exposures
Low Dynamic Range Exposures
Combination Yields High Dynamic Range
Ginosar and Zeevi, 88, Madden, 93, Mann and
Picard, 95, Debevec and Malik, 97, Mitsunaga and
Nayar, 99
5
The Camera Response
Linear Function (Optical Attenuation)
Image Irradiance
Scene Radiance
E
L
s
6
From Response To Measured Irradiance Levels
Response Function f
Brightness Levels B
Irradiance E
7
Where do you want your bits?
Response Function f
Coarse quantization
Brightness B
High Dynamic Range
Irradiance E
Response Function f
Fine quantization
Brightness B
Low Dynamic Range
Irradiance E
8
Effective Camera from Multiple Exposures
Acquired Images
Image from Effective Camera
Goal
Capture High Dynamic Range
9
Flexible Dynamic Range Imaging

• Can we create an effective camera with a desired
  response?
• How many exposures are needed?
• Which exposures to acquire?
• How to combine the acquired images?

10
Irradiance Levels From Multiple Exposures
f(E)
Exposure e1 1
Irradiance E
f(e2E)
Exposure e2
h(E)
Effective Camera
11
Response of the Effective Camera
Number of exposures
Camera Response
Effective Response
Exposures
Irradiance
Theorem The sum of a set of images of a scene
taken at different exposures includes all the
information in the individual exposures.
12
Camera Response Emulation
Emulated Response depends on
h
f ,
e (e1, ,en)
Brightness Levels B
g
Desired Response

• How can we tell if h emulates g well?
• Naïve answer h g lt e

Irradiance E
Level spacing characterizes similarity
13
How Response Determines Level Spacing
h
Observation The derivative determines the
distances between levels.
Larger Derivative
Brightness Levels B
Dense Spacing
Irradiance E
14
The Objective Function
Spacing Based Comparison
Desired Response
Effective Response
Number of Exposures
Exposure Values
Weight
Weight prevents penalizing success
15
Accounting for Camera Noise

• Add term to Penalize Variance
• Example Gaussian Noise
• Full Noise model depends on
• Thermal, Shot, or Read Noise
• Integration Time, Gain, Aperture
• Camera Response

Weight
Noise Term
Noise Variance
Exposure Values
16
Which Exposures and How Many?

• For fixed n, find minimizing exposure values e
• Choose min n such that error within tolerance
• Method Exhaustive search
• Objective function not continuous
• Only need to search actual settings
• Offline build table of exposures

17
Flexible Dynamic Range Imaging
18
Flexible Dynamic Range Imaging
19
Baseline Exposure Values

• Typically exposures are doubled
• Baseline
• Combine the exposures e (1,2,4)

Ginosar and Zeevi, 88, Madden, 93, Mann and
Picard, 95, Debevec and Malik, 97, Mitsunaga and
Nayar, 99
20
Increased Dynamic Range Linear Camera

• Real Camera f linear
• Desired Camera g linear (greater dynamic range)

Brightness
Brightness
8-bit real camera
21
Linear Camera Synthetic Ramp Image
22
Linear Camera Synthetic Ramp Image
23
Linear Camera Image of Cloth
Computed Exposures (1,1.05,1.11)
24
Constant Contrast from Linear Cameras

• Real Camera f linear
• Desired Camera g log response (constant
  contrast)

25
Constant Contrast Image of Tiles
Baseline Exposures
Computed Exposures
26
Linear Camera from Non-linear Camera

• Real Camera f non-linear (Nikon 990)
• Desired Camera g linear

Brightness
Brightness
27
Summary

• Combine images using summation
• Method finds number of exposures and exposure
  values to use
• Emulation of a variety of cameras
• Flexible Dynamic Range Imaging

28
Colors