Temporal Dithering of Illumination for Fast Active Vision

1
Temporal Dithering of Illuminationfor Fast
Active Vision
Srinivasa G. Narasimhan
Sanjeev J. Koppal
Shuntaro Yamazaki
Sponsors NSF, ONR
2
Computational illumination for Vision
Separation of light transport
Structured-light scanning
Office of the Future
Dual Photography
Zickler et al, Raskar et al, Zhang et al, Levoy
et al, Debevec et al,Nayar et al
3
Programmable Illumination
image (60Hz)
projector
scene
active vision
computer
camera
4
Illumination from DLP projectors
Video acquired at 60Hz
Video acquired at 1000Hz
Fast way of encoding scene illumination
5
High-speed Illumination Dithering
DMD controller (9600Hz)
image (60Hz)
micro-mirrors
DLP projector
scene
fast active vision
higher fps sensor
computer
camera
Pulse-Width Modulation (PWM) Coding
6
Micro-mirror arrays for Imaging and Display
Programmable imaging Nayar et al 04
Stereoscopic display Jones et al, 2007
Imperceptible display Raskar et al 98
Scanning Cotting et al, 2004
7
Color wheel effect Number of brightnesses
Texas Instruments, DLP.com
8
Reverse engineering a DLP projector
Intensity mosaic input to projector
Video captured at 3000 Hz
Video captured at 500 KHz
Video captured at 1000 Hz
9
Reverse engineering a DLP projector
Projected value
Measured intensity
165
215
187
255
255
21000
Projected
value
0
2000
1
100
Frames
1
100
Frames
10
Applications
11
Fast structured-light based reconstruction
Scene
Camera
DLP Projector
Structured Pattern
12
Captured video (30Hz)
Captured video (3000Hz)
Reconstruction (30Hz)
Reconstruction (120Hz)
Reconstruction different view (120Hz)
13
Reconstruction (30Hz)
Reconstruction (120Hz)
Different View
14
Captured video (30Hz)
Captured video (3000Hz)
Reconstruction (30Hz)
Reconstruction (120Hz)
Reconstruction different view (120Hz)
15
Reconstruction (30Hz)
Reconstruction (120Hz)
Different View
16
Captured video (30Hz)
Captured video (3000Hz)
Reconstruction (30Hz)
Reconstruction (120Hz)
Reconstruction different view (120Hz)
17
Reconstruction (30Hz)
Reconstruction (120Hz)
Different View
18
Captured video (30Hz)
Captured video (3000Hz)
Reconstruction different view (120Hz)
Reconstruction (30Hz)
Reconstruction (120Hz)
19
Reconstruction (30Hz)
Reconstruction (120Hz)
Different View
20
Reconstruction (jiggling effect)
Captured Video (30Hz)
21
Patterns dithering at 3000 Hz
Moving cloth shown at 30 Hz
Reconstruction (120 Hz)
22
object speed
camera fps
1000 fps
3000 fps
500 fps
120 fps
23
Multiplexing Illumination Dithering
24
Multi-spectral Illumination Multiplexing
Multiplexed intensity
Multiplexed gray scale video
multi-spectral source intensities (R, G, B)
single source image
Scene illuminated by three projectors
Schechner 05, Grossberg, 07
25
Multi-spectral Illumination Multiplexing
Multiplexed intensity
De-multiplexed color video
multi-spectral source intensities (R, G, B)
single source image
Schechner 05, Grossberg, 07
26
Illumination Multiplexing
Multiplexed intensity
Multiplexed gray scale video
source intensities
single source image
Scene illuminated by three projectors
Schechner et al 03
27
High-speed Photometric Stereo
De-multiplexed intensity
single source image
computed source direction
Computed Shape (300Hz)
Horn 86
28
High-speed Photometric Stereo
Scene illuminated by three projectors
Surface normals (300Hz)
Frame 1
Frame 5
Frame 10
29
DLP Photography
DLP projector
scene
still camera
30
DLP Photography
Ballet Right leg kick (1s exposure)
31
DLP Photography
Ballet Hand and head motion (1s exposure)
32
DLP Photography
Ballet Body motion (1s exposure)
33
Motion Blurring under DLP illumination
Blurring under normal lighting
I
t
object motion
notice the smear
object
Blurring under DLP illumination
I
t
object
notice the high-frequencies
projector dithering color wheel
similar to Flutter shutter, Raskar et al, 06
34
Frequencies of Motion PSF
magnitude
dithering PSF
rectangular PSF
frequency
35
Conclusions

• High speed illumination control using
  micro-mirrors.
• Reverse engineering an off-the-shelf DLP
  projector.
• Speedup of virtually any active vision
  technique.

36
Motion Blurring under DLP illumination
Motion blur
Skylight
DLP
Deblurred result
Frequencies of Blur PSF
Dithering PSF
Skylight PSF
37
Motion Blurring under DLP illumination
Motion blur
Skylight
DLP
Deblurred result
Frequencies of Blur PSF
38
Direct and Global separation for dynamic scenes
High speed camera
Beam splitter
DLP projector
Frame 2719
Direct component
Global component
Frame 2727
Frame 2730
Separation Result
Flipped Checker Patterns
(Direct Global)
Nayar et al, 06
39
Co-locating projector and camera
High speed camera
Side-view with shadow
Co-located view without shadow
Beam splitter
DLP projector
Setup
Side-view shows distorted pattern
Co-located view shows no distortion
40
Reverse engineering a DLP projector
10000hz
3000hz
1000hz
500hz
41
High-speed Illumination dithering
spatio-temporal modulation (60Hz)
high speed dithering (10kHz)
Lens
DLP projector
scene
fast active vision
higher fps
Lens
camera
computer
42
Fast projector-camera pixel correspondences
Stripe correspondences
Patterns dithering at 3000 Hz
43
Fast reconstruction of rigid objects
Recovered 3D structure ( holes due to occlusions
and dark cast shadows )
Moving statue shown at 30Hz
44
Fast reconstruction of non-rigid objects
Patterns dither at 3000 Hz
Moving cloth at 30 Hz
Reconstruction shown at 120 Hz
Reconstruction shown at 30 Hz
45
(No Transcript)
46
Deblurring at 125Hz
Blurred image under DLP illumination
Deblurred image
47
High-speed photometric stereo
Instantaneous shape
Depth map of moving statue
Moving Statue
48
Fast photometric stereo Statue
Surface normals
Needle map
3D shape by integrating normals
49
Illumination de-multiplexing for dynamic scenes
Illumination multiplexed images
multiple shadows
source intensities
multiple specularities
single source image
Illuminated by three projectors
Illumination de-multiplexed images
Illuminated by projector 1
Illuminated by projector 2
Illuminated by projector 3
Schechner et al, 03
50
Fast reconstruction of rigid objects
Patterns dither at 3000 Hz
Ping-pong balls dropped, shown at 30 Hz
Reconstruction shown at 120 Hz
51
Box PSF
Measured PSF
Motion blur
Deblurred image
Deblurred image
(a) Skylight illumination
Measured PSF
Box PSF
Deblurred image
Deblurred image
Motion blur
(b) Fluorescent illumination
DLP point spread function
PSF for box function
Measured PSF
Measured PSF for skylight
Measured PSF for fluorescent light
Motion blur
Deblurred image
(c) DLP illumination
(d) Comparison of PSF frequencies
52
object speed
camera fps
1000 fps
3000 fps
500 fps
120 fps
53
object speed
camera fps
1000 fps
3000 fps
500 fps
120 fps
54
(No Transcript)
55
(No Transcript)
56
Micromirror arrays in DLP projectors
Mirrors (10 x 10 microns)
Measured intensity
Digital Micromirror Device (DMD)
Time
Pulse-Width Modulation (PWM) Coding
Texas Instruments, DLP.com