Flexible Mirror Imaging

1
Flexible Mirror Imaging
Sujit Kuthirummal Shree K. Nayar
Columbia University
ICCV Workshop on Omnidirectional Vision October
2007, Rio de Janeiro, Brazil
Supported by ONR
2
Conventional Camera
Captured Image
Exposure
Image Credit Nautile _at_ Flickr
3
Conventional Camera
Captured Image
Exposure
Focus
Image Credit Nautile _at_ Flickr
4
Conventional Camera
Captured Image
Field of View
Exposure
Focus
Image Credit Nautile _at_ Flickr
5
Conventional Camera
Wasted Pixels
Wasted Pixels
Captured Image
Field of View
Exposure
Focus
Control over the field of view is limited
Image Credit Nautile _at_ Flickr
6
(No Transcript)
7
Cameras with Flexible Fields of View

• Creative Photography

• Flexible Surveillance

8
System with a Flexible Field of View
Camera
Flexible Mirror Sheet
9
System with a Flexible FOV
10
Flexible Field of View
11
Flexible Optics
Adaptive Optics Tyson 98, Roddier 99
Digital Micro-Mirrors Nayar et al. 04, Hicks et
al. 07
Flexible Camera Arrays Nomura et al. 07
12
Analyzing a Captured Image
Mirror Shape
Captured Image
Geyer Daniilidis 99, Kang 00, Micusik
Pajdla 04, Savarese et al. 05, Grossberg
Nayar 05
13
Analyzing a Captured Image
Automatically Detected 2D Boundary
Mirror Shape
Captured Image
3D Boundary
3D Shape
Decaudin et al. 06
14
Analyzing a Captured Image
Automatically Detected 2D Boundary
Mirror Shape
Captured Image
T
2D Mirror Boundary
3D Mirror Shape
15
Characterizing 2D Mirror Boundary
Mirror Boundary Descriptor D
16
Modeling 3D Mirror Shape
1D Spline Basis Functions
of Spline Basis Functions
Spline Coefficients
Normalized Image Coordinates
3D Point on the Mirror
Spline Coefficients Vary with Mirror Shape
17
Modeling 3D Mirror Shape
1D Spline Basis Functions
of Spline Basis Functions
Spline Coefficients
Image Coordinates
3D Point on the Mirror
Spline Coefficients Vary with Mirror Shape
18
Estimating 3D Mirror Shape for an Image
T
2D Mirror Boundary
3D Mirror Shape
19
Estimating 3D Mirror Shape for an Image
T
Boundary Descriptor D
20
Offline Calibration
Calibration Set 30,753 frames
21
Offline Calibration
Compute Mirror Boundary Descriptor D
Estimate Spline Coefficients
Lookup Table T
22
Estimating 3D Mirror Shape for an Image
Mirror Boundary Descriptor D
Mirror Spline Coefficients
Lookup Table T
Captured Image
Estimated Mirror Shape
23
Estimating 3D Mirror Shapes for a Sequence
Captured Video
Estimated Mirror Shapes
RMS Percentage Error in 3D Shape Estimation
0.0926
24
Properties of the Imaging System

• Viewpoint Locus

• Field of View

• Resolution

25
Viewpoint Locus
Captured Image
Estimated Mirror Shape
and Viewpoint Locus
Burkhard and Shealy 73, Grossberg and Nayar 05
Captured Images are Multi-Perspective
26
Field of View
Effective Viewing Direction
27
Field of View
Effective Viewing Direction
FOV of the System
1, if sees the mirror and 0 otherwise
28
Field of View
Cameras Field of View Horizontal 28o
Vertical 21o
29
Resolution
Captured Image
Resolution Map
Captured Images have Spatially Varying Resolution
30
Undistorting a Captured Image
Multi-Perspective
Varying Resolution
Swaminathan et al. 03
31
Undistorting a Captured Image
Multi-Perspective
Varying Resolution
Unknown Scene Depth
Map a Captured Image to an Equi-resolution Image
32
Undistorting a Captured Image
ICs Pixel Grid
Captured Image IC

• Horizontal Resolution
• Vertical Resolution

33
Undistorting a Captured Image
ICs Pixel Grid
Captured Image IC

• Horizontal Resolution
• Vertical Resolution

34
Undistorting a Captured Image
ICs Pixel Grid
Captured Image IC

• Horizontal Resolution
• Vertical Resolution

35
(No Transcript)
36
Undistorting a Captured Image
ICs Pixel Grid
Captured Image IC
Samples of IH
Horizontal Stretching
Stretch IC horizontally so that horizontal
resolution at very pixel is equal (to )
Horizontally Stretched Image IH
37
Undistorting a Captured Image
ICs Pixel Grid
Captured Image IC
Samples of IH
Horizontal Stretching
Stretch IC horizontally so that horizontal
resolution at very pixel is equal
Horizontally Stretched Image IH
38
Undistorting a Captured Image
IHs Pixel Grid
Captured Image IC
Samples of Equi-resolution Image
Horizontal Stretching
Stretch IH vertically so that every pixel has
the same field of view
Horizontally Stretched Image IH
39
Undistorting a Captured Image
Captured Image IC
Vertical Stretching
Horizontal Stretching
Equi-resolution Image
Horizontally Stretched Image IH
40
Creating Undistorted Rectangular Image
Equi-Resolution Image
Rectangular Image
Thin-plate Spline based Image Warp
41
Undistorting a Captured Image
Captured Image
Equi-resolution Image
Rectangular Image
42
Dynamic Fields of View
43
Street Monitoring
44
Conversation
45
Conversation
46
Panning up
47
Panning up
48
Birthday
49
Birthday
50
(No Transcript)
51
Limitations and Future Work

• Defocus due to curvature of the mirror

52
Limitations and Future Work

• Defocus due to Mirror Curvature

Point Spread Functions (PSFs)
Spatially varying PSF
Captured Image
Focused Image
53
Limitations and Future Work

• Defocus due to Mirror Curvature

Point Spread Functions (PSFs)
Spatially varying PSF
Captured Image
Focused Image
54
Limitations and Future Work

• Defocus due to Mirror Curvature
• Limited Mirror Deformation Space

55
Limitations and Future Work

• Defocus due to Mirror Curvature
• Limited Mirror Deformation Space

56
Contributions
Novel System with Flexible FOV
Self-Calibration Procedure
Analysis of Systems Properties
Distortion Minimization Algorithm
Creative Photography, Flexible Surveillance
57
(No Transcript)
58
Evaluating Equi-resolution Algorithm
Convex Cylindrical Mirror
Concave Cylindrical Mirror
Rendering
Rendering
59
Evaluating Equi-resolution Algorithm
Convex Cylindrical Mirror
Concave Cylindrical Mirror
Rendering
Initial Resolution
Rendering
Initial Resolution
Equi-Resolution Image
Equi-Resolution Image
Resolution
Resolution
60
Undistorting a Captured Image
Captured Image Grid
Sphere of Viewing Directions
Equi-resolution Image Grid
Equi-Areal Projection of a Sphere onto a Plane
Mollweide, Hammer, etc.
Synder 97
61
Make Captured Images Rectangular?
Captured Image
Undistorted Rectangular Image
62
(No Transcript)