Ramesh Raskar, Michael S. Brown, Ruigang Yang, Wei-Chao Chen, Greg Welch, Herman Towles, Brent Seales, Henry Fuchs - PowerPoint PPT Presentation

About This Presentation
Title:

Ramesh Raskar, Michael S. Brown, Ruigang Yang, Wei-Chao Chen, Greg Welch, Herman Towles, Brent Seales, Henry Fuchs

Description:

Flight sims, Panoram, Trimensions [Raskar98] Single wide ... (2) Render Model. 2-pass Rendering. Project this 'warped' image, the viewer will see it correctly ... – PowerPoint PPT presentation

Number of Views:173
Avg rating:3.0/5.0
Slides: 85
Provided by: ramesh48
Category:

less

Transcript and Presenter's Notes

Title: Ramesh Raskar, Michael S. Brown, Ruigang Yang, Wei-Chao Chen, Greg Welch, Herman Towles, Brent Seales, Henry Fuchs


1
Multi-Projector Displays usingCamera-based
Registration
  • Ramesh Raskar, Michael S. Brown, Ruigang Yang,
    Wei-Chao Chen, Greg Welch, Herman Towles, Brent
    Seales, Henry Fuchs
  • University of North Carolina at Chapel Hill

2
Multi-Projector Panoramic Displays
Traditional Display Setups
3
Multi-Projector Panoramic Displays
Traditional Display SetupsPrecise Geometry,
Well-defined overlaps
4
Generalized Panoramic Display
Flexible Display SetupsCasually aligned
projectors and screens
5
Problem
  • Traditionally rigid design
  • Painstaking construction
  • Room must be modified
  • Projector alignment is tedious
  • High maintenance cost
  • Constant adjustments
  • Projectors become unaligned over time

6
Problem Instant Panoramic Display
  • Generalized display environment
  • Setup display surface approximately
  • Align projectors casually
  • Render with what you have

7
Multi-Projector Goal
  • Seamless imagery
  • Immersive 3D images, moving user
  • Casually setup projectors and surfaces
  • Low maintenance
  • Auto-Calibration
  • Perceptually single logical projector

8
Goal Seamless Display
  • (a) Projector Registration
  • Geometric alignment
  • (b) Intensity blending
  • Smooth transition

Projection 2
Projection 1
9
Goal Seamless Display
  • (a) Projector Registration
  • Geometric alignment
  • (b) Intensity blending
  • Smooth transition

10
Goal Seamless Display
  • (a) Projector Registration
  • Geometric alignment
  • (b) Intensity blending
  • Smooth transition

11
Goal Seamless Display
  • (a) Projector Registration
  • Geometric alignment
  • (b) Intensity blending
  • Smooth transition

12
Goal Seamless Display
  • (a) Projector Registration
  • Geometric alignment
  • (b) Intensity blending
  • Smooth transition

13
Previous approachesMulti-projector setups
  • Sweet spot
  • Static user
  • Moving User
  • Head-tracked user

14
Previous approaches Sweet spot (static user)
  • Multiple projectors
  • (well-defined overlaps)
  • Flight sims,
  • Panoram, Trimensions
  • Raskar98
  • Single wide-field-of-view projector
  • OmniMax
  • Alternate Realities

User
15
Previous approaches Moving user
  • CAVE
  • Rigid setup
  • Well-defined geometry
  • Large support structure

Projector
User
16
Previous approaches Moving user
  • Office of the Future Raskar98
  • Relaxed construction constraints
  • Assumes precise display geometry recovery
  • Demonstrated for a single projector

17
Related work Image Mosaicing
  • QuickTime VR
  • Pair-wise mosaicing of photographs
  • Generalized mosaicing

18
Goal for this paper
  • Seamless image
  • Immersive 3D images, moving user
  • Arbitrary projectors and surfaces
  • Eliminate maintenance
  • Self-calibration
  • Perceptually single logical projector

19
Basic Approach Panoramic Display
  • Position projectors approximately
  • Use video cameras to recover
  • display geometry
  • projector configuration
  • Track user
  • Render images of 3D scenes
  • Warp and blend projected images to achieve
    registration

20
Outline
  • Single projector
  • Calibration, Display recovery, Rendering
  • Multiple projector
  • Registration, Blending
  • New techniques
  • Post rendering warp, mesh unification

21
Video
22
Outline
  • Single projector
  • Rendering, Calibration, Display recovery
  • Multiple projector
  • Registration, Blending
  • New techniques
  • Post rendering warp, mesh unification

23
2-pass Rendering
3D Display Surface Model
Projector
User
24
2-pass Rendering Project this warped image,
the viewer will see it correctly
3D Display Surface
Projector
(2) Render Model
(1) Desired Image
User
25
Single projector display2-pass unknowns
  • Arbitrary display surface
  • Arbitrary projector location
  • Need to determine these for 2 pass rendering

- How?
User
26
Single projector displayCamera calibration
1. Calibrate stereo pair
Stereo camera pair
27
Single projector displayDisplay surface
estimation
1. Calibrate stereo pair
2. Determine display surface
Stereo camera pair
28
Single projector displayProjector estimation
1. Calibrate stereo pair
2. Determine display surface
3. Determine projector
Stereo camera pair
29
Single projector displayProjector estimation
1. Calibrate stereo pair
2. Determine display surface
3. Determine projector
Display Surface
Stereo camera pair
Projector
30
Single projector display Display Surface
Estimation
  • Display surface is recovered by cameras (via
    projector)
  • Surface mesh is created from the 3D points
  • Allows us to calibrate the projector
  • Approximate speed
  • 720 surface point samples (1 minute)
  • 14,000 surfaces point samples (15 minutes)

31
Single projector displayCoordinate Frame
  • Display surface and projector are in the
    coordinate frame of the Calibration Pattern

32
Multiple ProjectorsPanoramic display environments
  • A single stereo-pair cannot see entire display
  • Must use several stereo-pairs
  • Each in its own coordinate frame
  • Must register data into common coordinate frame
  • Advantage
  • We control the light
  • Makes registration easier

33
Multiple ProjectorsSurface mesh registration
Display Surface
34
Multiple ProjectorsDisplay surfaces are in
different coordinate frames
3
Display Surfaces
2
1
35
Multiple ProjectorsMust register into common
coordinate frame
3
Display Surfaces
2
1
36
Multiple ProjectorsSurface meshes are registered
into the same coordinate frame
3
Display Surfaces
2
1
37
Multiple ProjectorsRe-calibrate projectors so
they are registered
3
Display Surfaces
2
Re-calibrate the projectors so they
are registered into a common coordinate frame
1
38
Multiple ProjectorsRe-calibrate projectors so
they are registered
3
Display Surfaces
2
With display surfaces and projectors registered
we can achieve geometric correctness . . .
1
39
Multiple ProjectorsIntensity Blending
Need to attenuate intensities in the
overlapped region
40
Intensity BlendingUse camera to find overlaps
Camera observes overlapped region
41
Intensity BlendingAssigning intensity weights
(alpha)
Camera image plane
Assign intensity weights based on distance
between projector boundaries
42
Intensity BlendingAssigning intensity weights
(alpha)
Camera image plane
Assign intensity weights based on distance
between projector boundaries
43
Intensity BlendingAlpha-masks
Projector alpha-masks
44
Achieving our goal Seamless display
  • Display surface and projector registration
  • Geometrically alignment imagery
  • Intensity blending
  • Smooth intensity transition between projectors

45
Registration Issues
  • Errors in estimation
  • user location
  • 3D display surface
  • projector parameters

46
Registration Issues
  • Errors in estimation
  • user location
  • 3D display surface
  • projector parameters
  • Single projector, errors not visible
  • Multi-projector overlap
  • Geometric mis-registrations
  • Gaps, Discontinuities

47
Sources of errors
  • Head-tracker for user location
  • Display surface recovery
  • Camera Calibration
  • Radial Distortion
  • Pattern Imperfection
  • Feature detection
  • Projector calibration
  • Relies on display surface recovery

48
Re-projection error
M
Projector framebuffer
m'
m
m'
m
P
P
P
49
Sources of re-projection error
  • Minimization residual
  • Projector not pin-hole device
  • Inaccuracies in estimated 3D pts on display
    surface

50
Effect of errors
  • User location
  • Display surface meshes
  • difficult to register
  • incorrect surface representation
  • Projector parameters
  • re-projection error
  • incorrect projection matrix

51
Geometric Error Compensation
  • Objective
  • Neighboring projectors use same display surface
    representation
  • Each projection matrix maps 3D screen pts to 2D
    pixels that illuminate them
  • Overcoming inaccuracies
  • Geometric continuity everywhere for display
  • Fidelity of projector model

52
Outline
  • Single projector
  • Calibration, Display recovery, Rendering
  • Multiple projector
  • Registration, Blending
  • New techniques
  • Post rendering warp, mesh unification

53
Geometric Error Compensation
  • Objective
  • Neighboring projectors use same display surface
    representation
  • Surface Mesh Unification
  • Each projection matrix maps 3D screen pts to 2D
    pixels that illuminate them
  • Post Rendering Warp

54
Surface mesh unification
  • Merge multiple recovered meshes
  • Create a single representation of display
    surfaces
  • Rigid transformation creates near alignment
  • Errors lead to discontinuities
  • Use smooth 3D transition

55
Surface mesh unification
Actual surface
P2
P1
56
Surface mesh unification
57
Surface mesh unification
58
Surface mesh unification
59
Surface Unification
New surface mesh
60
Surface Unification
Original surface
New surface mesh
Both projectors use same (incorrect) surface
geometry
61
Geometric Error Compensation
  • Objective
  • Neighboring projectors use same display surface
    representation
  • Surface Mesh Unification
  • Each projection matrix maps 3D screen pts to 2D
    pixels that illuminate them
  • Post Rendering Warp

62
Modified re-projection error
Actual Pt
M
M
Estimated Pt
P2
P1
63
Modified re-projection error
M
M
P2
P2
P1
P1
64
Modified re-projection error
M
M
P2
P2
P1
P1
65
Modified re-projection error
M
M
m2
m2
m2
m2
m1
m1
P2
P2
m1
P1
P1
m1
66
Effect of reprojection error
  • Physically m1 and m2 create image at M
  • In estimated model, m1 and m2 correspond
  • Features of a virtual object gets rendered at
    m1 and m2
  • But, m1 and m2 are not likely to correspond

67
Effect of Re-projection error
V
M
M
V
User
m2
m1
m2
m1
M
68
Eliminating re-projection error
  • We have
  • m1 P1 M
  • m2 P2 M

69
Eliminating re-projection error
  • We have
  • m1 P1 M
  • m2 P2 M
  • We need a nonlinear projective function
  • m1 F1 (M)
  • m2 F2 (M)

70
Eliminating re-projection error
  • We have
  • m1 P1 M
  • m2 P2 M
  • We need a nonlinear projective function
  • m1 F1 (M)
  • m2 F2 (M)
  • Cannot be achieved using single projection matrix
  • Lack of hardware support

71
Post Rendering warp
  • Goal
  • m1 F1(M) m2 F2(M)
  • (a) Render using estimated projection matrix
  • m1 P1 M m2 P2 M
  • (b) Warp resultant image
  • m1 --gt m1 m2 --gt m2

72
Post Rendering warp
  • Goal
  • m1 F1(M) m2 F2(M)
  • (a) Render using estimated projection matrix
  • m1 P1 M m2 P2 M
  • (b) Warp resultant image
  • m1 --gt m1 m2 --gt m2
  • Expensive to specify per-pixel warp
  • Use piecewise planar approximation

73
Post Rendering warp (using texture mapping)
  • Define grid of pixels in projector framebuffer
    (m)
  • Project and record 3D surfaces pts (M)
  • After surface unification (M),
  • find reprojected points (m)
  • Tesselate the framebuffer
  • Load rendered image in texture memory
  • Render vertices m with texture coordinates as m

74
Post Rendering warp (using texture mapping)
Record 3D pts
Project pixel grid
Find reprojected pixels
75
Post Rendering warp (using texture mapping)
Compute required post-rendering warp
76
Summary of TechniquesBasic Approach
  • At each projector
  • calibrate camera pair
  • project dots and find 3D display surface
  • (find surface illuminated by neighboring
    projectors)
  • find projector parameters
  • Find intensity blending functions
  • Register surface meshes (near alignment)

77
Summary of TechniquesCompensating the errors
  • Unify meshes using smooth 3D transition
  • Use camera overlap as transition region
  • Re-compute all best-fit projector matrices
  • Find projective function
  • piecewise planar approximation using multiple
    triangles
  • Specify all rendering tasks using a display list

78
Summary of Techniques
  • At run time, at each projector
  • Read current user location
  • Render virtual object using two-pass rendering
  • Post-rendering 2D warp
  • Intensity blending

79
Restricted cases
  • Static User (Sweet spot)
  • VSMM98
  • Planar display surface
  • Tech report
  • Available at project web page

80
Features
  • Task of each projector is independent
  • 2-pass rendering, post-rendering warp is
    pre-defined
  • Can be specified using display list
  • Traditional hardware

81
Implementation
  • 5 Single lens 1024x768 projectors
  • Multiple 640x480 cameras
  • SGI InfiniteReality2 for each projector
  • Matlab for calibration, OpenGL for rendering

82
Issues
  • Resampling artifacts due to limited resolution
  • Color matching
  • Viewer-dependent parameters
  • Limited depth-of-field

83
Acknowledgements
  • Office of the Future group
  • NSF STC
  • NTII (Advanced Network Systems)
  • Intel

84
Summary
  • Multiple projectors to create panoramic imagery
  • Geometric registration is major challenge
  • Video cameras to create 3D representation
  • Generalized display configuration
  • www.cs.unc.edu/stc
Write a Comment
User Comments (0)
About PowerShow.com