Title: Ramesh Raskar, Michael S. Brown, Ruigang Yang, Wei-Chao Chen, Greg Welch, Herman Towles, Brent Seales, Henry Fuchs
1Multi-Projector Displays usingCamera-based
Registration
- Ramesh Raskar, Michael S. Brown, Ruigang Yang,
Wei-Chao Chen, Greg Welch, Herman Towles, Brent
Seales, Henry Fuchs - University of North Carolina at Chapel Hill
2Multi-Projector Panoramic Displays
Traditional Display Setups
3Multi-Projector Panoramic Displays
Traditional Display SetupsPrecise Geometry,
Well-defined overlaps
4Generalized Panoramic Display
Flexible Display SetupsCasually aligned
projectors and screens
5Problem
- Traditionally rigid design
- Painstaking construction
- Room must be modified
- Projector alignment is tedious
- High maintenance cost
- Constant adjustments
- Projectors become unaligned over time
6Problem Instant Panoramic Display
- Generalized display environment
- Setup display surface approximately
- Align projectors casually
- Render with what you have
7Multi-Projector Goal
- Seamless imagery
- Immersive 3D images, moving user
- Casually setup projectors and surfaces
- Low maintenance
- Auto-Calibration
- Perceptually single logical projector
8Goal Seamless Display
- (a) Projector Registration
- Geometric alignment
- (b) Intensity blending
- Smooth transition
Projection 2
Projection 1
9Goal Seamless Display
- (a) Projector Registration
- Geometric alignment
- (b) Intensity blending
- Smooth transition
10Goal Seamless Display
- (a) Projector Registration
- Geometric alignment
- (b) Intensity blending
- Smooth transition
11Goal Seamless Display
- (a) Projector Registration
- Geometric alignment
- (b) Intensity blending
- Smooth transition
12Goal Seamless Display
- (a) Projector Registration
- Geometric alignment
- (b) Intensity blending
- Smooth transition
13Previous approachesMulti-projector setups
- Sweet spot
- Static user
- Moving User
- Head-tracked user
14Previous approaches Sweet spot (static user)
- Multiple projectors
- (well-defined overlaps)
- Flight sims,
- Panoram, Trimensions
- Raskar98
- Single wide-field-of-view projector
- OmniMax
- Alternate Realities
User
15Previous approaches Moving user
- CAVE
- Rigid setup
- Well-defined geometry
- Large support structure
Projector
User
16Previous approaches Moving user
- Office of the Future Raskar98
- Relaxed construction constraints
- Assumes precise display geometry recovery
- Demonstrated for a single projector
17Related work Image Mosaicing
- QuickTime VR
- Pair-wise mosaicing of photographs
- Generalized mosaicing
18Goal for this paper
- Seamless image
- Immersive 3D images, moving user
- Arbitrary projectors and surfaces
- Eliminate maintenance
- Self-calibration
- Perceptually single logical projector
19Basic Approach Panoramic Display
- Position projectors approximately
- Use video cameras to recover
- display geometry
- projector configuration
- Track user
- Render images of 3D scenes
- Warp and blend projected images to achieve
registration
20Outline
- Single projector
- Calibration, Display recovery, Rendering
- Multiple projector
- Registration, Blending
- New techniques
- Post rendering warp, mesh unification
21Video
22Outline
- Single projector
- Rendering, Calibration, Display recovery
- Multiple projector
- Registration, Blending
- New techniques
- Post rendering warp, mesh unification
232-pass Rendering
3D Display Surface Model
Projector
User
242-pass Rendering Project this warped image,
the viewer will see it correctly
3D Display Surface
Projector
(2) Render Model
(1) Desired Image
User
25Single projector display2-pass unknowns
- Arbitrary display surface
- Arbitrary projector location
- Need to determine these for 2 pass rendering
- How?
User
26Single projector displayCamera calibration
1. Calibrate stereo pair
Stereo camera pair
27Single projector displayDisplay surface
estimation
1. Calibrate stereo pair
2. Determine display surface
Stereo camera pair
28Single projector displayProjector estimation
1. Calibrate stereo pair
2. Determine display surface
3. Determine projector
Stereo camera pair
29Single projector displayProjector estimation
1. Calibrate stereo pair
2. Determine display surface
3. Determine projector
Display Surface
Stereo camera pair
Projector
30Single projector display Display Surface
Estimation
- Display surface is recovered by cameras (via
projector) - Surface mesh is created from the 3D points
- Allows us to calibrate the projector
- Approximate speed
- 720 surface point samples (1 minute)
- 14,000 surfaces point samples (15 minutes)
31Single projector displayCoordinate Frame
- Display surface and projector are in the
coordinate frame of the Calibration Pattern
32Multiple ProjectorsPanoramic display environments
- A single stereo-pair cannot see entire display
- Must use several stereo-pairs
- Each in its own coordinate frame
- Must register data into common coordinate frame
- Advantage
- We control the light
- Makes registration easier
33Multiple ProjectorsSurface mesh registration
Display Surface
34Multiple ProjectorsDisplay surfaces are in
different coordinate frames
3
Display Surfaces
2
1
35Multiple ProjectorsMust register into common
coordinate frame
3
Display Surfaces
2
1
36Multiple ProjectorsSurface meshes are registered
into the same coordinate frame
3
Display Surfaces
2
1
37Multiple ProjectorsRe-calibrate projectors so
they are registered
3
Display Surfaces
2
Re-calibrate the projectors so they
are registered into a common coordinate frame
1
38Multiple ProjectorsRe-calibrate projectors so
they are registered
3
Display Surfaces
2
With display surfaces and projectors registered
we can achieve geometric correctness . . .
1
39Multiple ProjectorsIntensity Blending
Need to attenuate intensities in the
overlapped region
40Intensity BlendingUse camera to find overlaps
Camera observes overlapped region
41Intensity BlendingAssigning intensity weights
(alpha)
Camera image plane
Assign intensity weights based on distance
between projector boundaries
42Intensity BlendingAssigning intensity weights
(alpha)
Camera image plane
Assign intensity weights based on distance
between projector boundaries
43Intensity BlendingAlpha-masks
Projector alpha-masks
44Achieving our goal Seamless display
- Display surface and projector registration
- Geometrically alignment imagery
- Intensity blending
- Smooth intensity transition between projectors
45Registration Issues
- Errors in estimation
- user location
- 3D display surface
- projector parameters
46Registration Issues
- Errors in estimation
- user location
- 3D display surface
- projector parameters
- Single projector, errors not visible
- Multi-projector overlap
- Geometric mis-registrations
- Gaps, Discontinuities
47Sources of errors
- Head-tracker for user location
- Display surface recovery
- Camera Calibration
- Radial Distortion
- Pattern Imperfection
- Feature detection
- Projector calibration
- Relies on display surface recovery
48Re-projection error
M
Projector framebuffer
m'
m
m'
m
P
P
P
49Sources of re-projection error
- Minimization residual
- Projector not pin-hole device
- Inaccuracies in estimated 3D pts on display
surface
50Effect of errors
- User location
- Display surface meshes
- difficult to register
- incorrect surface representation
- Projector parameters
- re-projection error
- incorrect projection matrix
51Geometric Error Compensation
- Objective
- Neighboring projectors use same display surface
representation - Each projection matrix maps 3D screen pts to 2D
pixels that illuminate them - Overcoming inaccuracies
- Geometric continuity everywhere for display
- Fidelity of projector model
52Outline
- Single projector
- Calibration, Display recovery, Rendering
- Multiple projector
- Registration, Blending
- New techniques
- Post rendering warp, mesh unification
53Geometric Error Compensation
- Objective
- Neighboring projectors use same display surface
representation - Surface Mesh Unification
- Each projection matrix maps 3D screen pts to 2D
pixels that illuminate them - Post Rendering Warp
54Surface mesh unification
- Merge multiple recovered meshes
- Create a single representation of display
surfaces - Rigid transformation creates near alignment
- Errors lead to discontinuities
- Use smooth 3D transition
55Surface mesh unification
Actual surface
P2
P1
56Surface mesh unification
57Surface mesh unification
58Surface mesh unification
59Surface Unification
New surface mesh
60Surface Unification
Original surface
New surface mesh
Both projectors use same (incorrect) surface
geometry
61Geometric Error Compensation
- Objective
- Neighboring projectors use same display surface
representation - Surface Mesh Unification
- Each projection matrix maps 3D screen pts to 2D
pixels that illuminate them - Post Rendering Warp
62Modified re-projection error
Actual Pt
M
M
Estimated Pt
P2
P1
63Modified re-projection error
M
M
P2
P2
P1
P1
64Modified re-projection error
M
M
P2
P2
P1
P1
65Modified re-projection error
M
M
m2
m2
m2
m2
m1
m1
P2
P2
m1
P1
P1
m1
66Effect of reprojection error
- Physically m1 and m2 create image at M
- In estimated model, m1 and m2 correspond
- Features of a virtual object gets rendered at
m1 and m2 - But, m1 and m2 are not likely to correspond
67Effect of Re-projection error
V
M
M
V
User
m2
m1
m2
m1
M
68Eliminating re-projection error
69Eliminating re-projection error
- We have
- m1 P1 M
- m2 P2 M
- We need a nonlinear projective function
- m1 F1 (M)
- m2 F2 (M)
70Eliminating re-projection error
- We have
- m1 P1 M
- m2 P2 M
- We need a nonlinear projective function
- m1 F1 (M)
- m2 F2 (M)
- Cannot be achieved using single projection matrix
- Lack of hardware support
71Post Rendering warp
- Goal
- m1 F1(M) m2 F2(M)
- (a) Render using estimated projection matrix
- m1 P1 M m2 P2 M
- (b) Warp resultant image
- m1 --gt m1 m2 --gt m2
72Post Rendering warp
- Goal
- m1 F1(M) m2 F2(M)
- (a) Render using estimated projection matrix
- m1 P1 M m2 P2 M
- (b) Warp resultant image
- m1 --gt m1 m2 --gt m2
- Expensive to specify per-pixel warp
- Use piecewise planar approximation
73Post Rendering warp (using texture mapping)
- Define grid of pixels in projector framebuffer
(m) - Project and record 3D surfaces pts (M)
- After surface unification (M),
- find reprojected points (m)
- Tesselate the framebuffer
- Load rendered image in texture memory
- Render vertices m with texture coordinates as m
74Post Rendering warp (using texture mapping)
Record 3D pts
Project pixel grid
Find reprojected pixels
75Post Rendering warp (using texture mapping)
Compute required post-rendering warp
76Summary of TechniquesBasic Approach
- At each projector
- calibrate camera pair
- project dots and find 3D display surface
- (find surface illuminated by neighboring
projectors) - find projector parameters
- Find intensity blending functions
- Register surface meshes (near alignment)
77Summary of TechniquesCompensating the errors
- Unify meshes using smooth 3D transition
- Use camera overlap as transition region
- Re-compute all best-fit projector matrices
- Find projective function
- piecewise planar approximation using multiple
triangles - Specify all rendering tasks using a display list
78Summary of Techniques
- At run time, at each projector
- Read current user location
- Render virtual object using two-pass rendering
- Post-rendering 2D warp
- Intensity blending
79Restricted cases
- Static User (Sweet spot)
- VSMM98
- Planar display surface
- Tech report
- Available at project web page
80Features
- Task of each projector is independent
- 2-pass rendering, post-rendering warp is
pre-defined - Can be specified using display list
- Traditional hardware
81Implementation
- 5 Single lens 1024x768 projectors
- Multiple 640x480 cameras
- SGI InfiniteReality2 for each projector
- Matlab for calibration, OpenGL for rendering
82Issues
- Resampling artifacts due to limited resolution
- Color matching
- Viewer-dependent parameters
- Limited depth-of-field
83Acknowledgements
- Office of the Future group
- NSF STC
- NTII (Advanced Network Systems)
- Intel
84Summary
- Multiple projectors to create panoramic imagery
- Geometric registration is major challenge
- Video cameras to create 3D representation
- Generalized display configuration
- www.cs.unc.edu/stc