A Distributed Cooperative Framework for Continuous MultiProjector Pose Estimation

1
A Distributed Cooperative Framework for
Continuous Multi-Projector Pose Estimation
Tyler Johnson, Greg Welch, Henry Fuchs, Eric La
Force, and Herman Towles Department of Computer
Science University of North Carolina at Chapel
Hill
IEEE VR 2009 - March 16, 2009
2
Funding

• ONR Behavior Analysis and Synthesis for
  Intelligent Training (BASE-IT), Dr. Roy
  Stripling, Program Manager
• ONR Virtual Technologies and Environments
  Program (VIRTE), CDR Dylan Schmorrow, Program
  Manager
• ONR-NAVAIR Deployable Intelligent Projection
  Systems for Training, SBIR contract with
  Renaissance Sciences Corporation
• IARPA Mockup Future Analyst Workspace (A-Desk),
  Dr. Jeff Morrison, Program Manager
• NSF Integrated Projector-Camera Modules for the
  Capture and Creation of Wide-Area Immersive
  Experiences, CRIIAD grant

3
Adaptive Multi-Projector Displays
An Intelligent Projector Unit (IPU)
4
Challenges

• Geometric
• Compensating for display surface shape
• Co-registration of projection images
• Photometric
• Intensity blending in image overlaps
• Matching colors between projectors

No Compensation
Compensation
5
Geometric Calibration
Before Display Use
During Display Use
Render Imagery
Geometric Image Correction
Project Structured Light
Calibrate Projectors
Continuous Calibration
Estimate Display Surface
6
Continuous Calibration
A Calibrated Two-Projector Display
Projectors Moved or Bumped
A Recalibrated Two-Projector Display
7
Cooperative Calibration

• We propose a distributed, Kalman filter-based
  approach to continuous calibration where
  intelligent projector units interact as peers to
  cooperatively estimate the poses (orientation
  position) of all projectors during actual display
  use

8
Related Work

• Continuous Calibration
• Active (Calibration Patterns)
• Imperceptible Structured Light Cotting04,05
• Passive (Application Imagery)
• Continuous Display Surface Estimation
  YangWelch01
• Single Projector Pose EstimationJohnsonFuchs07
• Multi-Projector Pose Estimation Zhou08
• Hybrid
• Automatic switch from passive to active
  Zollmann06
• Distributed Upfront Calibration
• Bhasker06

9
Contributions

• Our Kalman filter-based distributed cooperative
  framework provides
• Continuous pose estimation for multiple
  projectors
• Compatible with both active and passive feature
  collection
• All projectors may move simultaneously
• Temporal filtering
• Fault tolerance scalability

10
Cooperative Calibration

• Peer-to-Peer based, Single Program, Multiple Data
• Each IPU considers itself to be the local IPU
• Other IPUs are considered remote IPUs
• Each IPU is responsible for calculating its own
  pose using local and remote correspondences
• Assumptions
• The internal calibration of each IPU is fixed and
  known
• The geometry of the display surface is static and
  known
• Projectors remain mostly stationary, however they
  may drift over time or occasionally be moved by
  the user

11
Local Correspondences

• Measured for each IPU between its primary and
  secondary cameras
• Provides an estimate of pose

12
Remote Correspondences

• Measured between the primary camera of the local
  IPU and a remote IPU
• Remote IPU acts as a reference in estimating pose
  of local IPU

13
Collection of Measurements
Display Surface
14
Kalman Filter
Measurement Function
Display Surface Model
15
Kalman Filter
Pose of
Pose of
Pose of
Error Covariance
State
Pose of
16
Filter Update
Predict IPUs remain stationary
Add additional uncertainty
Correct state based on residual
17
Distributed Operation

• Each IPU has local access to
• Its own intrinsic calibration pose
• Its own camera images
• Display surface model
• Kalman filter update requires remote access to
• Intrinsic calibration poses of remote IPUs
• Error process noise covariance of remote IPUs
• Images from primary cameras of remote IPUs
  captured at time

18
Distributed Operation

• Request/response mechanism for exchanging camera
  images, pose information etc

19
Results
Before Movement
During Movement
After Movement
?
x
1240
2.85
1160
2.7
y
?
-95
-0.3
mm
rad
-120
-0.7
f
z
0.35
2640
2560
0.15
20
Video

• Distributed Cooperative Pose Estimation in a
  Two-IPU display

21
Discussion

• Observability Drift
• Surface geometry may not fully constrain pose
• Possible to anchor solution in unobservable
  directions
• Cooperative Estimation
• Not required for a working system
• Ensures imagery is registered between projectors,
  especially when pose may be unobservable

22
Future Work

• Continuous calibration of display surface
• Dynamic projector refocusing
• Dynamic photometric blending
• Improve scalability

23
Future Applications
24
In Conclusion