3D TeleCollaboration

1

• 3D Tele-Collaboration
• over Internet2
• Herman Towles, UNC-CH
• representing members of the
• National Tele-Immersion Initiative (NTII)
• ITP 2002
• Juan-les-Pins, France
• 06 December 2002

2
NTII Collaborators Co-authors

• University of North Carolina at Chapel Hill
• Wei-Chao Chen, Ruigang Yang, Sang-Uok Kum, and
  Henry Fuchs
• University of Pennsylvania
• Nikhil Kelshikar, Jane Mulligan, and Kostas
  Daniilidis
• Brown University
• Loring Holden, Bob Zeleznik, and Andy Van Dam
• Advanced Network Services
• Amela Sadagic and Jaron Lanier

3
Clear Motivation to Provide

• Higher Resolution
• Larger, more immersive Field-of-View
• Participants at Accurate Geometric Scale
• Eye Contact
• Spatialized Audio (Group settings)
• More Natural Human-Computer Interfaces

4
Related Work

• Improved Resolution FOV
• Access Grid Childers et al., 2000
• Commerical, multi-channel extensions of 1-camera
  to 1-display
• Gaze-Awareness
• MONJUnoCHIE System Aoki et al., 1998
• Blue-C Project - Kunz and Spagno, 2001-2002
• VIRTUE Project Cooke, Kauff, Schreer et al.,
  2000-2002
• 3D Reconstruction/New Novel Views
• CMUs Virtualized Reality Project Narayanan,
  Kanade, 1998
• Visual Hull Methods Matusik, McMillan et al,
  2000
• VIRTUE Project Cooke, Kauff, Schreer et al.,
  2000-2002
• Human Computer Interfaces
• T-I Data Exploration (TIDE) Leigh, DeFanti et
  al., 1999
• VisualGlove Project - Constanzo, Iannizzotto,
  2002

5
XTP Xtreme Tele-Presence
UNC Office of the Future Andrei State 1998
6
Research Snapshots
7
Presentation Outline

• Motivation and Related Work
• NTII Tele-Collaboration Testbed
• Acquisition and 3D Reconstruction
• Collaborative Graphics User Interfaces
• Rendering Display
• Network
• Results
• Future Challenges

8
Scene Acquisition Reconstruction

• Foreground Real-Time Stereo Algorithm
• Frame Rate 2-3 fps (550MHz Quad-CPU) -
  REAL-TIME!
• Volume 1 cubic meter
• Resolution 320x240 (15K-25K foreground points)
• Background Scanning Laser Rangefinder
• Frame Rate 1 frame in 20-30 minutes - OFFLINE!
• Volume Room-size
• Resolution More data than you can handle!
• Composite
• Live Foreground Static Background

9
Real-Time Foreground Acquisition

• Trinocular Stereo Reconstruction Algorithm
• After background segmentation, find
  corresponding pixels in each image using MNCC
  method
• 3D ray intersection yields pixel depth
• Median filter the disparity map to reduce
  outliers
• Produce 320x240 Depth Maps (1/z, R,G,B)

Images courtesy of UPenn GRASP Lab
10
UNC Acquisition Array
Five Dell 6350 Quad-Processor Servers
Seven Sony Digital 1394 Cameras Five Trinocular
Views
11
Stereo Processing Sequence
Camera Views
Disparity Maps
3 Views of Combined Point Clouds
Images courtesy of UPenn GRASP Lab
12
Collaborative Graphics User I/F
13
Shared 3D Objects

• Scene Graph Sharing
• Distributed, Common Scene Graph Dataset
• Local Changes, Shared Automatically with Remote
  Nodes
• Object Manipulation with 2D 3D Pointers
• 3D Virtual Laser Pointing Device
• Embedded magnetic tracker
• Laser beam rendered as part of Scene Graph
• One event/behavior button

14
Rendering System Overview
15
3D Stereo Display

• Passive Stereo Circular Polarization
• Custom Filters on Projectors
• Lightweight Glasses
• Silvered Display Surface
• Front Projection
• Usable in any office/room
• Ceiling-mounted Configurations
• Two Projector Stereo
• 100 Duty Cycle
• Brighter No flicker
• Permits multi-PC Rendering

16
View-Dependent Rendering

• HiBall? 6DOF Tracker
• 3D Position Orientation
• Accurate, Low latency noise
• Headband-mounted Sensor
• HiBall to Eyeball Calibration
• PC Network Server

17
Rendering Configurations

• One PC Configuration (Linux)
• Dual-channel NVIDIA graphics
• Three PC Configuration (Linux)
• Separate left right-eye rendering PCs w/NVIDIA
  graphics
• One PC used as network interface, multicasts
  depth map stream to rendering PCs
• Performance 933MHz PCs GeForce2
• Interactive Display Rates of 25-100fps
• Asynchronous updates of 3D Reconstruction (2-3Hz)
  Scene Graph (20Hz)
• Newest Rendering Configuration 10-20X
• 2.4GHz, GeForce4, Multi-Threaded, VAR Arrays

18
Network Considerations

• All Tests over Internet2
• Data Rates of 20-75 Mbps from Armonk, NY and
  Philadelphia into Chapel Hill
• 320 x 240 Resolution
• Up to 5 Reconstruction Views per site
• Frame Rates 2-3 fps
• TCP/IP
• Latency of 2-3 seconds typical

19
Presentation Outline

• Motivation and Related Work
• NTII Tele-Collaboration Testbed
• Acquisition and 3D Reconstruction
• Collaborative Graphics User Interfaces
• Rendering Display
• Network
• Results
• Future Challenges

20
Results

• Roll the Tape

21
Summary

• One-on-One 3D Tele-Immersion Testbed
• Life-size, view-dependent, passive stereo display
• Interact with shared 3D Objects using a virtual
  laser pointer
• Half-Duplex Operation today
• Operation over Internet2 between Chapel Hill,
  Philadelphia and Armonk
• Audio over H.323 or POTS

22
Future Challenges

• Improved 3D Reconstruction Quality
• Larger Working Volume, Faster Frame Rates 60
  cameras
• Fewer Reconstruction Errors (using structured
  light and adaptive correlation kernels)
• Reduce System Latency and Susceptibility to
  Network Congestion
• Pipelined architecture
• Shunt Protocol (between TCP/UDP and IP layers)
  that allows multiple flows to do coordinated
  congestion control
• Full Duplex Operation
• Unobtrusive Operation
• No headmounts, No eyeglasses!

23
Thank You

• Research funded by
• Advanced Network and Services, Inc. and
• National Science Foundation (USA)

24
UPenn Acquisition Array
Fifteen Sony Digital 1394 Cameras Five
Trinocular Views
25
System Overview
26
Past Experiments
With Collaboration
w/o Collaboration