Teleimmersion , the next generation of communication and collaboration technologies

1
Tele-immersion , the next generation of
communication and collaboration technologies

• Ruzena Bajcsy
• EECS Department, University of California Berkeley

2
Collaborators

• Oliver Krylos, UC Davis
• Lisa Wymore ,UC Berkeley
• Klara Nahrstedt, UIUC
• Renata Sheppard UIUC

3
Other Collaborators

• Edgar Lobaton
• Ram Vasudevan
• Gregorij Kurillo
• Tracy Wang
• Peggy Hackney
• Sumitra Ganesh
• Allen Yang
• Philip Krylyos

4
Acknowledgment

• National Science Foundation
• HP Research Labs
• CITRIS
• Mellon Foundation

5
Overview

• Recent accomplishments in our tele-immersive
  research and system
• New reconstruction algorithm
• New Integration algorithm
• Dealing with occlusions
• New applications

6
Overview The Teleimmersive Environment lab set up

• Virtual dance space

Berkeley
Illinois
In virtual cyberspace
7
TI System Overview

• Goal Collaboration between geographically
  distributed users.
• Models are constructed locally and interact in a
  virtual environment.

8
Tele-Immersive features

• The cameras must be geometrically calibrated
• Both for internal and external parameters.
• For geographical connectivity, the assumption is
  that both labs have the visual capturing
  capabilities.
• The next step is to combine the two or more sites
  into a coherent Virtual world . This requires
  even further geometric calibration of the two
  distant worlds.

9
Viewing the Joint world

• All sites have the common 3D joint integrated
  world and then by simple graphical manipulation
  of rendering , each site can view the joint
  world.
• The bottom line is Calibration,Calibration

10
TI System Overview

• Main Challenges in the System
• Calibration
• Real-Time 3D Reconstruction
• Communication
• Visualization

11
Computational issues

• REAL TIME Challenges
• Real time 3D reconstruction
• Real time rendering
• Real time interaction with the Virtual
  Environment
• Networking, delays and compression
• Understanding of Human Dynamics

12
How to Represent Data?
13
Desired Properties of Representation

• Ease of Manipulation
• We want to be able to apply filters with low
  computational cost
• We would like a representation which is somewhat
  driven by the way how we will be performing
  stereo matching in order to reduce computation.
• Compression
• We want the data to be as small as possible.
• Wavelets present a good alternative while
  allowing for a multi-scale representation.
• Visualization
• Ultimately, we want to render our representation
  in 3D space.
• A common approach is to consider a triangulation
  of an object to be displayed.

14
The Representation

• We choose to come represent our data by
  triangulating our domain using a triangular
  wavelet structure. This particular triangulation
  scheme is know as Maubachs bisection scheme.
• We also maintain the triangulation conforming
  (i.e. no hanging nodes) as to allow for ease on
  interpolation in the domain.

15
Sample Triangulation
16
Disparity Computation
Original Image 320 x 240 (76 800
pixels) Points for Matching 300 Triangles
in Rep. 2000
17
Results Accomplished

• By considering the processing, communication,
  and visualization aspects of our system at hand
  we choose an appropriate triangular wavelet-based
  representation.
• Our representation has shown to decrease the
  number of operations required for disparity
  computation dramatically. However, our
  implementation still needs to be optimized.
• Compression gains from using representation are
  promising but still need to be verified.

18
TI System Stereo Reconstruction

• Disparity values are computed from multiple
  views.
• Meshes are projected into 3D and texture is
  mapped.

19
Occlusion Detection

• Occlusion detect whenever topological constraints
  are violated.
• Formulation allows for detections between moving
  objects without any correspondence or tracking.
• Applications
• Background Subtraction
• Motion Segmentation

20
Representational issues

• Given all the available data there are several
  ways of considering representations of the human
  body motion
• 1. Consider Skeleton of the Human body , the
• Angles of joints and their velocity and the
  transformations over time from pose to pose
• Using Twist representation

21
Data Compression
22
Skeletonization from the data
23
Networking Problems

• Current problems
• Each camera has to send data to multiple
  renderers more traffic
• Reliability of connection
• Reliability of transfer speed

24
Network Configuration
Internet (TCP/IP)
100Mbit / 1 GBit
25
New Network Configuration
Internet (TCP/IP)
100Mbit / 1 GBit
26
(No Transcript)
27
Rendering

• 3D graphic representation of captured data
• OpenGL-based application
• Renderer is a server, cameras are clients
• Real-time (10FPS) rendering of 75,000 3D points
• Different types of rendering
• Quads
• Splatting

28
Rendering
29
Why bother? What is the motivation?

• Fundamentally there are two
• Scientific challenges
• Applications, service to the world

30
Scientific challenges

• REAL TIME Large data processing
• REAL TIME Large data rendering
• REAL TIME Large data transmission, implies data
  compression.
• Finally, Understanding peoples activities and
  their interaction ( This is the Signal to
  Symbol problem ).

31
Applications are many

• First and foremost , this technology enables
  geographically distributed people to INTERACT and
  COMMUNICATE is if they would be in the same
  physical space.
• Examples can be
• In Health care ( Monitoring rehab patients or
  nay other patients where one needs to physically
  interact with the subject)

32
(No Transcript)
33
Applications cont.,

• Scientific Collaborations where distributed
  scientist need to discuss 3D data
• (examples geophysicist, but also archeologists
  and their like)
• Designers in aerospace , automobile industries
  where again they need to explore changes in CAD
  Models

34
Applications, cont.

• Teaching and training where you wish to teach
  either by example (Taichi, or dance) or by
  demonstration of some physical phenomenon (take
  physics , chemistry or biology classes).
• ARTS where distributed artists can explore
  different choreographies and training.

35
Recognition of Human activities

• Recognition capabilities of human activity is of
  great interest as a fundamental problem of
  understanding how to assign labels to dynamic
  observations. Surveillance, to prevent crime, but
  also monitoring elderly, disabled and children.

36
Application in learning , Control

• How do you transfer the human movement
  capabilities to a machine?
• This is closely related to representation of
  human movement and how do we use it.
• Applications are many automated car driver
• Pilot, helper at home, etc.

37
Summary

• We have presented Tele-immersive technology
• Its development, technical challenge, the science
  that had to go with the development,
• And the possible applications.
• We have ways to go and there are many unsolved
  problems
• Science Real time data compression
  Understanding Human activities
• Technology Dealing with multiple sites.