Interaction and Distribution in Virtual Environments

1
Interaction and Distribution in Virtual
Environments

• Emmanuel Frécon - emmanuel_at_sics.se
• Swedish Institute of Computer Science

2
Abstract

• Multi-user virtual environment (VE) systems
  offer completely new ways to interact between
  users and computer applications by enabling 3D
  presentation and interaction. We believe that the
  man-machine interfaces of the future will be
  based on this new interaction paradigm. This talk
  will outline some technological issues concerning
  interaction in multi-user VE systems, as well as
  describing our research in this area.

3
Aim of talk

• This is not a visionary talk
• Real systems and applications
• It is not a computer graphics talk
• New ways of interaction and communication
• Research and concepts
• Man-machine interface of the future ?

4
Contents

• Introduction
• What is Multi-user Virtual Environments?
• Interaction Techniques
• Configurations and models for interaction in VEs.
• Application Examples
• DIVE applications and demonstrations
• Systems and Architecure
• Distributed VE systems, VRML, Dive
• Objects, Scripts, Avatars
• Communication and Networking
• Wide-area issues Decreasing bandwidth use

5
Introduction

• Multi-user virtual environments

6
Multi-user virtual environments

• Collaborative Virtual Environments - CVEs.
• Three dimensions - 3D
• presentation
• interaction
• Multiple participants
• Shared environments
• Highly interactive

7
Distributed VE systems - 1

• Large scale distributed simulation
• DIS-based systems - http//www.sisostds.org/doclib
  /doclib.cfm?SISO_CID_16
• NPSNET - http//www.npsnet.org/
• Military use

8
Distributed VE systems - 2

• CSCW oriented systems for group interaction
• Dive - http//www.sics.se/dive/
• Massive CVE - http//www.crg.cs.nott.ac.uk/resea
  rch/systems/
• Spline - http//www.merl.com/projects/spline/index
  .html

Massive2
Spline
9
Distributed VE systems - 3

• Games, MUDs and on-line systems
• Quake, Doom, - http//www.idsoftware.com/
• Alpha World - http//www.activeworlds.com/
• Blaxxun - http//www.blaxxun.com/
• More Info
• Contact Consortium - http//www.ccon.org/

Quake
Active Worlds
Blaxxun
10
SICS research

• Interaction models
• Applications
• Distribution using group communication
• Not focused on 3D rendering
• The Dive platform
• A distributed VE system
• First version in 1991
• http//www.sics.se/dive/

11
Enabling techniques

• Internet growth
• Multicast IP Mbone
• WWW
• VRML
• VRML 1.0 Static 3D scene description language
• VRML 2.0 Interaction, sensors, events, scripts
• OpenGL API
• Platforms
• Game-oriented cards for PCs offer performance
  that was available on high-end workstations a few
  years ago
• High-end workstations SGI Onyx

12
What is special with VEs?

• Compared with traditional desktop metaphor
• WIMP (window, icon, mouse, pointer)
• The world is 3D
• Real-world metaphors
• Spatial properties are central
• Exploit distances and room
• Attenuation
• Audio
• Space-based audio models
• Body motion, gestures

13
3D Objects

• Complex rendering.
• Selecting and focusing on 3D objects
• Moving and modifying 3D objects.
• Active objects
• behaviour defined by scripts
• triggered by events

14
A shared multi-user environment

• User are actors represented by avatars
• Viewpoint follows avatar
• Objects may be interacted with by any actor
• Objects are shared and active
• Actors may change worlds
• 3D audio
• conference
• object-based

15
Navigation in 3D

• Navigation How do you move the viewpoint (your
  avatar) in general 3D space?
• General motion in 3D
• translation x, y, z
• yaw, pitch roll
• Easy to get lost
• Restrained motion
• x-z plane
• Reference plane
• Vehicles

16
Partitioning of 3D worlds

• Many actors in a 3D coordinate system
• But they need not be aware of each other
• Spatial areas can be used to partition worlds and
  traffic
• Disjoint worlds

17
What do we want to do?

• Use spatial proximity and real-world metaphors as
  a new direct interaction paradigm
• The interface should be seamless
• Common gestures and body movements
• Pointing with hand and eye-gaze
• No wires or clumsy devices
• Sensors in environment
• Smart cameras watching user
• Speech recognition
• Smart objects, artificial agents
• Reactive rooms and active workplaces

18
Interaction Techniques

• in Multi-user VE systems consisting of
• a shared 3D environment

19
Different Setups

• Non-immersive
• Mouse/Keyboard/Screen
• Immersive
• Embedded inside the environment
• Personal equipment
• Semi-immersive with special rooms Cave
• Embedded without personal presentation equipment
• Mixed setup
• Using 3D interaction device with workstation

20
Non-immersive techniques

• How do you use WIMP to interact and navigate in
  3D?
• Mouse,
• Keyboard,
• Screen,
• Microphone
• Headset
• Stereo glasses

21
Mouse and Keyboard

• Keyboard navigation and interaction
• Doom navigation - widespread use
• Arrow keys and modes (plane pitch roll)
• Complex for 3D
• Mouse interaction
• DIVE uses interaction icons
• Vehicles
• walking motion
• flying
• helicopter
• jumping

22
Immersive techniques

• Immersion means to be surrounded by the synthetic
  environment
• Personal, often cumbersome, equipment
• Isolation and weight problems
• Head-mounted display
• Dataglove
• 3D Mouse

HMD
BOOM Binocular Omni Orientation Monitor
Data glove
wand
23
Semi-immersive techniques

• Special purpose room
• Large screens
• Cave
• No personal equipment
• Group use
• Immersion
• High cost

SICS grotto
ImmersaDesk
CAVETM
24
SICS grotto experiments

• Casual interaction without WIMP
• Magnetic bird control pointer on screen
• Speech recognition for command control
• Camera tracking and camera vision
• gesture recognition
• navigation and interaction

25
Abstract spatial models for interaction

• Area of interest (AOI)
• Aura
• Spatial area enabling interaction
• Awareness, Focus and Nimbus
• Focus - Degree of interest a user brings to bear
  on another.
• Nimbus - Degree of attention a user pays to
  another

26
Application Examples

• DIVE system experiments
• illustrating several aspects of
• interaction in virtual spaces

27
Application project examples

• Distributed conferencing and education
• Distributed engineering
• Social and tourist applications
• Data visualization and Web browsing
• Command, control and telepresence
• Telemedicine
• Help for persons with disabilities
• Military applications

28
Virtual conferencing

• Travel is expensive
• Personal face-to-face contact important
• Global participation via Internet/Mbone
• Interaction using normal daily metaphors
• Magic tools
• Shared documents
• Powerful whiteboard

29
Conferencing examples

• Distributed conferencing using tools
• Immersive conferencing
• 3D Audio
• World-wide-web interface
• Shared documents

30
Distributed Engineering

• Distributed Meetings with teams
• Design, Construction, Production
• Distributed versioning
• Visualising models
• animations crash tests, FEM

with LuTH, FramKom, ...
31
Social tourist applications

• Integrated tourist information
• Cities, museums, transports and services
• Virtual agents provide help

European ACTS project
32
Browsing information spaces

• Assign spatial and visual properties to data
• Enhance legibility
• Natural collaboration in browsing and selecting

VR-VIBE
Library Demo.
WebPath
33
Command and control

• Controlling robots
• Virtual models of real environments
• Mapping images to virtual objects
• Monitoring systems
• Future possibilites for disabled?
• Telemedicine?

34
Mapping the real world
What the robot sees...
is mapped to surfaces in the model.
35
Military applications

• Radar control and surveillance
• Command and control room
• Battle field simulation

36
Speech recognition and agents

• VEs benefits from speech as a natural mode of
  communication
• Speech recognization systems requires a context
• A VE defines a context - a world to reason about.
• Talking agents
• Restricted set of capabilities
• and understanding
• Speech controlled
• Example the pump scales objects

37
Other Domains

• Architecture
• Mechanical Design
• Psychotherapy
• Art
• Medicine
• Entertainment
• Scientific Visualisation

38
Other Domains
Architecture
Mechanical Design
Scientific Visualisation
Psychotherapy
Art
Education
Medicine
Entertainment
39
DIVE

• An example multi-user virtual environment system

40
DIVE

• Distributed Interactive Virtual Environment
• In development at SICS since 1991
• General-purpose multi-user VR system
• Tuned for rapid prototyping of highly-interactive
  applications
• Integration of wide range of media
• 3D spaces
• Video, audio
• WWW / Internet
• Wide range of interaction configurations
• Desktop, Multi-Screen Displays, HMDs, etc.
• Binaries freely available

41
DIVE concepts
Programming and Conceptual Model
application 2

• DIVE entities define the environment
• Views, transformations, lights, worlds
• Entities are replicated to all processes
• Everything is distributed by multicasting
• Worlds define a 3D environment
• Disjoint geometric universes
• Connected by gateways
• Users are represented by avatars
• Move freely within worlds and between worlds
• Changes in the environment cause events
• User interaction
• Collisions, object signaling
• Events trigger behaviour in active objects
• DIVE/TCL Programming Language

application 4
application 1
application 5
World 1
World 2
application 3
Process 4
Process 2
Process 3
Host 1
Process 1
Host 3
Process 5
Network
Host 2
Process 6
Implementation
42
Avatars

• Representation of users
• Rendering viewpoint follows avatar eyes
• 3D audio spatialized from ears
• Animation of limbs and motion
• Vehicles, sensors
• Avatars can be changed dynamically
• Any object can be avatar
• Usually simple representations
• Complex human models are expensive but need to be
  extremely good.

43
Avatar examples
44
File Example
object material RED_NEON_Mmaterial
BLUE_NEON_Mview material_index 0 RBOX v
-2 -2 -2 v 0 0 0view material_index 1
RBOX v 0 0 0 v 2 2 2object material
GREEN_M gouraud on translation v 0 3 0
view SPHERE 1 1 1
45
Active objects and scripts

• Active objects have behaviour scripts
• Dive Tcl/Tk
• Scripts are triggered by events in the system
• Scripts are replicated and evaluated on every
  node
• Scripts are powerful
• can effect every aspect of the system

Dive DB
Events
Actions
Actions
Events
46
Example Dive script

• Tcl/Tk
• begin.tcl
• proc bounce
• move 0 1 0
• register interaction_signal bounce
• end.tcl

47
Entity class hierarchy
Entity
Node
Actor
Light
View
...
...
World
Object
Point Light
Spot Light
Direct. Light
Box
Sphere
Multi Poly
Lod
48
Entity hierarchy - Example
49
Communication Channels

• VE data is sent using reliable multicast (SRM)
• Positions and geometry
• Events
• Audio and video use unreliable multicast (IP
  multicast)

50
Communication and Networking

• Concepts and techniques for distribution in
  Multi-user Virtual Environments

51
Distributed applications

• Application distributed over a network
• Users, clients, servers, resources
• Multicast one-to-many
• Internet
• IP Multicast
• Multicast Backbone (Mbone)
• Bandwidth and latency requirements

52
Client-Server vs. Peer-to-Peer
Client-Server
Peer-to-Peer
peer
peer
update
peer
peer
peer

• Server manages data
• Easy consistency model
• Clients may cache data

• Peers replicate data
• Updates may be multicast
• Low latency

53
IP Multicast routing
LAN
Sender
Router
LAN
Router
Target group
Multicast packets are forwarded from sender to
group of receivers.
54
Reliable Multicast

• Reliability with multiple receivers is more
  intricate than point-to-point
• Ordering source, causal, total.
• Atomicity all or none.
• Group synchronization
• Failure semantics
• Positive feedback protocol
• Ack implosion Every receiver sends acks.
• Negative acknowledge protocol
• Last packet problem Detect loss of packets.
• Sender-oriented vs Receiver-oriented multicast

55
Reliable Multicast - Ack implosion
B
C
A
acks
D
A receives one ack from each receiver
56
Reliable Multicast - Last packet problem
xi!
Point of Failure

• If A does not send another packet, how do C and
  D know of the failure?

57
SRM Scalable Reliable Multicast

• Updates (!) and nacks (?) are multicast to
  suppress duplicates.
• Host closest to point of failure send nack.
• Host closest to responder send update.
• Anyone should be able to supply data data and
  sender identity distinct (object-based
  communication)
• Objects have sequence numbers.
• Round-trip estimation to determine closest host.
• Delay request and response in proportion to RTT.
• To fix case of equal RTT, a random time factor is
  introduced.

58
Application dependent issues

• How to detect a packet loss?
• Sequence number gap.
• Periodic pings with current sequence number.
• Application must be able to send updates on
  demand.
• Advanced (two-way) API.
• Object-based model
• Application level framing (ALF) application and
  protocol uses same objects.
• Granularity.
• Identifiers.
• Who can modify objects?

59
SRM example - 1
xi!
Point of Failure

• When C and D are notified of loss of xi!,
  request timers are set.

60
SRM Example - 2
xi
xi?
xi

• D does not request xi, since xi? from C arrives
  before D timeouts.
• A and B set update timers.
• Failure xi? is periodically resent.

61
SRM Example - 3
xi
xi!
xi

• A does not send update, since xi! from B arrives
  before A timeouts.

62
How to decrease bandwidth use.
Bandwidth requirements per source

• Audio 10kbit/s-1Mbit/s
• Conference audio usually only one source
• Video 1Mbit/s - 20Mbit/s per source
• VE data small packets (1kbit)
• But frequently by many sources.

63
How to decrease bandwidth use - VE

• Suppose n hosts with f position updates per sec
• Example each update 1kbit, n 10, f 20 Hz
• Decrease f
• Dead reckoning, kinetic models
• Decrease receiving group size n
• Spatial partitioning.

64
Dead reckoning, Kinetic models

• Dead reckoning extrapolates from previous values
• Send updates only when model deviates from actual
  data.
• Kinetic models.
• Define complex motion with kinetic equations.
• Velocity, acceleration, angular velocity,
• Scripts defining motion.

65
Spatial partitioning

• Send data only to receivers that need it.
• Assumption spatial proximity decides interest.
• Add several multicast groups and partition
  traffic in network.
• Techniques
• Geographical areas.
• Object-based proximity.

66
Large scale distributed simulation

• Geographical area modelled by hexagons.
• NPSNET
• Each hexagon, one multicast group.
• Actor joins current and all adjacent groups.
• When actor changes hexagon, joins new group.

67
Space-based partitioning

• More general model based on general connected
  spaces.
• SPLINE, DIVE 2.
• Each space, one multicast group.
• Actor joins current and all neighbouring groups
• When actor changes space, joins new group.

member of blue and yellow groups
68
Object-based partitioning

• Object based partitioning with (adjustable)
  auras.
• Massive, light-weight groups in DIVE.
• Each actor owns multicast group.
• Actor joins adjacent actor groups.
• Aura extendors actor adjusts aura to current
  space.

A and B mutually join groups.
69
Proposal

• Combined object-based and spatial partitioning.
• Light-weight groups within crowded world-spaces.

A
B
C
70
Audio Communication

• Conference and object-based 3D audio in Virtual
  Environments

71
Audio Architecture
Acquisition/Read
Playback
Spatialisation
Source Processing
Sending Peers
Receiving Peers
Ordering
Encoding
Decoding
Coding
Send
Receive
Distribution
Network
72
Audio Components

• Acquisition user-defined quality
• Read Sun .au files, AIFF, WAV,
• Encoding methods supported
• Linear 8, 16, 24, 32
• u-law and A-law (compression 12)
• GSM (compression 329)
• DVI ADPCM (compression 14)
• More in the future LPC, MPEG,
• Distribution on a separate group and using
  unreliable multicast. Packets reordered on
  arrival.
• Audio streams associated to a source object,
  location used when spatialising. One ring buffer
  for each active source.
• Spatialisation is two-dimensional, and achieved
  using a simple set of filters based on
  inter-aural distance, distance from source, etc.