WPI HIVE: Introduction to Virtual Reality Interesting Issues, Open Problems

1
WPI HIVEIntroduction to Virtual
RealityInteresting Issues, Open Problems

• Prof. Robert W. Lindeman
• Worcester Polytechnic Institute
• Department of Computer Science
• gogo_at_wpi.edu

2
Overview

• Administrative Stuff
• What is Virtual Reality?
• Why study Interaction in VR?
• Open Problems

3
Motivation

• Much excitement (and hype) about how VR was going
  to change things
• VR has not made inroads into everyday life
• Lagging technology
• Lack of understanding of usability issues
• Lack of killer app
• Still remains mainly in research labs
• Video games show great promise
• Training scenarios - surgery, military, therapy
• Long-Term Goal
• Make VR more usable

4
Nothing New?SENSORAMA by Morton Heilig (1960)
5
Background

• VR defined
• Fooling the senses into believing they are
  experiencing something they are not actually
  experiencing
• Virtual reality systems consist of
• Graphical/audio/haptic/... rendering
• Content
• Tracking of people and objects
• Collision detection
• Interaction techniques
• Optional, but common
• Networking
• Autonomous agents

6
Keys to Success

• High fidelity (or realism)
• Graphics, audio, haptics, behaviors, etc.
• Low latency
• Tracking
• Collision detection
• Rendering
• Networking
• Ease of use
• Low cumber for users
• Easy integration for programmers
• Compelling Content

7
The Senses

• See (Visual Sense)
• Visuals are excellent!
• Hear (Aural Sense)
• Spatialized audio is very good!
• Smell (Olfactory Sense)
• Very hard! Too many types of receptors.
• Touch (Haptic Sense)
• Application specific and cumbersome
• Taste (Gustatory Sense)
• We know the base tastes, but that is it!

8
See Head-Mounted Displays
9
See Projection-Based Environments
10
See Projection-Based Environments (cont.)
11
Hear Sound in VR

• Display techniques
• Multi-speaker output (sound cube)
• Headphones
• Waveform filtering
• Simple balance volume control
• Head-Related Transfer Functions
• Software "Standards"
• OpenAL
• A3D from Aureal (RIP!)
• VRSonic.com

12
Smell Olfactory Sense

• Two main problems
• Scent generation
• Tens of thousands of receptor types
• Scent delivery
• Easier problem

13
Smell Air Cannon (Yanagida, 2004)
14
Touch Haptic Feedback in VR

• Tactile Surface properties
• Most densely populated area is the fingertip
  (okay, it's the tongue)
• Kinesthetic Muscles, Tendons, etc.
• Also known as proprioception

15
Vibrotactile Feedback Projects
GW TactaBoard and TactaVest
Navy TSAS Project
16
The TactaBoard 2.1
17
Empirical Studies

• TactaChair experiments
• Vibration location identification
• Priming for a visual search task
• Intensity matching
• Current experiments
• Exposure during room clearing tasks
• Spatial awareness
• Team member location for team training
• Robot tele-operation

18
TactaChair Experiments
19
Exposure Experiments

• Looking at the use of spatialized vibrotactile
  feedback as a training aid on "victim" search
  tasks

20
Exposure Experiments (cont.)

• Use vibration to convey exposure
• Results to appear in ACM CHI 2005

21
Issues to be Addressed

• Transfer effects from virtual to real
  environments
• How do subjects perform after training in VR?
• Psychophysical issues
• Sensory substitution
• Cognitive Issues
• Does the addition of haptic cues increase
  cognitive load?
• Multi-modal integration

22
Interaction in VR

• Use of a keyboard and mouse is not tractable
• Can't see them
• Want to move around
• No good 3D mappings
• How can we allow easy interaction that takes
  advantage of real-world experience?
• This is the problem that we need to solve!

23
Primary Interface Tasks

• Navigation and Wayfinding
• Object Selection
• Object Manipulation
• Direct Manipulation
• Indirect Manipulation
• Can compare different techniques empirically

24
Desktop InteractionSensAble PHANToM
http//www.sensable.com/
25
Direct Manipulation
Courtesy Virginia Tech
26
Can We Do WIMP in VR?
Courtesy Virginia Tech
27
Wearable Interaction with HapticsImmersion
CyberGrasp
http//www.immersion.com/
28
Wearable InteractionRob's Hand-Held Windows
http//www.cs.wpi.edu/gogo/
29
How Do We Do Menus?
Courtesy Virginia Tech
30
Interface Devices
Courtesy Virginia Tech
31
Augmented Reality (AR)
Courtesy Virginia Tech
32
Applications

• Data perceptualization
• Map variables to tactors
• Spatial awareness
• Driver warning system (vibrotactile Bott's dots)
• Navigational aid
• Firefighter guidance
• Non-verbal communication
• Map hand signals to vibrotactile patterns

33
The Unified Scene Graph

• A way of specifying visual, audio, and tactile
  object properties
• Design goals
• Provide programmer with a single scene graph
• Need to retain speed
• Need to support different "renderers"
• Main concepts
• Material node on steroids
• Compile/flatten routines

34
The Unified Scene Graph (cont.)

• Associate all properties in Material Node

35
Summary of Research Activities

• Human-Computer Interaction
• Usability studies
• Computer Graphics
• Virtual Reality and Gaming
• Interesting input devices
• Haptic/tactile feedback
• Multi-modal interaction and integration
• Pervasive and Wearable Computing
• Robot Tele-Operation

36
Open Problems in VR

• Tracking sucks!
• Optical, magnetic, ultrasonic, inertial, hybrid
• Integration of sensory stimuli
• Good combination of direct and indirect user
  interaction
• User studies!
• Tough to design
• Easy to get subjects

37
More Info

• gogo_at_wpi.edu
• http//www.cs.wpi.edu/gogo/
• http//www.cs.wpi.edu/gogo/hive/