An Overview of Virtual Reality in Rehabilitation

1
An Overview of Virtual Reality in Rehabilitation

• David L. Jaffe, MS
• VA Palo Alto Health Care System
• Rehabilitation RD Center

1
2
Topics

• Definition of terms
• VR Uses
• VR Benefits
• VR Hardware and Software
• VR Applications in Rehabilitation
• Specific VR Rehabilitation Projects
• Stepping-over Responses
• Driving Simulator

2
3
Definition of VR

• Virtual Reality is a technology that permits
  users to interact with elements outside the
  immediate physical world
• Usually computer-based
• Using one or more of the 5 senses
• Interaction can occur at a distance
• Elements can include objects, virtual people, and
  other VR users

3
4
Different types of interactions

• Virtual reality interaction with mostly virtual
  elements
• Augmented reality interaction with mostly
  physical elements
• Simulation interaction with processes that are
  well-defined or modeled in reality
• Visualization interaction with elements normally
  hidden from view
• Game interaction is for entertainment purposes

4
5
VR elements

• Physical people and objects
• Virtual people and objects
• Other VR users
• A combination of above

5
6
VR applications

• Entertainment and games
• Exploration of alternate environments
• Training or practice on equipment
• Control of devices at a distance
• Investigation of design alternatives
• Visualization of micro to macroscopic elements
  (2D and 3D)

6
7
VR benefits - 1

• Protects users from real physical risks and
  dangers by creating a safe environment for
  training
• Produces scenarios that are difficult or
  impossible to create in reality
• Monitors users responses and measure
  improvements in performance

7
8
VR benefits - 2

• Develops graded scenarios to challenge user
• Provides real-time sensory feedback
• Controls at a distance
• Provides a different sensory perspective
• Provides a different temporal perspective
• Occupies users attention
• Distracts users from pain

8
9
VR technology

• Displays
• Head mounted displays
• Force feedback devices
• Haptic (tactile or vibro-tactile) devices
• Actuators
• Trackers
• Camera systems

9
10
Head mounted display
The Virtual Research V8 HMD has active matrix
LCDs with 640 by 480 pixel resolution and
headphones.
10
11
Position tracking

• The 3D-BIRD delivers smooth tracking of
  orientation angles without range limitation or
  line-of-sight restrictions.
• 160 measurements per second
• Provides orientation tracking for real-time
  viewing of 3D graphical images and scenes
• Suited for head-tracking applications in which
  one needs to look around virtual environments and
  simulated worlds.

11
12
Camera tracking

• Non-contact, accurate motion measurement

12
13
Microvision Nomad
The Microvision Nomad superimposes the equivalent
of a 17 display in the users field of view.
13
14
Mobile VR
Wearable PCs can make the VR experience portable.
Wireless technology can also be employed to
connect users to the Internet.
14
15
CAVE display

• A multi-person virtual visual environment
• 3 meter cube
• Projections on 3 walls and floor
• Users heads and hands are tracked
• No force or tactile feedback
• Can walk around in the virtual space
• Not quite a Star Trek Holo-suite

15
16
CAVE display
The CAVE has been made relatively small by using
mirrors to deflect the images from the video
projectors to the back-projection screens.
16
17
Immersadesk
The ImmersaDesk is a drafting-table format
virtual prototyping device. Using stereo glasses
and sonic head and hand tracking, this
projection-based system offers a semi-immersive
intreaction. The ImmersaDesk features a 2 by 2.5
meter rear-projected screen at a 45-degree angle.
17
18
P5 controller glove

• The glove beams finger and hand movements to a
  base station
• Used for computer gaming
• Essential Reality - 129

18
19
CyberGrasp by Immersion

• Force feedback exoskeleton system for fingers and
  hand
• Grasp computer-generated or tele-manipulated
  objects

19
20
Software

• Gesture Extreme Software by VividGroup - creates
  dynamic computer applications that allow computer
  users to step inside the screen and have their
  live video image replace the mouse as the
  controlling interface for the computers operation.

20
21
VR in rehabilitation

• Stake-holders
• Researchers
• Practitioners
• Policy makers and advocates
• Persons living with a disability

21
22
Why should rehabilitation researchers be
interested in VR?

• VR technology can provide a means for individuals
  with disability to perceive and interact with
  objects in a non-physical world.
• Current applications include movement training,
  exploration of accessible environments,
  manipulation of virtual workplaces, assessment
  of mobility and cognitive functions, practice in
  operating a virtual wheelchair or mobile robot,
  treatment of phobias, and behavior modification.

22
23
Why should rehabilitation practitioners be
interested in VR?

• VR can provide a safe and consistent means of
  assessing and improving a wide variety of
  functional and cognitive abilities. These systems
  can be tailored to an individual's changing
  needs, preferences, and abilities. The degree of
  difficulty of a particular task can be programmed
  by a clinician and the user's performance can be
  measured and recorded. VR applications using a
  head-mounted display can provide an immersive
  visual environment that focuses the user's
  attention on the programmed task and can offer a
  unique viewing perspective.

23
24
Why should rehabilitation policy makers and
advocates be interested in Virtual Reality?

• VR applications may be able to improve functional
  outcomes in a cost-effective fashion. VR systems
  can be physically compact, taking up minimal
  space in a clinic and can be used with a minimum
  complement of personnel. The instrumentation can
  allow direct data streaming into a patient chart
  and provide objective evidence of effectiveness.

24
25
Why should persons living with a disability be
interested in Virtual Reality?

• Physical interactions with environments that are
  partially or wholly computer-generated allow a
  person to focus on a particular aspect of
  functional recovery without exposure to
  potentially dangerous situations.
• For example, practicing stepping over virtual
  obstacles during harnessed treadmill walking is
  safer than stepping over blocks in the clinic
  while aided by a therapist. As proficiency,
  coordination, and strength improve, the more
  complex realities of real-world situations can be
  attempted with increased confidence.

25
26
VR applications in rehabilitation

• Sensory impairments
• Physical impairments
• Cognitive and behavioral impairments
• Orthopedic and mobility impairments

26
27
Parkinsons and AkinesisTom Reiss and Suzanne
Weghorst

• Enables initiation of locomotion without visual
  targets through augmented reality
• Static and moving images provide visual flow
  stimulus
• http//www.hitl.washington.edu/research/parkinsons
  /

27
28
Exposure Therapy for Spider PhobiaHunter
Hoffman, University of Washington
http//www.hitl.washington.edu/research/exposure/
28
29
Integrated Wearable Orientation and Wayfinding
System for People with Vision LossDavid A. Ross,
Altanta VA Rehab RD Center

• The wearable system being developed acts as an
  agent to integrate information from five existing
  location technologies into a seamless
  presentation of orientation and way-finding
  information for the blind traveler. The five
  technologies employed include GPS,
  Dead-Reckoning, Talking Signs, Talking Lights,
  and Relume Pedestrian Signals.

http//www.varrd.emory.edu/personnel/ross.htm
29
30
Retractable Cane for the Visually
ImpairedVincent K. Ramsey, Altanta VA Rehab RD
Center

• This project focuses on the development and
  testing of a retractable long cane that
  accurately simulates the experience of
  encountering a drop-off without exposing a
  visually impaired individual to the risk of
  falling. This device would permit these
  individuals to learn to appropriately respond to
  the kinesthetic feedback produced from a drop-off
  encounter in a safe environment. Three prototype
  canes are now being field tested with a
  videography motion system that analyzes subjects'
  gait parameters and reaction times.

http//www.varrd.emory.edu/personnel/ramsey.htm
30
31
VR Driving SimulatorMaria T. Schultheis, Kessler
Medical Research

• This project employs a driving simulator to
  evaluate driving capacity following a stroke or
  traumatic brain injury.

http//www.kmrrec.org/KM/bios/mschultheis.php3
31
32
Real-Time 3D Virtual Training in SCI
RehabilitationTom MacLaughlin, Motion Reality,
Inc.

• The system consists of a state-of-the-art
  real-time 3D Full Body Tracking, Dynamics
  Analysis and Computer Graphics Animation of
  spinal cord injury patient body motion and
  performance data. It provides a unique virtual
  reality/visual biofeedback capability for SCI
  clinical rehabilitation.
• Additional novel capabilities include the ability
  to immerse the subject inside the body of a
  virtual "model' performer so the subject can
  "feel" what to do in a target task. Real-time EMG
  is converted to changing colors of graphical body
  muscles to provide visual feedback on
  instantaneous levels of muscle activation.
• http//www.motionrealityinc.com/

Real-time 'step-in transfer skill learning
process
32
33
Tele-Rehabilitation Functional
RehabilitationJared Baer, 5DT Inc.

• This project involves the design, development,
  and application of a tele-rehabilitation
  application focusing on hand injuries. The system
  can also be used for the tele-rehabilitation of
  other disabilities.
• http//www.5dt.com/index.html

33
34
Exposure Therapy for Vietnam Veterans with
Posttraumatic Stress Disorder (PTSD)Ken Graap,
Virtually Better, Inc.

• Two virtual Vietnam environments have been
  constructed to allow exposure based treatment of
  combat related PTSD in an open clinical trial
  with Vietnam Veterans who served in combat.
  Results suggested that many participants
  benefited from this treatment.
• http//www.virtuallybetter.com/

34
35
Face Attention in Preschoolers with AutismCheryl
Trepagnier, The Catholic University of America

• Accumulating evidence suggests that failure to
  attend to and process face-borne information
  plays a major contributing role in the autistic
  syndrome. In this environment their gaze is
  monitored using eye tracking, and reward
  attention to people with enjoyable visual and
  auditory displays and rides in the 'pod' in which
  the training is delivered. The goal of the
  training is to provide the child with lots of
  experiences of positive consequences of looking
  at others, and to produce generalization of this
  behavior to peers.
• http//www.atnrc.org/rd/rd_vrapp/prag/pragmatics.
  html

35
36
Making Exercise FunRory A. Cooper, University of
Pittsburgh

• Virtual reality provides a safe and effective
  means of developing and testing new interface and
  mobility technology for electric powered
  wheelchairs. In a virtual environment data can be
  gathered to tune devices to match the needs of
  individuals, and the virtual environment can
  serve as a training tool. By capturing the
  immersion offered by virtual gaming environments,
  the project has worked to make exercise and
  manual wheelchair mobility skills training more
  enticing and effective.

36
http//www.herlpitt.org/personnel-main.htmExecuti
ve Staff
37
VR Applications for Patients with Neurologic and
Orthopedic DeficitsLynda S. Savard, Sister Kenny
Rehabilitation Institute

• Using the IREX VR non-immersive interactive
  technology, clinicians from multiple
  rehabilitation disciplines can present
  challenging and engaging therapeutic activities
  and exercises to patients with neurologic and/or
  orthopedic deficits.
• mailtolsavard_at_allina.com

37
38
Robotic Neurorehabilitation using Augmented
Reality DisplaysJames Patton, The Rehabilitation
Institute of Chicago

• The PARIS augmented reality system allows the
  subject to view their own arm while receiving
  enhanced feedback of movement errors, forces, or
  visual distortions that can trick the nervous
  system into recovering faster.

http//www.smpp.northwestern.edu/robotLab/
38
39
Cognitive Virtual Environments in Assessment and
RetrainingMark Dubin, University of Colorado

• The CoVE project uses a recently opened Immersive
  Virtual Environment at the University of
  Colorado's Visualization Center to develop
  procedures for testing and retraining individuals
  with cognitive disabilities. Work in progress
  consists of the development of a visual, 3D test
  for early stage Alzheimer Disease. A conceptual
  plan and prototype for retraining of attentional
  mechanisms after brain injury is being addressed
  in this project.
• http//spot.colorado.edu/dubin/

39
40
VR Pain Distraction System

• Distract the attention of patients who undergo
  painful procedures, useful when anesthesia is not
  an option.
• User wears a HMD and plays a distraction game
  with a game pad or joystick.
• HMD isolates user from the sight of the
  procedure.
• The multi-level games provides a challenging and
  rich visual and auditory experience.

http//www.5dt.com/products/pvrpds.html
40
41
Virtual interactive environments

• Exposure therapy for phobias spiders, heights,
  fear of flying, fear of driving, fear of
  darkness, fear of small spaces
• Behavior modification systems for eating
  disorders and teaching proper social behaviors

41
42
VA RRD Projects

• Using Augmented Reality to Improve Gait of Stroke
  Survivors
• Driving Simulator to Assess and Improve
  Performance of Individuals after a Brain Injury

42
43
Using Augmented Reality to Improve Gait of Stroke
Survivors
43
44
Stroke

• A stroke occurs when a rupture or blood clot
  reduces blood flow to an area of the brain,
  killing brain cells and disrupting the abilities
  or functions they control.
• Many survivors experience weakness and poor
  coordination, which impairs their ability to walk
  and use their hands.

44
45
Strokes impact on walking

• Shortened stride length
• Decreased walking speed
• Balance problems
• Increased risk of falling
• Difficulty bending joints
• Lower walking confidence
• Diminished ability to climb stairs
• Lower walking endurance
• Difficulty stepping over obstacles

45
46
Project hypothesis
Training people with stroke to step over objects
will improve their walking

• Stride length
• Walking speed
• Balance
• Ability to step over obstacles

• Walking endurance
• Confidence
• Fear of falling
• Quality of life

46
47
AR training system
47
48
Computer setup
48
49
Camera
49
50
Footswitches
50
51
Vibro-tactile transducer
51
52
Interface box
52
53
Head-mounted display
53
54
All wired up
54
55
Step over
55
56
Toe collision
56
57
Heel collision
57
58
VR training video 1/5
4 by 6 obstacles and stills
This video shows the view through the
head-mounted display. The video pauses to show
the different range of motion of each knee.
58
59
VR training video 2/5
This video shows the assistance provided by the
therapist.
Manual assistance
59
60
VR training video 3/5
The subject steps over 4 by 12 inch virtual
obstacles.
4 by 12 obstacles
60
61
VR training video 4/5
The final VR training session for this subject
shows the best performance.
4 by 14 obstacles
61
62
VR training video 5/5
The subject steps over the 4 by 4 inch virtual
obstacle. The stripe on the pants provide knee
flex information.
4 by 4 obstacle with striped pants
62
63
Subject population
63
64
Outcome summary
64
65
65
66
66
67
Results

• Statistically significant improvements with both
  training groups
• Small differences between groups, but the AR
  group showed slightly better trends
• All participants expressed positive experiences

67
68
Conclusion

• This training intervention
• provides a safe environment for practicing new
  movement strategies
• provides multiple immediate feedback channels
• visual, auditory, tactile
• involves a trained professional (PT)

68
69
Future work

• New populations
• Stroke
• Parkinsons Disease
• Traumatic Brain Injury
• Incomplete Spinal Cord Injury
• New methodology
• 4 weeks training
• Addition outcome measures
• New equipment and upgraded software

69
70
Methodology for Using a Simulator to Assess and
Improve the Driving of Individuals after Brain
Injury
70
71
Background - 1

• Driving requires a complex set of skills and
  abilities
• mobility, vision, visual-motor coordination, and
  multiple levels of cognitive function
• A stroke, TBI, or SCI may result in diminished
  driving ability.
• After an acute injury, the DMV may suspend
  drivers license upon notification from a
  physician.

71
72
Background - 2

• The ability to drive is an essential component of
  independent living and increases ones quality of
  life.
• Disabled veterans and active duty personnel,
  including those undergoing rehabilitation, are
  entitled by Public Law 93-538 to have the
  opportunity to pursue a return to driving.
• The PMR Service within the VA is responsible for
  providing driving evaluations and training to
  patients with TBI, stroke, SCI, and other
  disabling conditions.

72
73
Driving tests after a CNS injury

• Tests of vision, reaction time, range of motion,
  strength
• On-road driving assessment
• Up to 10 hours of in-car training
• Prescription of adapted driving equipment

73
74
Considerations

• Patients may not have driven since accident
• Accident may have reduced driving skill
• Patients may be unaware of diminished skill
• Patients may be stressed, anxious, fearful
• Patients may lose control of car

74
75
Problems to be addressed

• In-car assessment and training activities create
  potentially hazardous situations for individuals
  with impaired cognitive function and diminished
  driving skills, their instructor, other cars, and
  pedestrians. Many close calls have been reported.
• In-car assessments are essentially subjective and
  lack a standardized scoring system.
• It is difficult to provide a graded driving
  training experience, repetitively practice a
  specific driving task, or test a patients
  ability to respond correctly to an emergency
  situation.

75
76
Ideal solution

• Means of assessing driving performance without a
  car
• Method should correlate with existing assessments

76
77
Project goal

• The long-term goal of this research effort is to
  investigate the effectiveness of a high-quality
  interactive driving simulator to safely evaluate
  and improve the driving ability of individuals
  following brain injury.
• Returning these individuals to driving satisfies
  the rehabilitation goals of community re-entry,
  improves their independence, and enhances their
  quality of life.

77
78
What is a driving simulator?

• A computer-based system that reproduces driving
  conditions and elicits driving responses in a
  clinical setting
• Interactive drivers actions influence display

78
79
Specific project goals

• Screen for unsafe drivers before an in-car
  assessment is performed
• Improve patients' awareness of their ability to
  drive safely
• Reduce time required for assessments and training
• Provide more effective training
• Train better drivers
• Improve patients' cognition, perception, and
  motor responses

79
80
Deficits addressed

• Cognition
• Left-right orientation
• Attention
• Distraction
• Visual motor tracking
• Perceptual motor tracking

• Spatial judgment depth perception
• Multiple tasks
• Divided attention
• Visual perception
• Judgment
• Motor responses

80
81
Photograph of simulator
81
DriveSafety Model 550C 3-Channel Simulator with
Saturn cab.
82
Primary benefits of using a driving simulator

• Safety the patient is not exposed to real
  traffic hazards
• Reproducibility identical customized road
  courses can be presented to measure improvements
  in proficiency after training
• Graded challenge training difficulty can be
  increased as patients driving performance
  improves
• Versatility various road and weather conditions
  can be simulated, including emergency situations

82
83
Benefits of using a simulator - 1

• Increased safety for patients, pedestrians, and
  the driving evaluator cant run over
  pedestrians, hit other cars, or damage property
• Ability to reduce fear, anxiety, anger, and
  frustration of veterans with brain injury who
  want to return to driving less stressful for
  those who havent driven since their accident
• Ability to individually tailor training by
  targeting specific driving performance
  deficiencies

83
84
Benefits of using a simulator - 2

• Ability to safely and repetitively practice
  specific driving problems at the subject's own
  pace
• Ability to objectively and consistently measure
  driving performance and track improvements
  including identifying learning plateaus
• Ability to alter the difficulty of road courses
  to enhance the training experience and challenge
  subjects

84
85
Benefits of using a simulator - 3

• Ability to avoid inclement weather conditions
  through indoor operation
• Ability to present unique driving situations such
  as a child running into the street
• Ability to modify driving parameters such as
  steering effort to accommodate varying abilities

85
86
Benefits of using a simulator - 4

• New technology provides a high degree of realism
• Potential time and money savings over the on-road
  assessment
• Can be fitted with adaptive driving equipment to
  accommodate patients with physical limitations

86
87
Research study

• The efficacy of using a computer-based driving
  simulator to provide objective data on driving
  ability and driving-related cognitive function
  will be investigated.
• The study will
• compare in-car and simulator driving evaluations
• compare the effectiveness of in-car and simulator
  training to improve driving performance
• follow-up subjects driving in the community to
  establish correlations with evaluations and
  training

87
88
Methodology

• Administer questionnaires, neuropsychological
  tests
• Perform simulator and in-car driving evaluations
• Train BI subjects based on their in-car
  evaluation
• Passed
• Failed, judged trainable
• Failed, judged not trainable
• Randomize simulator and in-car training
  intervention
• Perform post-training simulator and in-car
  evaluations
• Re-administer tests and questionnaire
• Analyze differences between in-car and simulator
  assessments and training
• Monitor driving after 6 months (family member or
  friend)

88
89
Evaluation courses

• Instruction / Practice course
• Hospital grounds
• Residential
• Commercial
• Freeway
• Hills and mountains

89
90
Training courses

• Several driving challenges presented in the
  context of a road course
• Driving challenges can be graded from simple to
  difficult
• The simulator can measure driving performance and
  alter difficulty dynamically

90
91
Course design Step 1

• Videotape and provide descriptive narration for
  the current on-road evaluation courses
• The driving instructor was accompanied on the
  five on-road courses used to evaluate drivers.
  The courses were videotaped through the
  windshield. As he drove, the instructor described
  the driving challenges as they presented
  themselves and related the level of driving
  performance he expected from a proficient driver.
  He annunciated the verbal commands he gives the
  subjects as well as the mistakes poor drivers
  make.
• Verbal notes of the road elements were made
  driveways, trees, buildings, schools, and
  pedestrians. The comments included specific
  driving situations such as people crossing in the
  middle of the street, joggers on the side of the
  road, and other unusual occurrences.

91
92
Photograph from video
92
93
Course design Step 2

• View the videotape and compose a log of the
  course elements and driving challenges, and
  evaluation items to be measured.
• The evaluation course videotape is reviewed and a
  chronological diary of the course elements are
  created. They consist of the general road
  conditions, the scenery, and the driving
  challenges experienced during the course. The
  driving challenges are organized into a list of
  specific observational evaluation items. Options
  for increasing the driving challenge are also
  noted. An example is
• 12. pull into and back out of a diagonal parking
  space
• 2 lane roadway in a parking lot
• narrowness of the parking space
  increases difficulty
• steering wheel control
• gas/ brake coordination
• use of turn signal

93
94
Course design Step 3

• Add / delete specific driving challenges to the
  individual course.
• Additional driving challenges are added to the
  course list. They include items not easily
  reproduced on the road, such as interaction with
  emergency vehicles and unexpected situations such
  as a child running into the street. Other
  challenges present in the original video may be
  deleted if they have been presented previously.
  Other items can modified to increase their
  difficulty.

94
95
Course design Step 4

• Draw a roadway course on paper that somewhat
  conforms to the on-road evaluation courses.
• The purpose of this task is to create a basic
  course layout that can be easily modified before
  it is constructed within the simulators
  authoring environment. The map would include
  roadway types, intersections, and turns.

95
96
Paper course template
96
97
Course design Step 5

• Create the basic roadway on simulator using
  appropriate roadway tiles
• The paper map is used as a template to create the
  course on the simulator. Road elements are
  selected and dragged/dropped onto the simulated
  world. Roadway and intersections types are
  designated and connected end-to-end.

97
98
Overhead view of an intersection
98
99
Course design Step 6

• Add scenery items to the basic roadway
• Additional buildings, driveways, signs, parked
  cars, pedestrians, and other static items are
  added to the basic roadway to create the proper
  environment for the driving challenges. Items are
  selected, positioned, and dropped into the
  roadway. Orientation and precision placement are
  specified.

99
100
Car pulling out into road
100
101
Perspective view of intersection
101
102
Close-up of House Party
102
103
Perspective view of course
103
104
Close-up of objects
104
105
Course design Step 7

• Animate vehicles, pedestrians, to create specific
  driving challenges
• Specific programming is required to specify time
  and location triggers that initiate movement of
  other vehicles and pedestrians and specify the
  path they will follow and the speed at which they
  will move. Sounds files can be added for
  increased realism or presented as instructions.

105
106
Car pulling out of driveway
106
107
Course design Step 8

• Create programming that will measure performance
  during the driving challenges
• Additional programming is created to monitor
  specific driving performances. This includes
  minimum/maximum speeds, vehicle following
  distance, quality of turns, lane position,
  stopping distance, brake or accelerator reaction
  time, and judgment during merging, right turn on
  red, etc. This information is written to a file
  during the simulator run and collected for
  subsequent analysis.

107
108
Programming process
108
109
Data collection

• Speed
• Acceleration
• Lane position
• Steering, throttle, and brake inputs
• Collisions
• Failure to stop
• Speed limit violations
• Divided attention response time
• Following speed and distance
• Reaction time
• Other, programmable

109
110
Future studies

• Practice use of hand-control and adaptive driving
  aids
• Evaluate performance of older drivers

Funding for this project has been provided by VA
Merit Review - B3288R.
110
111
Questions

• In the Stepping-over VR Project, why was a side
  view of the subject used?
• What is cybersickness? What causes it? How can
  it be minimized?
• In the driving simulator project, did anyone
  experience cybersickness?
• What populations of VR users are more susceptible
  to cybersickness?

111
112
Thanks to

• VA investigators
• VR investigators outside VA

112
113
Contact information

• David L. Jaffe, MS
• jaffe_at_roses.stanford.edu
• http//guide.stanford.edu

113