Augmented reality based affect adaptive neurocognitive rehabilitation

1
Augmented reality based affect adaptive
neurocognitive rehabilitation

• Intermediate report of Msc. Thesis project
• A.S.Panic

2

• Student Media Knowledge Engineering (EEMCS) -
  Man Machine Interaction

• 5 months project work at ETH Zurich

• Financial support provided by
• Imperial College, London, TU Delft, ETH Zurich,
  RWTH Aachen, ParisTech

http//www.idealeague.org/
3
In this presentation
Augmented reality based affect adaptive
neurocognitive rehabilitation
4
Thesis position

• Neurocognitive rehabilitation is possible to a
  higher extent than previously assumed
• Therapeutic exercises can be made more motivating
  for the patient
• In the near future more people will need
  cognitive (re)habilitation than have access to,
  or can be treated in the existing clinics
• Human factors play a significant and undervalued
  role when designing technology

Augmented reality based affect adaptive
neurocognitive rehabilitation
5
Surveying the literature

• Three questions drive the literature survey
• Which concepts are used to understand cognitive
  processes and neurocognitive rehabilitation?
• How can neurocognitive rehabilitation be
  supported by virtual reality applications?
• How can therapeutic games be created so that they
  promote a motivated learner?

Augmented reality based affect adaptive
neurocognitive rehabilitation
6
Main topics for literature survey
Augmented reality based affect adaptive
neurocognitive rehabilitation
7
Basic stages of cognition

• Lack or loss of language severely impairs
  cognition
• Sound and music can significantly influence
  emotion
• Emotion significantly influences cognition

Source Groome Dewart, 1999, An introduction to
cognitive psychology processes and disorders
Augmented reality based affect adaptive
neurocognitive rehabilitation
8
Emotion and cognition
One example of cognitive influences of emotion

• In a study at Tufts University
• Test subjects were blindfolded and physically
  moved a random distance between two houses
• The test subject were then asked to which of the
  two houses he or she was closer to

The conclusion the reported distance estimates
were significantly different if the two houses
were religious in nature (e.g. church, mosque)
versus neutral
Source Qui Wang at the Psychology department of
Tufts University
Augmented reality based affect adaptive
neurocognitive rehabilitation
9
Stages not easily summarizable

• Stages of visual perception include processing
  horizontal lines, vertical lines, arcs, edge
  detection, shape detection,
• Stages of auditory perception include pitch,
  timbre, rhythm, melody, localization, so

A multi-stage model of memory

• In reality cognitive processes are not
  identifiable by independent and distinct stages,
  but composed of many (overlapping) substages
• Cognition operates at the levels of basic skills,
  executive functions and general intelligence
  (fluid and crystallized)

Source Groome Dewart, 1999, An introduction to
cognitive psychology processes and disorders
Augmented reality based affect adaptive
neurocognitive rehabilitation
10
Loss of cognitive skills

• A loss of cognitive skills can be caused by

• Typical aging process
• Neurodegenerative diseases (see picture)
• Traumatic brain injury

• Similar symptoms exist in young and old people,
  but with different names
• Mostly pathological causes

Source Fisk et al, Designing for older adults
Augmented reality based affect adaptive
neurocognitive rehabilitation
11
Cognitive (behavioral) therapy

• Cognitive therapy (CT) aims at rehabilitating any
  of the basic cognitive skills
• Cognitive behavioral therapy (CBT) aims at
  rehabilitating one or more Activities of Daily
  Living (ADL)
• Issues with CT and CBT
• Treatment or assessment can be dependant on
  administrator (therapist)
• Lack of standardized stimuli or treatment content
• Therapeutic exercises can be boring

Source Buschert, 2009, Kognizionsbezogene
Interventionen bei Alzheimer-Krankheit
Augmented reality based affect adaptive
neurocognitive rehabilitation
12
Cognitive rehabilitation strategies

• Cognitive rehabilitation can aim to
• Reinforce, strengthen or reestablish previously
  learned skills or behavior
• Establish patterns of compensatory cognitive
  activity to cope with impairment of cognitive
  systems
• Establish environmental compensation that allow
  new patterns of activity
• Enable the patient to adapt to their cognitive
  impairment in order to improve overall level of
  functioning and quality of life

Source Cicerone, 2000, 2005, Evidence based
cognitive rehabilitation, recommendations for
clinical practice
Augmented reality based affect adaptive
neurocognitive rehabilitation
13
Brain plasticity and cognitive reserve

• Since the 1980s two (theoretical) constructs
  became adopted when explaining how a loss of
  cognitive skills can be rehabilitated
• Brain plasticity the changing of neurons, the
  organization of their networks, and their
  function via new experiences.
• Cognitive reserve the brains resilience
  to neuro pathological damage

Augmented reality based affect adaptive
neurocognitive rehabilitation
14
Gaming stimulates brain plasticity

• MRI assessments show that structural engagement
  with the Tetris game increases formation of gray
  matter

• Lack of conclusive evidence for other games, e.g.
  Dr Kawashimas Brain Training for Nintendo DS.
• However many of these games are based on standard
  instruments of cognitive assessment.

Source Haier et al, 2009, MRI Assessment of
cortical thickness and functional activity
changes in adolescent girls, following three
months of practice on a visual-spatial task
Augmented reality based affect adaptive
neurocognitive rehabilitation
15
Cognitive reserve

• Cognitive reserve is based on two mechanisms
• Neural reserve
• Neural compensation

• Evidence for cognitive reserve Nuns have been
  diagnosed with Alzheimers Disease
  (neurodegenerative!), but without any noticeable
  loss of cognitive skills

Sources Stern, 2006, Cognitive reserve and
Alzheimers disease Snowdon, 2004, Healthy aging
and dementia findings from the Nun study
Augmented reality based affect adaptive
neurocognitive rehabilitation
16
Case studies and explanations
Brain plasticity, Discovery channel clip (5
min) This video clip, from the Discovery Channel
and McGraw-Hill Higher Education, details how the
brain can "repair" itself. It introduces a case
study of young Jody Miller, who had half of her
brain removed, in an attempt to control her
epileptic seizures.
http//www.youtube.com/watch?vTSu9HGnlMV0
Re-wiring the brain, 2008 TED talk (20
min) Neuroscientist Michael Merzenich looks at
one of the secrets of the brain's incredible
power its ability to actively re-wire itself.
He's researching ways to harness the brain's
plasticity to enhance our skills and recover lost
function.
http//www.ted.com/talks/michael_merzenich_on_the_
elastic_brain.html
Augmented reality based affect adaptive
neurocognitive rehabilitation
17
Brain plasticity a case study
stroke of insight, 2008 TED talk (20 min) Brain
researcher Jill Bolte Taylor studied her own
stroke as it happened. She has become a
spokesperson for stroke recovery and for the
possibility of coming back from severe brain
injury stronger than before.
http//www.ted.com/talks/jill_bolte_taylor_s_power
ful_stroke_of_insight.html
Augmented reality based affect adaptive
neurocognitive rehabilitation
18
Stimulating cognitive reserve

• At any point in ones lifetime, cognitive reserve
  results from a combination of exposures
• Socio-economic status (e.g. occupation)
• Educational attainment
• Leisure activities
• Physical activity stimulates neurogenesis

Source Stern, 2009, Cognitive reserve
Augmented reality based affect adaptive
neurocognitive rehabilitation
19
The virtuality continuum

• Allows the classification of applications that
  create an illusion of an alternate reality

Reality
Virtual reality
Mixed Reality
Augmented reality, more real than virtual (e.g.
HMD)
Augmented virtuality, more virtual than real
Non or partially-immersive VR (e.g computer
monitor)
Source Milgram Kishino, 1994, A taxonomy of
mixed reality visual displays
Augmented reality based affect adaptive
neurocognitive rehabilitation
20
Creating a fake reality

• Sensorama (1957) by the godfather of virtual
  reality

• One to four simultaneous users
• 3D motion picture, smell, stereo sound,
  vibrations, wind
• The holy grail of virtual reality is to fool all
  the senses and create a suspension of disbelief
  for the virtual environment

Source Morten Heilig, 1962, Sensorama patent
Augmented reality based affect adaptive
neurocognitive rehabilitation
21
VR based cognitive rehabilitation

• When using virtual reality for neurocognitive
  rehabilitation, what are the
• Strengths?
• Weakness?
• Opportunities?
• Threats?
• Assumptions
• Administration in home environment

Source Adapted from Rizzo Kim, 2005, A SWOT
analysis of the field of virtual reality
rehabilitation and therapy
Augmented reality based affect adaptive
neurocognitive rehabilitation
22
SWOT of particular interest

• Strength
• High fidelity training and assessment
  environments can be created
• Economy of scale
• Gaming factors can be used to increase patient
  motivation and adherence
• Weakness
• Interface challenge wires, displays and
  peripherals
• Side effects of exposure to virtual environments
  may not be unavoidable
• Engineering challenge proper support for data
  mining, extraction analysis (for therapists!)

Source Adapted from Rizzo Kim, 2005, A SWOT
analysis of the field of virtual reality
rehabilitation and therapy
Augmented reality based affect adaptive
neurocognitive rehabilitation
23
SWOT of particular interest

• Opportunities
• Emerging technologies
• Unobtrusive and natural interaction devices,
  wires
• Wearable computing, sensor networks
• Situated cognitive rehabilitation /
  telerehabilitation
• Preventive training during preclinical stage
  increases cognitive reserve and resilience to
  pathology
• Threats
• Privacy issues with telerehabilitation and
  medical data
• Potential for lawsuits because of side-effects
• Ethical challenges

Source Adapted from Rizzo Kim, 2005, A SWOT
analysis of the field of virtual reality
rehabilitation and therapy
Augmented reality based affect adaptive
neurocognitive rehabilitation
24
Motivation in instructional games

• The goal is to create a game cycle which
• fulfills learning objectives and
• creates and sustains a motivated learner

Source Paras, 2005, Games, motivation and
effective learning an integrated model for
educational game design
Augmented reality based affect adaptive
neurocognitive rehabilitation
25
Affective gaming

• Affective computing computers and programs which
  take the players affective (emotional) state
  into account
• Affective games affect-adaptive games adapt
  their content based on the players emotional
  state. E.g. adapt gameplay difficulty, offer more
  or less feedback and support on progress, offer
  tips and suggestions on how to continue, increase
  or decrease number of concurrent tasks etc.
• High level design heuristics Emote me, assist
  me, challenge me

Sources Picard, 2000, Affective
Computing Gilleade et al, 2005, Affective
videogames and modes of affective gaming
Augmented reality based affect adaptive
neurocognitive rehabilitation
26
Affective game engines

• Core requirements
• A knowledge base which (hierarchically) describes
  emotions, used for emotion generation or
  expression.
• An affective user model (AUM) of the player,
  which facilitates recognition of emotion and
  transitions between them.
• Modeling of game characters emotions.
• No computational model of cognitive processes
  required, a few simple rules can already have a
  measurable effect

Source Hudlicka, 2009, Affective game engines
motivation requirements
Augmented reality based affect adaptive
neurocognitive rehabilitation
27
Thesis position (revisited)

• Neurocognitive rehabilitation is possible to a
  higher extent than previously assumed
• This may be explained by theories of brain
  plasticity and cognitive reserve
• Therapeutic exercises can be made more motivating
  for the patient
• This can be achieved by designing therapeutic
  games using criteria from affective computing and
  affective gaming

Augmented reality based affect adaptive
neurocognitive rehabilitation
28
Thesis position (revisited)

• In the near future more people will need
  cognitive (re)habilitation than have access to,
  or can be treated in the existing clinics
• Using telerehabilitation and commonly available
  (cheap) computing technology, virtual reality
  based therapy can be offered to a larger part of
  the patient population
• Human factors play a significant and undervalued
  role when designing technology
• Human Computer Interaction design should target
  natural and embodied interaction as much as
  possible.
• Differences in technological aptitude should not
  prevent people from accessing healthcare

Augmented reality based affect adaptive
neurocognitive rehabilitation
29
Experimental research questions

• Principal research questions
• Does the use of affect-adaptive training
  influence the rate of learning and motivation on
  a commonly used instrument of cognitive
  performance, when compared to a standard computer
  based training?
• Does the use of human computer interaction
  devices which support more natural and embodied
  interaction lead to a higher rate of acceptance
  by the targeted population?

Augmented reality based affect adaptive
neurocognitive rehabilitation
30
The mental rotation task

• Research vehicle the mental rotation task (MRT)
• Response time depends on
• Difference in rotation between object pair
• Object complexity (but this diminishes with
  practice!)

Are these two objects the same?
Sources Shepard Metzler, 1972, Mental
rotation of three dimensional objects, Bethell-Fox
Shepard, 1988, Mental Rotation effects of
stimulus complexity and familiarity
Augmented reality based affect adaptive
neurocognitive rehabilitation
31
Why use the mental rotation task?

• Rationale for using the MRT in this experiment
• As a widely used instrument for cognitive
  assessment it is particularly suitable for
  investigating gaming factors that influence
  motivation
• As a visuo-spatial task it is particularly
  suitable for virtual and augmented reality
• Stimulus complexity can be controlled precisely

Sources Rizzo et al, 1998, the virtual reality
mental rotation spatial skills project
Augmented reality based affect adaptive
neurocognitive rehabilitation
32
Experiment setup
IR LED glasses allow tracking of head position
gaze direction
Training, Assessment
Wii remote allows gestural interaction with
virtual objects (pointing, dragging, selecting)
and tactile and auditory output
With headtracking support, a TV screen allows
virtual objects to appear in front or behind of it
Augmented reality based affect adaptive
neurocognitive rehabilitation
33
Experiment protocol

• Targeted population
• Elderly people with mild cognitive impairment
• Vision corrected to standard, no history of
  postural instabilities, no motion sickness
• In a clinical session the participants complete
• A training round (using 1 of 3 game modes)
• A feedback round (motivation questionnaire)
• An assessment round (of task performance)

Augmented reality based affect adaptive
neurocognitive rehabilitation
34
Project status

• Dec 2009 Mar 2010
• Project work _at_ ETH
• Mar 2010 Jul 2010
• Finish writing Msc. Thesis _at_ DUT
• Thesis defense
• Complete 2 remaining human factors in aerospace
  courses from my honors program

Augmented reality based affect adaptive
neurocognitive rehabilitation
35
Do you want to know more?

• Y. Stern,Cognitive Reserve, Neuropsychologia
  vol 47, 2009

http//www.cumc.columbia.edu/dept/sergievsky/cnd/p
dfs/sdarticle-1.pdf

• Rizzo G. Kim, A SWOT analysis of the Field of
  Virtual Reality Rehabilitation and Therapy,
  Presence vol 14, 2005

http//vrpsych.ict.usc.edu/PDF/1Rizzo_2005_A20swo
t20analysis20of20the20field20of20VR.pdf
E. Hudlicka, Affective Game Engines Motivation
and Requirements, ICDFG proceedings, 2009
http//www.imgd.wpi.edu/speakers/0910/Hudlicka_ICF
DG_09_Dist.pdf