Inventing Virtual Teachers and Therapists Promises, Systems

1
Inventing Virtual Teachers and Therapists
Promises, Systems Challenges

• Ron Cole
• the CSLR Virtual Human Research Group
• September 2, 2005
• SPACE meeting Leuven Belgium

2
Outline

• TALK ONE Vision, Rationale Systems
• The promise of virtual humans
• Scientific rationale
• System overviews and demos
• TALK TWO Inventing Virtual Humans
• Technologies implemented to date
• Technical challenges and missing science
• Conclusions and recommendations

3
What is a Virtual Human?

• A Believable Computer Character
• with personality and attitude
• that engages users in natural face-to-face
  conversation
• to produce great learning experiences

4
Marge
5
The Promise of Virtual Humans

• Effective teachers, therapists, assistants
• A virtual human is patient and tireless
  learning can be more engaging, motivating,
  personal and effective
• Accessible to nearly everyone, anywhere, anytime
• Via multilingual natural dialog interaction on
  networked computers
• Awesome benefits to individuals and society
• Humans are expensive and often inaccessible,
  Virtual Humans are inexpensive
• Research tools for acquiring missing knowledge
• they can be programmed to behave in predictable
  ways whereas people are often guided by
  unconscious behaviors

6
Theoretical Empirical FoundationsThe media
equation media real lifeCliff Nass Byron
Reeves

• We have found that individuals interaction with
  computers are fundamentally social and natural,
  just like interactions in real life.
• All these rules come from the world of
  interpersonal interaction, and from studies of
  how people interact with the real world. But all
  of them apply equally well to media.
• The more a media technology is consistent with
  social and physical rules, the more enjoyable the
  technology will be to use. including feelings of
  accomplishment, competence, and empowerment.

7
The Persona Effect(Lester et al., 1997)

• Hypothesis Interfaces that use voice and/or face
  to foster social agency produce more satisfactory
  and effective experiences.
• Students form a social bond with virtual humans
  and are motivated to learn and succeed
• Better experiences reported in hundreds of
  experiments
• Students give top ratings to Do you think Marni
  is a good teacher? Does Marni act like a real
  human teacher? and How well does Marni help you
  learn to read?
• Better outcomes supported by dozens of
  experiments
• Several experiments show benefits of talking head
  compared to voice alone

8
The power of one-on-one tutoring

• Benjamin Bloom (1982) posed the two sigma
  challenge
• Demonstrated a two sigma benefit of one-on-one
  tutoring relative to classroom instruction
• Meta-analysis of 100s of tutoring studies
  confirms benefits of tutoring (Cohen, Kulik
  Kulik, 2982)

9
Cognitive Theory of Multimedia LearningMayer
(2001)

• Mayer examined how presentation of words and
  pictures affected learning in transfer tasks
• Best learning occurs when voice is used to
  explain phenomena displayed simultaneously in
  pictures or animation
• Atkinson demonstrated benefits of talking head
  relative to voice alone

10
Recipe for a Virtual Human System

• Develop a deep understanding of the task
• Theory, research and practice
• Analyze and model the performance of human
  experts
• Develop the system
• In collaboration with experts and users
  (participatory design)
• Design and test, design and test, test, test,
  test
• Evaluate the system (formative evaluation /
  clinical trials)
• Scale Up and Sustain the system

11
Virtual Teachers and Therapists

• Theoretical Empirical Rationale
• Demos
• Baldi teaches vocabulary to students with hearing
  loss
• Marni teaches children to read
• Marni conducts speech therapy for individuals
  with Parkinson disease and aphasia
• Research funded by grants from NSF and NIH

12
A Virtual Teacher for Students with Hearing Loss

• 1998-2001 Baldi teaches vocabulary to students
  at Tucker Maxon School
• Rapid acquisition of vocabulary
• gt50 retention several months later
• Dramatic improvements in speech production
• Featured on national TV and NSF Home page

13
Foundations to Literacy

• A comprehensive, scientifically-based reading
  program designed to
• Teach children to read and comprehend text
• Through conversational interaction with Marni, a
  virtual teacher
• That behaves like a sensitive and effective
  reading teacher

14
Cognitive theory scientifically- based reading
research

• Skilled reading is
• Word recognition processes comprehension
  processes
• This is called the Simple Model of Reading
  (Gough et al., 1996)
• Word recognition processes
• Alphabet, Phonological awareness, Encoding,
  Decoding, Sight words
• Reading in context until fluent automatic
• Evidence-based pedagogy SBRR (Rayner et al,
  2001)
• Comprehension processes
• Train fluent and expressive reading
• Train comprehension through thinking questions

15
Main Components of FtL

• Foundational Skills Tutors
• Teach underlying reading skills
• Interactive Books
• Teach fluent reading comprehension
• Managed Learning Environment
• Enrolls students, tracks and displays progress,
  manages individual study plans for each student

16
FtL Status

• Now in 50 Colorado classrooms
• Formal assessment in 40 K-2 classrooms 2
  computers per classroom, 20 min per day per
  designated students
• About 1/3 ESL students (one school has ALL native
  Spanish speaking students)
• 10 Special Ed classrooms for remedial instruction
  of students with cognitive disabilities
• Learning gains in kindergarten and first grade
  classrooms kids love the program

17
LSVT Lee Silverman Voice Treatment
If only we could hear and understand her --
family of Lee Silverman
18

Parkinsons disease 1.5 Million individuals US
alone Over 6 million worldwide 89 have a speech
or voice problem(Logemann et al.,1978)4
receive traditional speech therapy(Hartelius
Swenson, 1994 Oxtoby, 1982)1990 Consensus
Speech treatment does not work(Sarno, 1968
Allan, 1970 Green,1980 Aronson, 1990 Weiner
Singer, 1989)

19
Perceptual Characteristics of Speech
Reduced loudness Hoarse voice quality Monotone Imp
recise articulation Vocal tremor Some patients
report volume, hoarse voice or monotone as the
first PD symptom (Aronson, 1990)
20
If you dont talk loud enough, people stop
listening -Individual with Parkinson
Disease Boston, May 1996
21
(No Transcript)
22
(No Transcript)
23

• To a patientmajor life impact
• My voice is alive again
• I can talk to my grandchildren!
• I feel like my old self
• I am confident I can communicate!

24
(No Transcript)
25
Unexpected outcomesSystem-wide spread of effect

• Benefits to
• Articulation Swallowing
  
• Speech Rate Face
• Speech Motor Stability PET
• (Spielman, et al. 2002 El-Sharkawi, 2002
  Spielman et al., 2003 Kleinow et al., 2001
  Liotti et al., 2003)

26

• To a patientmajor life impact
• My voice is alive again
• I can talk to my grandchildren!
• I feel like my old self
• I am confident I can communicate!

27
Phonation Task - PD N5
Pre-LSVT
SMA
DLPF9
R a Ins
R Put
60
34
4
34
10
4
Post-LSVT
L
R
34
60
10
4
34
z-score
-4 2.25 4
28
LSVT Applications

• Parkinson Plus (Countryman et al., 1994)
• Post Surgery, Fetal cell (Countryman, et al.,
  1993)
• Stroke (Fox et al, 2002 Will et al., 2002)
• Multiple Sclerosis (Sapir et al., 2001)
• Ataxia (Sapir et al., 2003)
• Cerebral palsy (Fox, 2002)
• Down Syndrome (Robinson et al., 2004)
• Aging (Ramig et al., 2001)

29
LSVTVT Hi-Lo pitch
LSVTVT-Loud AH
LSVTVT-Read Aloud
LSVTVT-Loud Phrases
30
(No Transcript)
31
LSVT VT Status

• System being tested on three individuals with
  Parkinson disease
• Patients use system during 10 of 16 1-hour
  sessions using system
• Patients enjoy using the system
• STATUS Clinical trials begin Sept 14 05

32
Virtual Therapists for Aphasia

• The three systems described next were based on
  speech and language treatments developed for
  individuals with Aphasia
• C-COSTA (Developed by Leora Cherney, Chicago
  Rehabilitation Institute
• ORLA (Leora Cherney)
• TUF-T (Developed by Cynthia Thompson,
  Northwestern University

33
C COSTAComputerized Oral Scripts for Teach
Aphasia

• Develop conversational scripts personally
  relevant to the individual with aphasia
• A sequence of sentences that a person typically
  speaks in routine communication situations
• Ordering pizza over the phone
• Making a doctors appointment
• Implement and evaluate the computerized
  intervention relative to human therapists
• STATUS 5 subjects tested

34
Oral Reading for Language in AphasiaORLA

• ORLA is a speech-language treatment protocol for
  patients with aphasia (Cherney et al., 1986),
• A multi-modality stimulation approach that
  involves
• listening to a sentence, tapping along with the
  rhythm of the sentence, repeated practice saying
  the sentence in unison with the clinician and
  then independently.
• Studies indicate improvements in oral expression,
  auditory comprehension, and written expression
  (Cherney et al., 1986).
• STATUS 5 subjects tested

35
Treatment of Underlying Forms (TUF) for Aphasia
by Cynthia Thompson at Northwestern University

• Individuals with Brocas aphasia have difficulty
  comprehending and producing sentences,
  particularly sentences with complex syntax.
• Training simple sentence types fails to
  generalize to untrained sentence types and
  contexts.
• Research with TUF demonstrated generalization
  occurs to sentences that involve similar movement
  properties.
• For example, It was the thief who chased the
  artist results in improved production and
  comprehension of wh-questions such as Who did
  the thief chase?
• STATUS Testing Prototype