Immersive Virtual Humans for Interpersonal Skills Education - PowerPoint PPT Presentation

Loading...

PPT – Immersive Virtual Humans for Interpersonal Skills Education PowerPoint presentation | free to view - id: 1d80fe-ZDc1Z



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Immersive Virtual Humans for Interpersonal Skills Education

Description:

Immersive Virtual Humans for Interpersonal Skills Education – PowerPoint PPT presentation

Number of Views:104
Avg rating:3.0/5.0
Slides: 49
Provided by: benjam66
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Immersive Virtual Humans for Interpersonal Skills Education


1
Immersive Virtual Humans for Interpersonal Skills
Education
  • Benjamin Lok
  • Computer and Information Science and Engineering
    Department
  • University of Florida
  • University of Central Florida
  • November 30th, 2007

2
Virtual Humans
  • A virtual character
  • Human form
  • Application domains
  • Games
  • Movies
  • Simulation

3
Virtual Human Experiences
  • Not as a substitute interface
  • But the experience is the interaction with the
    virtual human itself
  • Fundamentally different than other VEs
  • Research goal Through interacting with virtual
    humans, can we
  • Teach
  • Affect
  • Change

4
Immersive Virtual Humans for Educating Medical
and Pharmacy Communication Skills
  • K. Johnsen, A. Raij, B. Rossen, A. Kotranza, X.
    Wang, B. Lok
  • Computer and Information Science and Engineering
  • J. Cendan
  • Community Health and Family Medicine D. Beck, C.
    Kimberlin
  • Pharmacy
  • R. Ferdig
  • Education
  • A. Deladisma, D. S. Lind
  • Surgical Oncology
  • S. Chapman, L. Bracegirdle
  • Pharmacy

5
Can Virtual Humans Enable
6
VOSCE Project Overview
  • Started Spring 04
  • n gt 327 students
  • Medical
  • Nursing
  • Physician Assistant
  • Pharmacy
  • One of the most popular VH/VR experiences
  • Three institutes
  • University of Florida
  • Medical College of Georgia
  • Keele University (U.K.)
  • Team
  • VR/HCI/CS 1 PhD, 6 grad students, 2 undergrads
  • Medicine 6 MDs, 2 medical students
  • Education 2 PhDs
  • Research focus on interfaces

7
T Interpersonal Simulator (IPS)
  • Platform for exploring interpersonal interactions
    with VHs
  • Natural interaction with a high level of
    immersion
  • Real-world application Medical interview
    training
  • Simulate a standardized patient encounter
  • Virtual patient, DIANA
  • Virtual instructor, VIC
  • Communication skills

8
  • Play Video XXX

9
Why Virtual Humans?
  • Students
  • Repetition
  • Feedback
  • Longitudinal learning
  • Educators
  • Standardization
  • Dynamic
  • Abnormal findings
  • Cultural competency
  • Aggregate performance
  • Researchers
  • Study the extent of impact of VHs
  • Easy to run studies (Twiddle one thing)

10
IPS System
  • Inputs
  • Natural speech
  • Tracking data (head, hands, chair, tools)
  • Video
  • Physiological measures
  • Outputs
  • Speech and animation
  • Life-size projection (or HMD)
  • Perspective correct rendering
  • Reactive virtual human
  • COTS components
  • 3 PCs
  • 2-4 video cameras
  • Data projector, large-screen TV, or HMD
  • Wireless microphone
  • Bodymedia Sensewear
  • lt 10,000(USD)
  • Potential
  • Every Hospital

11
Natural Interaction
  • No Keyboard, No Mouse
  • Speech Recognition
  • Dragon Naturally Speaking 9 Pro
  • Accuracy 90 with 10 minutes training
  • 50-70 match to database
  • Track Communication Cues
  • Non-Verbal (posture)
  • Verbal
  • Physiological
  • Gestures (pointing, handshake)
  • DIANA and VIC look at user
  • Why this works
  • Does not rely on complete sentences
  • Constrained scenario
  • Students trained on specific questions
  • Johnsen VR2005, Presence

12
Studies
  • (- July 2007) 14 studies
  • Focus
  • Validity
  • Similarity/differences
  • Course integration

13
VP ? SP
  • How is experiencing an interpersonal scenario
    with a virtual human similar to and different
    from experiencing an interpersonal scenario
    with a real human?
  • Clearly different, but in what important ways?

14
VP ? SP
  • n 57
  • Two studies
  • (between-subjects) Students interviewed VP or SP
  • (within-subjects) Students interviewed both VP
    and SP
  • Measures
  • Content of interaction
  • Expert video coding (student behavior, empathy,
    rapport)
  • Post-experience questionnaires
  • Goal is to identify
  • Extent to which interpersonal scenarios can be
    simulated with VH
  • Component technologies which need to improve to
    enable effective VH systems
  • Raij IEEE VR 2006,TVCG 2006
  • Stevens, Southern Group on Educational Affairs
    2006

15
VP cSP (c0..1)
Group V n 49
Group R n 33
  • Overall content similar
  • Questions asked
  • Pain sharp and stabbing (R 100, V 80)
  • Sexually active (R 54, V 45)
  • Global measures
  • Education goals met
  • Students rated educational merits similarly
  • Students rated difficulty similarly
  • Tool to augment SPs

10 minute Interview VH
10 minute Interview SP
Experts assess participants
16
Social Conventions
  • Conversation flow is rapid-fire
  • Confirmatory phrases (SP 20, VP 3.5)
  • Rapport-building different
  • Empathetic moments
  • Im scared, can you help me?
  • Could this be cancer?
  • Sneezing
  • Responses
  • Exists but reduced quantity quality
  • Expert ratings higher of SP experiences
  • Spontaneous empathy (R 84, V 24)
  • Nonverbal behavior different
  • Body lean (R 2.9 0.9 , V 2.1 0.6, p lt
    0.001)
  • Deladisma, American Journal of Surgery 2007
  • Cohen, Southern Group on Educational Affairs
    2006, Association for Surgical Education 2006

17
Validity
  • n33
  • Each student interviewed VP and SP with similar
    complaint
  • Interaction was rated by expert
  • Information (r0.34, r20.12)
  • Process (r0.19, r20.03)
  • Quality (r0.38, r20.14)
  • Overall (r0.50, r20.25, plt0.005)
  • Predictor
  • Good w/ VP Good w/ SP
  • Bad w/ VP Bad w/ SP
  • Valid experience
  • Despite technology, believability
  • Proven valid VH experience!
  • But measures can be misleading
  • Global vs. local
  • Highlights current limitations
  • Johnsen, CHI 2007
  • Bernard, Society of Medical Simulation Meeting
    2006

18
Characterizing a Human-Virtual Human Interaction
  • Self-reporting measures
  • Easy
  • Minimal power
  • Expert measures
  • Video coding and evaluation
  • Powerful
  • Logistical issues
  • Need a tool/metric that is
  • Quantitative
  • Encompassing
  • Why?
  • Evaluate What is a good
  • How good is it?
  • What makes it good?
  • Limits How good can it be?
  • For each interaction
  • Sensors log
  • Analyze log file for data

19
Sample the Interaction
Interaction Log File
Tracking Data User (posture, gaze) Tools Physiolog
ical Data
20
Interaction Log File
Tracking Data User (posture, gaze) Tools Physiolog
ical Data
21
Interaction S(Interaction Signals)
  • Challenges
  • What signals matter?
  • How to sample signal?
  • Data can be
  • Missing
  • Noisy
  • Discritized
  • How to visualize signals for users?
  • Evaluate?
  • Interpret interaction as a signal
  • Gain insight from signals via
  • Signal analysis
  • Pattern recognition
  • Visualization
  • Data mining

22
Interaction S(Interaction Signals)
  • User and virtual human talking
  • What topics does the user focus on?
  • Does the user interrupt the virtual human?
  • Empathy
  • How well does the user respond empathically?
  • User behavior
  • How much does posture vary?
  • Where does the user gaze?
  • Evaluation
  • How does an interaction compare to previous
    interactions (student portfolio)?
  • How does an interaction compare to others?
  • Can we identify poor performing students?
  • Goal Automatic evaluation and feedback

23
Interactive Visualization
  • Explore Signals
  • Display
  • Playback
  • Multiple views
  • Since we log the conversation
  • We can recreate any moment of the conversation
  • Novel views

24
IPSViz
  • n 27
  • After an interaction, students receive email and
    dload interaction
  • Analogous to flight simulators
  • Users
  • Students
  • Self-reflection
  • Feedback
  • Educators
  • Quantitative analysis
  • Identify trends and outliers
  • Researchers
  • Capturing H-VH interaction
  • Analysis
  • Process, Filter, Visualize

25
After Action Review of H-VH Interactions
  • Users do not remember how they behave in H-VH
    interactions.
  • Participants self-reported that (7-point Likert)
  • they did not appear as friendly (pre 5.5, post
    4.6, plt0.001)
  • or act naturally to the VH
  • (pre 3.8, post 2.7, plt0.001)
  • Students reevaluated their empathy with a VH
    (scale 1-4)
  • (pre 2.5, post 2.2, plt0.001)
  • Students reported specific ways they would act
    differently in future real-world interactions.
  • It was good to watch myself. See how I act, and
    to hear my voice, how I ask questions the
    intonation in my voice. And after doing it I
    can think about it more, you know, things I could
    have done. And then when I go back and view, I
    can see ways I could have changed.

Raij VR 2008
26
See What It Was Like Talking To Yourself!
  • Viz Vid

27
IPSVizn Log files for a student
  • Student
  • Portfolio of VH interactions
  • Accreditation, track learning
  • Educator
  • Evolution of performance
  • Researcher
  • Longitudinal learning

28
IPSVizn Multiple students log files
  • Educator
  • Aggregate performance
  • Identify trends/outliers
  • Researcher
  • Examine logistically complicated topics
  • E.g. Uncanny Valley

29
Physiological Measures
  • We need new measures to characterize H-VH
    interactions
  • Physiological measures
  • Blood pressure
  • Heart rate
  • Heat Flux
  • Skin Temperature
  • Galvanic Skin Response
  • Bodymedia Sensewear
  • We can then look at
  • Are VH affecting people?
  • Are VH changing people?
  • Real-time input to simulation
  • Scenario
  • Anxiety (sex history)
  • Empathy challenges
  • Study n27
  • Experience correlated to anxiety measures w/ a VH!

VHs challenge Can you help me? Im really
scared.
VHs challenge Could this be cancer?
30
Sexual History
  • n9, SP only
  • n10, VPSP
  • Measures Blood pressure, heart rate
  • ? SBP 9.31 (SP only), -0.47 (VPSP)
  • ? HR 7.76 (SP only), -0.47 (VPSP)
  • Taking a sexual history with DIANA makes a
    student less nervous with taking a sexual history
    with a SP
  • Self-reported anxiety was not different
  • DIANA is changing people, but they dont know it
  • Mack, Society for Simulation in Healthcare 2007

31
Bias IPSViz
Interact with VP 1
Interact with VP 2
32
Bias
  • Multiple log files per student
  • Visualize bias trends differences
  • Study n 27
  • Interact with either a Caucasian or
    African-American VH
  • Explicit and Implicit Bias
  • Psychology measures (IAT, Friendliness scales)
  • Correlation between users opinion of VH w/
    measures
  • VHs can elicit racial biases
  • Study (n9), people reported that the
    African-American VH had less education and money
  • Identical animations, words spoken
  • Differences skin-tone, voice
  • Other biases to study
  • Race/Ethnicity
  • Gender
  • Age
  • Weight
  • Social hot topics

33
Signal Analysis of Bias
  • Audio signal analysis
  • A students similar audio with two VHs
  • Identify differences

34
Component Evaluation
  • Study how system components impact interaction
    with a VH
  • Recorded Speech vs. Text-to-Speech
  • Similar content
  • Less engagement with TTS
  • Surprisingly, speech understanding does not need
    to be that good
  • Dickerson, SIMCHI 2006, MMVR14
  • HMD vs. Projector Displays
  • HMD more engagement and immersion
  • Increased encumbrance
  • Perceptions of DIANA affected
  • Self-evaluation of empathy
  • Johnsen VR 2008

35
Mixed Reality Humans
  • Virtual humans have limitations
  • Open research problems (e.g. AI, speech)
  • No tactile feedback
  • Merge the real and virtual spaces
  • Real tools
  • Real simulators
  • Complete patient interaction
  • Physical Exams (e.g. eye exam)
  • Point at eye chart
  • Ophthalmoscope
  • Follow my finger
  • 1 or 2 fingers?

36
Integrate VP with Simulators
  • Medical simulators (e.g. mannequins) are becoming
    accepted educational tools
  • However, they
  • Focus on a small set of tasks
  • Interaction is different than clinical practice
  • Integrate virtual patient with simulators
  • Interview virtual patient
  • Interact with simulator
  • VP responds to simulator

37
Breast Simulator Integration
  • Breast simulator integrated
  • Dr. Carla Pugh, Northwestern
  • Student does a patient history
  • Asks to remove gown (physiological measures)
  • Performs a breast exam
  • VP winces at too much pressure
  • Future work
  • Pelvic simulator
  • Central line simulator
  • Kotranza VR2008

38
Abnormal Findings
  • Conditions difficult to represent with existing
    education methods
  • Conditions
  • Psychomotor
  • Neurological
  • Age/race/ethnicity dependent conditions
  • Blurry vision scenario
  • Cranial Nerve III (due to brain tumor)
  • Corneal ulcer
  • Retinal detachment
  • Benefits
  • Curricular planning of medical student exposure
  • Supplement SP experiences
  • Leverage the dynamic nature of VPs
  • Play Video

39
Current Work
  • Classroom incorporation
  • Communication course
  • 2nd year MS
  • Surgical rotation 3rd year MS
  • Pharmacology 1st year
  • ngt120 per year
  • Potentially 30,000 interactions a year _at_ existing
    institutions
  • Virtual Instructor
  • Real-time response to tracked cues
  • Physiological measures
  • Posture cues
  • Verbal cues

40
Museum of Science and Industry
  • Science museum in Tampa, FL
  • 5th largest in US (75,000 sq ft.)
  • 1 million guests
  • Integrate a VH interaction
  • The Amazing You exhibit
  • Lifelong wellness information
  • Fall 08
  • Inspire, Inform
  • Explore race, VH bias

41
Scenarios and VPs
  • Acute abdominal pain
  • Breast mass
  • Dyspepsia
  • Sexual history
  • Eye exam
  • Patients
  • DIgital ANimated Avatar (DIANA)
  • Elderly Diana (Edna)
  • Manniquin Diana (Mandi)
  • Building the following VPs
  • Male
  • Personality (e.g. irate)
  • Intelligence (e.g. mentally retarded)
  • Appearance (e.g. disfigurement, limbs, burns)

42
Pain Studies
  • Prof. Mike Robinson, Clinical and Health
    Psychology
  • Perception of pain
  • 1 Independent variable

43
Augmented Reality Medical Devices
  • Combine the abstract (Virtual Anesthesia Machine)
  • With the concrete (Anesthesia Machine)
  • Bridge learning

44
Milestones
  • August 2004 First participant
  • October 2005 First angry student
  • May 2007 First participant who expressed a
    behavior change
  • October 2007 First crier

45
Virtual Experiences Research Group
  • PhD Students
  • Kyle Johnsen, Aaron Kotranza, John Quarels,
    Andrew Raij, Xiyong Wang, Brent Rossen
  • Undergraduates
  • Harold Rodriguez, Anna Vittone
  • Funding
  • National Science Foundation (CAREER, REU),
    University of Florida Colleges of Engineering and
    Medicine, Medical College of Georgia, Keele
    University, School of Pharmacy

46
Graduating Students (looking for post-docs)
  • Kyle Johnsen
  • Validation
  • Tracking
  • Study Design
  • http//www.cise.ufl.edu/kjohnsen
  • Andrew Raij
  • Virtual Human Human?
  • Visualization
  • http//www.cise.ufl.edu/raij
  • Harold Rodriguez
  • Graduate school

47
Join Us!
  • We need your expertise!
  • Looking for grad students
  • Additional locations to install and test system
  • Thank you!
  • http//www.cise.ufl.edu/research/vegroup
  • Questions?

48
Studies
  • 2004 (n20)
  • April Project initiated
  • August Prototype (n7) UF
  • October Experts (n3) UF
  • December Pilot Test (n10) UF
  • 2005 (n81)
  • June Two Institutions (n16) UF/MCG
  • July VP vs SP (n16, n8) UF/MCG
  • October Cultural Bias (n16) MCG
  • October Class Integration (n33) UF
  • 2006 (n27)
  • November (n12) MCG, (n15) Keele
  • 2007 (n199)
  • February (n16) MCG
  • Spring Keele (n47)
  • May (n27) MCG, 23 med, 4 pa
  • July (n10) MCG, pa, testing aarons stuff
  • July-October (n60) MCG, basic system (Adeline
    running them)
  • October (n39) UF Pharmacy, WoZ
About PowerShow.com