Some Usability Issues of Augmented and Mixed Reality for e-Health Applications in the Medical Domain

1
Some Usability Issues of Augmented and Mixed
Reality for e-Health Applications in the
Medical Domain

• Reinhold Behringer
• Johannes Christian
• Andreas Holzinger
• Steve Wilkinson

2
Purpose of this Paper

• Highlight AR and MR applications in the Medical
  Domain.
• Point to usability issues.
• Not a complete survey.

3
Augmented Reality (AR)

• Principle
• Computer generates output, which is fused with
  the human perception of the environment.
• Appears to emanate from the environment itself,
  to be a part of it registration with real
  environment.
• Visual
• 3D computer graphics, employing Virtual Reality
  (VR) concepts.
• Acoustic
• Spatial audio.

4
Mixed Reality (MR)

• Real environment is not necessarily the basic
  framework.
• Virtual Environment (VE) is the main framework.
• Objects in MR can be computer-generated or real.

5
AR and MR in Medical Domain

• AR
• Data visualisation directly on patient.
• Guidance to physician for medical procedures.
• MR
• Simulation and training.
• Patient itself can be simulated.
• Benefits improved situational awareness

6
Display Types for AR

• AR fusion of visual sense
• Head-worn displays
• Ideal merging of computer output with visual
  impression of surroundings.
• Hand-held displays
• Either acting as frame into the real world, or
• Showing representation of real world by video /
  images, onto which information is overlaid.
• Projective displays
• Project information directly onto object
  (patient).
• 3D is only correct for one single user at a time.
• Requires consideration of projection surface.

7
Tracking of User

• Users viewing position and orientation needs to
  be tracked for correct display rendition.
• Technologies
• Active illumination outside of human-visible
  spectrum (e.g. IR).
• Computer vision approaches, capturing either
• Environment as seen from user (head-worn camera),
  or
• User himself, seen from cameras in the
  environment.

8
Examples of Applications in Medical Domain

• Data projection onto patients.
• Support of surgery.
• Simulation tools for teaching and practising.
• Therapy and rehabilitation.

9
Visualisation of Ultrasound and CT

• UNC, A. State (1992).
• Live image stream of ultrasound scans onto body
  pre-natal care, obstetric examination.
• Initially 2D scans, later volumetric scans.
• Technique also used for needle biopsies and
  minimally invasive surgery.
• Stockmans (2005).
• Correction of bone disformities.
• Comparison of before-after procedure.

UNC AR demonstration
10
Simulation

• Sielhorst et al., TU Munich (2004)
• Birth simulator, using MR techniques.
• Real object model of torso.
• Graphical simulation baby.
• Birth pliers as haptic interface for interaction.
• Nestler et al., TU Munich (2007)
• Virtual patients for large-scale disaster
  training.
• Table-top display, individual patients with
  different injuries.
• Wilkinson (2005)
• Using purely VR for simulation of surgical
  procedure on hand.
• Goal to educate patients and reduce fear.

11
Aiding Surgery

• Not yet practical in clinical applications, but
  promising improvements of spatial orientation,
  allowing more radical operative therapy.
• MEDARPA project
• AR and VR supporting minimally invasive surgery,
  to aid the navigation of the surgeon
  (bronchoscopy and brachytherapy).
• Future use with surgical robot.
• Reitinger, TU Graz (2005)
• Virtual liver surgery planning system, aiding in
  providing more precise measurements for tumor
  treatment.

12
Therapy, Rehabilitation

• Using VR to cure phobias.
• Learning in controlled graphical environment how
  to deal with phobia-causing situations.
• Rehabilitation of motoric skills
• AR system evokes motor images,
• Guides motoric practise
• Healthy living
• Persuasive Mirror shows future projection of
  self, under influence of varying live style.

13
Edutainment

• Teaching anatomy
• electronic book for interactive display of 3D
  anatomic perspective of humans.
• Organs can be viewed from different angles.
• Simulation
• To reduce fear of the unknown.

14
Registration and Tracking

• Requirements
• View of patients body needs to be unobstructed
  for the physician.
• Tracking needs to take into account deformable
  tissue of human body.
• Calibration effort needs to be small, so as not
  to distract from medical procedure.
• Tracking needs to be resilient to occlusion.
• Markers for visual tracking can be attached to
  human body.
• Magnetic tracking can provide seamless tracking
  of physician.

15
Displays

• Head-attached
• Allows hands-free activity.
• Low resolution and viewing angle.
• Example Variscope (Birkfellner, TU Vienna)
• Hand-held
• Is currently available at reasonable cost and
  specifications.
• May interfere with actual procedures.
• Tablet display on boom
• Allows hands-free operation, while at the same
  time being cost effective.
• Can be semi-transparent or video-see-through.
• Both display and physician need to be tracked for
  correct visual view.
• Spatial.

Variscope
MEDARPA display
16
Interaction

• Traditional interfaces (mouse, keyboard) are not
  suitable, as they distract from the task.
• Preferable
• Automatic ubiquitous interface, acting in the
  background.
• E.g. speech recognition, gesture recognition.
• Possibly integrated into medical instrument.

17
User-Centred Development

• Potential benefits of AR/MR are obvious.
• But for studying usability, practical workflows
  need to be studied.
• AR application developers need to understand the
  workflows of medical professionals.
• Usability Engineering Methods (UEM) are becoming
  more important, but are still not applied very
  often
• Software engineers work on the implementation,
  usability experts work on design.
• Rarely do they collaborate.

18
User-Centred Development

• In most cases
• Usability of AR environments is evaluated.
• E.g. Hix
• Criteria for assessing VE design.
• Usability of standard user interfaces for VE.
• Effectiveness of different interactive devices.
• Needed
• User-Centred Development.

19
Presence and Errors in VE

• Specific to VEs
• Sense of presence (immersion).
• Assessed through questionnaires, focusing on a
  persons sense of presence.
• Presence is not associated with task performance
  (Slater).
• But measures of presence have concentrated on
  user perception of VE technology.
• Errors in VEs can impact significantly the
  perception of presence

20
Immersion

• Effects of being immersed
• Investigated in a variety of VE system
  configurations
• Collaborative VEs, viewpoint of self, perception
  of others presence.
• Fully immersive VEs have been well investigated,
  focusing on head-worn displays.
• Missing
• Techniques for measuring presence.
• Questionnaire-based assessment may be biased.

21
Concerns about AR

• Problematic
• Information overload can be accompanied by
  sensory overload.
• Overwhelming experience in AR environment.
• Possible lack of acceptance of this technology.
• Social effects (beliefs, attitudes, feelings).

22
Suggestions for Further Study

• Which are the most appropriate interaction
  metaphors in AR for medical domain?
• Which influences do AR applications have on
  performance of the end users in the medical
  domain?
• What are the effects of adaptation that people
  might have to make cognitively to believe in and
  cope within an AR environment?
• How will multimodal interaction through a number
  of input and output channels enhance or detract
  from the reality/Virtuality experience?

23
Specific to AR

• AR must deal with
• physical interactions,
• social interaction,
• cognitive interactions.
• Is difficult to isolate variables.
• Less performance constraints.
• Lower predictability of behaviors.

24
Successful AR Application

• All involved research domains have to be
  considered.
• User-centered design focus is important.
• Need to be
• Accessible.
• Usable for everyday end users.
• Follow notions of pervasive and ubiquitous
  computing.
• Implement basic ideas of social software.
• Designed for users without deep IT knowledge.

25
Summary

• AR and MR have excellent potential in medical
  applications.
• Usability issues need to be addressed in
  application design
• Not only study usability in experience of VE
  environments.
• Need to apply usability engineering methods in
  design.

26
Acknowledgment

• Johannes Christian
• Andreas Holzinger
• Steve Wilkinson