Collaborative Annotation, Archival and Visualization in a Biofeedback Rehabilitation system

1
Collaborative Annotation, Archival and
Visualization in a Biofeedback Rehabilitation
system

• Hari Sundaram
• Arts Media and Engineering
• Arizona State University

2
Introduction

• Motivation
• Every 45 seconds, someone in the United States
  suffers a stroke. It results in functional
  deficits of neuropsychological and physical
  functions in post-stroke survivors.
• Up to 85 of patients have a sensorimotor deficit
  in the arm, such as muscle weakness, abnormal
  muscle tone, abnormal movement synergies, and
  lack of coordination during voluntary movement
• Goal
• Design a real time multimodal biofeedback system
  for stroke patient rehabilitation.
• Archival / annotation and information
  visualization to provide insight.

3
The Biofeedback system

• To appear in acm mm 2006

4
System Overview

• The Biofeedback system system situates
  participants in a multi-sensory engaging
  environment, where physical actions of the right
  arm are closely coupled with digital feedback.
• The Biofeedback system integrates five
  computational subsystems.
• Motion capture
• Motion analysis
• Audio feedback
• Visual feedback
• Database for archival and annotation
• All five subsystems are synchronized with respect
  to a universal time clock.

5
Action Analysis

• Arm Representation
• 11 labeled markers on arm and torso
• 3 labeled markers on the back of chair
• Feature Extraction
• 3D hand trajectory / 3D hand trajectory relative
  to the predefined straight line
• Shoulder / Elbow extension
• Hand Orientation
• Shoulder rotation / abduction/elevation
• Trunk flexion / rotation / lean and shoulder
  trajectory
• Wrist extension
• Multi-goal Framework
• Reaching
• Opening
• Flow

6
Coupling Action to Feedback

• Engagement
• Aesthetically attractive, easy to use and
  intuitive.
• Message and Mapping
• Reaching - visual target, an image
  completion/reassembly task, and an accompanying
  musical progression.
• Flow - pointalistic sound clouds in the main
  musical line, flowing particles in the visuals
• Opening - a rich, resonant musical accompaniment.
  
• Environment
• Introduction ( visual )
• Abstract I (visualaudio)
• Abstract II (visualaudio), more variation

7
Audio Feedback

• Dynamic mapping of the normalized distance to
  target along the z coordinate to harmonic
  progression.
• Map the hand trajectory velocity in the z
  direction to event density.
• Joint Synchrony and Harmonic Progression.
• Shoulder - woodwind sounds (flute, clarinet,
  bassoon) through the progression
• Elbow - string sounds (a violin section of
  tremolo, a violin section, and a pizzicato
  violincello section).
• Mapping of Shoulder and Elbow Extensions
• Midi velocity (Mv)
• Duration (td)
• The probability of an octave doubling (Pd)

8
Visual Feedback

• Transition Environment
• 3D virtual environment
• Physical movement will control the virtual
  environment.
• Abstract Environment
• A picture in a frame
• Explosion
• Turbulence
• Horizontal and Vertical Pull

9
Validation

• Offline Segmentation
• Reaction
• Reaching
• Grasping
• Returning
• Spatial Error
• Target-Hand Distance
• Hand Orientation
• Arm Openness
• Should Openness
• Elbow Openness
• Reaching Duration
• Flow Error
• Zero crossing number
• Polynomial curve fitting error
• Consistency

10
Results

• Flow - the smooth of speed curve means three
  things
• Subjects are clearer the goal and they need not
  hesitant what will happen.
• Subjects are clearer about the feedback cue.
  Based on the current feedback and their memory,
  they can easily find the way to reach the target.
• Subjects start following the rhythm, that is
  mapped in the audio feedback.

• Openness - our audio feedback design for the
  abstract environment can help subjects with more
  openness.

• Reaching - our visual-audio feedback design can
  guide the normal subject to do the reaching as
  accurately as they did in real world.

11
Archival Sub-system
12
Overview

• Challenges
• Continuous data streams and large datasets
• Real-time annotation has high cognitive load
• We are integrating an archival subsystem into a
  team with different domain experts.
• Our Approach
• Continuous multimodal archival
• Real-time collaborative annotation
• Offline information visualization

13
Archival Subsystem Design

• Part of our overall Biofeedback system
• Manage multimodal data streams
• Different data transport rates (total 1.89MBps)
  
• Scalable Multicast Network
• raises synchronization problem

14
Continuous Multimodal Archival

• We split computational and storage resources into
  two archival subsystems
• Archiving parametric system models
• Raw motion capture data
• Motion analysis parameters
• Audio-visual synthesis parameters
• Data was multicast
• Contextual media capture
• Seven channels
• Actual audio-visual feedback data
• Three microphones
• Video camera
• Hardware soundboard, microphones, VGA monitor
  scan-converter, video camera, mpeg hardware
  encoder, due-core server

max / msp graphical program
15
Database design

• Indexing Scheme
• The patient / session / set / trial hierarchy
• Universal time stamp of synchronized subsystems
• Structural DB tables
• Motion capture and analysis parameters
  categorization
• Group audio-visual data by feedback semantics
• We first stream parametric data into a
  multi-buffered queue, then write to DB using bulk
  insert in parallel.
• We keep reference of multimedia data.
• Privacy issue

16
Real-Time Collaborative Annotation

• Why emphasize real-time collaborative?
• Annotations are time critical
• Each trial is short (5 sec.)
• there can be many unexpected events in this
  period can cause cognitive overload.
• Team is focused on the experiment!
• Design goals of the annotation tool
• Distributive
• Personalized (Multi-disciplinary team)
• Collaborative

17
Annotation Interface

• Design Elements
• Dynamic experiment progression indicators
• Domain specific checklist
• Collaborative annotation sharing
• We multicast annotations from one client to
  others
• Random query, retrieval and modification
• User feedback is very positive

18
Information Visualization

• Offline visualization for review / annotation of
  archived data.
• Our design goals
• Hierarchical and selectable motion parameters /
  evaluation metrics navigation
• Synchronized contextual information playback
• Facilitates annotation modification
• Helps domain experts share information and
  improve their subsystems

19
Visualization Prototype

• Features
• Allows navigation through our trial hierarchy on
  motion analysis results
• Contextual media playback with parametric motion
  analysis visualization
• Provides offline annotation facilities

20
Open Issues

• Event Model
• Event definition by domain experts
• Event detection
• Event Network Modeling
• Pre-emptive Annotation
• Show events with high priority
• Event log
• SenseCam Integration
• SenseCam pictures can be integrated into our
  visualization framework

21
thanks

• Team Weiwei Xu, Yinpeng Chen, Richard Wallis,
  Thanassis Rikakis, Hari Sundaram, Todd Ingalls,
  Loren Olson, Jiping He, Sharon Liu