Tradeoffs between automatic postural adjustments and orienting responses as indices of cognitive eng

1
Trade-offs between automatic postural
adjustments and orienting responses as indices of
cognitive engagement

• Carey D. Balaban
• Departments of Otolaryngology, Neurobiology and
  Communication Sciences Disorders
• University of Pittsburgh

2
Postural Control and Cognitive States

• Multimodal integration of visual, vestibular,
  proprioceptive, haptic and auditory information
• Automatic
• Context-dependent
• Predictive or reactive
• Reflects trade-off between voluntary movement and
  postural maintenance

3
Real-time Dynamic Analysis of Seated Posture

• Detect voluntary movements
• Task-related
• Spontaneous (including fidgeting)
• Identify automatic postural responses
• Reactive (vehicle, self or substrate movt)
• Predictive (vehicle, self or substrate movt)
• Background activity

4
Dynamic Postural Assessment Chair

• Computer Workstation Environment
• 16 X 16 arrays of capacitive pressure sensors in
  seat and seat back
• 1 inch sensor spacing
• Sampling rate 4.5 Hz
• Ultrasonic head tracker
• 6 degrees of freedom (x,y,z,roll angle, pitch
  angle and yaw angle)
• Sampling rate 50 Hz
• Referenced to computer monitor
• Motion sensor (to be added)

5
Dynamic Postural Assessment Chair

• DCC S-class Mercedes
• 16 X 16 arrays of capacitive pressure sensors in
  seat and seat back
• 1 inch sensor spacing
• Sampling rate 4.5 Hz
• Ultrasonic head tracker
• 6 degrees of freedom (x,y,z,roll angle, pitch
  angle and yaw angle)
• Sampling rate 50 Hz
• Transmitter mounted on ceiling above driver

6
Dynamic Postural Assessment Chair Patent
Application Serial No. PCT/US04/14158
7
Dynamic Postural Assessment ChairPatent
Application Serial No. PCT/US04/14158
8
Dynamic Postural Assessment Chair Patent
Application Serial No. PCT/US04/14158
9
Approach

• Dynamic rather than static analysis of posture
• Use normal automatic behavioral templates
• Postural sway (e.g., spontaneous anteroposterior
  and lateral weight shifts in seat)
• Anticipatory shifts (e.g., leaning prior to turns
  or accelerating of a car)
• Cognitive engagement responses (e.g., orienting
  movements relative to computer monitor)
• Integrate sensors with Cognitive Models Active
  contextual gauges

10
Sensors versus Gauges

• Sensors
• Single modality
• Concrete read-out
• Ambiguous without context
• Gauges
• Integrated from multiple sensors
• Requires context to infer cognitive state
• Can vary in degree of abstraction

11
Gauge Development Stage 1

• Identify automatic postural responses during task
• Static platform
• Warship Commander Task environment
• Balaban et al., Int. J. Human-Computer
  Interaction, 17 (2), in press.
• Transferred to LM-ATL Aegis platform

12
Seat Sensor Example TH1 Pseudocolor of
instantaneous pressure
13
Engagement Response Head position and seat
center of pressure (COP) a linear function of
number of tracks
14
Engagement Response Head position
15
Epochs (8 sec duration) where at least 50 of
variance (r0.71) explained by linear
relationship between AP head position and number
of tracks or by left (blue dots) or right (red
circles) seat COP. Head and body engagement are
partially independent
16
Back Bracing Gauge SD of dP/dt across Back Pad
17
Transfer to LMATL Aegis Task

• For WCT the waves are
• Cued
• Dominant feature of display
• Rapid changes in number of tracks (mean ? 2 sec)
• For LMATL Aegis Environment the waves are
• Uncued (from operators perspective)
• Embedded in a more complex display
• Slow change in number of tracks (mean ? 12 sec)

18
AugCog December LM CVE

• Postural engagement gauge
• Detected linear association between head position
  or seat COP (center of pressure) movement (re
  screen) and the number of tracks
• Time window 12 sec (mean median target change
  duration)
• Criterion for engagement coefficient of
  determination 0.50 (explains 50 of variance)
• Calculated the percentage of sub-sampled time
  points (at 2 Hz) exceeding criterion

19
Engagement Gauge Predicts Next User Action
Overall engagement gauge Maximum of head and
seat COP values Example episodic engagement
during scenario (upper left). Predicts with
high probability user action within next 5
seconds (upper right). Distribution by type of
next action (lower left) and hazard plots (lower
right).
20
Engagement Gauge Predicts Next User Action

• Engagement gauge value 0.5 indicates that 50 of
  COP or head motion relative to computer monitor
  is explained by the number of targets
• Positive engagement value predicts user action
  with high probability and reliability across
  subjects and scenarios
• Hook, Select or ID within 2 seconds median
  p0.616
• Hook, Select or ID within 3 seconds median
  p0.766
• Hook, Select or ID within 4 seconds median
  p0.860
• Hook, Select or ID within 5 seconds median
  p0.946

21
AugCog DCC CVE

• Seat sensors and head tracker in Mercedes S500
• Road driving conditions
• Autobahn
• Two lane highway
• Urban
• Interactive gauge development and cognitive
  (contextual) modeling with Sandia National
  Laboratories

22
Chair Signals for Contextual Modeling

• Statistical properties of time derivatives of
  seat and back pressure from left and right sides
  of body
• Body COP torsion (yaw) gauges
• Angle of body in seat from left and right body
  centers of pressure/derivatives of centers of
  pressure
• Linear translation of global COP
• Correlation between seat and back velocity RMS in
  a time window
• Head yaw velocity and RMS(time window)

23
Seat Sensor Example TH1 Pseudocolor of
instantaneous pressure
24
Left
Right
Front
Back
25
New Algorithms

• Stability gauges
• Based upon statistical properties of time
  derivatives of seat and back pressure from left
  and right sides of body
• Body torsion (yaw) gauges
• Angle of body in seat from left and right body
  centers of pressure/derivatives of centers of
  pressure
• Linear translation from global center of pressure
• Correlation of head yaw and body yaw

26
Body Torsion (Yaw) Gauge

• Angle of body in seat from left and right body
  centers of pressure/derivatives of centers of
  pressure

Right COP
q
Left COP
Seat back
27
Seat Yaw re Left A-P COP
28
Seat Yaw re Right A-P COP
29
Seat COP Yaw

• Multiple regression analysis 98 VAF (r2) by
  front-back COP shift on right and left seat
• Seat yaw angle -0.1733 (0.0006) left COP
  .1676 (0.001)right COP 0.0566(0.0098)
• No significant contribution of lateral COP

30
Seat Yaw Deterministic and Residual
31
Temporal Relationships
32
Head and Body Yaw Example in Traffic and Lane
Change
1
Seat
Head
Prepare
Change
Lanes
Overtaken
0.5
1155-1161
repeatedly
by traffic
1100-1136
Yaw (radians re leftward positive)
0
Execute
Looking toward
Lane Change
blind spot
1158-1161
-0.5
1100
1110
1120
1130
1140
1150
1160
1170
1180
Time (sec)
33
3600 Emerge from tunnel in right lane on
Autobahn (2 lanes) 3606 Overtaken (car) 3644
Left lane to pass RV 3651 To right 3667 Left
lane, Pass car/trailer (3676) 3680 Right lane
TS1
34
3600 Emerge from tunnel in right lane on
Autobahn (2 lanes) 3606 Overtaken (car) 3644
Left lane to pass RV 3651 To right 3667 Left
lane, Pass car/trailer (3676) 3680 Right lane
TS1
35
6772 Merge on Autobahn behind truck, overtaken
by cars 6777 To center lane to pass truck pass
truck at 6780 6800 Pass car 6802 Move to left
lane 6812 Pass car
36
In left lane on Autobahn Pass car at 6831, 6845,
6851, 6860, 6878, 6883, 6892, 6906, 6909, 6925,
6932
37
In left lane on Autobahn, Tunnel at 6949-53
Light traffic To center lane 6960 Left 6967
Center 6980 Left 6988 Pass car (6992), truck
(6994), car (7000), car (7010) To center lane
7018
38
Postural Measurements in Driving

• Chair sensors detect orienting responses to
  vehicles, signs and situations requiring
  vigilance
• Being overtaken by vehicles
• Lane change scenarios
• Passing large vehicles
• Chair sensors detect relative quiescent periods
  with lower vigilance demands

39
Postural Measurements in Driving

• Need to optimize sensor-derived gauges with
  cognitive model
• Considerations
• Different time scales (e.g., derivatives,
  correlation history, RMS history)
• Quasi-independent measurements to facilitate
  optimization
• Physiologically sensible and plausible