Human Robot Teams: Concepts, Constraints, and Experiments

1
Human Robot TeamsConcepts, Constraints, and
Experiments
Michael A. Goodrich Dan R. Olsen Jr. Brigham
Young University
2
Research Agenda

• Evaluation Technology
• Neglect Tolerance
• Behavioral Entropy
• Fan-Out
• Interface Design
• Mixed Reality Displays
• Principles
• HF Experiments
• Autonomy Design
• Team-Based Autonomy
• UAVs
• Perceptual Learning

3
The Presentation Agenda

• The types of questions
• Neglect tolerance Is a team feasible?
• How do we compute neglect tolerances?
• Tradeoffs workload and performance
• Is a team optimal?
• The problem with switch costs
• Some limits, ideas, and proposals

4
A Special Case The Robotics Specialist

• One soldier
• Two UAVs
• One UGV
• Can one person manage all three assets?
• At what level of performance?
• At what level of engagement?

5
A More General CaseSpan of Control

• How many things can be managed by a single
  human?
• How many robots?
• How do we measure Span of Control in HRI?
• Relationships between NT and IT
• How do we compare possible team configurations?
• Evaluate performance-workload tradeoffs
• Identify performance of feasible configurations

6
The Most General Case Multiple Robots
Multiple Humans

• How many people are responsible for a single
  robot?
• How many robots can provide information to a
  single human?

Platoon Headquarters Organization
1 CL I UAV System
ARV-A (L)
ICV
1 CL I UAV System
7
The Presentation Agenda

• The types of questions
• Neglect tolerance Is a team feasible?
• How do we compute neglect tolerances?
• Tradeoffs workload and performance
• Is a team optimal?
• The problem with switch costs
• Some limits, ideas, and proposals

8
Neglect ToleranceNeglect Time and Interaction
Time

• How long can the robot go without needing human
  input?
• How long does it take for a human to give
  guidance to the robot?

Neglect Time (NT)
Interaction Time (IT)
9
Fan-Out (Olsen 2003,2004) How many homogeneous
robots?

• How many interaction periods fit into one
  neglect period
• Two other robots can be handled while robot 1 is
  neglected
• Fan-out 3

1
NT
IT
IT
IT
2
3
4
10
Can a human manage team T ? Fan-out and
Feasibility

• Fan-out (homoeneous teams)
• Feasibility (heterogeneous teams)
• These are upper bounds

11
The Presentation Agenda

• The types of questions
• Neglect tolerance Is a team feasible?
• How do we compute neglect tolerances?
• Tradeoffs workload and performance
• Is a team optimal?
• The problem with switch costs
• Some limits, ideas, and proposals

12
Neglect Impact Curves

• A task is Neglected if attention is elsewhere
• Neglect impacts task performance 2ndary tasks

13
Not Neglect Tolerant Enough
14
Too Neglect Tolerant

• Old Glory Insurance

15
Interface Efficiency Curves

• Recovery from zero point
• Imprecise switch costs

16
Efficient Interfaces

• PDA-based UAV control (versus command line)

17
Efficient Interfaces

• Phycon-based UAV control (versus command line)

18
Finding NT and IT from the curves
19
Example

• Vary minimum performance level
• Measure
• Average performance
• Neglect time
• Interaction time

20
Validation of Method Complexity

• As complexity goes up, NT goes down and IT goes
  up
• Feasibility using NT/IT needs more work

21
The Presentation Agenda

• The types of questions
• Neglect tolerance Is a team feasible?
• How do we compute neglect tolerances?
• Tradeoffs workload and performance
• Is a team optimal?
• The problem with switch costs
• Some limits, ideas, and proposals

22
Existing Tradeoffs
Ideal
23
Types of Autonomy
24
Using Tradeoffs to Select a Configuration
Ideal
Ideal
Ideal
25
Tradeoffs Galore

• Higher workload means shorter missions
• Higher performance requires higher workload
• Higher workload implies smaller span of control
• Lower risk tolerance implies shorter neglect
  times
• Longer neglect times imply greater risk of
  unperceived failures
• Longer neglect times imply more complicated
  other tasks
• More complicated other tasks imply greater
  switch costs

26
The Presentation Agenda

• The types of questions
• Neglect tolerance Is a team feasible?
• How do we compute neglect tolerances?
• Tradeoffs workload and performance
• Is a team optimal?
• The problem with switch costs
• Some limits, ideas, and proposals

27
Predicting Performance of a Heterogeneous Team

• Each robot may have multiple autonomy modes and
  interaction methods
• Each interaction scheme yields NT, IT, and
  average performance values

28
Predicting Performance continued
29
Using Predicted Performance

• Eliminate the infeasible (SIT gt NT)
• Find the maximum performance
• Use interaction schemes that maximize performance

30
Accuracy of Predictions in a Three-Robot Team

• Two interaction schemes
• Point to point (P)
• Region of Interest (R)
• Three robots
• Experiment
• 23 subjects
• 148 trials
• 3 world complexities

31
The Presentation Agenda

• The types of questions
• Neglect tolerance Is a team feasible?
• How do we compute neglect tolerances?
• Tradeoffs workload and performance
• Is a team optimal?
• The problem with switch costs
• Some limits, ideas, and proposals

32
What are switch costs?

• The biggest unknown influence on span of control
• They come in several flavors
• Time to regain situation awareness
• Time to prepare for switch
• Errors and Change Blindness

What really happens here?
33
Before and After
34
Getting a Feel for the Experiment
35
The Experiment

• Primary task Control a robot
• Vary type and duration of secondary task
• Measure speed and accuracy of change detection
• Measure speed and accuracy of change diagnosis

36
Preliminary Results

• 6 subjects, none naïve
• 207 correct change detections
• One-sided T-test, equal variances

37
Important Trends

• Differences not just from time away
• blank and tetris have same time
• UAV and tone have same time
• Averages nearly identical
• Differences not just from counting
• UAV and tone both count
• Differences not just from motor channel
• UAV and tone both select
• Tetris requires interaction
• Probably spatial reasoning and changing
  perspectives

38
Summary of Preliminary Results

• If it takes longer than 20 seconds to diagnose a
  change, the subject has probably failed
• Need to gather failure rate
• Averages show very strong trend
• We conclude
• The test is sensitive enough to detect
  differences
• The type of secondary task affects recovery

39
The Presentation Agenda

• The types of questions
• Neglect tolerance Is a team feasible?
• How do we compute neglect tolerances?
• Tradeoffs workload and performance
• Is a team optimal?
• The problem with switch costs
• Some limits, ideas, and proposals

40
How Many Robots?

• Assumptions
• Goal Gather battle-related information while
  minimizing risk
• Media Mostly camera/video information
• Prediction
• Interpreting camera information difficult
• High robot autonomy wont help enough

41
A Special Case The Robotics Specialist

• Can one person manage multiple robot assets?
• At what level of performance?
• Goal gather information
• Media visual (camera/video)
• Belief autonomy will help, but not enough

42
A Research Agenda

• Phase 1 Refine assessment technique
• Validate sensitivity
• Assess feasibility with switch costs
• Phase 2 Study interfaces
• Select plausible secondary tasks
• Compare information presentation techniques
• Compare PDAs, tablets, workstations
• Compare interaction while stationary with
  in-motion
• Phase 3 Study autonomy
• Vehicle control
• Team playbooks
• Interactive perception

43
Pushing the limits

• Mixed reality displays
• Let the operator control the camera, not the
  robot
• robot controls its motion to support camera
• Highlight changes
• Mitigate change blindness effects
• TiVO
• Support task switching
• Improve robot perception by teaching it
• Automate image understanding
• Effects of false alarms on the human?
• Costs of missed detections on the mission?
• 2 to 3, or 3 to 5 ratios redundancy and
  responsibility swapping

44
Mixed Reality Displays

• Eliminate The world through a soda straw
• Integrate vision with active sensors
• Integrate display with autonomy
• Include sensor uncertainty
• Control pan-and tilt
• Study time delay effects

45
Real World Results

• Objective
• 51 Faster (p lt .01)
• 93 Less Safeguarding (p lt .01)
• 29 Lower Entropy (p lt . 05)
• 10 Better on Memory Task (p lt .05)
• Subjective
• 64 Less Workload / Effort (p lt .001)
• 70 More Learnable (p lt .0001)
• 46 More Confident (p lt .05)

46
Several Thousand Words
47
Experiment Results
48
Ecological Display Experiment Results

• Mixed-Reality better for
• Time to complete
• Number of collisions
• Subjective workload
• Entropy
• Even stronger results for real-world
• Twice as long
• Better performance on secondary task

49
Mixed Reality Displays (Pan and Tilt)
50
Control the Information Source, Not the Robot

• Phlashlight Concept
• What will UAV see?

51
Semantic Maps and Change Highlighting

• Video in context
• Icon-based maps w/ semantic labels
• That was then, this is now comparison ---
  change highlighting
• Information decay

52
Information in Context
53
Support Timely Shifts

• Prompt prospective memory
• Shift in a timely way
• Give time to prepare

Situation Awareness
54
Supporting Task Switching Etc.

• History trails. Knowing recent past helps
• Tail on a map-based interface
• Virtual descent into video-based interface
• Change highlighting/morphing
• Plans Knowing intention helps
• Planned path on map-based interface
• Predicted trajectory on video-based interface
• Quickened displays
• Task relationships Knowing relationship between
  two tasks helps
• Relative spatial location on map-based interface
• Picture-in-picture on video-based interface
• Progress bar of task X on task Ys display

55
Improve Perception and Scene Interpretation
(Olsen)

• Use interaction and machine learning to make this
  robust

56
Future Concept (Proposed)

• Safe/Unsafe occupancy grids
• Evolutionary image classifier
• Evolutionary integration of vision and lasers
• Particle-based inverse perspective transform
• Path planning
• Uncertainty-based triggers for retraining

• Learning interface mappings from implicit user
  cues

57
Conclusions

• We can evaluate team feasibility
• We can predict team performance
• We need to understand task switching better
• We need to support realistic task switching
• Via interfaces
• Via autonomy

58
Near-Term Future Work

• Complete validation of task switching experiment
  paradigm
• Compare new and improved interfaces against
  baseline
• Compare effects of type and size of interface
• Answer the questions for the special case