Human-Computer Decision Making: The View from Psychology

1
Human-Computer Decision Making The View from
Psychology

• Earl Hunt
• University of Washington

2
Acknowledgments

• Collaborators on research project
• Susan Joslyn UW Psychology
• Karla Schweitzer UW Psychology
• David Jones UW Applied Physics Laboratory
• General inspiration and ideas-writings and talks.
• Lee Beach U. of Arizona (formerly UW)
• Gary Klein Klein Associates
• Steven Poltrock Boeing Corporation
• Steven Hunt Unicru Inc.

3
Support of Research, Preparation of Paper

• Department of Defense Multidisciplinary Research
  Initiative
• Office of Naval Research Grant N00014-01-10745 to
  the University of Washington
• Adrian Raftery (Prof. of Statistics, UW)
  Principal Investigator

4
Possible Mixed Mode Decision Making

• Computers tell humans what to do
• Rental car check in.
• Not relevant
• Humans tell computer what to do
• Some aspects of fly by wire
• Not relevant
• Computers advise and gather data, humans decide
• Common
• Thats what this is about.

5
Point of Talk

• There is always an interface
• Computer system designers must consider
• Cognitive psychological aspects, not just human
  engineering
• Social context of decision making situation

6
Plan of Talk

• Psychological theories of decision making
• Two studies of computer-human decision making
• Heath and Luff study of British medical records
  system
• Research of my colleagues and myself on Naval
  Aviation weather forecasting
• Concluding comments.

7
Models of Decision Making
8
Von Neumann-Morgenstern Model

• Life is a lottery
• Evaluate the rewards (utility)
• Factor in the probabilities
• Choose action with highest expected value
• This is often seen as the normative model
• Ideal for computerization!

9
Objections

• Kahneman and Tversky (modifications)
• People are poor estimators of probability
• Subjective scale needed
• May not be a probability calculus
• Prospect theory-modify utility scale
• Kahneman and Tversky (basic objection)
• Framing. Decision situation changes if you focus
  on profit or loss
• Merchants advertise discounts for cash. How many
  advertise charges for credit?

10
Radical Alterations

• Richard Wagner Rejection of lottery model
• LeeRoy Beach Image Theory
• People make decisions based upon goals and their
  view of permissible ways to attain them
• Example University cheating
• Progress by scenario creation and monitoring
• Gary Klein Pattern driven decision making
• Decision making is largely forward driven
• Depends upon recognizing situation, taking
  action.
• Example Fire commanders action as fire
  progresses.

11
Implications

• Von Neumann-Morgenstern approach good for
  computer-alone decision making
• Example programmed stock trading.
• Does not fit into human thought process

12
Computers supporting human thought

• Provide human with information in way human wants
  it.
• Do not restrict human thought processes.
• Dont rely on applying mathematical decision
  theory except for autonomous computing

13
Medical Study

• Ref Heath, C. Luff, P. (2000) Technology in
  action. Cambridge, Cambridge U. Press
• Topic Studied conversion of medical clinical
  records from written notes to computer forms
• Forms consulted and/or filled out by the
  physician during the consultation.

14
Key points

• Computer forms supported decision making by a
  high-level expert
• Fully computerized systems proposed
• We aren t there yet
• Decision made
• In social situation
• Under considerable time pressure

15
Written Records

• Flexibility
• Spontaneously developed
• Humans are trained to write standard forms
• But, concept of defeasability
• Most of the time standard forms
• Any rule should have exceptions
• Example (mine, not C and L)
• Required Case History, Diagnosis, Prognosis
• Written Drunk Again.

16
Computer Records

• Standardized forms
• Probably better for standard or anticipated
  issues Made sure physician did not overlook
  something
• Do not support defeasability
• Or for special notes to other physicians
• Interview was distorted
• Physicians turned away from patient
• Did not distract decision making about presented
  facts
• Did distract physician-patient interchange,
  chance for spontaneous remarks by patient

17
Decision Making by Naval Weather Forecasters

• Naval Aviation Forecast (Whidbey Island NAS,
  Whidbey Island, WA)
• Writing Terminal Aerodrome Forecast (TAF)
• Anticipated conditions at local airport
• Similar TAF is filed for commercial airports by
  National Weather Service

18
Conditions of TAF

• TAF written by PO1/c
• Has experience as observer
• Has gone to Navy METOC school.
• Far less training that NWS meteorologist
• (Navy does have more trained people at Fleet
  Weather Service)
• TAF is written while other things are in process
• Weather briefing for flights

19
information available to forecaster

• Recent observations in region
• Satellite observations
• Numerical model predictions
• More than one model is available
• Models do not always agree
• Different initializations
• Different physics in model
• One model can do best for awhile, then another
• NO ONE could look at all of these variables!

20
Observations and formal experiments

• Spent time observing how people used models
• Forecasters would select favorite model, then
  adjust it from observations and satellite
  patterns
• Did not spend much time on other models
• Did not look at model history

21
(No Transcript)
22
Example Forecaster D compared the observed
pressure to what the model had predicted. Access
current pressure 29.69 Calculate difference
29.69-29.64.05between current and
forecast (error) Access predicted
pressure 29.57for forecast time Adjust
predicted pressure 29.57-.0529.52based on
current error Adjust predicted pressure
29.52.0229.54 based on model bias
23
How effective is this

• Have compared model and TAF
• General results, observers are quite good

24

• Wind Speed From Winter, Spring 2003
• Correlation between Navy Forecasters prediction
  and the observed wind speed was greater than the
  correlation between numerical models and observed
  wind speed.

25

• The variance in actual wind speed accounted for
  by the human forecast subsumed that accounted for
  by the numerical model.

Observed1
Numerical Model alone.15
Human Forecast.46
26
How general are these results?

• Similar results have been obtained for other
  parameters (e.g. barometric pressure), other time
  periods.
• Work rather difficult in summer!
• Are continuing studies, for stormy period
• Have observed similar results in artificial
  studies using UW undergrads, made up cover
  story
• Select best recommendation
• Adjust slightly, but actually pay little
  attention to non-favored recommenders.

27
What is being missed

• Statistical studies show that the information in
  non-preferred models is valuable
• But studies in literature are consistent with our
  findings
• We do see multiple models in NWS
• As did Hoffman and colleagues, studying NWS
• Differences
• Better trained individuals
• Much less time pressure.

28
Probability Product MM5 ACME Ensemble
Probability of winds greater than 20 knots
29
Applied Physics Lab (UW) approach
30
We are studying simplified models, to see how
much history checking is actually done
31
YOUR TASK You need to make a forecast for
atmospheric pressure for airline pilots.
Pressure is used by the altimeter in determining
the altitude of the plane. Pilots set the
altimeter based on the predicted pressure.
Pilots use altimeter readings to determine their
altitude, which is especially important when
landing the plane. If the pressure forecast is
too high, the altimeter will read ground level
too late and the plane may crash. It is
important for your pressure forecast to be
accurate, but you want to predict the lowest
pressure, without going below, in order to safely
land. You will make the forecast using computer
model predictions alone. You have no other
information. There are 3/7 models you may use.
You may use as many or as few of them as you
wish. The important thing is to make your best
forecast. Clicking on the prediction button
will give you information regarding each models
prediction. Clicking on the history button
will give you information regarding that
forecasters previous 10 forecasts. Please make
your best judgment of what the pressure will be.
This chart will show trends in error- if the
model is consistently high or low, large error or
small error
32
Questions to consider in design

• Peachy keen computer systems not enough
• Consider context of use
• Look especially at time pressures and social
  situations
• How much flexibility should the human have?
• Can system designer anticipate everything?
• French bureaucrats ncest posible !

33
Things that reduce flexibility

• Situational factors
• Time pressures, time sharing
• Lack of training by individuals
• Drowning people with information
• The computer system itself
• These are specific cases of -gt

34
The curse of functionality

• Added functionality may increase difficulty of
  useactually reduces flexibility
• Added probability information in weather
  forecasting seems to make little difference
• MICROSOFT WORD, UNIX changes?
• Research required on methods of increasing
  functionality and information for decision making
  without increasing complexity of information
  search

35
Take home message

• These issues are research issues
• Local decisions in specific situation
• More general principles needed.
• Focus groups and informal observations should be
  treated with suspicion
• Random choice of users
• Formal experimentation statistical issues
• Army MANPRINT guidance
• Support your local cognitive psychologist