Components of judgmental skill

1
Components of judgmental skill
Public Administration and Policy PAD634 Judgment
and Decision Making Behavior

• Thomas R. Stewart, Ph.D.
• Center for Policy Research
• Rockefeller College of Public Affairs and Policy
• University at Albany
• State University of New York
• T.STEWART_at_ALBANY.EDU

2
Fundamental tenet of correspondence research

• "Human competence in making judgments and
  decisions under uncertainty is impressive.
  Sometimes performance is not. Why? Because
  sometimes task conditions degrade the accuracy of
  judgment."
• Hammond, K. R. (1996). Human Judgment and Social
  Policy Irreducible Uncertainty, Inevitable
  Error, Unavoidable Injustice. New York, Oxford
  University Press (p. 282).

3
Brunswik's lens model
Cues
Distal variable
Judgment
X
4
Expanded lens model
Distal variable
5
Components of skill and the lens model
True
Subjective
Cues
Descriptors
Cues
Distal variable
Judgment
Environmental predictability
Reliability of information processing
Fidelity of the information system
Reliability of information acquisition
Match between environment and judge
components-of-skill.ppt
5
6
Alternative measures of performance(correspondenc
e approach)

• Area under ROC curve
• Correlation (rYO)
• O is the observation, or gold standard
• Y is the judgment
• Mean square error
• Skill score

7
Decomposing accuracy--the skill score

• Murphy (1988)
• Skill Correlation - Conditional
  bias - Unconditional bias
• score (regression bias)
  (base rate bias)

8
Decomposing accuracy The Lens Model Equation
(Tucker 1964)
9
(No Transcript)
10
Decomposition of skill score
Skill score
11
components-of-skill.ppt
11
12
1. Environmental predictability
Components of skill

• Environmental predictability is conditional on
  current knowledge and information. It can be
  improved through research that results in
  improved information and improved understanding
  of environmental processes.
• Environmental predictability determines an upper
  bound on performance and therefore indicates how
  much improvement is possible through attention to
  other components.

13
Environmental predictability limits accuracy of
judgment
13
14
2. Fidelity of information system
Components of skill

• Judgmental skill may be degraded if the
  information system that brings data to the judge
  does not accurately represent actual conditions,
  i.e., if the cues do not accurately measure the
  true descriptors. Fidelity of the information
  system refers to the quality, not the quantity,
  of information about the cues that are currently
  being used.
• Fidelity is improved by developing better
  measures, e.g., though improved instrumentation
  or increased density in space or time.

15
3. Match between environment and judge
Components of skill

• The match between the model of the judge and the
  environmental model is an estimate of the
  potential skill that the judge's current strategy
  could achieve if the environment were perfectly
  predictable (given the cues) and the judgments
  were unbiased and perfectly reliable.
• This component might be called knowledge. It
  is addressed by training and experience. If the
  judge learns to rely on the most relevant
  information and ignore irrelevant information,
  this component will generally be good.

16
Reliability
Components of skill

• Reliability is high if identical conditions
  produce identical judgments.
• Humans are rarely perfectly reliable.
• There are two sources of unreliability
• Reliability of information acquisition
• Reliability of information processing

17
Reliability
Components of skill

• Reliability decreases as amount of information
  increases.

Theoretical relation between amount of
information and accuracy of judgment
18
Reliability decreases as environmental
predictability decreases
Components of skill
18
19
4. Reliability of information acquisition
Components of skill

• Reliability of information acquisition is the
  extent to which the judge can reliably interpret
  the objective cues.
• It is improved by organizing and presenting
  information in a form that clearly emphasizes
  relevant information.

20
Visual display of information
Components of skill
Books by Edward R. Tufte The Visual Display of
Quantitative Information. Envisioning Information
Visual Explanations Beautiful Evidence The
Cognitive Style of Powerpoint Pitching Out
Corrupts Within Cheshire, Connecticut Graphics
Press
21
Visual display of information
Components of skill
Image from the videotape Study of a Numerically
modeled severe storm, National Center for
Supercomputing applications, University of
Illinois (Tufte, Visual Explanations, p. 20)
Tuftes questions How big is the cloud? What
direction is it moving? What are the dimensions
of the grid?
22
Visual display of information
Components of skill
Redesign by Tufte and Bushell (Tufte, Visual
Explanations, p. 21)
Restored quantitative information Directional
arrows Grid size given (and reduce the dominance
of the grid pattern) Time scale, with small
clouds depicting storm history
23
Visual display of information
Components of skill
Chart prepared by Morton Thiokol for Challenger
commission. Tufte, Visual Explanations, p. 47

• Visual display problems
• Disappearing legend.
• The O-ring damage legend was on another slide.
• Chartjunk
• Obscures cause and effect
• Temperatures turned sideways
• O-ring anomalies depicted by scattered little
  marks
• Wrong order
• Date of launch rather than temperature

24
Visual display of information Scatterplot of
temperature vs. O-ring damage
Components of skill
Tufte, Visual Explanations, p. 45
25
Visual display of information influences mode of
thought
Components of skill

• Use of images tends to induce intuition
• Use of numbers tends to induce analysis

26
5. Reliability of information processing
Components of skill

• Decreases with increasing information and with
  increasing environmental uncertainty
• Methods for improving reliability of information
  processing
• Limit the amount of information used in making
  judgments. Use a small number of very important
  cues.
• Use mechanical methods to process information.
• Combine several judgments.
• Require justification of judgments.

27
Theoretical relation between amount of
information and accuracy
components-of-skill.ppt
27
28

• The relation between information and accuracy
  depends on environmental uncertainty

- - - - - Theoretical limit of accuracy
Actual accuracy
- - - - - Theoretical limit of accuracy
Actual accuracy
29
6 and 7. Bias -- Conditional (regression bias)
and unconditional (base rate bias)
Components of skill

• Together, the two bias terms measure judgment
  "calibration.
• Reducing bias
• Experience
• Statistical training
• Feedback about nature of biases in judgment
• Search for discrepant information
• Statistical correction for bias

30
Calibration of judgments depends on the task
Calibration data for precipitation forecasts
(Murphy and Winkler, 1974)
Heideman (1989)
31
Conclusion

• Problem Improving judgmental accuracy
• Understanding and improving judgment requires
  understanding the task and the environment.
• Decomposing skill can aid in identifying the
  factors that limit judgmental accuracy.