Title: Understanding and Using Uncertainty Information in Weather Forecasting
1Understanding and Using Uncertainty Information
in Weather Forecasting
- Susan Joslyn
- University of Washington
2Acknowledgements
- Earl Hunt
- David Jones
- Limor Nadav-Greenberg
- John Pyles
- Adrian Raftery
- Karla Schweitzer
- McLean Slaughter
- Meng Taing
- Jeff Thomasson
- This research was supported by the DOD
Multidisciplinary University Research Initiative
(MURI) program administered by the Office of
Naval Research under Grant N00014-01-10745
3Forecast Uncertainty
- Available for some time
- Rarely communicated in public forecasts
-
- Underused by weather forecasters
4Forecast Uncertainty
- Difficult to understand
- - Forecasters claim
- People make mistakes when reasoning with
probability - Format Frequency (1 time in 10) is better than
- Probability (10 chance)
5Forecast Uncertainty
- Useful for deterministic forecasts decision?
- Theoretically
- Practically useful?
- It doesnt matter how good the information if
people cant or wont make use of it.
6Goals for Psychology Team
- Establish uncertainty information is useful
- Threshold forecast (forecasters general
public) - - high wind advisory for boater safety
- What is best presentation format to enhance
- Understanding?
- Decisions?
-
-
7Three Major Lines of Inquiry
- 1. Does probability information improve threshold
forecast? Study 1 - 2. Does display format (visualization) matter?
- Study 2
- 3. Does the wording matter? Studies 3-4
- (probability/ frequency)
8Study1 Does Probability Information Improve
Threshold Forecast?
- Participants
- Advanced atmospheric science students
- Task
- Forecast wind speed and direction
- Decide whether to issue high wind advisory
-
(winds gt 20 knots)
9Within Subject Design
Condition 2
Condition 1
- Historical data
- Radar Imagery
- Satellite Imagery
- TAFs and current METARs
- Model output
- (AVN, MM5 NGM)
- Historical data
- Radar Imagery
- Satellite Imagery
- TAFs and current METARs
- Model output
- (AVN, MM5 NGM)
-
- Chart showing probability
- of winds gt 20 k
10Probability of Winds 20k
11Within Subjects Design
Condition 2
Condition 1
- Historical data
- Radar Imagery
- Satellite Imagery
- TAFs and current
- METARs
- Model output (AVN,
- MM5 NGM)
-
- Chart showing probability
- of winds gt 20 k
- Historical data
- Radar Imagery
- Satellite Imagery
- TAFs and current
- METARs
- Model output (AVN,
- MM5 NGM)
Same participants, same weather Only
difference is probability product
12Results
- Threshold Forecast
- People posted fewer wind advisories with
probability product. -
- Similar ability to discriminate between high wind
and low wind event (sensitivity).
13Results Percent Advisories
- Y times
- forecasters posted advisory
- X probability
- of winds
- gt 20K
-
14Conclusion Uncertainty Information IS Beneficial
for Threshold
- Increased advisories when high winds were very
likely - Decreased advisories when high winds were
unlikely-fewer false alarms - Increase trust in warnings!
15 Study 2 Does Display Format Matter?
- 3 different visualizations of 90 predictive
interval - Range of likely wind speeds
- All conditions included median wind speed chart
- deterministic forecast
163 Visualizations Between subjects
- 1. 90 Upper bound
- warmer colors higher wind speed
- observed wind speeds will be higher
only 1 time in 10 - worse case scenario highest likely winds
-
-
173 Visualizations
- 1. 90 Upper bound
- wind speeds will be higher only 1 time in 10
- warmer colors higher wind speed
- 2. Margin of error
- range of wind speeds between UB median
- display of uncertainty in the forecast
- warmer colors more uncertainty
-
183 Visualizations
- 1. 90 Upper bound
- wind speeds will be higher only 1 time in 10
- warmer colors higher wind speed
- 2. Margin of error
- range of wind speeds between upper bound and
median - warmer colors more uncertainty
- 3. Box plot
Wind speed in knots
Wind Speed in knots
90 Upper bound
median
90 lower bound
19Method
- Participants
- Atmospheric Science students
- (replicated on NOAA Forecasters)
- Practice Learned how to read charts
- Test
- - Forecast wind speeds
- - Threshold high wind advisory (winds gt20
knots) - - Rate uncertainty in forecast
20Results Wind Speed Forecast
Box Plot
1.17
Upper bound
2.02
1.55
Margin of Error
Knots above the Median
UB forecast significantly higher wind speeds
Display provided a high anchor (Tversky
Kahneman, 1982)
21Results High Wind Advisories
Likelihood of high winds Box Plot Upper Bound Margin of Error
HIGH Median gt 20K 98.44 94.45 91.67
MEDIUM Median 15-20K 32.40 31.24 27.95
LOW Median lt15 K 3.57 3.97 2.38
People in the box plot condition posted
significantly more advisories most in high
likelihood situations
22Results Uncertainty Rating
MoE best for detecting relative
uncertainty They learned The wider
the range the greater the uncertainty
correlation
Box plot .81
Upper Bound .89
Margin of Error .97
Ratings in the MoE significantly more highly
correlated to range
23Conclusion Format Matters
- Box Plot better threshold forecast
wind speed no bias - (salient high and low
anchors) - MoE detect relative uncertainty in
- forecast
- Upper higher winds speeds bias (anchor)
- Bound no benefit to threshold forecast
24Study 3 4 Does Wording Matter?
- Participants
- Psychology undergraduates
- Frequency is easier to understand than
probability (Gigerenzer, 1995, 1999, 2000) - Research on complex problems
- Is that true of simple expressions of uncertainty?
25Does Wording Matter?
- There is a 10 chance that the wind
speeds will be greater than 20 knots.
26Method
- Procedure
- Fill out questionnaire rating expressions of
uncertainty - Decide whether or not to post a high wind
advisory - Suppose that there is a 10 chance that the wind
speeds will be greater than 20 knots. - How likely are the wind speeds to be
greater than 20 knots? (please fill in a bubble) - Very Unlikely
Very Likely - O-------O-------O-------O-------O-------
O--------O-------O-------O-------O-------O - Would you issue a small craft advisory
(winds equal or greater than 20 knots)? -
___Yes ___No
27Method
- Procedure
- Fill out questionnaire rating expressions of
uncertainty - Decide weather to post a wind advisory
- Suppose that there is a 10 chance that the wind
speeds will be greater than 20 k. - How likely are the wind speeds to be
greater than 20 knots? (please fill in a bubble) - Very Unlikely
Very Likely - O-------O-------O-------O-------O-------
O--------O-------O-------O-------O-------O - Would you issue a small craft advisory
(winds equal or greater than 20 knots)? -
___Yes ___No
28Method
- Procedure
- Filled out questionnaire rating expressions of
uncertainty - Decide weather to post a wind advisory
- Suppose that 1 time in 10 the wind speeds will be
greater than 20 knots. - How likely are the wind speeds to be greater
than 20 knots? (please fill in a bubble) - Very Unlikely
Very Likely - O-------O-------O-------O-------O-------O-------
-O-------O-------O-------O-------O - Would you issue a small craft advisory (winds
equal or greater than 20 knots)? -
___Yes ___No
29Study 32 Variables Wording Likelihood
- Probability Frequency
- 10 chance 1 time in 10
-
- 90 chance 9 times in 10
30Study 3 Likelihood of High Wind Held Constant
-
- 1 time in 10 wind speeds 9 times in
10 wind speeds - will be greater than 20 knots will be less
than 20 knots
31Results Reversal Error
- Rate from wrong side of scale
- Suppose that there is a 90 chance that the wind
speeds will be less than 20 knots. - How likely are the wind speeds to be less than
20 knots? (please fill in a bubble) -
-
-
- O-------O-------O-------O-------O-------O------
--O-------O-------O-------O-------O - lt---very unlikely
very likely
------gt - They completely misunderstand the phrase
- Most in 90 (9 in 10) less than wording
- Which is it? High likelihood? Less than?
Reversal error
32Study 4 Manipulated Less / Greater
- Less Greater
- 10 chance less 10 chance greater
33Added 2 levels of likelihood
- Less Greater
- 10 chance less 10 chance greater
- 1 in 10 less 1 in 10 greater
- 30 chance less 30 chance greater
- 3 in 10 less 3 in 10 greater
- 70 chance less 70 chance greater
- 7 in 10 less 7 in 10 greater
- 90 chance less 90 chance greater
- 9 in 10 less 9 in 10 greater
34Equivalent Expressions
- Less Wording Greater Wording
- 10 chance less 10 chance greater
- 1 in 10 less 1 in 10 greater
- 30 chance less 30 chance greater
- 3 in 10 less 3 in 10 greater
- 70 chance less 70 chance greater
- 7 in 10 less 7 in 10 greater
- 90 chance less 90 chance greater
- 9 in 10 less 9 in 10 greater
35Equivalent Expressions
- Less Wording Greater Wording
- 10 chance less 10 chance greater
- 1 in 10 less 1 in 10 greater
- 30 chance less 30 chance greater
- 3 in 10 less 3 in 10 greater
- 70 chance less 70 chance greater
- 7 in 10 less 7 in 10 greater
- 90 chance less 90 chance greater
- 9 in 10 less 9 in 10 greater
36Results Reversal Error
- More often in less than wording (4x as likely)
-
-
Mean reversal error -
per person
Less than .41
Greater than .10
High vs. low likelihood does not matter
Frequency wording does not help
37Results Wind Advisories
10
30
70
90
38Results Wind Advisories
10
30
70
90
39Results Wind Advisories
10
30
70
90
40Results Probability less is worst
10
30
70
90
10
30
70
90
Reversal error subjects eliminated from this
analysis
41Conclusion Wording Matters
- Less than wording is difficult (reversal
errors) - Wind speed advisories in probability less
- - too many advisories in low ranges
- - too few in high ranges
- Frequency protects against posting errors
generated by less than wording
42Conclusions
- Probability information improves threshold
forecasts - Many end-user weather decisions are yes/no
threshold decisions - The right display format
- Improves understanding
- MoE communicates relative uncertainty
- Improves weather decisions
- Box Plot increases warnings in high likelihood
- Box Plot unbiased wind speed forecast
- Wording matters
- Less than is confusing
- Frequency helps sometimes
- NOT in reversal errors
- HELPS in posting advisories
43The End
44Results Percent Advisories
- Y times
- forecasters posted advisory
- X probability
- of winds
- gt 20K
-
45Results Percent Advisories
- Y times
- forecasters posted advisory
- X probability
- of winds
- gt 20K
-
46Results Percent Advisories
- Y times
- forecasters posted advisory
- X probability
- of winds
- gt 20K
-
47Results Percent Advisories
- Y times
- forecasters posted advisory
- X probability
- of winds
- gt 20K
-
48Study 1 Rating
- 10 was rated significantly higher
- Probability condition
- 10 chance (M1.32) 90 chance (M.99)
- O-------O-------O-------O-------O-------O-------
-O-------O-------O-------O-------O - Frequency condition
- 1 in ten (M1.06) 9 out of 10
(M.98) - O-------O-------O-------O-------O-------O--------
O-------O-------O-------O-------O -
49Study 2 Rating
- 10 was rated higher--did not reach significance
- 10 (1 in 10) greater (M1.25) 90 (9 in
10)less (M.97) - O-------O-------O-------O-------O-------O-------
-O-------O-------O-------O-------O - 10 (1 in 10) less (M.98) 90 (9 in
10)greater (M.88) - O-------O-------O-------O-------O-------O--------
O-------O-------O-------O-------O -
50Study 1 Reversal Error
- Mean reversal
- error per person
90 (9 times) less than .83
10 (10 times) greater than .33
51User Needs Understanding
Naval Forecasters Terminal Aerodrome
Forecast (TAF) posted at regular
intervals while fulfilling other duties
52Method
Talk-aloud while creating TAF
Microphone recorder
53Synoptic Pattern Comparison
- 1. Compare position of low in the model
satellite - 2. Assess differences in movement and position
- 3. Adjust forecast accordingly
54Compare Predicted to Observed Values
- 1. Access NOGAPS predicted pressure for
current time 29.69 - 2. Access current local pressure and
29.69 - subtract from NOGAPS
- 29.64 - .05
- 3. Access NOGAPS predicted pressure for
29.59 - forecast period and subtract error amount
- .05 - 4. Forecast
29.54
55Results
- Naval forecasters rely heavily on models
- (1/3-1/2 source statements referred to
models) - Statements implying understanding of model
uncertainty - Model biases and
strengths - Initialization of model run
- Strategies for determining uncertainty
- Evaluation of degree of uncertainty
- Adjusting model predictions
56Conclusions
- Uncertainty?
- Error in deterministic forecast?
- Subsequent questionnaire study confidence is
related to their assessment of model performance
57Probability Problem
- The probability that a woman getting a mammogram
has breast cancer is 1. If the woman has breast
cancer the probability is 80 that she will have
a positive mammogram. - If the woman does not have breast cancer the
probability that she will still have a positive
mammogram is 10. - You have a patient that has a positive mammogram
(no symptoms)--what is the probability she has
breast cancer.
58Frequency Problem
- Ten out of every 1,000 women have breast cancer
- Of those 10 women with breast cancer 8 will have
a positive mammogram - Of the remaining 990 women without breast cancer,
95 will still have a positive monogram - You have a sample of women who have positive
mammograms in your screening (no symptoms) - How many of these women will actually have breast
cancer?
59Results Probability less is worst
10
30
70
90
Reversal error subjects eliminated from this
analysis