Understanding and Using Uncertainty Information in Weather Forecasting - PowerPoint PPT Presentation

1 / 59
About This Presentation
Title:

Understanding and Using Uncertainty Information in Weather Forecasting

Description:

The probability that a woman getting a mammogram has breast cancer is 1%. If the ... Of those 10 women with breast cancer 8 will have a positive mammogram ... – PowerPoint PPT presentation

Number of Views:32
Avg rating:3.0/5.0
Slides: 60
Provided by: susanj3
Learn more at: https://stat.uw.edu
Category:

less

Transcript and Presenter's Notes

Title: Understanding and Using Uncertainty Information in Weather Forecasting


1
Understanding and Using Uncertainty Information
in Weather Forecasting
  • Susan Joslyn
  • University of Washington

2
Acknowledgements
  • Earl Hunt
  • David Jones
  • Limor Nadav-Greenberg
  • John Pyles
  • Adrian Raftery
  • Karla Schweitzer
  • McLean Slaughter
  • Meng Taing
  • Jeff Thomasson
  • This research was supported by the DOD
    Multidisciplinary University Research Initiative
    (MURI) program administered by the Office of
    Naval Research under Grant N00014-01-10745

3
Forecast Uncertainty
  •  Available for some time
  •  Rarely communicated in public forecasts
  • Underused by weather forecasters

4
Forecast Uncertainty
  •  Difficult to understand
  • - Forecasters claim
  • People make mistakes when reasoning with
    probability
  • Format Frequency (1 time in 10) is better than
  • Probability (10 chance)

5
Forecast Uncertainty
  •  Useful for deterministic forecasts decision?
  • Theoretically
  • Practically useful?
  •  It doesnt matter how good the information if
    people cant or wont make use of it.

6
Goals for Psychology Team
  •   Establish uncertainty information is useful
  • Threshold forecast (forecasters general
    public)
  • - high wind advisory for boater safety
  •  What is best presentation format to enhance
  • Understanding?
  • Decisions?

7
Three Major Lines of Inquiry
  • 1. Does probability information improve threshold
    forecast? Study 1
  • 2. Does display format (visualization) matter?
  • Study 2
  • 3. Does the wording matter? Studies 3-4
  • (probability/ frequency)

8
Study1 Does Probability Information Improve
Threshold Forecast?
  • Participants
  • Advanced atmospheric science students
  • Task
  • Forecast wind speed and direction
  • Decide whether to issue high wind advisory

  • (winds gt 20 knots)

9
Within Subject Design
Condition 2
Condition 1
  • Historical data
  • Radar Imagery
  • Satellite Imagery
  • TAFs and current METARs
  • Model output
  • (AVN, MM5 NGM)
  • Historical data
  • Radar Imagery
  • Satellite Imagery
  • TAFs and current METARs
  • Model output
  • (AVN, MM5 NGM)
  • Chart showing probability
  • of winds gt 20 k

10
Probability of Winds 20k
11
Within Subjects Design
Condition 2
Condition 1
  • Historical data
  • Radar Imagery
  • Satellite Imagery
  • TAFs and current
  • METARs
  • Model output (AVN,
  • MM5 NGM)
  • Chart showing probability
  • of winds gt 20 k
  • Historical data
  • Radar Imagery
  • Satellite Imagery
  • TAFs and current
  • METARs
  • Model output (AVN,
  • MM5 NGM)

Same participants, same weather  Only
difference is probability product
12
Results
  • Threshold Forecast
  • People posted fewer wind advisories with
    probability product.
  • Similar ability to discriminate between high wind
    and low wind event (sensitivity).

13
Results Percent Advisories
  • Y times
  • forecasters posted advisory
  • X probability
  • of winds
  • gt 20K

14
Conclusion Uncertainty Information IS Beneficial
for Threshold
  • Increased advisories when high winds were very
    likely
  • Decreased advisories when high winds were
    unlikely-fewer false alarms
  • Increase trust in warnings!

15
Study 2 Does Display Format Matter?
  • 3 different visualizations of 90 predictive
    interval
  •  Range of likely wind speeds
  • All conditions included median wind speed chart
  • deterministic forecast

16
3 Visualizations Between subjects
  • 1. 90 Upper bound
  • warmer colors higher wind speed
  • observed wind speeds will be higher
    only 1 time in 10
  • worse case scenario highest likely winds

17
3 Visualizations
  • 1. 90 Upper bound
  •  wind speeds will be higher only 1 time in 10
  • warmer colors higher wind speed
  • 2. Margin of error
  •  range of wind speeds between UB median

  • display of uncertainty in the forecast
  • warmer colors more uncertainty

18
3 Visualizations
  • 1. 90 Upper bound
  • wind speeds will be higher only 1 time in 10
  • warmer colors higher wind speed
  • 2. Margin of error
  • range of wind speeds between upper bound and
    median
  • warmer colors more uncertainty
  • 3. Box plot

Wind speed in knots
Wind Speed in knots
90 Upper bound
median
90 lower bound
19
Method
  • Participants
  • Atmospheric Science students
  • (replicated on NOAA Forecasters)
  • Practice Learned how to read charts
  • Test
  • - Forecast wind speeds
  • - Threshold high wind advisory (winds gt20
    knots)
  • - Rate uncertainty in forecast

20
Results Wind Speed Forecast
Box Plot
1.17
Upper bound
2.02
1.55
Margin of Error
Knots above the Median
UB forecast significantly higher wind speeds
Display provided a high anchor (Tversky
Kahneman, 1982)
21
Results High Wind Advisories
Likelihood of high winds Box Plot Upper Bound Margin of Error
HIGH Median gt 20K 98.44 94.45 91.67
MEDIUM Median 15-20K 32.40 31.24 27.95
LOW Median lt15 K 3.57 3.97 2.38
People in the box plot condition posted
significantly more advisories most in high
likelihood situations
22
Results Uncertainty Rating
MoE best for detecting relative
uncertainty  They learned The wider
the range the greater the uncertainty

correlation
Box plot .81
Upper Bound .89
Margin of Error .97
Ratings in the MoE significantly more highly
correlated to range
23
Conclusion Format Matters
  • Box Plot better threshold forecast
    wind speed no bias
  • (salient high and low
    anchors)
  • MoE detect relative uncertainty in
  • forecast
  • Upper higher winds speeds bias (anchor)
  • Bound no benefit to threshold forecast

24
Study 3 4 Does Wording Matter?
  • Participants
  • Psychology undergraduates
  • Frequency is easier to understand than
    probability (Gigerenzer, 1995, 1999, 2000)
  • Research on complex problems
  • Is that true of simple expressions of uncertainty?

25
Does Wording Matter?
  • There is a 10 chance that the wind
    speeds will be greater than 20 knots.

26
Method
  • Procedure
  • Fill out questionnaire rating expressions of
    uncertainty
  • Decide whether or not to post a high wind
    advisory
  • Suppose that there is a 10 chance that the wind
    speeds will be greater than 20 knots.
  • How likely are the wind speeds to be
    greater than 20 knots? (please fill in a bubble)
  • Very Unlikely

    Very Likely
  • O-------O-------O-------O-------O-------
    O--------O-------O-------O-------O-------O
  • Would you issue a small craft advisory
    (winds equal or greater than 20 knots)?

  • ___Yes ___No

27
Method
  • Procedure
  • Fill out questionnaire rating expressions of
    uncertainty
  • Decide weather to post a wind advisory
  • Suppose that there is a 10 chance that the wind
    speeds will be greater than 20 k.
  • How likely are the wind speeds to be
    greater than 20 knots? (please fill in a bubble)
  • Very Unlikely

    Very Likely
  • O-------O-------O-------O-------O-------
    O--------O-------O-------O-------O-------O
  • Would you issue a small craft advisory
    (winds equal or greater than 20 knots)?

  • ___Yes ___No

28
Method
  • Procedure
  • Filled out questionnaire rating expressions of
    uncertainty
  • Decide weather to post a wind advisory
  • Suppose that 1 time in 10 the wind speeds will be
    greater than 20 knots.
  • How likely are the wind speeds to be greater
    than 20 knots? (please fill in a bubble)
  • Very Unlikely

    Very Likely
  • O-------O-------O-------O-------O-------O-------
    -O-------O-------O-------O-------O
  • Would you issue a small craft advisory (winds
    equal or greater than 20 knots)?

  • ___Yes ___No

29
Study 32 Variables Wording Likelihood
  • Probability Frequency
  • 10 chance 1 time in 10
  • 90 chance 9 times in 10

30
Study 3 Likelihood of High Wind Held Constant
  • 1 time in 10 wind speeds 9 times in
    10 wind speeds
  • will be greater than 20 knots will be less
    than 20 knots

31
Results Reversal Error
  • Rate from wrong side of scale
  • Suppose that there is a 90 chance that the wind
    speeds will be less than 20 knots.
  • How likely are the wind speeds to be less than
    20 knots? (please fill in a bubble)
  • O-------O-------O-------O-------O-------O------
    --O-------O-------O-------O-------O
  • lt---very unlikely

    very likely
    ------gt
  • They completely misunderstand the phrase
  • Most in 90 (9 in 10) less than wording
  • Which is it? High likelihood? Less than?

Reversal error
32
Study 4 Manipulated Less / Greater
  • Less Greater
  • 10 chance less 10 chance greater

33
Added 2 levels of likelihood
  • Less Greater
  • 10 chance less 10 chance greater
  • 1 in 10 less 1 in 10 greater
  • 30 chance less 30 chance greater
  • 3 in 10 less 3 in 10 greater
  • 70 chance less 70 chance greater
  • 7 in 10 less 7 in 10 greater
  • 90 chance less 90 chance greater
  • 9 in 10 less 9 in 10 greater

34
Equivalent Expressions
  • Less Wording Greater Wording
  • 10 chance less 10 chance greater
  • 1 in 10 less 1 in 10 greater
  • 30 chance less 30 chance greater
  • 3 in 10 less 3 in 10 greater
  • 70 chance less 70 chance greater
  • 7 in 10 less 7 in 10 greater
  • 90 chance less 90 chance greater
  • 9 in 10 less 9 in 10 greater

35
Equivalent Expressions
  • Less Wording Greater Wording
  • 10 chance less 10 chance greater
  • 1 in 10 less 1 in 10 greater
  • 30 chance less 30 chance greater
  • 3 in 10 less 3 in 10 greater
  • 70 chance less 70 chance greater
  • 7 in 10 less 7 in 10 greater
  • 90 chance less 90 chance greater
  • 9 in 10 less 9 in 10 greater

36
Results Reversal Error
  • More often in less than wording (4x as likely)

  • Mean reversal error

  • per person

Less than .41
Greater than .10
High vs. low likelihood does not matter
Frequency wording does not help
37
Results Wind Advisories
10
30
70
90
38
Results Wind Advisories
10
30
70
90
39
Results Wind Advisories
10
30
70
90
40
Results Probability less is worst
10
30
70
90
10
30
70
90
Reversal error subjects eliminated from this
analysis
41
Conclusion Wording Matters
  • Less than wording is difficult (reversal
    errors)
  • Wind speed advisories in probability less
  • - too many advisories in low ranges
  • - too few in high ranges
  • Frequency protects against posting errors
    generated by less than wording

42
Conclusions
  • Probability information improves threshold
    forecasts
  • Many end-user weather decisions are yes/no
    threshold decisions
  • The right display format
  • Improves understanding
  • MoE communicates relative uncertainty
  • Improves weather decisions
  • Box Plot increases warnings in high likelihood
  • Box Plot unbiased wind speed forecast
  • Wording matters
  • Less than is confusing
  • Frequency helps sometimes
  • NOT in reversal errors
  • HELPS in posting advisories

43
The End
44
Results Percent Advisories
  • Y times
  • forecasters posted advisory
  • X probability
  • of winds
  • gt 20K

45
Results Percent Advisories
  • Y times
  • forecasters posted advisory
  • X probability
  • of winds
  • gt 20K

46
Results Percent Advisories
  • Y times
  • forecasters posted advisory
  • X probability
  • of winds
  • gt 20K

47
Results Percent Advisories
  • Y times
  • forecasters posted advisory
  • X probability
  • of winds
  • gt 20K

48
Study 1 Rating
  • 10 was rated significantly higher
  • Probability condition
  • 10 chance (M1.32) 90 chance (M.99)
  • O-------O-------O-------O-------O-------O-------
    -O-------O-------O-------O-------O
  • Frequency condition
  • 1 in ten (M1.06) 9 out of 10
    (M.98)
  • O-------O-------O-------O-------O-------O--------
    O-------O-------O-------O-------O

49
Study 2 Rating
  • 10 was rated higher--did not reach significance
  • 10 (1 in 10) greater (M1.25) 90 (9 in
    10)less (M.97)
  • O-------O-------O-------O-------O-------O-------
    -O-------O-------O-------O-------O
  • 10 (1 in 10) less (M.98) 90 (9 in
    10)greater (M.88)
  • O-------O-------O-------O-------O-------O--------
    O-------O-------O-------O-------O

50
Study 1 Reversal Error
  • Mean reversal
  • error per person

90 (9 times) less than .83
10 (10 times) greater than .33
51
User Needs Understanding
 Naval Forecasters Terminal Aerodrome
Forecast (TAF) posted at regular
intervals while fulfilling other duties
52
Method
 Talk-aloud while creating TAF
Microphone recorder
53
Synoptic Pattern Comparison
  • 1. Compare position of low in the model
    satellite
  • 2. Assess differences in movement and position
  • 3. Adjust forecast accordingly

54
Compare Predicted to Observed Values
  • 1. Access NOGAPS predicted pressure for
    current time 29.69
  • 2. Access current local pressure and
    29.69
  • subtract from NOGAPS
    - 29.64
  • .05
  • 3. Access NOGAPS predicted pressure for
    29.59
  • forecast period and subtract error amount
    - .05
  • 4. Forecast
    29.54

55
Results
  • Naval forecasters rely heavily on models
  • (1/3-1/2 source statements referred to
    models)
  • Statements implying understanding of model
    uncertainty
  • Model biases and
    strengths
  • Initialization of model run
  • Strategies for determining uncertainty
  • Evaluation of degree of uncertainty
  • Adjusting model predictions

56
Conclusions
  • Uncertainty?
  • Error in deterministic forecast?
  • Subsequent questionnaire study confidence is
    related to their assessment of model performance

57
Probability Problem
  • The probability that a woman getting a mammogram
    has breast cancer is 1. If the woman has breast
    cancer the probability is 80 that she will have
    a positive mammogram.
  • If the woman does not have breast cancer the
    probability that she will still have a positive
    mammogram is 10.
  • You have a patient that has a positive mammogram
    (no symptoms)--what is the probability she has
    breast cancer.

58
Frequency Problem
  • Ten out of every 1,000 women have breast cancer
  • Of those 10 women with breast cancer 8 will have
    a positive mammogram
  • Of the remaining 990 women without breast cancer,
    95 will still have a positive monogram
  • You have a sample of women who have positive
    mammograms in your screening (no symptoms)
  • How many of these women will actually have breast
    cancer?

59
Results Probability less is worst
10
30
70
90
Reversal error subjects eliminated from this
analysis
Write a Comment
User Comments (0)
About PowerShow.com