Getting better value from forecasting software: -where can the improvements come from?

1
Getting better value from forecasting software
-where can the improvements come from?

• Paul Goodwin
• University of Bath

Supported by Robert Fildes, Kostas Nikolopoulos,
Wing Yee Lee, Michael
Lawrence
2
Project aim

• To recommend practical improvements to
• forecasting software support systems through
• system design changes
• process improvements
• statistical modelling improvements

3
Main companies involved in research

• Pharmaceutical company
• Food manufacturer
• Domestic cleaning products manufacturer
• Major U.K. Retailer
• Brewer

4
Company approaches to forecasting

• Most has very large number of series to forecast
  - either monthly or weekly
• All used a software system that embodied
  statistical time series forecasting methods
• All judgmentally adjusted a large number of these
  forecasts
• -most adjustments were made by groups of
  managers in review meetings

5
Data supplied by companies

1. Software forecast
2. Final forecast (i.e. software forecast after
   judgmental adjustment)
3. Actual outcome

We also sat in on forecasting review meetings and
observed discussed use of software with
individual forecasters
6
Percentage of system forecasts that are adjusted
7
What we have found

• Many examples of good practice
• But some common issues.
• How can software use design address these
  issues?

8
On average did judgmental adjustments improve
accuracy?

• Where forecasts were adjusted (excluding the
  retailer)

9
But evidence that adjustment process can be
improved

• E.g. In one company While adjustments lead to
  mean improvement in absolute error of 4.31
• Half improvements less than 0.37
• Only 51.3 of system forecasts improved
• through MI adjustment

• What type of adjustments improve accuracy?
• Which are the most damaging to accuracy?

10
1. Only larger adjustments tend to improve
accuracy
Data from one typical companyCompany X
Size of adjustments measured as absolute
adjustment as of system forecast
11
2. Negative adjustments are more effective, on
average.
(Data is from all companies except retailer)
Some very large errors distort the mean here
12
3. Wrong direction adjustments are particularly
damaging

• E.g.

Adjusted forecast
System forecast
Actual demand
Mean absolute error worsened by 57.5 in
company 1 67.8 in company 2 101.5 in
company 3
13
Positive adjustments are more likely to be in the
wrong direction
(Data is from all companies except retailer)
14
Smaller adjustments are more likely to be in the
wrong direction
(Data is from all companies except retailer)
15
Optimism bias?

• Final adjusted forecasts tend to be too high by
  an average of 18.1
• System forecasts that are unadjusted on average
  are to high by 44. 3
• -so people seemed happy with these
• high forecasts

(Data is from all companies except retailer)
16
The retailer
17
Distinguishing between forecasts and decisions

• Make the forecast first
• e.g. I think well sell 200 units
• Then you can turn it into a decision
• e.g. I think we ought to produce 250
  units,
• in case demand is unexpectedly
  high

18
Summary of data analysis

• Avoid small adjustments
• -they tend to reduce accuracy
• Be careful with positive adjustments
• -they are often made when the system
• forecast actually needs to be reduced
• Beware optimism bias
• Be careful to distinguish between forecasts
  decisions

19
Can software help to solve the observed problems?

• Analogies to tackle optimism bias and improve
  accuracy adjustments
• Profiles to reduce wrong sided adjustments
• Advice to reduce small adjustments etc
• Restrictiveness -to reduce small adjustments
• Less severe restrictiveness -to deter trivial or
  unnecessary adjustments

20
Your views

• After presentation of each idea please indicate
  your views on the questionnaire

21
Analogies

• E.g. a promotion campaign will take place next
• month
• Software has access to a database of past
  promotions
• It selects 3 most similar promotions to
  forthcoming promotion and displays their
  estimated effect on demand
• It allows you to estimate the effect on demand
  of differences between forthcoming promotion
  the selected similar promotions

22
Adaptation judgment support
Click here for the interface in the demo
Similarity judgment support
Memory support
23
Profiles of the effects of events

• E.g. a promotion campaign will take place next
  month for 3 weeks
• Our forecasts need to take into account the
  timing of its effects

Pre-promotion dip
24
Time series graph
Free-plotting allowed
Gallery of promotion demand profiles
The demand profiles in the gallery are averages
of the past promotions of the same promotion
type, e.g. 40 Off. Experimental evidence has
shown that using these profiles could improve
forecast accuracy significantly regardless of the
predictability of the environment
25
On screen guidance

• System tells you when it thinks you should or
  should not make adjustments

26
(No Transcript)
27
Restrictiveness

• System will not allow judgmental adjustments
  below a certain size

28
(No Transcript)
29
Less severe restrictiveness
System requires user to give a reason for
adjustment before it allows the adjustment to be
made
30
(No Transcript)
31
Questions for breakout sessions

1. Are you satisfied with the way your software
   produces its statistical forecasts? If not what
   improvements would you like to see?
2. Do you think that your forecasting software could
   provide better support for including market
   information through your group's judgmental
   adjustments? If so how?
3. Are there any improvements you would like to see
   in the way your software can be used to analyse
   past errors and report forecasting accuracy?

32
End
33
Back to main slides
Future promotion
Step 1 Memory support
Step 2 Similarity judgment support
Step 3 Adaptation judgment support
(Details of the three support features may be
viewed by clicking on the labels.)
34
1. Memory Support

• The memory support consists of a database listing
  all of the past sales promotions (within a
  pre-determined time period) on a particular
  product so that there is no need to remember
  their details.

Interface diagram
2. Similarity judgment support
3. Adaptation judgment support
35
2. Similarity Judgment Support

• To search for the relevant cases in the database,
  you can use the combo boxes above the database to
  sort the past events.
• You can also select the cases you find useful and
  store them in the table above the combo boxes.

Interface diagram
3. Adaptation judgment support
1. Memory support
36
3. Adaptation Judgment Support

• The adaptation judgment support provides an
  analysis that helps you adapt past promotions to
  the future promotion.
• The ratios indicate the average changes to the
  promotional effect on sales if the promotion
  characteristic is changed from one alternative to
  another (e.g. from "Buy One Get One Free" to "3
  For 2").
• If you click the Details button next to a
  ratio, a bar chart will appear to show you the
  input values to that ratio. For instance, if
  there are only very few input values and/or they
  vary a lot, the ratio may not be a reliable
  reflection of the relationship between the
  promotion characteristic alternatives.
• An input value of a ratio is the relative
  difference in promotional effect between a pair
  of cases that have the same promotion
  characteristics apart from one. For example, If
  the ratio is between cases of Buy One Get One
  Free and 3 For 2, the pair of cases would
  differ likewise and only on this aspect.

Interface diagram
1. Memory support
2. Similarity judgment support