Title: Improving accuracy in inefficient firm level forecasts: with lessons for macro-forecasters.
1Improving accuracy in inefficient firm level
forecasts with lessons for macro-forecasters.
- Robert Fildes Lancaster University Centre for
Forecasting - Paul Goodwin, University of Bath
- Supported by Kostas Nikolopoulos
- Wing Yee Lee, University of Bath
An EPSRC Research Project A collaboration between
Lancaster Centre for Forecasting,, and 10
companies, including McBride, Interbrew, Heinz,
Cow and Gate
2Outline
- How and why forecasting is carried out at the
company level - Similarities and differences compared to
macroeconomic forecasting - Analysis of company level forecasts
- Explaining the excessive use of judgmental
interventions - Potential lessons for macroeconomic forecasting
3- Focus
- on short and medium term demand forecasting in
supply chain companies - Forecasts in these companies are used in
decisions relating to - - logistics,
- - human resource planning,
- - stock control,
- - purchasing,
- - cash flow management.
Service - inventory investment tradeoff curves
The wrong product in the wrong place at the wrong
time - forecasters motivated to improve accuracy
4Information affecting the supply chain
The full model Ordersf(past orders, Sales,
forecast sales, promotions, Events) The basic
model Ordersf(past orders) judgemental
estimates of promotions
5A complete statistical model?Is it possible?
Marketing factors
- The Problems
- Too complex
- Incomplete data on many drivers
- Unique events
- No available statistical expertise
- Management understanding acceptance
- Belief that managerial expertise is best
- Cost of cleaning past data to remove unusual
events
The Practical Solution Capture the unusual
complexity by managerial judgement
6Complementary nature of statistical forecasts
management judgment
Organisationally based Forecasting combines
statistical analysis with managerial judgement
- humans are adaptable and can take into account
one-off events, but they are inconsistent and
suffer from cognitive biases - statistical methods are rigid, but consistent,
and can take into account large volumes of
information
7How companies make their forecasts
- 1 Package embodying simple robust time
series methods is used (e.g. simple exponential
smoothing, Holt-Winters) - Statistical forecast can then be judgmentally
adjusted, usually at a meeting. - ostensibly for special events like
promotion campaigns
8Differences between macroeconomic and company
forecasting
- Very large numbers of series need to be forecast
regularly in companies - Macro forecasters are usually statistically
trained, company forecasters are not - Unlike macro-economic forecasting there are
usually no company forecasts that compete to
forecast the same variable - Macro forecast variables are at a higher level
of aggregation, but lower frequency - Macro forecasters employ explanatory models,
company forecasters tend to use univariate methods
9Similarities with macroeconomic forecasting
- Process of forecasting similar
- Judgment used in model selection and fitting and
in subsequent adjustment of resulting forecast - Difficult to tease out these separate effects of
judgment - Starting conditions often not known for certain
10Company Evidence Data (4 U.K. based companies)
The EPSRC Research Project
- 578 SKUs, 3668 Months
- Company A Major UK Manufacturer of Laundry,
household cleaning and personal care products - - 234 SKUs x 9 months -gt 908 triplets
- Company B Major International Pharmaceutical
Manufacturer - - 134 SKUs x 24 months -gt 2023 triplets
- Company C Major International canned Food
Manufacturer - 210 SKUs x 6 months -gt 908 triplets
- 548 SKUs, 2039 Weeks
- Company D Major UK Retailer (over 26000 SKUs)
- - 548 SKUs x 52 weeks -gt 2039 triplets
11Research Issues and Hypotheses (practical and
theoretical)
- Does adjustment improve accuracy?
- Under what circumstances
- Are the organisational forecasts rational?
- Do inefficiencies arise from organisational
processes - Can the forecasts be improved?
- By Model based improvements
- By software improvements
- By process improvements
12Evidence on frequency of judgmental adjustments
from 4 of our companies
- A significant of forecasts are judgmentally
adjusted
Companies Data N Adjusted Adjusted
A Monthly 3264 2034 62
B Monthly 873 744 85
C Monthly 1416 942 67
D1 Weekly 12789 1851 14
D2 Weekly 44899 4392 10
Total 63241 9963 16
13Results MAPE by Adjustment Size
Small Large Adjustments Adjustments
ALL Monthly data - Final vs. System Size
Error
Final Forecast System Forecast
14Results
Company D - Final vs. System Size
15For first two companies larger adjustments tend
to lead to the greatest gains in accuracy
Companies Small Small Medium Medium Large Large Very Large Very Large
Companies RelAdjlt10 RelAdjlt10 10-50 10-50 50-150 50-150 gt150 gt150
Companies SFC Improvement SFC Improvement SFC Improvement SFC Improvement
A 11.8 -0.2 20.6 5.5 44.0 13.4 60.0 11.8
B 1 0.3 19.8 2.3 79.4 44.4 71.4 54.9
C 12.7 -0.1 22.0 5.7 36.6 -5.5 51.3 -30.2
D1 12.5 0 19.0 -7.9 18.3 -48.3 43.6 -144.4
D2 27.9 1.0 26.9 -1.2 25.3 -36.8 No Obs. No Obs.
Do not make small adjustments!
SFC statistical forecast
16The Types of Error
- Adjustments in the wrong direction
- With expected positive information
- Adjust up, but actual is below system Finalgt
Systemgt Actual - With expected negative information
- Adjust down, but actual is above system
- Adjustment in the right direction
- Adjustment is not strong enough
- With positive info, SystemgtFinalgt Actual
- With negative info, ActualltFinalltSystem
- Adjustment is too strong
- With Positive info, SystemltActuallt Final
- With Negative info, FinalltActualltSystem
17Adjustments made in the wrong direction reduce
accuracy
Wrong Direction adjust from SFC Point improved Overshoot SFC Point improved Undershoot SFC Point improved
A 12.90 -9.60 10.10 -2.00 27.00 14.50
B 11.90 -13.10 11.80 -1.30 30.00 16.80
C 13.50 -22.40 13.60 -2.10 22.70 10.00
D1 15.70 -34.10 12.10 -15.10 26.10 -0.30
D2 32.90 -36.40 13.00 -7.60 36.60 16.30
SFC statistical forecast
18Bias
Fcasts
Fcasts
19Modelling the forecastsStatistical Issues
For unbiasedness
- Errors heteroscedastic with outliers
- Can firms be pooled?
- Solutions
- Errors normalised by standard deviation of
actuals and analysed by size of adjustment
20Bias of final forecasts
Company N (no. dropped) a, d b Coefficient Std. Error
A 1810 (7) Significant -0.12 0.35
B 326 (0) Significant -0.13 0.52
D1 887 (1) Significant -0.35 0.51
D2 2630(0) Significant -0.47 0.63
C 535 (5) Significant -0.15 0.37
Also evidence that for A,B and C companies bias
is reduced through judgmental adjustment
21Can we model the error to ensure an efficient
forecast?? improved forecast error
The models
Adjust is proxy for Market intelligence
Efficiency all available information is being
used effectively
The last- the 50/50 model Blattberg Hoch
22The Effects of Information
- Define Positive (negative) information as when
the system forecast is adjusted upward
(downward) - Questions
- Are there differences in accuracy?
- Are there differences in efficiency?
- yes affecting both mean bias and regression bias
- Can these be capitalised on to design rules to
improve accuracy?
23Comparative Results Major gains with some
companies
Final Forecast System forecast Optimal combination
MAPE (Median APE) MAPE (Median APE) MAPE (Median APE)
A Positive Info 34.5 (12.5) 27.5 (14.0) 28.1 (12.7)
Negative Info 25.8 (14.0) 45.7 (19.8) 27.2 (13.7)
B Positive Info 71.6 (18.2) 55.8 (17.1) 49.0 (14.5)
Negative Info 27.4 (10.9) 69.5 (15.6) 19.4 (11.2)
D1 Positive Info 68.9 (35.8) 35.5 (17.8) 28.4 (14.2)
Negative Info 48.2 (29.7) 58.7 (20.0) 34.9 (18.4)
D2 Positive Info 72.1 (50.0) 40.3 (24.6) 21.9 (16.1)
Negative Info 30.2 (19.8) 43.5 (30.3) 15.0 (10.9)
C Positive Info 28.8 (15.9) 25.2 (19.2) 23.1 (13.8)
Negative Info 28.8 (12.7) 39.2 (16.9 26.4 (13.0)
24Weighting the Information Sources
25Conclusions from the Empirical Analysis
- 1 out of 3 adjustments is in the wrong direction!
- They are very costly
- b) Inefficiencies persist
- effects are exaggerated for positive info
- Double counting? Wishful thinking?
- c) Small adjustments (lt10-20) should not be
made! - d) Final forecasts are (usually) more accurate
than system forecasts - e) Reweighting of system forecast and market
adjustment leads to improved accuracy -
- is it the data, the drivers or the forecasting
process that makes the difference? Can we by
understanding the process, develop an FSS that
supports improved accuracy
26- The adjusted forecasts are biased and
inefficient - the process through which the forecasts are
adjusted and the Market Intelligence estimated
leads to inaccuracies - Hence there may be gains in improving the
quality of judgmental inputs to forecasts - But most forecasting software do not provide
facilities to support judgment. - Instead, packages promote their statistical
power..! - To understand the process of adjustment
- To influence the adjustments through better FSS
design
Aim
27Why do managers adjust forecasts excessively?
- The advice perspective
- We can regard the computer systems forecasts
- as a form of advice
- Our willingness to accept advice from a machine
can vary
28Theories of why we accept advice
- Yaniv Kleinberger (2000)
- People trust their own forecasts more because
they have greater access to the rationale for
these - - No rationale or explanation is provided in
most forecasting software packages employing
univariate methods -
29Theories of why we accept advice
- Yaniv Kleinberger (2000)
- Weight attached to advice is dependent on
reputation of adviser - - but negative information about an adviser is
perceived to be more diagnostic than positive -
- -In forecasting, noise special events may
contribute to a negative perception - Kaplan, et al (2001) found people were more
likely to rely on a system when its accuracy was
not disclosed.
30Why do managers adjust forecasts excessively?
- Misconceptions of randomness
- We tend to see systematic patterns in what are
really random movements in graphs.
31Would you want to adjust the statistical forecast
for month 21 (below)?
32- Ability to manipulate the system and carry out
what if analyses leads to an illusion of
control -
(Davis et al 1994) - -manipulation involves effort and is
perceived - to involve skill
- -people become overconfident in their
- judgments
33Manipulations
Change the responsiveness of the forecasts
Change the forecasting method...
Change the amount of past data used..
34Result of overconfidence
- Over weighting of judgmental forecasts relative
to statistical forecasts - -even when evidence shows judgment is less
accurate.
In one study people continued to rely on their
judgment despite receiving messages from the
computer like Please be aware that you are
18.1 LESS ACCURATE than the statistical forecast
provided to you.
35Why do managers adjust forecasts excessively?
- 4. Confusion between forecasts and
- -targets
- -decisions
-
-
-
- -politically acceptable numbers
A forecast I think demand will be 200
units A decision I think we ought to produce
250 units, in case demand is
unexpectedly high
36Why do managers adjust forecasts excessively?
- 5. Need for ownership of the forecasts
- Demonstration that youve contributed to the
forecasting process and earned your salary - Demonstration of your marketing expertise to
your colleagues at meetings
37- 6. Base-rate information vs Case-specific
information - Kahneman and Tversky
- Group A This description has been drawn
randomly from a folder containing descriptions of
30 engineers 70 lawyers - Group B This description has been drawn
randomly from a folder containing descriptions of
70 engineers 30 lawyers - -What is the probability that Tom W is an
engineer?
"Tom W. is of high intelligence, although lacking
in true creativity. He has a need for order and
clarity, ...... in which every detail finds its
appropriate place. His writing is rather dull and
mechanical, enlivened by somewhat corny puns
He has a strong drive for competence. He seems to
feel little sympathy for other people .
Self-centered, he nonetheless has a deep moral
sense.
38- People over-emphasise case specific information
at the expense of base-rate information - Our company forecasters focused on each demand
figure on the graph as being a special case. - -the base rate information represented by the
statistical forecasts was thus under weighted
39Why do managers adjust forecasts excessively?
- 7. Recency bias
- - only the recent past was thought to
- be relevant
- Statistical methods were fitted to
- very short sets of data (often 6
- months to 2 years)
-
- Statistical methods not given much of a chance
- -reinforcing the forecasters confidence in
the relative accuracy of judgment
40Potential lessons for macroeconomic forecasting
- 1. Lessons relating to when to intervene
- 2. Lessons relating to improving judgmental
estimation
41Lesson relating to when to intervene
- Beware misconceptions of randomness,
underweighting of base-rate information recency
biases - -require recording of reasons for judgmental
intervention - Beware overconfidence in judgment
- - restrict what-if analyses?
- Beware numbers masquerading as forecasts
- Build explanations into software
- so its advice carries greater weight
42 Lessons relating to improving judgmental
estimation - Supporting the use of analogies
- Involves identifying similar products, or similar
promotion campaigns, and using these as a basis
for the judgmental forecast -
- E.g. We have a 2 week B.O.G.O.F promotion in the
North region starting on 5 June 2006 - Database identifies most similar cases
-
-
43 Supporting the use of analogies
- Support for different stages of judgmental
process - Memory support so forecaster avoids need to
recall past cases - Similarity support helps forecaster to identify
most similar past cases - Adaptation support -helps forecaster to adapt
from similar past cases to specifics of current
case - Eg. -Most similar past BOGOF promotion may
have lasted for 3 weeks, - - Forthcoming BOGOF promotion
lasts for only 2 weeks. - - Database will estimate effect of
2 weeks rather than 3 - from all past promotions..
44The interface
Adaptation judgment support
Similarity judgment support
Memory support
45Lessons relating to improving judgmental
estimation
- Using profiles to estimate effects of special
events over time.. - E.g. timing effects of unexpected financial
crisis in SE Asia
46(No Transcript)
47Lessons relating to improving judgmental
estimation
- Using decomposition to reduce the demands of
holistic estimation
With some obvious macro analogies
48Macroeconomic forecasting errors
Can the lessons learnt in micro forecasting help?
- Evidence of forecast failure (Fildes Stekler,
2002) - Cyclical turns
- They are missed
- Conservatism
- in underestimating growth in periods of
expansion, overestimating in periods of
contraction - Inflation underestimated when accelerating,
overestimating when slowing - Rationality
- Observed inefficiencies
- Failure to improve over time
49Explaining the errors (Stekler, 2007)
- Model derived errors
- Structural change
- Model and data interactions
- Incompatibility between model / data vintages
- Forecasters loss functions
- Institutional effects
- Judgemental effects
- Bias
- anchors
50Our focus judgemental adjustments In macro
forecasting, why are they made? (Donihue, J.
Forecasting, 1993)
- incompleteness in model
- dotcom
- structural change,
- New exchange rate regime
- missing variable in historic data base
- SE Asia financial crisis
- data inadequacies
- Current data or revised data
- recent level errors
51Common Features - micro forecasting in macro
forecasting
- Biased and inconsistent forecasts
- Reputational effects
- Herding (Smith, 2002 most actuals outside range
of forecasts) - Political pressues (?)
- Limited historic record keeping
- No monitoring of reasons for judgement
- Final forecast error monitoring
- No learning
- But
- Judgemental adjustments add value
52Conclusions
- Judgment is employed in both macroeconomic and
company forecasting - This leads to improved accuracy in both domains
- but still much scope for improvement
- Macro forecasters should
- Monitor judgemental forecast accuracy (perhaps
they do?) - Use a notes system to justify adjustments
- Use support systems to understand the
effectiveness of their interventions