Composite Indicators The Controversy and the way forward

1
Composite Indicators - The Controversy and the
way forward Andrea Saltelli, Michela Nardo,
Michaela Saisana and Stefano Tarantola European
Commission Joint Research Centre of
Ispra andrea.saltelli_at_jrc.it Statistics,
Knowledge and Policy OECD World Forum on Key
Indicators Palermo November 2004
2
Prepared with Michela Nardo, Michaela
Saisana Stefano Tarantola
Based on 3 Saisana M., Saltelli A., Tarantola
S., 2005, Uncertainty and Sensitivity analysis
techniques as tools for the quality assessment of
composite indicators, J. R. Stat. Soc. A, 168(2),
1-17. 11 Joint OECD JRC handbook on good
practices in composite indictors building.
3

• Outline
• CI controversy
• Composite Indicators as models
• Wackernagels critique of ESI
• Putting the critique into practice the TAI
  example
• Conclusions

4

• CI controversy
• EU structural indicators scoreboards versus
  indices

5
Report from the Commission to the Spring European
Council 2004, Annex 1
Relative Performance
Relative Improvement in Performance (av. since
1999)
6
Relative Performance
7
Assessing policies Green Country policy on a
good path Yellow Country policy on a bad path
(expert judgment)
8
Enter the FT analysts
Source Spring Report, European Commission 2004
Source Financial Times Thursday January 22 2004
9
Categorisation (star rating) in three
groups LEADERS UK, NL SE, DK, AT,LU MIDDLE OF
THE ROAD DE, FI, IE, BE, FR LAGGARDS IT, GR,
ES, PT done by FT and based likely on same
synoptic performance and improvement tables in
the Spring Report, 2004, Annex 1 (yellow-green
boxes) Like in the UK NHS hospital rating
10
Can league tables be avoided? Or are they an
ingredient of an overall analysis and
presentational strategy Long list of
107 Short List of 14 Synoptic tables
League tables
11
ltltThe aggregators believe there are two major
reasons that there is value in combining
indicators in some manner to produce a bottom
line. They believe that such a summary statistic
can indeed capture reality and is meaningful, and
that stressing the bottom line is extremely
useful in garnering media interest and hence the
attention of policy makers. The second school,
the non-aggregators, believe one should stop once
an appropriate set of indicators has been created
and not go the further step of producing a
composite index. Their key objection to
aggregation is what they see as the arbitrary
nature of the weighting process by which the
variables are combined.gtgt Literature Review of
Frameworks for Macro-indicators, Andrew Sharpe,
2004, Centre for the Study of Living Standards,
Ottawa, CAN.
12
Reviews on methodologies and practices on
composite indicators State-of-the-art Report
on Current Methodologies and Practices for
Composite Indicator Development (2002) Michaela
Saisana Stefano Tarantola, European Commission,
Joint Research Centre Composite indicators of
country performance a critical assessment (2003)
Michael Freudenberg, OECD. Literature Review of
Frameworks for Macro-indicators (2004), Andrew
Sharpe, Centre for the Study of Living Standards,
Ottawa, CAN. Measuring performance An
examination of composite performance indicators
(2004) Rowena Jacobs, Peter Smith, Maria Goddard,
Centre for Health Economics, University of York,
UK. Methodological Issues Encountered in the
Construction of Indices of Economic and Social
Well-being (2003) Andrew Sharpe Julia Salzman
Methodological Choices Encountered in the
Construction of Composite Indices of Economic
and Social Well-Being, Julia Salzman , (2004)
Center for the Study of Living Standards ,
Ottawa, CAN. http//farmweb.jrc.cec.eu.int/ci/
13

• Pros Cons (Saisana and Tarantola, 2002)
• Pros
• Composite indicators can be used to summarise
  complex or multi-dimensional issues, in view of
  supporting decision-makers.
• Composite indicators provide the big picture
  . They facilitate the task of ranking
  countries on complex issues.
• Composite indicators can help attracting public
  interest
• Composite indicators could help to reduce the
  size of a list of indicators .

14

• Cons
• Composite indicators may send misleading,
  non-robust policy messages if they are poorly
  constructed or misinterpreted or may invite
  politicians to draw simplistic policy conclusions
  
• The construction of composite indicators
  involves stages where judgement has to be made
  the selection of sub-indicators, choice of model,
  weighting indicators and treatment of missing
  values etc.
• There could be more scope for disagreement among
  Member States about composite indicators than on
  individual indicators .

15
Pros Cons (JRSS paper) it is hard to
imagine that debate on the use of composite
indicators will ever be settled official
statisticians may tend to resent composite
indicators, whereby a lot of work in data
collection and editing is wasted or hidden
behind a single number of dubious significance.
On the other hand, the temptation of stakeholders
and practitioners to summarise complex and
sometime elusive processes (e.g. sustainability,
single market policy, etc.) into a single figure
to benchmark country performance for policy
consumption seems likewise irresistible.
16

• Composite indicators as models and the critique
  of models

17
Indicators as models and the critique of models
The nature of models, after Rosen
18
The critique of models
After Rosen, 1991, World (the natural system)
and Model (the formal system) are internally
entailed - driven by a causal structure.
Efficient, material, final for world formal
for modelNothing entails with one another
World and Model the association is hence the
result of a craftsmanship.
19

• Wackernagels critique of ESI

20
The critique of indicators
Environmental sustainability Index, figure from
The Economist, Green and growing, The Economist,
Jan 25th 2001, Produced on behalf of the World
Economic Forum (WEF), and presented to the annual
Davos summit this year.
21
The critique of indicators Robustness
Mathis Wackernagel, mental father of the
Ecological Footprint and thus an authoritative
source in the Sustainable Development expert
community, concludes an argumented critique of
the study done presented at Davos by noting
22
"Overall, the report would gain from a more
extensive peer review and a sensitivity analysis.
The lacking sensitivity analysis undermines the
confidence in the results since small changes in
the index architecture or the weighting could
dramatically alter the ranking of the nations.
The critique of indicators Robustness
23
The quality of a composite indicator is in its
fitness or function to purpose. The economist
A. K. Sen, Nobel prize winner in 1998, was
initially opposed to composite indicators but was
eventually seduced by their ability to put into
practice his concept of Capabilities, the
range of things that a person could do and be in
her life.Sen A., 1989, Development as
Capabilities Expansion, Journal of Development
Planning 19, 41-58.
The critique of indicators Fitness
24
The example of the capabilities is relevant to
the issue CI are supposedly good at capture
complex (someone would say poorly defined)
concepts such as sustainability, welfare,
achievement of an EU internal market,
competitiveness, etc. Said otherwise, complex
processes call for scoreboards, and scoreboards
cry for an index.
The critique of indicators Fitness
25
In discussing pedigrees matrices for statistical
information Funtowicz and Ravetz note (in
Uncertainty and Quality in Science for Policy,
1990 6) any competent statistician knows
that "just collecting numbers" leads to nonsense
so in "Definition and Standards" we put
"negotiation" as superior to "science", since
those on the job will know of special features
and problems of which an expert with only a
general training might miss. We would add
that, however good the scientific basis for a
given composite indicator, its acceptance relies
on negotiation and peer acceptance.
The critique of indicators Fitness
26
(1) A composite constructed on the basis of
underlying indicators with high internal
correlation will give a very robust CI, whose
values and ranking are moderately affected by
changes in the selection of weights, the
normalisation method and other steps involved in
the analysis (see paper, this conference).
Open issues in CI Building 1 Variables
correlation
27
(2) When building composite indicators using
automated tools such as factor analysis, one
seeks to obtain a set of totally uncorrelated new
variables. While this can be a powerful tool to
benchmark countries performance, or to produce
e.g. leading or lagging synthetic indicators, the
interpretation in terms of original variables
becomes more difficult.
Variables correlation
28
(2) At the same time, it would be very difficult
to imagine a composite indicator made of truly
orthogonal variables.(3) In a multicriteria
context, one would consider the existence of
correlation among the attributes of an issue as a
feature of the issue, not to be compensated for.
A cars speed and beauty are likely correlated
with one another, but this does not imply that we
are willing to trade speed for design.
Variables correlation
29
(1) Munda, and Nardo, 2003 12, noticed how
weights, customarily conceived as importance
measures, act in practice as substitution rates,
e.g. wi/wj is the ratio of substitution (or
compensation) of indicator i with indicator
j.
Open issues in CI Building 2 Compensability
30
(2) This may be perceived as an important
limitation of a CI (e.g. literacy should not be
traded with GDP per capita). When one is not
willing to accept this kind of trade offs, e.g.
when the variable cannot be compensated with
another, a multi criteria approach can be
applied.
Compensability
31
(2) This may be perceived as an important
limitation of a CI (e.g. literacy should not be
traded with GDP per capita). When one is not
willing to accept this kind of trade offs, e.g.
when the variable cannot be compensated with
another, a multi criteria approach can be
applied. See paper for a simplified description
of a Condorcet-type of ranking procedure based on
Munda, 1995, 13. This approach produces
rankings (ordered sequence of countries) instead
of an index.
Compensability
32
(3) The ordering thus obtained is only based on
the weights, and on the sign of the difference
between countries values for a given indicator,
the magnitude of the difference being ignored.
(4) With this approach no compensation occurs.
To exemplify, a country that does marginally
better on many indicators comes out better than a
country that does a lot better on a few ones
because it cannot compensate deficiencies in some
dimensions with outstanding performances in
others.
Compensability
33
Points touched upon in this brief discussion of
open issues in CI building are tackled in a
forthcoming joint paper from OECD and JRC on
composite indicators building. It aims to be a
guide to the construction and use of CI.
Ongoing work the OECD JRC handbook
34
Theoretical framework - What is badly defined is
likely to be badly measured. Data selection
The quality of composite indicators depends
largely on the quality of the underlying
indicators. Multivariate analysis Multivariate
statistic is a powerful tool for investigating
the inherent structure in the indicators set.
Imputation of missing data The idea of
imputation is both seductive and dangerous.
Ongoing work the OECD JRC handbook
35
Normalisation Avoid adding up apples and pears.
Weighting and aggregation Relative importance
of the indicators and compensability issues.
Robustness and sensitivity The iterative use
of uncertainty and sensitivity analysis during
the development of a composite indicator can
contribute to its well-structuring.
Ongoing work the OECD JRC handbook
36
Link to other variables Correlation with other
simple indicators or composite indicators.
Visualisation If arguments are not put into
figures, the voice of science will never be heard
by practical men.Back to the real data
Deconstructing composite indicators for
analytical purposes.
Ongoing work the OECD JRC handbook
37
http//farmweb.jrc.cec.eu.int/ci/