Ottawa, 79 November 2005 http:farmweb'jrc'cec'eu'intci 141

1
Advocacy and Narratives on Composite
Indicators Michaela Saisana michaela.saisana_at_jr
c.it European Commission Joint Research Centre
Ispra, Italy Composite Indicators
Workshop Ottawa, 7-9 November 2005
2
Prepared with Andrea Saltelli
Based on Andrea Saltelli (2005) Developing
Effective Data Based Lisbon Strategy Narratives,
XVII Villa Mondragone International Economic
Seminar on " CAPITALISM AND ENTREPRENEURSHIP
DYNAMICS BENCHMARKING EUROPES GROWTH, CEIS -
University of Rome "Tor Vergata July 6th 7th
2005, available at http//farmweb.jrc.cec.eu.int/
CI/
3
Hits on Composite Indicators 992 on
Scholar Google 35,500 on Google and
growing! A UNDP recent compilation of existing
CI lists some 120 indices. Most of these in the
last 4 years.
4
Several reviews of CIs News Workshops
http//farmweb.jrc.cec.eu.int/CI/
5

• Financial Times, October 27th, 2005
• 7 articles with narratives based on composite
  indicators!
• FTSE Eurofirst 300 Index, since merger (Thomson
  Datastream source)
• p.3 According to Global Witness, 8.45bn (7bn)
  of public money went unaccounted for between 1997
  and 2001 in Angola, which the United Nations
  ranks 160th out of 177 countries in terms of
  human development. Watchdog group Transparency
  International s corruption perceptions index,
  published last week, listed it as one of the
  worlds 10 most corrupt countries.
• p.4 Italian business confidence high. The
  Rome-based Isae Institutes confidence index
  increased to 89.5 from a revised 89.3 in
  September, the fifth consecutive monthly rise.
  (survey of about 4,000 manufacturers)

6

• Financial Times, October 27th, 2005
• p.4 US mortgage applications down. The industry
  trade groups seasonally adjusted index of
  mortgage applications for the week ended October
  21 declined 7.9 to 679.1, its lowest level since
  mid-April when the index stood at 672.6
• p.4 US activity index falls in September. The
  Federal Reserve Bank of Chicago said its gauge of
  the US economy fell in September as the impact of
  Hurricane Katrina was felt on production and
  employment. Its National Activity index fell to
  minus 0.71 in September .
• p. 8 Witness to change Milton Moskowitz thinks
  the best company rankings will have to monitor
  labour standards abroad to survive long term.
• p. 11 All lanes to London. European arrivals find
  hope and glory in a global metropolis.
• p.13 by drawing our attention to it. Having
  shot the messenger, we ought to consider the
  message

7
Center for Creative Leadership (CCL)
2005 News Release, May 16, 2005 Financial Times
Survey Ranks Center for Creative Leadership
(CCL) Among Top 10 Worldwide in Executive
Education
In 2003, BusinessWeek's biennial special report
on executive education ranked the Center 1st
worldwide in leadership education for the third
consecutive time. The Center also ranked 4th
worldwide among open-enrollment programs and 7th
among custom program providers in that report.
In a 2005 Financial Times survey, the Center
earned an overall Top 10 ranking worldwide among
providers of executive education. In addition,
CCL ranked 3rd worldwide for open-enrollment
programs. This marks the fourth consecutive year
that the Center has ranked in the Top 10 for
open-enrollment programs.
8
Technical definition Composite indicators are
mathematical combinations (or aggregations) of a
set of indicators. Conceptual definition
Composite indicators are based on
sub-indicators that have no common meaningful
unit of measurement and there is no obvious way
of weighting these sub-indicators (Source Note
on composite indicators, EC, Brussels, March
2002)
9

• European Commission, 2005, Information Note to
  the College of Economic Finance DG
• A list of new structural indicators to be
  developed includes
• Price convergence between EU Members States
• Healthy Life Years
• Biodiversity
• Urban population exposure to air pollution by
  ozone and
• Urban population exposure to air pollution by
  particles (PM10)
• Consumption of toxic chemicals
• Generation of hazardous waste
• Recycling rate of selected materials
• Resource productivity
• E-business indicator
• Can you guess how many of these are composite?

10

• ALL OF THEM! (One is a ratio of composites)
• Price convergence between EU Members States
• Healthy Life Years
• Biodiversity
• Urban population exposure to air pollution by
  ozone and
• Urban population exposure to air pollution by
  particles (PM10)
• Consumption of toxic chemicals
• Generation of hazardous waste
• Recycling rate of selected materials
• Resource productivity The definition of this
  indicator has now been established as the ratio
  of Gross Domestic Product (GDP, at constant
  prices) over Domestic Material Consumption (DMC).
  
• E-business indicator
• and yet their use within and without the EC is
  controversial.

11
ltlt it is hard to imagine that debate on the
use of composite indicators will ever be settled
official statisticians may tend to resent
composite indicators, whereby a lot of work in
data collection and editing is wasted or
hidden behind a single number of dubious
significance. On the other hand, the temptation
of stakeholders and practitioners to summarise
complex and sometime elusive processes (e.g.
sustainability, single market policy, etc.) into
a single figure to benchmark country performance
for policy consumption seems likewise
irresistible. gtgt Source Saisana M., Saltelli
A., Tarantola S. (2005) Uncertainty and
Sensitivity analysis techniques as tools for the
quality assessment of composite indicators,
Journal of the Royal Statistical Society - A,
168(2), 307-323.
12
ltltThe aggregators believe there are two major
reasons that there is value in combining
indicators in some manner to produce a bottom
line. They believe that such a summary statistic
can indeed capture reality and is meaningful, and
that stressing the bottom line is extremely
useful in garnering media interest and hence the
attention of policy makers. The second school,
the non-aggregators, believe one should stop once
an appropriate set of indicators has been created
and not go the further step of producing a
composite index. Their key objection to
aggregation is what they see as the arbitrary
nature of the weighting process by which the
variables are combined.gtgt Source Andrew
Sharpe, 2004, Literature Review of Frameworks
for Macro-indicators, report of the Centre for
the Study of Living Standards, Ottawa, CAN.
13
Handbook on Constructing Composite Indicators
Methodology and User GuideNardo, M. M. Saisana,
A. Saltelli and S. Tarantola (EC/JRC), A. Hoffman
and E. Giovannini (OECD), OECD Statistics Working
Paper JT00188147, STD/DOC(2005)3.http//www.olis
.oecd.org/olis/2005doc.nsf/LinkTo/std-doc(2005)3
The OECD - JRC handbook
14

• On the handbook on CI the pros
• Summarise complex, multi-dimensional issues to
  support decision-makers .
• Yet perhaps individual variables are more
  important for policy
• Easier to interpret than trying to find a trend
  in many separate indicators.
• For the layperson?
• Facilitate the task of ranking countries on
  complex issues in a benchmarking exercise .

15
Can assess progress of countries over time
on complex issues . CI are not famous for
this! Reduce the size of a set of indicators
or include more information within the existing
size limit . Place issues of country
performance and progress at the centre of the
policy arena . Advocacy Facilitate
communication with general public (i.e. citizens,
media, etc.) and promote accountability .
Advocacy
16

• while CIs cons are
• May send misleading policy messages if they are
  poorly constructed or misinterpreted .
• Poorly should be qualified
• May invite simplistic policy conclusions .
• May be misused, e.g., to support a desired
  policy, if the construction process is not
  transparent and lacks sound statistical or
  conceptual principles. Advocacy

17
and (cons) The selection of indicators
and weights could be the target of political
challenge . When a CI exacerbates disagreement
rather than focusing minds May disguise
serious failings in some dimensions and increase
the difficulty of identifying proper remedial
action . Hardly for policymakers CI more for
awareness than for action May lead to
inappropriate policies if dimensions of
performance that are difficult to measure are
ignored . As above. Hardly the fault of a CI, as
we do not know what we do not know
18
but two more pros not in the handbook
Constructing/underpinning narratives for lay or
literate audiences . More in this talk
19
and Comparing effectively complex
dimensions with one another . Andre
Sapirs latest work (Globalisation and the
Reform of European Social Models, 2005)
20
A crucial criticism of CI is the con CI May
be misused, e.g., to support a desired policy, if
the construction process is not transparent and
lacks sound statistical or conceptual principles.
Advocacy In a criticism to Cavenaugh et al.s
Alternatives to Economic Globalisation, Martin
Wolf notes (Why Globalisation Works,
2004) ltltPerhaps to disguise the failure of such
closed communities, measurements of successful
performance would no longer be based on
traditional economic growth figures like GDP and
GNP, but rather on more subjective social and
environmental characteristics gtgt How about
psychological characteristcs?
21
ltltComposite indicators are much like mathematical
or computational models. As such, their
construction owes more to the craftsmanship of
the modeller than to universally accepted
scientific rules for encoding. As for models, the
justification for a composite indicator lays in
its fitness to the intended purpose and the
acceptance of peers (Rosen, 1991) gtgt
22
As suggested by Box (1979) all models are
wrong, some are useful. The quality of a
composite indicator is thus in its fitness or
function to purpose. The economist A. K. Sen,
Nobel prize winner in 1998, was initially opposed
to composite indicators but was eventually
seduced by their ability to put into practice his
concept of Capabilities (the range of things
that a person could do and be in her life) in
the UN Human development index . Sen A.
1989 Development as Capabilities Expansion,
Journal of Development Planning, 19, 41-58
23
The need for a summary measure

• Give me a measure a single measure which is
  as vulgar as GNP per capita, but not as
  insensitive to broader aspects of human lives.
• Mahbubul Haq

24
According Cherchye et al., the lack of
consensus is a defining property of composite
indicators, which goes hand in hand with CI
suitability to advocacy.
One market, one number. A composite
indicator assessment of EU internal market
dynamics, KEI project, University of Leuven,
2005, L. Cherchye, C.A. Knox Locell, W. Moesen,
T. Van Puyenbroeck
25

• A short story on CI

26
Report from the EC to the Spring European Council
2004, Annex 1
27
Enter the FT analysts
Source Spring Report, European Commission 2004
Source Financial Times January 22 2004
28
From structural indicators (EUROSTAT, short, long
lists) to league table (Financial Times) Long
list of indicators (gt 100) Short List of 14
Synoptic tables League tables
29
CI for Advocacy CI and Lisbon narratives

30
As noted by A. K. Sen, composite indicators are a
powerful tool for advocacy. And the hunger of the
economically literate press for statistic based
narrative is noticeable (interview of FT with
Jean Philippe Cotis, OECD).
31

• Wim Kok warns in its now famous report
• An ambitious and broad Lisbon reform agenda
  needs a clear narrative
• Barrosos Growth and jobs go in the direction
  of a possible narrative for a simplified Lisbon
  yet the production of anti-Lisbon narratives seem
  to have been more intense
• The stability pact strangles the EU economies,
• EU regulations are a systemic hindrance to
  business,
• Services directive fosters social dumping

32
That a consistent segment of
the EU elites rejects these narratives hasnt
helped defuse them, as the results from
constitutional referendums have shown. And, as
a counter proof, where are there today narratives
in favour of the EU reform?
33
Narratives in the EU Could EU
leaders make more effective use of statistical
information to build effective narratives to
promote structural reform and growth in the EU?
it is a pity that attempts to use even
comparatively bland measures - such as the
"naming and shaming" of laggards - have been
dropped. In other areas, such as the
implementation of single-market legislation or
state-aid controls, "scoreboards" have played a
useful role in bringing peer pressure to bear on
national decision-makers. Mario Monti, FT, March
21, 2005.
Mario Monti
34

• Narratives in the EU
• Could statistical information help to build sound
  narratives
• To the effect that excessive deficits are passed
  on to future generation (the OECD example)
• To the fact that simultaneously limiting labour
  and capital mobility within EU 25 implies capital
  migrating elsewhere (service directive)
• To the impact of globalisation and ageing on the
  EU as a whole (e.g. wage level in the EU
  depending from wage level elsewhere)
• To the impact of systemic rigidities or
  distributional coalitions (in Olsons sense) on
  growth?

35

• Narratives in the EU (continued)
• The stricter the employment protection
  legislation of a model, the lower its employment
  rate, and
• The lower the level of secondary education
  attainment, the higher the risk of poverty. By
  contrast, the extent of redistributive policy
  only plays a secondary role

Andre Sapir, 2005, Globalisation and the Reform
of European Social Models
36

• Caveat These are just examples, on which an
  abundant statistical and economic literature
  already exists, clearly not a manifesto for
  political advocacy.

37
A counter example (in the sense of not being data
driven) the ambitious new meta-narrative of a
redefined human mission to underpin the European
Dream invoked by Jeremy Rifkin (2004). In this
work the author defines EU style of inclusiveness
and tolerance against the American Dream of
individual self-achievements and materialistic
consumption. Data driven? Economically
defensible?
Jeremy Rifkin
38
At the same time activists advocacy may lead to
the so-called Rhetoric Selection of statistical
information whereby feelings and facts are
merged in reach for the audiences empathy and
wallets (Rosling et al., 2005). In other
words, there is no guarantee that a Narrative
would not precede the data. On the contrary,
isnt this always the case? The separabily versus
non-separability of feelings and facts is one of
the distinctive features of normal versus
post-normal Science (Funtowicz and Ravetz,
1990).
Hans Rosling, Professor of International Health,
Karolinska Institute, Sweden
39
We are not arguing that the best (in the
fitness sense) narratives are those based on
measurements. The ghost of the Polish plumber was
apparently an apt protagonist in the French Non
campaign (no cost benefit analysis or CI needed).
Yet a narrative could have been built on
available data to negotiate with voters on the
impact of globalisation and the role of EU
enlargement in it.
40
More to the point, the (missing) discourse on
enlargement could not have been developed without
data, and a debate on those. any competent
statistician knows that "just collecting numbers"
leads to nonsense so in "Definition and
Standards" we put "negotiation" as superior to
"science", since those on the job will know of
special features and problems of which an expert
with only a general training might miss.
(Funtowicz and Ravetz in Uncertainty and Quality
in Science for Policy, 1990)We would add that,
however good the scientific basis for a given
composite indicator, its acceptance relies on
negotiation and peer acceptance.
41

• What is a composite indicator?
• A simplistic presentation and comparison of
• performance in a given area to be used as
• a starting point for future analysis.
• - JRC, an information server on composite
  indicators
• provides a clue to a matter of larger
• significance or make perceptible a trend or
• phenomenon that is not immediately
• detectable.
• - Hammond et al, 1995