Ottawa, 79 November 2005 http:farmweb'jrc'cec'eu'intci 135

1
Steps in the Construction of a Composite
Indicator Michaela Saisana michaela.saisana_at_jrc
.it European Commission Joint Research Centre
Ispra, Italy Composite Indicators
Workshop Ottawa, 7-9 November 2005
2
Handbook on Constructing Composite
IndicatorsMethodology and User GuideNardo, M.
M. Saisana, A. Saltelli and S. Tarantola
(EC/JRC), A. Hoffman and E. Giovannini (OECD),
OECD Statistics Working Paper JT00188147,
STD/DOC(2005)3.http//www.olis.oecd.org/olis/200
5doc.nsf/LinkTo/std-doc(2005)3
The OECD - JRC handbook
3
Steps in the Development of an Index
stakeholder involvement
Step 1. Developing a theoretical framework Step
2. Selecting indicators Step 3. Multivariate
analysis Step 4. Imputation of missing
data Step 5. Normalisation of data Step 6.
Weighting and aggregation Step 7. Robustness and
sensitivity Step 8. Links to other
variables Step 9. Back to the details Step 10.
Presentation and dissemination
4
Participatory Techniques
Decision-making Processes DEPENDENT on SOCIETY
The involvement of society in policy debates
where scientific issues are relevant is being
recognised as essential in new governance
processes
() it is clear that once that they are in
possession of all the facts, ordinary citizens
can engage high-quality dialogue with the
experts and arrive at a reasonable consensus with
them. Ph. Busquin Santander. 2000
5
Step 1. Developing a theoretical framework
What is badly defined is likely to be badly
measured Excerpt For example, the Growth
Competitiveness Index (GCI) developed by the
World Economic Forum is founded on the idea that
the process of economic growth can be analysed
within three important broad categories the
macroeconomic environment, the quality of public
institutions, and technology."
6
Step 1. Developing a theoretical framework
After Step 1. the constructor should have
A clear understanding and definition of the
multidimensional phenomenon to be measured. A
nested structure of the various sub-groups of the
phenomenon. A list of selection criteria for
the underlying variables, e.g., input, output,
process. Clear documentation of the above.
7
Step 2. Selecting variables
A composite indicator is above all the sum of
its parts Excerpt The strengths and
weaknesses of composite indicators largely derive
from the quality of the underlying variables.
While the choice of indicators must be guided by
the theoretical framework for the composite, the
data selection process can be quite subjective as
there may be no single definitive set of
indicators.
8
Step 2. Selecting variables
After Step 2. the constructor should have
Checked the quality of the available
indicators. Discussed the strengths and
weaknesses of each selected indicator. Made
scale adjustments, if necessary. Created a
summary table on data characteristics, e.g.,
availability (across country, time), source, type
(hard, soft or input, output, process).
9
Step 3. Multivariate analysis
Analysing the underlying structure of the data
is still an art Excerpt Check as not to be
indicator rich but information poor. ..
Indicators are often chosen with little attention
paid to the interrelationships between them.
Identify statistical dimensions in data
set Eliminate highly correlated indicators
Correlation analysis
10
Step 3. Multivariate analysis
Depending on one school of thought, one may see
a high correlation among sub-indicators as
something to correct for, e.g. by making the
weights inversely proportional to the strength of
the overall correlation for a given
sub-indicator, e.g. Index of Relative Intensity
of Regional Problems in the EU (Commission of the
European Communities, 1984).
11
Step 3. Multivariate analysis
Practitioners of multicriteria decision analysis
would instead tend to consider the existence of
correlations as a feature of the problem, not to
be corrected for, as correlated sub-indicators
may indeed reflect non-compensable different
features of the problem. e.g. A cars speed
and beauty are likely correlated with one
another, but this does not imply that we are
willing to trade speed for design.
12
Step 3. Multivariate analysis

• Look at the indicators as an entity, with a view
  to investigate its structure.
• Multivariate statistic suitability of dataset,
  understanding of implications of methodological
  choices (e.g. weighting, aggregation) during the
  construction phase of the composite indicator.
• In the analysis, the statistical information
  inherent in the indicators set can be dealt with
  grouping information along the two dimensions of
  the dataset, i.e. along indicators and along
  constituencies (e.g. countries, regions, sectors,
  etc.), not independently of each other.

13
Step 3. Multivariate analysis

• Factor Analysis can be used to group the
  information on the indicators.
• The aim is to explore whether the different
  dimensions of the phenomenon are well balanced
  -from a statistical viewpoint- in the composite
  indicator. The higher the correlation between the
  indicators, the fewer statistical dimensions will
  be present in the dataset. However, if the
  statistical dimensions do not coincide with the
  theoretical dimensions of the dataset, then a
  revision of the set of the sub-indicators might
  be considered.

14
Step 3. Multivariate analysis

• Cluster Analysis can be applied to group the
  information on constituencies (e.g. countries) in
  terms of their similarity with respect to the
  different sub-indicators.
• This type of analysis can serve multiple
  purposes, and it can be seen as
• a purely statistical method of aggregation of the
  indicators,
• a diagnostic tool for impact assessment of the
  methodological choices made during the
  construction phase of the composite indicator,
• a method of disseminating information on the
  composite indicator, without losing the
  information on the dimensions of the indicators,
• a method for selecting groups of countries to
  impute missing data with a view to decrease the
  variance of the imputed values.

15

• e.g. Environmental Sustainability Index 2005
• Cluster analysis
• better basis for benchmarking environmental
  performance within peer groups
• a good place to start in the search for best
  practices
• avoidance of policies that did not deliver the
  expected results in other countries.

The geographic pattern of the clusters is
striking (no geographical data were used in the
analysis) ? result of the many similarities of
countries in close geographical proximity in
regard to environmental conditions and pressures,
economic and trade linkages, as well as with
respect to social and cultural communalities.
16
Just to give an example Cluster 1 and 3
represent developed countries (24 out of 29 OECD
countries). Comparable per capita incomes and
good environmental governance BUT the average ESI
scores for cluster 1 and 3 are markedly different
(mainly due to the Environmental Stress
component). Clearly, developed countries with
large land area, low population densities (by far
the lowest of all 7 clusters) and a rich natural
resource base enjoy a comparative advantage
because the absorptive capacity of their
environments is bigger than that of smaller
sized, high population density, developed
countries. Although variables underlying the
indicators were corrected for the most prevalent
distortions due to size, the cluster results
indicate that large area size is advantageous for
environmental sustainability.
17

• e.g. Environmental Performance Index 2006
• Cluster analysis
• The EPI is not able to distinguish between the
  countries (masks differences?)
• A complementary table with the standard
  deviations could reveal further information

18
e.g. Technology Achievement Index 2001

• Cluster analysis
• main difference between the leaders and the
  potential leaders is on Receipts Exports.
• the dynamic adopters are lagging behind the
  potential leaders due to their lower performance
  on Internet, Electricity Schooling. They are,
  however, performing better on Exports.
• Two of the sub-indicators, i.e. Patents
  Enrolment, are not useful in distinguishing
  between these 3 groups (cluster means are very
  close).

19
Step 3. Multivariate analysis
After Step 3, the constructor should have
Checked the underlying structure of the data
along various dimensions, i.e., sub-indicators,
countries. Applied the suitable multivariate
methodology, e.g., PCA, FA, cluster analysis.
Identified sub-groups of indicators or groups of
countries that are statistically similar.
20
Step 4. Imputation of missing data
The idea of imputation could be both seductive
and dangerous

• Missing data are present in almost all composite
  indicators
• Dealing with missing data (3 generic approaches)
• case deletion
• removes either country or indicator from the
  analysis (- it ignores possible systematic
  differences between complete and incomplete
  sample and may produce biased estimates if
  removed records are not a random sub-sample of
  the original sample. Furthermore, standard
  errors will, in general be larger in a reduced
  sample given that less information is used).

21
Step 4. Imputation of missing data
The idea of imputation could be both seductive
and dangerous

• The other two approaches see the missing data as
  part of the analysis and therefore try to impute
  values
• single imputation (e.g. Mean/Median/Mode
  substitution, Regression Imputation,
  Expectation-Maximisation Imputation, etc.)
• multiple imputation (e.g. Markov Chain Monte
  Carlo algorithm).
• The advantages of imputation include the
  minimisation of bias and the use of expensive to
  collect data that would otherwise be discarded.

22
Step 5. Normalisation of data
Avoid adding up apples and peers

• Ranking
• Standardization
• Re-scaling
• Distance to reference country
• Categorical scales
• Cyclical indicators
• Balance of opinions

23
Step 6. Weighting (and aggregation)
The relative importance of the indicators is a
source of contention

• Weights based on statistical models
• Principal component and factor analysis
• Data envelopment analysis
• Regression approach
• Unobserved components models
• Weights based on opinions participatory methods
• Budget allocation
• Public opinions
• Analytic hierarchy process
• Conjoint analysis

24
Step 6. (Weighting and) aggregation

• Aggregation rules
• Linear aggregation
• implies full (and constant) compensability
• rewards sub-indicators proportionally to the
  weights
• Geometric mean
• entails partial (non constant) compensability
• rewards more those countries with higher scores
• The absence of synergy or conflict effects among
  the indicators is a necessary condition to admit
  either linear or geometric aggregation
• weights express trade-offs between indicators
• Multi-criteria analysis
• different goals are equally legitimate and
  important (e.g. social, economic dimensions),
  thus a non-compensatory logic

25
Step 7. Robustness and sensitivity
Sensitivity analysis can be used to assess the
robustness of composite indicators
The Economist, 1998 comments Cynics say
that models can be built to conclude anything
provided that suitable assumptions are fed into
them.
26
Step 7. Robustness and sensitivity
Uncertainty analysis results
Sensitivity analysis results
27
Step 8. Links to other variables
Composite indicators can be linked to other
variables and measures
Break away from the GDP dominance

• Strictly, GDP and HDI ranks are not directly
  comparable
• One cannot really say country A ranks 120th on
  GDP per capita, but because of its social
  programmes it is 60th on HDI ranking
• Yet that is done
• Possible solution introducing a complementary
  index consisting only of indicators reflecting
  current flows

28
Step 8. Links to other variables

• Innovation, a key driver of economic welfare
• a modest positive correlation coefficient (r²
  0.77 t 10.47)
• ? best innovation performance comes at the
  highest income level
• More complex if countries are split in two (a
  high-income group above and a low-income group
  below the EU average GDP).
• ? correlation coefficient declines for both
  groups, not statistically significant for the
  high-income group (r² 0.23 t 0.96) and a
  very moderate correlation for the low-income
  group (r² 0.39 t 1.64).

The relationship between income and innovation
performance, and possible policy choices, become
more differentiated at higher GDP levels.
Countries that combine a very high innovation
performance with moderate GDP performance are
particularly concerned by these results (e.g. SE
recently created a Growth Policy Institute to
provide advice for the integration of innovation
and growth policies)
29
Step 9. Back to the details
De-constructing composite indicators can help
extend the analysis
More disaggregated data are useful to bring out
the variations among regions, social groups,
gender
30
Step 10. Presentations and dissemination
A well-designed graph can speak louder than
words

• Composite indicators must be able to communicate
  the picture to decision-makers and users quickly
  and accurately
• Transparency of the indicator make composite
  indicators available via the web, along with the
  data, the weights and the documentation of the
  methodology
• ? allow users to change variables, weights, etc.
  and to replicate sensitivity tests.

The four-quadrant model of the Sustainable
Project Appraisal Routine (SPeAR), see EUR
report 2005.
31
A good example

• European Innovation Scoreboard (DG ENTR)
• One of the most well-organised sites of the EC
  benchmarking exercises
• Executive Summary,
• Country performance (graphs, tables)
• Economic Performance (comparison with GDP)
• Innovation in sectors companies
• Innovation over time
• Conclusions
• Indicators (databases)
• Definitions
• Full reports

32
JRC references on composite indicators

• 2002
• Saisana M. and S. Tarantola (2002)
  State-of-the-Art Report on Current Methodologies
  and Practices for Composite Indicator
  Development, EUR 20408 EN
• Tarantola S., Saisana M., Saltelli A., Schmiedel
  F. and N. Leapman (2002) Statistical techniques
  and participatory approaches for the composition
  of the European Internal Market Index 1992-2001,
  EUR 20547 EN.
• 2003
• Saisana, M., S. Tarantola, and A. Saltelli (2003)
  Exploratory Research Report the Integration of
  Thematic Composite Indicators, EUR 20682 EN
• 2004
• Tarantola S., R. Liska and A. Saltelli (DG JRC -
  Unit G09), M. Donnay (DG ECFIN - Unit E02) (2004)
  Structural Indicators of the Lisbon agenda
  robustness analysis and construction of composite
  indicators, EUR 21287EN.
• Tarantola S., Liska R., Saltelli A., Leapman N.,
  Grant C. (2004) The Internal Market Index 2004,
  EUR 21274EN
• Nardo M., S. Tarantola, A Saltelli, C.
  Andropoulos, R Buescher, G. Karageorgos, A.
  Latvala, F. Noel (2004) The e-business readiness
  composite indicator for 2003 a pilot study, EUR
  21294EN

33
JRC references on composite indicators

• 2005
• Saltelli A., Funtowicz A., Guimarães-Pereira A.
  and Malingreau J-P, Munda G., Giampietro M.,
  (2005) Develping effective Lisbon Strategy
  Narratives, EUR 21644 EN.
• Nardo, M. M. Saisana, A. Saltelli and S.
  Tarantola (JRC), A. Hoffman and E. Giovannini
  (OECD) (2005), Handbook On Constructing Composite
  Indicators Methodology And User Guide, OECD
  Statistics Working Paper JT00188147,
  STD/DOC(2005)3.
• Nardo M., Saisana M., Saltelli A. and Tarantola
  S. (2005) Tools for Composite Indicators
  Building. European Commission, EUR 21682 EN, JRC
  Ispra, Italy, pp. 131.
• Saisana M., Saltelli A., Tarantola S., 2005,
  Uncertainty and Sensitivity analysis techniques
  as tools for the quality assessment of composite
  indicators, J. R. Stat. Soc. A, 168(2), 1-17
• Saisana M., Nardo M., Saltelli A. (2005)
  Uncertainty and Senstivity Analysis for the
  Environmental Sustainability Index (in
  collaboration with the Yale Center for
  Environmental Law and Policy the Center for
  International Earth Science Information Network
  at Columbia University), presented to the January
  2005 conference in Davos, http//www.ciesin.columb
  ia.edu/indicators/ESI/ .

34
JRC references on composite indicators

• 2005
• Sajeva M., Gatelli D., Tarantola S. (JRC) and
  Hollanders (MERIT) (2005) Methodology Report on
  European Innovation Scoreboard 2005, European
  Commission, Enterprise Directorate-General, A
  discussion paper from the Innovation/SMEs
  Programme.
• Saisana M., Saltelli A., Schulze N., Tarantola
  S., Duchene V. (2005) Uncertainty and Sensitivity
  Analysis for the Knowledge-based Economy Index,
  Conference on Medium-Term Economic Assessment
  (CMTEA), Sofia, September 29-30.
• Munda M. and Nardo M. (2005) Constructing
  Consistent Composite Indicators the Issue of
  Weights, manuscript submitted to Economics
  Letters.
• Munda G. and Nardo M. (2005) Non-Compensatory
  Composite Indicators for Ranking Countries A
  Defensible Setting, manuscript submitted to
  Economica.
• Munda G. (2005) Social Multi-Criteria Evaluation
  (SMCE) Methodological Foundations and
  Operational Consequences, forthcoming, J. of
  Operational Research.

35
Reviews on methodologies and practices on
composite indicators State-of-the-art Report
on Current Methodologies and Practices for
Composite Indicator Development (2002) Michaela
Saisana Stefano Tarantola, European Commission,
Joint Research Centre Composite indicators of
country performance a critical assessment (2003)
Michael Freudenberg, OECD. Methodological Issues
Encountered in the Construction of Indices of
Economic and Social Well-being (2003) Andrew
Sharpe Julia Salzman Literature Review of
Frameworks for Macro-indicators (2004), Andrew
Sharpe, Centre for the Study of Living Standards,
Ottawa, CAN. Measuring performance An
examination of composite performance indicators
(2004) Rowena Jacobs, Peter Smith, Maria Goddard,
Centre for Health Economics, University of York,
UK. Methodological Choices Encountered in the
Construction of Composite Indices of Economic
and Social Well-Being, Julia Salzman , (2004)
Center for the Study of Living Standards ,
Ottawa, CAN.