Ottawa, 79 November 2005 http:farmweb.jrc.cec.eu.intci 1

1
Weighting issues in the development of a
Composite Indicator Michaela
Saisana michaela.saisana_at_jrc.it European
Commission Joint Research Centre Ispra,
Italy Composite Indicators Workshop Ottawa,
7-9 November 2005
2
Prepared with Michela Nardo
Based on Handbook on Constructing Composite
IndicatorsMethodology and User GuideNardo, M.
M. Saisana, A. Saltelli and S. Tarantola
(EC/JRC), A. Hoffman and E. Giovannini (OECD),
OECD Statistics Working Paper JT00188147,
STD/DOC(2005)3.http//www.olis.oecd.org/olis/200
5doc.nsf/LinkTo/std-doc(2005)3
3
Step 6. Weighting (and aggregation)
No agreed methodology exists to weight individual
indicators
4
Step 6. Weighting (and aggregation)

• Equal weights
• Weights based on statistical models
• Principal component/Factor analysis
• Data envelopment analysis
• Regression approach
• Unobserved components models
• Weights based on opinions participatory methods
• Budget allocation
• Public opinion
• Analytic hierarchy process
• Conjoint analysis

5
Equal weights

• Considerations
• Proper normalisation of indicators is needed
• Works well if all dimensions (e.g. economic,
  social, environmental, etc.) have the same number
  of indicators (e.g. TAI example)
• Otherwise, equal weighting implies a higher
  weight to the dimension represented by the larger
  number of indicators.
• Appealing when high correlation of indicators
  does not mean redundancy of information in the
  composite, i.e. when correlated components
  explain different aspects of the picture the
  composite aims to capture.

6
Weights based on Principal Component Analysis

• Tool to examine thoroughly the relationships
  among indicators and statistical dimension of the
  dataset.
• Two crucial problems
• 1. weights assigned to sub-indicators are based
  on correlations which do not necessarily
  correspond to the underlying relationships
  between the indicators and the phenomena being
  measured.
• confusion between correlation-redundancy
  redundancy implies correlation but the reverse is
  not necessarily true
• 2. disciplines homogeneity rather than
  representing plurality (PCA only applied when
  variables are correlated).

7
Weights based on Principal Component Analysis
E.g. Environmental Sustainability Index 2005
Conclusions 1. The ESI is a multi-D index and
environmental sustainability is a multi-D
concept. 2. The assumption that if the ESI were
based on a small number of indicators such as the
UNDP - HDI, it would not fully describe all
dimensions of environmental sustainability.
Eigenvalues and scree plot to specify number of
PCs for the 21 indicators
8
Weights based on Principal Component Analysis
Environmental Sustainability Index 2005
The higher the loading of an indicator, the more
useful it is for explaining variation in the
direction of the PC.
Conclusions 1. Most indicators load highly on
the first three PCs 2. No indicator has low
loadings on all six PCs, thus none of them is
redundant. 3. Very good interpretation of PCs ?
distinct different aspects of environmental
sustainability captured by each component
9
Weights based on Principal Component Analysis
Environmental Sustainability Index 2005
Loadings ? inherent weights
Conclusion Weights based on PCA for the 21
indicators are 1/21. This supports the choice
of equal weights on the indicator level for
calculating the ESI.
10
Weights based on Data Envelopment Analysis
Employs linear programming tools to retrieve an
efficiency frontier Two main issues 1.
construction of a benchmark (frontier) 2.
measurement of the distance between countries in
a multi-d framework.
Rearranged from Mahlberg and Obersteiner (2001)

• countries on the frontier (a, b, c) are
  classified as the best performing, country d is
  the worse performing.
• performance indicator d/d
• set of weights of a country depends on its
  position with respect to the frontier.
• benchmark INTERNAL or EXTERNAL (e.g. Korhonen
  et al. 2001, performance indicator of academic
  research target in the efficiency frontier
  having the most preferred combination of
  sub-indicators.

11
Table 6.3. BOD approach applied to the TAI
dataset (23 countries). Columns 1 to 8 contain
weights, column 9 displays the countrys
composite indicator.
Weights based on Data Envelopment Analysis
E.g. Technology Achievement Index 2001
The optimal set of weighs (if it exists)
guarantees the best position for the country vis
á vis all other countries in the sample- any
other weights profile would worsen the relative
position of that country
Notice that 1. weights are country specific 2.
benchmark is country-dependent (no unique
benchmark, unless a country is better-off in all
sub-indicators) 3. indicators must be comparable
(same unit of measurement)
12
Weights based on Regression

• Multiple regression models can handle a large
  number of indicators.
• input indicators (e.g. related to various policy
  actions) ? output indicator (e.g. target)
• The regression model could
• a. quantify the relative effect of each policy
  action on the output
• b. be used for forecasting purposes
• In a more general case of multiple output
  indicators, canonical correlation analysis
  (generalization of multiple regression) could be
  applied.
• However, in any case, there is always the
  uncertainty that the relations, captured by the
  regression model for a given range of inputs and
  output, may not be valid for different ranges.

13
Weights based on Unobserved Components

• Unobserved components is similar in spirit to
  the multiple regression models, it does not need
  an explicit value for the dependent variable as
  it treats it like another unknown variable to
  estimate.
• This advantage is counter-balanced by the
  inconvenient of the complexity in estimation and
  the computational cost

14
Weights based on Budget Allocation
Expert 1
Royalties
Internet
Technology exports
Telephones
Electricity
Patents
Schooling years
University Students
allocate 100 points
5 5 5 20 10 5 20 30
Total
5 5 5 20 10 5 20 30
15
Weights based on Budget Allocation
Expert 2
Royalties
Internet
Technology exports
Telephones
Electricity
Patents
Schooling years
University Students
allocate 100 points
15 15 10 15 10 5 20 10
Total
5 5 5 20 10 5 20 30
20 20 15 35 20 10 40 40
Based on an internal JRC survey of 21 experts...
16
Weights based on Budget Allocation

• Phases
• 1. Selection of experts for the valuation
• 2. Allocation of budget to the sub-indicators
• 3. Calculation of the weights
• 4. Iteration of the budget allocation until
  convergence is reached (optional).
• Essential to bring together experts that have a
  wide spectrum of knowledge, experience and
  concerns ?proper weighting system
• Example A case study in which 400 German experts
  in 1991 were asked to allocate a budget to
  several environmental indicators related to an
  air pollution problem showed very consistent
  results, in spite of the fact that the experts
  came from opposing social spheres like the
  industrial sector and the environmental sector
  (Jesinghaus in Moldan and Billharz, 1997).
• Advantages
• Participatory technique
• Limitations
• Design of the survey - choice of experts
  (number, background)
• Circular thinking
• Up to 8-10 indicators
• Not transferable from one area to another

17
Weights based on Analytic Hierarchy Process
USING PAIRWISE COMPARISONS, THE RELATIVE
IMPORTANCE OF ONE CRITERION OVER ANOTHER CAN BE
EXPRESSED
1 EQUAL 3 MODERATE 5 STRONG 7 VERY STRONG
9 EXTREME
18
Weights based on Analytic Hierarchy Process
USING PAIRWISE COMPARISONS, THE RELATIVE
IMPORTANCE OF ONE CRITERION OVER ANOTHER CAN BE
EXPRESSED
1 EQUAL 3 MODERATE 5 STRONG 7 VERY STRONG
9 EXTREME
19
Weights based on Analytic Hierarchy Process
USING PAIRWISE COMPARISONS, THE RELATIVE
IMPORTANCE OF ONE CRITERION OVER ANOTHER CAN BE
EXPRESSED
1 EQUAL 3 MODERATE 5 STRONG 7 VERY STRONG
9 EXTREME
Weights
solve for the Eigenvector
Inconsistency 17.4
20
Weights based on Analytic Hierarchy Process
Based on an internal JRC survey of 18 experts...
Inconsistencies range from 1.1 - 45.5
(desired lt 10-20 )
21
Weights based on Analytic Hierarchy Process

• Advantages
• Participatory technique
• Tolerates inconsistency (part of human thinking)
• Pairwise comparisons can be easily handled by
  human mind
• Limitations
• More time consuming than budget allocation
• (N.(N-1)/2 pairs for comparison)
• Design of the survey - choice of experts
  (number, background)
• Recommended for less than 10 indicators
• Not transferable from one area to another

22
Weights based on BAL AHP

Budget allocation provides weights that are
closer to the average than Analytic Hierarchy
Process
23
Weights based on Conjoint Analysis

• Conjoint analysis derives the worth of the
  indicators from the worth of a composite, i.e. it
  reverses the process of AHP, with which it shares
  advantages and disadvantages.
• Further complication is the need to specify and
  estimate and utility function

24
Table 6.6. Weights for the sub-indicators
obtained using 4 different methods equal
weighting (EW), factor analysis (FA), budget
allocation (BAL), and analytic hierarchy process
(AHP)
Weights based on different weighting methods
E.g. Technology Achievement Index 2001
weights
ranks
() For example USA ranks first according to BOD,
second according to EW, FA, and BAL and third
according to AHP.