league tables as policy instruments: the political economy of accountability in tertiary education

1
league tables as policy instruments the
political economy of accountability in tertiary
education

• Jamil Salmi and Alenoush Saroyan
• CIEP, 18-20 June 2006

2
Lexus-Nexus index on rankings
1981-85 1986-90 1991-95 1996-00 2001-06

Asia/Pacific Asia/Pacific 0 0 6 18 27

Middle East/Africa Middle East/Africa 0 0 0 1 9
Europe 0 0 2 24 68

N. S. America N. S. America 0 3 17 23 68
3
The rankings business
A ranking of league tables September 10, 2005
4
outline of the presentation

• typology of rankings
• a world of controversies
• do rankings measure quality?
• policy implications

5
typology of rankingshow is it done?

• statistical indicators
• produced by institutions
• publicly available
• survey of stakeholders
• employers
• professors
• students
• combination of both

6
typology of rankings what does it apply to?

• entire institution or specific program
• gives a global score or measures several
  dimensions separately
• research or teaching / learning

7
cluster of indicators in league tables as
measures of quality

• beginning characteristics
• learning inputs- staff
• learning inputs- resources
• learning outputs
• final outcomes
• research
• reputation

8
who prepares the ranking?

• A government agency (Ministry of Higher
  Education, Higher Education Commission,
  University Grants Council, etc.)
• B independent organization / professional
  association / university
• C newspaper / magazine / media
• D accreditation agency
• I International ranking (IA, IB, IC and ID
  linking the international dimension to the type
  of institution conducting the ranking)

9
ranking systems in 2006
Region National and International Ranking System
Eastern Europe and Central Asia Poland (C), Slovakia (B), Russia (B), Ukraine (B)
East Asia and Pacific Australia (B), China (B, IB), Hong Kong (C), Japan (C), New Zealand (A), Thailand (A)
Latin America and the Caribbean Argentina (D)
Middle East and North Africa
North America Canada (C), United States (C)
South Asia India (D), Pakistan (A)
Sub-Saharan Africa Nigeria (A)
Western Europe Germany (B/C), Italy (C), Netherlands (A), Spain (B), United Kingdom (A, B, IC)

10
outline of the presentation

• typology of rankings
• a world of controversies

11
a thin line between love and hate
12
(No Transcript)
13
a thin line between love and hate

• disagreement with principle

14
a thin line between love and hate

• disagreement with principle
• criticism of methodology

15
a thin line between love and hate

• disagreement with principle
• criticism of methodology
• boycotts

16
boycotts
17
boycotts

• Asiaweek
• US News and World Report
• McLeans

18
a thin line between love and hate

• disagreement with principle
• criticism of methodology
• boycotts
• court actions (New Zealand, Holland)

19
(No Transcript)
20
outline of the presentation

• typology of rankings
• a world of controversies
• do rankings measure quality?

21
but do they measure quality?

• quality a moving target Illusive definition
• mutlidimensional construct unidimensional score
  (subjective weights to indicators)
• theoretical justification of measures and
  methodology
• empirical support for indicators

22
other shortcomings

• methodological flaws
• lesser emphasis on outcome indicators
• few meaningful indicators to assess teaching
  quality
• one size fits all general disregard for
  non-research universities and non-university
  institutions
• encourages universities to adjust method of data
  reporting

23
and the winner is
24
the Anglo-Saxon factor
25

• the English factor in the 2005 rankings

26
outline of the presentation

• typology of rankings
• a world of controversies
• do rankings measure quality?
• policy implications

27
usefulness of rankings?

• for the Government?
• for the institutions?
• for the public?

28
government use of rankings

• Pakistan case
• promoting a culture of accurate and transparent
  information
• promoting a culture of quality

29
from the viewpoint of institutions

• sensitive to factors that affect their rankings
  (benchmarking)
• goal setting for strategic planning purposes
• forming strategic partnerships
• mergers

30
applying public pressure

• Provão

31
applying public pressure

• Provão
• France

32
(No Transcript)
33
applying public pressure

• Provão
• France
• Colombia

34
(No Transcript)
35
conclusion divisive or helpful?
36
conclusion divisive or helpful?
37
conclusion divisive or helpful?

• rankings are here to stay
• useful for prospective students
• useful in the absence of an established
  evaluation and/or accreditation system
• useful for benchmarking, goal-setting and
  self-improvement purposes
• useful to conduct a healthy debate on issues and
  challenges
• useful to promote a culture of accountability

38
principles of an appropriate ranking instrument

• compare similar institutions
• better to focus on program than on entire
  institution
• better to rank by indicator than wholesale
  (Germany Pakistan)
• better to focus on outcomes/outputs/results
  rather than inputs (labor market outcomes,
  publications, patents)
• better if used for self-improvement purposes
• better to advertise results publicly than to keep
  them secret

39
(No Transcript)