league tables as policy instruments: the political economy of accountability in tertiary education

1
league tables as policy instruments the
political economy of accountability in tertiary
education

• Jamil Salmi and Alenoush Saroyan
• 2nd IREG Meeting
• Berlin, 18-21 May 2006

2
Lexus-Nexus index on rankings
1981-85 1986-90 1991-95 1996-00 2001-06

Asia/Pacific Asia/Pacific 0 0 6 18 27

Middle East/Africa Middle East/Africa 0 0 0 1 9
Europe 0 0 2 24 68

N. S. America N. S. America 0 3 17 23 68
3
The rankings business
A ranking of league tables September 10, 2005
4
and the winner is

5
outline of the presentation

• typology of rankings
• a world of controversies
• do rankings measure quality?
• policy implications

6
typology of rankingshow is it done?

• statistical indicators
• produced by institutions
• publicly available
• survey of stakeholders
• employers
• professors
• students
• combination of both

7
typology of rankings what does it apply to?

• entire institution or specific program
• gives a global score or measures several
  dimensions separately
• research or teaching / learning

8
who prepares the ranking?

• A Ranking prepared by government agency
  (Ministry of Higher Education, Higher Education
  Commission, University Grants Council, etc.)
• B Ranking prepared by independent organization
  / professional association / university
• C Ranking prepared and published by newspaper /
  magazine
• D Ranking prepared by accreditation agency
• I International ranking (IA, IB, IC and ID
  linking the international dimension to the type
  of institution conducting the ranking)

9
ranking systems in 2006
Region National and International Ranking System
Eastern Europe and Central Asia Poland (C), Slovakia (B), Russia (B), Ukraine (B)
East Asia and Pacific Australia (B), China (B, IB), Hong Kong (C), Japan (C), New Zealand (A), Thailand (A)
Latin America and the Caribbean Argentina (D)
Middle East and North Africa
North America Canada (C), United States (C)
South Asia India (D), Pakistan (A)
Sub-Saharan Africa Nigeria (A)
Western Europe Germany (B/C), Italy (C), Netherlands (A), Spain (B), United Kingdom (A, B, IC)

10
outline of the presentation

• typology of rankings
• a world of controversies

11
a thin line between love and hate
12
(No Transcript)
13
a thin line between love and hate

• disagreement with principle (Anglo-Saxon
  exercise)

14
a thin line between love and hate

• disagreement with principle (Anglo-Saxon
  exercise)
• criticism of methodology

15
a thin line between love and hate

• disagreement with principle (Anglo-Saxon
  exercise)
• criticism of methodology
• boycotts

16
boycotts
17
boycotts

• Asiaweek
• US News and World Report

18
a thin line between love and hate

• disagreement with principle (Anglo-Saxon
  exercise)
• criticism of methodology
• boycotts (Asiaweek, USA)
• court actions (New Zealand, Holland)

19
(No Transcript)
20
outline of the presentation

• typology of rankings
• a world of controversies
• do rankings measure quality?

21
the Anglo-Saxon factor
22
(No Transcript)
23
shortcomings

• methodological flaws
• design
• choice of indicators (very few meaningful
  measures of quality of teaching and learning
• weight among indicators
• reliability of data
• no true measure of quality
• wrongly used as one size fits all
• encourage universities to adjust to the ranking
  criteria

24
outline of the presentation

• typology of rankings
• a world of controversies
• do rankings measure quality?
• policy implications

25
usefulness of rankings?

• for the Government?
• for the institutions?
• for the public?

26
government use of rankings

• Pakistan case

27
government use of rankings

• Pakistan case
• promoting a culture of accurate and transparent
  information

28
government use of rankings

• Pakistan case
• promoting a culture of accurate and transparent
  information
• promoting a culture of quality

29
from the viewpoint of institutions

• sensitive to factors that affect their rankings
  (benchmarking)
• goal setting for strategic planning purposes
• forming strategic alliances

30
applying public pressure

• Provão

31
applying public pressure

• Provão
• France

32
(No Transcript)
33
applying public pressure

• Provão
• France
• Colombia

34
(No Transcript)
35
conclusion divisive or helpful?
36
conclusion divisive or helpful?
37
conclusion divisive or helpful?

• rankings are here to stay
• useful for prospective students
• useful in the absence of an established
  evaluation and/or accreditation system
• useful for benchmarking, goal-setting and
  self-improvement purposes
• useful to conduct a healthy debate on issues and
  challenges
• useful to promote a culture of accountability

38
principles of an appropriate ranking instrument

• compare similar institutions
• better to focus on program than on entire
  institution
• better to rank by indicator than wholesale
  (Germany Pakistan)
• better to focus on results rather than inputs
  (labor market outcomes, publications, patents)
• better if used for self-improvement purposes
• better to advertise results publicly than to keep
  them secret

39
(No Transcript)