Examining Major Rankings According to Berlin Principles

1
Examining Major Rankings According to Berlin
Principles

• Ying Cheng a, b and Nian Cai Liu a
• a Institute of Higher Education, Shanghai Jiao
  Tong University, China
• b Observatoire des Sciences et des Techniques,
  France
• October 29, 2007

2
Background

• Berlin Principles on Ranking of Higher Education
  Institutions
• 16 principles of quality and good practice in
  rankings of HEIs
• Regarding four aspects purpose, indicators, data
  and presentation of results
• It is expected these principles will be useful
  for the improvement and evaluation of rankings.

3
Purpose of this Study

• To analyze selected rankings according to Berlin
  Principles
• To provide recommendations on further efforts on
  the basis of Berlin Principles

4
From Berlin Principles to Measurable
Criteria

• Why difficult to use principles to analyze and
  compare rankings directly
• How based on authors understanding of Berlin
  Principles
• Each measurable criterion is a good practice of
  rankings which corresponds to one or more items
  of Berlin Principles

5
Proposed Criteria Sample 1

• 1st Item of Berlin Principles Ranking should be
  one of a number of diverse approaches and not
  the main approach.
• Criterion 1 A desirable ranking should claim
  that it is just one source of reference which
  might help users to make their judgment.

6
Proposed Criteria Sample 2

• 3rd Item of Berlin Principles Ranking should
  recognize the diversity of institutions and take
  the different missions of institutions into
  account.
• Criterion 3 A desirable ranking should use
  distinguishing indicator systems to rank HEIs
  with different missions and provides separate
  ranking lists.

7
Proposed Criteria Sample 3

• Criteria 6 - 8 A desirable ranking should
  provide
• 6 definitions and statistical methods of each
  indicator.
• 7 weights of each indicator.
• 8 sources of each indicator and links them
  clearly.
• Berlin Principles
• Ensure transparency (6th, 9th Item).
• Help users to understand results(15th Item),
• Help users to judge Data collection Process (12th
  Item)
• Help users to verify the data and inform
  errors(16th Item)

8
Fourteen Criteria of Desirable Rankings
9
Items of BP not Represented

• 7th Item Choose indicators according to their
  relevance and validity.
• People might hold different or even opposite view
  on the relevance of the same indicators.
• 8th Item Measure outcomes in preference to
  inputs whenever possible.
• How much is sufficient?
• 11th Item Use audited and verifiable data
  whenever possible.

10
Analysis Rankings by Proposed Criteria

• Consideration on selection of samples
• Institutional rankings Rankings of university
  units or disciplines are excluded
• Ranking published in EnglishRankings publishe in
  other language are excluded

11
Ranking Samples
12
Analysis of Rankings by Criteria
13
Problems worth noticing

• A few methodological criteria (principles) can
  not be well implemented by ranking practice
• For international rankings Specify the
  educational (National) systems being ranked. (
  5th Item of BP) No good practice
• For all rankings Take the different missions of
  HEIs into account. (3rd Item of BP) What is the
  best way?
• Objective criteria for precaution but subjective
  judgment of ranking practice
• How to identify that the information of
  precaution was really provided and can be noticed
  by users ?

14
Rethink what is a good ranking

• If Good Ranking Eligible Operation
  Appropriate Methodology
• Eligible Operation
• In order to be understood thoroughly and
  correctly
• Can be judged by its transparency and
  precaution
• Appropriate Methodology
• Right idea, relevant indicators, reliable source,
  ..
• Sometimes controversial, hard to fairly judge

15
Findings and Discussions

• Criteria for transparency and their application
• Criteria for precaution and their application
• Criteria for methodology and their application
• Items of Berlin Principles can not be transformed

16
Transparency

• Consist with BP very well
• Successful application for analysis of rankings
• Essential to any scientific study
• Easy to operate and follow

17
Precaution

• Consist with BP very well
• Need subjective judgments to rankings
• Essential to any scientific study
• Not difficult to operate and follow

18
Methodology

• Consist with BP constrainedly

• Can not be well implemented by ranking practice

19
Items of BP can not be transformed

• Also methodology related
• Not available for evaluation purpose

20
Back to the Berlin Principles

• Purpose Improvement of rankings
• Merit a milestone which symbolized the
  initiation of collaborative efforts by rankers,
  users and scholars
• Acknowledging rankings rationality
• Try to consider problems constructively

21
What we should do to on the basis of Berlin
Principles to improve rankings?

• Keep the merit of Berlin Principles
• Develop BP towards two directions
• Set requirements for the transparency and
  precaution
• Give more operable suggestions and
  recommendations on the methodology

22
Set requirements

• Transparency to determine absolutely necessary
  information
• Precaution to create a good model
• E.g U. S. News
• Do use the rankings as one tool to select and
  compare schools.
• Don't rely solely on rankings to choose a college.

23
Give more operable suggestions and
recommendations on the methodology

• It is much more helpful to rankers if we can
  continually work together to
• Summarize examples of good ranking practices
• Make operable suggestions to difficult problems
  with wide interests (eg. Diversity of HEIs)
• Develop an evaluation system of ranking
  indicators according to their relevance and
  validity to different purpose, as well as the
  proper method of gathering data

24

• Thank you for your attention!