Research Excellence Framework Consultation

1
Research Excellence FrameworkConsultation

• Peter Jones
• January 2008

2
Headlines

• STEM subjects six broad subject divisions
• Assessment and funding for STEM driven by
  quantitative indicators (bibliometric data)
• Non STEM lighter touch peer review informed by
  metrics in 2013
• Also make use of other data research income
  students
• Expert panels will advice on development of new
  systems
• A single quality profile (like RAE 2008) will be
  produced for each subject area
• Aims to reduce admin burden

3
6 Subject groups

• Clinical Medicine
• Health Sciences
• Subjects Allied to Health
• Biological Sciences
• Physical Sciences
• Engineering and Computer Science

4
STEM subjects

• 2 options for producing quality profiles
• Bibliometric analysis alone (funding allocations
  also driven by other indicators including
  research income and student data)
• Developing a compound quality indicator based on
  biblometrics and other indicators

5
STEM subjects

• HEIs must state in early 2009, which staff are
  considered research active
• Bibliometrics piloted in 2008
• Assumed that highly cited research greatest
  intellectual influence
• Citations over 5 or 10 year period
• Thomson Web of Science primary data source
• HEIs will verify data about staff outputs this
  will require HEIs maintaining info on staff
  publications
• Metrics will concentrate on citation rates per
  paper, aggregated for each subject group

6
Humanities Social Sciences (and Maths)

• Anticipate fewer and larger groups than RAE 2008
  Units of Assessment
• Looking to develop lighter touch less
  burdensome approaches
• Aim to give higher weighting to quantitative
  indicators within peer review process
• Make greater use of HESA data

7
Timetable
8
HEPI analysis - key points (1)

• Metrics and citations are a given element in new
  arrangements
• Improvement on previous proposals based entirely
  on research income?
• Citation analysis measures impact rather than
  quality
• Leiden report Optimal research evaluation is
  through combination of metrics and peer review
  (therefore there may need to be an extended peer
  review role than proposed)
• Evidence report correlation between metrics and
  RAE grades at broad level, but variance between
  units within any grade is high. Hence HEFCEs
  proposed broad subject groups.

9
HEPI analysis - key points (2)

• Citation proposals take out peer review process
  and place assessment of quality with
  technicians
• Citations based on older retrospective data than
  current RAE
• Recent research is bound to have fewer citations
  than research of the same quality produced years
  before.
• Metrics alone without peer judgement will have
  detrimental impact on equality and diversity
  issues, e.g. early career
• Larger subject groupings will provide less
  detailed MI for HEIs
• HEPI suggest that RC income should not appear in
  metrics as game playing may concentrate on
  aspects which can be controlled e.g. RC grants
  not citations, and there is already a 70 failure
  rate.

10
Other points to consider (1)

• Does not take into account contribution/position
  of author might encourage multiple authors!
• Encourages researchers to write more reviews,
  rather than primary papers, since these are most
  often cited
• Methods papers" will often be cited - though
  important they pale into insignificance relative
  to some less frequently cited papers
• If researchers decide to submit to a "lower
  ranked journal in most cases it will be cited
  less
• Discourages younger researchers from establishing
  their academic credentials through lesser
  journals
• It will no longer be worth publishing books or
  chapters and science will be the poorer for it
• Encourages fewer publications of higher impact,
  reducing the apparent productivity of UK
  scientist - other countries place a high value
  not only on the number of citations but on the
  number of papers published as well.

11
Other points to consider (2)

• Supervisors may be reluctant to publish early
  career work as it will have detrimental impact on
  citations average
• Any research with a lot of adventure and a longer
  term aim might well be discouraged
• There appears to be no mechanism for considering
  esteem indicators
• Referees may want their papers cited when the
  justification might well be questionable
• Highly cited papers are often so, due to errors
  in the research
• Alternative option would be to give all QR
  funding to Research Councils (although unlikely
  due to Dual Support system).
• What does the data verification entail? Does
  Keele have the MI systems in place to support
  this?
• How will HEIs decide who is being submitted
• No consideration or allowances are suggested for
  early career researchers or those with personal
  circumstances

12
Consultation questions

• 1a Do you endorse our proposals for defining the
  broad group of science-based disciplines, and for
  dividing this into six main subject groups, in
  the context of our new approach to assessment and
  funding?
• 1b Are there issues in relation to specific
  disciplines within this framework that we should
  consider?
• 2a Do you agree that bibliometric indicators
  produced on the basis that we propose can provide
  a robust quality indicator in the context of our
  framework?
• 2b Are there particular issues of significance
  needing to be resolved that we have not
  highlighted?

13
Consultation questions

• 3a What are the key issues that we should
  consider in developing light touch peer review
  for the non science-based disciplines?
• 3b What are the main options for the form and
  conduct of this review?
• 4 Is there additional quantitative information
  that we should use in the assessment and funding
  framework to capture user value or the quality of
  applied research, or other key aspects of
  research excellence? Please be specific in terms
  of what the information is, what essential
  element of research it casts light on, how it may
  be found or collected, and where and how it might
  be used within the framework.

14
Consultation questions

• 5 Are our proposals for the role of expert
  panels workable within the framework? Are there
  other key issues on which we might take their
  advice?
• 6 Are there significant implications for the
  burden on the sector of implementing our new
  framework that we have not identified? What more
  can we do to minimise the burden as we introduce
  the new arrangements?
• 7 Do you consider that the proposals in this
  document are likely to have any negative impact
  on equal opportunities? What issues will we need
  to pay particular attention to?
• 8 Do you have any other comments about our
  proposals, which are not covered by the above
  questions?