Introduction: the policy context

1
(No Transcript)
2
The Research Excellence FrameworkMURG, 19
March 2009Graeme Rosenberg, REF Manager
3
REF overview

• REF is a unified framework for funding and
  assessing research
• Its primary focus is still excellence, but will
  take more explicit account of the impact of
  research on the economy and society

4
REF overview (2)

• REF is a work in progress following previous
  consultations we have already moved some way from
  where we started
• REF is not only about basic research, nor is it
  only about bibliometrics
• We are developing a range of assessment
  approaches which can be combined in different
  ways to fit particular subject fields and types
  of research

5
REF overview (3)

• We see REF as a further development of what we do
  now, not something completely new
• We aim to build upon the lessons of RAE
  including examples of good practice in RAE that
  we wish to carry forward into REF
• We are committed to reducing the burden on HEIs,
  but we need to discuss how to achieve this
  without losing responsiveness to sector concerns

6
Framework for assessment
Our starting points are

• National policy imperatives
• Responses to last years consultation unanimous
  that there may be a problem but much less so on
  the solution
• Lessons to be learnt from the RAE experience
  especially from the enhanced provision made by
  panels in 2008

7
Tools for assessment
The toolkit

• Bibliometric analysis
• Expert review of outputs
• Other available indicators
• Submission of information by HEIs
• All of these would be collected and interpreted
  with advice and input from expert panels as
  required

8
Main elements of assessment

• The quality of research outputs remains the
  dominant element of assessment
• We also aim to take more explicit account of the
  impact of research on the economy, society and
  quality of life
• The quality and sustainability of the research
  environment remains important
• Broad consensus that esteem is not needed as a
  distinct element, though some aspects can be
  retained

9
The REF framework
Outputs
Impact
Environment
Quality and sustainability of the environment
assessed through narrative and indicators
Quality of outputs assessed through a
combination of bibliometrics and expert review
Engagement with users and the public assessed
through narrative and indicators
Impact of research assessed through a portfolio
of evidence
10
Outputs bibliometrics and expert review

• Through the pilot we are testing models for
  producing bibliometric indictors and combing them
  with expert review on in the REF
• Some related questions
• Is there a real alternative to HEIs selecting
  staff?
• Should the HEI or the author get the credit?
• Panel configuration what scope is there for
  fewer, broader panels?

11
Model 1 Top down approach

• Citation indicators are produced by associating
  outputs in WoS/SCOPUS to HEIs through
  address/affiliation data, and to fields based
  on journal categories
• Implications for the REF
• HEIs not authors get the credit
• Outputs not linked to staff. What is the volume
  measure?
• How would expert panels use/interpret the
  indicators and combine with expert review?
• Implications for HEIs
• Potentially very little effort but HEIs may want
  to ensure data is robust

12
Model 2 Bottom up approach

• For a defined group of staff, citation indicators
  are produced based on (all their indexed?)
  outputs
• Implications for the REF
• Better fit with other aspects of REF which are
  staff-based
• How would expert panels use/interpret the
  indicators and combine with expert review?
• Implications for institutions
• Selecting staff
• Collecting publications data, which must be
  linked to staff and to WoS/SCOPUS

13
Model 3 Indicative profiling

• Citation rates can provide an indicative star
  rating for submitted (journal) outputs expert
  panels review samples and supplementary (non
  journal) outputs
• Implications for the REF
• Fits with other elements of the REF across all
  subjects
• How useful are citation rates for individual
  outputs? How much sampling would be needed?
• Implications for institutions
• Submissions similar to RAE

14
Impact and engagement (1)

• In the RAE, user significance and impact informed
  panels judgements in a number of ways
• HEIs have developed portfolios of work that bring
  a range of benefits to their regions, the UK and
  internationally
• These include economic, social, cultural, quality
  of life and public policy benefits
• In REF we intend that HEIs should more explicitly
  demonstrate their successes in this, in a way
  that complements the achievements in moving the
  discipline forward

15
Impact and engagement (2)

• Metrics can be informative but cannot capture the
  full range of impacts
• Qualitative approaches will also be needed
• There are challenges relating to time-lags,
  attribution and corroboration
• How best to involve users in the assessment
  process

16
Impact possible approaches
Impact statements for (some) outputs submitted
for output quality
Additional (applied or user-focused) outputs to
be assessed for impact
Narrative and indicators of engagement with users
and the public, dissemination, collaboration,
income, KT activity, etc.
Case studies of longer term impact from prior
research
17
Criteria for assessment

• Outputs could be assessed in terms of (academic)
  excellence significance, originality and rigour
  against international standards
• Impact could be assessed against criteria for the
  breadth and depth of social, economic or quality
  of life benefits
• while meeting a minimum threshold for rigour and
  originality
• assessment by a combination of user and academic
  experts

18
Environment

• Particular interest in sustainability and forward
  looking elements
• REF will build on the approach taken in the RAE,
  with a number of possible refinements
• More structured narrative (RA5)
• Can indicators be more standardised across
  panels?
• Can HESA data (PGRs, research income) be used?
• Should the criteria and ratings scale be more
  tailored?

19

Timetable