Introduction: the policy context - PowerPoint PPT Presentation

1 / 19
About This Presentation
Title:

Introduction: the policy context

Description:

The quality and sustainability of the research environment remains important ... users and the public, dissemination, collaboration, income, KT activity, etc. ... – PowerPoint PPT presentation

Number of Views:46
Avg rating:3.0/5.0
Slides: 20
Provided by: HEF9
Category:

less

Transcript and Presenter's Notes

Title: Introduction: the policy context


1
(No Transcript)
2
The Research Excellence FrameworkMURG, 19
March 2009Graeme Rosenberg, REF Manager
3
REF overview
  • REF is a unified framework for funding and
    assessing research
  • Its primary focus is still excellence, but will
    take more explicit account of the impact of
    research on the economy and society

4
REF overview (2)
  • REF is a work in progress following previous
    consultations we have already moved some way from
    where we started
  • REF is not only about basic research, nor is it
    only about bibliometrics
  • We are developing a range of assessment
    approaches which can be combined in different
    ways to fit particular subject fields and types
    of research

5
REF overview (3)
  • We see REF as a further development of what we do
    now, not something completely new
  • We aim to build upon the lessons of RAE
    including examples of good practice in RAE that
    we wish to carry forward into REF
  • We are committed to reducing the burden on HEIs,
    but we need to discuss how to achieve this
    without losing responsiveness to sector concerns

6
Framework for assessment
Our starting points are
  • National policy imperatives
  • Responses to last years consultation unanimous
    that there may be a problem but much less so on
    the solution
  • Lessons to be learnt from the RAE experience
    especially from the enhanced provision made by
    panels in 2008

7
Tools for assessment
The toolkit
  • Bibliometric analysis
  • Expert review of outputs
  • Other available indicators
  • Submission of information by HEIs
  • All of these would be collected and interpreted
    with advice and input from expert panels as
    required

8
Main elements of assessment
  • The quality of research outputs remains the
    dominant element of assessment
  • We also aim to take more explicit account of the
    impact of research on the economy, society and
    quality of life
  • The quality and sustainability of the research
    environment remains important
  • Broad consensus that esteem is not needed as a
    distinct element, though some aspects can be
    retained

9
The REF framework
Outputs
Impact
Environment
Quality and sustainability of the environment
assessed through narrative and indicators
Quality of outputs assessed through a
combination of bibliometrics and expert review
Engagement with users and the public assessed
through narrative and indicators
Impact of research assessed through a portfolio
of evidence
10
Outputs bibliometrics and expert review
  • Through the pilot we are testing models for
    producing bibliometric indictors and combing them
    with expert review on in the REF
  • Some related questions
  • Is there a real alternative to HEIs selecting
    staff?
  • Should the HEI or the author get the credit?
  • Panel configuration what scope is there for
    fewer, broader panels?

11
Model 1 Top down approach
  • Citation indicators are produced by associating
    outputs in WoS/SCOPUS to HEIs through
    address/affiliation data, and to fields based
    on journal categories
  • Implications for the REF
  • HEIs not authors get the credit
  • Outputs not linked to staff. What is the volume
    measure?
  • How would expert panels use/interpret the
    indicators and combine with expert review?
  • Implications for HEIs
  • Potentially very little effort but HEIs may want
    to ensure data is robust

12
Model 2 Bottom up approach
  • For a defined group of staff, citation indicators
    are produced based on (all their indexed?)
    outputs
  • Implications for the REF
  • Better fit with other aspects of REF which are
    staff-based
  • How would expert panels use/interpret the
    indicators and combine with expert review?
  • Implications for institutions
  • Selecting staff
  • Collecting publications data, which must be
    linked to staff and to WoS/SCOPUS

13
Model 3 Indicative profiling
  • Citation rates can provide an indicative star
    rating for submitted (journal) outputs expert
    panels review samples and supplementary (non
    journal) outputs
  • Implications for the REF
  • Fits with other elements of the REF across all
    subjects
  • How useful are citation rates for individual
    outputs? How much sampling would be needed?
  • Implications for institutions
  • Submissions similar to RAE

14
Impact and engagement (1)
  • In the RAE, user significance and impact informed
    panels judgements in a number of ways
  • HEIs have developed portfolios of work that bring
    a range of benefits to their regions, the UK and
    internationally
  • These include economic, social, cultural, quality
    of life and public policy benefits
  • In REF we intend that HEIs should more explicitly
    demonstrate their successes in this, in a way
    that complements the achievements in moving the
    discipline forward

15
Impact and engagement (2)
  • Metrics can be informative but cannot capture the
    full range of impacts
  • Qualitative approaches will also be needed
  • There are challenges relating to time-lags,
    attribution and corroboration
  • How best to involve users in the assessment
    process

16
Impact possible approaches
Impact statements for (some) outputs submitted
for output quality
Additional (applied or user-focused) outputs to
be assessed for impact
Narrative and indicators of engagement with users
and the public, dissemination, collaboration,
income, KT activity, etc.
Case studies of longer term impact from prior
research
17
Criteria for assessment
  • Outputs could be assessed in terms of (academic)
    excellence significance, originality and rigour
    against international standards
  • Impact could be assessed against criteria for the
    breadth and depth of social, economic or quality
    of life benefits
  • while meeting a minimum threshold for rigour and
    originality
  • assessment by a combination of user and academic
    experts

18
Environment
  • Particular interest in sustainability and forward
    looking elements
  • REF will build on the approach taken in the RAE,
    with a number of possible refinements
  • More structured narrative (RA5)
  • Can indicators be more standardised across
    panels?
  • Can HESA data (PGRs, research income) be used?
  • Should the criteria and ratings scale be more
    tailored?

19

Timetable
Write a Comment
User Comments (0)
About PowerShow.com