August 2004

1
RATING WORKSHOP

• Presentation
• August 2004

2
This presentation

• What is an NRF rating?
• Why should I get rated?
• What are the rating categories?
• When should I submit for rating?
• How do I apply for rating?
• What happens to my application?
• Is there any thing else I must know?

3
What is an NRF rating?

• Outcome of an assessment of the recent track
  record,
• through an international peer review process,
  using the
• quality of research outputs of the last seven
  years as the basis

4
This presentation

• What is an NRF rating?
• Why should I get rated?
• What are the rating categories?
• When should I submit for rating?
• How do I apply for rating?
• What happens to my application?
• Is there any thing else I must know?

5
Why should I get rated?

• Benchmarking
• Against international standards
• Personal dimension
• Institutional dimension
• Programme dimension
• National dimension
• Access to NRF funding for five years (NB
  conditional, depending on the approval of project
  proposal)

6
Example of national benchmarking

• Institutions
• All technikons
• All universities
• Natural history museums

• 1998 2003
• 20 46
• 910 1260
• 37 18

7
Example of national benchmarking (cont)

• Number of persons in academic/research positions
  in HEI
• Number of rated researchers
• Rated researchers

• -14000
• 1306
• lt10

8
This presentation

• What is an NRF rating?
• Why should I get rated?
• What are the rating categories?
• When should I submit for rating?
• How do I apply for rating?
• What happens to my application?
• Is there any thing else I must know?

9
Rating categories
Young high-flier with exceptional potential
P
Y
Promising young researcher
L

• Late entrant with potential

A
Leading international scholar
Researcher with considerable international
recognition
B
Established researcher
C
10
Rating categories - definitions

• P
• Young researchers (normally younger than 35
  years of age), who have held the doctorate or
  equivalent qualification for less than five years
  at the time of application and who, on the basis
  of exceptional potential demonstrated in their
  published doctoral work and/or their research
  outputs in their early post-doctoral careers are
  considered likely to become future leaders in
  their field.

11
Rating categories - definitions

• Y
• Young researchers (normally younger than 35
  years of age), who have held the doctorate or
  equivalent qualification for less than five years
  at the time of application, and who are
  recognised as having the potential to establish
  themselves as researchers within a five-year
  period after evaluation, based on their
  performance and productivity as researchers
  during their doctoral studies and/or early
  post-doctoral careers.

12
Rating categories - definitions

• L
• Persons (normally younger than 55 years) who
  were previously established as researchers or who
  previously demonstrated potential through their
  own research products, and who are considered
  capable of fully establishing or re-establishing
  themselves as researchers within a five-year
  period after evaluation. Candidates should be
  South African citizens or foreign nationals who
  have been resident in South Africa for five years
  during

13
Rating categories - definitions

• L
• which time they have been unable for practical
  reasons to realise their potential as
  researchers. 
• Candidates who are eligible in this category
  include black researchers, female researchers,
  those employed in a higher education institution
  that lacked a research environment as well as
  those who were previously established as
  researchers and have returned to a research
  environment.

14
Rating categories - definitions

• A
• Researchers who are unequivocally recognised by
  their peers as leading international scholars in
  their field for the high quality and impact of
  their recent research outputs.
• B
• Researchers who enjoy considerable international
  recognition by their peers for the high quality
  and impact of their recent research outputs.

15
Rating categories - definitions

• C
• Established researchers with a sustained recent
  record of productivity in the field who are
  recognised by their peers as having
• produced a body of quality work, the core of
  which has coherence and attests to ongoing
  engagement with the field
• demonstrated the ability to conceptualise
  problems and apply research methods to
  investigating them.

16
Rating sub-categories

• A
• B
• C
• P
• Y
• L

• A1, A2
• B1, B2, B3
• C1, C2, C3
• Y1, Y2

17
Definition of research

• For purposes of the NRF, research is original
  investigation undertaken to gain knowledge and/or
  enhance understanding.
• Research specifically includes
• the creation and development of the intellectual
  infrastructure of subjects and disciplines (e.g.
  through dictionaries, scholarly editions,
  catalogues and contributions to major research
  databases)
• the invention or generation of ideas, images,
  performances and artefacts where these manifestly
  embody new or substantially developed insights
• building on existing knowledge to produce new or
  substantially improved materials, devices,
  products, policies or processes.

18
Definition of research

• It specifically excludes
• routine testing and analysis of materials,
  components, instruments and processes, as
  distinct from the development of new analytical
  techniques
• the development of teaching materials and
  teaching practices that do not embody substantial
  original enquiry.

19
This presentation

• What is an NRF rating?
• Why should I get rated?
• What are the rating categories?
• When should I submit for rating?
• How do I apply for rating?
• What happens to my application?
• Is there any thing else I must know?

20
When should I submit for rating?

• There is no universal answer. It will depend
  on the research profile of the person asking the
  question. However, there are some commonalities
• Doctoral degree should be completed and articles
  from the PhD study should have appeared in the
  public literature (a must for the Y and P
  categories, ideally also for the other
  categories).
• There must be research outputs in the public
  domain. If these outputs are in written form they
  must be published (not in press etc).

21
When should I submit for rating?(cont)

• In the rating system quality is supreme. It is
  therefore not a question of the quantity of
  outputs but a question of the quality of outputs.
  
• The research outputs should include a meaningful
  number of those outputs which are considered to
  be among the most important in that research
  field.
• The research outputs should have appeared in the
  public domain for public scrutiny over a number
  of years.

22
When should I submit for rating? (cont)

• The outputs should reflect engagement in the
  international arena.
• If you are aiming for the Y and P category the
  application must be submitted before five years
  have elapsed since obtaining the doctoral degree.
• If you are tempted to list your unpublished PhD
  thesis as one of your research outputs you are
  not yet ready to submit for evaluation and
  rating!

23
When should I submit for rating? (cont)
Research outputs and their importance may
differ considerably from discipline to
discipline. It is therefore not possible to say
exactly how many and which research outputs are
necessary before applying for rating.
24
This presentation

• What is an NRF rating?
• Why should I get rated?
• What are the rating categories?
• When should I submit for rating?
• How do I apply for rating?
• What happens to my application?
• Is there any thing else I must know?

25
How do I apply for rating?

• Electronic application - application form
  http//nrfonline.nrf.ac.za
• Must be submitted by applicant to the institution
• Must be submitted by institution to the NRF

26
Information required from applicant

• Personal details
• Career profile
• Qualifications obtained
• Other biographic information
• Assessment panel(s) to consider application
• Nominated reviewers
• Application for L category?
• All research outputs of last seven years

• Five best recent research outputs published
  during the last seven years
• Ten best research outputs before that
• Description of completed research
• Self-assessment
• Postgraduate students
• Other research-based contributions
• Future research

27
Research outputs of the last seven years

• Publications in peer-reviewed journals
• Books/chapters in books
• Peer-reviewed published conference proceedings
• Other significant conference outputs

• Patents, artefacts and products
• Technical reports
• Keynote/Plenary addresses
• Other recognised research outputs

28
Nomination of reviewers
Applicants are given the opportunity to nominate
their own reviewers. They are also given the
opportunity to indicate who should not be
approached.   A mix of national and international
reviewers is appropriate.  
29
Sources of information

• Evaluation Centre website (http//www.nrf.ac.za/ev
  aluation/)
• Brochure on the NRFs evaluation and rating of
  the research performance of researchers in SA
  (http//www.nrf.ac.za/evaluation/Content/Documents
  /Rating/Evaluation_Brochure_2004_July.doc)
• Application form (http//nrfonline.nrf.ac.za)

30
This presentation

• What is an NRF rating?
• Why should I get rated?
• What are the rating categories?
• When should I submit for rating?
• How do I apply for rating?
• What happens to my application?
• Is there any thing else I must know?

31
What happens to my application?
32
Evaluation and Rating Process
Submission of scholarly achievements
Not accepted
Specialist Committee
Selection of 6 peers (reviewers)
Reviewers Reports
Assessor
Specialist Committee
Joint meeting Rating
33
Evaluation and Rating Process
Joint meeting Rating
No Consensus
Consensus
B, C, Y, L
A, P recommendation
Inform Candidate
Executive Evaluation Committee
Appeal
Appeals Committee
34
Assessment Panels

• Anthropology, Development Studies, Geography,
  Sociology and Social Work
• Animal and Veterinary Sciences
• Biochemistry
• Chemistry
• Communication, Media Studies Library and
  Information Sciences

• Earth Sciences
• Economics, Management, Administration and
  Accounting
• Education
• Engineering
• Forestry and Agricultural Sciences
• Health Sciences
• Historical Studies

35
Assessment Panels (cont)

• Law
• Literary Studies, Languages and Linguistics
• Mathematical Sciences
• Microbiology and Plant Pathology
• Performing and Creative Arts, and Design

• Physics
• Plant Sciences
• Psychology
• Political Sciences, Policy Studies and Philosophy
• Religious Studies and Theology

36
Tasks of Specialist Committees

• Selecting reviewers
• Assessing reviewers reports
• Recommending a rating for each applicant based on
  the comments by reviewers
• Identifying feedback
• Rating reports by reviewers
• Advising NRF

37
Selection of reviewers by Specialist Committees
Committees are provided with detailed
guidelines Committees select three names from
the applicants list and select three other
names   A mix of national and international
reviewers is appropriate.  
38
Guidelines to reviewers

• Comment on
• Quality of research outputs over the last seven
  years
• Standing as a researcher both nationally and
  internationally

39
This presentation

• What is an NRF rating?
• Why should I get rated?
• What are the rating categories?
• When should I submit for rating?
• How do I apply for rating?
• What happens to my application?
• Is there any thing else I must know?

40
Is there anything else I must know?
41
Critical success factors for the evaluation and
rating system

• Quality of documents submitted by applicant
• Selection of appropriate peers
• Composition of specialist panels
• Quality of reports by peers
• Clear definition of categories
• Fair and equitable procedures
• Goodwill of academic community, locally and abroad

42
Further clarification on

• Timing of first submission
• Policy on feedback
• Appeals process
• Alignment of rating and funding proposal
  processes
• Re-evaluation and special re-evaluations

43
Questions

• Thank you