The ESF EUROCORES international, multistage peer review system

1
The ESF EUROCORES international, multi-stage peer
review system

• The international,
• multi-stage
• peer review system
• of the ESF
• EUROCORES
• Scheme
• involves
• up to 80 ESF MOs
• in gt 30 states,
• external, non-ESF
• partners,
• all scientific areas
• (interdisciplinarity)

2
Philosophy of the EUROCORES Scheme

• Collaborative research programme that combines
  bottom-up principle (theme proposals) with
  strategic priorities of funding agencies, by
  mutual agreement on
• Joint programme development (ESF manages dialogue
  between scientists and multiple funding
  agencies),
• Joint common Call for Proposals for
  Collaborative Research Projects, published by ESF
  / funding agencies,
• Common multistage international peer review of
  Collaborative Research Projects, managed by ESF
  through independent Review Panel and external
  referees,
• Acceptance of international ranking of
  Collaborative Research Projects by Review Panel,
• Joint management of the programme (Management
  Committee).
• N.B. EUROCORES retains the principle of juste
  retour

3
EUROCORES review process Actors and Stages

• Actors
• Meetings of the independent, interdisciplinary,
  international Review Panel
• (members suggested by funding agencies
  appointed by ESF can include non-Europeans)
• Detailed written online assessments by external
  referees
• (suggested by funding agencies, RP and ESF
  should include 1/3 non-Europeans per programme)
• Main stages
• Outline Proposals are selected by RP (two
  rapporteurs per proposal panel discussion gt
  consensus)
• Full Proposals are assessed by minimum 3 external
  referees (depending on disciplinary composition
  of proposal) and presented by two RP rapporteurs
• Applicants have right to rebuttal to referee
  assessment
• RP ranks Full Proposals, gives budgetary
  recommendation

4
EUROCORES Review Panel /-

• Plus
• RP meetings allow to discuss interdisciplinary
  merit and to overrule, by consensus, specialist
  referees
• RP gives detailed feedback to applicants
• RP has life cycle control of science progress
  throughout the programme duration
• Potential Minus
• RP members may act as national representative in
  project selection and ranking process
• Problematic
• RP do not have full insight into funding rules of
  national funding agencies
• ESF does not pay RP members

5
EUROCORES online assessment (OA) by external
referees /-

• Plus
• Comfortable access for RP, applicants (feedback)
  and funding agencies (transparency)
• Efficiently processed and stored
• Possibility for quality control of referee
  performance over time
• Potential Minus
• External referees suggested by funding agencies
  not always appropriate (level of English
  conflict of interest decline service)
• Non-European referees may misunderstand questions
  regarding European added value (scientific gain
  by transnational cooperation)
• Referees cant always judge compliance with
  specific national requirements (e.g.
  intellectual merit, broader impact)
• Problematic
• External referees seeing only one proposal cant
  compare
• Expert referees able to judge only one aspect of
  interdisciplinary proposals
• Referee fatigue ca. 30 good quality returns
• ESF does not pay external referees

6
EUROCORES Declaration of Interest

• Conflict of interest (incl. competing interest)
  a person involved in the assessment or evaluation
  of a proposal may benefit professionally or
  personally by the success, or failure, of that
  proposal.
• RP members and referees declare that they will
  abide by the Conflict of Interest guidelines and
  perform their task in full confidentiality.
• Guidelines
• 1. RP members must not be applicant
• 2. RP members must notify if
• - applicants are from RP members institution
  (institute),
• - applicants are close scientific collaborators
  of RP member,
• - applicants and RP members have jointly
  published in the last 3 years.
• 3. RP member declaring an interest leaves the
  room for the discussion of that proposal, will
  not receive referee details and is expected to
  refrain from intervening directly or indirectly
  on behalf of that proposal in any subsequent RP
  discussion on prioritisation.
• 4. All actions taken under these guidelines are
  recorded in the RP minutes to ensure full
  transparency.

7
Transparency of the EUROCORES review process

• Funding agencies see on dedicated, protected
  website
• All Proposals (outline and full)
• Anonymous referee reports and responses from
  proponents
• Review Panel records
• Master list on referees
• Applicants receive via dedicated, protected
  website
• Written RP comments on decisions taken regarding
  their proposal (and recommendations, if any)
• Online access to anonymous referee reports of
  their proposal (for rebuttal)
• Ranking position of their proposals (upon
  completion of the review process)
• Scientific community sees on programme website
• Detailed review process guidelines
• List of Review Panel members
• List of referees used (upon completion of the
  review process)

8
EUROCORES review process Two stages /-

• Plus
• Sifting of Outline Proposals helps to prevent
  applicants from elaborating complex Full Proposal
  not fitting the Call
• Assessment of Full Proposals by min. 3 external
  referees and 2 rapporteurs from Review Panel
  ensures multiple level quality control (covering
  disciplinary composition of proposal)
• External referees assigned by expert scientific
  staff at ESF
• Potential Minus
• 2 stage process adds ca. 2 months to procedure
• 2 level (Panel online assessment) process adds
  ca. 2 months to procedure
• Problematic
• Costly high input of expert scientific staff
  labour at ESF
• Time consuming high degree of transparency
  requires extensive discussions (to reach
  fully-documented and accepted consensus )