On the Use of Probing-Question Approach in Grant Application Peer Reviews

1
On the Use of Probing-Question Approach in
Grant Application Peer Reviews

• ????????????????????
• Ping Sun (? ?)
• ISTIC / ORI, MOST
• Dalian,China May 21-23, 2012

2

• 1. Targeted problems
• 2. The fuci of relevant researches in China
• 3. Concrete measures are needed
• 4. Simple strategies might also be effective
• 5. Summary

3
1. Targeted problems

• As is the case with any process, peer review is
  not an infallible system and to a large extent
  depends on the integrity and competence of the
  people involved and the degree of editorial
  oversight and quality assurance of the peer
  review process itself.
• A UK House of Commons Committee report

4
Lack of Integrity and competence

• Agreeing to review an application beyond his/her
  competence.
• Entrusting review tasks to someone else without
  adequate approval.
• Not disclosing conflict of interest.
• Irresponsible behaviors, e.g. scores given and
  comments made are inconsistent comments made are
  not based on careful scrutiny of the applications
  

5
A mail reviewer is always right!

• Characteristics of the applications
• Extensive presentation?
• Brief presentation?
• Theoretically focused?
• Details included?
• Focus emphasized?
• Many factors listed?

• Possible comments of the reviewers
• Not focused
• Lack of in-depth understanding
• Lack of details
• Focus to much on tools and methods
• Other factors ignored
• Too ambitious, not focused

6
The suspicions of an failed applicant

• (NSFC now provides full-text comments)
• The first reviewer seems not read the research
  proposal carefully.
• The second reviewer is not familiar with the
  concepts and contents.
• The third reviewer is an expert in the field and
  did read the proposal, but the inquiry made
  (previous articles were almost all published on
  a single journal) is not reasonable.

7
Some causes of the problems

• Insufficient communication between applicants and
  reviewers
• The review process is not transparent, and the
  outcomes are un-appealable.
• The supervision of the peer review process is not
  in place.
• Personal factors of reviewers (COI, etc)
• The accountability of reviewers is not
  emphasized.

8
(Continued) Struggling Reviewers

• Many peers have to review more than 50 grant
  applications altogether (Xu Gong).
• Some reviewers are not familiar with the novice
  theories, methods, tools, and have to take more
  time.
• Some applications are not well presented, so the
  decisions are hard to make.
• Some reviewers are not quite sure about their
  decisions due to various reasons.

9
2. The Foci of relevant researches in China

• The development of principles and norms related
  to peer review
• The effectiveness and fairness of peer review
  process and outcomes
• The criteria and selection of reviewers
• The assessment of reviewers credit (integrity,
  competence, performance, etc)
• The control of conflict of interest (COI),
  including the COI of review organizers

10
3. Concrete measures are needed

• To address some major problems
• The mismatch of the criteria and the practical
  selection of reviewer, resulting in the
  qualifications of reviewers are not always
  guaranteed.
• The credit records of reviewers are scarce and
  not necessarily fit for use.
• The unpredictable nature of reviewers behaviors
  in a particular review process.

11
4. Simple strategies might also be effective

• The proposed practice
• To encourage reviewer to pose one or two
  questions to the applicants to clarify or explain
  something, if necessary.
• The questions will be sent via an automatic email
  relay system, with email address of the sender
  concealed.
• The communications between reviewers and
  applicants will be archived.

12
(Continued)

• What can be achieved
• A serious reviewer will be able to obtain
  additional information (interdisciplinary),
  without increasing the burden of anyone.
• A less serious reviewer will be more careful, at
  least with the questions asked.
• The records of the communications can be analyzed
  and examined for different purposes, if desirable.

13
(Continued)

• What differences can be made
• A source of reviewers credit information,
  collected and stored centrally
• The assessment and adjustment of reviewers can be
  made more easily.
• The reviewers awareness of accountability might
  be raised.
• The quality of peer review might be improved with
  extra information.

14
5. Summary

• 1. The use of probing-question approach is
  simple, but potentially effective in many
  aspects.
• 2. The approach will add positive personal
  experiences for both reviewers and applicants (to
  be supervised vs. the applications to be taken
  seriously).
• 3. The approach can promote integrity and the
  review quality at the same time.

15
Any comment?

• Do the problems listed also exist in Western
  countries?
• Is the approach a practical one?

16

• Thank you!