Thoughts on how to improve reviews

1
Thoughts on how to improve reviews

• Paul Francis
• (Presented by Robbert Van Renesse)
• Cornell

2
Main problem with current system

• Reviews are often of poor quality
• I suspect main reason is that PC members are
  overworked
• Both ideas in this presentation are focused on
  this one problem alone

3
Idea One

• Allow PC members to ask authors simple where in
  the paper can I find this? questions
• Authors can answer ONLY with page, column, and
  line numbers
• Why?
• Often when I reject a paper, it is based on one
  or a few specific flaws. It would be good to
  verify that Im not overlooking something.

4
A reviewer comment

• But you can do this today...just ask the PC
  chairs to forward an email
• My rebuttal
• Bothering the PC chairs is a significant
  deterrent
• Should really limit author reply to text already
  in paper...otherwise author is effectively
  submitting more than what is in the paper

5
Today
6
Proposal One Pass reviews on
7
Observations

• Reviewer gets the benefit of previous reviewers
• Various ways this can be exploited
• To improve review Reviewer first does complete
  unbiased review, then uses previous reviews as
  sanity check
• To reduce work Reviewer scans paper, previous
  reviews, and rebuttal, and then does only enough
  work to decide if author has overcome previous
  criticisms
• This can be done in good faith

8
Observations

• Good paper should be able to overcome earlier
  criticisms
• Without having to torque paper to satisfy future
  reviewers....
• Instead author can use rebuttal
• Truly bad paper should not be able to overcome
  earlier criticisms
• Prevents bad paper from retrying until it gets
  lucky
• Might discourage hail-mary submissions

9
A reviewer comment

• This will promote reviewer laziness
• My rebuttal
• A good-faith reviewer will benefit from earlier
  reviews
• A lazy reviewer already has many ways of being
  lazy (form instant opinion, do minimal work to
  support opinion...)