Paper Reviewing

1
Paper Reviewing
2
(No Transcript)
3
(No Transcript)
4

• Please note
• These ideas are not mine alone.
• I have gleaned the content from others.

5

• It is important to remember that a reviewer is
  asked to provide an informed opinion about a
  manuscript.
• The decision whether the manuscript will be
  published is made solely by the editor.

6
Why is a review necessary?

• The peer review serves several roles, although
  the precise combination varies with the type of
  review. The most important reasons for review
  include finding deficiencies in
• technical approach and analysis
• computation
• ignorance of related research.

7
Types of reviews

• There are three types of reviews
• "anonymous",
• In an anonymous review, the editor solicits a
  referee to review the article. The referee
  returns the review to the editor who, after
  removing any identification, gives it to the
  author. Academic journals typically use the
  anonymous review, but it is also used for books,
  articles in proceedings, and some reports.
• "friendly", and
• Many authors send drafts of articles or reports
  to other experts and solicit their comments. This
  is called a "friendly" review. In such cases, the
  reviewer is known to the author. The timid
  reviewer may be reluctant to harshly criticize a
  paper, so these are less valued than an anonymous
  review (although a true friend should be the
  severest critic in private).
• "internal".
• Many laboratories and research institutes require
  that all papers be internally reviewed prior to
  submission to a journal or proceedings. The
  quality of such reviews is highly variable, from
  extremely rigorous to worthless beyond protecting
  the author from the most outrageous errors.
• In all cases, however, the procedure to review a
  paper is fundamentally similar.

8
Most reviews have four parts

• referee's review form
• additional comments
• original paper
• cover letter to editor.

9
Goals of a Review

• Guide the program committee in selection process
• Help authors
• to revise paper for acceptance,
• to understand rejection,
• to improve further research and future projects

10
Structure of a Review

• 1/2 to 1 page of text (2 - 4 paragraphs)
• Longer reviews are generally given for better
  papers, shorter reviews for bad papers
• 1 paragraph executive summary
• what is the paper trying to do?
• what is potential contribution of paper?
• short summary of strengths and weaknesses
  (sentence or 2)
• accept/reject (hard, because you don't
  necessarily see the entire sample)
• several paragraphs of details (listed in order of
  importance)
• technical flaws?
• structure of paper?
• are key ideas brought out?
• don't want to just describe system, also need
  motivation and justification of approach
• presentation? (ex undefined terms, confusing
  wording, unclear sections...)
• justification -- can they say why ideas are
  important?
• comparison with other systems? For bigger
  conferences (SOSP, ISCA, ASPLOS) need
  quantitative evidence of ideas
• grammar? (usually only point out consistent
  errors)

11
Process of a Review

• When reviewing a paper it is recommended that a
  reviewer read it through three times
• the first for general overview and familiarity
  with the subject,
• the second for argument detail, and
• the third for recording information about the
  paper to be used in the review. 
• With three reads a reviewer should be able to
  make not only a decision about which papers are
  strong enough to include in the conference, but
  also be able to make a convincing argument about
  why the decision was made. 

12
Possible question subjects for review of an
abstract

• The relevance of the argument to the conference
  theme
• The importance of the research
• The originality of the topic
• The quality of the abstract

13
Possible question subjects for review of a paper

• The same questions used for abstract review plus
  
• Source usage
• Clarity of argument
• The use of images
• Sentence structure
• Paper length
• Standard of the conference language, such as
  English or French
• Appropriateness of the title

14
(No Transcript)
15
Some tips on wording the review

• If you identify a shortcoming in the paper, you
  should suggest a solution.
• ("Experiments would be more convincing than
  simulation.")
• This is the heart of constructive criticism if
  you find a problem but can't think of a
  suggestion or solution, then you're not being
  helpful, you're being destructive.

16
Some tips on wording the review

• Your criticisms should be directed at the paper,
  not the author.
• "The paper did not cover" is preferred to "The
  authors did not cover"

17
Some tips on wording the review

• Sometimes you will get a paper that had a good
  idea but was hopelessly executed and written.
• The right thing is to end your review with
  something to the effect that you thought it was a
  good idea, though possibly premature, and you
  look forward to reading a future version.
• If it's a bad idea and badly written, don't
  encourage them to rewrite it.

18
Some tips on wording the review

• You should not indicate who you are.
• Some people may be tempted to take out a bad
  review on you.
• I know someone who once had to listen to a person
  complain about a review his advisor had written
  of his article for about 15 minutes. It was clear
  the guy was out for revenge. What he didn't
  realize (thanks to blind reviewing) was that the
  student he was talking to had written the
  review. But that student had made reference to a
  couple of things only people associated with his
  advisor's lab would know about, so the angry
  author was able to figure out that the review
  originated at Georgia Tech.

19
Review Philosophy

• When we get a paper to review, at the beginning
  we should always have as the default that we
  accept the paper.
• While reading the paper, we may start raising
  specific objections along the issues in the
  review form, namely the goals are not stated, the
  system is not well described, the approach is not
  novel or not validated, etc. etc.

20
Review Philosophy

• Each objection weighs a little against our
  initial default acceptance.
• Rejecting a paper is to see if these objections
  weigh more than our threshold, based on our
  experience with other other conferences, papers,
  and advice from the specific conference. (Of
  course, the review can also raise the initial
  default acceptance, and then it's even a clearer
  accept.)

21
Review Philosophy

• One word of care that I recall It may happen
  that we raise either unjustified objections or
  support to a paper.
• In particular in AI, it may be rather common that
  we completely "disagree" or "agree" with the
  paper's approach/results. We have to be very
  careful, as much as possible, not to include
  objections or support that corresponds to
  subjective, or sometimes dogmatic, opinions about
  the work.

22
Twelve Tips for Reviewers by Henry L. Roediger,
III

• Know your mission.
• Be speedy.
• Read carefully.
• Say positive things in your review.
• Dont exhibit hostility or mean-spiritedness in
  your review.
• Keep it brief.

23
Twelve Tips for Reviewers by Henry L. Roediger,
III

• Dont nitpick.
• Develop a good reviewing style.
• Be careful in recommending further
  experimentation.
• Watch for egocentrism.
• Make a recommendation about the paper,
• unless the instructions from the editor tell you
  not to.
• Sign your review.

24
A Common Question

• How many papers should I be expected to referee
  per year?
• you should do at least as much work for the
  community as the community does for you.
• (2-3 reviews per paper)

25
(No Transcript)
26
Benefits for the reviewer

• Enhanced reputation
• for junior reviewers editors may eventually be
  references qualify as candidates for appointments
  to PCs / Editorial boards
• Being informed
• finger on the pulse of the field (rather for
  conference papers) but confidential
• Learning from others peoples mistakes
• how to write better papers yourself
• More work
• but also more routine and experience

27
(No Transcript)
28
Class Reviewing Process

• We have N students,
• In every round, N papers will be reviewed.
• Two students who are assigned the same paper are
  called peers of each other for this paper.

29

• First round
• Student 1 and Student 2 are peers of each other
  for papers 1 and 2

30
One Round of paper Reviews

• Week 1
• Read first and second paper
• Week 2
• Write review of first paper and send it to me.
• Week 3
• Write a review for your second paper and send it
  to me.
• Before Class at the beginning of Week 4 send your
  two reviews to your peer,
• Week 4
• Send feedback to your peer (cc me) for the
  reviews you received.
• Week 5
• Revise your reviews and submit them to me

31
(No Transcript)
32
Taxonomy of (acceptable) research papers

• Breakthrough
• solving a fundamental problem
• e.g. parallel sorting in logarithmic time
  AKS83
• Ground-breaking
• pioneering papers exploring new fields of
  research
• Progress (most papers)
• (raises and) solves new problem
• e.g. new algorithm with improvement in asymptotic
  complexity
• Reprise
• more elegant or clear solution / proof for an
  already solved problem

33
Taxonomy of (acceptable) research papers

• Tinkering
• extend a known result by a more careful but
  non-obvious analysis
• Debugging
• elucidates and repairs a previously undiscovered
  flaw in published work
• Survey / Comparison
• surveys and unifies several existing approaches
  for a specialized subject

34
(No Transcript)