Reviews

1
Reviews
ICML2003 Minitutorial on Research, Riting, and
Reviews

• Rob Holte
• University of Alberta
• holte_at_cs.ualberta.ca

2
Purpose

• Evaluate the papers scientific merit
• Check the validity of the papers claims and
  evidence
• Judge the papers relevance and significance
• Provide constructive feedback to the author

3
Example IJCAI99 review form

• 1. How RELEVANT is this paper to AI researchers?
• 2. How SIGNIFICANT is this paper?
• 3. How ORIGINAL is this paper?
• 4. Is this paper technically SOUND?
• 5. How well is this paper PRESENTED?
• Further comments, advice or explanations (Please
  be specific and
• constructive, especially with respect to any
  negative judgements
• above. Point to the section(s) where an error
  occurs, cite omitted
• references, etc.)

4
Excerpts from ICML03 form
GOALS/MOTIVATION. Does the submission state the
goals of the research, including the criteria by
which readers should evaluate the results? Is the
learning problem well-motivated?
CLAIMS/EVIDENCE. Do the authors make explicit
claims or draw clear conclusions, and do they
present reasonable evidence to support their
position? If the claims are theoretical, are the
proofs correct? If the claims are empirical, are
the experiments appropriate, and do the results
reveal the underlying reasons or causes for
phenomena? How might the authors strengthen
their claims, evidence, or reasoning?
5
Example COLT97 instructions
FOUNDATIONAL/CONCEPTUAL CONTRIBUTION Note
things like a new model, new notion, new
definition, new approach. Note the significance
and reasons for this novelty (and note the
absence of such a novelty). TECHNICAL
DEVELOPMENT - Introduces a new technique -
Novel use of known technique - Talented use of
known technique - Traditional use of known
technique - Trivial technically
6
COLT97 continued
RELATION TO OPEN PROBLEMS Does the paper solve
completely/partially/ or does it address an open
question? How important is this
question? (Central/ important/ interesting/
legitimate/ stupid). How much effort has been
invested before in solving it and by whom?
7
COLT97 continued
SOCIAL INTEREST IN PAPER Is it potentially
interesting to the whole COLT community, to a
major subarea, to everyone in a restricted area,
interesting only to the authors. HOW WILL IT
CONTRIBUTE Fertilization, satisfy curiosity, who
knows? PAPER TYPE Is it a - First step
(opens a new area) - Last step (closes an
important area) - Giant step (makes essential
progress) - None of the above.
8
Good Reviews

• Polite
• Fair
• Concise
• Clear
• Constructive
• Specific
• Well-documented
• Represent the scientific community

9
Anonymity

• Reviewers are supposed to be anonymous
• BUT, sometimes their identity becomes known or is
  at least guessed by the authors
• Write your reviews so that you would not be
  embarrassed if your identity was revealed

10
In my opinion

• Reviews are necessarily subjective
• A paper must convince its readers that its claims
  are valid and significant
• A reviewer is more careful, thorough and patient
  than an average reader

11
Reviews that miss the point

• Not uncommon, very frustrating
• Dont blame the reviewer. Fix your paper.

12
Conference reviews

• short time frame
• each reviewer has several papers to review
• Expect reviews to be terse, less thorough, less
  satisfying