Scholarly Communications: Changes to Peer Review

1
Scholarly CommunicationsChanges to Peer Review

• Bradley Hemminger
• School of Information and Library Science
• University of North Carolina at Chapel Hill

2
Scholarly Communications Process
Present to colleagues V2
Present at conference V3
Idea V1
Submit to journal V4
Revision to include additional new results V8
Revision to update analysis V7
Referees Revision for journal V5
Journal Final Revision V6
3
Scholarly Communications Process
formulate
discussion
discussion, revision
Present to colleagues V2
Present at conference V3
Idea V1
Submit to journal V4
comments
comments
comments
Author revision
Revision to include additional new results V8
Revision to correct analysis V7
Referees Revision for journal V5
Journal Final Revision V6
Copyproofing
Criticisms, new thoughts, revision
new results, revision
Two peer reviews
4
Scholarly Communications Process Whats Produced
Journal Final Revision V6
5
Scholarly Communications ProcessWhat Id like to
see saved!
formulate
discussion
discussion, revision
Present to colleagues V2
Present at conference V3
Idea V1
Submit to journal V4
comments
comments
comments
Author revision
Revision to include additional new results V8
Revision to correct analysis V7
Referees Revision for journal V5
Journal Final Revision V6
Copyproofing
Criticisms, new thoughts, revision
new results, revision
Two peer reviews
6
Peer Review Output
With Respect to XYZ Accept reject revise
Review (Peer)
Qualitative Grade
Article
Qualitative Comments
Comments to Author
7
Generalized Review Model
Score (1-10)
Quantitative Grades
Review (open, peer, machine)
Accept, Reject, Revise, With respect to XYZ
Qualitative Grade
Article
Qualitative Comments
Comments to Author
8
Overview of Peer Review
Quantitative Grade
Score (1-10)
Filter
Published Article
Review Peer, Open, Machine
Accept, reject, revise with respect to XYZ
standards
Article submitted
Send elsewhere
Qualitative Grade
Comments to Author
Reject
Qualitative Comments
9
General Review Model Parallels

• In general, you have sample (material) which is
  judged/scored quantitatively and qualitatively by
  an identified observer.
• Current Peer Review
• Automated Scoring Systems (lab tests)
• Moderated email lists (announce)
• Moderated Eprints servers (arXiv)

10
Peer Review Options
Comment
Quantitative Score (1-10) Score (1-10) Score
(1-10) Score (1-10) Score (1-10) citations hit
s number of related discussions
Qualitative Rel Yes/No Group Rel Yes/No
Group Absolute Absolute Rel Yes/No
Group Rel Yes/No Group
Y Y Y Y Y ?

• Human Judgement
• Expert peer review (status quo)
• Certified expert peer review
• Open Peer Review BMJ, BioMed
• Open comment review pyscprints
• Computer Judgement
• Computer peer review
• Human Usage
• Citation-based (CiteSeer)
• Usage counts (CiteSeer) Example
• Quantity of discussion
• Coarse Categorization
• Two Tier (grey/gold)
• Moderator (current arXiv)
• No review (old arXiv)

11
Judgment based on some combination of
reviews/comments
formulate
discussion
discussion, revision
Present to colleagues V2
Present at conference V3
Idea V1
Submit to journal V4
comments
comments
comments
Author revision
Revision to include additional new results V8
Revision to correct analysis V7
Referees Revision for journal V5
Journal Final Revision V6
Copyproofing
Criticisms, new thoughts, revision
new results, revision
Two peer reviews
12
What areas of improvement?

• Review Process Change
• Search, Retrieval Process Change
• Service Provider Process Change

13
Review Process Changes

• Include open reviews and comments to get
  additional feedback.
• Support normal scholarly discourse, allowing give
  and take with author responding.
• Add quantitative scores to allow better filtering
  based on quality during retrieval
• Add machine (automated) reviews

14
Search and Retrieval Changes

• Universal Archive all material freely available.
• Universal Searching standardized metadata
  (Dublin Core) for general searching.
• Automated agents to bring material of interest to
  your attention.
• Use additional review scores (public reviews,
  machine) to help filter search.
• Example article scores gt 7.0, refereed,
  citation count above X, typeresearch article,
  search terms schizophrenia, geneX)

15
Provider Service Change

• What is worth paying for?
• Quality review (Faculty of 1000)
• Proofing, citation linking, professional
  presentation (CiteSeer, Cite-base)
• Archival (JStor)
• Who hosts material
• Society (arXiv)
• Commerical Publishers (Elesiever,BioMedCentral)
• University Library (MIT Dspace)

16
New Frameworks for Peer Review

• As an enabling technology frameworks like NeoRef
  supports all of the above models in any
  combination at the same time, while eliminating
  many of the costs.
• Requirements Based on OAI and Dublin Core, and
  expectation of logical universal archive, an
  universal unique object IDs (URL, DOIs) and
  person IDs.

17
Example Model (NeoRef)

• All material and metadata are author contributed
  to a public OAI archive (author retains
  ownership).
• All materials universally available via search
  engines that harvest metadata from OAI archives.
• OAI archives have automated or manual moderator
  to filter out junk.
• Everything--articles, reviews, comments,
  indexings, etc., are stored as digital content on
  archive using the same mechanism. Reviews contain
  quantitative score, qualitative grade,
  qualitative comments. Logically (although not
  physically), a two tier (Grey Gold) system for
  materials
• High quality keep forever material reviewed by
  known entity
• Grey material (everything else)

18
NeoRef for Movies, Dates, DocSouth

• The same process used by NeoRef to support
  Scholarly Communication could be used for almost
  any purpose. All that is required is storage of
  Digital Content Items, and linking of reviews,
  comments, etc to them.
• Movies Grey is everyones reviews Gold is
  Siskel and Ebert reviews
• DocSouth self cataloged and indexed items are
  Grey librarian/archivist cataloged and indexed
  items are Gold.

19
Can we save the Gold and Grey?
formulate
discussion
discussion, revision
Present to colleagues V2
Present at conference V3
Idea V1
Submit to journal V4
comments
comments
comments
Author revision
Revision to include additional new results V8
Revision to correct analysis V7
Referees Revision for journal V5
Journal Final Revision V6
Copyproofing
Criticisms, new thoughts, revision
new results, revision
Two peer reviews
20
NeoRef Storage Model
Auto-indexing
Revision to include additional results and
analyses V8
Author Indexing
Journal Final Revision V6
Comments on V3
Journal Submission V4
Comments on V6
Conference paper (v3)
Material expressing content
Local powerpoint Presentation v2
Two peer reviews
Machine Review
Digital Archive
Filter (Moderate)
Author
Automated
Grey Literature
Recognized Expert
Top Tier (Keep Forever)
Open (anyone)
21
What do users want?
22
The Association of Learned and Professional
Society Publishers (ALPSP) SurveyAuthors and
Electronic Publishing

• Scholarly research communication has seen
  far-reaching developments in recent years.
• Most journals are now available online as well as
  in print, and numerous electronic-only journals
  have been launched
• the Internet opens up new ways for journals to
  operate.
• Authors have also become conscious of alternative
  ways to communicate their findings, and much has
  been written about what they ought to think.

23
ALPSP felt that it would be timely to discover
what they actually thought and what they actually
did. This survey aimed to discover the views of
academics, both as authors and as readers. Some
14,000 scholars were contacted across all
disciplines and all parts of the world, and
nearly 9 responded their detailed comments make
thought-provoking reading.
Alma Swan and Sheridan Brown. Authors and
Electronic Publishing The ALPSP Research Study
on Authors' and Readers Views of Electronic
Research Communication. (West Sussex, UK The
Association of Learned and Professional Society
Publishers, 2002). http//www.alpsp.org/pub5.ht
m
24
Importance of the Peer Review Process
http//www.alpsp.org/pub5.ppt
25
Importance of journal features
26
Importance of the peer review process
27
Importance of publishers roles
28
Importance of future dissemination channels
29
http//www.update-software.com/Cochrane/MR000016.p
df
30
Cochrane Methodology Review

• Despite its widespread use and costs, little hard
  evidence exists that peer review improves the
  quality of published biomedical research.
• There had never even been any consensus on its
  aims and that it would be more appropriate to
  refer to it as competitive review.

Caroline White, Little Evidence for
Effectiveness of Scientific Peer Review, BMJ
326 (February 1, 2003) 241 http//bmj.com/cgi/rep
rint/326/7383/241/a.pdf
31
Cochrane Methodology Review

• On the basis of the current evidence, the
  practice of peer review is based on faith in its
  effects, rather than on facts,' state the
  authors, who call for large, government funded
  research programmes to test the effectiveness of
  the classic peer review system and investigate
  possible alternatives.

Caroline White, Little Evidence for
Effectiveness of Scientific Peer Review, BMJ
326 (February 1, 2003) 241 http//bmj.com/cgi/rep
rint/326/7383/241/a.pdf
32
Cochrane Methodology Review

• The use of peer-review is usually assumed to
  raise the quality of the end-product (i.e. the
  journal or scientific meeting) and to provide a
  mechanism for rational, fair and objective
  decision-making. However, these assumptions have
  rarely been tested.

Tom O. Jefferson, Phil Alderson, Frank Davidoff,
and Elizabeth Wager, Editorial Peer-review for
Improving the Quality of Reports of Biomedical
Studies. (Middle Way, Oxford Update Software
Ltd, 2003). http//www.update-software.com/Cochran
e/MR000016.pdf
33
Cochrane Methodology Review

• The available research has not clearly identified
  or assessed the impact of peer-review on the more
  important outcomes (importance, usefulness,
  relevance, and quality of published reports)
• Given the widespread use of peer-review and
  its importance, it is surprising that so little
  is known of its effects

Tom O. Jefferson, Phil Alderson,Frank Davidoff,
and Elizabeth Wager, Editorial Peer-review for
Improving the Quality of Reports of Biomedical
Studies. (Middle Way, Oxford Update Software
Ltd, 2003). http//www.update-software.com/Cochran
e/MR000016.pdf
34
FURTHERMORE

• 16 said that the referees would no longer be
  anonymous
• 27 said that traditional peer review would be
  supplemented by post-publication commentary
• 45 expected to see some changes in the
  peer-review system within the next five years

Fytton Rowland, The Peer-Review Process,
Learned Publishing 15 no. 4 (October 2002)
247-258.
Report version http//www.jisc.ac.uk/uploaded_doc
uments/rowland.pdf