Alternative Peer Review: Quality Management for 21st Century Scholarship

1
Alternative Peer ReviewQuality Management for
21st Century Scholarship
http//www.public.iastate.edu/gerrymck/APR.ppt

• Gerry McKiernan
• Science and Technology Librarian
• and Bibliographer
• Iowa State University Library
• Ames IA
• USA

gerrymck_at_iastate.edu
2
Workshop on Peer Review in the Age of Open
Archives

• International School for Advanced Studies
• Interdisciplinary Laboratory
• Trieste, Italy
• May 23-24, 2003

3
THANK YOU!

• Workshop Advisory Board (Marco Fabbrichesi
  (INFN/SISSA Italy), Stevan Harnad (University of
  Southampton, UK), Stefano Mizzaro (University of
  Udine, Italy) and Corrado Pettenati (CERN
  Library, Geneva, Switzerland)
• Iowa State University, Faculty Senate, Committee
  on Recognition and Development
• European Commission
• Iowa State University Library
• Heike Kross, Ph.D.

4
DISCLAIMER (1) The screen prints selected for
this presentation are for educational purposes
and their inclusion does not constitute an
endorsement of an associated product, service,
place, or institution.
5
DISCLAIMER (2) The views and opinions expressed
in this presentation are those of the presenter
and do not constitute an endorsement by Iowa
State University or its Library.
6
NOTICE
No editors, authors, or referees were harmed in
the preparation of this presentation.
7
http//www.sciencecartoonsplus.com/
8
Giuseppe De Nittis (1846-1884) The Macchiaioli /
Italian Impressionists
9
Campo di Biche (1875)
10
PEER REVIEW DEFINITION

• Peer review is the assessment by an expert of
  material submitted for publication.

Carin M. Olson, Peer Review of the Biomedical
Literature, American Journal of Emergency
Medicine 8 no.4 (July 1990) 356-358.
11
PEER REVIEW PURPOSES
Peer review helps to ensure that published
research is
Important Original
Timely Technically-reliable
Internally-consistent Well-presented
Benefited from guidance by experts
Carin M. Olson, Peer Review of the Biomedical
Literature, American Journal of Emergency
Medicine 8 no.4 (July 1990) 356-358.
12
PEER REVIEW STRENGTHS
The underlying strength of peer review isthe
concerted effort by large numbers of researchers
and scholars who work to assure that valid and
valuable works are published and conversely to
assure that invalid or non-valuable works are not
published .
Anne C. Weller, Editorial Peer Review Its
Strengths and Weaknesses. (Medford, NJ
Information Today, 2001).
13
Houston, We Have a Problem!
14
PEER REVIEW PROBLEMS

• Subjectivity
• Bias
• Abuse
• Detecting defects
• Fraud and Misconduct
• Delay

Fytton Rowland, The Peer-Review
Process, Learned Publishing 15 no. 4 (October
2002) 247-258.
Report version http//www.jisc.ac.uk/uploaded_doc
uments/rowland.pdf
15
SUBJECTIVITY

• Summary rejections by editor without sending the
  paper to referees
• Choice of referees by the editor (choosing for
  example, a known harsh referee for a paper the
  editor wishes to see rejected)

16
BIAS

• Discrimination against authors because of their
  nationality, native language, gender or host
  institution
• Situations where author and referee are
  competitors in some sense, or belong to warring
  schools of thought

17
ABUSE

• Too many articles out of one piece of research,
  or duplicate publication
• Intellectual theft omission or downgrading of
  junior staff by senior authors
• Plagiarism (stealing others yet unpublished work
  that has been sent for review)
• Delaying publication of potentially competing
  research

18
DETECTING DEFECTS

• Identification of factual errors within submission

19
FRAUD and MISCONDUCT

• Fabrication of results
• Falsification of data
• False claim of authorship for results

20
DELAY

There is much muttering about publication delay,
a real enough problem, especially in paper
publication, but peer review itself is often
responsible for as much of the delay as the paper
publication and distribution process
itself. Stevan Harnad
Stevan Harnad, Implementing Peer Review on the
Net Scientific Quality Control in Scholarly
Electronic Journals, in Scholarly Publication
The Electronic Frontier, edited by Robin P. Peek
and Gregory B. Newby (Cambridge MA MIT Press,
1996).
http//www.ecs.soton.ac.uk/harnad/Papers/Harnad/h
arnad96.peer.review.html
21
http//www-marketing.wharton.upenn.edu/ideas/pdf/p
eerpaper.pdf
22
Peer review is slow, expensive, profligate of
academic time, highly subjective, prone to bias,
easily abused, poor at detecting gross defects,
and almost useless in detecting fraud.
Richard SmithEditor, BMJ

• Richard Smith, Opening Up BMJ Peer Review,
• BMJ 318 (7175) (January 2 1999) 4-5

23
Stephen Lock, A Difficult Balance Editorial Peer
Review in Medicine (Philadelphia, PA ISI Press,
1986).
24
(No Transcript)
25
EXAMPLE

• Jan Hendrik Schön

26
-
http//archive.salon.com/tech/feature/2002/09/16/p
hysics/
27
Science
Nature
http//physicsweb.org/article/news/6/9/15/
28
http//agenda.cern.ch/fullAgenda.php?idaa01193
29
RECOMMENDATIONSWorkshop on the Open Archives
Initiative (OAI)and Peer Review Journals in
Europe, CERN, Geneva Switzerland, March 22-24,
2001

• The participants were unanimous in their belief
  that the certification of scholarly work remains
  a fundamental part of a system for scholarly
  communication.
• It was also generally believed that the
  electronic environment allows for novel
  approaches to accord a stamp of quality to
  scholarly works.

Alison Buckholtz, Raf Dekeyser, Melissa
Hagemann, Thomas Krichel, and Herbert Van de
Sompel, Open Access Restoring Scientific
Communication to Its Rightful Owners, European
Science Foundation Policy Briefing 21 (April
2003) 1-8.
http//www.arl.org/sparc/SPB21_OAI.pdf
30
Let us be imaginative in exploring the
remarkable possibilities of this brave new
medium.

• Stevan Harnad, Implementing Peer Review on the
  Net Scientific Quality Control in Scholarly
  Electronic Journals, in Scholarly Publication
  The Electronic Frontier, edited by Robin P. Peek
  and Gregory B. Newby (Cambridge MA MIT Press,
  1996).

http//www.ecs.soton.ac.uk/harnad/Papers/Harnad/h
arnad96.peer.review.html
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
Let us be more imaginative in exploring the
remarkable possibilities of this brave new
medium. With Apologies to Stevan Harnad
36
(No Transcript)
37
TM
http//lockss.stanford.edu/
38
LOCKSS

• For centuries libraries and publishers have had
  stable roles publishers produced information
  libraries kept it safe for reader access. There
  is no fundamental reason for the online
  environment to force institutions to abandon
  these roles.
• The LOCKSS model capitalizes on the traditional
  roles of libraries and publishers.  LOCKSS
  creates low-cost, persistent digital "caches" of
  authoritative versions of http-delivered
  content. 

39
LOCKSS

• The LOCKSS software enables institutions to
  locally collect, store, preserve, and archive
  authorized content thus safeguarding their
  community's access to that content. 
• The LOCKSS model enforces the publisher's access
  control systems and, for many publishers, does no
  harm to their business models.

40
LAMPSS
Lots of Alternative Models Provide Sensible
Solutions
41
(No Transcript)
42
DISCLAIMER
ON

• The alternative peer review models profiled are
  for informational and educational purposes only
  and do not necessarily constitute an endorsement.

43
ALTERNATIVE PEER REVIEW

• Institution-based
• Citation-based
• Index-based
• Metadata-based
• Computer-assisted
• NO Peer Review
• Moderator-based
• Tier-based

• Neo-Classical
• Certification-based
• Open Peer Review
• Commentary-based
• Collaborately-filtered

44
NEO-CLASSICAL PEER REVIEW
45
Neo-Classical Peer Review
46
Neo-Classical Peer Review
47
Neo-Classical Peer Review
48
CERTIFICATION-BASED
49
Certification-Based

• The process of pre-publication peer review
  could be improved and become a more reliable
  indicator of manuscript quality if reviewers were
  trained in, and subsequently applied
  systematically, critical skills and use of a
  hierarchy of evidence to classify submitted
  articles being reviewed.

Stephen Pritchard , Peer Review - a Proposal for
Change, Paper presented at Thinking Globally -
Acting Locally Medical Libraries at the Turn of
an Era, 8th European Conference of Health and
Medical Libraries, September 16-21, 2002,
Deutsche Zentralbibliothek für Medizin, Köln,
Germany. http//www.zbmed.de/eahil2002/abstracts/p
ritchard.pdf
50
OPEN PEER REVIEW
51
Open Peer Review

• IDENTIFICATION OF REVIEWERS / SIGNED REVIEWS
• BMJ
• bmj.com
• BioMed Central
• biomedcentral.com
• eMJA (Medical Journal of Australia)
• www.mja.com.au/public/information/project.html

52
COMMENTARY-BASED
53
Commentary-based

• Readers can comment before and/or after classic
  peer review, or instead of classic peer review
• Electronic Transactions on Artificial
  Intelligence
• (www.etaij.org)
• OPEN REVIEW / REFEERING
• Journal of Interactive Media in Education
• (www-jime.open.ac.uk)
• PRE- AND POST- COMMENTARY
• Psycoloquy
• (psycprints.ecs.soton.ac.uk)
• POST PEER REVIEW COMMENTARY)

54
Electronic Transactions on Artificial
Intelligence
55
Journal of Interactive Media in Education
56
http//psycprints.ecs.soton.ac.uk/
57
(No Transcript)
58
(No Transcript)
59
COLLABORATIVELY-FILTERED
60
Collaboratively-Filtered

• DEFINITION Guiding people's choices of what
  to read, what to look at, what to watch, what to
  listen to (the filtering part) and doing that
  guidance based on information gathered from some
  other people (the collaborative part)."
• Paul Resnick

http//www.cni.org/Hforums/cni-announce/1996/0031.
html
61
ResearchIndex / CiteSeer
http//www.researchindex.com
62
INSTITUTION-BASED
63
Institution-Based

• INSTITUTIONAL INITIATIVES
• DSpace (MIT)
• www.dspace.org
• eScholarship (University of California)
• escholarship.cdlib.org
• Glasgow ePrint Service (University of Glasgow)
• eprints.lib.gla.ac.uk

64
CITATION-BASED
65
Citation-Based

• Citations to Open Access / OAI-compliant
• documents are indicators of document importance

http//citebase.eprints.org/
66
(No Transcript)
67
(No Transcript)
68
(No Transcript)
69
(No Transcript)
70
INDEX-BASED
71
Index-Based

• INDEXING OF EPRINTS BY COMMERICAL ABSTRACTING AND
  INDEXING SERVICE
• Chemical Abstracts (American Chemical Society)
  (CAS) indexes select appropriate e-prints from
  the arXiv.org eprint server as well from the
  Chemical Preprint Service (Elsevier)
• Its selection criteria for this kind of
  electronic document are essentially the same as
  for the traditional printed documents they must
  report new information of chemical or
  chemistry-related interest and must be original
  publications. Also, the electronic publication
  must be publicly available and have some relative
  permanence .
• Eric Shively / Chemical Abstracts Service

72
Index-Based

• CAB Abstracts doesnt currently include Eprints
  or Preprints, but we are looking at the
  implications and possible mechanisms for
  accessing and indexing Eprints and/or Preprints
  related to the applied life sciences.
• Tracy Shaw / CAB International

73
METADATA-BASED
74
Metadata-Based
ltoai-qualitygt ltcategorygtinternallt/categorygt ltproc
essgt peer review lt/processgt ltorganizationgt CERN lt/
organizationgt ltpoliciesgt http//www.cern.ch/polici
es/review.html lt/policiesgt lt/oai-qualitygt
William Y. Arms, Quality Control in Scholarly
Publishing On The Web. What Are the Alternatives
to Peer Review? PowerPoint presentation given at
the Workshop on the Open Archives Initiatives
(OAI) and Peer Review Journals in Europe, March
22-24, 2003, CERN, Geneva, Switzerland. http//www
.cs.cornell.edu/wya/papers/CERN-2001.ppt
75
COMPUTER-ASSISTED
76
Computer-Assisted (1)

• SOFTWARE THAT ASSISTS IN THE EVALUATION OF A
  SUBMITTED MANUSCRIPT
• A Software Program to Aid in Peer Review
• Alvar Loria and Gladys Faba
• Objective To characterize a personal
  computer-based software program developed as an
  aid to peer review of medical papers.
• Design The software is a Windows-based
  application that records automatically a numeric
  score to a series of questions related to 8
  sections of scientific papers (introduction,
  methods, results, and discussion, plus 4 other
  sections). The questions and sections vary
  according to type of paper (original reports,
  case reports, or reviews), and the final output
  is a score with a maximum of 100 for a "perfect"
  paper. The software was tested using a single
  reviewer to judge 289 papers (169 original
  reports, 50 case reports, and 70 reviews) from 44
  Mexican medical journals. All statistical
  analysis of scores were done with nonparametric
  tests.

77
Computer-Assisted (2)

• Results The paper scores ranged from 29 to 97
  with slightly higher median and less dispersion
  of scores for reviews as compared with original
  reports and case reports, but these differences
  did not reach significance. Two observations
  suggest that the software operated reasonably
  well a) there were some differences in the
  section scores by type of paper that agreed well
  with differences in their complexity b) the
  journal scores showed an association with their
  number of original papers and their percentage of
  original papers (Kruskal-Wallis test, P.06 and
  0.07, respectively).
• Conclusions The software operated reasonably
  well when used to compare the relative quality of
  289 papers. The validity of the program is
  restricted in this study to the experience of 1
  reviewer. An analysis of the raw scores helped in
  detecting some ambiguous and redundant questions
  that have been modified in an improved version.
  The program has a potential as a training tool
  for inexperienced reviewers or as a scorekeeper
  for experienced peer reviewers.

Alvar Loria and Gladys Faba, A Software Program
to Aid in Peer Review, Abstract of paper
presented at the Third International Congress on
Biomedical Peer Review and Global Communications,
September 18-20, 1997, Prague, Czech Republic.
http//www.ama-assn.org/public/peer/arev.htm
78
NO PEER REVIEW
79
NO Peer Review
http//xxx.arXiv.cornell.edu
80
MODERATOR-BASED
81
Moderator-based (1)

• The intent of this model is to allow the widest
  range of scientific manuscripts to be archived,
  searched, and distributed electronically at the
  lowest possible cost.
• This would be accomplished through very minimal
  filtering and subsequent placement of eprints on
  a non-commercial archival server by a
  subject-specific Moderator appointed by a society
  (or consortia of societies).
• A society-appointed Editorial Board (with
  double-blind peer review approved by the
  non-profit Peer Review Inc. organization) would
  then the identify the most important materials
  from among these archived items, and the stamp of
  approval for these items would be included in a
  secondary Virtual Collection.

82
Moderator-based (2)

• There are no direct submissions to the Editorial
  Board manuscripts would be directed to the
  Editorial Board in one of three ways
• 1. nominated by the eprint Moderator upon receipt
  for the archival server,
• 2. notification sent to the Editorial Board when
  a threshold number of hits are generated by any
  one manuscript on the archive server, and
• 3. nominated by readers of material from the
  archive this process requires a letter of
  support outlining the importance of the work to
  the Editorial Board.

83
Moderator-based (3)

• The Virtual Collection could be produced as a
  variety of products
• enhanced abstracts
• email threads (with comments)
• virtual reviews of sub-disciplines
• SDIs (selective dissemination of information)
  current awareness tools
• This process
• reduces the load on the Editorial Boards, which
  results in a faster review process
  differentiates those items worthy of higher
  recognition from those worthy of archiving,
  making it easier for a reader to filter material,
  based upon a society and discipline authority
  (rather than commercial reasoning)
• provides for search/browse/sdi from the Virtual
  Collection for filtered info, reducing this more
  expensive option for only those items recognized
  as of the highest quality.

84
3
2
1
David Stern, The eprint Moderator Model,
Newsletter on Serials Pricing Issues no. 214
(February 8, 1999). http//www.lib.unc.edu/prices/
1999/PRIC214.HTML214.5
85
TIER-BASED
86
Tier-based

• Two separate domains
• Standard Tier
• Any and all submissions would be accepted after
  a cursory examination of or other pro forma
  certification.
• The review process could be minimally
  labor-intensive, perhaps relying primarily on an
  automated check of author institutional
  affiliation, prior publication record, research
  grant status, or other related background and
  involve human labor primarily to adjudicate
  incomplete or ambiguous results of an automated
  pass.

87
Tier-based

• Upper Tier
• At some later point (which could vary from
  article to article, perhaps with no time limit),
  a much smaller set of articles would be selected
  for the full peer review process. The initial
  selection criteria for this smaller set could be
  any of a variety of impact measures, to be
  determined, and based explicitly on their prior
  widespread and systematic availability and
  citability e.g., reader nomination or rating,
  citation impact, usage statistics, editorial
  selection, ... .

Paul Ginsparg, Can Peer Review be Better
Focused?, Science Technology Libraries 22
No. 3/4 (In press). http//arxiv.org/blurb/pg02pr.
html
88
DISCLAIMER
OFF

• Presented for Your Consideration

89
FREEDOM OF IDEAS
http//www.nrm.org/exhibits/current/four-freedoms.
html
90
FUTURE OF IDEAS

• The explosion of innovation we have seen in
  the environment of the Internet was not conjured
  from some new, previously unimagined
  technological magic instead, it came from an
  ideal as old as the nation. Creativity flourished
  there because the Internet protected an
  innovation commons.

91
FUTURE OF IDEAS
The Internets very design built a neutral
platform upon which the widest range of creators
could experiment. The legal architecture
surrounding it protected this free space so that
culture and information the ideas of our
eracould flow freely and inspire an
unprecedented breadth of expression.

Lawrence Lessig, The Future of Ideas The Fate
of the Commons in a Connected World. (New York
Random House, c2001).
92

• ITS
• NOT
• ABOUT
• PUBLICATION
• ITS
• ABOUT
• IDEAS.

93
The Association of Learned and Professional
Society Publishers (ALPSP) SurveyAuthors and
Electronic Publishing

• Scholarly research communication has seen
  far-reaching developments in recent years.
• Most journals are now available online as well as
  in print, and numerous electronic-only journals
  have been launched
• the Internet opens up new ways for journals to
  operate.
• Authors have also become conscious of alternative
  ways to communicate their findings, and much has
  been written about what they ought to think.

94
ALPSP felt that it would be timely to discover
what they actually thought and what they actually
did. This survey aimed to discover the views of
academics, both as authors and as readers. Some
14,000 scholars were contacted across all
disciplines and all parts of the world, and
nearly 9 responded their detailed comments make
thought-provoking reading.
Alma Swan and Sheridan Brown. Authors and
Electronic Publishing The ALPSP Research Study
on Authors' and Readers Views of Electronic
Research Communication. (West Sussex, UK The
Association of Learned and Professional Society
Publishers, 2002). http//www.alpsp.org/pub5.ht
m
95
When asked to predict what would be the most
common form of quality control in five years
time, only a bare majority answered traditional
peer review.Fytton Rowland, The Peer-Review
Process, Learned Publishing 15 no. 4 (October
2002) 247-258.
Report version http//www.jisc.ac.uk/uploaded_doc
uments/rowland.pdf
96
FURTHERMORE

• 16 said that the referees would no longer be
  anonymous
• 27 said that traditional peer review would be
  supplemented by post-publication commentary
• 45 expected to see some changes in the
  peer-review system within the next five years

Fytton Rowland, The Peer-Review Process,
Learned Publishing 15 no. 4 (October 2002)
247-258.
Report version http//www.jisc.ac.uk/uploaded_doc
uments/rowland.pdf
97
Importance of the Peer Review Process
http//www.alpsp.org/pub5.ppt
98
What is Gray/Grey Literature ?

• Papers are often written to inform funding bodies
  about the results of research projects, to
  support grant applications, to inform rapidly a
  specific scientific community, to present
  preliminary results at conferences or as
  dissertations.
• Such material is disseminated quickly, often in
  limited numbers, before or without the formal
  publication process. Such documents are called
  non-conventional or grey literature.

http//www.kb.nl/infolev/eagle/what_is_gl.htm
99
The Value of Grey Literature Grey literature is
really a type of informal communication, which on
a scale of formality, fits in somewhere between
conversation and normal publication. A formal
publication may follow later but in many cases -
contrary to the common assumption - these papers
may not been made publicly available at all.
http//www.kb.nl/infolev/eagle/what_is_gl.htm
100
Nevertheless, grey publications may contain
comprehensive, concrete and up-to-date
information on research findings, and
investigations have shown, that even when grey
documents are published officially at a later
stage, detailed information on techniques,
methods, measured values and details of
experiments are frequently omitted. For these
details of importance for further research, the
non-conventional literature is then the first and
only source of information.
http//www.kb.nl/infolev/eagle/what_is_gl.htm
101
Veterinary Medicine

• 12 Major Veterinary Medicine Journals
• Overall, 6.38 of cited literature was Gray/Grey
  Literature
• The figures for individual journals ranged from
  about 2.5 to 10 gray/grey literature
• Research journals cited a higher percentage of
  Gray/Grey Literature than did Clinical titles
• William H. Weise and Nancy Pelzer, Bibliometric
  Study of Grey Literature in Core Veterinary
  Medicine Journals, Journal of the Medical
  Library Association 91 no. 4 (October 2003) In
  press.

102
Indexing and Abstracting Services

• SIGLE System for Information on Grey Literature
• National Technical Information Service (NTIS)
• PsychINFO (Psychological Abstracts)

103
SIGLESystem for Information on Grey Literature

• Grey literature documents covered by SIGLE are
  technical or research reports, preprints,
  committee reports, working papers, dissertations,
  conference papers,  discussion and policy papers,
  government reports, market surveys, etc.

http//www.kb.nl/infolev/eagle/frames.htm
104
SIGLESystem for Information on Grey Literature

• No. of Records Category
• 4,158 Aeronautics
• 17,044 Agriculture, plant veterinary sciences
• 17,668 Environmental pollution, protection
  control
• 256,657 Humanities, psychology social
  sciences
• 81,269 Biological medical sciences
• 25,089 Chemistry

http//www.kb.nl/infolev/eagle/frames.htm
105
NTIS (National Technical Information Service)
The NTIS Database produced by the National
Technical Information Service, is the preeminent
resource for accessing the latest U.S.
government-sponsored research and worldwide
scientific, technical, engineering, and
business-related information.
http//www.csa2.com/csa/factsheets/ntis.shtml

• NTIS Database provides bibliographic data and
  abstracts of unclassified and publicly available
  information from research reports, journal
  articles, data files, computer programs and audio
  visual products, from U.S. and non-U.S.
  governmental, organizational, and commercial
  sources

106
Subject Coverage
Administration and Management Aeronautics Aerodynamics Agriculture Behavior Society
Business Chemistry Communications Computer Science
Education Energy Engineering Environmental Sciences
Health Care International Trade Library Information Science Materials Sciences
Mathematical Sciences Natural Resources Earth Sciences Nuclear Science Physics
Regulations Technology Tele-communications Transportation
107
PsycINFO
PsycINFO provides access to international
literature in psychology and related disciplines.
Unrivaled in its depth of psychological coverage
and respected worldwide for its high quality, the
database is enriched with literature from an
array of disciplines related to psychology such
as psychiatry, education, business, medicine,
nursing, pharmacology, law, linguistics, and
social work.
http//www.csa2.com/csa/factsheets/psycinfo.shtml
PsycINFO includes psychological research and its
applications the database is of prime relevance
to many industries and research establishments
worldwide. The sources include over 1,400
professional journals, chapters, books, reports,
theses and dissertations, published
internationally.
108
Subject Coverage
Applied Psychology Communication Systems Developmental Psychology Educational Psychology
Experimental Psychology Personality Psychological and physical disorders Professional personnel issues
Physiological Psychology and Neuroscience Psychometrics And Statistics Social Psychology Treatment and Prevention
109
DATABASE COVERAGE AND SIZE
Database Coverage Size
SIGLE 1976 Present 781,410 records (November 2002)
NTIS 1964 - Present 2,168,400 records (October 2001)
PsycINFO 1872- Present 1,870,180 records (September 2002)
110
http//www.neci.nec.com/lawrence/papers/online-na
ture01/
111
Conference papers are typical gray/grey
literature!
112
EPrints are Gray/Grey Literature

• Daniela Luzi (1998) E-Print Archives a New
  Communication Pattern for Grey Literature,
  Interlending Document Supply 26 no. 3 (1998)
  130-139.

113
Gray/Grey Literature

• Its good enough,
• its smart enough,
• and
• doggone it, people use it!

With apologies to Stuart Smalley
114
http//www.openarchives.org/Register/BrowseSites.p
l
115
(No Transcript)
116
(No Transcript)
117
(No Transcript)
118
June 2003
http//software.eprints.org/
119
http//eprints.anu.edu.au/
120
http//caltechcstr.library.caltech.edu/
121
http//eprints.lub.lu.se/
122
http//ndltdpapers.dlib.vt.edu9090/
123
http//dspace.org/index.html
124
http//rocky.dlib.vt.edu/etdunion/cgi-bin/browse.
pl
125
http//www.ncstrl.org/
126
(No Transcript)
127
Quality Control in Scholarly Publishing on the
Web

• "Most of the high quality materials on the Web
  are not peer-reviewed and much of the
  peer-reviewed literature is of dubious quality.
• William Y. Arms, "What Are the Alternatives to
  Peer Review? Quality Control in Scholarly
  Publishing On The Web."
• Journal of Electronic Publishing, 8 no. 1 (August
  2002).
• http//www.press.umich.edu/jep/08-01/arms.html

128
http//www.update-software.com/Cochrane/MR000016.p
df
129
Cochrane Methodology Review

• Despite its widespread use and costs, little hard
  evidence exists that peer review improves the
  quality of published biomedical research.
• There had never even been any consensus on its
  aims and that it would be more appropriate to
  refer to it as competitive review.

Caroline White, Little Evidence for
Effectiveness of Scientific Peer Review, BMJ
326 (February 1, 2003) 241 http//bmj.com/cgi/rep
rint/326/7383/241/a.pdf
130
Cochrane Methodology Review

• On the basis of the current evidence, the
  practice of peer review is based on faith in its
  effects, rather than on facts,' state the
  authors, who call for large, government funded
  research programmes to test the effectiveness of
  the classic peer review system and investigate
  possible alternatives.

Caroline White, Little Evidence for
Effectiveness of Scientific Peer Review, BMJ
326 (February 1, 2003) 241 http//bmj.com/cgi/rep
rint/326/7383/241/a.pdf
131
Cochrane Methodology Review

• The use of peer-review is usually assumed to
  raise the quality of the end-product (i.e. the
  journal or scientific meeting) and to provide a
  mechanism for rational, fair and objective
  decision-making. However, these assumptions have
  rarely been tested.

Tom O. Jefferson, Phil Alderson, Frank Davidoff,
and Elizabeth Wager, Editorial Peer-review for
Improving the Quality of Reports of Biomedical
Studies. (Middle Way, Oxford Update Software
Ltd, 2003). http//www.update-software.com/Cochran
e/MR000016.pdf
132
Cochrane Methodology Review

• The available research has not clearly identified
  or assessed the impact of peer-review on the more
  important outcomes (importance, usefulness,
  relevance, and quality of published reports)
• Given the widespread use of peer-review and
  its importance, it is surprising that so little
  is known of its effects

Tom O. Jefferson, Phil Alderson,Frank Davidoff,
and Elizabeth Wager, Editorial Peer-review for
Improving the Quality of Reports of Biomedical
Studies. (Middle Way, Oxford Update Software
Ltd, 2003). http//www.update-software.com/Cochran
e/MR000016.pdf
133
Royal Society
http//www.biomedcentral.com/news/20030203/04/
134
UNCITEDNESS
David P. Hamilton, "Publishing by and for? --
the numbers, Science (New Series) 250 (4986)
(December 7 1990) 1331-1332. http//www.garfield.
library.upenn.edu/papers/hamilton1.html
135
David P. Hamilton,Research Papers Whos Uncited
Now?, Science (New Series) 251 (4989) (January
4, 1991) 25 http//garfield.library.upenn.edu/pap
ers/hamilton2.html
136
David P. Hamilton,Research Papers Whos Uncited
Now?, Science (New Series) 251 (4989) (January
4, 1991) 25 http//garfield.library.upenn.edu/pap
ers/hamilton2.html
137
REJECTED CITATION CLASSICS

• NOBEL PRIZE RESEARCH
• Severo Ochoa
• Polynucleotide phosphorylase
• Hans Krebs
• Citric acid cycle
• Rosalind Yalow
• Radioimmunoassay
• Harmut Michel
• Photosynthetic processes

Juan Miguel Campanario, Commentary On
Influential Books and Journal Articles
Initially Rejected Because of Negative Referees
Evaluations, Science Communication 16 no. 3
(March 1995) 306-325
138
PEER REVIEW PURPOSES
Peer review helps to ensure that published
research is
Important ? Original ?
Timely ? Technically-reliable ?
Internally consistent ? Well-presented ?
Benefited from guidance by experts ?
Carin M. Olson, Peer Review of the Biomedical
Literature, American Journal of Emergency
Medicine 8 no.4 (July 1990) 356-368.
139
FILTERING (1)

• UpStream / DownStream

Researchers look at certain types of
electronic publications because, despite being
tentative, may be relevant to their work.
Researchers are expected to do their own
downstream-filtering of relevant information,
which in the electronic world can be facilitated
by providing meta-information.
140
FILTERING (2)

• UpStream / DownStream

• Some have expressed the concern that having
  non-peer reviewed documents with peer-reviewed
  documents on the same server would contaminate
  the latter and compromise its quality
• Readers could have trouble in distinguishing the
  different sections
• Making non-peer-reviewed as well as
  peer-reviewed material will confuse both
  scientists and the public .

141
FILTERING (3)

• UpStream / DownStream
• However, this perhaps belittles the ability of
  scientists to recognize different levels of
  evidence and to be able to interpret quality
  labels that could make clear that certain
  materials is non-peer-reviewed content after all
  this is the age of transparency rather than
  paternalism .

Gunther Essenbach, The Impact of Preprint
Servers and Electronic Publishing on Biomedical
Research, Current Opinion in Immunology 12 no.
5 (October 2000) 499-503.
http//yi.com/home/EysenbachGunther/scans/Eysenbac
h2000e_CurrOpImmunol_preprint_servers.pdf
142
INVISIBLE HAND OF CLASSICAL PEER REVIEW

• The refereed journal literature needs to be
  freed from both paper and its costs, but not from
  peer review, whose invisible hand is what
  maintains its quality.
• Stevan Harnad

http//www.presidentmoron.com
143
INVISIBLE HAND OF CLASSICAL PEER REVIEW
144
INVISIBLE HAND OF CLASSICAL PEER REVIEW
Human nature being what it is, it cannot be
altogether relied upon to police itself.
Individual exceptions there may be, but to treat
them as the rule would be to underestimate the
degree to which our potential unruliness is
vetted by collective constraints, implemented
formally.
Stevan Harnad, The Invisible Hand of Peer
Review, Exploit Interactive no. 5 (April 2000).
http//www.exploit-lib.org/issue5/peer-review/
145
INVISIBLE HAND OF CLASSICAL PEER REVIEW
The system is not perfect, but it is what has
vouchsafed us our refereed journal literature to
date, such as it is, and so far no one has
demonstrated any viable alternative to having
experts judge the work of their peers, let alone
one that is at least as effective in maintaining
the quality of the literature as the present
imperfect one is.
146
INVISIBLE HAND OF CLASSICAL PEER REVIEW
Remove that invisible constraint -- let the
authors be answerable to no one but the general
users of the Archive arXiv.org (or even its
self-appointed "commentators") -- and watch human
nature take its natural course, standards eroding
as the Archive devolves toward the canonical
state of unconstrained postings the free-for-all
chat-groups of Usenet , that Global Graffiti
Board for Trivial Pursuit -- until someone
re-invents peer review and quality control.
Stevan Harnad, The Invisible Hand of Peer
Review, Exploit Interactive no. 5 (April
2000). http//www.exploit-lib.org/issue5/peer-revi
ew/
147
INVISIBLE HANDS
148
INVISIBLE HANDS

• Personal reputation
• Institutional reputation
• Pride
• Self-respect
• Professional respect
• Peer pressure

• Critical Peer Response
• Invisible College
• Self-Archiving-Process-Itself
• Open access
• Common Sense
• Self-correcting dynamics

149
RECOMMENDATIONSWorkshop on the Open Archives
Initiative (OAI)and Peer Review Journals in
Europe, CERN, Geneva Switzerland,March 22-24,
2001

• It was also generally believed that the
  electronic environment allows for novel
  approaches to accord a stamp of quality to
  scholarly works.

Alison Buckholtz, Raf Dekeyser, Melissa
Hagemann, Thomas Krichel, and Herbert Van de
Sompel, Open Access Restoring Scientific
Communication to Its Rightful Owners, European
Science Foundation Policy Briefing 21 (April
2003) 1-8.
http//www.arl.org/sparc/SPB21_OAI.pdf
150
Examples of new metrics that can be extracted
from a fully electronic communication system are

• Usage counts of a work
• Automatically extracted citation information with
  a scope beyond the ISI- core journals
• Amount of discussion generated by a paper
  submitted in a system with open peer review and
  peer comment
• Etc.

Alison Buckholtz, Raf Dekeyser, Melissa
Hagemann, Thomas Krichel, and Herbert Van de
Sompel, Open Access Restoring Scientific
Communication to Its Rightful Owners, European
Science Foundation Policy Briefing 21 (April
2003) 1-8.
http//www.arl.org/sparc/SPB21_OAI.pdf
151
Scientific Publishing as Rhetoric

• The problems with peer review become evident
  once the fact that science has a rhetorical
  element is accepted.
• On the one hand, the traditional mode of peer
  review obscures the problems of reference and the
  rhetorical dimension of science. The rhetorical
  process which is at the heart of science and peer
  review conveniently disappears with the final
  publication of the manuscript. In its place is an
  ideal typical representation (the scientific
  paper) of the realist assumptions about empirical
  reference. All the academic world sees is a
  polished manuscript where the personal
  involvement of the researcher and reviewers has
  been systematically eliminated.

Mike Sosteric, Interactive Peer Review A
Research Note, Electronic Journal of Sociology 2
no. 1 (1996). http//socserv.socsci.mcmaster.ca/EJ
S/vol002.001/SostericNote.vol002.001.html
152
IDEAL SPEECH SITUATION
Jürgen Habermas

• A theoretical construct that describes the ideal
  type of interpersonal interaction that should
  exist in a rhetorical situation.

153

• IDEAL SPEECH SITUATION
• the ideal speech situation permits each
  interlocutor an equal opportunity to initiate
  speech
• there is mutual understanding between
  interlocutors
• there is space for clarification
• all interlocutors are equally free to use of any
  speech act
• there is equal power over the exchange.
• Applied in the context of peer, the Ideal Speech
  Situation would permit unimpeded authorial
  initiative, endless rounds of give and take,
  and unchecked openness among authors, editors,
  and referees.

Mike Sosteric, Interactive Peer Review A
Research Note, Electronic Journal of Sociology 2
no. 1 (1996). http//socserv.socsci.mcmaster.ca/EJ
S/vol002.001/SostericNote.vol002.001.html
154
CONTINUA

• Continuum of PUBLICATION
• (Scholarly Skywriting)
• WEAK
• MEDIUM
• Continuum of REVIEW
• (Scholarly Skyreading)

STRONG
155
OPEN ACCESS and OPEN RETRIEVAL without OPEN
USE

• Incongruent
• Contradictory

• Ironic
• Paradoxical

• Cognitively Dissonant

156
Three-Legged Stool

• ACCESS -- RETRIEVAL -- USE

157
(No Transcript)
158
ACCESS

• OPEN ACCESS INITIATIVES

159
RETRIEVAL

• OPEN ARCHIVES INITIATIVE FOR METADATA HARVESTING

160
USE

• OPEN SCHOLARSHIP

161

• ACCESS -- RETRIEVAL -- USE

162
Lots of Alternative ModelsProvide Sensible
Solutions
163
Four-Legged Stool

• ACCESS -- RETRIEVAL -- USE -- NAVIGATION

164
INFORMATION OVERLOAD
165
OAIster

• A search engine for freely available,
  difficult-to-access, academically-oriented
  digital resources that are OAI -compliant

http//oaister.umdl.umich.edu/
166
(No Transcript)
167
University of Michigan Digital Library
Production Service

• institutional repositories
• departmental repositories
• e-Journal collections
• technical reports
• dissertations and theses

• discipline eprint collections
• working papers
• Internet resources
• audio
• video
• images

168
(No Transcript)
169
Cognitive Psychology
170
(No Transcript)
171
Open Archives Initiative Protocol for Metadata
Harvesting(OAI-PMH)
ADD

METADATA ELEMENT for QUALITY
172
Within the framework of OAI, there is a need for
a new protocol for certification. There was
strong support for the extension of the usage of
the OAI protocol beyond discovery-related metadata
. Given the focus of the 1st OAI workshop on
peer review, concrete actions were suggested to
address the exchange of certification-related meta
data using the OAI protocol in a trusted
environment.
Alison Buckholtz, Raf Dekeyser, Melissa
Hagemann, Thomas Krichel, and Herbert Van de
Sompel, Open Access Restoring Scientific
Communication to Its Rightful Owners, European
Science Foundation Policy Briefing 21 (April
2003) 1-8.
http//www.arl.org/sparc/SPB21_OAI.pdf
173
QUALITY METADATA (1)
ltoai-qualitygt ltcategorygtinternallt/categorygt ltproc
essgt peer review lt/processgt ltorganizationgt CERN lt/
organizationgt ltpoliciesgt http//www.cern.ch/polici
es/review.html lt/policiesgt lt/oai-qualitygt
William Y. Arms, Quality Control in Scholarly
Publishing On The Web. What Are the Alternatives
to Peer Review? PowerPoint presentation given at
Workshop on the Open Archives Initiatives (OAI)
and Peer Review Journals in Europe, March 22-24,
2003, CERN, Geneva, Switzerland http//www.cs.corn
ell.edu/wya/papers/CERN-2001.ppt
174
QUALITY METADATA (2)
ltoai-qualitygt ltcategorygtinternallt/categorygt ltrati
nggt lt/ratinggt ltorganizationgt 42 lt/organizati
ongt ltpoliciesgt http//www.cern.ch/policies/review.
html lt/policiesgt lt/oai-qualitygt
175
(No Transcript)
176
CERTIFICATION SERVICES
Faculty of 1,000,000
177
CERTICATION SERVICES

• New roles for Indexing and Abstracting Services
• Expanded Role for Learned and Professional
  Societies
• Establishment of Formal/Commercial Reviewing
  Services

178
PEER REVIEW STRENGTHS
The underlying strength of peer review isthe
concerted effort by large numbers of researchers
and scholars who work to assure that valid and
valuable works are published and conversely to
assure that invalid or non-valuable works are not
published .
Anne C. Weller, Editorial Peer Review Its
Strengths and Weaknesses. (Medford, NJ
Information Today, 2001).
179
Faculty of 1000 / BioMed Central

• BioMed Central (biomedcentral.com) publishes
  Faculty of 1000 (F1000), the leading literature
  evaluation service and new online research tool
  that highlights the most interesting papers in
  biology, based on the recommendations of over
  1000 leading scientists. F1000 is managed by
  scientists for scientists . and provides a
  rapidly updated consensus map of the important
  papers and trends across biology.

www.facultyof1000.com
180
Faculty of 1000 / BioMed Central

• Among its many benefits, F1000
• systematically organizes and evaluates the mass
  of information within scientific literature
• provides scientists with a continuously updated
  insider's guide to the most important papers
  within any given field of research
• highlights papers on the basis of their
  scientific merit rather than the journal in which
  they appear
• offers the researcher a consensus of
  recommendations from well over 1000 leading
  scientists and,
• offers an immediate rating of individual papers
  by the authors' peers, and an important
  complement to the indirect assessment provided by
  the journal impact factor.

181
Faculty of 1000 / BioMed Central

• Within the F1000, the entire field of biology
  is divided into 16 subject areas
  (Faculties)(e.g., Biochemistry,, Cell
  Biology, Microbiology). Each Faculty is
  subdivided into three (3) to twelve (12)
  Sections, (e.g., Biochemistry Biocatalysis,
  Molecular evolution, Protein folding), with each
  section comprised of between 10 to 50 faculty
  members. F1000 seeks to invite the best
  internationally known scientists in each
  represented field and to involve both experienced
  and younger investigators.

182
Peer review is a quality-control and
certification (QC/C) filter necessitated by the
vast scale of learned research today. Without it,
no one would know where to start reading in the
welter of new work reported every day, nor what
was worth reading, and believing, and trying to
build ones own further research upon.

• Stevan Harnad, Free at Last The Future of
  Peer-Reviewed Journals,
• D-Lib Magazine 5 no. 12 (December 1999)
• http//www.dlib.org/dlib/december99/12contents.htm
  l

183
SEIZE THE E!

• Embrace the potential of the digital environment
  to facilitate access, retrieval, use, and
  navigation of electronic scholarship.

184
OPEN NAVIGATION
185
New Age Navigation

• Innovative Interfaces for Electronic Journals
• Gerry McKiernan
• The Serials Librarian
• Fall 2003

186
SUMMARY. While it is typical for electronic
journals to offer conventional search features
similar to those provided by electronic
databases, a select number of e-journals have
also made available higher-level access options
as well. In this article, we review several novel
technologies and implementations that creatively
exploit the inherent potential of the digital
environment to further facilitate use of
e-collections.

• Gerry McKiernan, New Age Navigation Innovative
  Interfaces for Electronic Journals,
• The Serials Librarian, Fall 2003.

http//www.coleonline.us/serialslibrarian/
187
http//www.highwire.org
188
Topic Map
189
Topic Map
190
Topic Map
191
Topic Map
192
Topic Map
Secondary Screen
193
Topic Map
194
Topic Map
TopicMap is based on the Hyperbolic Tree SDK for
Java, licensed from Inxight Software, Inc., a
spin-off company from Xerox PARC, and leading
provider of Unstructured Data Management
solutions for accessing, analyzing and delivering
information.
http//www.inxight.com
195
Semio
Automated categorization software technology
http//www.entrieva.com/
196
Kohonen Self-Organizing Maps
http//websom.hut.fi/websom/milliondemo/html/root.
html
197
Kohonen Self-Organizing Maps
http//websom.hut.fi/websom/milliondemo/html/1_cx5
.html
198
Kohonen Self-Organizing Maps
http//websom.hut.fi/websom/milliondemo/html/2_gx1
0.html
199
Kohonen Self-Organizing Maps
http//websom.hut.fi/websom/milliondemo/getnd.cgi?
32323
200
Kohonen Self-Organizing Maps
http//websom.hut.fi/websom/cgi-bin/getfile.cgi/co
mp.ai/39340
201
http//www.springer.de/books/toc/3540679219-c.pdf
202
There are some excuses, but at the bottom it
will be seen to be the sluggishness of human
nature and its superstitious cleavage to old
habits.

• Stevan Harnad, Free at Last The Future of
  Peer-Reviewed Journals,
• D-Lib Magazine 5 no. 12 (December 1999)
• http//www.dlib.org/dlib/december99/12contents.htm
  l

203
In the digital world, the evaluation process
stands ready to be reinvented in a clear,
rational way by the relevant research communities
themselves.

• Jean-Claude Guédon,
• In Oldenburgs Long Shadow Librarians, Research
  Scientists, Publishers, and the Control of
  Scientific Publishing.
• (Washington, D.C. Association of Research
  Libraries, 2001), 54.

http//www.arl.org/arl/proceedings/138/guedon.html

204
lt/ENDQUOTEgt

• The Medium is the Message
• And
• the Method.
• With apologies to Marshall McLuhen

205
THANK YOU FOR YOUR ATTENTION!
206
Alternative Peer ReviewQuality Management for
21st Century Scholarship

• Gerry McKiernan
• Science and Technology Librarian
• and Bibliographer
• Iowa State University Library
• Ames IA
• USA

gerrymck_at_iastate.edu
207
OPEN MIND
208
REVISED VERSION 1.01

• July 27, 2003
• 1130 AM