Alternative Peer Review: Quality Management for 21st Century Scholarship - PowerPoint PPT Presentation

Loading...

PPT – Alternative Peer Review: Quality Management for 21st Century Scholarship PowerPoint presentation | free to download - id: 613f08-NzNhM



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Alternative Peer Review: Quality Management for 21st Century Scholarship

Description:

Alternative Peer Review: Quality Management for 21st Century Scholarship http://www.public.iastate.edu/~gerrymck/APR.ppt Gerry McKiernan Science and Technology Librarian – PowerPoint PPT presentation

Number of Views:919
Avg rating:3.0/5.0
Slides: 209
Provided by: Gerr157
Learn more at: http://www.ils.unc.edu
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Alternative Peer Review: Quality Management for 21st Century Scholarship


1
Alternative Peer ReviewQuality Management for
21st Century Scholarship
http//www.public.iastate.edu/gerrymck/APR.ppt
  • Gerry McKiernan
  • Science and Technology Librarian
  • and Bibliographer
  • Iowa State University Library
  • Ames IA
  • USA

gerrymck_at_iastate.edu
2
Workshop on Peer Review in the Age of Open
Archives
  • International School for Advanced Studies
  • Interdisciplinary Laboratory
  • Trieste, Italy
  • May 23-24, 2003

3
THANK YOU!
  • Workshop Advisory Board (Marco Fabbrichesi
    (INFN/SISSA Italy), Stevan Harnad (University of
    Southampton, UK), Stefano Mizzaro (University of
    Udine, Italy) and Corrado Pettenati (CERN
    Library, Geneva, Switzerland)
  • Iowa State University, Faculty Senate, Committee
    on Recognition and Development
  • European Commission
  • Iowa State University Library
  • Heike Kross, Ph.D.

4
DISCLAIMER (1) The screen prints selected for
this presentation are for educational purposes
and their inclusion does not constitute an
endorsement of an associated product, service,
place, or institution.
5
DISCLAIMER (2) The views and opinions expressed
in this presentation are those of the presenter
and do not constitute an endorsement by Iowa
State University or its Library.
6
NOTICE
No editors, authors, or referees were harmed in
the preparation of this presentation.
7
http//www.sciencecartoonsplus.com/
8
Giuseppe De Nittis (1846-1884) The Macchiaioli /
Italian Impressionists
9
Campo di Biche (1875)
10
PEER REVIEW DEFINITION
  • Peer review is the assessment by an expert of
    material submitted for publication.

Carin M. Olson, Peer Review of the Biomedical
Literature, American Journal of Emergency
Medicine 8 no.4 (July 1990) 356-358.
11
PEER REVIEW PURPOSES
Peer review helps to ensure that published
research is
Important Original
Timely Technically-reliable
Internally-consistent Well-presented
Benefited from guidance by experts
Carin M. Olson, Peer Review of the Biomedical
Literature, American Journal of Emergency
Medicine 8 no.4 (July 1990) 356-358.
12
PEER REVIEW STRENGTHS
The underlying strength of peer review isthe
concerted effort by large numbers of researchers
and scholars who work to assure that valid and
valuable works are published and conversely to
assure that invalid or non-valuable works are not
published .
Anne C. Weller, Editorial Peer Review Its
Strengths and Weaknesses. (Medford, NJ
Information Today, 2001).
13
Houston, We Have a Problem!
14
PEER REVIEW PROBLEMS
  • Subjectivity
  • Bias
  • Abuse
  • Detecting defects
  • Fraud and Misconduct
  • Delay

Fytton Rowland, The Peer-Review
Process, Learned Publishing 15 no. 4 (October
2002) 247-258.
Report version http//www.jisc.ac.uk/uploaded_doc
uments/rowland.pdf
15
SUBJECTIVITY
  • Summary rejections by editor without sending the
    paper to referees
  • Choice of referees by the editor (choosing for
    example, a known harsh referee for a paper the
    editor wishes to see rejected)

16
BIAS
  • Discrimination against authors because of their
    nationality, native language, gender or host
    institution
  • Situations where author and referee are
    competitors in some sense, or belong to warring
    schools of thought

17
ABUSE
  • Too many articles out of one piece of research,
    or duplicate publication
  • Intellectual theft omission or downgrading of
    junior staff by senior authors
  • Plagiarism (stealing others yet unpublished work
    that has been sent for review)
  • Delaying publication of potentially competing
    research

18
DETECTING DEFECTS
  • Identification of factual errors within submission

19
FRAUD and MISCONDUCT
  • Fabrication of results
  • Falsification of data
  • False claim of authorship for results

20
DELAY

There is much muttering about publication delay,
a real enough problem, especially in paper
publication, but peer review itself is often
responsible for as much of the delay as the paper
publication and distribution process
itself. Stevan Harnad
Stevan Harnad, Implementing Peer Review on the
Net Scientific Quality Control in Scholarly
Electronic Journals, in Scholarly Publication
The Electronic Frontier, edited by Robin P. Peek
and Gregory B. Newby (Cambridge MA MIT Press,
1996).
http//www.ecs.soton.ac.uk/harnad/Papers/Harnad/h
arnad96.peer.review.html
21
http//www-marketing.wharton.upenn.edu/ideas/pdf/p
eerpaper.pdf
22
Peer review is slow, expensive, profligate of
academic time, highly subjective, prone to bias,
easily abused, poor at detecting gross defects,
and almost useless in detecting fraud.
Richard SmithEditor, BMJ
  • Richard Smith, Opening Up BMJ Peer Review,
  • BMJ 318 (7175) (January 2 1999) 4-5

23
Stephen Lock, A Difficult Balance Editorial Peer
Review in Medicine (Philadelphia, PA ISI Press,
1986).
24
(No Transcript)
25
EXAMPLE
  • Jan Hendrik Schön

26
-
http//archive.salon.com/tech/feature/2002/09/16/p
hysics/
27
Science
Nature
http//physicsweb.org/article/news/6/9/15/
28
http//agenda.cern.ch/fullAgenda.php?idaa01193
29
RECOMMENDATIONSWorkshop on the Open Archives
Initiative (OAI)and Peer Review Journals in
Europe, CERN, Geneva Switzerland, March 22-24,
2001
  • The participants were unanimous in their belief
    that the certification of scholarly work remains
    a fundamental part of a system for scholarly
    communication.
  • It was also generally believed that the
    electronic environment allows for novel
    approaches to accord a stamp of quality to
    scholarly works.

Alison Buckholtz, Raf Dekeyser, Melissa
Hagemann, Thomas Krichel, and Herbert Van de
Sompel, Open Access Restoring Scientific
Communication to Its Rightful Owners, European
Science Foundation Policy Briefing 21 (April
2003) 1-8.
http//www.arl.org/sparc/SPB21_OAI.pdf
30
Let us be imaginative in exploring the
remarkable possibilities of this brave new
medium.
  • Stevan Harnad, Implementing Peer Review on the
    Net Scientific Quality Control in Scholarly
    Electronic Journals, in Scholarly Publication
    The Electronic Frontier, edited by Robin P. Peek
    and Gregory B. Newby (Cambridge MA MIT Press,
    1996).

http//www.ecs.soton.ac.uk/harnad/Papers/Harnad/h
arnad96.peer.review.html
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
Let us be more imaginative in exploring the
remarkable possibilities of this brave new
medium. With Apologies to Stevan Harnad
36
(No Transcript)
37
TM
http//lockss.stanford.edu/
38
LOCKSS
  • For centuries libraries and publishers have had
    stable roles publishers produced information
    libraries kept it safe for reader access. There
    is no fundamental reason for the online
    environment to force institutions to abandon
    these roles.
  • The LOCKSS model capitalizes on the traditional
    roles of libraries and publishers.  LOCKSS
    creates low-cost, persistent digital "caches" of
    authoritative versions of http-delivered
    content. 

39
LOCKSS
  • The LOCKSS software enables institutions to
    locally collect, store, preserve, and archive
    authorized content thus safeguarding their
    community's access to that content. 
  • The LOCKSS model enforces the publisher's access
    control systems and, for many publishers, does no
    harm to their business models.

40
LAMPSS
Lots of Alternative Models Provide Sensible
Solutions
41
(No Transcript)
42
DISCLAIMER
ON
  • The alternative peer review models profiled are
    for informational and educational purposes only
    and do not necessarily constitute an endorsement.

43
ALTERNATIVE PEER REVIEW
  • Institution-based
  • Citation-based
  • Index-based
  • Metadata-based
  • Computer-assisted
  • NO Peer Review
  • Moderator-based
  • Tier-based
  • Neo-Classical
  • Certification-based
  • Open Peer Review
  • Commentary-based
  • Collaborately-filtered

44
NEO-CLASSICAL PEER REVIEW
45
Neo-Classical Peer Review
46
Neo-Classical Peer Review
47
Neo-Classical Peer Review
48
CERTIFICATION-BASED
49
Certification-Based
  • The process of pre-publication peer review
    could be improved and become a more reliable
    indicator of manuscript quality if reviewers were
    trained in, and subsequently applied
    systematically, critical skills and use of a
    hierarchy of evidence to classify submitted
    articles being reviewed.

Stephen Pritchard , Peer Review - a Proposal for
Change, Paper presented at Thinking Globally -
Acting Locally Medical Libraries at the Turn of
an Era, 8th European Conference of Health and
Medical Libraries, September 16-21, 2002,
Deutsche Zentralbibliothek für Medizin, Köln,
Germany. http//www.zbmed.de/eahil2002/abstracts/p
ritchard.pdf
50
OPEN PEER REVIEW
51
Open Peer Review
  • IDENTIFICATION OF REVIEWERS / SIGNED REVIEWS
  • BMJ
  • bmj.com
  • BioMed Central
  • biomedcentral.com
  • eMJA (Medical Journal of Australia)
  • www.mja.com.au/public/information/project.html

52
COMMENTARY-BASED
53
Commentary-based
  • Readers can comment before and/or after classic
    peer review, or instead of classic peer review
  • Electronic Transactions on Artificial
    Intelligence
  • (www.etaij.org)
  • OPEN REVIEW / REFEERING
  • Journal of Interactive Media in Education
  • (www-jime.open.ac.uk)
  • PRE- AND POST- COMMENTARY
  • Psycoloquy
  • (psycprints.ecs.soton.ac.uk)
  • POST PEER REVIEW COMMENTARY)

54
Electronic Transactions on Artificial
Intelligence
55
Journal of Interactive Media in Education
56
http//psycprints.ecs.soton.ac.uk/
57
(No Transcript)
58
(No Transcript)
59
COLLABORATIVELY-FILTERED
60
Collaboratively-Filtered
  • DEFINITION Guiding people's choices of what
    to read, what to look at, what to watch, what to
    listen to (the filtering part) and doing that
    guidance based on information gathered from some
    other people (the collaborative part)."
  • Paul Resnick

http//www.cni.org/Hforums/cni-announce/1996/0031.
html
61
ResearchIndex / CiteSeer
http//www.researchindex.com
62
INSTITUTION-BASED
63
Institution-Based
  • INSTITUTIONAL INITIATIVES
  • DSpace (MIT)
  • www.dspace.org
  • eScholarship (University of California)
  • escholarship.cdlib.org
  • Glasgow ePrint Service (University of Glasgow)
  • eprints.lib.gla.ac.uk

64
CITATION-BASED
65
Citation-Based
  • Citations to Open Access / OAI-compliant
  • documents are indicators of document importance

http//citebase.eprints.org/
66
(No Transcript)
67
(No Transcript)
68
(No Transcript)
69
(No Transcript)
70
INDEX-BASED
71
Index-Based
  • INDEXING OF EPRINTS BY COMMERICAL ABSTRACTING AND
    INDEXING SERVICE
  • Chemical Abstracts (American Chemical Society)
    (CAS) indexes select appropriate e-prints from
    the arXiv.org eprint server as well from the
    Chemical Preprint Service (Elsevier)
  • Its selection criteria for this kind of
    electronic document are essentially the same as
    for the traditional printed documents they must
    report new information of chemical or
    chemistry-related interest and must be original
    publications. Also, the electronic publication
    must be publicly available and have some relative
    permanence .
  • Eric Shively / Chemical Abstracts Service

72
Index-Based
  • CAB Abstracts doesnt currently include Eprints
    or Preprints, but we are looking at the
    implications and possible mechanisms for
    accessing and indexing Eprints and/or Preprints
    related to the applied life sciences.
  • Tracy Shaw / CAB International

73
METADATA-BASED
74
Metadata-Based
ltoai-qualitygt ltcategorygtinternallt/categorygt ltproc
essgt peer review lt/processgt ltorganizationgt CERN lt/
organizationgt ltpoliciesgt http//www.cern.ch/polici
es/review.html lt/policiesgt lt/oai-qualitygt
William Y. Arms, Quality Control in Scholarly
Publishing On The Web. What Are the Alternatives
to Peer Review? PowerPoint presentation given at
the Workshop on the Open Archives Initiatives
(OAI) and Peer Review Journals in Europe, March
22-24, 2003, CERN, Geneva, Switzerland. http//www
.cs.cornell.edu/wya/papers/CERN-2001.ppt
75
COMPUTER-ASSISTED
76
Computer-Assisted (1)
  • SOFTWARE THAT ASSISTS IN THE EVALUATION OF A
    SUBMITTED MANUSCRIPT
  • A Software Program to Aid in Peer Review
  • Alvar Loria and Gladys Faba
  • Objective To characterize a personal
    computer-based software program developed as an
    aid to peer review of medical papers.
  • Design The software is a Windows-based
    application that records automatically a numeric
    score to a series of questions related to 8
    sections of scientific papers (introduction,
    methods, results, and discussion, plus 4 other
    sections). The questions and sections vary
    according to type of paper (original reports,
    case reports, or reviews), and the final output
    is a score with a maximum of 100 for a "perfect"
    paper. The software was tested using a single
    reviewer to judge 289 papers (169 original
    reports, 50 case reports, and 70 reviews) from 44
    Mexican medical journals. All statistical
    analysis of scores were done with nonparametric
    tests.

77
Computer-Assisted (2)
  • Results The paper scores ranged from 29 to 97
    with slightly higher median and less dispersion
    of scores for reviews as compared with original
    reports and case reports, but these differences
    did not reach significance. Two observations
    suggest that the software operated reasonably
    well a) there were some differences in the
    section scores by type of paper that agreed well
    with differences in their complexity b) the
    journal scores showed an association with their
    number of original papers and their percentage of
    original papers (Kruskal-Wallis test, P.06 and
    0.07, respectively).
  • Conclusions The software operated reasonably
    well when used to compare the relative quality of
    289 papers. The validity of the program is
    restricted in this study to the experience of 1
    reviewer. An analysis of the raw scores helped in
    detecting some ambiguous and redundant questions
    that have been modified in an improved version.
    The program has a potential as a training tool
    for inexperienced reviewers or as a scorekeeper
    for experienced peer reviewers.

Alvar Loria and Gladys Faba, A Software Program
to Aid in Peer Review, Abstract of paper
presented at the Third International Congress on
Biomedical Peer Review and Global Communications,
September 18-20, 1997, Prague, Czech Republic.
http//www.ama-assn.org/public/peer/arev.htm
78
NO PEER REVIEW
79
NO Peer Review
http//xxx.arXiv.cornell.edu
80
MODERATOR-BASED
81
Moderator-based (1)
  • The intent of this model is to allow the widest
    range of scientific manuscripts to be archived,
    searched, and distributed electronically at the
    lowest possible cost.
  • This would be accomplished through very minimal
    filtering and subsequent placement of eprints on
    a non-commercial archival server by a
    subject-specific Moderator appointed by a society
    (or consortia of societies).
  • A society-appointed Editorial Board (with
    double-blind peer review approved by the
    non-profit Peer Review Inc. organization) would
    then the identify the most important materials
    from among these archived items, and the stamp of
    approval for these items would be included in a
    secondary Virtual Collection.

82
Moderator-based (2)
  • There are no direct submissions to the Editorial
    Board manuscripts would be directed to the
    Editorial Board in one of three ways
  • 1. nominated by the eprint Moderator upon receipt
    for the archival server,
  • 2. notification sent to the Editorial Board when
    a threshold number of hits are generated by any
    one manuscript on the archive server, and
  • 3. nominated by readers of material from the
    archive this process requires a letter of
    support outlining the importance of the work to
    the Editorial Board.

83
Moderator-based (3)
  • The Virtual Collection could be produced as a
    variety of products
  • enhanced abstracts
  • email threads (with comments)
  • virtual reviews of sub-disciplines
  • SDIs (selective dissemination of information)
    current awareness tools
  • This process
  • reduces the load on the Editorial Boards, which
    results in a faster review process
    differentiates those items worthy of higher
    recognition from those worthy of archiving,
    making it easier for a reader to filter material,
    based upon a society and discipline authority
    (rather than commercial reasoning)
  • provides for search/browse/sdi from the Virtual
    Collection for filtered info, reducing this more
    expensive option for only those items recognized
    as of the highest quality.

84
3
2
1
David Stern, The eprint Moderator Model,
Newsletter on Serials Pricing Issues no. 214
(February 8, 1999). http//www.lib.unc.edu/prices/
1999/PRIC214.HTML214.5
85
TIER-BASED
86
Tier-based
  • Two separate domains
  • Standard Tier
  • Any and all submissions would be accepted after
    a cursory examination of or other pro forma
    certification.
  • The review process could be minimally
    labor-intensive, perhaps relying primarily on an
    automated check of author institutional
    affiliation, prior publication record, research
    grant status, or other related background and
    involve human labor primarily to adjudicate
    incomplete or ambiguous results of an automated
    pass.

87
Tier-based
  • Upper Tier
  • At some later point (which could vary from
    article to article, perhaps with no time limit),
    a much smaller set of articles would be selected
    for the full peer review process. The initial
    selection criteria for this smaller set could be
    any of a variety of impact measures, to be
    determined, and based explicitly on their prior
    widespread and systematic availability and
    citability e.g., reader nomination or rating,
    citation impact, usage statistics, editorial
    selection, ... .

Paul Ginsparg, Can Peer Review be Better
Focused?, Science Technology Libraries 22
No. 3/4 (In press). http//arxiv.org/blurb/pg02pr.
html
88
DISCLAIMER
OFF
  • Presented for Your Consideration

89
FREEDOM OF IDEAS
http//www.nrm.org/exhibits/current/four-freedoms.
html
90
FUTURE OF IDEAS
  • The explosion of innovation we have seen in
    the environment of the Internet was not conjured
    from some new, previously unimagined
    technological magic instead, it came from an
    ideal as old as the nation. Creativity flourished
    there because the Internet protected an
    innovation commons.

91
FUTURE OF IDEAS
The Internets very design built a neutral
platform upon which the widest range of creators
could experiment. The legal architecture
surrounding it protected this free space so that
culture and information the ideas of our
eracould flow freely and inspire an
unprecedented breadth of expression.

Lawrence Lessig, The Future of Ideas The Fate
of the Commons in a Connected World. (New York
Random House, c2001).
92
  • ITS
  • NOT
  • ABOUT
  • PUBLICATION
  • ITS
  • ABOUT
  • IDEAS.

93
The Association of Learned and Professional
Society Publishers (ALPSP) SurveyAuthors and
Electronic Publishing
  • Scholarly research communication has seen
    far-reaching developments in recent years.
  • Most journals are now available online as well as
    in print, and numerous electronic-only journals
    have been launched
  • the Internet opens up new ways for journals to
    operate.
  • Authors have also become conscious of alternative
    ways to communicate their findings, and much has
    been written about what they ought to think.

94
ALPSP felt that it would be timely to discover
what they actually thought and what they actually
did. This survey aimed to discover the views of
academics, both as authors and as readers. Some
14,000 scholars were contacted across all
disciplines and all parts of the world, and
nearly 9 responded their detailed comments make
thought-provoking reading.
Alma Swan and Sheridan Brown. Authors and
Electronic Publishing The ALPSP Research Study
on Authors' and Readers Views of Electronic
Research Communication. (West Sussex, UK The
Association of Learned and Professional Society
Publishers, 2002). http//www.alpsp.org/pub5.ht
m
95
When asked to predict what would be the most
common form of quality control in five years
time, only a bare majority answered traditional
peer review.Fytton Rowland, The Peer-Review
Process, Learned Publishing 15 no. 4 (October
2002) 247-258.
Report version http//www.jisc.ac.uk/uploaded_doc
uments/rowland.pdf
96
FURTHERMORE
  • 16 said that the referees would no longer be
    anonymous
  • 27 said that traditional peer review would be
    supplemented by post-publication commentary
  • 45 expected to see some changes in the
    peer-review system within the next five years

Fytton Rowland, The Peer-Review Process,
Learned Publishing 15 no. 4 (October 2002)
247-258.
Report version http//www.jisc.ac.uk/uploaded_doc
uments/rowland.pdf
97
Importance of the Peer Review Process
http//www.alpsp.org/pub5.ppt
98
What is Gray/Grey Literature ?
  • Papers are often written to inform funding bodies
    about the results of research projects, to
    support grant applications, to inform rapidly a
    specific scientific community, to present
    preliminary results at conferences or as
    dissertations.
  • Such material is disseminated quickly, often in
    limited numbers, before or without the formal
    publication process. Such documents are called
    non-conventional or grey literature.

http//www.kb.nl/infolev/eagle/what_is_gl.htm
99
The Value of Grey Literature Grey literature is
really a type of informal communication, which on
a scale of formality, fits in somewhere between
conversation and normal publication. A formal
publication may follow later but in many cases -
contrary to the common assumption - these papers
may not been made publicly available at all.
http//www.kb.nl/infolev/eagle/what_is_gl.htm
100
Nevertheless, grey publications may contain
comprehensive, concrete and up-to-date
information on research findings, and
investigations have shown, that even when grey
documents are published officially at a later
stage, detailed information on techniques,
methods, measured values and details of
experiments are frequently omitted. For these
details of importance for further research, the
non-conventional literature is then the first and
only source of information.
http//www.kb.nl/infolev/eagle/what_is_gl.htm
101
Veterinary Medicine
  • 12 Major Veterinary Medicine Journals
  • Overall, 6.38 of cited literature was Gray/Grey
    Literature
  • The figures for individual journals ranged from
    about 2.5 to 10 gray/grey literature
  • Research journals cited a higher percentage of
    Gray/Grey Literature than did Clinical titles
  • William H. Weise and Nancy Pelzer, Bibliometric
    Study of Grey Literature in Core Veterinary
    Medicine Journals, Journal of the Medical
    Library Association 91 no. 4 (October 2003) In
    press.

102
Indexing and Abstracting Services
  • SIGLE System for Information on Grey Literature
  • National Technical Information Service (NTIS)
  • PsychINFO (Psychological Abstracts)

103
SIGLESystem for Information on Grey Literature
  • Grey literature documents covered by SIGLE are
    technical or research reports, preprints,
    committee reports, working papers, dissertations,
    conference papers,  discussion and policy papers,
    government reports, market surveys, etc.

http//www.kb.nl/infolev/eagle/frames.htm
104
SIGLESystem for Information on Grey Literature
  • No. of Records Category
  • 4,158 Aeronautics
  • 17,044 Agriculture, plant veterinary sciences
  • 17,668 Environmental pollution, protection
    control
  • 256,657 Humanities, psychology social
    sciences
  • 81,269 Biological medical sciences
  • 25,089 Chemistry

http//www.kb.nl/infolev/eagle/frames.htm
105
NTIS (National Technical Information Service)
The NTIS Database produced by the National
Technical Information Service, is the preeminent
resource for accessing the latest U.S.
government-sponsored research and worldwide
scientific, technical, engineering, and
business-related information.
http//www.csa2.com/csa/factsheets/ntis.shtml
  • NTIS Database provides bibliographic data and
    abstracts of unclassified and publicly available
    information from research reports, journal
    articles, data files, computer programs and audio
    visual products, from U.S. and non-U.S.
    governmental, organizational, and commercial
    sources

106
Subject Coverage
Administration and Management Aeronautics Aerodynamics Agriculture Behavior Society
Business Chemistry Communications Computer Science
Education Energy Engineering Environmental Sciences
Health Care International Trade Library Information Science Materials Sciences
Mathematical Sciences Natural Resources Earth Sciences Nuclear Science Physics
Regulations Technology Tele-communications Transportation
107
PsycINFO
PsycINFO provides access to international
literature in psychology and related disciplines.
Unrivaled in its depth of psychological coverage
and respected worldwide for its high quality, the
database is enriched with literature from an
array of disciplines related to psychology such
as psychiatry, education, business, medicine,
nursing, pharmacology, law, linguistics, and
social work.
http//www.csa2.com/csa/factsheets/psycinfo.shtml
PsycINFO includes psychological research and its
applications the database is of prime relevance
to many industries and research establishments
worldwide. The sources include over 1,400
professional journals, chapters, books, reports,
theses and dissertations, published
internationally.
108
Subject Coverage
Applied Psychology Communication Systems Developmental Psychology Educational Psychology
Experimental Psychology Personality Psychological and physical disorders Professional personnel issues
Physiological Psychology and Neuroscience Psychometrics And Statistics Social Psychology Treatment and Prevention
109
DATABASE COVERAGE AND SIZE
Database Coverage Size
SIGLE 1976 Present 781,410 records (November 2002)
NTIS 1964 - Present 2,168,400 records (October 2001)
PsycINFO 1872- Present 1,870,180 records (September 2002)
110
http//www.neci.nec.com/lawrence/papers/online-na
ture01/
111
Conference papers are typical gray/grey
literature!
112
EPrints are Gray/Grey Literature
  • Daniela Luzi (1998) E-Print Archives a New
    Communication Pattern for Grey Literature,
    Interlending Document Supply 26 no. 3 (1998)
    130-139.

113
Gray/Grey Literature
  • Its good enough,
  • its smart enough,
  • and
  • doggone it, people use it!

With apologies to Stuart Smalley
114
http//www.openarchives.org/Register/BrowseSites.p
l
115
(No Transcript)
116
(No Transcript)
117
(No Transcript)
118
June 2003
http//software.eprints.org/
119
http//eprints.anu.edu.au/
120
http//caltechcstr.library.caltech.edu/
121
http//eprints.lub.lu.se/
122
http//ndltdpapers.dlib.vt.edu9090/
123
http//dspace.org/index.html
124
http//rocky.dlib.vt.edu/etdunion/cgi-bin/browse.
pl
125
http//www.ncstrl.org/
126
(No Transcript)
127
Quality Control in Scholarly Publishing on the
Web
  • "Most of the high quality materials on the Web
    are not peer-reviewed and much of the
    peer-reviewed literature is of dubious quality.
  • William Y. Arms, "What Are the Alternatives to
    Peer Review? Quality Control in Scholarly
    Publishing On The Web."
  • Journal of Electronic Publishing, 8 no. 1 (August
    2002).
  • http//www.press.umich.edu/jep/08-01/arms.html

128
http//www.update-software.com/Cochrane/MR000016.p
df
129
Cochrane Methodology Review
  • Despite its widespread use and costs, little hard
    evidence exists that peer review improves the
    quality of published biomedical research.
  • There had never even been any consensus on its
    aims and that it would be more appropriate to
    refer to it as competitive review.

Caroline White, Little Evidence for
Effectiveness of Scientific Peer Review, BMJ
326 (February 1, 2003) 241 http//bmj.com/cgi/rep
rint/326/7383/241/a.pdf
130
Cochrane Methodology Review
  • On the basis of the current evidence, the
    practice of peer review is based on faith in its
    effects, rather than on facts,' state the
    authors, who call for large, government funded
    research programmes to test the effectiveness of
    the classic peer review system and investigate
    possible alternatives.

Caroline White, Little Evidence for
Effectiveness of Scientific Peer Review, BMJ
326 (February 1, 2003) 241 http//bmj.com/cgi/rep
rint/326/7383/241/a.pdf
131
Cochrane Methodology Review
  • The use of peer-review is usually assumed to
    raise the quality of the end-product (i.e. the
    journal or scientific meeting) and to provide a
    mechanism for rational, fair and objective
    decision-making. However, these assumptions have
    rarely been tested.

Tom O. Jefferson, Phil Alderson, Frank Davidoff,
and Elizabeth Wager, Editorial Peer-review for
Improving the Quality of Reports of Biomedical
Studies. (Middle Way, Oxford Update Software
Ltd, 2003). http//www.update-software.com/Cochran
e/MR000016.pdf
132
Cochrane Methodology Review
  • The available research has not clearly identified
    or assessed the impact of peer-review on the more
    important outcomes (importance, usefulness,
    relevance, and quality of published reports)
  • Given the widespread use of peer-review and
    its importance, it is surprising that so little
    is known of its effects

Tom O. Jefferson, Phil Alderson,Frank Davidoff,
and Elizabeth Wager, Editorial Peer-review for
Improving the Quality of Reports of Biomedical
Studies. (Middle Way, Oxford Update Software
Ltd, 2003). http//www.update-software.com/Cochran
e/MR000016.pdf
133
Royal Society
http//www.biomedcentral.com/news/20030203/04/
134
UNCITEDNESS
David P. Hamilton, "Publishing by and for? --
the numbers, Science (New Series) 250 (4986)
(December 7 1990) 1331-1332. http//www.garfield.
library.upenn.edu/papers/hamilton1.html
135
David P. Hamilton,Research Papers Whos Uncited
Now?, Science (New Series) 251 (4989) (January
4, 1991) 25 http//garfield.library.upenn.edu/pap
ers/hamilton2.html
136
David P. Hamilton,Research Papers Whos Uncited
Now?, Science (New Series) 251 (4989) (January
4, 1991) 25 http//garfield.library.upenn.edu/pap
ers/hamilton2.html
137
REJECTED CITATION CLASSICS
  • NOBEL PRIZE RESEARCH
  • Severo Ochoa
  • Polynucleotide phosphorylase
  • Hans Krebs
  • Citric acid cycle
  • Rosalind Yalow
  • Radioimmunoassay
  • Harmut Michel
  • Photosynthetic processes

Juan Miguel Campanario, Commentary On
Influential Books and Journal Articles
Initially Rejected Because of Negative Referees
Evaluations, Science Communication 16 no. 3
(March 1995) 306-325
138
PEER REVIEW PURPOSES
Peer review helps to ensure that published
research is
Important ? Original ?
Timely ? Technically-reliable ?
Internally consistent ? Well-presented ?
Benefited from guidance by experts ?
Carin M. Olson, Peer Review of the Biomedical
Literature, American Journal of Emergency
Medicine 8 no.4 (July 1990) 356-368.
139
FILTERING (1)
  • UpStream / DownStream

Researchers look at certain types of
electronic publications because, despite being
tentative, may be relevant to their work.
Researchers are expected to do their own
downstream-filtering of relevant information,
which in the electronic world can be facilitated
by providing meta-information.
140
FILTERING (2)
  • UpStream / DownStream
  • Some have expressed the concern that having
    non-peer reviewed documents with peer-reviewed
    documents on the same server would contaminate
    the latter and compromise its quality
  • Readers could have trouble in distinguishing the
    different sections
  • Making non-peer-reviewed as well as
    peer-reviewed material will confuse both
    scientists and the public .

141
FILTERING (3)
  • UpStream / DownStream
  • However, this perhaps belittles the ability of
    scientists to recognize different levels of
    evidence and to be able to interpret quality
    labels that could make clear that certain
    materials is non-peer-reviewed content after all
    this is the age of transparency rather than
    paternalism .

Gunther Essenbach, The Impact of Preprint
Servers and Electronic Publishing on Biomedical
Research, Current Opinion in Immunology 12 no.
5 (October 2000) 499-503.
http//yi.com/home/EysenbachGunther/scans/Eysenbac
h2000e_CurrOpImmunol_preprint_servers.pdf
142
INVISIBLE HAND OF CLASSICAL PEER REVIEW
  • The refereed journal literature needs to be
    freed from both paper and its costs, but not from
    peer review, whose invisible hand is what
    maintains its quality.
  • Stevan Harnad

http//www.presidentmoron.com
143
INVISIBLE HAND OF CLASSICAL PEER REVIEW
144
INVISIBLE HAND OF CLASSICAL PEER REVIEW
Human nature being what it is, it cannot be
altogether relied upon to police itself.
Individual exceptions there may be, but to treat
them as the rule would be to underestimate the
degree to which our potential unruliness is
vetted by collective constraints, implemented
formally.
Stevan Harnad, The Invisible Hand of Peer
Review, Exploit Interactive no. 5 (April 2000).
http//www.exploit-lib.org/issue5/peer-review/
145
INVISIBLE HAND OF CLASSICAL PEER REVIEW
The system is not perfect, but it is what has
vouchsafed us our refereed journal literature to
date, such as it is, and so far no one has
demonstrated any viable alternative to having
experts judge the work of their peers, let alone
one that is at least as effective in maintaining
the quality of the literature as the present
imperfect one is.
146
INVISIBLE HAND OF CLASSICAL PEER REVIEW
Remove that invisible constraint -- let the
authors be answerable to no one but the general
users of the Archive arXiv.org (or even its
self-appointed "commentators") -- and watch human
nature take its natural course, standards eroding
as the Archive devolves toward the canonical
state of unconstrained postings the free-for-all
chat-groups of Usenet , that Global Graffiti
Board for Trivial Pursuit -- until someone
re-invents peer review and quality control.
Stevan Harnad, The Invisible Hand of Peer
Review, Exploit Interactive no. 5 (April
2000). http//www.exploit-lib.org/issue5/peer-revi
ew/
147
INVISIBLE HANDS
148
INVISIBLE HANDS
  • Personal reputation
  • Institutional reputation
  • Pride
  • Self-respect
  • Professional respect
  • Peer pressure
  • Critical Peer Response
  • Invisible College
  • Self-Archiving-Process-Itself
  • Open access
  • Common Sense
  • Self-correcting dynamics

149
RECOMMENDATIONSWorkshop on the Open Archives
Initiative (OAI)and Peer Review Journals in
Europe, CERN, Geneva Switzerland,March 22-24,
2001
  • It was also generally believed that the
    electronic environment allows for novel
    approaches to accord a stamp of quality to
    scholarly works.

Alison Buckholtz, Raf Dekeyser, Melissa
Hagemann, Thomas Krichel, and Herbert Van de
Sompel, Open Access Restoring Scientific
Communication to Its Rightful Owners, European
Science Foundation Policy Briefing 21 (April
2003) 1-8.
http//www.arl.org/sparc/SPB21_OAI.pdf
150
Examples of new metrics that can be extracted
from a fully electronic communication system are
  • Usage counts of a work
  • Automatically extracted citation information with
    a scope beyond the ISI- core journals
  • Amount of discussion generated by a paper
    submitted in a system with open peer review and
    peer comment
  • Etc.

Alison Buckholtz, Raf Dekeyser, Melissa
Hagemann, Thomas Krichel, and Herbert Van de
Sompel, Open Access Restoring Scientific
Communication to Its Rightful Owners, European
Science Foundation Policy Briefing 21 (April
2003) 1-8.
http//www.arl.org/sparc/SPB21_OAI.pdf
151
Scientific Publishing as Rhetoric
  • The problems with peer review become evident
    once the fact that science has a rhetorical
    element is accepted.
  • On the one hand, the traditional mode of peer
    review obscures the problems of reference and the
    rhetorical dimension of science. The rhetorical
    process which is at the heart of science and peer
    review conveniently disappears with the final
    publication of the manuscript. In its place is an
    ideal typical representation (the scientific
    paper) of the realist assumptions about empirical
    reference. All the academic world sees is a
    polished manuscript where the personal
    involvement of the researcher and reviewers has
    been systematically eliminated.

Mike Sosteric, Interactive Peer Review A
Research Note, Electronic Journal of Sociology 2
no. 1 (1996). http//socserv.socsci.mcmaster.ca/EJ
S/vol002.001/SostericNote.vol002.001.html
152
IDEAL SPEECH SITUATION
Jürgen Habermas
  • A theoretical construct that describes the ideal
    type of interpersonal interaction that should
    exist in a rhetorical situation.

153
  • IDEAL SPEECH SITUATION
  • the ideal speech situation permits each
    interlocutor an equal opportunity to initiate
    speech
  • there is mutual understanding between
    interlocutors
  • there is space for clarification
  • all interlocutors are equally free to use of any
    speech act
  • there is equal power over the exchange.
  • Applied in the context of peer, the Ideal Speech
    Situation would permit unimpeded authorial
    initiative, endless rounds of give and take,
    and unchecked openness among authors, editors,
    and referees.


Mike Sosteric, Interactive Peer Review A
Research Note, Electronic Journal of Sociology 2
no. 1 (1996). http//socserv.socsci.mcmaster.ca/EJ
S/vol002.001/SostericNote.vol002.001.html
154
CONTINUA
  • Continuum of PUBLICATION
  • (Scholarly Skywriting)
  • WEAK
  • MEDIUM
  • Continuum of REVIEW
  • (Scholarly Skyreading)

STRONG
155
OPEN ACCESS and OPEN RETRIEVAL without OPEN
USE
  • Incongruent
  • Contradictory
  • Ironic
  • Paradoxical
  • Cognitively Dissonant

156
Three-Legged Stool
  • ACCESS -- RETRIEVAL -- USE

157
(No Transcript)
158
ACCESS
  • OPEN ACCESS INITIATIVES

159
RETRIEVAL
  • OPEN ARCHIVES INITIATIVE FOR METADATA HARVESTING

160
USE
  • OPEN SCHOLARSHIP

161
  • ACCESS -- RETRIEVAL -- USE

162
Lots of Alternative ModelsProvide Sensible
Solutions
163
Four-Legged Stool
  • ACCESS -- RETRIEVAL -- USE -- NAVIGATION

164
INFORMATION OVERLOAD
165
OAIster
  • A search engine for freely available,
    difficult-to-access, academically-oriented
    digital resources that are OAI -compliant

http//oaister.umdl.umich.edu/
166
(No Transcript)
167
University of Michigan Digital Library
Production Service
  • institutional repositories
  • departmental repositories
  • e-Journal collections
  • technical reports
  • dissertations and theses
  • discipline eprint collections
  • working papers
  • Internet resources
  • audio
  • video
  • images

168
(No Transcript)
169
Cognitive Psychology
170
(No Transcript)
171
Open Archives Initiative Protocol for Metadata
Harvesting(OAI-PMH)
ADD

METADATA ELEMENT for QUALITY
172
Within the framework of OAI, there is a need for
a new protocol for certification. There was
strong support for the extension of the usage of
the OAI protocol beyond discovery-related metadata
. Given the focus of the 1st OAI workshop on
peer review, concrete actions were suggested to
address the exchange of certification-related meta
data using the OAI protocol in a trusted
environment.
Alison Buckholtz, Raf Dekeyser, Melissa
Hagemann, Thomas Krichel, and Herbert Van de
Sompel, Open Access Restoring Scientific
Communication to Its Rightful Owners, European
Science Foundation Policy Briefing 21 (April
2003) 1-8.
http//www.arl.org/sparc/SPB21_OAI.pdf
173
QUALITY METADATA (1)
ltoai-qualitygt ltcategorygtinternallt/categorygt ltproc
essgt peer review lt/processgt ltorganizationgt CERN lt/
organizationgt ltpoliciesgt http//www.cern.ch/polici
es/review.html lt/policiesgt lt/oai-qualitygt
William Y. Arms, Quality Control in Scholarly
Publishing On The Web. What Are the Alternatives
to Peer Review? PowerPoint presentation given at
Workshop on the Open Archives Initiatives (OAI)
and Peer Review Journals in Europe, March 22-24,
2003, CERN, Geneva, Switzerland http//www.cs.corn
ell.edu/wya/papers/CERN-2001.ppt
174
QUALITY METADATA (2)
ltoai-qualitygt ltcategorygtinternallt/categorygt ltrati
nggt lt/ratinggt ltorganizationgt 42 lt/organizati
ongt ltpoliciesgt http//www.cern.ch/policies/review.
html lt/policiesgt lt/oai-qualitygt
175
(No Transcript)
176
CERTIFICATION SERVICES
Faculty of 1,000,000
177
CERTICATION SERVICES
  • New roles for Indexing and Abstracting Services
  • Expanded Role for Learned and Professional
    Societies
  • Establishment of Formal/Commercial Reviewing
    Services

178
PEER REVIEW STRENGTHS
The underlying strength of peer review isthe
concerted effort by large numbers of researchers
and scholars who work to assure that valid and
valuable works are published and conversely to
assure that invalid or non-valuable works are not
published .
Anne C. Weller, Editorial Peer Review Its
Strengths and Weaknesses. (Medford, NJ
Information Today, 2001).
179
Faculty of 1000 / BioMed Central
  • BioMed Central (biomedcentral.com) publishes
    Faculty of 1000 (F1000), the leading literature
    evaluation service and new online research tool
    that highlights the most interesting papers in
    biology, based on the recommendations of over
    1000 leading scientists. F1000 is managed by
    scientists for scientists . and provides a
    rapidly updated consensus map of the important
    papers and trends across biology.

www.facultyof1000.com
180
Faculty of 1000 / BioMed Central
  • Among its many benefits, F1000
  • systematically organizes and evaluates the mass
    of information within scientific literature
  • provides scientists with a continuously updated
    insider's guide to the most important papers
    within any given field of research
  • highlights papers on the basis of their
    scientific merit rather than the journal in which
    they appear
  • offers the researcher a consensus of
    recommendations from well over 1000 leading
    scientists and,
  • offers an immediate rating of individual papers
    by the authors' peers, and an important
    complement to the indirect assessment provided by
    the journal impact factor.

181
Faculty of 1000 / BioMed Central
  • Within the F1000, the entire field of biology
    is divided into 16 subject areas
    (Faculties)(e.g., Biochemistry,, Cell
    Biology, Microbiology). Each Faculty is
    subdivided into three (3) to twelve (12)
    Sections, (e.g., Biochemistry Biocatalysis,
    Molecular evolution, Protein folding), with each
    section comprised of between 10 to 50 faculty
    members. F1000 seeks to invite the best
    internationally known scientists in each
    represented field and to involve both experienced
    and younger investigators.

182
Peer review is a quality-control and
certification (QC/C) filter necessitated by the
vast scale of learned research today. Without it,
no one would know where to start reading in the
welter of new work reported every day, nor what
was worth reading, and believing, and trying to
build ones own further research upon.
  • Stevan Harnad, Free at Last The Future of
    Peer-Reviewed Journals,
  • D-Lib Magazine 5 no. 12 (December 1999)
  • http//www.dlib.org/dlib/december99/12contents.htm
    l

183
SEIZE THE E!
  • Embrace the potential of the digital environment
    to facilitate access, retrieval, use, and
    navigation of electronic scholarship.

184
OPEN NAVIGATION
185
New Age Navigation
  • Innovative Interfaces for Electronic Journals
  • Gerry McKiernan
  • The Serials Librarian
  • Fall 2003

186
SUMMARY. While it is typical for electronic
journals to offer conventional search features
similar to those provided by electronic
databases, a select number of e-journals have
also made available higher-level access options
as well. In this article, we review several novel
technologies and implementations that creatively
exploit the inherent potential of the digital
environment to further facilitate use of
e-collections.
  • Gerry McKiernan, New Age Navigation Innovative
    Interfaces for Electronic Journals,
  • The Serials Librarian, Fall 2003.

http//www.coleonline.us/serialslibrarian/
187
http//www.highwire.org
188
Topic Map
189
Topic Map
190
Topic Map
191
Topic Map
192
Topic Map
Secondary Screen
193
Topic Map
194
Topic Map
TopicMap is based on the Hyperbolic Tree SDK for
Java, licensed from Inxight Software, Inc., a
spin-off company from Xerox PARC, and leading
provider of Unstructured Data Management
solutions for accessing, analyzing and delivering
information.
http//www.inxight.com
195
Semio
Automated categorization software technology
http//www.entrieva.com/
196
Kohonen Self-Organizing Maps
http//websom.hut.fi/websom/milliondemo/html/root.
html
197
Kohonen Self-Organizing Maps
http//websom.hut.fi/websom/milliondemo/html/1_cx5
.html
198
Kohonen Self-Organizing Maps
http//websom.hut.fi/websom/milliondemo/html/2_gx1
0.html
199
Kohonen Self-Organizing Maps
http//websom.hut.fi/websom/milliondemo/getnd.cgi?
32323
200
Kohonen Self-Organizing Maps
http//websom.hut.fi/websom/cgi-bin/getfile.cgi/co
mp.ai/39340
201
http//www.springer.de/books/toc/3540679219-c.pdf
202
There are some excuses, but at the bottom it
will be seen to be the sluggishness of human
nature and its superstitious cleavage to old
habits.
  • Stevan Harnad, Free at Last The Future of
    Peer-Reviewed Journals,
  • D-Lib Magazine 5 no. 12 (December 1999)
  • http//www.dlib.org/dlib/december99/12contents.htm
    l

203
In the digital world, the evaluation process
stands ready to be reinvented in a clear,
rational way by the relevant research communities
themselves.
  • Jean-Claude Guédon,
  • In Oldenburgs Long Shadow Librarians, Research
    Scientists, Publishers, and the Control of
    Scientific Publishing.
  • (Washington, D.C. Association of Research
    Libraries, 2001), 54.

http//www.arl.org/arl/proceedings/138/guedon.html

204
lt/ENDQUOTEgt
  • The Medium is the Message
  • And
  • the Method.
  • With apologies to Marshall McLuhen

205
THANK YOU FOR YOUR ATTENTION!
206
Alternative Peer ReviewQuality Management for
21st Century Scholarship
  • Gerry McKiernan
  • Science and Technology Librarian
  • and Bibliographer
  • Iowa State University Library
  • Ames IA
  • USA

gerrymck_at_iastate.edu
207
OPEN MIND
208
REVISED VERSION 1.01
  • July 27, 2003
  • 1130 AM
About PowerShow.com