The Usage Factor Project Richard Gedye, Sales Director, Oxford Journals

1
The Usage Factor Project Richard
Gedye, Sales Director, Oxford Journals

• OUP NLC Day
• 27 February 2008

2
5 years ago, COUNTER set out to make usage
statistics

• Consistent
• Credible
• Compatible
• How successful has COUNTER been?

Standard format usage reports now adopted by over
60 publishers and hosts
Formal auditing process starts in 2007
Yes, but
3
The challenge.

• COUNTER statistics provide a reliable comparison
  of amount of use between journals
• But they dont provide a meaningful usage-based
  measure of relative quality or value
• All other things being equal, a journal
  publishing 2000 articles a year will generate
  significantly more downloads than one publishing
  50.

4

• If I was guided solely by usage statistics, I
  would cancel all my subscriptions to humanities
  journals, which tend to publish far fewer issues
  per year than the monster science titles
• Terry Bucknell, Electronic Resources Manager at
  Liverpool University

5
Addressing the challenge

• ISI's Impact Factor compensates for the fact that
  larger journals will tend to be cited more than
  smaller ones
• Can we do something similar for usage?
• In other words, should we seek to develop a
  Usage Factor as an additional measure of
  journal quality/value?

6
For example..

• Usage Factor
• Total usage over period x of articles published
  during period y
• Total articles published during period y

7
Basis for Usage Factor

• A possible basis for the calculation of Usage
  Factor could be-
• COUNTER Journal Report 1 Number of Successful
  Full-text Article Requests by Month and Journal

8
Reality check

• Is there a demand for it among-
• Publishers
• Authors
• Librarians
• What are the practical issues that would need to
  be addressed?
• How much would it cost to develop and maintain?

9
Some initial evidence
From New journal publishing models an
international survey of senior researchers Ian
Rowlands and Dave Nicholas, A CIBER report for
the Publishers Association and the International
Association of STM Publishers, 22 September 2005
10
Some initial evidence
From New journal publishing models an
international survey of senior researchers Ian
Rowlands and Dave Nicholas, A CIBER report for
the Publishers Association and the International
Association of STM Publishers, 22 September 2005
11
Some initial evidence

• Note that the question explored the measurement
  of utility rather than quality but this is
  nonetheless a surprising finding and it may
  indicate that download metrics would have
  considerable credibility amongst the author
  community.
• Alternatives to the traditional impact factor,
  based on article downloads and modeled using the
  same time windows as are used to construct impact
  factors might offer a very interesting and
  worthwhile direction for future research and
  development they would certainly be of great
  appeal to librarians and many publishers.

From New journal publishing models an
international survey of senior researchers Ian
Rowlands and Dave Nicholas, A CIBER report for
the Publishers Association and the International
Association of STM Publishers, 22 September 2005
12
The Research.

• In 2007 the UK Serials Group and COUNTER
  published the results of research which
  examined-
• the various ways in which journal quality is
  currently assessed
• the degree to which any additional usage-based
  metrics might prove valuable to each stakeholder
  community
• practical ways in which such metrics might be
  derived and constructed to provide the maximum
  utility for all, within defined resource
  constraints

13
Research..

• Phase 1 October 2006 January 2007
• COUNTER Director Peter Shepherd carried out a
  series of in-depth interviews with-
• 7 authors
• 9 librarians
• 13 publishers
• Phase 2 March 2007
• Broader web based survey of-
• 155 librarians
• 1400 academic authors
• May 2007
• Final report published by UKSG

14
Phase 1 Headline Results Users

• Would Journal Usage Factors be helpful to you in
  assessing the value, status and relevance of a
  journal?
• Librarians YES 100
• Authors - YES 100

15
Phase 1 Perceptions

• Usage Factor
• Advantages
• What are the optimal variables to measure

16
Phase 1 Usage factor advantages

• A useful counterweight to Impact Factors
• Especially helpful for journals and fields not
  covered by ISI
• Especially helpful for journals with high
  undergraduate or practitioner use
• Especially helpful for journals publishing
  relatively few articles
• Data available potentially sooner than with
  Impact Factors

17
Phase 1 Usage factor advantages

• Authors select journals that will give their
  articles prestige and reach. Impact Factor is a
  widely used surrogate for the former, while
  perceived circulation and readership reflect the
  latter. But usage is becoming more important as a
  measure of reach
• Carol Tenopir
• Many of the publications in which I publish and
  in which I would like to publish do not have IFs
  and the current system almost requires serious
  authors to publish in journals that have IFs.
• 
  Author

18
Phase 1 Usage factor advantages

• UF would definitely benefit publishers.
• First, it would be a useful balance for IFs,
  which are not always accurate reflections of the
  value of a journal.
• Second, UF is a simple metric that would make
  usage more understandable to editors and authors
  as a measure of value. There is currently much
  talk of usage and a lot of data, which the
  non-librarians find confusing.
• Publisher

19
Phase 1 What to measure

• Many views on how UF should be calculated
• In particular, how to define
• total usage
• specified usage period
• total number of articles published online
• Tests with real usage data will be required to
  refine the definitions for these terms.

20
Research..

• Phase 1 October 2006 January 2007
• COUNTER Director Peter Shepherd carried out a
  series of in-depth interviews with-
• 7 authors
• 9 librarians
• 13 publishers
• Phase 2 March 2007
• Broader web based survey of-
• 155 librarians
• 1400 academic authors
• May 2007
• Final report delivered to UKSG

21
Phase 2 Author results - support for a new,
usage based measure
22
(No Transcript)
23
Phase 2 Librarian results new journals
24
Phase 2 Librarian results existing journals
I would view Usage Factor as an aid for
collection rather than cancellation decisions.
Usage per se is a more suitable tool for us when
considering cancellation.
25
Recommendations.

• That the UF concept be developed further, with a
  view to testing it as a practical, implementable
  measure of journal quality, value and status-
• Test each of the individual elements in the UF
  equation using real publisher usage data
• Compare UF journal rankings with IF journal
  rankings
• Refine and investigate further the different
  workflow/organizational scenarios for the
  definition, calculation and dissemination of UFs

26
Recommendations.

• If satisfactory results are obtained from the
  above investigations and tests, it is likely that
  the system would have to be tested for one or two
  years to check the large scale validity of the
  outcomes before a comprehensive list of journal
  UFs is published.

27
Recommendations Accepted

• On May 18, 2007 the UKSG Committee accepted these
  recommendations.
• It asked the Project Working Group to draw up a
  set of practical proposals for putting the
  recommendations into practice.
• We held our first meeting on June 27

28
The Next Stage

• Testing and modelling the Usage Factor concept
  with real data

29
Stage 2 Essential preliminaries

• Working Group expands, becomes Project Steering
  Group
• Wider geographical library representation plus-
• 6 publishers
• 1 aggregator
• 1 hosting service
• All interested in the insights to be gained from
  contributing data to the project

30
Stage 2 The Plan

• Usage logs to be ingested and converted to
  uniform standard report format for analysis by
  expert third party
• RFP currently being drafted for third party
  selection process

31
Stage 2 The Plan

• RFP published March 2008 after outstanding issues
  addressed to ensure data consistency, integrity,
  and fitness for purpose.
• For example-
• Measuring number of items published
• Assigning a correct publication year for each
  item
• Excluding spiders, crawlers, etc

32
Stage 2 The Deliverables

• A report (Summer 2008) which will-
• Outline the various metrics assessed
• Recommend which of them prove consistent and
  robust enough to be adopted for scaled up onward
  monitoring
• Suggest any ways in which data providers might
  amend the way they capture, structure, label, and
  maintain their data which would make the
  measurement of Usage Factors-
• Easier
• More reliable

33
Usage Factor Project

• More information at-
• http//www.uksg.org/usagefactors
• Or contact-
• Richard Gedye
• richard.gedye_at_oxfordjournals.org