Next Generation Statistics for Libraries: CAUL Forum 20 January 2003 Sydney, NSW The ARL EMetrics Pr

1
Next Generation Statistics for Libraries CAUL
Forum20 January 2003Sydney, NSWThe ARL
E-Metrics Project

• Mary E. Jackson
• Senior Program Officer for Access Services
• Association of Research Libraries
• mary_at_arl.org

2
ARL New Measures Initiative

• Collaboration among member leaders with strong
  interests
• Specific projects explore different models
• Self-funded projects

3
ARL New Measures Initiative

• Make resulting tools and methodologies available
  to full membership and wider community
• Continue to collect, but limit/freeze
  modifications to existing descriptive measures

4
ARL New Measures Projects

• Demonstration project for service effectiveness
  measures
• LibQUAL?
• Identification of measures that demonstrate a
  librarys contribution to student learning
  outcomes
• Investigation of role libraries play in support
  of the research process

5
ARL New Measures Projects

• Development of tools to address cost
  effectiveness of library operations
• staff allocation, Assessing ILL/DD Services study
• Project to define usage measures for electronic
  information resources
• E-metrics Project

6
Electronic Serials Expenditures
7
Percentages of Acquisitions Dollars Devoted to
Electronic Resources
8
Average Yearly Increases in Electronic Resources
and Total Library Materials Expenditures
9
Project Leaders

• Co-Chairs
• Rush Miller, University of Pittsburgh, and
  Sherrie Schmidt, Arizona State University
• Information Use Management and Policy Institute,
  Florida State University
• Charles R. McClure, Wonsik Jeff Shim, and John
  Carlo Bertot
• ARL
• Duane Webster and Martha Kyrillidou

10
Project Participants

• Alberta
• Arizona State
• Auburn
• Chicago
• Connecticut
• Cornell
• Illinois - Chicago
• Library of Congress
• Manitoba
• Maryland
• Massachusetts
• Nebraska

• New York Public
• Notre Dame
• Pennsylvania
• Penn State
• Pittsburgh
• Purdue
• Southern California
• Texas A M
• Virginia Tech
• Western Ontario
• Wisconsin
• Yale

11
Vendor Statistics Working Group

• 12 major ARL vendors met with project team in
  Denver prior to 2000 ACRL Meeting

Academic Press/IDEAL Elsevier/Science
Direct Lexis/Nexis Ovid Bell Howell Gale Group
ISI
netLibrarySilver Platter EBSCO JSTOROCLC/First
Search
Unable to attend Denver Meeting
Nonparticipant in project.
12
Working Definition of Networked Services

• Electronic information resources and/or services
  that users access electronically via a computer
  network
• From on-site in the library
• Remote to the library but from a campus
  facility
• Remote from the library campus

13
Networked Information Resources

• Locally Licensed Databases
• Regional or Statewide Consortia Licensed
  Databases
• Aggregated Databases
• Publishers Databases
• Publicly Available (Web) Resources

14
ARL E-Metrics Project Phase 1

• May-October 2000
• What do we know?
• Inventory of current practices at ARL libraries
  of statistics, measures, processes, and
  activities that pertain to networked resources
  and services.

15
ARL E-Metrics Project Phase 2

• November 2000-June 2001
• What can we collect?
• Identified and field tested an initial draft set
  of statistics and measures.

16
ARL E-Metrics Project Phase 3

• July 2001-December 2001
• What difference does this make?
• Build linkages to educational outcomes/impact,
  research, technical infrastructure.

17
Recommended Statistics Measures

• Patron Accessible Electronic Resources (R1-3)
• Use of Networked Resources Services (U1-5)
• Expenditures for Networked Resources Related
  Infrastructure (C1-3)
• Library Digitization Activities (D1-3)
• Performance Measures (P1-3)

18
Patron Accessible Electronic Resources

• R1 Number of electronic full-text journals
• R2 Number of electronic reference sources
• R3 Number of electronic books

19
Use of Networked Resources Related
Infrastructure

• U1 Number of electronic reference transactions
• U2 Number of logins (sessions) to electronic
  databases (db)
• U3 Number of queries (searches) in electronic
  db
• U4 Items requested in electronic db
• U5 Virtual visits to librarys website and
  catalog

20
Expenditures for Networked Resources Related
Infrastructure

• C1 ? Cost of electronic full-text journals
• C2 ? Cost of electronic reference sources
• C3 ? Cost of electronic books
• C4 ? Library expenditures for bibliographic
  utilities, networks, consortia
• C5 ? External expenditures for bibliographic
  utilities, networks, consortia

21
Library Digitization Activities

• D1 Size of library digital collection
• D2 Use of library digital collection
• D3 Cost of digital collection construction
  management

(Collecting these data requires staff familiar
with the digital environment.)
22
Performance Measures

• P1 Percentage of electronic reference
  transactions of total reference
• P2 Percentage of virtual visits of all library
  visits
• P3 Percentage of electronic books to all
  monographs

23
Characteristics of Each Recommended Measure

• Definition
• Rationale
• Unit of Measure
• Data source
• Frequency
• Process
• Related Issues

24
R1 Number of Electronic Full-text Journals

• Definition - Number of electronic full-text
  journal subscriptions by individual institution
  or consortia licensing.
• Rationale Documents degree of expansion of
  electronic subscriptions available can be used
  to show good coverage need for more funding.
• Unit of measure the journal subscription

25
R1 Number of Electronic Full-text Journals

• Data source local or vendors.
• Frequency annual, monthly, etc.
• Process parse into database or spreadsheet,
  update dynamically from local catalog or vendor
  record.

26
U1 Number of Electronic Reference Transactions

• Definition - number of electronic reference
  transactions via e-mail, WWW form, etc.
• Rationale libraries are interested in tracking
  the development of new electronic services.
  Attempt to measure reference transactions through
  new electronic tools and services.
• Unit of Measure request count, time it took.

27
U1 Number of Electronic Reference Transactions

• Data Source local server, manual tally, e-mail
  count.
• Frequency daily, monthly, annually, etc.
• Process clarify process, identify activity
  points, identify collectors of data, consolidate
  data.
• Related Issues This measure may have to broken
  down into additional data types time, type of
  query, type of interaction, scheduling issues,
  measures of quality and reliability.

28
Outcomes of the Investigation

• Mixed results
• Participants wanted more definitive answers
• Confirmed how difficult, but not impossible, to
  come to consensus
• Realized how dependent libraries are for vendors
  to supply data
• Recognized need for ongoing effort
• Reaffirmed importance of using data for decisions

29
E-Metrics Next Steps

• Call for participation among members to test
  proposed measures for 2002/2003 (over 35
  participants to date)
• Examination of the deliverables from the first
  phases
• Collect FY02 totals
• Compilation
• Data analysis
• Distribution for discussion
• Analysis of approaches
• Best practices for work processes
• From E-metrics project
• Locally developed

30
E-Metrics Next Steps

• Continued work with vendors through international
  COUNTER project
• Continued work with international standards
  activities
• Workshops and training to develop necessary data
  analysis skills

31
Project Documents

• Measures for Electronic Resources (E-Metrics)
• Part 1 Project Background and Phase One Report
• Part 2 Phase Two Report
• Part 3 E-Metrics Instructional Module
• Part 4 Data Collection Manual
• Part 5 Library and Institutional Outcomes
• www.arl.org/stats/newmeas/emetrics/

32
Additional Questions?

• Please contact
• Martha Kyrillidou
• Senior Program Officer for Statistics and
  Measurement
• Association of Research Libraries
• martha_at_arl.org