Measuring%20the%20Impact%20of%20Networked%20Electronic%20Services:%20Developing%20Assessment%20Infrastructure%20for%20Libraries,%20State,%20and%20Other%20Types%20of%20Consortia

1
Measuring the Impact of Networked Electronic
Services Developing Assessment Infrastructure
for Libraries, State, and Other Types of Consortia

• Presented By
• Terry Plum, Simmons GSLISBrinley Franklin,
  University of ConnecticutMartha Kyrillidou, ARL
  Gary Roebuck, ARL
• Raynna Bowlby, ARL Consultant Simmons
  GSLISMaShana Davis, ARL Kristina Justh, ARL

Library Assessment Conference 2008 University of
Washington Seattle, WA
2
ARL New Measures Toolkit StatsQUAL
3
What is MINES?

• Action Research
• Historically rooted in indirect cost studies
• Set of recommendations for research design
• Set of recommendations for web survey
  presentation
• Set of recommendations for information
  architecture in libraries
• Plan for continual assessment of networked
  electronic resources
• An opportunity to benchmark across libraries

4
What is MINES? (contd)

• Measuring the Impact of Networked Electronic
  Services (MINES)
• MINES is a research methodology that measures the
  usage of networked electronic resources of a
  library or consortium by a specific category of
  the patron population.
• MINES is a Web-based survey form consisting of 5
  questions that is administered at the time of
  transaction.
• MINES measures
• User status and discipline/affiliation (who)
• Physical location (where)
• Primary purpose and reason of use (why)
• MINES a part of Association of Research
  Libraries New Measures Assessment Initiatives
  (since 2003).
• MINES is different from other electronic resource
  usage measures that quantify total usage (e.g.,
  Project COUNTER, E-Metrics) or measure how well a
  library makes electronic resources accessible
  (LibQUAL).

5
Early Data Collection Activities
Data was collected at seven main campus libraries
and seven academic health sciences libraries in
the U.S. between 2003 and 2005.

• Academic Medical Libraries
• U of Connecticut Health Center
• U of North Carolina
• U of Texas Medical Branch
• U of Texas Southwestern
• U of Utah
• U of Virginia
• Washington U

• Main Universities
• U of Colorado
• U of Connecticut
• U of North Carolina
• Oregon State U
• U of Utah
• U of Virginia
• Washington U

6
Early Data Collection Activities

• More than 45,000 networked electronic services
  uses were surveyed.
• At each library, the MINES survey was one
  component of a comprehensive cost analysis study
  that assigned all library costs to sponsored
  research, instruction/education/non-sponsored
  research, patient care, other sponsored
  activities and other activities.

7
Recent Data Collection Activities via ARL

• Ontario Council of University Libraries (OCUL)
• University of Iowa Libraries
• University of Macedonia

8
Questions Addressed

• How extensively do sponsored researchers use the
  new digital information environment?
• Are researchers more likely to use networked
  electronic resources from inside or outside the
  library?
• Are there differences in usage of electronic
  information based on the users location (e.g.,
  in the library on-campus, but not in the
  library or off-campus)?
• What is a statistically valid methodology for
  capturing electronic services usage both in the
  library and remotely through web surveys?
• Are particular network configurations more
  conducive to studies of digital libraries patron
  use?

9
Questions Addressed by OCUL Implementation

• How extensively do sponsored researchers use
  OCULs Scholars Portal? How much usage is for
  non-funded research, instruction/education,
  student research papers, and course work?
• Are researchers more likely to use the Scholars
  Portal from inside or outside the library? What
  about other classifications of users?
• Are there differences in Scholars Portal based on
  the users location (e.g., in the library
  on-campus, but not in the library or
  off-campus)?
• Could MINES, combined with usage counts, provide
  an infrastructure to make Scholars Portal usage
  studies routine, robust, and easily integrated
  into OCULs administrative decision-making
  process for assessing networked electronic
  resources?

10
MINES for Libraries Survey FormFive Questions
and a Comments Box
11
Methodological ConsiderationsExperience with
the MINES Survey

• Terry Plum
• Assistant Dean
• Simmons GSLIS

Library Assessment Conference 2008 University of
Washington Seattle, WA
12
Issues with Web surveys

• Research design
• Coverage error
• Unequal access to the Internet
• Internet users are different than non-users
• Response rate
• Response representativeness
• Random sampling and inference
• Non-respondents
• Data security

13
MINES Strategy

• A representative sampling plan, including sample
  size, is determined at the outset. Typically,
  there are 48 hours of surveying over 12 months at
  a medical library and 24 hours a year at a main
  library.
• Random moment/web-based surveys are employed at
  each site.
• Participation is usually mandatory, negating
  non-respondent bias, and is based on actual use
  in real-time.
• IRB waiver or approval
• Libraries with database-to-web gateways or proxy
  re-writers offer a comprehensive networking
  solution for surveying all networked services
  users during survey periods.

14
MINES Strategy

• Placement
• Point of use
• Not remembered, predicted or critical incident
• Usage rather than user
• What about multiple usages
• Time out ?
• Cookie or other mechanism with auto-population or
  more recently counting invisibly with a time out.
  
• Distinguish patron association with libraries.
• For example, medical library v. main library.
• But what if the resources are purchased across
  campus for all. Then how to get patron
  affiliation?

15
Web Survey Design Guidelines

• Web survey design guidelines that MINES followed
  
• Presentation
• Simple text for different browsers no graphics
• Different browsers render web pages differently
• Few questions per screen or simply few questions
• Easy to navigate
• Short and plain
• No scrolling
• Clear and encouraging error or warning messages
• Every question answered in a similar way -
  consistent
• Radio buttons, drop downs
• ADA compliant
• Introduction page or paragraph
• Easy to read
• Must see definitions of sponsored research.
• Can present questions in response to answers
  for example if sponsored research was chosen,
  could present another survey

16
Quality Checks

• Target population is the population frame
  surveyed the patrons who were supposed to be
  surveyed - except in libraries with outstanding
  open digital collections.
• Check usage against IP. In this case, big
  numbers may not be good. May be seeing the
  survey too often.
• Alter order of questions and answers,
  particularly sponsored and instruction.
• Spot check IP against self-identified location
• Spot check undergraduates choosing sponsored
  research measurement error
• Check self-identified grant information against
  actual grants
• Content validity discussed with librarians and
  pre-tested.
• Turn-aways number who elected not to fill out
  the survey
• Library information architecture -- Gateway v.
  HTML pages there is a substantial difference in
  results.

17
(No Transcript)
18
Documenting the Purpose and Use of Electronic
ResourcesExperience with the MINES Survey

• Brinley Franklin
• Vice Provost for University Libraries
• University of Connecticut

Library Assessment Conference 2008 University of
Washington Seattle, WA
19
It is useless to tell the acquisitions librarian
that half the monographs ordered will never be
used, unless we can specify which 50 to avoid
buying. (Galvin and Kent, 1977)
20
Reliance on Vendor Statistics
Vendor statistics, while more reliable than
in the past, are still maturing.
21
Measuring Digital Content Use

• The most popular current method of measuring
  usage of electronic resources by libraries is not
  through web-based usage surveys, but through
  vendor supplied data of library patron usage or
  transaction usage.
• Web-based usage surveys are increasingly relevant
  in the collection of usage data to make
  collection development and service decisions, to
  document evidence of usage by certain patron
  populations, and to collect and analyze
  performance outputs.
• Brinley Franklin and Terry Plum, Successful Web
  Survey Methodologies for Measuring the Impact of
  Networked Electronic Services (MINES for
  Libraries) IFLA Journal 32 (1) March, 2006

22
Measuring Digital Content Use

• A web-based transactional survey that collects
  data on users demographics and their purpose of
  use. It is administered in real time over the
  course of at least a year using a random moments
  sampling plan.
• MINES for Libraries has been administered at 40
  North American universities in the last four
  years. More than 100,000 North American
  networked services users have been surveyed using
  a standard protocol.

23
Library User Survey
24
Library User Survey Patron Status
25
Library User Survey Affiliation
26
Library User Survey Location
27
Library User Survey Purpose
28
Sample Survey Data File Generated

• Other UConnInstruction/Education/Departmental
  (Non-Funded) Researchhttp//newfirstsearch.oclc.or
  g/donerefererdbnameWorldCatautho100122319FSI
  P12365012/3/2004Off CampusUConn
  Faculty12.101.104.84
• Family StudiesInstruction/Education/Departmental
  (Non-Funded) Researchhttp//www.jstor.org/cgi-bin/
  jstor/gensearch12374312/3/2004Off CampusUConn
  Undergraduate Student12.101.104.84Non-UConnOther
  Activitieshttp//homerweb.lib.uconn.edu/cgi-bin/Pw
  ebrecon.cgi?DBlocalPAGEFirst13084112/3/2004Of
  f CampusNon-UConn12.18.36.40
• Non-UConnInstruction/Education/Departmental
  (Non-Funded) Researchhttp//magic.lib.uconn.edu/in
  dex_real.html13312912/3/2004Off
  CampusNon-UConn12.76.131.246
• Non-UConnOther Activitieshttp//magic.lib.uconn.ed
  u/index_real.html12110612/3/2004Off
  CampusNon-UConn130.132.86.156Agriculture
  Natural ResourcesInstruction/Education/Departmenta
  l (Non-Funded) Researchhttp//magic.lib.uconn.edu/
  index_real.html12335712/3/2004
• Off CampusNon-UConn131.128.89.54EducationInstructi
  on/Education/Departmental (Non-Funded)
  Researchhttp//www.euromonitor.com/womdas/125744
  12/3/2004
• Off CampusNon-UConn134.241.135.70Non-UConnInstruct
  ion/Education/Departmental (Non-Funded)
  Researchhttp//homerweb.lib.uconn.edu/cgi-bin/Pweb
  recon.cgi?DBlocalPAGEFirst13285212/3/2004
• Off CampusNon-UConn134.74.42.82Business
  AdministrationOther Activitieshttp//homerweb.lib.
  uconn.edu/cgi-bin/Pwebrecon.cgi?DBlocalPAGEFirs
  t12564612/3/2004In the LibraryUConn
  Faculty137.99.1.122
• Liberal Arts SciencesOther Activitieshttp//www.
  siam.org/journals/simax/simax.htm12521712/3/2004
  On Campus - StorrsUConn Graduate
  Student137.99.1.122
• EngineeringInstruction/Education/Departmental
  (Non-Funded) Researchhttp//homerweb.lib.uconn.edu
  /cgi-bin/Pwebrecon.cgi?DBlocalPAGEFirst120431
  12/3/2004On Campus - StorrsUConn Graduate
  Student137.99.1.219
• Business AdministrationInstruction/Education/Depar
  tmental (Non-Funded) Researchhttp//proquest.umi.c
  om/pqdweb?RQT31812163312/3/2004On Campus -
  StorrsUConn Graduate Student137.99.100.105
• Business AdministrationInstruction/Education/Depar
  tmental (Non-Funded) Researchhttp//www.jstor.org/
  journals/07322399.html12165212/3/2004On Campus
  - StorrsUConn Graduate Student137.99.100.105
• Business AdministrationInstruction/Education/Depar
  tmental (Non-Funded) Researchhttp//homerweb.lib.u
  conn.edu/cgi-bin/Pwebrecon.cgi?DBlocalPAGEFirst
  12295312/3/2004On Campus - StorrsUConn Graduate
  Student137.99.100.175
• Business AdministrationInstruction/Education/Depar
  tmental (Non-Funded) Researchhttp//homerweb.lib.u
  conn.edu/cgi-bin/Pwebrecon.cgi?DBlocalPAGEFirst
  12484112/3/2004On Campus - StorrsUConn Graduate
  Student137.99.100.175
• Business AdministrationInstruction/Education/Depar
  tmental (Non-Funded) Researchhttp//proquest.umi.c
  om/login?COPTSU5UPTAmVkVSPTImREJTPTE3MjErMysxNkJD
  clientId4899612042312/3/2004On Campus -
  StorrsUConn Graduate Student137.99.100.175

29
MINES for Libraries
Demographics by Location of User U.S. Main
Libraries
On Campus, Not in the Libraryn 6,391
Inside the Libraryn 9,172
Off-Campus n 4,953
30
MINES for Libraries
Demographics by Location of User Ontario
Council of University Libraries
Off-Campus n 9,163
On Campus, Not in the Library n 7,090
Inside the Library n 4,047
31
Purpose of Use
Are users engaged in coursework, funded (or
unfunded) research, public service,
patient care, or other activities?
32
MINES for Libraries
Purpose of Use by Location U.S. Main Campus
Libraries 2003-05
In the Library n 9,733
On-Campus, Not in the Library n 9,460
66
Overall Use n 26,983
Off-Campus n 7,790
72 of sponsored research usage of electronic
resources occurred outside the library 83 took
place on campus.
33
MINES for Libraries
Purpose of Use OCUL Scholars Portal Users
In a sample of 20,300 electronic resources uses
at OCUL Libraries, there were four uses outside
the library for each use in the library.
34
Analysis

• Web deliverables
• Cross-tabulations in HTML for all institutional
  data
• Interactive crosstabs for all institutional
• Print deliverables
• Summary tables
• Final report

35
(No Transcript)
36
Web Interface
37
OCUL Scholars Portal Usage Affiliation
38
OCUL Scholars Portal Usage User Status
39
OCUL Scholars Portal UsageLocation
40
OCUL Scholars Portal UsagePurpose of Use
41
Cross Tabulations Affiliation by Purpose of Use
42
Cross Tabulations User Status by Purpose of Use
43
Cross TabulationsLocation by Purpose of Use
44
OCUL Scholars Portal Usage Reason for Use
45
(No Transcript)
46
(No Transcript)
47
(No Transcript)
48
Resources

• Web interface
• http//minesforlibraries.org
• ARL New Measures and Assessment Initiatives
  MINES for Libraries
• http//www.arl.org/stats/initiatives/mines/index.
  shtmlplace
• Articles Presentations
• http//www.arl.org/stats/initiatives/mines/minesr
  esources.shtml

49
Contact Us

• ARL Staff
• MaShana Davis, Technical Communications Liaison
• Martha Kyrillidou, Director, ARL Statistics and
  Service Quality Programs
• Gary Roebuck, Technical Operations Manager
• Project Management
• Brinley Franklin, Vice Provost for University
  Libraries, University of Connecticut Libraries
• Toni Olshen, Business Librarian, Peter F.
  Bronfman Business Library, York University
• Terry Plum, Assistant Dean for Technology and
  Director, Simmons Graduate School of Library and
  Information Sciences

50
Questions