Assessing Service Effectiveness: The UConn Libraries Experience

1
Assessing Service EffectivenessThe UConn
Libraries Experience

• Simmons GSLIS
• Brinley Franklin
• April 17, 2004

April 17, 2004
2
Assessment Efforts at the UCL

• Encouraged by 1995 Initiative to Develop
  methods for learning more about the information
  and service needs of our constituencies and
  integrate this understanding into decision-making
  at all levels of the Libraries.
• User surveys
• Organizational Assessment
• LibQual
• E-metrics
• Focus groups
• Written and Electronic Comments
• Responding to User Feedback

3
User Surveys

• Faculty survey (1997) census mailing to 1100
  faculty received 400 responses
• Undergraduate student survey (1999)
  administered a written survey in class to a
  representative sample of students
• Graduate Student Survey (2000) census mailing to
  4500 students received 1550 responses (95
  confidence level /- 2 error rate
• Web-based survey (2001) communicated via email
  to all faculty and graduate students and a
  representative random sample of undergraduates

4
Organizational Assessment

• When the Libraries reorganized in 1996, we
  developed 18 objectives to measure how successful
  our reorganization efforts were. These included
  empowering staff making the decision-making
  structure clear to staff and integrating the
  regional campus libraries into the overall
  library system.
• Every three years, starting in 1999, we
  administer an organizational assessment survey to
  determine library-wide progress toward the
  objectives.
• Between 1999 and 2002, the overall score for the
  Libraries as an organization improved by 5 and
  the library-wide average score improved on 14 of
  the 18 measures.

5
LibQual

• UConn was one of 12 participants in a pilot study
  to prepare the instrument for broader
  distribution
• UConn average perceived score about 7. Minimum
  expectation tended to be about 6, perceived 6-7,
  desired about 8.
• Only gap at research libraries (33,000
  respondents) between minimum tolerable and
  perceived level of service is Complete runs of
  journal titles (minimum6.56, desired 8.05,
  perceived 6.51) Gap is -.05 outside zone of
  tolerance.

6
E-Metrics

• There are a number of initiatives underway to
  standardize data elements and reporting
  requirements for electronic materials and
  services usage statistics, including
• E-METRICS (Association of Research Libraries)
  www.arl.org/stats/newmeas/emetrics/
• PROJECT COUNTER Counting Online Usage of
  NeTworked Electronic Resources a consortium of
  publishers, intermediaries, and library
  organizations www.projectcounter.org
• ICOLC International Coalition of Library
  Consortia Guidelines for Statistical Measures
  of Usage of Web-based Information Resources
  www.library.yale.edu/consortia/2001webstats.htm
• NISO National Information Standards
  Organization Z39.7 Library Statistics
  Standard includes network services statistics
• ISO International Organization for
  Standardization ISO 2789 Library Statistics
• NCLIS National Commission on Libraries and
  Information Science Task Force on Network
  Performance Measures

7
E-Metrics Statistics and Measures

• Number of electronic full-text journals
• Number of electronic reference sources
• Number of electronic books
• Number of electronic reference transactions
• Number of logins (sessions) to electronic
  databases
• Number of queries (searches) in electronic
  databases
• Items requested in electronic databases
• Virtual visits to librarys website and catalog
• Cost of electronic full-text journals
• Cost of electronic books

• Library expenditures (and external expenditures)
  for bibliographic utilities, networks and
  consortia
• Size of library digital collection
• Use of library digital collection
• Cost of digital construction and management
• Percentage of electronic reference transactions
  of total reference
• Percentage of virtual visits of all library
  visits
• Percentage of electronic books to all monographs
• Percentage of electronic journals to serial
  subscriptions

8
Focus Groups

• Used to follow-up user surveys
• Good opportunities to get user input on a variety
  of subjects (e.g., public interfaces, student
  areas in the library, new software products)

9
Written and Electronic Comments

• Library users can send comments via email or the
  Librarys web page
• Comments boxes are placed in the main library
• Library staff personally respond to each comment

10
Responding to User Feedback

• The 2001 User Survey as a Case Study
• Brinley Franklin
• Director of University Libraries
• The University of Connecticut
• brinley.franklin_at_uconn.edu