Making Library Assessment Work Practical Approaches for Developing and Sustaining Effective Assessme

1
Making Library Assessment WorkPractical
Approaches for Developing and Sustaining
Effective Assessment
Steve Hiller Director, Assessment Planning,
University of Washington Libraries ARL Visiting
Program Officer 2004-06 Jim Self Director,
Management Information Services, University of
Virginia Library ARL Visiting Program Officer
2004-06 University of Minnesota Libraries March
23, 2006

2
Why Assess?

• Accountability and justification
• Measure progress/achievement
• Improvement of services
• Comparison with others
• Identify changing use and needs patterns
• Remote/In-person
• Relevancy of current services and resources
• New services
• Marketing and promotion
• Decisions based on data, not assumptions
• Assumicide!

3
Customer-Centered Library and the Culture of
Assessment

• Customer-Centered Library
• All services and activities are viewed through
  the eyes of the customers
• Customers determine quality
• Library services and resources add value to the
  customer

• Culture of Assessment
• Organizational environment in which decisions are
  based on facts, research and analysis,
• Services are planned and delivered to maximize
  positive customer outcomes

User-centered libraries collect data and use
them as the basis for decision-making rather than
rely on subjective impressions and opinions
(Stoffle, Renaud and Veldof, Choosing Our
Futures, CRL, 1996)

4
New Measures . . . But Libraries Have Trouble
Using Data Effectively

• Reasons include
• Library leadership
• Organizational culture
• Library priorities
• Resource sufficiency
• Data infrastructure
• Assessment skills and expertise
• Sustainability
• Presenting results
• Using results to improve libraries

5
Making Library Assessment Work Practical
Approaches for Developing and Sustaining
Effective Assessment

• ARL program to assist libraries with moving
  assessment efforts forward
• Recognition that libraries are aware of the
  value of assessment but find it difficult to
  sustain and use results
• Conduct site visits to evaluate assessment and
  consult on developing practical approaches to
  effective assessment
• Identify common barriers and facilitators to
  assessment
• 7 libraries participated in Phase I Spring 2005
  18 are part of Phase II (2005 - 2006)

6
Good Assessment Starts Before You Begin . . .
Some Questions to Ask

• Define the question
• What do you need to know and why
• What will it cost who will do the work
• How will you use the information
• Where will you get the information
• Existing data
• New data (where or who will you get it from)
• How will you get the information
• How will you analyze and present the information
• Who will act upon the information

7
Effective Assessment

• Is customer-focused
• Uses diverse samples/representative groups of
  users
• Asks fair and unbiased queries
• Assesses what is important
• Develops criteria for success
• Uses multiple assessment methods
• Uses corroboration from other sources
• Provides results that can be used to improve
  libraries

8
Sustainable Assessment Needs . . .

• Organizational leadership
• Sufficient resources
• Supportive organizational culture
• Identifiable organizational responsibility for
  assessment
• Link to strategic planning/library priorities
• Iterative process of data collection, analysis,
  and use
• Customers, staff and stakeholders involved

9
Making Sense of the Numbers

• You look at a number and dont say thats a
  fact. You want to say where did it come from,
  who generated it, why, is it consistent with what
  we would get from looking at other sources, does
  it make sense? What sort of insight can the
  quantitative record give to the qualitative one.
• Susan Carter, Co-Editor, Historical Statistics of
  the United States (2006), as quoted in the New
  York Times, February 22, 2006

10
Practical Assessment Results That Can Be Used!

• Keep it simple and focused less is more
• Know when good nuff is good enough
• Use assessment where it can add value to
  customers
• Present results so they are understandable
• Libraries organized to act on results
• A frequent mistake is to overanalyse and
  overcomplicate. . . The key is to remember what
  questions you are asking and pick your analysis
  to fit. Above all, keep it simple!
• Claire Creaser, Deputy Director and Senior
  Statistician, LISU, Loughborough University, UK

11
Two Approaches to Effective and Sustainable
Assessment

• User needs assessment
• Methodological diversity
• Distributed responsibility
• Management Information
• Balanced scorecard
• Methodological diversity

12
University of WashingtonA Typical Day in Seattle

• Located in beautiful Seattle metro pop. 2.5
  million
• Comprehensive research university
• 27,000 undergraduate students
• 11,000 graduate and professional students
• 4,000 research/teaching faculty
• 800 million annually in federal research
  funding
• Large comprehensive research library system
• 35 million annual budget
• 140 librarians

13
Growing Assessment at UWFrom Project-Based to
Ongoing and Sustainable

• Libraries first strategic plan in 1991 called
  for survey
• Initial large scale library survey done in 1992
  as one-time project
• Library Services Committee formed in 1993
• Conducted in-library use surveys in 1993,1996,
  triennial survey in 1995
• Developed Libraries Service Policy in 1995
• Library Assessment Group appointed in 1997
• Focus groups, observation studies, in-library and
  triennial surveys
• Assessment efforts arise in other library areas
  1997-
• E-Metrics, usability, Web design
• Library Assessment Coordinator (50) appointed
  1999
• Assessment Planning (1.5 librarians)
  established 2006

14
Value of Organizational Structure for Assessment

• Awareness/coordination
• Responsibility
• Advocacy/support
• Best practices/expertise
• Prioritization
• Communication/reporting internally and externally
• Move from project-based to ongoing, sustainable
  program

15
UW Libraries AssessmentMethodological Diversity

• Large scale user surveys every 3 years
  (triennial survey) 1992, 1995, 1998, 2001,
  2004
• In-library use surveys every 3 years beginning
  1993
• LibQUAL each year 2000-2003
• Focus groups on varied topics (annually since
  1998)
• Interviews
• Observation (guided and non-obtrusive)
• Usability
• E-Metrics
• Internal surveys and focus groups

16
UW Triennial Library Use Survey Number of
Respondents and Response Rate 1992-2004Large
number of respondents allows for analysis within
groups
17
Use Libraries at Least Once Per Week(Remote and
In-Person) 1998 - 2004 by Group
18
Use Libraries at Least Twice Per Week(Combined
Remote/In-Person) 1998 - 2004 by Group
19
Resources Used Remotely Faculty and Grad
Students 2001/2004 (at least twice per week)
20
2004 Top Library Priorities at UWComparison of
Faculty and Students
21
2004 Priorities Vary by Academic AreaFaculty and
Graduate Students
22
Data-Based Decision Making

• Service Location Evaluation (2002-03)
• Change in use patterns of science branch
  libraries
• Viability of branch libraries
• Identify library to close and consolidate
• Biosciences Resources and Services (2005-06)
• Current support split between Health
  Sciences/Sciences
• Growth in interdisciplinary research
• Significant change in use patterns
• Libraries connection to the research enterprise

23
UW Libraries Print/Media Circulation 1995-96 To
2004-05
24
Assessing Branch Library ViabilityData-Based
Decision Making

• Performance Indicators developed in 4 broad
  categories
• Use
• Print materials use/loans, photocopies, reference
  questions, gate counts
• Primary user population
• Faculty, grad students, undergraduates, change
  over time
• Facility quality
• Quality of space for users, collections, and
  staff
• Physical library dependency of primary user group
• Importance of books and older journals, use of
  other libraries, frequency of in-person library
  visits, core journals available electronically

25

26
Science/Engineering Faculty Libraries Used
Regularly (2004 Survey Data)
27
(No Transcript)
28
Health Sciences Faculty Library Use By Funding
Source (2004, at least 2x per week)
29
Biosciences Review Task Force (2005-06)

• Review Libraries support and connection to
  bioscience teaching, learning and research
• Recommend how UW Libraries can best support
  bioscience programs
• Employ a customer-centered approach consulting
  widely with the biosciences community
• Examine how other research libraries provide
  resources and services to the biosciences

30
Biosciences Review Process

• Define biosciences Dec 2005
• Mine existing data
• Survey, use, institutional, peer Jan-May
  2006
• Acquire new information
• Interviews (library staff) Jan-Apr 2006
• Interviews (bioscience faculty Feb 2006
• Focus groups (biosci faculty students) Mar-Apr
  2006
• Peer library surveys/interviews
  April-May 2006
• Synthesis and first report draft May-June
  2006
• Reaction and revision
• Final report and recommendations June-Aug
  2006

31
Themes from Faculty Interviews

• 10 interviewed
• 8 in Health Sciences/Basic Sciences and 2 in AS
  Science
• Library seen primarily as E-Journal provider
• Especially by lab researchers
• Few non-prompted ideas about new services
• Physical library used only for items not
  available online
• The information search starts with Google and Pub
  Med
• Too busy for training, instruction etc.
• Faculty who teach seem to use libraries
  differently
• Follow up with 7 focus groups
• 3 faculty, 2 grad students, 2 undergrads

32
Data Mining Library Use by Group and Academic
Area
33
How UW Has Used Assessment Information

• Make our physical libraries student places
• Extend hours in Undergraduate Library
• Create more diversified student learning spaces
• Address student information technology needs
• Eliminate print copies of journals received
  online
• Enhance resource discovery tools
• Provide standardized service training for all
  staff
• Stop doing activities that do not add value to
  users
• Consolidate and merge branch libraries
• Improve internal and external communications

34
Overall Library Satisfaction by UW Group
1995-2004
35
The University of Virginia

• 13,000 undergraduates
• 65 from Virginia, 35 out of state
• Most notable for liberal arts
• 7,000 graduate and professional students
• Prominent in humanities, law, business
• Located in Charlottesville
• Small city in metro area of 160,000

36
University of Virginia LibraryOrganizing for
Assessment

• Management Information Services unit
• Established in 1996
• Currently 3 staff
• Resource for library management and staff
• Advocates for sustainable assessment
• Centralized data collection, analysis and
  compilation
• Multifaceted approaches

37
Collecting the Data at U.Va.

• Customer Surveys
• Staff Surveys
• Mining Existing Records
• Comparisons with peers
• Qualitative techniques

38
but to suppose that the facts, once established
in all their fullness, will speak for
themselves is an illusion.

• Carl Becker
• Annual Address of the President of the
• American Historical Association, 1931

39
Making the facts Meaningful

• Summarize
• Compare
• Analyze
• Present

40
at their most basic, statistics are simply
numbers with context.

• Claire Creaser
• Statistics for the Faint-Hearted
• Statistics in Practice Measuring and Managing
  2002

41
Customer Surveys

• Faculty
• 1993, 1996, 2000, 2004
• Separate analysis for each academic unit
• Response rates 59 to 70
• Students
• 1994, 1998, 2001, 2005
• Separate analysis for grads and undergrads
• Undergrad response rates 43 to 50
• Grad response rates 54 to 63

42
(No Transcript)
43
Faculty Priorities1993 to 2004
44
Top Priorities at UVaComparison of Faculty and
Students
45
Priorities for increased or improved library
space
46
Data Mining

• Acquisitions
• Circulation
• Finance
• University Records

47
Acquisitions Expenditures by Format
University of Virginia Library
48
University of Virginia Library Serving the
Customer
49
University of Virginia Library Serving the
Customer
50
The Balanced ScorecardManaging and Assessing Data

• The Balanced Scorecard is a layered and
  categorized instrument that
• Identifies the important statistics
• Ensures a proper balance
• Organizes multiple statistics into an
  intelligible framework

51
Metrics

• Specific targets indicating full success, partial
  success, and failure
• At the end of the year we know if we have met our
  target for each metric
• The metric may be a complex measure encompassing
  several elements

52
What Do We Measure?

• Customer survey ratings
• Staff survey ratings
• Timeliness and cost of service
• Usability testing of web resources
• Success in fundraising
• Comparisons with peers

53
Metric U.1.A Overall Rating in Student and
Faculty Surveys.

• Target1 A score of at least 4.00 (out of 5.00)
  from each of the major constituencies.
• Target2 A score of at least 3.90 from each of
  the major constituencies.
• FY05 Result Target1
• Undergraduates 4.08
• Graduate Students 4.13

54
Metric U.4.B Turnaround time for user requests

• Target1 75 of user requests for new books
  should be filled within 7 days.
• Target2 50 of user requests for new books
  should be filled within 7 days.
• Result FY05 Target2.
• 62 filled within 7 days.

55
Metric U.3.A Circulation of New Monographs

• Target1 60 of newly cataloged cataloged
  monographs should circulate within two years.
• Target2 50 of new monographs should circulate
  within two years.
• Result FY05 Target2.
• 59 circulated.

56
Using Data for Results at UVa

• Additional resources for the science libraries
  (1994)
• Redefinition of collection development (1996)
• Initiative to improve shelving (1999)
• Undergraduate library open 24 hours (2000)
• Additional resources for the Fine Arts Library
  (2000)
• Support for transition from print to e-journals
  (2004)

57
in conclusion Assessment is not

• Free and easy
• A one-time effort
• A complete diagnosis
• A roadmap to the future

58
Assessment is

• A way to improve
• An opportunity to know our customers
• A chance to tell our own story
• A positive experience

59
Moving Forward

• Keep expectations reasonable and achievable
• Strive for accuracy and honestynot perfection
• Assess what is important
• Use the data to improve
• Keep everyone involved and informed
• Focus on the customer

60
For more information

• Steve Hiller
• hiller_at_u.washington.edu
• www.lib.washington.edu/assessment/
• Jim Self
• self_at_virginia.edu
• www.lib.virginia.edu/mis
• www.lib.virginia.edu/bsc
• ARL Assessment Project
• www.arl.org/stats/Hiller_Self.html