Making Incremental Improvements to Public Library Comparative Statistical Practices Ray Lyons Jason Holmes Library Assessment Conference Seattle, Washington August 5, 2008 - PowerPoint PPT Presentation

Loading...

PPT – Making Incremental Improvements to Public Library Comparative Statistical Practices Ray Lyons Jason Holmes Library Assessment Conference Seattle, Washington August 5, 2008 PowerPoint presentation | free to download - id: 64ccbf-MDRlM



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Making Incremental Improvements to Public Library Comparative Statistical Practices Ray Lyons Jason Holmes Library Assessment Conference Seattle, Washington August 5, 2008

Description:

Title: Using Comparative Public Library Statistics to Your Library's Benefit Ray Lyons Ohio Library Council Management Conference May 21-22, 2008 – PowerPoint PPT presentation

Number of Views:24
Avg rating:3.0/5.0
Date added: 10 June 2020
Slides: 38
Provided by: ray8162
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Making Incremental Improvements to Public Library Comparative Statistical Practices Ray Lyons Jason Holmes Library Assessment Conference Seattle, Washington August 5, 2008


1
Making Incremental Improvements to Public
Library Comparative Statistical PracticesRay
LyonsJason HolmesLibrary Assessment
ConferenceSeattle, Washington August 5, 2008
2
The Basic Problem
  • The ultimate goal of the library is public
    enlightenment.
  • It is difficult to assess our impact on
    enlightenment of the community because we have no
    way to measure it.
  • We measure what we CAN measure
  • We compare what we CAN compare

3
Context of Comparative Statistics
  • Assessment
  • Measures used as part of a more general process
    of assessing library value and effectiveness
  • Management practice
  • Measures intended for iterative and ongoing
    process of performance measurement

4
Public Library Assessment
  • Library profession traditionally applies systems
    or industrial model
  • Inputs resources supplied to the library
  • Outputs products and services

Throughput
Input
Output
5
Performance Measurement Model
RESULTS
EFFORTS
Intermediate Outcomes
End Outcomes
Inputs
Outputs
Outcome Measures
Outcome Measures
6
Performance Measurement Steps
  1. Define long term goals (desired outcomes)
  2. Define medium and short term objectives
  3. Develop programs aimed at objectives
  4. Specify measurement indicators
  5. Monitor indicators to track accomplishments

7
Rationale for Standardized Statistics
  • PLA Planning-for-Results approach to library
    management
  • Abandonment of established operational and
    performance standards
  • ALA / PLA 1987 publication, Output Measures for
    Public Libraries A Manual of Standardized
    Procedures, defines standard statistics and
    collection procedures

8
ALA / PLA Approach to Standardized Statistics
  • Useful for self-evaluation based on service
    response choices library makes
  • Should be interpreted with respect to library
    mission, goals, objectives
  • Interpretation left up to the library

9
Current Practices in Comparative Library
Statistics
  • How are library statistics currently used for
    comparing public libraries?
  • What are the bases for these uses?
  • What purposes do they serve?

10
Survey of Ohio Public Libraries on Use of
Comparative Statistical Measures
  • Exploratory study
  • Available at http//www.plstatreports.com/c
    ompare/
  • UsePerceptComparStats.pdf
  • Responses via interview or online questionnaire
  • Stratified random sample of 90 Ohio public
    libraries
  • Two strata urban rural counties
  • Response rate 47 (42 libraries)

11
Survey Findings
12
Survey Findings
13
Survey Findings Frequency of Managerial Review
of Input Measures
Frequency Input Measure
Annually Quarterly Monthly Weekly
Rarely Not Sure
Operating expenditures 12 9 65 15 0 0
Print Mat. Expenditures 12 18 62 9 0 0
Electronic Mat. Expenditures 39 21 36 3 0 0
Print Materials 56 9 29 3 3 0
Print Subscriptions 74 12 15 0 0 0
Audio/Video Materials 58 9 30 0 3 0
Databases 63 14 6 6 6 6
Internet terminals 51 11 14 9 6 9
FTE Staff 79 6 12 0 3 0
NOTE Sky blue highlighting indicates measures
that 50 or more library managerial teams review
periodically. Light blue highlighting indicates
higher frequencies that, combined, total 50 or
more.
14
Survey FindingsFrequency of Managerial Review
of Output Measures
Frequency Output Measure
Annually Quarterly Monthly Weekly
Rarely Not Sure
Circulation 9 0 83 9 0 0
In-house Mat. Use 21 12 26 3 34 0
Interlibrary loan 23 6 66 6 0 0
Visits 21 15 54 6 3 0
Reference Transactions 32 24 34 0 9 0
Program attendance 32 9 51 6 0 0
Electronic Mat. Use 19 6 60 0 9 0
Internet Terminal Use 15 6 66 3 3 3
Website Use 12 9 63 0 9 3
NOTE Sky blue highlighting indicates measures
that 50 or more library managerial teams review
periodically. Light blue highlighting indicates
higher frequencies that, combined, total 50 or
more.
15
Survey Findings
16
Survey Findings
17
Survey Findings
18
Survey Findings Statistical Measures Libraries
Usein Comparisons with Other Libraries (Table
Format)
Measure of
Libraries
Using Measure
Measure of
Libraries
Using Measure
Material Expenditures 100.0
Circulation 96.8
Operating Expenditures 90.3
FTE Staff 77.4
Print Material Counts 48.4
Audio/Video Material Counts 41.9
Databases Available 41.9
Visits 38.7
Subscriptions 35.5
Interlibrary Loans 35.5
Electronic Materials Expenditures 32.3
Librarians 32.3
Program attendance 29.0
Internet terminals 25.8
Reference Transactions 25.8
Electronic Materials Usage 12.9
Other (borrowers, salaries, etc.) 12.9
In-house Material Usage 9.7
Internet Terminal Usage 9.7
Website Usage 9.7
19
  • Interpreting Library Measures

There are no right or wrong scores on an
output measure high and low values are
relative. The scores must be interpreted in terms
of library goals, scores on other measures, and a
broad range of other factors. - Van
House, Weill, and McClure (1990)
20
Interpreting Library Measures
  • ALA / PLA policy since 1980s
  • Leave data interpretation
  • to local library
  • Each library staff should decide for
    them-selves whether the statistical findings
    for that library were acceptable in terms of
    performance expectations.
  • - Ellen Altman (1990) describing the Public
    Library Performance Measurement Study by Deprospo
    et al. (1973)

21
Key Problems with Library Statistics
  • Lack of criteria for evaluating measures
  • Collection of standard statistics assumes all
    library resources/activities counted to be
    equivalent

22
Key Problems with Library Statistics
  • Standardization ignores differences in

- Complexity - Sophistication - Relevance -
Quality (Merit)
- Value (Worth) - Effectiveness - Efficiency -
Significance
23
Key Problems with Library Statistics
  • Data imprecision due to
  • Inconsistent collection methods
  • Mistakes
  • Sampling error
  • Gaming
  • Statistical imputation
  • Imprecision makes individual library comparisons
    less valid

24
Key Problems with Library Statistics
  • Lack of reliable methods for identifying peer
    libraries
  • Comparisons are either approximate or inaccurate
  • Can result in incorrect or misleading conclusions
  • Data are self-reported and unaudited

25
Key Problems with Library Statistics
  • The More-is-Better Myth
  • Views higher numbers as favorable performance,
    lower as unfavorable
  • More activity does not necessarily mean better
    activity
  • - Van House, Weill, and McClure (1990)
  • Striving to earn higher numbers may compromise
    service quality

26
Key Problems with Library Statistics
  • Statistics say nothing about performance
    adequacy, quality, effectiveness, or efficiency
    of library resources/activities
  • No consensus on constructs that statistics can
    realistically reflect
  • Difficult to determine remedies for problems
    which statistics might reveal

27
Key Problems with Library Statistics
  • Variety of reasons for insufficient scores

- Inadequate knowledge of community needs -
Staff skill deficiencies - Inadequate staffing
  • - Inefficient workflows
  • Inadequate planning
  • - Limited user
  • competencies
  • . . . and others

Adapted from Poll and te Boekhorst (2007)
28
  • Output measures reflect the interaction of
    users and library resources, constrained by the
    environment in which they operate. The meaning of
    a specific score on any measure depends on a
    broad range of factors including the librarys
    goals, the current circumstances of the library
    and its environment, the users, the manner in
    which the measure was constructed, and how the
    data were collected. emphasis added
  • - Van House, Weill, and McClure
    (1990)

29
Policy-Level Problems withLibrary Statistics
  • PLA managing-for-results approach has produced
    undesirable results
  • Confusion about meanings of statistical
    indicators
  • Expectations that local libraries are able to
    interpret data productively have been too
    optimistic

30
Policy-Level Problems withLibrary Statistics
  • Exaggerated or inaccurate advocacy campaigns
    undermine credibility of assessment process,
    methods, and data
  • Biased advocacy studies at cross-purposes with
    need for accountability

31
Negative Side-Effects of Library Advocacy
Practices
  • Advocacy narratives can dumb down data analysis
    and assessment efforts
  • Encourage absurd interpretations of library
    statistics
  • Misuse key assessment terms and concepts
  • Promote unjustifiable conclusions drawn from
    studies that have employed limited research
    methods

32
Improvements Needed
  • Commit to ensuring credibility of assessment data
    and performance measurement findings
  • Discourage naïve, disingenuous, and unsupportable
    use of statistics or assessment findings
  • Specify professions ideology regarding rules of
    evidence

33
Improvements Needed
  • Fuller understanding of limitations of
    statistical indicators and comparison methods
  • Discourage describing performance solely based on
    standard statistics
  • The measures are best used with other
    information about the library.
  • - Van House,
    Weill, and McClure (1990)

34
Improvements Needed
  • Relate levels of resources/services to levels of
    community needs
  • Explore relationships among entire set
  • of standard statistical indicators, i.e.
    complementary and conflicting dimensions

35
Improvements Needed
  • Identify peer libraries using multiple
    indicators
  • Community population library budget
  • key demographic characteristics
  • Explore feasibility of alternative sets of
    indicators depending on library type, size,
    mission, etc.

36
Improvements Needed
  • Increased understanding of measurement and
    interpretation
  • Draw reasonable conclusions and interpretations
  • Basic behavioral science measurement practices

37
Behavioral Science Measurement Model
Conceptualization
Nominal Definition
Operational Definition
Measurement in Real World
Babbie (2007)
38
References
Ellen Altman. Reflections on Performance
Measures Fifteen Years Later In Library
Performance, Accountability, and Responsiveness
Essays in honor of Ernest R. DeProspo, C.C.
Curran and F.W. Summers, eds. (Norwood, NJ
Ablex, 1990) Earl Babbie, The Practice of
Social Research, 11th ed. (Beaumont, California
Thomson, 2007) Roswitha Poll and Peter te
Boekhorst. Measuring Quality Performance
Measurement in Libraries, 2nd ed. (Munich KF
Saur, 2007) Nancy A. Van House et al. Output
Measures for Public Libraries A Manual of
Standardized Procedures, 2nd ed. (Chicago
American Library Association, 1987) Nancy A.
Van House, Beth T. Weill, and Charles R. McClure,
Library Performance A Practical Approach
(Chicago American Library Association, 1990)
About PowerShow.com