Electronic Performance Measures:

1

• Electronic Performance Measures
• How are we really doing?
• Saturday, October 13
• Karen Howells, City of Calgary
• Bonnie Gray, Alberta Community Development
• Hetty Wilderdijk, Hinton Municipal Library
• Karen Adams, University of Alberta

2
(No Transcript)
3
Why measure use of electronic services?

• Funders and stakeholders expect evidence
• Demonstrating value, cost effectiveness, etc.
• Can no longer rely on the general perceived good
  of libraries.
• Need to make a case for new / increased funding.
• Need to justify ongoing funding.

• To assist library boards and staff to make better
  decisions.
• Determining priorities.
• Monitoring trends.
• Anticipating customer needs and expectations.

4

• To determine if goals of project / service are
  being achieved.

• Leveling off (or in many cases decline) of use of
  traditional resources and services
• With electronic performance measures you can
  demonstrate where a shift rather than just a
  leveling off or decline is occurring.

5
What to Measure
Dont measure something just because you can!
According to our research, 100 of our customers
resent being surveyed at 2 am.!
6
What to Measure

• Electronic equivalents of traditional library use
  measures
• Can be described as separate measures.
• Can be combined with traditional measures.

Choose a few measures that give you a Balanced
Score Card of the outcomes being achieved
through investment in electronic resources and
services.
7
Examples

• Number of customers
• Customer penetration.
• Customer uses.
• Profile of users.

• Qualitative indicators
• Value / impact.
• Satisfaction.
• Importance.
• Customer comments / anecdotes

QUALITY

• Costing
• Cost per customer / transaction.

8
When to Measure

• Before / after implementation.
• Over time.
• Time sample where appropriate.

9

• Dont wait until someone comes up with the
  perfect measure!
• We dont have perfect measures for traditional
  library activities.
• Look for reasonable, reliable estimates.
• or in the words of Charles Babbage

Errors using inadequate data are much less than
errors using no data at all.
10

• How to Measure
• Look for measures and methods already being
  developed.
• APLEN / Province
• CALUPL
• Other libraries

11

• Determine what we need to know
• Then consider methods that meet need.

• Rank importance of measure
• Invest measurement resources into highest
  priorities.
• Cost of more sophisticated methods may or may not
  be justifiable by value of results.

• Cross-section of methods increases reliability
• Every method has inherent strengths and
  weaknesses.

12

• Who Needs to Know?
• Telling the Online Story
• Management/administration
• Boards
• Local government
• Provincial government
• Special grant funders
• Customers

13

• Know your audience and target the message
• Level of detail.
• Which measures (mandated and strategic).
• Choose appropriate and compelling style.

• Carefully choose your words, charts, tables, or
  statistics!
• Accurately describe what results actually
  reflect.
• Particular caution with website log statistics.

• Dont make Benjamin Disraelis famous quote apply
  to your use of electronic performance measures

(There are 3 kinds of lies lies, damn lies,
andstatistics.)
14
STRATEGIC DIRECTION
Summary of Success, Value and Return-on-Investment
Library Jurisdictions
analysis, evaluation planning
APLEN / Provincial analysis, evaluation
planning
PERFORMANCE INDICATORS
Core and Optional Electronic Measures
Customer Satisfaction
staff e-training
formal customer e-training
reasons for use of e-services
informal customer e-training / coaching
virtual library visits
e-reference questions
website pages viewed
in-library e-users
e-resources per capita
subscribed database sessions
DATA
Measurement Tools and Methods
Computer Signup Sheets
Satisfaction Surveys
Website log analysis
In-library use surveys
Questions counts
Training counts
APLEN database tracking
15
More about electronic performance
measures Electronic Performance Measures
Implementation Guide Coming Soon to a Library
Near You!

• For more information on
• Web Logs and Log Analysis Terminology.
• Web Measurement Strategies and Methods.
• EMAIL karen.howells_at_gov.calgary.ab.ca

16
Electronic Performance Measures

• The Provincial Perspective

17
Performance Measures

• Libraries have a long tradition of identifying
  performance indicators, establishing measures,
  collecting statistics and using the statistics to
  plan their future program initiatives.
• Operations such as circulation, reference
  questions, size of collections, program
  attendance are all carefully and thoroughly
  documented and compared to other libraries.

18
Performance Measures

• Measures allow the library to compare its
  performance
• over a number of years
• with other similar libraries

19
Performance Measures

• These measures are used for the annual Public
  Library Statistics.
• Albertas statistics can be compared with those
  of other provinces.

20
Electronic Performance Measures

• The arrival of APLEN has added a new range of
  services to the public library portfolio.
• These new services require a new set of
  information and standard performance measures at
  the provincial level.
• There have to date been few standard measures for
  networked services.

21
Electronic Performance Measures

• Why do we need measures for networked services
  at the provincial level?
• Standard reporting across province
• Management tool for planning and budgeting at the
  provincial level
• Measure and evaluate program
• Justify ongoing funding and service
• Model for other provinces

22
Electronic Performance Measures

• APLEN created an opportunity to identify and test
  performance measures for networked services
• Alberta Community Development contracted with TAL
  on the Statistics and Performance Measures
  Project carried out by Dr. Charles McClure and
  Bruce Fraser

23
Unveiling the NEW Electronic Performance Measures

• The measures have been developed and tested over
  the past year and one half. They include seven
  core performance measures (CPMs) and four
  optional performance measures (OPMs).

24
Unveiling the NEW Electronic Performance Measures

• CORE PERFORMANCE MEASURES (CPMs)
• CPM1- Customer In-Library Electronic Resource
  Sessions
• CPM2 Customer Satisfaction with Electronic
  Library Services and Resources
• CPM3 Internet Workstations per Legal Population

25
Unveiling the NEW Electronic Performance Measures

• CPM4 Electronic Reference Transactions per all
  Reference Transactions
• CPM5 Virtual visits to Librarys Website and
  Catalogue
• CPM6 Number of Public Technology Customer
  Training Sessions
• CPM7 Number of Database Sessions conducted by
  Users

26
Unveiling the NEW Electronic Performance Measures

• OPTIONAL PERFORMANCE MEASURES (OPMs)
• OPM1 Network resources utilized by non-TAL
  members as a percentage of all users
• OPM2- Number of web pages viewed
• OPM3 Staff training
• OPM4 Reasons for internet use by library
  customers

27
Unveiling the NEW Electronic Performance Measures

• Training sessions will be held in each system
  area. System staff will be encouraged to act as
  local trainers.
• The Electronic Performance Measures
  Implementation Guide will be distributed in the
  near future.

28
The NEW Electronic Performance Measures

• Librarians will be asked to do a sample
  collection of the CPMs this year as part of their
  annual library survey.
• CPMs will be incorporated into the Annual Report
  and Survey of Public Libraries in Alberta
  starting in 2002.

29
Electronic Performance Measures
Karen Adams University of Alberta Netspeed 2001
30
Traditional measures

• Item counts
• Service counts
• Expenditures by category

31
So what?
32
ARL New Measures

• Higher Education Outcomes
• Research Review
• Lib(QUAL)
• Investigation of Cost drivers
• A Guide for Measuring Performance of ILL/DD
• E-metrics (measures for electronic resources)

33
E-metrics

• develop, test, and refine selected statistics
• engage in a collaborative effort with selected
  database vendors
• to develop a proposal for external funding to
  maintain the development and refinement of
  networked statistics and performance measures

34
Phases of the project
Hired Information Use Management and Policy
Institute (Chuck McClure) Identify state of the
art re networked services (survey, site visits,
vendor reports) Field test possible statistics
and meet with selected vendors Develop proposal
for external funding
0
1
2
3
35
Phase 2

• Field test to work towards core statistics has
  taken place
• Second draft of manual being written after review
  over summer, 2001

36
Issues

• Culture of assessment
• Systems for data collection analysis
• Staff development training
• Network activities not stand-alone
• Distribution of data among library, consortia and
  vendors
• Modifications to infrastructure

37
Issues (contd)

• Relationship of e-stats to other statistics
  initiatives, and to other ARL research
  activities.
• Standards definitions
• Internal systems
• Organizational structures

38
5 categories to be measured

• Resources
• Use
• Cost
• Local digital collections
• Performance measures

39
Common measures

• Number of logins (sessions) to electronic
  databases
• of electronic reference transactions of total
  reference transactions

40
More information

• www.arl.org/stats/newmeas/newmeas.html
• Karen.Adams_at_ualberta.ca