Building a resource for practical assessment: adding value to value and impact

1
Building a resource for practical assessment
adding value to value and impact

• Stephen Town
• University of York, UK
• Library Assessment Conference, Seattle
• Wednesday 6th August, 2008

2
Summary

• The SCONUL Value Impact Measurement Program
  (VAMP) recap
• The Performance Portal
• Value options
• UK drivers (TRAC)
• Institutional Case The Open Universitys Best
  Value project
• Benchmarking national statistics

3
Introduction recap
4
The University Context (from the 2006 Library
Assessment Conference, after Lombardi)

• Universities have two bottom lines
• Financial (as in business)
• Academic, largely through reputation in
• Research (the priority in leading Universities)
• Teaching ( maybe Learning)

5
Library Pressures for Accountability

• The need is therefore to demonstrate the Library
  contribution in these two dimensions
• Financial, through value for money or related
  measures
• Impact on research, teaching and learning
• This also implies that competitive data will be
  highly valued

6
The Aim Role of Universities their Libraries
cautions for measurement

• Research, Teaching Reductionism
• Mode 1 Research impact transcendental
• Mode 2 Research impact instrumental
• Value, Price Mandarinisation of research and
  its support
• Interdisciplinary research
• Collaborative research across institutions
• Learning as a set of discreet assessed modules
• All of this may damage the idea of Libraries as
  transcendent, collective and connective services

7
SCONUL Member Survey Findings

• 70 undertaken value or impact measurement
• Main rationales are advocacy, service
  improvement, comparison
• Half used in-house methodologies half used
  standard techniques
• Main barrier is lack of tools,
• Creating issues of time and buy-in

8
Member Survey Conclusions

• There is a need to demonstrate value and that
  libraries make a difference
• Measurement needs to show real value
• Need to link to University mission
• Libraries are, and intend to be, ahead of the
  game
• Impact may be difficult or impossible to measure
• All respondents welcomed the programme, and the
  prospect of an available toolkit with robust and
  simple tools

9
VAMP Objectives

• New missing measurement instruments frameworks
• A full coherent framework for performance,
  improvement and innovation
• Persuasive data for University Senior Managers,
  to prove value, impact, comparability, and worth

10
Missing methods

• An impact tool or tools, for both teaching
  learning and research (from the LIRG/SCONUL
  initiative?)
• A robust Value for Money/Economic Impact tool
• Staff measures
• Process operational costing tools

11
Program Benefits

1. Attainment retention of Library institutional
   income
2. Proof of value and impact on education and
   research
3. Evidence of comparability with peer institutions
4. Justification of a continuing role for libraries
   and their staff
5. Meeting national costing requirements for
   separating spend on teaching and research

12
Communities of Practice

• groups of people who share a passion for
  something that they know how to do,and who
  interact regularly to learn how to do it better
• coherence through mutual engagement
• Etienne Wenger, 1998 2002

13
VAMP Project Structure

• Analysis March-June 2006
• Tools I (Impact ) - June 2007
• Site Development - June 2007
• Tools II (Value) - ?
• CoP development
• Maintenance

14
The Performance Portal
15
Community of Practice
Techniques
Members Forum (Blog?Chat?)
VAMP Home Page
Simple Introductions
Detailed Techniques
Techniques in Use (Wiki?)
16
The Performance Portal

• A Wiki of library performance measurement
  containing a number of approaches, each
  (hopefully) with
• A definition
• A method or methods
• Some experience of their use in libraries (or
  links to this)
• The opportunity to discuss use

17
(No Transcript)
18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
Content submission
23
User guide
24
Discussion Tools

• An experiment in social networking Web 2.0
  technologies

25
(No Transcript)
26
(No Transcript)
27
The Ontology of Performance

• Frameworks
• Impact
• Quality
• Statistics
• Value
• A visual Mind map?

28
(No Transcript)
29
Frameworks

• Mounted
• European Framework for Quality Management (EFQM)

• Desired
• Key Performance Indicators
• The Balanced Scorecard
• Critical Success Factors
• The Effective Academic Library

30
Impact

• Mounted
• Impact tools

• Desired
• Detailed UK experience from LIRG/SCONUL
  Initiatives
• Outcome based evaluation
• Information Literacy measurement
• More on research impact

31
(No Transcript)
32
Quality

• Mounted
• Charter Mark
• Customer Surveys
• LibQUAL
• SCONUl Survey
• Priority Research
• Investors in People

• Desired
• Benchmarking
• Quality Assurance
• ISO 9000s
• Investors in People experience
• Opinion meters
• Quality Maturity Model

33
(No Transcript)
34
Statistics

• Mounted
• SCONUL Statistics interactive service
• HELMS statistics

• Desired
• Institutional experience of using SCONUL
  statistics for local advocacy
• COUNTER
• E-resource tools

35
Value

• Mounted

• Desired
• Contingent valuation
• Transparency costing
• Staff process costing, value contribution
• E-resource value tools

36
Value
37
What is value?

• Cost efficiency
• Cost effectiveness
• Cost comparison (Case 3)
• Financial management process standards audit
• Financial allocation (Case 1)
• Valuation
• Value added
• Return on investment
• Best value (Case 2)

38
Case 1 TRAC

• UK Higher Education Transparency initiative
  2000-09

39
Transparent approach to costing

• The standard method for costing in UK HEIs
• Government requirement
• Ending of cross-subsidy (T vs R)
• Research funding based on full economic costing
  (fEC)
• Positive effects on funding
• Positive effect on pricing

40
Implications

• All activity to be identified as research,
  teaching or other
• Library as other? or
• All library activities either research or
  teaching, or a simplistic apportioning to each
• Libraries omitted as a component of research
  costs, and therefore as a share recipient

41
Case 2 the UK Open University Librarys Best
Value Program
42
OU Best Value Program Objectives

• To increase the business skills of library
  managers staff
• To develop skills to support customer-focused,
  cost-efficient management decision making
• To develop benchmarking evaluation skills that
  balance quality, value and cost efficiency

43
Strands

• Business reporting
• Process costing and continuous improvement
• Service planning
• Benchmarking
• to generate real accountability

44
Business reporting elements

• Library business areas
• Five PIs per area, including cost, quality
  customer impact
• Forecast, variance remedial action
• Has improved use of management information,
  efficiency, prioritisation and expenditure control

45
Process Costing

• Complete process and stage costing
• Average times and skill levels
• Included enquiries, cataloguing, e-resources, IT
  support, document delivery, counter services
• Has delivered justification for staffing levels
  against activity, staffing formulae, redeployment
  to priority areas, and process improvements

46
Service plans

• Costed service plans to achieve medium term
  improvement and development through a rolling
  program
• Included document delivery, enquiries,
  information literacy, and e-resources

47
Program benefits and outcomes

• Staff development
• cost-conscious decision-making
• business skills
• Management information improvement
• Clarity about customers and use
• Improved quality
• Ability to sell benefits

48
Case 3 Financial benchmarking
49
International Benchmarking initiatives

• OU able to engage and lead an exercise against
  distance education Universities worldwide
• In one 2008 international benchmarking study
• Only one institution (out of eight) had a
  comprehensive costing model

50
Financial Statistical Convergence

• York Meeting, 2008
• OCLC/RLG
• ARL
• SCONUL
• CAUL

51
Conclusion Questions

• What do mean by value?
• Why do we not yet have a collective view on
  costing approaches?
• Skills deficiency?
• Lack of real need or real financial performance
  accountability?
• We would rather not know?
• Are we more intent on increasing budgets than
  seeking efficiency improvement?

52
Acknowledgments

• The VAMP Subgroup of SCONUL WGPI
• Maxine Melling, Philip Payne, Rupert Wood
• The Cranfield VAMP Team, Darien Rossiter, Michael
  Davis, Selena Lock, Heather Regan
• The Open University, Ann Davies
• Value Consultants, Sue Boorman, Larraine Cooper
• Attendees at the York Statistics meeting

53
(No Transcript)