The SCONUL Value and Impact Measurement Programme VAMP: a progress report

1
The SCONUL Value and Impact Measurement Programme
(VAMP) a progress report

• Stephen Town
• (Cranfield University)
• Project Manager, VAMP
• SCONUL Autumn Conference
• British Library Conference Centre28th November,
  2006

2
Introduction

• Advocacy Drivers VAMP
• Report on progress to date
• Definition of the next steps

3
The University Context (from the Library
Assessment Conference, Charlottesville, Va,
September 2006)

• Universities have two bottom lines
• Financial (as in business)
• Academic, largely through reputation in
• Research (the priority in leading Universities)
• Teaching ( maybe Learning)

4
Library Pressures for Accountability

• The need is therefore to demonstrate the Library
  contribution in these two dimensions
• Financial, through value for money or related
  measures
• Impact on research, teaching and learning
• This also implies that competitive data will be
  highly valued

5
The SCONUL Response

• The SCONUL Working Group on Performance
  Improvement
• Ten years of toolkit development to assist in
  performance measurement and improvement (for both
  management and advocacy)
• SCONUL Top concern survey 2005, leading to VAMP

6
Examples of tools developed 1

• Integration
• Efficiency Comparability

• Quality assurance Guidelines
• SCONUL Statistics interactive service
• HELMS national performance indicators
• E-measures project
• Benchmarking Manual

7
Examples of tools developed 2

• Satisfaction
• Impact

• SCONUL Satisfaction Survey
• SCONUL LibQUAL Consortium
• LIRG/SCONUL Impact Initiative
• Information Literacy Success Factors

8
VAMP Objectives

• New missing measurement instruments tools
• A full coherent framework for performance,
  improvement and innovation
• Persuasive data for University Senior Managers,
  to prove value, impact, comparability, and worth

9
Missing methods?

• An Impact tool, for both teaching learning and
  research (based on the LIRG/SCONUL Impact
  Initiative?)
• A robust Value for Money/Economic Impact tool
• Staff measures
• Process operational costing methods

10
Benefits?

• Attainment retention of Library institutional
  income
• Proof of value and impact on education and
  research
• Evidence of comparability with peer institutions
• Justification of a continuing role for libraries
  and their staff

11
VAMP Project Structure

• Phase 1 (March-June 2006)
• Critical review
• SCONUL Member Survey
• Gap analysis synthesis
• SCONUL Conference Workshops
• Phases 2 3 (July 2006 - April 2007)
• Development of new measures techniques
• Review and re-branding of existing tools
• Web site development
• Dissemination maintenance strategy

12
Critical Review
13
Critical Review Method

• Review of
• SCONUL initiated or promoted services
• Other UK European initiatives
• Initiatives from other UK library sectors
• International initiatives
• starting from the perspective of The Effective
  Academic Library, 1995

14
Review Findings

• The Impact Initiative work will be key, but needs
  to be solidified and embedded
• Eight JISC/EU projects in the last eight years
  relevant to the assessment of value and impact,
  many relating to e-resources
• Significant work in the USA, Australia and South
  Africa

15
Review Conclusions

• Vast amount of relevant work, but without wide
  take up
• More critical analysis required of most products
  and tools
• Further development and simplification required
  to create credible and applicable instruments for
  SCONUL members

16
SCONUL Member Survey
17
Member Survey Findings

• 38 respondents 27 of population
• 70 undertaken value or impact measurement
• Main rationales are advocacy, service
  improvement, comparison
• Half used in-house methodologies half used
  standard techniques
• Main barrier is lack of tools, making time an
  issue
• Buy-in of stakeholders is an issue

18
Member Survey Conclusions

• There is a need to demonstrate value and that
  libraries make a difference
• Measurement needs to show real value
• Need to link to University mission
• Libraries are, and intend to be, ahead of the
  game
• Impact may be difficult or impossible to measure
• All respondents welcomed the programme, and the
  prospect of an available toolkit

19
Synthesis, Expert Commentary, SCONUL Conference
Workshops
20
Synthesis

• Some terminological confusion?
• Is impact effect or to outcome?
• Higher order effects and level
• Individual, course, institutional, vocational,
  societal, national, international
• Value and impact are not an item?
• Value, adding value, value for money,
  cost-effectiveness

21
Expert Comment

• Return to why?
• Advocacy or management?
• Critical Gap in measurement
• Effect of the Library on educational attainment
• Effect of the Library on research attainment
• Robust and simple tools
• Only a few existing tools effective, so simplify
  and focus the range of offerings

22
SCONUL Conference Workshops

• Accountability to a variety of structures and
  individuals
• therefore a range of approaches required
• SCONUL Statistics heavily used
• Directors want help with all VAMP lines
• Pedagogic Big project needed?
• Re-engineer processes rather than measure!

23
Overall conclusions of Phase 1

• Wider sectoral involvement?
• Health Evidence-based methods
• National MLA Measures
• British Library Contingent Valuation
• Not only new measures
• but also supporting directing processes

24
VAMP Phases 2 3
25
Next Steps 1

• Content Products
• 2.1 Value Impact Guidelines
• 2.1.1 Institutional Value (eg VFM Economic
  Impact)
• 2.1.2 Impact on Teaching Learning
• 2.1.3 Impact on Research

26
Next Steps 2

• Content Products
• 2.2 Staffing Operational Measures Guidelines
• 2.2.1 Staff Costing
• 2.2.2 Staff Added Value measures
• 2.2.3 Other operational costing methods
• 2.3 Re-branding packaging of existing tools

27
Next Steps 3

• Process Products
• 3.1 Web Site
• 3.2 Community of practice establishment
• 3.3 Maintenance sustainability strategy

28
Progress on Content 1

• 2.1.1 Institutional Value (eg VFM Economic
  Impact)
• No source identified for VFM tool
• Contingent Valuation method in negotiation
• 2.1.2 Impact on Teaching Learning
• 2.1.3 Impact on Research
• Tools for both areas commissioned (Information
  Management Associates)

29
Progress on Content 2

• 2.2.1 Staff Costing
• 2.2.2 Staff Added Value measures
• 2.2.3 Other operational costing methods
• Staff operational costing method in negotiation
  
• No source identified for Staff Added-Value tool
• 2.3 Re-branding packaging of existing tools
• Mainly included within the Web Site development

30
Existing or Identified Methods/Tools

• Benchmarking
• Charter Mark
• Critical Success factors
• Economic Impact
• HELMS
• Impact Measurement
• Information Literacy Measurement
• Investors in People
• JISC Studies

• KPIs
• LibQUAL
• Process Costing
• Quality Assurance
• Quality Maturity Model
• Research Impact
• SCONUL Satisfaction Survey
• SCONUL Statistics
• Teaching Learning Impact
• Transparency Costing

31
Progress on Process

• 3.1 Web Site
• Web Site commissioned (Cranfield)
• 3.2 Community of practice establishment
• Invitations in progress

32
Communities of Practice

• groups of people who share a passion for
  something that they know how to do,and who
  interact regularly to learn how to do it better
• coherence through mutual engagement
• Etienne Wenger, 1998 2002

33
Community of Practice
Techniques
Members Forum (Blog?)
VAMP Home Page
Simple Introductions
Detailed Techniques
Techniques in Use (Wiki?)
34
Acknowledgments

• Angela Conyers, Evidence Base, UCE
• Claire Creaser Suzanne Lockyer, LISU,
  Loughborough University
• Professor Peter Brophy, Manchester Metropolitan
  University
• The VAMP Subgroup of SCONUL WGPI
• Maxine Melling, Philip Payne, Rupert Wood

35
(No Transcript)