What difference has it made?

1
What difference has it made?

• Evaluation and impact assessment for NOF-digitise
  projects
• Susi Woodhouse, Senior Network Adviser

2
Why evaluation?

• NOF digitise is the largest programme of
  publicly-funded content creation so far supported
  by the Government and as such will deliver a
  ground-breaking framework of resources to support
  learning in its broadest sense and will provide a
  foundation for the future.

3
Proof of concept

• Demonstrate value and impact
• a political tool
• a financial lever
• a service development baseline
• added value and additionality
• Learn from our actions
• Share lessons and management models
• Plan for a sustainable future

4
Many stakeholders

• Government (national and local)
• Strategic agencies (Resource, Regional bodies)
• Funding bodies (NOF)
• Cross-sectoral involvement (libraries, museums,
  archives, education, community,voluntary...)
• and USERS, users, users...

5
Measuring Performance
Learning packages used
Fulfilled Citizens
Items digitised
No. of web visits
spent
6
What is Impact?

• An effect on an individual or group which
• may be positive or negative
• may be what was intended or unintentional
• may result in changed
• attitudes
• behaviours
• products
• may be short or long term
• may be critical or trivial
• .and so on

7
Impact and Learning

• Surface learning
• Skim, memorise, regurgitate for tests
• Little long-term impact
• Deep learning
• Involved, questioning, interactive, integration
  with own knowledge
• Significant long-term impact
• A habit of critical thinking

8
Learning and Critical Thinking

• Problem identification
• A triggering event arouses interest
• Problem definition
• The subject is clarified ways of exploring it
  are identified links are made to personal
  experience
• Problem exploration
• New approaches new solutions are explored
  issues are understood ideas are disentangled
• Problem applicability
• Solutions are judged practical knowledge is
  assessed
• Problem integration
• Solutions are acted upon ideas are applied

(After Garrison, 1992)
9
Levels of Impact
6 Changed action 5 Changed world view 4
Changed perception 3 Improved knowledge 2
Better Informed 1 Awareness raised 0
None -1 Dismissive -2 Hostility
10
Measuring Impact

• Who is making judgements?
• the user?
• the institution?
• the government?
• society?
• What does each of them mean by good?
• What does each of them think the project is for?

11
What are digi projects for?What would positive
impacts look like?

• The website
• A statement which draws attention to its
  producers
• Happy users, made so by a well-designed,
  easy-to-use product that engages interest

12
What are digi projects for?What would positive
impacts look like?

• Sustainability and visibility
• Objects still available, in good condition, in
  100 years time
• Use of digital surrogates raises awareness of
  originals and creates greater demand for access
  to them and other items
• promotes role and purpose of host institutions
• other institutions wish to contribute materials
  to the site

13
What are digi projects for?What would positive
impacts look like?

• To provide access knowledge
• Sought information immediately available
• Unknown sources brought to users attention
• Facilities and services universally available
• materials re-purposed by others (e.g. schools,
  broadcasting, private sector)

14
What are digi projects for?What would positive
impacts look like?

• Supporting learning
• Increased literacy of users who are not exposed
  to other learning
• learners using digi materials pass examinations
• evidence of increased confidence and sense of
  wellbeing in users

15
What are digi projects for?What would positive
impacts look like?

• Interpretation, selection and context-setting of
  materials
• Users not left to fend for themselves.
• No information overload and therefore less
  stressed than the average googlebug or Yahoo
  Henrietta
• CONCLUSION Before you analyse your project, be
  clear what it is for!

16
What do we measure?

• Impacts
• We CANNOT measure impact directly
• We CAN measure surrogates of impact
• The real questions are about determining which
  surrogates are adequate for our purposes

17
Some surrogates

• How big the website is
• How much it cost
• How many digital objects it contains
• How much they cost
• How many staff you employ
• How much they cost
• How many letters the project staff have after
  their names

18
Some other surrogates

• What the users say about the service
• warm-tummy stories
• What users actually do
• understanding how the site is used
• What users actually produce
• what the impact of using the site is (e.g. new
  skills, new job, coursework, their own
  contribution to the site)

19
(No Transcript)
20
Stories from the web

• Site providing a service which encourages
  development of children as fulfilled readers and
  creative writers, and develops their IT skills
• 7.8million hits over 12 months, 28 stayed longer
  than 19 minutes
• Stories Clubs in many libraries

21
Stories impacts

• Children attending the clubs have increased IT
  skills and book knowledge
• Children are using the library more
• Family members are using the library more
• skills and confidence of children and staff all
  increased

22
The warm-tummy factor

• I think your web page is wonderful! I even have
  our site on Favourites because it is one of my
  favourite sites! (Samantha, aged 10)
• This is the best web ever, Ive started to write
  stories now (Jane, aged 8)
• http//www.storiesfromtheweb.org/