Testing the Impact Assessment Framework in Lisbon Municipal Libraries Network

1
Testing the Impact Assessment Framework in
Lisbon Municipal Libraries Network
Libraries for Lifelong Learning

• Leonor Gaspar Pinto
• Library Management Division
• Lisbon Municipality
• PORTUGAL

2
The agenda

• Lisbon Municipal Libraries Network a brief
  introduction
• Testing the Impact Assessment Framework
• Test 1 Basic ICT Learning Skills
• Test 2 - Story Telling
• Lessons learnt

3
LISBON Municipal Libraries Network
The Network

• LISBON, the capital city of Portugal

• 17 libraries 1 central library, 10 branch
  libraries, 3 special libraries, 2 mobile
  libraries and 1 public garden service point
• First municipal library 1887

S. Lázaro Library
4
LISBON Municipal Libraries Network

• To respond and anticipate users information and
  knowledge needs
• Incentive them to use the information recourses
  in a Lifelong Learning perspective
• Promote reading and information literacy as basic
  competencies for an active citizenship
• Give them the necessary support in leaser
  activities.

L M D Goals
5
Testing the Impact Assessment Framework
Selection of 2 Different Activities

• Basic ICT Learning Sessions (ULISSES Programme)
• Adults and elders

• Give me a Book I will tell you a story
• Reading promotion initiative story telling
• Children, Primary School classes

6
Testing the Impact Assessment Framework

• 5 Learning Areas

Attitudes Values
Knowledge Understanding
Skills
Activity, behaviour progression
Enjoyment, Inspiration Creativity

• 5 Steps 1) Developing your Story 2)
  Selecting SMART Indicators
  3) Deciding on the right Data Collection Methods
  4) Analyse
  Results 5) Present and Use Findings

7
Testing the Impact Assessment Framework
Basic ICT Learning Sessions

• ULISSES Programme

• Aims
• To reinforce the role of Municipal Libraries as
  places for Lifelong Learning, as well as
  privileged access points to information.
• To help users developing the competencies they
  need to use information in an effective and
  critical manner.
• To promote users information literacy
  competences, improving their ability to access,
  use and evaluate information in every format.
• To contribute to the development of computer
  basic skills among library users and citizens in
  general.

Named after Ulysses, Homers hero Lisbon
Olissipo, the city of Ulisses
8
Testing the Impact Assessment Framework
ULISSES Programme Outputs - 2008

• 524 users/learners
• 103 learning sessions
• 226 leaning hours
• 27 elements of Library Staff directly involved
• 52 computers used

9

Testing the Impact Assessment Framework
Basic ICT Learning Sessions

• Step 1) Developing your Story

• Retrospective testing
• ULISSES Programme available performance data
• Time scope February-December 2008
• 5 Libraries
• At least 10 Sessions
• At Least 50 Participants
• Expected Outcomes
• Basic ICT competences development
• Comprehension of sessions content
• Users/learners satisfaction
• Change in users/learners attitude and behaviour
  towards ICT

10
Testing the Impact Assessment Framework

• Step 2) Selecting SMART Indicators
• Step 3) Deciding on the right Data Collection
  Methods

Knowledge and Understanding
Indicator
Collection Method
Outcome
Nº of participants that point a level 2, in a
scale from 1 (min) to 3 (max.), in the skills
framework of each learning Module
At least half of the participants will understand
the sessions contents
Skills Evaluation Form
11
Testing the Impact Assessment Framework
Basic ICT Learning Sessions

• Step 3) Deciding on the right Data Collection
  Methods

• Interview Guide
• Control Sheets
• Enrolment Forms
• Learning Resources Evaluation Form
• Skills Evaluation Form

12
Testing the Impact Assessment Framework
Basic ICT Learning Sessions

• Step 4) Analyse Results

Per Generic Learning Outcome
13
Testing the Impact Assessment Framework
Basic ICT Learning Sessions
Per Generic Learning Outcome
14
Testing the Impact Assessment Framework
Basic ICT Learning Sessions
Per Generic Learning Outcome
15
Testing the Impact Assessment Framework

• Step 5) Present and Use Findings

Testimonials
It is very nice to be able to send and receive
E-mails or Chat with my grandchildren living
abroad.
16
Testing the Impact Assessment Framework
Story Telling

• Reading Promotion Programme

• Aims
• To consolidate and expand the social and
  educational role Lisbon Municipal Libraries of
  Lisbon play on the development of reading habits
  and information literacy skills.
• To strengthen the external image of Lisbon
  Municipal Libraries as information and knowledge
  hubs and reading promoters.

• Action Lines
• I Promotion of Reading and Library Collections
• II Promotion of Information Literacy
• III Promotion of Citizenship

17

Testing the Impact Assessment Framework
Story Telling

• Step 1) Developing your Story

• Give me a book... I will tell you a story a
  reading promotion activity targeted at children
  (primary school classes)
• Sample
• 3 municipal libraries, 50 participants (47
  children, 3 teachers/facilitators), 3 sessions
• Expected outcomes
• The activity promoted reading skills and habits
  of children
• Children understood the story and the characters
  roles
• The activity stimulated children's fondness of
  reading
• Children found the activity fun

18
Testing the Impact Assessment Framework
Story Telling

• Step 2) Selecting SMART Indicators
• Step 3) Deciding on the right Data Collection
  Methods

Skills
Ex.
Indicator
Collection Method
Outcome
Nº of teachers/ facilitators that point a level
2, in a scale from 0 (min) to 4 (max.) in the
item Contribution to reading skills improvement
At least ¾ of the teachers/facilitators think the
activity helped to improve children reading skills
Teacher/ facilitator Evaluation Form
19
Testing the Impact Assessment Framework
Story Telling

• Step 3) Deciding on the right Data Collection
  Methods

• Adapting the collection methods to the target
  audience
• Redefining the story
• Groups of children participating in summer-camps
• Different age groups
• Accompanied by summer-camp facilitators

20
Testing the Impact Assessment Framework
Story Telling

• Step 4) Analyse Results

Per Generic Learning Outcome
21
Testing the Impact Assessment Framework
Story Telling
Per Generic Learning Outcome
22
Testing the Impact Assessment Framework
Story Telling
Per Generic Learning Outcome
Enjoyment, inspiration, creativity

• 61,3 of the children considered the activity was
  very fun (a rating of 3, in a 1 to 3 scale)
• All adult considered the activity useful to
  improve children's fondness of reading (a rating
  of 3, in a 0 to 4 scale)

Activity, behaviour progression

• 47,7 of the children were willing to read more
  stories after the activity (a rating of 3, in a 1
  to 3 scale)

23
Lessons learnt
Selection of 2 Different Activities

• Basic ICT Learning Sessions

• Give me a Book I will tell you a story

• It was easier to use the IAF in Test 1 than in
  Test 2
• The more structured an activity is in terms of
  its planning and evaluation methods and routines,
  the easier the use of IAF.
• The less pedagogical structured the activities
  are, the bigger the effort needed to (re)think
  and frame them into IAF
• All learning initiatives and activities in
  libraries should be goal-oriented.

24
The ENTITLE Impact Assessment Framework TEST
TEAM
We welcome any comments Learning Promotion
Service spa.dgb_at_cm-lisboa.pt