The why in evaluation: learning as part of the development and change process

1
The why in evaluation learning as part of the
development and change process

• Dr Tina Cook
• Northumbria University
• January 2008 UKES Seminar

2
Two whys

• Why do the evaluation?
• Why are we being asked to do this evaluation
  why are we doing it?
• The purpose of asking why as part of the
  evaluation process
• as opposed to what are we doing and how is it
  being done

3
What is evaluation?

• Historically evaluation has been seen as a
  quantitative procedure consisting of the task
  of generating unbiased, precise estimates of the
  causal consequences of programmes (T.D. Cook
  1997)

4

• The process of determining whether an item or
  activity meets specified criteria.sparc.airtime.c
  o.uk/users/wysywig/gloss.htm
• an assessment of the project's success in meeting
  its intended outcomes usually includes a
  formative evaluation and summative evaluation
  (see formative and summative evaluation).www.brow
  ard.k12.fl.us/grants/html/resources/definitions.ht
  ml

5

• Judging the value of material based on personal
  values/opinions, resulting in an end product,
  with a given purpose, without real right or wrong
  answers. appraises compares contrasts
  concludes criticizes critiques decides
  defends interprets judges justifies reframes
  supports.faculty.washington.edu/krumme/guides/blo
  om.html

6
Summary of approaches for conducting evaluations
7
Chelimsky and Shadish (1997)

• Evaluation for development
• Evaluation for knowledge building
• Evaluation for accountability

8
Early Excellence Centres

• Help establish and maintain communication amongst
  project participants support enabling
  conversations
• Help clarify indicators and criteria for success
• Assist reflective stance
• Provide evidence and raise issues for discussion
  - suggest foci for shared deliberations
• Assist in seeing possibilities for change of
  direction in the project

9
Categories of indicators

• Enabling what was set up so far
• Process how things were being done what was
  being tried to translate principles of project
  into action
• Outcome performance achievements of the project

10
Best Value

• Experience of evaluating the governments Best
  Value programme has led to questions as to
  whether it is possible for evaluators to
  reconcile the multiple political, managerial and
  societal objectives associated with such projects
• (Martin and Sanderson 1999)

11

• It was clear to us, and this was reinforced when
  discussing these issues with partner evaluators
  across the country, that the relationship between
  policy, policy makers, the national and local
  evaluations was under-rehearsed
• Questions about why we were doing the
  evaluation had been particularly avoided.
• Biott and Cook (2000) and Owen, Cook and Jones
  (2005)

12
Outcome

• Disastrous!
• Local evaluation stalled
• National Evaluation not completed
• Findings from all evaluations (11 starter, 25
  second wave, 100 final) not embedded into
  practice at either local or national level.

13
Why within the evaluation

• Education Action Zones Evaluating Early Years
  Practice
• Used same principles for evaluation as with EEC

14
Action Research Approach

• Joint reflection about the relationship in
  particular circumstances between process and
  products (Elliott, 1991)
• A continuous cycle in which you select an area of
  your work to investigate, collect information
  about, reflect upon, then perhaps develop new
  ideas about ways of working and act upon these.
  (Goldsmiths, 1995)

15
Aim of the meetings

• To develop
• The capacity for discrimination and judgement in
  particular complex, human situations and to
  unify inquiry, the improvement of performance
  the development of persons in their professional
  role. (Elliott, 1991)

16
Participatory Action Research

• Evaluator facilitated meetings were devised to
• Support participants in investigating their own
  practice (used research methods) and to encourage
  in-depth discussion about issues raised with all
  the other participants carrying out
  investigations into their own projects.
• Provide an arena for data collection for the
  overall evaluation, making an account of, and
  accounting for, practice and the development of
  that practice.

17
Main topics of discussion

• What is good about our work and the way we
  practice it what is difficult?
• How do we know what is worth developing and where
  the barriers are?
• Are there institutional/professional/personal
  barriers to development and change?
• What is the difference between evidence and
  knowing gleaned through experience?
• How does evidence, professional knowledge and
  general knowing fit together?
• Can merely asking questions of ourselves and
  others lead to change?

18
The 3P approach to questioning Project One

• Philosophies
• Inclusion
• What does inclusion mean?
• What is the difference between inclusion and
  integration, exclusion and segregation?
• Principles
• No one will be excluded on the basis of perceived
  abilities
• What does exclusion mean?
• Practice
• Children with special education needs had a
  special box of equipment, carefully chosen, and
  kept separately from the other equipment
• Why?
• What might inclusion look like?

19
The 3P approach to questioning Project Two

• Philosophies
• Parents habits effect their childrens behaviours
• Principles
• More parents in libraries will ultimately lead to
  more children
• Libraries should be welcoming place for parents
• Practice
• Parents were not coming to certain libraries
  despite the same programme being put in place
• Why?

20
Outcomes

• In the first project
• who they consulted and collaborated with changed
  their criteria for success changed who they
  asked if it was successful changed practice
  changed
• In the second project
• Who they who they consulted and collaborated with
  changed their criteria for success changed who
  they asked if it was successful changed practice
  changed
• Early Years Library service changed its way of
  developing services

21
Evaluation

• Asking why had enabled people to articulate
  their understandings and beliefs
• What is articulated is known
• Began to take a critical stance on taken for
  granted knowing
• Had been supported in delving below comfort
  structures
• Developed their ability to see what was really
  there rather than what they hoped was there
• Found their own way of moving forward described
  indicators
• Had learned and developed for themselves
• Had implications for future practice beyond
  evaluation framework.

22
Project One

• when we first met I was saying oh I know we
  shouldn't really have these in the boxI know its
  not OK but it will probably have to say because I
  dont know any other way and I havent any money
  for change and this is all we can afford to buy
  but now, 6 months down the line, yes we can,
  because we have listened to how others think and
  their ideas have become just as important and let
  us see new ways of doing it .

23
Project Two

• How silly had I been. I was only asking the ones
  who came what they liked, and obviously they
  liked it or they would not have come there. So
  when I asked those that dont, I found out that
  some parents, even young parents, have ideas
  about libraries that are so out of date - and
  that so are my ideas of what makes people feel
  comfortable but I had to go through a lot of
  discomfort myself to get to that!! rueful
  laughter on tape

24
Early Years Library Loan Service

• Little did we know that this evaluation would
  have such a positive effect within our own
  service . It has had an impact on our Early Years
  Strategy and has informed our choice of stock for
  the new Early Years Library Loan Service.

25
Bridge between knowing and change

• Facilitated enquiry into own practice and own
  means of knowing.
• Letting go of previous knowns
• Real learning is not easy messy
• Real learning brings about change and development
• Evaluation reported on real meaning and
  worthwhileness in context and identified key
  issues for engagement and development