An Evaluation Tool for Development Education Consultation Day

1
An Evaluation Tool for Development
EducationConsultation Day 

• September 25th 2009, University of Limerick

2
Why Participate?

• Irish Aid funded project to develop DE/ESD
  Evaluation Tool for possible use as a standard
  tool to compare programmes across Ireland
• Help to design it to the specific needs of the
  Irish DE/ESD Sectors
• This tool will increase transparency,
  communication, and possible comparisons
• Allow for much-needed benchmarks and baselines to
  be established

3
Objectives of the day

• become familiar with general tool parameters and
  strengths and limitations of quantitative tools
• Identify and prioritise outcomes of successful
  DE/ESD programmes
• Identify optimal operational factors of a tool,
  e.g. optimal time to complete the tool
• Consider piloting this tool within your
  organisation

4
Project Background

• No one standard assessment to measure impact of
  work carried out in formal or non-formal DE/ESD
  sectors.
• RCE-Irelands research project aims to develop a
  quantitative evaluation tool for DE/ESD
  programmes
• Focus on values and attitudes, and critical
  thinking - two key skills identified in DE/ESD
  literature for action-orientation, (IRISH AID,
  1999, Tormey, 2003, Tilbury and Wortman, 2004)
• Acknowledgement of this tool as a significant
  undertaking preliminary stages. By February
  2010 the first pilot will be completed. Report
  back to Irish Aid for possible further funding
  and development of this invaluable tool

5
Project Timeframe

• Sept. - ESD Evaluation Consultation Day
• Oct. - Finalisation of tool format
• Nov. - Evaluation Tool Pilot
• Jan. - Data analysis
• Feb. - Finalise results and tool

6
Factors influencing decision-making on
development and sustainability Issues

• Situational factors (e.g if supports and
  incentives exist to make sustainable choices like
  recycle banks)
• knowledge and ability factors (e.g. knowing about
  energy saving and how to do it
• intrapersonal factors (e.g. attitudes, values,
  acceptance of responsibility)
• (Hines, Hungerford and Tomera,1986)
• The intrapersonal factor MUST be dealt with for
  LASTING behavioural change

7
What to test for?
Knowledge
Skills
Attitudes
8
Introduction to Critical Thinking

• Critical thinking is reasonable, reflective
  thinking that is focused on deciding what to
  believe or do.
• (Ennis, 1993)
• Some critical thinking skills are applicable
  across contexts and others are context-specific

• DE/ESD is cross-curricular
• Some critical thinking skills are more relevant
  to DE/ESD than others, e.g.
• being open-minded
• being able to judge the credibility of sources
• ask clarifying questions and draw conclusions
• identify conclusions, reasons, and assumptions
• (Ennis, 1993)
• Critical thinking tests general-content based,
  subject specific, multiple choice tests exist
  (Illinois Critical Thinking Project), essay-style
  tests (simulated situations) (The Ennis-Weir
  Critical Thinking Essay Test)
• Prioritising key critical thinking skills for
  DE/ESD - break-out session II

9
Evaluation Tool Design Basis

• July 2008 - April 2009 RCE-Ireland Project
  Situated Learning and DE/ESD Activism -
  questionnaire tool developed based on 3 variables
  identified by Hines, Homerford and Tomera (1986),
  (situational, knowledge ability, intrapersonal)
• Defining Issues Test (Bebeau and Thoma, 2003)
• moral reasoning psychometric test
• Story-format or scenarios
• participants rank statements in order of
  importance, in relation to decision they made in
  each scenario
• based on the work of Lawrence Kohlberg.
  (Kohlberg, 1973). The DIT measures moral
  reasoning on a scale corresponding to these
  stages
• Adapting scenarios model for new DE/ESD
  evaluation tool to test for values and attitudes
  as well as priority critical thinking skills
• Develop DE/ESD scale for action-orientation

10
Evaluation Tool Parameters

• Quantitative
• Comparative
• less time to use and analyse
• Addresses pre-determined questions
• Statistically analytic
• Dependent on evaluator knowledge
• Operational
• Reliable and valid
• Useable data for the end user
• Implementable in a reasonable timeframe
• Web-based tool or pencil and paper
• Self-corrected Vs. externally evaluated
• Measuring increase in levels of critical thinking
  and values and attitudes (i.e., cognitive and
  affective dimensions), before and after a DE/ESD
  programme

• Acknowledging Tool Limits
• Action-orientation is measured indirectly
• Prerequisites for making long-term behavioural
  change are, having corresponding values and
  attitudes, and critical thinking skills
• Final tool used in conjunction with qualitative
  evaluation tools. Tool kit to accompany the final
  tool

11
Over to you

• Break-out Session I Appreciative Enquiry
• Outcome to identify successful factors of a
  quantitative evaluation tool
• share general experience of quantitative
  evaluation tools small groups, to categorise and
  summarise these and briefly feedback to the whole
  group
• Break-out Session II - Success Indicators of
  DE/ESD
• Outcome to identify key indicators to measure
  how successful a DE/ESD programme was
• If you had a quantitative tool, what would you
  want it to measure?
• What are your hoped-for outcomes for
  participants?

12

• Thank-you

RCE-IRELAND Nancy L. Serrano, RCE Research
Assistant, Department of Education and
Professional Studies, University of
Limerick  Phone [phone] E-mail
nancy.l.serrano_at_ul.ie