Measuring Outcomes Welcome to Kindergarten program Rotman Nexus Symposium on Measuring the Impact of

1
Measuring OutcomesWelcome to Kindergarten
program Rotman Nexus Symposium on
Measuring the Impact of Doing GoodJanuary 21,
2009Veronica Lacey and Helle TosineThe
Learning Partnership
2

• The Learning Partnership (TLP), a national
  not-for-profit organization, brings together
  business, education, government, policy makers
  and the community to strengthen public education.
  
• Our four strategic objectives are to engage
  Canadians in discussion on important educational
  issues affect public policy develop world class
  educational leaders and deliver programs, in
  partnership, which enrich public education.
• Welcome to Kindergarten program , started in
  2004, has the goal of preparing children for
  school, from immigrant, hi-risk families, from
  Aboriginal, and northern communities, and from
  French boards/districts right across Canada.

3

• Welcome to Kindergarten program objectives
• Close achievement gap between advantaged and
  disadvantage children
• Address key areas of early childhood development
• Engage and build trusting relationships between
  schools, community partners, families
• Build cross-sectoral partnerships with
  government, businesses and organizations that
  serve children and families
• Welcome to Kindergarten program 2008/09
• Across Canada, the program is delivered in 2000
  schools, 85 school boards/districts affecting
  over 75,000 families
• Materials are available in gt 20 languages
  interpreters are available to overcome language
  barriers.
• Over 550 Community Partners across Canada,
  including, e.g. public libraries, public health,
  early years and parenting centres, social
  services, YMCA, childcare agencies, universities,
  local businesses, Native Friendship Centres,
  Canadian Settlement Workers in the Schools,
  service clubs, government ministries, early
  intervention programs, Best Start Networks

4

• How is Welcome to Kindergarten program
  delivered?
• Recruitment and Engagement of Community Partners
• Professional Development of Partners/Educators
• Parent/Caregiver and Child Orientation Workshops
  (spring) introducing early learning resources
  with WTK bag
• Follow up workshops for Parent/Caregivers (fall)
• Evaluation

5

• Welcome to Kindergarten program evaluation /
  measurement
• Started in 2004 with the first pilot in the GTA)
  in partnership with OISE/UofT
• As the program evolved and expanded, subsequent
  evaluations testing specific program areas
  included Nipissing University -- North Bay
  Project (2006/07) in partnership with Ontario
  Trillium Foundation Project
• As the program expanded across Canada, evaluation
  was conducted in British Columbia in partnership
  with UBC (2007/08)
• WTK Evaluation uses a mixed method approach,
  deliberately combining different tools and
  techniques, e.g.
• Qualitative tools, e.g. parent teacher focus
  groups, surveys, questionnaires
• Quantitative tool Early Development Index (EDI)
  a nationally accepted and validated tools for
  evaluating childrens school readiness

6
Workshop discussion.

• In most cases, the difficulties of causality,
  aggregation, and long time horizons mean that the
  total social benefit of programs may not be
  directly measurable.or measurement may be cost
  prohibitive
• Attribution is a key challenge that can be
  anticipated in completing program evaluations in
  an educational setting.
• One of WTK key goals isto build cross-sectoral
  partnerships with government, businesses and
  organizations that serve children and families
• As the WTK program expands, how can the
  attribution issue be addressed? How can WTK be
  sure that it is the WTK program that helps meet
  the goal of building effective cross-sectoral
  partnerships with government, businesses and
  organizations.

7
(No Transcript)
8
The Learning Partnerships Outcome Measurement
Framework
9

• In developing an outcome measurement framework,
  some of the challenges we found include, e.g
• Obtaining longitudinal change data, and adequate
  participation in control groups is difficult
• Competing demands in schools for principals and
  teachers may make that part of the evaluation too
  costly (takes too much effort for little result)
  program evaluation is not their first priority
• Often it is too costly to obtain data to prove a
  hypothesis some concepts have been tested over
  time and can be accepted as true
• Educating children is a complex topic need to
  very clear and focused on TLP specific role and
  objectives, matching evaluation to those focused
  objectives.

10

• Some of the best practices for program evaluation
  for educational programs included, e.g
• Be very clear on who needs to know the answer and
  what do they need to know
• Keep it simple use as few parameters as possible
  to answer the question What do we have to
  prove?
• Not always feasible to evaluate all programs to
  the same level of detail necessary look to
  overall impacts on a few key areas
• Although it is always preferable to obtain
  quantitative evidence, not always able to obtain
  this data find acceptable indicators that can be
  used as proxy for the outcomes, e.g. attitudinal
  shift can demonstrate capacity building with
  data obtained through a survey
• Although proof is preferable, evidence is
  acceptable.indirect indicators can.demonstrate
  progress even though they dont provide absolute
  proof1

1Indicators of Effectiveness the Centre for
Effective Philanthropy
11

• Our outcome measurement framework is a critical
  component of all our programs and projects and is
  used
• to evaluate program effectiveness, strengths,
  weaknesses, towards continuous improvement and
  informed business decision making
• The outcome measures can be used as part of the
  Return on Investment (ROI) report on achievement
  of our vision and as the basis of communication
  of key achievements to our partners, sponsors,
  funders and supporters

12

• Principles
• Useful
• the evaluation will serve the information needs
  of intended users.
• Feasible
• the evaluation will be realistic, diplomatic,
  and cost effective.
• Appropriate
• the evaluation will be conducted legally,
  ethically, and with due regard for the welfare of
  those involved in the evaluation, as well as
  those affected by its results
• Accurate
• the evaluation will reveal and convey
  technically adequate information about the
  features that determine worth or merit of the
  program being evaluated.

13
Established ethical conduct guidelines
incorporated into TLP outcome measurement
framework.

• Tri-Council Guidelines
• Developed by
• Canadian Institutes of Health Research
• Natural Sciences and Engineering Council of
  Canada
• Social Sciences and Humanities Research Council
  of Canada
• Goals for use
• To establish high standards for research and
  program evaluations within TLP
• To offer a consistent protocol for reviewing
  internal and external research.

14
An example.how the Outcome Measurement Framework
is used for WTK in meeting the goal of closing
the achievement gap
Inputs
Outputs
Outcomes
Short Term
Participation
Activities
Long Term
What TLP does invest towards achieving the
goals? Staff Time Money Materials Equipment
After Years 1 - 5 what are indicators of success
against the broader societal goal? - 10 of WTK
children have low readiness scores compared to
24 of non-WTK children
After 5-10 years what are indicators of success
against the broader societal goal? - Research
underway

Who does TLP want to target through this
program? -children, parents, teachers, community
partners, government
What does TLP do to achieve the
goals? -Recruit, engage Community
Partners -provide professional development of
Partners /Educators -provide parent/care-giver,
child orientation workshops - Follow up
What are TLP Program or Initiative Goals (limit
to lt6)? Close achievement gap address key areas
of early childhood development, engage partners
What is the Social Benefit Created or Societal
Goal? Success for Student close the
achievement gap
EVALUATION Focus Collect Data Analyse
and Interpret - Report
15
Quantitative WTK has the advantage of a national
evaluation tool EDI, which allows the program
to assess against whether the program impacted
school readiness using 5 key domains of child
development
16

• Qualitative Testimonials from Sponsors and
  Participants
• The Welcome to Kindergarten bags are not only a
  very appealing gift for each child to take home,
  they serve as a valuable tool for interactive
  learning. Using the WTK materials and
  information, parent and child can engage together
  in shared activity. This program is extremely
  important for school readiness. The Hon.
  Margaret McCain
• Thank you, just a big thank you. It just puts
  tears in my eyes when I think of what one mother
  said, This is the first book that my child has
  ever owned that wasnt previously used, and was
  his alone. Teacher, Toronto
• This is a terrific program. Its in keeping with
  all the research that talks about the importance
  of the early years and if we can give these kids
  a head start, I think it is time and money
  well-invested.
• Principal, Calgary
• Kids arent as self-conscious going into class.
  There is a link between the bag and interest in
  school. They know more and have more of a focus.
  
• Parent, New Brunswick

17

• Key learnings
• Multiple measurement techniques may be needed
• Pilot projects allow experimentation
• Replicable programs are ones where there is
  evidence supporting the hypothesis
• Key Success Factors..
• Willing site for piloting
• Willingness to participate in evaluation and
  measurement
• An effective program
• Thank you, to Rotman and Nexus for the
  opportunity!