Using the Logic Model to Develop Evaluation Research Design and Measures

1
Using the Logic Model to Develop Evaluation
Research Design and Measures
2
Program Planning and Evaluation

• Solid program planning is vital to a well
  designed evaluation.
• Community partners (stakeholders) need to have a
  meaningful way to provide input to program
  planning.
• Extension staff must be able to communicate
  outcomes to funders and stakeholders.

3
Logic Models

• Well developed, easy to use tool for planning
  educational programs, using stake holder input.
• Provide a common language
• Simple, graphic way to communicate program theory
• Provides planners with a road map asking them
  to determine where they want to end up (goals)
  and then chart their course of action

4
Outcomes based

• Outcomes must be clearly written and the
  relationship between learning, action and
  long-term outcomes must be logical.
• Forces planners to think about and articulate
  their goals and objectives.
• Is an easy model for stakeholders to understand
  and provide feedback on.

5
Generic logic model
Strategy
Impact
C O N T E X T

A diagram of the theory of how a program is
supposed to work A graphic depiction of
relationships between activities and results
Source University of Wisconsin
Extension http//www.uwex.edu/ces/pdande/evaluatio
n/evallogicmodel.html
6
University of New Hampshire Cooperative Extension

• Began staff development efforts in 2001
• Provided in-services, mouse pads, web support and
  even one-on-one support to staff
• Began presenting the model to advisory committees
• Some resistance from staff, but many liked the
  model
• More staff are using it now to obtain grant
  funding
• Suggested model for communities proposing CYFAR
  New Communities projects

7
Benefits of logic modeling to partnerships

• Provides framework to discuss and articulate
  joint work
• Keeps end outcome upfront and center
• Helps facilitate conversation with focus on
  agreed upon goal that might not happened
  otherwise
• Utilizes local needs assessments and works with
  community partners to inform Logic Model
  development
• Separates indicators of achievement - papers
  published, and type of experiments completed
  from theory of change. Indicators of achievement
  are part of evaluation plan for the partnership.

8
Case Examples

• Seacoast Youth Leadership Program

• Hillsboro After School Program

9
Seacoast Youth Leadership Program

• This project will evaluate the efficacy of the
  youth diversion programs ability to achieve
  improvements in the following objectives
• Leadership
• Future Aspirations/Goal Setting
• Social Responsibility
• Decision Making/Social Skills
• Social Connectedness
• Media Literacy

10
SYLP Logic Model
Inputs Outputs Participants Activities Outputs Participants Activities Learning Outcomes Action Outcomes Condition Outcomes
H.S. graduation rates and living wage crime social capital among youth and adults safe, positive environments for youth
11
SYLP Logic Model, cont
Inputs Outputs Participants Activities Outputs Participants Activities Learning Outcomes Action Outcomes Condition Outcomes
Improved school attendance performance Healthy, drug free lifestyle Demonstrate care and concern for others Positive decision making H.S. graduation rates and living wage crime social capital among youth and adults safe, positive environments for youth
12
SYLP Logic Model, cont
Inputs Outputs Participants Activities Outputs Participants Activities Learning Outcomes Action Outcomes Condition Outcomes
life skill knowledge competencies technology intelligence media literacy Improved pro-social norms, citizenship commitment to peers, family community Goal setting, Can do attitude Improved school attendance performance Healthy, drug free lifestyle Demonstrate care and concern for others Positive decision making H.S. graduation rates and living wage crime social capital among youth and adults safe, positive environments for youth
13
SYLP Logic Model, cont
Inputs Outputs Participants Activities Outputs Participants Activities Learning Outcomes Action Outcomes Condition Outcomes
Research-based, best practice curriculum Life skills through experiential learning leadership and service learning Support learning for parents Mobilize community partnership life skill knowledge competencies technology intelligence media literacy Improved pro-social norms, citizenship commitment to peers, family community Goal setting, Can do attitude Improved school attendance performance Healthy, drug free lifestyle Demonstrate care and concern for others Positive decision making H.S. graduation rates and living wage crime social capital among youth and adults safe, positive environments for youth
14
SYLP Logic Model, cont
Inputs Outputs Participants Activities Outputs Participants Activities Learning Outcomes Action Outcomes Condition Outcomes
Youth/ parents High-risk, low-income families communities Lower Seacoast Youth Collaborative Research-based, best practice curriculum Life skills through experiential learning leadership and service learning Support learning for parents Mobilize community partnership life skill knowledge competencies technology intelligence media literacy Improved pro-social norms, citizenship commitment to peers, family community Goal setting, Can do attitude Improved school attendance performance Healthy, drug free lifestyle Demonstrate care and concern for others Positive decision making H.S. graduation rates and living wage crime social capital among youth and adults safe, positive environments for youth
15
SYLP Logic Model, cont
Inputs Outputs Participants Activities Outputs Participants Activities Learning Outcomes Action Outcomes Condition Outcomes
Human Resources Material Resources Community Partnership Youth/ parents High-risk, low-income families communities Lower Seacoast Youth Collaborative Research-based, best practice curriculum Life skills through experiential learning leadership and service learning Support learning for parents Mobilize community partnership life skill knowledge competencies technology intelligence media literacy Improved pro-social norms, citizenship commitment to peers, family community Goal setting, Can do attitude Improved school attendance performance Healthy, drug free lifestyle Demonstrate care and concern for others Positive decision making H.S. graduation rates and living wage crime social capital among youth and adults safe, positive environments for youth
16
Hillsboro After school Program

• This project will evaluate the efficacy of the
  after school programs ability to achieve
  improvements in the following objectives
• Academic Performance
• Future Aspirations
• Social Connectedness
• Social Responsibility
• Sense of Self
• Parenting Efficacy
• Social Skills/Decision Making
• Community Involvement with Youth/School

17
Hillsboro After School Future Aspirations
Inputs Outputs Participants Activities Outputs Participants Activities Learning Outcomes Action Outcomes Impact
Youth pursue post-secondary education and apprenticeship-type experiences to attain what they need to enter and succeed in a chosen field.
18
Future Aspirations, cont
Inputs Outputs Participants Activities Outputs Participants Activities Learning Outcomes Action Outcomes Impact
Youth articulate a positive view of their future, including planned career/work goals/life goals. Youth make connections/ network within the community. Youth pursue post-secondary education and apprenticeship-type experiences to attain what they need to enter and succeed in a chosen field.
19
Future Aspirations, cont
Inputs Outputs Participants Activities Outputs Participants Activities Learning Outcomes Action Outcomes Impact
Youth learn about various career options related to their interests, talents, skills. Youth develop a new skill and explore related careers. Youth articulate a positive view of their future, including planned career/work goals/life goals. Youth make connections/ network within the community. Youth pursue post-secondary education and apprenticeship-type experiences to attain what they need to enter and succeed in a chosen field.
20
Future Aspirations, cont
Inputs Outputs Participants Activities Outputs Participants Activities Learning Outcomes Action Outcomes Impact
7-8 week series of special interest 4-H projects using 4-H CCS, JASON, Robotics, NASA and computer literacy curriculum Field trips within community to learn about local careers and jobs. Youth learn about various career options related to their interests, talents, skills. Youth develop a new skill and explore related careers. Youth articulate a positive view of their future, including planned career/work goals/life goals. Youth make connections/ network within the community. Youth pursue post-secondary education and apprenticeship-type experiences to attain what they need to enter and succeed in a chosen field.
21
Future Aspirations, cont
Inputs Outputs Participants Activities Outputs Participants Activities Learning Outcomes Action Outcomes Impact
Youth Program Staff 4-H Volunteers Business industry members from the community region 7-8 week series of special interest 4-H projects using 4-H CCS, JASON, Robotics, NASA and computer literacy curriculum Field trips within community to learn about local careers and jobs. Youth learn about various career options related to their interests, talents, skills. Youth develop a new skill and explore related careers. Youth articulate a positive view of their future, including planned career/work goals/life goals. Youth make connections/ network within the community. Youth pursue post-secondary education and apprenticeship-type experiences to attain what they need to enter and succeed in a chosen field.
22
Future Aspirations, cont
Inputs Outputs Participants Activities Outputs Participants Activities Learning Outcomes Action Outcomes Impact
UNHCE staff volunteers Access to school library/ Computers Program space 4-H CCS, JASON, NASA computer literacy curriculum Material, supplies Snacks Transportation College student interns Youth Program Staff 4-H Volunteers Business industry members from the community region 7-8 week series of special interest 4-H projects using 4-H CCS, JASON, Robotics, NASA and computer literacy curriculum Field trips within the community to learn about local careers and jobs. Youth learn about various career options related to their interests, talents, skills. Youth develop a new skill and explore related careers. Youth articulate a positive view of their future, including planned career/work goals/life goals. Youth make connections/ network within the community. Youth pursue post-secondary education and apprenticeship-type experiences to attain what they need to enter and succeed in a chosen field.
23
Hillsboro After School Community Involvement
Inputs Outputs Participants Activities Outputs Participants Activities Learning Outcomes Action Outcomes Impact
Citizens recognize and act to ensure community wide support and involvement in school, after school and youth/family programs After school program is expanded and sustained
24
Hillsboro After School Community Involvement,
cont
Inputs Outputs Participants Activities Outputs Participants Activities Learning Outcomes Action Outcomes Impact
Youth and adults work together to set and reach goals. youth serving as equal partners on community boards committees Farmers Market participation with growers accepting Food Stamp program Citizens recognize and act to ensure community wide support and involvement in school, after school and youth/family programs After school program is expanded and sustained
25
Hillsboro After School Community Involvement,
cont
Inputs Outputs Participants Activities Outputs Participants Activities Learning Outcomes Action Outcomes Impact
knowledge of jobs, careers resources in community. Youth adults learn to work in partnership. Adults learn the importance value of youth as a resource to community. Youth learn components of an effective healthy community. Youth and adults work together to set and reach goals. youth serving as equal partners on community boards committees Farmers Market participation with growers accepting Food Stamp program Citizens recognize and act to ensure community wide support and involvement in school, after school and youth/family programs After school program is expanded and sustained
26
Hillsboro After School Community Involvement,
cont
Inputs Outputs Participants Activities Outputs Participants Activities Learning Outcomes Action Outcomes Impact
Field trips / visits with community leaders. Training in community youth development Steering committee reviews/ sets program goals and shares vision Adults youth partner on after school program Teen Center development knowledge of jobs, careers resources in community. Youth adults learn to work in partnership. Adults learn the importance value of youth as a resource to community. Youth learn components of an effective healthy community. Youth and adults work together to set and reach goals. youth serving as equal partners on community boards committees Farmers Market participation with growers accepting Food Stamp program Citizens recognize and act to ensure community wide support and involvement in school, after school and youth/family programs After school program is expanded and sustained
27
Hillsboro After School Community Involvement,
cont
Inputs Outputs Participants Activities Outputs Participants Activities Learning Outcomes Action Outcomes Impact
Community adults Youth Program Staff 4-H volunteers Social services health care workers T own government Business leaders, regional planning commission Main Street project leaders Field trips / visits with community leaders. Training in community youth development Steering committee reviews/ sets program goals and shares vision Adults youth partner on after school program Teen Center development knowledge of jobs, careers resources in community. Youth adults learn to work in partnership. Adults learn the importance value of youth as a resource to community. Youth learn components of an effective healthy community. Youth and adults work together to set and reach goals. youth serving as equal partners on community boards committees Farmers Market participation with growers accepting Food Stamp program Citizens recognize and act to ensure community wide support and involvement in school, after school and youth/family programs After school program is expanded and sustained
28
Hillsboro After School Community Involvement,
cont
Inputs Outputs Participants Activities Outputs Participants Activities Learning Outcomes Action Outcomes Impact
Program space Snacks Tech AV, computers Community development plan CYD resources Community gardens Staff from various disciplines Community adults Youth Program Staff 4-H volunteers Social services health care workers Town government Business leaders, regional planning commission Main Street project leaders Field trips / visits with community leaders Training in community youth development Steering committee reviews/ sets program goals and shares vision Adults youth partner on after school program Teen Center development knowledge of jobs, careers resources in community Youth adults learn to work in partnership Adults learn the importance value of youth as a resource to community Youth learn components of an effective healthy community Youth and adults work together to set and reach goals youth serving as equal partners on community boards committees Farmers Market participation with growers accepting Food Stamp program Citizens recognize and act to ensure community wide support and involvement in school, after school and youth/family programs After school program is expanded and sustained
29
Seacoast Youth Leadership Program

• This project will evaluate the efficacy of the
  youth diversion programs ability to achieve
  improvements in the following objectives
• Leadership
• Future Aspirations/Goal Setting
• Social Responsibility
• Decision Making/Social Skills
• Social Connectedness
• Media Literacy

30
Hillsboro After school Program

• This project will evaluate the efficacy of the
  after school programs ability to achieve
  improvements in the following objectives
• Academic Performance
• Future Aspirations
• Social Connectedness
• Social Responsibility
• Sense of Self
• Parenting Efficacy
• Social Skills/Decision Making
• Community Involvement with Youth/School

31
Collapsing extensive outcome goals and objectives
into manageable and meaningful evaluation
research conceptual categories for design and
measurement development
32
Identifying existing measures for conceptual
categories

• LEADERSHIP
• INDEPENDENCE (WRIGHTSMAN, 1964,1974)
• MASTERY (PEARLIN, 1981)
• SELF-ESTEEM (ROSENBERG, 1965)
• FUTURE ASPIRATIONS (PEER LEADER SURVEY DYS)
• SOCIAL CONNECTEDNESS
• SOCIAL SUPPORT (VAUX)
• ATTACHMENT TO PARENTS SCHOOL
• (RESNICK, 1997 ADD HEALTH SURVEY)

33
Collaboration with Extension staff to determine
appropriate evaluation design and measurement

• EVALUATION AS A PROCESS
• COMMUNICATION IS KEY

34
Utilization of triangulation of methods to
provide depth and breadth to the evaluation
research process
35
Contact InformationScott R. Meyer, MSW, Ph.D
- [phone] scottm_at_mail.plymouth.eduStepha
nie Halter,M.A., Research Associate, (603)
536-5725 stephanie.halter_at_unh.eduRick Alleva,
Extension Educator, [phone]
rick.alleva_at_unh.eduDan Riedy, Extension
Educator, [phone] dan.reidy_at_unh.eduPenny
Turner, Extension Educator, [phone],
penny.turner_at_unh.eduLisa Townson, Extension
Specialist, [phone] lisa.townson_at_unh.edu