Effective Planning and Evaluation of Active Community Programs

1
How to Plan and Evaluate a Great PA Program
Tammy Vehige, M.Ed, CHES Sarah Levin Martin, PhD.
2
Today's Presentation

• What is a Great Program?
• Community Development
• Planning Evaluation

3
What is a Great PA Program?

• One with community input
• Comprehensive/multi-component
• One that people will do

4
(No Transcript)
5
Health Promotion for the Individual
Traditional Approach
Community Barriers
6
Health Promotion and Community Design
Community Design Approach
Traditional
Community Barriers
7
(No Transcript)
8
Influences on Behavior
9
Community Development

• Community level work

10
Phase 1 The Community
11
Determine Community Capacity

• What do you want to accomplish?
• Who do you need as partners?

12
Determine Community Capacity
Talk with People Review Documents
and Existing Data Observe the Community
13
Determine Community Capacity

• Are the community leaders supportive?
• Is the community interested in the areas in which
  your agency can provide technical assistance?

14
Engage Stakeholders

• Those who have decision-making power
• Intended target population
• Those who partner with you, the implementer
• The implementer

15
Tips

• Collaborative Leader
• Inspires commitment and action
• Leads as a peer problem solver
• Builds broad-based
• involvement
• Sustains hope and
• participation

16
Assess the Community

• Areas to explore
• History of the community
• Previous community collaboration
• Community response to
• Economic conditions
• Major landmarks
• Physical activity opportunities
• Culture(s) of the community
• Influential people
• Norms and values of community members

17
Focus Priorities

• May need to gather additional information on
  identified community issues
• Gathering community specific information that is
  relevant will enable the development and
  implementation of programs that are of interest
  to the community
• Choose a solution(s) that will bring about the
  most desired results and open doors for bigger
  and better things
• The selected solution(s) should be obtainable and
  should have the resources to support it

18
Establish Workgroups

• Establish one or more workgroup for each priority
  area
• Example Community Coalition Workgroups
• Diabetes
• Domestic Violence
• Air quality
• Taffic/Safety

19
Create a Community Strategic Plan and Logic Model
Diabetes
Miracle Occurs
Domestic Violence
Healthier Community
Air Quality
Traffic/Safety
20
Planning and Evaluation

• Workgroup level work

21
Phase 2 The Workgroup
Engage Stakeholders
Assess Community Needs and Resources
Choose an Intervention
Describe Plan Your Intervention Evaluation
Implement Your Intervention Evaluation
Disseminate Results and Ensure Use
22
Engage Stakeholders
23
Assess the Community and Resources Related to PA
FORMATIVE Assessment designed to enhance your
understanding of the population
Needs Assessment What are the barriers?
Asset Mapping What are the strengths?
Find out key informant interviews, focus
groups, observations, audits, surveys, existing
local documents
24
Choose an Intervention
Guide to Community Prevention
Services RE-AIM Framework Social-Ecoglogical
Model
25

• Independent Non-federal Task Force
• A Process
• Systematic reviews of the literature
• Assessments of study quality
• Summary of outcomes
• Community Guide gold standard for reviews of
  population based interventions
• A Product
• Evidence-based recommendations
• Book
• Website (www.thecommunityguide.org)

26
Physical Activity

• Informational
• Behavioral and Social
• Environmental and Policy

27
RE-AIM Framework

• Reach
• Efficacy/Effectiveness
• Adoption
• Implementation
• Gasgow et al, AJPH 1999

28
Social Ecological Model
29
Describe and Plan Program
Influential Factors surrounding environmental
factors such as politics, SES, other on-going
programs, social norms, staff turnover etc. tat
can affect the programs success (positively or
negatively)
30
Plan Evaluation

• For Each Question, consider
• Indicators
• Data Sources - People, community, secondary
  sources
• Data Collection Method - Interviews, focus
  groups, observations, surveys, document review,
  measurements
• Who will Collect the Data? - Trained staff,
  graduate students, consultants

31
Plan Evaluation
32
Evaluation Grid (Example)
33
Implement and Evaluate Intervention

• Fidelity of Implementation
• Process evaluation during the implementation
• Tweaking

34
Collect Analyze Evidence
Quantity Quality Triangulation
35
Process Evaluation Measures
36
Outcome Measures
37
Analyze Data

• Enter data, clean data
• Do quality control
• Tabulate the data
• Stratify the data as appropriate
• Make comparisons
• Present data

38
Interpret Results

• Compare your multiple indicators
• Compare your results with objectives
• Are they what you expected?
• If not, what are some possible explanations
• If so, are there alternative explanations
• Compare your results compare with other similar
  programs
• What are the limitations of your evaluation

39
Justify Conclusions

• There are several things to consider as you make
  judgment about the programs success based on
  your findings
• Compare with objectives
• Compare with logic model
• Prioritize evaluation questions
• Re-engage stakeholders
• Build consensus before moving on to the next
  step!

40
Share Lessons Learned

• In order to ensure use make action oriented
  recommendations
• Expand the program
• Redesign the program
• Abandon the program

41
Recommendations

• Consider your stakeholders values
• Share draft recommendations with stakeholders
  first and solicit feedback
• Consider your original purpose and uses
• Target your recommendations for each audience

42
Follow-Up

• Ensuring use does not stop at making
  recommendations!
• Active follow-up is required
• Remind stakeholders and others of the intended
  uses of the evaluation results,
• Prevent lessons learned from being lost or
  ignored when broader decisions are made,
• Prevent misuse of results by ensuring evidence is
  applied the questions that were asked and results
  are not taken out of context.

43
Proposed Model for Developing a GREAT PA Program
THE COMMUNITY
THE WORKGROUPS