Evaluating Health Promotion Efforts

1
Evaluating Health Promotion Efforts

• May 21, 2009
• Webinar
• Jennifer D. Keith, MPH
• Public Health Management Corporation (PHMC)
• www.phmc.org
• Public Health Management Corporation (PHMC) is a
  nonprofit public health institute that builds
  healthier communities through partnerships with
  government, foundations, businesses and
  community-based organizations. PHMC has served
  the Greater Philadelphia region since 1972.

2
BE A BRIDGE

• A state-wide multi-disciplinary collaborative
  working to improve the health of Pennsylvanians
  disparately affected by tobacco related chronic
  disease
• Links organizations with the resources needed to
  successfully build capacity to improve and/or
  develop population-specific tobacco-related
  programs and interventions
• www.beabridge.org

BE A BRIDGE is funded through a PA DOH grant
3
CDC Evaluation Framework
Source www.cdc.gov/mmwr/preview/mmwrhtml/rr4811a1
.htm
4
Logic Model

• Needs to convey
• Purpose of initiative
• Expected actions
• Expected results
• Expected causal relationships
• Serves as a map for all involved

• Everything I needed to know about my program, I
  learned from my logic model.

Makes program theory clear, not true.
5
Evaluation Plan

• Needs to convey
• Data to be collected
• Collection methods
• Relationship to logic model
• Base plan on program logic model
• Logic model is the foundation!
• So evaluation plan may not change, but the
  evaluation focus will shift

• Youll need to answer
• What did we do?
• How well did we do it?
• What difference did our program make?

6
Evaluation Framework

• Standards Maximize all four
• Utility What information do we need?
• Feasibility What information can we get?
• Propriety Who should be involved?
• Accuracy How can we get accurate information?

7
Utility is King/Queen!

• Who needs what information?
• How will information be used?
• Feasibility is often Utilitys 1st reality check
• Given the money/time/effort we can contribute
  what proof can I get?
• But, evaluation must be ethical reliable too!

8
Public Health Model

• Impact Efficacy Participation

High Reach
Low Cost
ideal program
Effective
9
Socio-Ecological Model
Levels of Influence
Public Policy national, state, local laws
Community relationships among orgs, norms
Organizational orgs, social institutions
Interpersonal family, friends, networks
Individual knowledge, attitudes, skills
Relationships between individuals and their
environments
10
Why evaluate health promotion efforts?

• Measure if time money are well spent
• Describe a promotions success
• Improve understanding/services
• Inform decision making
• Be accountable
• Engage stakeholders

11
Health Promotion Planning

• Target Population
• Understand
• Engage
• Message
• Focus on a goal
• Test
• Medium Placement

• Can be informed by your needs assessment, part of
  your overall health promotion evaluation

12
Health Promotion Messages Examples

• Attend ABC class
• Visit XYZ web resource
• Schedule a 123 screening
• Get healthy!
• Dont ____________

13
Health Promotion Mediums

• Face to Face
• Print (e.g., newspapers, fliers, posters)
• Billboard
• Online
• Radio
• TV (e.g., PSA, paid commercial, earned media)

14
Medium Reach Exposure

• Media buys
• Saturation
• Coverage
• Dose effect
• Scope (reach length)
• Remember Utility, Feasibility, Accuracy,
  Propriety
• BALANCE

15
Promotion Plan Key Questions

• BEFORE the Promotion Effort
• What is the goal of the message?
• Who does the message need to reach?
• How should the message be delivered?
• AFTER the Promotion Effort
• Was the message delivered to the intended
  audience?
• Did it make a difference?

16
Evaluation Stages

• Needs Assessment
• Pre-testing/Testing
• Process Evaluation
• Outcome Evaluation
• Impact Evaluation
• Dont wait until the end of your
  program/initiative to evaluate!
• Use evaluation data throughout your
  program/initiative to keep improving efforts

17
Did it make a difference?

• Need to define success first
• What are you going for?
• Perception/Attitude shift?
• Knowledge change?
• Behavior change?
• Social norms shift?
• Policy change?
• Branding?
• Media effects or improved relationships?
• Health change (long-term)?

18
You NEED Feedback

• Ensure your feedback will be
• Systematic
• Intentional
• Logical
• Regular
• Triangulated or From mixed sources
• Use more than one method
• Consider qualitative and quantitative

19
Effects Direct Indirect

• Immediate learning
• People learn directly from the effort
• Delayed learning
• Information is processed later
• Generalize learning
• Idea of or idea in the message understood
• Social diffusion
• Promotes social discussion
• Institutional diffusion
• Message reinforced by agencies

Source NCI, Theory at a Glance, 2nd Ed, 2005
20
Message Testing

• Is the message/promotion
• Understandable
• Memorable
• Relevant
• Noticeable/Attractive
• Credible
• Appropriate
• Effective
• Use focus groups, storyboards, surveys, etc

21
Evaluation Tools

• Tracking
• distribution of items, attendance
• Surveys
• paper, online, in person, phone
• Interviews
• Focus Groups
• Content Analyses
• Surveillance

22
Selecting a Method of Collection

• Issues to consider
• Population
• Questions
• Cost
• Time
• Personnel
• Valid
• Reliable
• Bias

23
Evaluation Strategy Examples

• Call to Action
• Pre/Post Measure
• Recall
• Control Group
• Pilot

24
Call to Action

• A measure of reach and motivation
• Measure response to a specific task
• Website hits
• Hotline calls
• Votes
• Signatures
• Visits

25
Pre/Post Measure

• A measure of change in awareness, attitude,
  behavior, etc
• Survey target audience before and after
  campaign/event

26
Recall

• A measure of attention, retention and
  memorability
• Aided Recall (may or may not use a visual/audio
  aid)
• Have you heard of the ABC Clinic?
• Unaided Recall (may still include some cues)
• In the last week, did you see any TV commercials
  with health messages?

27
Control Group

• A measure of effectiveness in reach, exposure,
  recruiting and message
• Run a campaign in one of two similar areas then
  measure changes in both areas

28
Pilot

• A measure of fit and effectiveness
• Choose pilot audience and message
• Gather campaign feedback throughout
• Take advantage of a pilot and gather as much
  qualitative information as possible
• Evaluate and document lessons learned before
  expanding to additional mediums or populations

29
Dont Forget to

• Communicate with stakeholders!
• Coordinate efforts
• Be sure program/service knows it is being
  promoted
• Be sure timing/audience/message makes sense to
  different stakeholders
• Facilitate two way communication
• Engage stakeholders throughout the promotion
  process
• Get feedback and buy in regarding audience,
  medium, message, etc

30
Review

• Do your homework!
• It doesnt have to be all or nothing
• Evaluate key measures
• Evaluate campaigns you hope to replicate
• Be systematic
• Document your questions and evaluation plan
• Collect feedback from multiple sources
• Pretest

31
Hlth Promotion Tools/Examples

• Health Communication
• Pink Book Making Health Communication Work
• http//www.cancer.gov/pinkbook
• Designing and Implementing an Effective Tobacco
  Counter-Marketing Campaign
• http//www.cdc.gov/tobacco/media_communications/co
  untermarketing/campaign/index.htm
• Social Marketing
• Most of Us - http//www.mostofus.org/resources.php
  
• http//www.socialmarketing.org/resources/resources
  .htm
• Nonprofit Communications
• Communications Consortium Media Center
• www.mediaevaluationproject.org/Paper5.pdf

32
General Evaluation Resources

• Centers for Disease Control and Prevention (CDC)
• http//www.cdc.gov/eval/resources.htmlogic20mode
  l
• Community Tool Box
• http//ctb.ku.edu/en/tablecontents/
• Free Management Library
• http//www.managementhelp.org/evaluatn/evaluatn.ht
  m
• Innovation Network
• http//www.innonet.org/index.php?section_id62con
  tent_id142
• University of Wisconsin-Extension
• http//www.uwex.edu/ces/lmcourse/ and
  http//www.uwex.edu/ces/pdande/
• W.K. Kellogg Foundation
• http//wkkf.org/Default.aspx?tabid90CID385Item
  ID5000077NID5010077LanguageID0

My contact info Jen Keith jkeith_at_phmc.org
215-985-2527