Part I: Evaluation Strategies for Colorado Tobacco Programs

1
Part I Evaluation Strategies for Colorado
Tobacco Programs

• Arnold Levinson, Mike Wacker
• December 7, 2007

2
Session overview

• Evaluation framework overview
• R.E.I.
• Logic models
• Exercise writing a logic model
• S.M.A.R.T. objectives
• Exercise writing S.M.A.R.T. objectives

3
Who we are

• Amendment 35 Program Evaluation Group
• Tobacco Program Evaluation Group
• Tobacco Attitudes Behaviors Survey (TABS)

4
Our approach R.E.I.

• Reach
• Effectiveness
• Implementation

5
Our approach R.E.I.

• Reach
• The absolute number, proportion, and
  representativeness of individuals who participate
  in a given program.

6
Our approach R.E.I.

• Effectiveness
• The impact of an intervention on important
  outcomes.

7
Our approach R.E.I.

• Implementation
• At the setting level fidelity to the various
  elements of a protocol, including consistency of
  delivery as intended and the time and cost of the
  intervention.
• At the individual level clients use of the
  intervention strategies.

8
Why evaluate?

• The Five Ps
• Progress learn how to enhance strategies
• Promote develop success stories to tell the
  public, policymakers, peers
• Plan guide decisions to continue, increase,
  decrease, end specific strategies
• Pretest prepare a new or unproven strategy
• Prove establish how well something works

9

• Evaluate only if you know
• how youll use the results.

10
Working with logic models
11
--Lewis Carroll, Alice in Wonderland

• One day Alice came to a fork in the road
• and saw a Cheshire cat in a tree.
• Which road do I take? she asked.
• Where do you want to go? was his response.
• I don't know, Alice answered.
• Then, said the cat, it doesn't matter."

12
What Is a Logic Model?

• A visual representation of the relationship
  between
• What you have (needs, plans, strategies,
  resources)
• What you do with it (Implementation)
• What happens as a result (Reach, Effectiveness)

13
Purposes of a Logic Model

• shows how your program works
• enables evaluation
• facilitates consensus around what success means

14
R.E.I. logic model (generic to tobacco control)
Inputs
Reach
Effectiveness
Implementation
What we have needs plans strategies
resources
What we do convene, mobilize,
facilitate advocate serve educate
Who we affect populations intermediaries
-- providers -- policymakers
Short term awareness knowledge
attitudes skills motivations intentions
Long term reduced disease and death
Medium term behaviors policies services
systems norms
15
Reading a logic model

• From left to right, columns are in if ? then
  sequence
• If we undertake these activities, then we can
  achieve these short-term outcomes
• If we have these resources, then we can
  undertake these activities
• If we achieve these short-term outcomes, then we
  can achieve these long-term outcomes
• and so on.

16
R.E.I. logic model example Goal Reduce
smoking-related illness
Inputs
Reach
Effectiveness
Implementation
What we have telephone-based smoking cessation
counseling registry of patients who
participate in counseling clinic training
modules for screen and refer program
What we do train primary care practices to
screen patients train primary care clinicians
to refer patients who smoke to counseling refer
patients to counseling reports to practices
from registry
Who we affect primary clinicians and staff
adult primary care patients in Fumador and Tar
Counties
Short term clinical awareness of brief
screening methods knowledge of referral
resource patients referred to smoking
cessation understand pts who use counseling
service
Long term reduced disease and death
Medium term screening and referrals become
routine in primary care increase screen and
refer rates patients reduce or quite smoking
17
The meaning of success

• Most evaluations will not prove program services
  caused intermediate and long-term outcomes.
• A logic model shows how your strategies and
  activities logically lead to the intermediate and
  long-term outcomes.
• Choose proven (or promising) intervention
  strategies, then plan to evaluate implementation
  and reach.

18
Successful evaluation

• requires
• program theory
• literature
• best practices
• program design and implementation
• evaluation design and measurement

19
Summing up logic models

• demonstrate evaluability
• link activities to results, prevent mismatches
• integrate planning, implementation, evaluation,
  reporting
• create understanding and agreement
• promote learning

20

• Writing Measurable Objectives

21

• Goals vs. Objectives

22

• Goals ultimate purposes of the program
• Building a world where young people reject
  tobacco and anyone can quit
• American Legacy Foundation
• Reducing death and disease from smoking.

23

• Objectives Specific and measurable steps or
  outcomes that lead to the goal
• Objectives are used to
• help set program priorities
• monitor progress toward the goal
• set targets for accountability
• provide framework for program evaluation

24
Two Kinds of Objectives

• Outcome
• Implementation (process)
• Both kinds involve observation and measurement

25
Outcome Objectives

• specifically describe what will happen as a
  result of activities
• By June 30, 2008, retail tobacco sales to minors
  in Mytown will be reduced from 35 of attempts to
  5 of attempts.
• By December 31, 2008, 10 of primary care
  providers in Mytown will newly adopt and be
  practicing 2As/R with all their patients.

26
Implementation objectives

• Describe what you intend to do to achieve the
  desired outcomes
• By December 31, 2008, contract with the Mytown
  police department to conduct monthly compliance
  checks at all Mytown retail tobacco outlets.
• By July 31, 2008, establish practice-based
  systems that ensure all providers use 2As/R in
  thirty primary care offices.

27
Objectives-Evaluation Link

• Consider how youll measure your objectives when
  you are developing them.
• Measurement of objectives forms the basis for
  your evaluation plan.
• Evaluation is a key to sustainability.
• Good evaluation starts with well-written
  objectives.

28
Writing S.M.A.R.T. Objectives

• A well-written and clearly defined objective is a
  S.M.A.R.T. objective
• Specific
• Measurable
• Achievable
• Relevant
• Time-bound
• Both implementation and outcome objectives should
  be S.M.A.R.T.!

29
S.M.A.R.T. Specific

• Specific State specifically what you want to
  achieve.
• Describe the desired action, behavior or
  achievement using clearly defined and
  quantitative terms (number, percentage, rate,
  frequency)
• If change is desired, include both the current
  level and desired level
• Identify the specific target audience(s)

30
Getting Specific

• Example implementation objective
• Non-specific
• Enhanced tobacco services will be provided in
  Fumador County by June 30, 2008.
• Better
• Childcare workers in Fumador County will receive
  One Step training by June 30, 2008.
• Specific
• 75 childcare workers in Fumador County will
  receive One Step training by June 30, 2008.

31
S.M.A.R.T. Measurable

• Measurable Be able to determine how much of the
  action, behavior or achievement has been
  accomplished
• Identify a system or method to track and record
  the action, behavior or achievement
• Use an existing system or create one
• Needs to be reliable

32
Is it measurable, will it be measured?

• Necessary measurement not done
• Students in the local Boys and Girls Club will
  show an 50 increase in their knowledge of the
  risks of tobacco use by June 30, 2008.
  (Measurement cited evaluation at the end of the
  program)
• Assessable
• Students in the local Boys and Girls Club will
  show an 50 increase in their knowledge of the
  risks of tobacco use by June 30, 2008.
  (Measurement cited TTI Youth pre-post survey)

33
S.M.A.R.T. Achievable (Attainable)

• Achievable Can you actually accomplish it?
• Have a reasonable belief that you can achieve the
  objective.
• Consider resources and potential obstacles.
• Decide which to choose, the sure thing or the
  stretch

34
Is it Achievable?

• Unrealistic
• Smoking rates among Latinos in Humidor County
  will be reduced to the statewide average using
  the Los Padres tobacco cessation program by June
  30, 2008.
• Possibly attainable
• 250 Latinos enrolled in the Los Padres tobacco
  cessation program will quit smoking by June 30,
  2008.

35
S.M.A.R.T. Relevant

• Relevant Achievement of the objective should be
  meaningful.
• Important to your organization
• Directly impact the program goal and/or long-term
  outcomes

36
Is it Relevant?

• Off Topic
• 25 Altria High School students will stop smoking
  as a result of Liggett Groups adoption of a
  tobacco-free workforce policy by June 30, 2008.
• Applicable
• 25 Altria High School students will stop smoking
  as a result of participation in their local N-O-T
  program by June 30, 2008.

37
S.M.A.R.T. Time-bound

• Time-bound the objective has a deadline for
  achievement
• Clearly state the target date
• Often easiest to read if you begin the objective
  with the target date

38
A tool to help write SMART objectives
39
What we covered today

• Reach, Implementation, Effectiveness (REI)
• Concentrate on Reach, Implementation
• Purposes of evaluation (5 Ps)
• Evaluate only if you know how youll use results
• Logic models
• SMART objectives

40

• Break

41
Part II Writing a STEPP Grant Proposal

• Lori Zigich, MPH
• Public Health Division, CDPHE
• December 7, 2007

42
STEPP Grants Program

• To advance collaborative, innovative,
  culturally-appropriate and evidence-based
  solutions to reduce tobacco use among Coloradoans
  of all ages and populations, particularly those
  most at risk for tobacco use and tobacco-related
  disease.

43

• Preliminary Steps

44
Step 1

• Read the 2008 Request for Application (RFA)
  documents in its entirety.

45

• RFA Contents
• Guidelines
• Part I Applicant Information Form
• Part II Needs Statement and Project Narrative
• Part III Work Plan and Evaluation Plan
• Part IV Financial Assessment Questionnaire,
  Budget and Budget Narrative

46
Step 2

• Is the organization eligible to apply for a STEPP
  grant?

47
Step 3

• Is there a need(s)? If so, what is it, and how
  important is it?

48
Step 4

• Is the STEPP Grants Program a good match to
  address the need(s)?
• Note five different RFAs, each with their own
  funding requirements, priorities and suggested
  strategies.
• Competitive Request for Applications
• Non-competitive continuation for applications
• Local Health Agency Program
• Tobacco Disparities Grants
• TGYS Tobacco Initiative (TTI)

49
Step 5

• Need to also consider
• Funding needed
• Existing resources
• Funding exclusions
• Program time-frame

50
Step 6

• Compare the need(s) youve identified to the
  STEPP Grants Program funding priorities,
  suggested strategies, and exclusions as outlined
  in the Guidelines Section of the 2008 RFA.

51
Step 7

• Begin to consider infrastructure and other
  resources needed to meet the goals and objectives
  of the proposed project.

52

• Additional resources needed to meet goals and
  objectives of the proposed project may be found
  within or outside of your organization. Begin to
  contact potential partners.
• In-kind
• Collaborative Partners

53

• Questions

54
Step 8

• Writing the Application

55

• Scoring of applications will vary by by
  application type.
• Be familiar with the criteria and make sure all
  points are addressed.

56

• Adhere to Formatting Guidelines
• Include a cover page for each section
• Spell out acronyms the first time they are used.
• Define abbreviations of technical terminology
• Write in 12 point font.

57

• Double-spaced
• Typed on 8 1/2 x 11 paper
• One inch margins (top, bottom, left and right)
• One sided pages
• Include page numbers on lower right hand corner
  of page

58
RFA Part I

• Contents
• Agency and funding information
• Target populations(s) and county information
• Certification of Non-Acceptance of Tobacco
  Industry Funds and Resources
• Project Abstract
• Project Narrative

59
RFA Part I

• Several components in this section
• Read carefully, and respond to every question
  completely
• This sections helps STEPP program staff
  understand who you are, your experience with
  tobacco programs, and who you propose to serve

60
RFA Part I - Narrative

• Describe how your proposal satisfies an unmet
  need.
• Use of health and other data
• Special considerations (e.g., culturally
  competent interventions)
• Testimonials
• Unexpected opportunities

61
RFA Part I - Narrative

• Describe how your organization is qualified to
  address an unmet need.
• Experience in tobacco interventions and with the
  target population and community
• Staffing plan
• Experience with grants management
• Community support for the project
• Partnerships new and existing
• Major strengths and significant accomplishments

62

• Questions

63
RFA Part II

• Contents
• Work plan instructions and sample
• Evaluation Strategy Guidance
• Letter(s) of collaboration

64
RFA Part II

• Work Plan contains the scope of work to be
  performed as obligated under a contract agreement.

65
RFA Part II

• Work Plan
• Read carefully, and respond to every question
  completely
• Tie back with Dr. Levinsons presentation from
  earlier this afternoon (example)
• This section explains your goals and objectives,
  and the WHO is going to do WHAT and to WHAT
  EXTENT, and how they activities meet the needs
  you have identified

66

• Work Plan Sample

67
Instructions Respond to each section completely.
Refer to additional instructions throughout the
Work Plan sample.

• Grantee Name Agency X
• Grant Period Note A separate work plan must be
  submitted for each grant period for which funding
  is requested.
• Year One July 1, 2008-June 30, 2009
• Year Two July 1, 2009-June 30, 2010
• Year Three July 1, 2010-June 30, 2011
• Contact Name Jane Doe Project Name Agency X
  Tobacco Control Program
• Contact email address jane.doe_at_agencyx.org
• Copy and paste as many tables as needed to
  address each strategy for which funding is being
  requested.

68
(No Transcript)
69
(No Transcript)
70
(No Transcript)
71
RFA Part II

• Evaluation Strategy
• Evaluation is the process of collecting data to
  determine the reach, implementation and/or
  effectiveness of a program
• The State Board of Health Rules Pertaining to the
  Tobacco, Education, Prevention and Cessation
  Grant Program requires A description of all
  program evaluation activities. as part of grant
  applicant contents

72
RFA Part II

• Letters of Collaboration
• Roles and responsibilities that contribute to
  meeting project activities and goals
• Resources to be provided by partner organization

73

• Questions

74
RFA Part III

• Contents
• Applicant Financial Assessment Questionnaire
• Guidelines for the Budget and Budget Narrative
• Professional Development Calendar and Requirements

75
RFA Part III

• Guidelines for the Budget and Budget Narrative
• Sample budget
• Complete detailed Budget Form and Summary Sheet
• Sample budget narrative
• Complete budget narrative

76

• Final Steps
• Final review
• Application checklist
• Number of applications to submit
• Mail applications to CDPHE or
• Deliver applications in person by 200 pm on
  January 23, 2008 to the CDPHE

77

• Technical Assistance
• Technical Assistance Calls December-January
• Questions and Answers posted on the STEPP website
  beginning December-January
• Additional Resources outlined in the RFA
  Guidelines section

78

• Questions

79

• Thank you
• for your
• participation today.