Using Logic Models for Project Evaluation

1
Using Logic Models for Project Evaluation
Improvement

• MeHAF Integration Initiative
• Grantee Web Seminar
• March April, 2009
• Presented by Mary Ann Scheirer
• Maryann_at_ScheirerConsulting.com

2
Why Are We Here?

• What do you want to learn from this web seminar?
• Remember, for 2009 grantees, your projects
  Logic Model is due as part of your first 6-month
  report in August, 2009.
• Be sure to review these materials just before you
  draft your own Logic Model!

3
Definition of evaluation The use of
systematic data to aid users in their management
of a targeted set of activities
4
Planning Program Evaluation

• Evaluation (collecting using data) should be a
  continuing part of your project management.
• When planning evaluation, one needs a clear
  description of what the project is use a logic
  model!

5
Logic Models - Warm-Up Exercise

• Think about a recipe for a favorite dish
• How would you go about making that dish?

6
What is a Logic Model?

• Picture/graphic of how a project or program works
• A systematic, visual way to show the connections
  among parts of a project
• Shows a theory of change connecting what
  happens (activities) with outcomes

7
Format for a Logic Model
Participants
Resources (Inputs)
Activities (Outputs)
Short-term Outcomes
Long-term Outcomes
Environmental Context
8
Why a Logic Model?

• A program is a set of activities designed to
• achieve change, which should be based on
• IF - THEN LOGIC.
• If my organization provides X,
• Then the target audience will Y

9
Why is a Logic Model Important?
10
How Can You Use a Logic Model?

• For Project Planning
• To understand specify project elements
• By involving stakeholders - agreement?
• For Project Management
• Monitoring improving project implementation
• Communicating building consensus
• For Project Evaluation
• Shows assumptions in line of reasoning
• Suggests measures needed for evaluation
• For Communications
• Writing proposals for further funding ?
  sustainability

11
Simplified MeHAF Integration Grantee Logic Model
Participants N Providers and N patients in XYZ
health system

• Activities
• Create PC-BH team
• Train providers
• Screen/treat/ follow-up on patients

• Short-term Outcomes
• providers in integ. team
• pts recving integ. screening/
• treatment/f-up
• pts reporting integ./pt.-ctrd. care

• Long-term Outcomes
• Integrated care delivered consistently
• X of pts. w/ chronic diseases improve
• Care is more pt.- centered

• Resources
• MeHAF funding
• Evidence
• Staff experience

• Environmental Context
• Fragmented service delivery ?Licensing,
  regulation, reimbursement
• ?Workforce Shortages ? Translation of Research
  to Practice

12
Parts of a Logic Model - Resources

• Examples
• Staff Volunteers
• Time
• Money
• Space in agency
• Evidence Base
• Partners
• Materials Technology

• Resources are whatever the agency needs to
  deliver the program
• Usually are nouns
• Also called inputs

13
Parts of a Logic Model - Activities

• Examples
• Train providers
• Recruit clients - outreach
• Deliver MH/SA/PC treatment
• Develop products resources
• Create collaboration with partners
• Implement new clinical protocols
• Adopt shared clinical records

• A summary of the actions or steps an organization
  takes to deliver a program.
• what the projects does with its resources.
• use verbs like provide
• counseling or serve
• meals.
• ... show what you do.

14
Parts of Logic Model Short-Term
Outcomes

• Examples
• Providers implement integration components
• of new clients reached
• ? of Clients screened
• Clients engaged in treatment
• Clients receive Follow-up
• Partners implement agreed-on components
• Change in agency agendas

• Outcomes How the target audience changes or
  benefits after participating in programs
  activities.
• Short term Immediate actions desired among
  participants
• Measures are usually expressed quantitatively
• Shows what they do

15
LM Short Term Outcomes Examples

• Better
• 80 of PC sites improve scores on SSA tool
• ? of SA patients treated in primary care
• At least 150 patients assessed/year, using
  evidence-based tools
• 90 of providers practice integrated model
  consistently

• Weak
• Increased capacity of primary care to provide
  addiction treatment
• Patient assessment tools using SDOH evidence
  designed and utilized
• Increase in knowledge, attitudes and beliefs
  around integrated care for practice personnel,
  mental health providers and third-party payers.

16
Parts of Logic Model Long-Term
Outcomes

• Examples
• Integration components used consistently by all
  providers
• Increased client access to care
• clients with continued behavior change
• Higher retention rate in SA treatment
• New policies enacted
• Partners sustain integrated services

• Reflect your long-term goals for the project.
• What will they do then?
• Ideally, measured quantitatively, behaviorally.
• May be derived from evidence base.
• Include LT outcomes for agency, e.g.,
  sustainability

17
LM Long Term Outcomes Examples

• Weak
• Improved health status
• Evaluate sustainability of project financially
  set up data base for estimating revenue capacity
• Integrated quality health care for clients,
  leading to longer lives
• Add to the existing evidence base

• Better
• X of patients with chronic diseases improve on
  relevant heath status measures
• Revenue sources secured for sustaining
  integrated services
• High quality integrated care for clients
  continued after project (as per SSA)

18
Check the logic and assumptions underlying your
project!
19
Feedback Loops

• Information about the project is sent back as an
  input to the system
• Can lead to project improvements
• A key use for process evaluation, from your
  short-term outcome measures

20
Logic Model - MeHAF
Integration Initiative
Clients/Participants Health providers,
funders and consumers in Maine

• Short-Term Outcomes
• For MeHAF
• High quality pool of applicants developed
• Models/culture of pt.-centeredness and
  integration created
• Trends, issues opportunities...leveraged
• Systems change ....
• MeHAF viewed as partner
• For Grantees
• Increased .... level of pt.-centeredness
  integration
• Changed delivery systems
• Improved pt. outcomes
• Priority pop.s served
• Pts/families become advisors/advocates
• Data systems support integration

21
Conclusions

• The process of developing a Logic Model may
  require time multiple stakeholders.
• Prior literature and stakeholders should be
  consulted.
• Logic Models are powerful, living tools for
  planning, management, evaluation, and
  communication, at any stage of project
  implementation!

22
Logic Model Integrated Primary Care Sacopee
Valley Health Center Clients/Participants
Patients of and community members referred to
SVHC for services.
Environment Fragmented delivery system, Rural
area, Poverty, Workforce Shortages, Cost of Gas,
Reimbursement issues
23
Logic Model - Integrated Behavioral Health for
Farmworkers (Plan phase-year 1) Clients/Participan
ts Maine Migrant Health Program (MMHP), and
Migrant and Seasonal Farmworkers in Maine (MSFWs)
24
Your Turn!

• Questions?
• Work as a project team to develop your draft LM.
• Logic Models will be due with interim reports, in
  August
• Contact me, if questions arise.

25
Overview of Evaluation Plans for MeHAF
Integration Initiative

• Multiple methods used
• Combined with grantee level evaluation
• Focused on data collection and feedback over
  time, across grantees
• Includes assessing both provider system-level
  change and care received by clients
• Includes both process and outcome evaluation
  linked over time

26
Major Components of MeHAF Evaluation Plan

• Grantee Level Data Sources
• Do clinical services grantees make progress in
  implementing the components of integrated,
  patient-centered care?
• Measured by the Site Self Assessment tool
• How many what types of clients receive
  integrated care, across all projects?
• Data from Client Data Elements form
• Do clients receiving integrated care show
  improvement in outcomes measured by each site?
• Data submitted on Client Data Elements form

27
Major Components of MeHAF Evaluation Plan
(Continued)

• Grantee Level Data Sources
• How are grantees implementing integrated care?
• Qualitative data from site visits, annual
  reports conducted by external evaluator - JSI
• Are patients/families and providers engaged in,
  satisfied with their experience with integrated
  care?
• Method(s) to be developed (TBD)
• Can the costs of integrated care at grantee sites
  be documented?
• Method(s) to be developed (TBD)

28
Major Components of MeHAF Evaluation Plan
(Continued)

• From State-Level Data Sources
• (Methods to be developed)
• To what extent has integrated care spread among
  Maines population as a whole?
• What are the total costs of providing integrated
  care cost changes over time?
• Do providers and/or consumers better understand
  and utilize integrated care?
• Does the initiative prompt changes in
  reimbursement, regulatory and/or licensing
  policies?

29
Site Self Assessment Tool (SSA)

• Many of you have completed the tool
• Any comments after using the tool?
• Note that SSA can also serve as a measuring tool
  for site level progress in implementing
  integrated care - it should be included in your
  Logic Models

30
Client Data Elements Form (CDE)

• Developed with input from Round 1 grantees
• 4 sections
• I - Numbers of clients II Demographics of
  clients
• III - Follow-Up IV Client
  outcome measures
• Discuss additional comments or concerns
• Client data to be reported in CDE should be
  included as outcomes in your grantee-level logic
  models
• An evaluation data management contractor will be
  available, with TA if needed