Title: Using Logic Models for Project Evaluation
1Using Logic Models for Project Evaluation
Improvement
- MeHAF Integration Initiative
- Grantee Web Seminar
- March April, 2009
- Presented by Mary Ann Scheirer
- Maryann_at_ScheirerConsulting.com
2Why Are We Here?
- What do you want to learn from this web seminar?
- Remember, for 2009 grantees, your projects
Logic Model is due as part of your first 6-month
report in August, 2009. - Be sure to review these materials just before you
draft your own Logic Model!
3Definition of evaluation The use of
systematic data to aid users in their management
of a targeted set of activities
4Planning Program Evaluation
- Evaluation (collecting using data) should be a
continuing part of your project management. - When planning evaluation, one needs a clear
description of what the project is use a logic
model!
5Logic Models - Warm-Up Exercise
- Think about a recipe for a favorite dish
- How would you go about making that dish?
6What is a Logic Model?
- Picture/graphic of how a project or program works
- A systematic, visual way to show the connections
among parts of a project - Shows a theory of change connecting what
happens (activities) with outcomes
7Format for a Logic Model
Participants
Resources (Inputs)
Activities (Outputs)
Short-term Outcomes
Long-term Outcomes
Environmental Context
8Why a Logic Model?
- A program is a set of activities designed to
- achieve change, which should be based on
- IF - THEN LOGIC.
- If my organization provides X,
- Then the target audience will Y
9Why is a Logic Model Important?
10How Can You Use a Logic Model?
- For Project Planning
- To understand specify project elements
- By involving stakeholders - agreement?
- For Project Management
- Monitoring improving project implementation
- Communicating building consensus
- For Project Evaluation
- Shows assumptions in line of reasoning
- Suggests measures needed for evaluation
- For Communications
- Writing proposals for further funding ?
sustainability
11Simplified MeHAF Integration Grantee Logic Model
Participants N Providers and N patients in XYZ
health system
- Activities
- Create PC-BH team
- Train providers
- Screen/treat/ follow-up on patients
- Short-term Outcomes
- providers in integ. team
- pts recving integ. screening/
- treatment/f-up
- pts reporting integ./pt.-ctrd. care
- Long-term Outcomes
- Integrated care delivered consistently
- X of pts. w/ chronic diseases improve
- Care is more pt.- centered
- Resources
- MeHAF funding
- Evidence
- Staff experience
- Environmental Context
- Fragmented service delivery ?Licensing,
regulation, reimbursement - ?Workforce Shortages ? Translation of Research
to Practice
12Parts of a Logic Model - Resources
- Examples
- Staff Volunteers
- Time
- Money
- Space in agency
- Evidence Base
- Partners
- Materials Technology
- Resources are whatever the agency needs to
deliver the program - Usually are nouns
- Also called inputs
13Parts of a Logic Model - Activities
- Examples
- Train providers
- Recruit clients - outreach
- Deliver MH/SA/PC treatment
- Develop products resources
- Create collaboration with partners
- Implement new clinical protocols
- Adopt shared clinical records
- A summary of the actions or steps an organization
takes to deliver a program. - what the projects does with its resources.
- use verbs like provide
- counseling or serve
- meals.
- ... show what you do.
14Parts of Logic Model Short-Term
Outcomes
- Examples
- Providers implement integration components
- of new clients reached
- ? of Clients screened
- Clients engaged in treatment
- Clients receive Follow-up
- Partners implement agreed-on components
- Change in agency agendas
- Outcomes How the target audience changes or
benefits after participating in programs
activities. - Short term Immediate actions desired among
participants - Measures are usually expressed quantitatively
- Shows what they do
15LM Short Term Outcomes Examples
- Better
- 80 of PC sites improve scores on SSA tool
- ? of SA patients treated in primary care
- At least 150 patients assessed/year, using
evidence-based tools - 90 of providers practice integrated model
consistently
- Weak
- Increased capacity of primary care to provide
addiction treatment - Patient assessment tools using SDOH evidence
designed and utilized - Increase in knowledge, attitudes and beliefs
around integrated care for practice personnel,
mental health providers and third-party payers.
16Parts of Logic Model Long-Term
Outcomes
- Examples
- Integration components used consistently by all
providers - Increased client access to care
- clients with continued behavior change
- Higher retention rate in SA treatment
- New policies enacted
- Partners sustain integrated services
- Reflect your long-term goals for the project.
- What will they do then?
- Ideally, measured quantitatively, behaviorally.
- May be derived from evidence base.
- Include LT outcomes for agency, e.g.,
sustainability
17LM Long Term Outcomes Examples
- Weak
- Improved health status
- Evaluate sustainability of project financially
set up data base for estimating revenue capacity - Integrated quality health care for clients,
leading to longer lives - Add to the existing evidence base
- Better
- X of patients with chronic diseases improve on
relevant heath status measures - Revenue sources secured for sustaining
integrated services - High quality integrated care for clients
continued after project (as per SSA)
18Check the logic and assumptions underlying your
project!
19 Feedback Loops
- Information about the project is sent back as an
input to the system - Can lead to project improvements
- A key use for process evaluation, from your
short-term outcome measures
20 Logic Model - MeHAF
Integration Initiative
Clients/Participants Health providers,
funders and consumers in Maine
- Short-Term Outcomes
- For MeHAF
- High quality pool of applicants developed
- Models/culture of pt.-centeredness and
integration created - Trends, issues opportunities...leveraged
- Systems change ....
- MeHAF viewed as partner
- For Grantees
- Increased .... level of pt.-centeredness
integration - Changed delivery systems
- Improved pt. outcomes
- Priority pop.s served
- Pts/families become advisors/advocates
- Data systems support integration
21 Conclusions
- The process of developing a Logic Model may
require time multiple stakeholders. - Prior literature and stakeholders should be
consulted. - Logic Models are powerful, living tools for
planning, management, evaluation, and
communication, at any stage of project
implementation!
22Logic Model Integrated Primary Care Sacopee
Valley Health Center Clients/Participants
Patients of and community members referred to
SVHC for services.
Environment Fragmented delivery system, Rural
area, Poverty, Workforce Shortages, Cost of Gas,
Reimbursement issues
23Logic Model - Integrated Behavioral Health for
Farmworkers (Plan phase-year 1) Clients/Participan
ts Maine Migrant Health Program (MMHP), and
Migrant and Seasonal Farmworkers in Maine (MSFWs)
24Your Turn!
- Questions?
- Work as a project team to develop your draft LM.
- Logic Models will be due with interim reports, in
August - Contact me, if questions arise.
25Overview of Evaluation Plans for MeHAF
Integration Initiative
- Multiple methods used
- Combined with grantee level evaluation
- Focused on data collection and feedback over
time, across grantees - Includes assessing both provider system-level
change and care received by clients - Includes both process and outcome evaluation
linked over time
26Major Components of MeHAF Evaluation Plan
- Grantee Level Data Sources
- Do clinical services grantees make progress in
implementing the components of integrated,
patient-centered care? - Measured by the Site Self Assessment tool
- How many what types of clients receive
integrated care, across all projects? - Data from Client Data Elements form
- Do clients receiving integrated care show
improvement in outcomes measured by each site? - Data submitted on Client Data Elements form
27Major Components of MeHAF Evaluation Plan
(Continued)
- Grantee Level Data Sources
- How are grantees implementing integrated care?
- Qualitative data from site visits, annual
reports conducted by external evaluator - JSI - Are patients/families and providers engaged in,
satisfied with their experience with integrated
care? - Method(s) to be developed (TBD)
- Can the costs of integrated care at grantee sites
be documented? - Method(s) to be developed (TBD)
28Major Components of MeHAF Evaluation Plan
(Continued)
- From State-Level Data Sources
- (Methods to be developed)
- To what extent has integrated care spread among
Maines population as a whole? - What are the total costs of providing integrated
care cost changes over time? - Do providers and/or consumers better understand
and utilize integrated care? - Does the initiative prompt changes in
reimbursement, regulatory and/or licensing
policies?
29Site Self Assessment Tool (SSA)
- Many of you have completed the tool
- Any comments after using the tool?
- Note that SSA can also serve as a measuring tool
for site level progress in implementing
integrated care - it should be included in your
Logic Models
30Client Data Elements Form (CDE)
- Developed with input from Round 1 grantees
- 4 sections
- I - Numbers of clients II Demographics of
clients - III - Follow-Up IV Client
outcome measures - Discuss additional comments or concerns
- Client data to be reported in CDE should be
included as outcomes in your grantee-level logic
models - An evaluation data management contractor will be
available, with TA if needed