Logic Model Approach to Developing Performance Measures at the Program Level

1

• Logic Model Approach to Developing Performance
  Measures at the Program Level

http//www.oregon.gov/DAS/OPB [phone]
Presentation by Rita Conrad Government
Performance Summit West June 8, 2004
2
Overview

• Oregons framework for aligning public sector
  resources
• Oregons performance measure system
• State agency logic model examples

3
Oregons Framework for Aligning Public Resources
4

• The Unfocused Government
• Decentralized Approach

5

• The More Focused Government
• Internal Alignment

6

• The Well Focused Government
• Alignment with a Higher Vision

Shared Strategies
Statewide Vision
Meaningful Measures
Outcome-based evaluations
7

• Oregon Shines provides a unified vision
• for state government.

Shared Strategies
Unified Vision
Meaningful Measures
Outcome-based evaluations
8

• Strategic Planning in Oregon

9

• Oregon Shines has broad goals focusing on all of
  Oregon

• ? Economy Quality jobs for all Oregonians
• Community Safe, caring and engaged communities
• ? Environment Healthy, sustainable surroundings

10
Its underlying strategy is called the Circle of
Prosperity.




11

• Oregon Benchmarks are key to Oregon's performance
  measurement system.

Shared Strategies
Unified Vision
Meaningful Measures
Outcome-based evaluations
12

• Oregon Benchmarks are key to knowing how were
  doing.

Results Not Effort
13
Results Not Effort

• What Benchmarks
• Don't Measure
• School Expenditures
• Prison Beds
• Environmental Enforcement

• What Benchmarks
• Do Measure
• Literacy
• Crime Rates
• Air and Water Quality

14
Oregon has ninety benchmarks in three broad
categories.

• Economy
• Education
• Civic Engagement
• Social Support
• Public Safety
• Community Development
• Environment

Economy
Community
Surroundings
15
Oregon State Government and Benchmarks

• Oregon Benchmarks
• measure how well Oregon is doing.
• Performance measures
• gauge how well state government is doing.

16

• No organization can claim total control over
  high-level outcomes.

17
The underlying premise of all performance
measures systems is...

• we understand the cause and effect
  relationships that exist between the different
  types of measures used.
• When attempting to affect outcomes, this premise
  is usually untrue. Often we do not even know the
  strength of the correlations that exist between
  inputs, outputs and outcomes.

18
Evidence-based practices help solve that problem.

• Because evidence-based practices are proven to
  work in certain situations, we can lean on this
  knowledge when developing performance measures.

19
Logic Models Connect the Dots

• a framework for planning, managing, measuring and
  evaluating government programs.
  (www.performanceweb.org)
• the basic framework of program evaluation,
  linking day-to-day activities to results
  (www.gse.harvard.edu/hfrp/pubs/onlinepubs/rrb/lear
  ning.html)
• something that communicates the logic behind a
  program, its rationale. (www.insites.org/documents
  /logmod.htm )

20
Oregons Performance Measure System
21
Setting the Stage - The Basic Planning Model
Analysis
Goals
Measures
Strategies
22
The process is based on identifying a tiered set
of measures of success.
Goal Reduce recidivism.
So That
So That
So That
23
Oregons Performance Measure Criteria-based
Process

• Criteria-based review shared with agencies, which
  can opt to comply
• Review and measures travel with agencys budget
  through Ways and Means
• Ways and Means approves the measures and/or
  requests changes

24
Criterion 1. Standard language

• OUTCOME Result
• High-level (societal) Recidivism (OBM 65)
• Intermediate of treated youth with reduced
  risk factors.
• OUTPUT Product or service (widget)
• of youth completing treatment.
• EFFICIENCY Input per output/outcome
• Average number of staff hours per completed
  training.
• Governmental Accounting Standards Board (GASB)

25
Practical translation

• High-Level Outcomes Is the world you are
  attempting to change changing?

• Intermediate OutcomesAre strategies having the
  desired result?

• OutputsIs the work getting done?

26
Criterion 2. Gauge progress towards goals
high-level outcomes
of paroled offenders convicted of a new felony
within three years
Goal Reduce recidivism. (not a measure)
So That
High-Level Outcome (Benchmark 64)
of offenders showing a measurable improvement
in behavior and/or skill level
So That
of offenders engaged in work, training,
education and/or treatment
Intermediate Outcome
So That
of intake assessments completed
Intermediate Outcome
Output
27
Criterion 3 Few key measures - agencies decide
how high up to go.
Goal Prevent Juvenile Crime
So That
of served youth with mitigated risk
factorsINCREASES.
So That
of high risk youth completing program INCREASES.
Intermediate Outcome
So That
of grantees trained INCREASES .
Intermediate Outcome
Output
28
Criterion 4. Performance measures must have
targets.
2003 Actual 20

• TARGET Desired level at a given point in time
• Ambitious but realistic
• Set targets using
• trend data
• comparisons
• expert opinion

2007 Target 15
Example hours of travel delay per capita per
year
29
Criterion 5. Performance measure data must be
accurate and reliable.

• Trustworthy data is essential.Example
  verifiable records trump estimates
• Per measure - at least one data point, preferably
  several
• Data should match the measure.

30
Logic Model ExamplesOregon State Agencies
31
Oregons Logic Modelfor Performance Measures
32
Department of Human Services Mental Health and
Addiction Services
of Oregonians whose self-perceived health
status is very good or excellent. Source Oregon
Department of Human Services
33
Department of Human Services Self-Sufficiency
of covered Oregon workers with earnings of
150 or more of poverty for a family of four.
Source Oregon Employment Dept
34
Department of Human Services Child Welfare
of children, per 1,000 persons under 18, who
are a. abused or neglected, b. at substantial
risk of. Source Oregon Department of Human
Services
35
Department of Human Services Seniors and People
with Disabilities
Outputs receiving services a. in their own
homes b. In community-based care
3B
High-level outcome. Agencies can use societal
measures as alternatives or in addition to Oregon
Benchmarks. of Oregonians with a lasting
disability who could live on their own with
adequate support. Source Dept of Human Services
36
Department of Transportation Division of Motor
Vehicles
Intermediate Outcome of DMV customer service
targets met
ODOTs Mission
3C
2
Impact
3B
GoalProvide excellent customer services
Outputs Average field office wait drive test
waits title registration turnaround time
1
3
KPM of DMV customer service targets met
3A
Strategy Monitor reallocate resources daily to
meet demand.
Provide a safe, efficient transportation system
that supports economic opportunity and livable
communities for Oregon.
37
Health Licensing Office
of agency partners HLO has established a MOA.
of stakeholders who actively participate in
program discussions of HLO programs
recognizing reciprocity as qualification
HLOs Mission
2
3C
Impact
Goal Regulate in a manner that supports a
positive business climate.
Outputs of eligible applicants who are
licensed the same day as application/ testing is
completed of agency partnerships
1
3B
3
KPM 2. of stakeholders who actively
participate in program discussions
Strategies media education educational
materials, trainings and consultations
electronic business transactions reciprocal
agreements with other states
3A
create uniform public protection practices for
health professions and occupations pertaining
toathletic trainers, midwives, etc
38
www.oregon.gov/DAS/OPB

• Oregon Shines II
• Performance Measures for 90 State Agencies
• 10 Ideas for Improving Oregon State Government
• Oregon Business Plan How is Oregon Doing?
• Performance Measure Guidelines for State Agencies
• 2003 Benchmark Performance Report
• 2004 County Benchmark Data Book
• Achieving Better Health Outcomes The Oregon
  Benchmark Experience