Title: Understanding the Government-Wide Shift in Accountability: Using Performance Management to Plan for Outcomes
1Understanding the Government-Wide Shift in
Accountability Using Performance Management to
Plan for Outcomes Demonstrate Results
- Technical Assistance Guide
- Prepared for Web Cast on AT Act Outcomes
- Wednesday, October 22, 2003
- by Margaret L. Campbell, Ph.D.
- National Institute on Disability and
Rehabilitation Research - U.S. Department of Education
2Purpose of Technical Assistance Guide
- To provide NIDRR staff, grantees, applicants and
reviewers with a comprehensive resource for
understanding the government-wide shift in
accountability and how the concepts and tools of
performance management can be used to - Plan for outputs outcomes
- Improve programs
- Monitor progress, and
- Document results and communicate successes
3Part 1
- The What and the Why of the Accountability Shift
4What is the Government-Wide Shift in
Accountability?
- Shift from primary focus on
- Dollars (how much is spent on what)
- Activities processes (what you are doing
to/with whom and how well you are doing it) - Productivity (how much are you doing)
- To an expanded focus on
- Performance -- (are you meeting stated
objectives and standards of quality), and - Results (what you are achieving and whos
benefiting?)
5Shift in Accountability is Not Either- Or
Situation
- Rather amounts to an added dimension of
accountability for results. - Still must meet accepted standards of practice
for - Continuous quality improvement
- Program development and delivery of services, and
- Consumer involvement
- Systems change advocacy
6What Does Accountability for Results Mean?
- In general, accountability for results refers
to accomplishments that are under the direct or
indirect influence of project resources and
activities occur within the boundaries of a
funding/budget cycle. - Within the NIDRR context, this translates into
accountable for three types of results - Outputs planned and produced to date
- Short-Term Outcomes anticipated and actual, and
in some cases - Intermediate Outcomes anticipated or actual
7Origins of the Accountability Shift in
Government GPRA
- In 1993, Congress passed the Government Results
and Reporting Act, which established uniform
requirements for strategic planning, annual
planning and reporting, and agency-grantee
partnerships. - Key Provision of GPRA Provides a new definition
of program evaluation an assessment, through
objective measurement and statistical analysis,
of the manner and extent to which federal
programs achieve intended objectives.
8Recent Policy Initiatives Reinforcingthe
Accountability Shift
- The Presidents Management Agenda (PMA) FY 2002,
with its emphasis on - Performance management measurement
- Objective criteria to assess program results
(i.e., the PART) - Performance-based budgeting
- The Dept. of Educations New Strategic Goals,
2002-2007, with an emphasis on - Transforming Education into an Evidenced-based
Field.
9Part 2
- The Language and Logic of
- Outcomes Planning and Managing for Results
- (Key Concepts and Definitions)
10What is Meant by Managing for Results?
- Managing for results requires developing a new
mindset based on outcomes-oriented goals and
performance measures, as opposed to
activity-oriented goals and objectives . - The primary focus is on planning for what outputs
and outcomes will be achieved not on what
activities will be conducted. - For most of us, this involves re-expressing
existing program goals and objectives into the
language of planned outputs and anticipated
outcomes.
11Concepts and Tools You Need to Succeed
- Performance management
- Performance measurement
- Performance goals measures
- The Performance Spectrum
- Logic modeling i.e., reverse mapping
- Components of Performance measurement
- Target systems and/or populations
12What is Performance Management vs. Performance
Measurement?
- Performance management is an approach to
planning and evaluation that utilizes the
concepts and tools of performance measurement and
logic modeling to identify performance goals and
assess progress towards goals. - Performance measurement involves the ongoing
monitoring and reporting of program
accomplishments, particularly progress towards
pre-established goals (see GPRA requirements).
13What are Performance Goals?
- According to OMB, performance goals are the
desired levels of performance expressed as
measurable objectives against which actual
achievement can be compared. - To be complete, performance goals should
incorporate performance measures and performance
targets into timeframes.
14Definitions of Performance Goals (Contd.)
- Performance measures are the quantitative or
qualitative metrics and indicators that are used
to demonstrate progress towards goals and
document results/ accomplishments. - Performance targets are the quantifiable or
otherwise measurable characteristics that tell
how well a program or project must accomplish a
performance measure.
15The Spectrum of Performance Outcomes-Oriented
Goals
- Although the PART emphasizes outcome goals, that
focus on changes and/or improvements in the
target system, performance goals can also be
stated in terms of planned outputs (and
sometimes quality of activities). - The key All performance goals should clearly
distinguish between components of performance and
there should be a logical connections among them
with outputs supporting outcomes (i.e.,
outcome-oriented performance goals).
16What is Logic Modeling?
- A vehicle for dialogue, planning, program
improvement and evaluation - A graphic representation or blueprint of the
key elements of a program or project, and how it
will work under certain conditions to solve
identified problems. - A helpful tool for identifying outcomes and
depicting the chain of events that link inputs
and activities to outputs and outcomes.
17What Does A Basic Logic Model Look Like? Prepared
by the NIDRR PPBE Division1
HOW
WHY
Span of Accountability for Results
Short-Term Outcomes
Program Activities
Intermediate Outcomes
Longer Term Outcomes
Outputs
Inputs Infrastructure
Customers Who Use Outputs
PROGRAM DELIVERED
RESULTS FROM PROGRAM
EXTERNAL CONDITIONS INFLUENCING SUCCESS (/-)
ANTECEDENT/ MEDIATING
MEASURES
1 Source Adapted from McLaughlin, J. A.
Jordan, G. B. (In Press/2003) Logic models a
tool for describing program theory and
performance, in Wholey, et. al. Handbook of
Practical Evaluation. Jossey-Bass. The lead
author may be contacted by email
atmacgroupx_at_aol.com NoteThe above logic model
does not include the standards by which the
quality, relevance, and productivity of
performance will be measured and evaluated across
the spectrum. For NIDRR grantees this is provided
by the centers of excellence (CoE) criteria.
Updated on June 27, 2003
18 What are the Components of Performance
Measurement?
- Inputs Infrastructure The human and financial
resources and systems needed to conduct a high
quality, outcomes-oriented program or project. - Examples agency priorities and requirements,
host institutional support and leveraging, staff
competencies and expertise, management practices
and evaluation plans, partnerships and
collaborations, and previous accomplishments.
19 What are the Components of Performance
Measurement? (Contd)
- Program Activities Are the action steps, tasks,
procedures, and services performed in conjunction
with implementing your planned program of public
awareness, technical assistance and training,
Outreach, and interagency coordination to carry
out objectives and produce results.
20 What are the Components of Performance
Measurement? (Contd)
- Outputs Are the direct results of program
activities and consist of the findings or
conclusions, products and services produced and
reported to external audiences. - Outputs typically are reported using statistics
or quantitative counts e.g., the of
individuals trained, newsletters distributed,
publications produced, participants attended. - But they also can be described qualitatively in
terms of the nature of the findings or
discoveries.
21 What are the Components of Performance
Measurement? (Contd)
- More On Outputs
- Outputs are important in their own right as
indicators of productivity they also are the
essential building blocks of outcomes. - However, not every output will have a
corresponding outcome, nor should it.
22 What are the Components of Performance
Measurement? (Contd)
- Utility Although not an official component of
the performance spectrum as defined by GPRA,
NIDRR has included utility because of its
centrality to our Mission, and because
demonstrating the perceived usefulness of
outputs to sub-groups within the broader target
systems constitutes an important bridge between
outputs and short-term outcomes.
23What are the Components of Performance
Measurement (Contd)
- Outcomes Are the anticipated or actual effects
of program activities and outputs and constitute
changes or improvements in the target systems
being served. Although the emphasis is on
planning for outcomes, outcomes may also occur as
unexpected changes. - Outcomes are the most complex component of PM and
are difficult to define outside of the context of
a specific program and target system.
24What are the Components of Performance
Measurement (Contd)
- Factors to take into account in planning for and
documenting outcomes - The basic outcomes equation is Outcomes
change/improvement in the target population or
system, which is a function of - Outputs Documented External Use
- Outcomes typically occur sequentially over time
and, as a result, may affect more than one target
system. Because of this, we refer to chain of
outcomes, starting with short-term outcomes and
progressing to intermediate and longer-term
outcomes.
25Types of Outcomes
- Short-term outcomes are expected or actual
changes or improvements in the identified target
system that are more under the direct influence
of center activities. Short-term outcomes
represent the first level of change that must
occur in order to bring about intermediate
outcomes. Typically, short-term outcomes are
focused on changes/improvements in the learning
knowledge system at the individual or
environmental level of analysis.
26Types of Outcomes
- Intermediate outcomes are the gold-standard in
terms of accountability for results. They consist
of expected or actual changes or improvements in
the action system that occur in part as a
result of the use or adoption of project outputs.
Unlike short-term outcomes that occur under the
direct influence of program activities, there may
be other causal factors contributing to the
achievement of intermediate.
27Types of Outcomes (Contd)
- Longer-term outcomes are the desired end-results
of a program, and constitute changes or
improvement in the overall condition of a
population or system. Given their scope,
longer-term outcomes usually take more than one
funding cycle to achieve, and therefore are not
required under the accountability for results
umbrella. Their primary function is to serve as
critical anchor points in the planning process.
28Examples of Types of Outcomes (Changes or
Improvements) at the Individual Level
Focus on Learning and Knowledge Focus on Action Focus on Conditions
Awareness Knowledge Attitudes Skills Opinion Motivation Function Behavior Practice and/or Clinical care Advocacy Choice or decision-making Empowerment/Self-Efficacy Physical or Mental Health Status Employability Independent Living Community Integration, Participation Economic Self Sufficiency Quality of Life
29Examples of Types of Outcomes (Changes or
Improvements) at the Environmental Level
Focus on Learning and Knowledge Focus on Action Focus on Conditions
Level or Availability of New Knowledge Models, frameworks or guidelines Methods or Tools Products and/or devices Services Technologies Use or Adoption of New Knowledge Models, frameworks or guidelines Methods or Tools Products and/or devices Services Technologies Standards Policies, laws and/or regulations Capacity of systems or society Employment Rates Rates of poverty among PWD Rates of Secondary Conditions Rates of Premature Mortality Delivery Systems Environmental Access (barrier elimination) Rates of participation
30What is a Target System?
- Formally defined, a target system refers to a
unit of analysis or bounded set of interacting
elements. - In the NIDRR context, a target system or
population refers to the level(s) of the
environment and/or to the group(s) of individuals
that program activities and outputs are intended
to affect. - Specifying the particular sub-groups or sample
populations within the broader target system that
your program intends to affect is key to
identifying anticipated outcomes.
31Examples of the Types of Target Systems Typically
Involved in NIDRR-Funded Projects
- Individual level systems persons with disability
and/or their family members, clinicians, service
providers, other researchers, policy makers, and
industry representatives. - Environmental-level systems the knowledge base
in a field, treatment/practice standards, product
development standards, availability of new
methods, tools, products, services, and
technologies, policies and/or service delivery
systems, manufacturers, and societal conditions.
32Part 3
- How to Apply Performance Management Tools to Plan
for Outcomes Manage for Results
33Objective of Outcomes Planning
- ?To identify a limited number of high-priority,
problem-focused, reasonably ambitious and
measurable outcomes-oriented performance goals
that - Reflect NIDRR priorities or statutory
requirements, and - Capture the anticipated effects and benefits of
project activities outputs on identified target
populations and/or systems.
34Two Ways to Get to OutcomesProspective vs.
Retrospective Planning
- Prospective outcomes planning is the preferred
strategy, but it requires starting with a
managing for results mindset and identifying
performance goals and measures to guide
activities and monitor progress (slides 35- 42 ). - Retrospective outcomes planning is less than
ideal, but is how most of us beginning to get our
feet wet. It involves identifying outcomes
after-the-fact and collecting data to support
them (slides .
35Prospective Outcomes Planning Step 1
Preparation
- Describe the overall problem you are trying to
solve and specify what part or parts of the
problem will be addressed by your program
activities (i.e., theory of the problem clarifies
scope of your project). - Identify the specific sub-groups or sample
populations within the broader target systems
that you anticipate will change/ improve as a
result of your program. - Explain how your program activities outputs
will change or benefit the specific target
systems or populations you have identified (i.e.,
theory of the program)
36Prospective Outcomes Planning Step 2
Constructing a Logic Model
- Translate requirements and objectives of program
into anticipated outcomes, starting with
longer-term, end-results and working backwards to
identify intermediate and short-term outcomes
(see Worksheet below) - Identify the specific target populations and
systems to be affected by each type of outcome - Plan the outputs that will be produced to bring
about anticipated changes in target systems - Design the activities that must be conducted to
produce planned outputs - Identify the resources needed to implement
activities and - Document the contextual factors that could
influence the success of your program.
37Worksheet for Constructing a Logic Model of
Performance for Your Program Prepared by the
NIDRR PPBE Division1
Important Tip When planning, begin with the end
in mind and work from right to left. When
implementing, begin with inputs and
infrastructure and move from left to right
Span of Accountability for Results
Inputs and Infrastructure
Program Activities
Short-Term Outcomes
Intermediate Outcomes
Longer-term Outcomes
Outputs
Utility
What resources and management practices are you
using, or developing, to conduct planned
activities and produce desired results consistent
with the priority and program requirements?
What evidence do you have of the potential merit
and/or usefulness of your outputs?
What changes or improvements will occur in
awareness, understanding or knowledge and for
which target populations as a result of your
program activities and outputs? And, what
measurable indicators will you use to track
progress towards achieving these short-term
outcomes/ performance goals?
What activities are you currently conducting, or
plan to conduct, across program requirements to
fulfill objectives, produce intended outputs, and
assess their relevance and utility for target
audiences? How well are you implementing these
activities and what is your progress to date,
including problems encountered and actions taken?
What products, services, and information are you
providing, or do you plan to provide to target
audiences or customers to support anticipated
outcomes? And what is your productivity to date?
What changes or improvements do you anticipate
will occur in behavior, practice and/or policy,
and for which target populations as a partial
result of the use or adoption of new knowledge
generated by your program? And, what measurable
indicators will you use to track progress towards
achieving these intermediate outcomes/
performance goals?
What are the long-term changes or improvements in
societal conditions that you hope to contribute
to overtime through your program activities,
outputs and short-term and intermediate outcomes?
Internal and External Factors That Can Influence
Success
1 Source Adapted from McLaughlin, J. A.
Jordan, G. B. (In Press/2003) Logic models a
tool for describing program theory and
performance, in Wholey, et. al. Handbook of
Practical Evaluation. Jossey-Bass. The lead
author may be contacted by email
atmacgroupx_at_aol.com NoteThe above logic model
does not include the standards by which the
quality, relevance, and productivity of
performance will be measured and evaluated across
the spectrum. For NIDRR centers and model systems
this is provided by the centers of excellence
(CoE) criteria. Revised June 27, 2003
38Prospective Outcomes Planning Step 3 Formulate
Performance Goals
- Convert outcome statements into performance goals
by - Identifying or developing the performance
measures that will be used to document progress
and demonstrate accountability for results (i.e.,
outputs, short-term and intermediate outcomes). - Establishing a baseline and identifying
performance targets that must be met to achieve
outcomes. - Establishing the timeframe within which progress
results will occur.
39Prospective Outcomes PlanningStep 4 Evaluation
of Performance
- Develop an evaluation plan that identifies the
types of evidence you need to collect to
determine how well your program is working and if
it is achieving the anticipated results. - Specify the sources and methods of data
collection you will use to measure performance
and demonstrate progress towards
outcomes-oriented goals (see Table on next slide
for examples of data sources methods).
40Examples of Sources and Methods of Data
Collection for Demonstrating Outcomes
41Prospective Outcomes PlanningStep 5 Evaluation
of Performance
- Integrate your performance management plan into
the ongoing operation of your program activities
to measure performance, collect outcomes data
and monitor progress.
42Prospective Outcomes PlanningStep 6 Analysis
Documentation
- Analyze the qualitative and quantitative
performance data to document progress towards
anticipated outcomes. - Interpret the data to determine if an outcome has
occurred or is likely to occur within the
specified timeframe. Making this determination
relies on qualitative judgments of quantitative
and/or qualitative data. - Document results and communicate accomplishments
to external audiences is an evidence-based
claims-making process, in which grantees present
evidence to support the claim that their program
activities outputs have resulted in changes/
improvements in the identified target system.
43Retrospective Outcome Identification Problems
with Approach
- Missing pre-established performance goals to
guide activities and outputs - Missing specification of target systems where
outcomes are expected to occur - Missing a logic model that clarifies the
interconnections among activities, outputs and
outcomes and between short-term and intermediate
outcomes - Missing performance measures and systematic
outcomes data to monitor progress and demonstrate
results
44Retrospective Outcome Identification Suggested
Implementation Strategies
- Form an AT Act Work Group to translate statutory
requirements into a limited number of
high-priority (i.e., 2-3) and measurable
outcomes-oriented performance goals for each core
activity public awareness, TA training,
outreach, and interagency coordination.
(suggested of outcomes is - Review and categorize accomplishments produced to
date into outputs and types of outcomes
(short-term, intermediate and long-term).
45Retrospective Outcome Identification
Implementation Strategies (Contd.)
- Link accomplishments to newly formulated
performance goals and identify the types of data
available and the data elements needed to
demonstrate progress towards outcomes and
document the contribution of AT Act project
activities and outputs. - Revise web-based annual performance reporting
form for FY 2004 to collect necessary data and
rethink how existing data can be analyzed to
provide evidence of progress or achievement of
outcomes.
46Common Weaknesses in Prospective Retrospective
Outcomes Identification
- 1. Lack of specificity in what and where
- 2. Too many outcomes and expressed too narrowly
or too broadly - 3. Outcomes expressed as speculations vs. as
expectations - 4. Confusion between levels of the performance
spectrum (i.e., activities vs. outputs and
outcomes and short-term vs. intermediate
outcomes) - 5. Missing supporting links between outputs and
short-term outcomes - 7. Missing evidence of outcomes
47Exercise Whats Missing in the Following AT Act
Project Outcomes?
- Outreach Within the last grant cycle, State X
provided technical assistance to 4, 127
individual and demonstration of loan of
equipment to 1,415 individuals. - Training State X provided 102 training sessions
attended by 3,8288 which resulted in increased
knowledge levels that have prompted systems to
address AT needs and responsibilities within
policies and procedures.
48Exercise Illustration of Reformulated Outcome
Statement
- Original Statement Systems Change Intermediate
Outcome State AT program worked with the state
election officials on the HAVA state plan which
resulted in adoption of the FEC 2002 access
standards for voting machines. - Suggested Reformulation Conduct legislative
initiatives that result in the development
(short-term outcome) and adoption by states
(intermediate outcome) of new standards for
voting machines to increase access for adults
with moderate to severe disabilities.
49Part 4
- Lessons Learned
- The Utility and Challenges of Implementing
Performance Measurement
50Benefits of Utilizing Performance Management
- Provides a common language and integrated
framework to facilitate - Strategic planning program design
- Identification of performance goals
- Program integration coordinates the
contributions of multiple projects and tasks - Objective measurement of progress towards goals
- Program monitoring improvement
- Program accountability -- documentation of
results, and - Communication of success stories
51Challenges of Applying Performance Management
Concepts Tools
- Developing a managing for results mindset.
- Incorporating requirements of performance
measurement into the traditional standards of
excellence for service delivery, consumer
involvement, and productivity. - Learning to re-express core program objectives
into the language of outcomes-oriented
performance goals (outputs and outcomes). - Deciding what evidence you need to collect and
how to measure performance and determine how your
program is working (i.e., quality and
relevance/utility) and if you are achieving
planned outputs and anticipated outcomes.
52Challenges of Applying Performance Management
(Contd)
- Integrating the tools of performance management
into the operation of your program. - Using the information gathered to improve program
management, evaluate performance, achieve
results, and provide evidence to support claims
of short-term intermediate outcomes. - Documenting results and communicating successes
using a claims-making process based on
qualitative judgments of quantitative
and/qualitative data
53Lessons Learned from Implementing Performance
Management in Business Government
- What gets measured gets done. (Drucker)
- If you dont measure results, you cant tell
success from failure (C. Mindel). - According to OMB, the number one cause of federal
agencies getting a low score on the new Program
Assessment Reporting Tool (PART) is NOT having
results-oriented performance measures to gauge
the outcomes of a program (http//www.transparentg
overnment.org/tg/analysis.htm).
54Lessons Learned from Implementing Performance
Management (Contd)
- Accountability for results involves a partnership
-- NIDRR cant satisfy GPRA and PART alone! To
succeed in the new era of performance-based
budgeting, NIDRR needs data on accomplishments
from grantees like you. - Key Assumption All grantees have
accomplishments what missing are the performance
goals and outcome data. - Key Assumption If you can demonstrate results,
you can win public support (C. Mindel).
55Thank you