Understanding the Government-Wide Shift in Accountability: Using Performance Management to Plan for Outcomes - PowerPoint PPT Presentation

1 / 55
About This Presentation
Title:

Understanding the Government-Wide Shift in Accountability: Using Performance Management to Plan for Outcomes

Description:

Title: PowerPoint Presentation Author: William.Schutz Last modified by: Deb Wiley Created Date: 4/23/2003 4:25:19 PM Document presentation format – PowerPoint PPT presentation

Number of Views:307
Avg rating:3.0/5.0
Slides: 56
Provided by: Willi549
Category:

less

Transcript and Presenter's Notes

Title: Understanding the Government-Wide Shift in Accountability: Using Performance Management to Plan for Outcomes


1
Understanding the Government-Wide Shift in
Accountability Using Performance Management to
Plan for Outcomes Demonstrate Results
  • Technical Assistance Guide
  • Prepared for Web Cast on AT Act Outcomes
  • Wednesday, October 22, 2003
  • by Margaret L. Campbell, Ph.D.
  • National Institute on Disability and
    Rehabilitation Research
  • U.S. Department of Education

2
Purpose of Technical Assistance Guide
  • To provide NIDRR staff, grantees, applicants and
    reviewers with a comprehensive resource for
    understanding the government-wide shift in
    accountability and how the concepts and tools of
    performance management can be used to
  • Plan for outputs outcomes
  • Improve programs
  • Monitor progress, and
  • Document results and communicate successes

3
Part 1
  • The What and the Why of the Accountability Shift

4
What is the Government-Wide Shift in
Accountability?
  • Shift from primary focus on
  • Dollars (how much is spent on what)
  • Activities processes (what you are doing
    to/with whom and how well you are doing it)
  • Productivity (how much are you doing)
  • To an expanded focus on
  • Performance -- (are you meeting stated
    objectives and standards of quality), and
  • Results (what you are achieving and whos
    benefiting?)

5
Shift in Accountability is Not Either- Or
Situation
  • Rather amounts to an added dimension of
    accountability for results.
  • Still must meet accepted standards of practice
    for
  • Continuous quality improvement
  • Program development and delivery of services, and
  • Consumer involvement
  • Systems change advocacy

6
What Does Accountability for Results Mean?
  • In general, accountability for results refers
    to accomplishments that are under the direct or
    indirect influence of project resources and
    activities occur within the boundaries of a
    funding/budget cycle.
  • Within the NIDRR context, this translates into
    accountable for three types of results
  • Outputs planned and produced to date
  • Short-Term Outcomes anticipated and actual, and
    in some cases
  • Intermediate Outcomes anticipated or actual

7
Origins of the Accountability Shift in
Government GPRA
  • In 1993, Congress passed the Government Results
    and Reporting Act, which established uniform
    requirements for strategic planning, annual
    planning and reporting, and agency-grantee
    partnerships.
  • Key Provision of GPRA Provides a new definition
    of program evaluation an assessment, through
    objective measurement and statistical analysis,
    of the manner and extent to which federal
    programs achieve intended objectives.

8
Recent Policy Initiatives Reinforcingthe
Accountability Shift
  • The Presidents Management Agenda (PMA) FY 2002,
    with its emphasis on
  • Performance management measurement
  • Objective criteria to assess program results
    (i.e., the PART)
  • Performance-based budgeting
  • The Dept. of Educations New Strategic Goals,
    2002-2007, with an emphasis on
  • Transforming Education into an Evidenced-based
    Field.

9
Part 2
  • The Language and Logic of
  • Outcomes Planning and Managing for Results
  • (Key Concepts and Definitions)

10
What is Meant by Managing for Results?
  • Managing for results requires developing a new
    mindset based on outcomes-oriented goals and
    performance measures, as opposed to
    activity-oriented goals and objectives .
  • The primary focus is on planning for what outputs
    and outcomes will be achieved not on what
    activities will be conducted.
  • For most of us, this involves re-expressing
    existing program goals and objectives into the
    language of planned outputs and anticipated
    outcomes.

11
Concepts and Tools You Need to Succeed
  • Performance management
  • Performance measurement
  • Performance goals measures
  • The Performance Spectrum
  • Logic modeling i.e., reverse mapping
  • Components of Performance measurement
  • Target systems and/or populations

12
What is Performance Management vs. Performance
Measurement?
  • Performance management is an approach to
    planning and evaluation that utilizes the
    concepts and tools of performance measurement and
    logic modeling to identify performance goals and
    assess progress towards goals.
  • Performance measurement involves the ongoing
    monitoring and reporting of program
    accomplishments, particularly progress towards
    pre-established goals (see GPRA requirements).

13
What are Performance Goals?
  • According to OMB, performance goals are the
    desired levels of performance expressed as
    measurable objectives against which actual
    achievement can be compared.
  • To be complete, performance goals should
    incorporate performance measures and performance
    targets into timeframes.

14
Definitions of Performance Goals (Contd.)
  • Performance measures are the quantitative or
    qualitative metrics and indicators that are used
    to demonstrate progress towards goals and
    document results/ accomplishments.
  • Performance targets are the quantifiable or
    otherwise measurable characteristics that tell
    how well a program or project must accomplish a
    performance measure.

15
The Spectrum of Performance Outcomes-Oriented
Goals
  • Although the PART emphasizes outcome goals, that
    focus on changes and/or improvements in the
    target system, performance goals can also be
    stated in terms of planned outputs (and
    sometimes quality of activities).
  • The key All performance goals should clearly
    distinguish between components of performance and
    there should be a logical connections among them
    with outputs supporting outcomes (i.e.,
    outcome-oriented performance goals).

16
What is Logic Modeling?
  • A vehicle for dialogue, planning, program
    improvement and evaluation
  • A graphic representation or blueprint of the
    key elements of a program or project, and how it
    will work under certain conditions to solve
    identified problems.
  • A helpful tool for identifying outcomes and
    depicting the chain of events that link inputs
    and activities to outputs and outcomes.

17
What Does A Basic Logic Model Look Like? Prepared
by the NIDRR PPBE Division1
HOW
WHY
Span of Accountability for Results
Short-Term Outcomes
Program Activities
Intermediate Outcomes
Longer Term Outcomes
Outputs
Inputs Infrastructure
Customers Who Use Outputs
PROGRAM DELIVERED
RESULTS FROM PROGRAM
EXTERNAL CONDITIONS INFLUENCING SUCCESS (/-)
ANTECEDENT/ MEDIATING
MEASURES
1 Source Adapted from McLaughlin, J. A.
Jordan, G. B. (In Press/2003) Logic models a
tool for describing program theory and
performance, in Wholey, et. al. Handbook of
Practical Evaluation. Jossey-Bass. The lead
author may be contacted by email
atmacgroupx_at_aol.com NoteThe above logic model
does not include the standards by which the
quality, relevance, and productivity of
performance will be measured and evaluated across
the spectrum. For NIDRR grantees this is provided
by the centers of excellence (CoE) criteria.
Updated on June 27, 2003
18

What are the Components of Performance
Measurement?
  • Inputs Infrastructure The human and financial
    resources and systems needed to conduct a high
    quality, outcomes-oriented program or project.
  • Examples agency priorities and requirements,
    host institutional support and leveraging, staff
    competencies and expertise, management practices
    and evaluation plans, partnerships and
    collaborations, and previous accomplishments.

19
What are the Components of Performance
Measurement? (Contd)
  • Program Activities Are the action steps, tasks,
    procedures, and services performed in conjunction
    with implementing your planned program of public
    awareness, technical assistance and training,
    Outreach, and interagency coordination to carry
    out objectives and produce results.

20
What are the Components of Performance
Measurement? (Contd)
  • Outputs Are the direct results of program
    activities and consist of the findings or
    conclusions, products and services produced and
    reported to external audiences.
  • Outputs typically are reported using statistics
    or quantitative counts e.g., the of
    individuals trained, newsletters distributed,
    publications produced, participants attended.
  • But they also can be described qualitatively in
    terms of the nature of the findings or
    discoveries.

21
What are the Components of Performance
Measurement? (Contd)
  • More On Outputs
  • Outputs are important in their own right as
    indicators of productivity they also are the
    essential building blocks of outcomes.
  • However, not every output will have a
    corresponding outcome, nor should it.

22
What are the Components of Performance
Measurement? (Contd)
  • Utility Although not an official component of
    the performance spectrum as defined by GPRA,
    NIDRR has included utility because of its
    centrality to our Mission, and because
    demonstrating the perceived usefulness of
    outputs to sub-groups within the broader target
    systems constitutes an important bridge between
    outputs and short-term outcomes.

23
What are the Components of Performance
Measurement (Contd)
  • Outcomes Are the anticipated or actual effects
    of program activities and outputs and constitute
    changes or improvements in the target systems
    being served. Although the emphasis is on
    planning for outcomes, outcomes may also occur as
    unexpected changes.
  • Outcomes are the most complex component of PM and
    are difficult to define outside of the context of
    a specific program and target system.

24
What are the Components of Performance
Measurement (Contd)
  • Factors to take into account in planning for and
    documenting outcomes
  • The basic outcomes equation is Outcomes
    change/improvement in the target population or
    system, which is a function of
  • Outputs Documented External Use
  • Outcomes typically occur sequentially over time
    and, as a result, may affect more than one target
    system. Because of this, we refer to chain of
    outcomes, starting with short-term outcomes and
    progressing to intermediate and longer-term
    outcomes.

25
Types of Outcomes
  • Short-term outcomes are expected or actual
    changes or improvements in the identified target
    system that are more under the direct influence
    of center activities. Short-term outcomes
    represent the first level of change that must
    occur in order to bring about intermediate
    outcomes. Typically, short-term outcomes are
    focused on changes/improvements in the learning
    knowledge system at the individual or
    environmental level of analysis.

26
Types of Outcomes
  • Intermediate outcomes are the gold-standard in
    terms of accountability for results. They consist
    of expected or actual changes or improvements in
    the action system that occur in part as a
    result of the use or adoption of project outputs.
    Unlike short-term outcomes that occur under the
    direct influence of program activities, there may
    be other causal factors contributing to the
    achievement of intermediate.

27
Types of Outcomes (Contd)
  • Longer-term outcomes are the desired end-results
    of a program, and constitute changes or
    improvement in the overall condition of a
    population or system. Given their scope,
    longer-term outcomes usually take more than one
    funding cycle to achieve, and therefore are not
    required under the accountability for results
    umbrella. Their primary function is to serve as
    critical anchor points in the planning process.

28
Examples of Types of Outcomes (Changes or
Improvements) at the Individual Level
Focus on Learning and Knowledge Focus on Action Focus on Conditions
Awareness Knowledge Attitudes Skills Opinion Motivation Function Behavior Practice and/or Clinical care Advocacy Choice or decision-making Empowerment/Self-Efficacy Physical or Mental Health Status Employability Independent Living Community Integration, Participation Economic Self Sufficiency Quality of Life
29
Examples of Types of Outcomes (Changes or
Improvements) at the Environmental Level
Focus on Learning and Knowledge Focus on Action Focus on Conditions
Level or Availability of New Knowledge Models, frameworks or guidelines Methods or Tools Products and/or devices Services Technologies Use or Adoption of New Knowledge Models, frameworks or guidelines Methods or Tools Products and/or devices Services Technologies Standards Policies, laws and/or regulations Capacity of systems or society Employment Rates Rates of poverty among PWD Rates of Secondary Conditions Rates of Premature Mortality Delivery Systems Environmental Access (barrier elimination) Rates of participation
30
What is a Target System?
  • Formally defined, a target system refers to a
    unit of analysis or bounded set of interacting
    elements.
  • In the NIDRR context, a target system or
    population refers to the level(s) of the
    environment and/or to the group(s) of individuals
    that program activities and outputs are intended
    to affect.
  • Specifying the particular sub-groups or sample
    populations within the broader target system that
    your program intends to affect is key to
    identifying anticipated outcomes.

31
Examples of the Types of Target Systems Typically
Involved in NIDRR-Funded Projects
  • Individual level systems persons with disability
    and/or their family members, clinicians, service
    providers, other researchers, policy makers, and
    industry representatives.
  • Environmental-level systems the knowledge base
    in a field, treatment/practice standards, product
    development standards, availability of new
    methods, tools, products, services, and
    technologies, policies and/or service delivery
    systems, manufacturers, and societal conditions.

32
Part 3
  • How to Apply Performance Management Tools to Plan
    for Outcomes Manage for Results

33
Objective of Outcomes Planning
  • ?To identify a limited number of high-priority,
    problem-focused, reasonably ambitious and
    measurable outcomes-oriented performance goals
    that
  • Reflect NIDRR priorities or statutory
    requirements, and
  • Capture the anticipated effects and benefits of
    project activities outputs on identified target
    populations and/or systems.

34
Two Ways to Get to OutcomesProspective vs.
Retrospective Planning
  • Prospective outcomes planning is the preferred
    strategy, but it requires starting with a
    managing for results mindset and identifying
    performance goals and measures to guide
    activities and monitor progress (slides 35- 42 ).
  • Retrospective outcomes planning is less than
    ideal, but is how most of us beginning to get our
    feet wet. It involves identifying outcomes
    after-the-fact and collecting data to support
    them (slides .

35
Prospective Outcomes Planning Step 1
Preparation
  • Describe the overall problem you are trying to
    solve and specify what part or parts of the
    problem will be addressed by your program
    activities (i.e., theory of the problem clarifies
    scope of your project).
  • Identify the specific sub-groups or sample
    populations within the broader target systems
    that you anticipate will change/ improve as a
    result of your program.
  • Explain how your program activities outputs
    will change or benefit the specific target
    systems or populations you have identified (i.e.,
    theory of the program)

36
Prospective Outcomes Planning Step 2
Constructing a Logic Model
  • Translate requirements and objectives of program
    into anticipated outcomes, starting with
    longer-term, end-results and working backwards to
    identify intermediate and short-term outcomes
    (see Worksheet below)
  • Identify the specific target populations and
    systems to be affected by each type of outcome
  • Plan the outputs that will be produced to bring
    about anticipated changes in target systems
  • Design the activities that must be conducted to
    produce planned outputs
  • Identify the resources needed to implement
    activities and
  • Document the contextual factors that could
    influence the success of your program.

37
Worksheet for Constructing a Logic Model of
Performance for Your Program Prepared by the
NIDRR PPBE Division1
Important Tip When planning, begin with the end
in mind and work from right to left. When
implementing, begin with inputs and
infrastructure and move from left to right
Span of Accountability for Results
Inputs and Infrastructure
Program Activities
Short-Term Outcomes
Intermediate Outcomes
Longer-term Outcomes
Outputs
Utility
What resources and management practices are you
using, or developing, to conduct planned
activities and produce desired results consistent
with the priority and program requirements?
What evidence do you have of the potential merit
and/or usefulness of your outputs?
What changes or improvements will occur in
awareness, understanding or knowledge and for
which target populations as a result of your
program activities and outputs? And, what
measurable indicators will you use to track
progress towards achieving these short-term
outcomes/ performance goals?
What activities are you currently conducting, or
plan to conduct, across program requirements to
fulfill objectives, produce intended outputs, and
assess their relevance and utility for target
audiences? How well are you implementing these
activities and what is your progress to date,
including problems encountered and actions taken?
What products, services, and information are you
providing, or do you plan to provide to target
audiences or customers to support anticipated
outcomes? And what is your productivity to date?
What changes or improvements do you anticipate
will occur in behavior, practice and/or policy,
and for which target populations as a partial
result of the use or adoption of new knowledge
generated by your program? And, what measurable
indicators will you use to track progress towards
achieving these intermediate outcomes/
performance goals?
What are the long-term changes or improvements in
societal conditions that you hope to contribute
to overtime through your program activities,
outputs and short-term and intermediate outcomes?

Internal and External Factors That Can Influence
Success
1 Source Adapted from McLaughlin, J. A.
Jordan, G. B. (In Press/2003) Logic models a
tool for describing program theory and
performance, in Wholey, et. al. Handbook of
Practical Evaluation. Jossey-Bass. The lead
author may be contacted by email
atmacgroupx_at_aol.com NoteThe above logic model
does not include the standards by which the
quality, relevance, and productivity of
performance will be measured and evaluated across
the spectrum. For NIDRR centers and model systems
this is provided by the centers of excellence
(CoE) criteria. Revised June 27, 2003
38
Prospective Outcomes Planning Step 3 Formulate
Performance Goals
  • Convert outcome statements into performance goals
    by
  • Identifying or developing the performance
    measures that will be used to document progress
    and demonstrate accountability for results (i.e.,
    outputs, short-term and intermediate outcomes).
  • Establishing a baseline and identifying
    performance targets that must be met to achieve
    outcomes.
  • Establishing the timeframe within which progress
    results will occur.

39
Prospective Outcomes PlanningStep 4 Evaluation
of Performance
  • Develop an evaluation plan that identifies the
    types of evidence you need to collect to
    determine how well your program is working and if
    it is achieving the anticipated results.
  • Specify the sources and methods of data
    collection you will use to measure performance
    and demonstrate progress towards
    outcomes-oriented goals (see Table on next slide
    for examples of data sources methods).

40
Examples of Sources and Methods of Data
Collection for Demonstrating Outcomes
41
Prospective Outcomes PlanningStep 5 Evaluation
of Performance
  • Integrate your performance management plan into
    the ongoing operation of your program activities
    to measure performance, collect outcomes data
    and monitor progress.

42
Prospective Outcomes PlanningStep 6 Analysis
Documentation
  • Analyze the qualitative and quantitative
    performance data to document progress towards
    anticipated outcomes.
  • Interpret the data to determine if an outcome has
    occurred or is likely to occur within the
    specified timeframe. Making this determination
    relies on qualitative judgments of quantitative
    and/or qualitative data.
  • Document results and communicate accomplishments
    to external audiences is an evidence-based
    claims-making process, in which grantees present
    evidence to support the claim that their program
    activities outputs have resulted in changes/
    improvements in the identified target system.

43
Retrospective Outcome Identification Problems
with Approach
  • Missing pre-established performance goals to
    guide activities and outputs
  • Missing specification of target systems where
    outcomes are expected to occur
  • Missing a logic model that clarifies the
    interconnections among activities, outputs and
    outcomes and between short-term and intermediate
    outcomes
  • Missing performance measures and systematic
    outcomes data to monitor progress and demonstrate
    results

44
Retrospective Outcome Identification Suggested
Implementation Strategies
  • Form an AT Act Work Group to translate statutory
    requirements into a limited number of
    high-priority (i.e., 2-3) and measurable
    outcomes-oriented performance goals for each core
    activity public awareness, TA training,
    outreach, and interagency coordination.
    (suggested of outcomes is
  • Review and categorize accomplishments produced to
    date into outputs and types of outcomes
    (short-term, intermediate and long-term).

45
Retrospective Outcome Identification
Implementation Strategies (Contd.)
  • Link accomplishments to newly formulated
    performance goals and identify the types of data
    available and the data elements needed to
    demonstrate progress towards outcomes and
    document the contribution of AT Act project
    activities and outputs.
  • Revise web-based annual performance reporting
    form for FY 2004 to collect necessary data and
    rethink how existing data can be analyzed to
    provide evidence of progress or achievement of
    outcomes.

46
Common Weaknesses in Prospective Retrospective
Outcomes Identification
  • 1. Lack of specificity in what and where
  • 2. Too many outcomes and expressed too narrowly
    or too broadly
  • 3. Outcomes expressed as speculations vs. as
    expectations
  • 4. Confusion between levels of the performance
    spectrum (i.e., activities vs. outputs and
    outcomes and short-term vs. intermediate
    outcomes)
  • 5. Missing supporting links between outputs and
    short-term outcomes
  • 7. Missing evidence of outcomes

47
Exercise Whats Missing in the Following AT Act
Project Outcomes?
  • Outreach Within the last grant cycle, State X
    provided technical assistance to 4, 127
    individual and demonstration of loan of
    equipment to 1,415 individuals.
  • Training State X provided 102 training sessions
    attended by 3,8288 which resulted in increased
    knowledge levels that have prompted systems to
    address AT needs and responsibilities within
    policies and procedures.

48
Exercise Illustration of Reformulated Outcome
Statement
  • Original Statement Systems Change Intermediate
    Outcome State AT program worked with the state
    election officials on the HAVA state plan which
    resulted in adoption of the FEC 2002 access
    standards for voting machines.
  • Suggested Reformulation Conduct legislative
    initiatives that result in the development
    (short-term outcome) and adoption by states
    (intermediate outcome) of new standards for
    voting machines to increase access for adults
    with moderate to severe disabilities.

49
Part 4
  • Lessons Learned
  • The Utility and Challenges of Implementing
    Performance Measurement

50
Benefits of Utilizing Performance Management
  • Provides a common language and integrated
    framework to facilitate
  • Strategic planning program design
  • Identification of performance goals
  • Program integration coordinates the
    contributions of multiple projects and tasks
  • Objective measurement of progress towards goals
  • Program monitoring improvement
  • Program accountability -- documentation of
    results, and
  • Communication of success stories

51
Challenges of Applying Performance Management
Concepts Tools
  • Developing a managing for results mindset. 
  • Incorporating requirements of performance
    measurement into the traditional standards of
    excellence for service delivery, consumer
    involvement, and productivity.
  • Learning to re-express core program objectives
    into the language of outcomes-oriented
    performance goals (outputs and outcomes).
  • Deciding what evidence you need to collect and
    how to measure performance and determine how your
    program is working (i.e., quality and
    relevance/utility) and if you are achieving
    planned outputs and anticipated outcomes.

52
Challenges of Applying Performance Management
(Contd)
  • Integrating the tools of performance management
    into the operation of your program.
  • Using the information gathered to improve program
    management, evaluate performance, achieve
    results, and provide evidence to support claims
    of short-term intermediate outcomes.
  • Documenting results and communicating successes
    using a claims-making process based on
    qualitative judgments of quantitative
    and/qualitative data

53
Lessons Learned from Implementing Performance
Management in Business Government
  • What gets measured gets done. (Drucker)
  • If you dont measure results, you cant tell
    success from failure (C. Mindel).
  • According to OMB, the number one cause of federal
    agencies getting a low score on the new Program
    Assessment Reporting Tool (PART) is NOT having
    results-oriented performance measures to gauge
    the outcomes of a program (http//www.transparentg
    overnment.org/tg/analysis.htm).

54
Lessons Learned from Implementing Performance
Management (Contd)
  • Accountability for results involves a partnership
    -- NIDRR cant satisfy GPRA and PART alone! To
    succeed in the new era of performance-based
    budgeting, NIDRR needs data on accomplishments
    from grantees like you.
  • Key Assumption All grantees have
    accomplishments what missing are the performance
    goals and outcome data.
  • Key Assumption If you can demonstrate results,
    you can win public support (C. Mindel).

55
Thank you
Write a Comment
User Comments (0)
About PowerShow.com