Driving Quality: Making the Difference through Program Evaluation Processes - PowerPoint PPT Presentation


PPT – Driving Quality: Making the Difference through Program Evaluation Processes PowerPoint presentation | free to download - id: 66fb6c-Njc2Z


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Driving Quality: Making the Difference through Program Evaluation Processes


Driving Quality: Making the Difference through Program Evaluation Processes Ben Kearney, PhD Guidestone I think you should be more explicit here in Step Two ... – PowerPoint PPT presentation

Number of Views:2
Avg rating:3.0/5.0
Date added: 26 September 2019
Slides: 40
Provided by: BenjaminK153
Learn more at: http://umassociation.org


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Driving Quality: Making the Difference through Program Evaluation Processes

Driving Quality Making the Difference through
Program Evaluation Processes
  • Ben Kearney, PhD
  • Guidestone

  • I think you should be more explicit here in Step

  • thought Alice and she went on, Would you
    tell me, please, which way I ought to go from
  • That depends a good deal on where you want to
    get to, said the Cat.
  • I dont much care where said Alice.
  • Then it doesnt matter where you go, said the
  • so long as I get somewhere, Alice added as
    an explanation.
  • Oh, youre sure to do that, said the Cat, if
    you only walk long enough.
  • (Carroll, 1865, p. 51)


  • What gets measured gets done
  • If you dont measure results, you cant tell
    success from failure
  • If you cant see success, you cant reward it
  • If you cant reward success, youre probably
    rewarding failure
  • If you cant see success, you cant learn from it
  • If you cant recognize failure, you cant correct
  • If you can demonstrate results, you can win
    public support.
  • Osborne and Gaebler, 1992

Improvement Process
  • Data
  • Information
  • Knowledge
  • Wisdom

Inherent Questions
  • Does the program work as designedcorrelated
    process measures with outcome measures?
  • How does the program compare with other programs
    and against organizational/contractual/external
  • If effectiveness indicators are met, what
    efficiency changes can be made to maintain
    quality, but increase client saturation?

A priori assumptions
  • Therapy works
  • Common factors help facilitate change
  • Empathy, increasing awareness, therapeutic
  • Specific interventions permits change to happen
    even faster
  • We must be responsible stewards
  • Make an IMPACT

Components of Program Evaluation
  • Theory of Change
  • Logic Models
  • Process Measures (including Fidelity Measures)
  • Outcome Measures
  • Short termduring treatment
  • Mid term--At the end of treatment
  • Long termat some point following
    treatmentdepends on logic model and change
  • Status reports

Theory of Change vs. a Logic Model
  • a theory of change defines how and why you expect
    the desired outcomes to occur and usually applies
    to several programs or the whole organization
  • theories of change clarify why you are doing what
    you are doing
  • a logic model visually presents your
    understanding of the relationships among your
    programs resources, planned activities and
    anticipated results and usually applies to a
    single program.
  • Logic models clarify what you are doing

Theory of Change vs. a Logic Model
  • Theory of Change
  • Links outcomes and activities to explain how and
    why the expected change will occur
  • Usually starts with a goal before deciding on
    programmatic components
  • Requires justification for program components
    specifies the hypothesis about why something will
    cause something else
  • Requires identifying indicators to measure
  • Best used to design and evaluate a complex

Theory of Change vs. a Logic Model
  • Logic Model
  • Graphically illustrates program components,
    identifies, inputs, activities and outcomes
  • Usually starts with a program and illustrates its
  • Requires identification of program components,
    but doesnt show why activities are expected to
    produce outcomes
  • Sometimes includes indicators to measure outcomes
  • Best used to demonstrate you have carefully
    identified the inputs, outputs and outcomes of
    your work

  • A logic model is the first step in evaluation.
    Through evaluation, we test and verify the
    reality of the program theory how we believe
    the program will work.
  • A logic model helps us focus on appropriate
    process and outcome measures.

An Expanded Logic Model
Logic Model Components
  • Situation
  • Inputs
  • Outputsthe measurement of
  • Activities
  • Personnel
  • Outcomes

The situation
  • is the foundation for logic model development.
  • The problem or issue that the program is to
    address sits within a setting or situation--a
    complex of sociopolitical, environmental, and
    economic conditions.

Situational Questions
  • What is the problem/issue?
  • Why is this a problem? (What causes the problem?)
  • For whom (individual, household, group,
    community, society in general) does this problem
  • Who has a stake in the problem? (Who cares
    whether it is resolved or not?)
  • What do we know about the problem/issue/people
    that are involved? What research, experience do
    we have? What do existing research and experience

  • Inputs are the resources and contributions that
    you and others make to the effort, the resources
    invested that allow us to achieve the desired
    outputs. These include time, people (staff,
    volunteers), money, materials, equipment,
    partnerships, research base, and technology among
    other things.
  • These inputs allow us to create outputs

  • Outputs are the activities, services, events, and
    products that reach people (individuals, groups,
    agencies) who participate or who are targeted.
  • Outputs are "what we do" or "what we offer." They
    include workshops, services, conferences,
    community surveys, facilitation, in-home
    counseling, etc.
  • These outputs are intended to lead to specific

  • Outcomes are the direct results or benefits for
    individuals, families, groups, communities,
    organizations, or systems. Examples include
    changes in knowledge, skill development, changes
    in behavior, capacities or decision-making,
    policy development. Outcomes can be short-term,
    medium-term, or longer-term achievements.
    Outcomes may be positive, negative, neutral,
    intended, or unintended.

  • In the past, we've tended to focus on what is
    included in the outputs column - the "what we do
    and who we reach."
  • We are anxious to tell our clients, funders and
    community partners what it is that we do, the
    services we provide, how we are unique, who we
  • Now, the question is "What difference does it
    make?" This is a question about OUTCOMES.
  • The newest question is How does what you do
    produce the differences being made? This
    question links OUTPUTS to OUTCOMES.

  • the ultimate consequence or effects of the
    program--for example, increased economic
    security, reduced rates of teen smoking, improved
    air quality.
  • the ultimate, long-term outcome or your
    programmatic goal.
  • Impact refers to the ultimate, longer-term
    changes in social, economic, civic, or
    environmental conditions.

  • Assumptions are the beliefs we have about the
    program and the people involved and the way we
    think the program will work. This is the "theory"
    we are talking about the underlying beliefs in
    how it will work. These are validated with
    research and experience. Assumptions underlie and
    influence the program decisions we make.

More about Outcomes
  • Outcomes often fall along a continuum from
    shorter- to longer-term results. This continuum
    is called an "outcome line" (Mohr, 1995), the
    "outcome sequence chart" (Hatry, 1999), or
    "outcome hierarchy" (Funnell, 2000). This
    concept--a series of outcomes that are
    connected--is fundamental to a logic model.

More about Outcomes
  • For our process
  • short-termaccomplished during the program
  • medium-termaccomplished at the end of program
  • long-termaccomplished at some point after the
    program has ended

Starting at the End
  • When planning, start where you want to end.
  • Identify the long-term outcome(s).
  • What is your end goal?
  • What will be different?
  • How will the community, producers, local
    citizens, the environment, be different as a
    result of the program?
  • Once you have that long-term outcome (end result,
    goal) identified, then work backwards across the
    logic model

Outpatient Counseling Fidelity Checklist
  • The worker has utilized Nurturing Parenting
    principles and philosophies to guide their
    interaction with the referred child and/or
  • The therapist has established an optimal
    therapeutic alliance with the client and/or
    family by providing a secure base from which the
    client can explore the various unhappy and
    painful aspects of his/her life, past and
    present, through support, encouragement,
    sympathy, and guidance.
  • The therapist has assisted the client in
    identifying important attachment relationships
    in recognizing the role they play in their
    current functioning to rework unhealthy
    attachment related behaviors/patterns and to
    promote stronger relationships between the client
    and their children or loved ones in the present
    and for the future.
  • The therapist has promoted resiliency and
    generational change in the client and/or family.
  • The therapist has utilized Cognitive Behavioral
    Therapy (CBT) techniques to assist the client
    with monitoring and reduction of mental health

Outpatient Counseling Fidelity Checklist
  • The therapist has assessed for the clients
    commitment to change applied appropriate
    intervention technique and promoted client
    belief in the notion that therapy will help.
  • The therapist has facilitated the clients
    awareness of the factors that maintain his or her
  • The therapist has encouraged and assisted the
    client to consider the ways in which he or she
    engages in relationships with significant figures
    in his or her current life.
  • The therapist has helped the client recognize and
    change images of self and others that contribute
    to difficulties.
  • The therapist has encouraged the client to engage
    in corrective emotional experiences.
  • The therapist has emphasized ongoing reality
    testing in the clients life.
  • The therapist has worked to promote openness to
    empathy in the client.

Helpful evaluation tools
  • Website data entry
  • Built-in specific reports
  • Specific tools for analysis
  • Statistical change
  • Clinical change
  • Analysis tools
  • Run Charts
  • Pareto charts
  • Fishbone-cause and effect
  • Correlations/regressions

Pareto Chart
Pareto Chart-example
Run Chart--Example
Evaluation Potholes
  • Be aware where the evaluators purpose is
    different than the clinicians
  • Be aware when the evaluators purpose is the same
    as the clinicians
  • Be clear about roles and boundaries, but be
    willing to be flexible when needed as well
  • ex CCO, Evaluator

Evaluation Potholes
  • Use well established instruments
  • Computerization combines data gathering, editing,
    and entering into one step
  • Computerization permits automatic analysis on
    predetermined factors
  • Dont be afraid to pilotPilot when you can.
    Force field analysis for pilotdriving and
    restraining forces.

Current work
  • Spirituality
  • The openness to and the awareness of the
    transcendent, the sacred, that which is beyond
    us and by this knowledge, recognizing the
    connectedness and responsibility we have to each

Current Work
  • Moral emotional processes-build communication and
    collaboration, transmit culture
  • Quietness, openess, mindfulness, awareness,
    regulation, empathy
  • Transformational emotional processes
  • Hope, altruism, sympathy, truth (joy), narrative

  • Beckworth, L., (2000). Prevention science and
    prevention programs. In C. H. Zeanah, Jr. (Ed.),
    Handbook of infant mental health (pp 439-456).
    New York Gilford Press.
  • Breyfogle III, F. W. (2003). Implementing Six
    Sigma Smarter Solutions Using Statistical
    Methods (2nd ed.). Hoboken, NJ John Wiley
    Sons, Inc.
  • Brusse, W. (2004). Statistics for Six Sigma Made
    Easy! New York, NY McGraw-Hill.
  • Carey, Ph.D., R. G., Lloyd, Ph.D., R. C.
    (2001). Measuring Quality Improvement in
    Healthcare A Guide to Statistical Process
    Control Applications. Milwaukee, WI ASQ.

  • Cavanagh, R. R., Neuman, R. P., Pande, P. S.
    (2000). The Six Sigma Way How GE, Motorola, and
    Other Top Companies are Honing Their Performance.
    New York, NY McGraw-Hill.
  • Cavanagh, R. R., Neuman, R. P., Pande, P. S.
    (2002). The Six Sigma Way. New York, NY
  • Hayes, R. A., Stout, C. E. (Eds.) (2005). The
    Evidence-Based Practice Methods, Models, and
    Tools for Mental Health Professionals. Hoboken,
    NJ John Wiley Sons, Inc.

  • Langley, G. J., Moen, R. D., Nolan, K. M., Nolan,
    T. W., Norman, C. L., Provost, L. P. (2009).
    The Improvement Guide A Practical Approach to
    Enhancing Organizational Performance (2nd ed.).
    San Francisco, CA Jossey-Bass.
  • Parlakian, P. and Seibel, N.L. (2002). Building
    strong foundations Practical guidance for
    promoting the social-emotional development of
    infants and toddlers. Washington ZERO TO THREE
  • Stiffman, A. R. (Ed.) (2009). The Field Research
    Survival Guide. New York, NY Oxford University
About PowerShow.com