Program Evaluation - PowerPoint PPT Presentation


PPT – Program Evaluation PowerPoint presentation | free to view - id: 6a5d0-ZDc1Z


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Program Evaluation


The purpose of an evaluation is to assess the effects or effectiveness of an ... Fitz-Gibbon, C. T., & Morris, L. L. (1987). How to design a program evaluation. ... – PowerPoint PPT presentation

Number of Views:62
Avg rating:3.0/5.0
Slides: 47
Provided by: andre251


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Program Evaluation

Program Evaluation
  • What is it and
  • how do you do it?

Evaluation a purpose not a strategy
  • The purpose of an evaluation is to assess the
    effects or effectiveness of an innovation or
    intervention policy, practice, or service
  • Attempt to assess the worth of value of some
    innovation or intervention
  • Evaluation highlights change past and future

  • Hayes, S. C., Barlow, D. H., Nelson-Gray, R. O.
    (1999). The scientist-practitioner Research and
    accountability in the age of managed care (2nd
  • Rossi, P. H., Freeman, H. E. (1989). Evaluation
    A systematic approach (4th ed.). Newbury Park,
    Calif. Sage Publications.
  • Shaddish, W. R., Cook, T. D., Leviton, L. C.
    (1993). Foundations of program evaluation
    Theories of practice. Newbury Park, Ca. Sage.

Rossi Freemans Model
  • (i) a comprehensive evaluation,
  • (ii) a tailored evaluation, and
  • (iii) a theory-driven evaluation.

Novel (Innovative) Programs or Formative
  • Conceptualization
  • problem description
  • operationalising objectives
  • development of intervention model
  • define extent and distribution of target
  • specify delivery systems
  • Implementation
  • research and development
  • implementation monitoring
  • Utility
  • impact studies (determine program results and
  • efficiency analyses

Recent Programs (Requiring Fine Tuning)
  • Conceptualization
  • identifying needed program
  • redefining objectives
  • designing program modifications
  • Implementation
  • research and development
  • program refinements
  • Utility
  • monitoring program changes
  • impact studies
  • efficiency analyses

Established Programs or Summative Evaluation
  • Conceptualization
  • determining evaluability
  • developing evaluation model
  • identifying potential modification opportunities
  • determining accountability requirements
  • Implementation
  • program monitoring and accountability studies
  • Utility
  • impact studies
  • efficiency analyses

Program Evaluation
  • (Rossi Freeman)

Features of an evaluation
  • Utility must be useful to an audience
  • Feasibility practical in political, practical,
    and cost-effectiveness terms
  • Propriety must be ethical and fair
  • Technical Adequacy carry it out with technical
    skill and sensitivity

Evaluation Models (House, 1978)
  • Systems analysis inputs, outputs to look at
    effectiveness and efficiency
  • Needs-based evaluation examines extent to which
    clients needs are being met
  • Accreditation external accreditors determine
    the extent to which a program meets professional
  • Decision-making evaluation structured to assist
    decision making
  • Illuminative inductive, naturalistic enquiry
  • Responsive responds to all stakeholders

1. Comprehensive Evaluation
  • Purposes
  • conceptualization and design of evaluations,
  • monitoring of program implementation, and
  • assessment of program usefulness.

1. Comprehensive Evaluation Questions
  • is a new program cheaper?
  • does a new training improve outcomes?
  • is the new program getting sufficient
    administrative support?
  • how do the trainees and trainers feel about the
    new program?
  • are the contents of the program valid?
  • are some parts of the program more effective than

1.1 Conceptualization and Design
  • Program need be documented
  • It needs to be assured that the intervention
  • is targeted appropriately,
  • is reaching the target population, and
  • is ameliorating the need

1.2 Monitoring of Implementation
  • is the program reaching its target population?
  • is the structure of services consistent with the
    program specifications?
  • why is site X having poorer success with a
    program than expected?
  • what modifications is the program robust against
    and what modifications always lead to changes in

1.3 Assessment of usefulness
  • If the program has not demonstrable effect it
    should not be continued
  • Political and social forces may be more important
    in assuring the continuation of a program than
    its overall efficacy (e.g., sexual assault
  • In this context cost-benefit analyses often carry
    more weight than clear experimental
    demonstrations of effectiveness
  • Program evaluation is not the dispassionate
    investigation of the efficacy of treatment
    effectiveness, but a political activity that aims
    to ensure that programs improve the human
    condition but achieves this end by influencing
    policy and politics

1.3 Assessment of usefulness Purposes
  • Clear
  • Accessible
  • Useful
  • Relevant
  • Humane
  • Compatible
  • Worthwhile

2. Tailored Evaluation
  • Comprehensive evaluations are often impractical
    gt tailored evaluation
  • still comprehensive, as the evaluator continues
    to ask questions about conceptualization,
    implementation, and utility, but asks them in
    different ways or with different procedures
    depending on the stage of the program
  • Divide programs into
  • novel (or innovative programs),
  • recent (requiring fine-tuning), and
  • established programs

Aside Stuffelbeams approach
  • 1. Context evaluation Defines the situational
    context, identifies the target population, needs,
    solutions, and possible problems to implementing
    the solutions
  • 2. Input evaluation Aims to identify the
    system capabilities, alternative program
    strategies, and procedural designs and strategies
  • 3. Process evaluation Aims to identify or
    predict defects in the process of the program
  • 4. Product evaluation Collect descriptions and
    judgments of outcome and to relate them to the
    program's objectives

Aside Activities of Evaluation
  • Awareness evaluation who knows about the
    program? What do they know?
  • Cost-benefit
  • Cost-effectiveness
  • Criterion-referenced are specific objectives
    being met?
  • Quality assurance Minimum standards met? How can
    quality be monitored and demonstrated?

3. Theory-Driven Evaluation Why?
  • Social change is brought about by political
    decisions (expert witness)
  • Outcome of a program evaluation will rarely bring
    about change is that programs can do little to
    modify the structure of society and therefore the
    ultimate effects are always limited
  • Allows us to interpret a failed outcome
  • Without a theory to guide program evaluation, we
    are left to rely on trial-and-error-learning
  • Internal vs external validity causal modeling
    derived from a theory enables one to steer a path
    that maximizes internal and external validity
  • Addresses internal validity by ensuring adequate
    model specification within the context of a piece
    of evaluation research.
  • By entering all of variables that may lead one to
    misinterpret the effect of a confounding variable
    as a treatment effect, influence of these can be

4. Conducting a Program Evaluation
  • Cycle of questions to refine the evaluation
  • What are the evaluation questions?
  • What will be reported to target audiences?
  • What information will be gathered?
  • How will the information be analyzed?

4.1 Considerations in Program Evaluation
  • Who's The Boss?
  • Each stakeholder will have an investment in a
    certain outcome, which places the evaluator in a
    difficult position
  • Whose side to take? (e.g., the choice of measures
    will influence the outcomes observed)
  • Sponsors of the evaluation may turn on the
    evaluators if the outcomes are not as expected.
    Future funding may be withdrawn if results are
    not as desired or if policy decisions will be
    made regardless of the data.

4.1 Considerations in Program Evaluation
  • Evaluator can take side of one stakeholder or
    play an even-handed game
  • Impartial evaluators are likely to have a greater
    impact on the stakeholder and are more likely to
    be co-opted by the stakeholder again
  • Narrow loyalty is associated with being within an
    organization in best position to ensure
    organizational cooperation
  • Rossi and Freeman suggest that an evaluator must
    make a moral judgment.
  • Who are the relevant audiences?
  • sponsors or commissioners of the evaluation
    decision makers
  • target group from whom the information is
  • program planners/creators people running program
    interested parties
  • people with a right to the information
  • people whose roles may be affected by evaluation

4.1 Considerations in Program Evaluation
  • How long is too long?
  • Conflict between the time taken by evaluation and
    the time before policy decisions are made
  • Three principles to guide practice
  • Do not begin a long-term study if the outcome is
    needed before the study will be finished. It may
    be better to rely on expert opinion if the time
    for the study is not possible.
  • Avoid technically complex evaluations at these
    times (better in pilot research)
  • Attempt to second-guess the direction of programs
    and policies and conduct evaluations in advance.

4.1 Considerations in Program Evaluation
  • Quantitative Versus Qualitative Methods
  • There is a debate about whether the best measures
    are quantitative or qualitative

4.2 Evaluation of Novel Programs
  • Ensure that the program is well implemented and
    well evaluated
  • Evaluator need not be involved in the planning of
    the programming or issues to with targeting, etc.
    The evaluator is needed to provided refinement to
    permit evaluation
  • One important role of the evaluator is turning
    program goals into program objectives. Goals are
    abstract, idealized statements about the desired
    purpose of a program. Programs cannot be designed
    around goals.
  • Objectives are more detailed operational
    statements about measurable criteria of success
  • Discrepancies - that the aim of the program is to
    reduce a discrepancy between an undesired and a
    desired state of affairs
  • In order to do this, a needs assessment needs to
    be conducted

Needs Assessment.
  • Stufflebeam, D. L., McCormick, C. H., Binkerhoff,
    R. O., Nelson, C. 0. (1985). Conducting
    educational needs assessment. Boson

Needs Assessment
  • 1. Preparation.
  • Identification of questions to ask,
  • the method of asking,
  • the target participants,
  • the methods of information collection and
  • and the possible uses to which the results will
    be put
  • By the end of the preparation phase you should be
    able to answer
  • who will address what questions?
  • with what methods?
  • using what resources?
  • according to what schedule?
  • for what purposes?

Needs Assessment (Cont.)
  • 2. Information Gathering.
  • How
  • 3. Analysis.
  • The results must be accurate, timely,
    understandable, and relevant to target audiences.
  • 4. Reporting.
  • 5. Liaising and Applying.
  • The difficulty in program evaluation is
    identifying appropriate goals, which are relevant
    to the needs of the target sample but also
    address the goals of stakeholders in as unbiased
    manner as possible.

Steps in Carrying Out a Needs Assessment (Robson,
  • Identify possible objectives
  • Be comprehensive
  • Use literature, experts, and those involved in
    the program
  • Decide on important objectives
  • Get stakeholders to rate importance of potential

Steps in Carrying Out a Needs Assessment (Cont.)
  • Assess what is currently available to meet the
    important objectives
  • Survey, records, compare meeting of needs now
    with before
  • Select final set of objectives
  • Must be high in importance and low in current

Delivery System Design
  • For novel programs this will involve assessment
  • target problem and population service
  • implementation qualifications and competencies of
  • mechanisms of recruiting staff
  • means of optimizing access to the intervention
  • location and characteristics of delivery sites
  • referral and follow-up procedures
  • Therefore, the evaluator will need to consider
  • the characteristics of the organization,
  • methods of obtaining cooperation from relevant
  • relationships with other existing programs,
  • the political and social context of the program,
  • and the availability of required resources.

Delivery System Design (Cont.)
  • Specifically, the program should consider
  • Overcoverage (comparing individuals who
    participate in the program with those who drop
  • Undercoverage (comparing individuals who
    participate in the program with those who are
    eligible but do not participate),
  • Biased coverage (comparing individuals who
    participate in the program with those who drop
    out and those eligible non- participants)

Outcome Evaluation
  • Outcome evaluation is inappropriate without
    evaluation of targeting and implementation of a
  • The strengths of novel evaluations are that they
    are often involve only partial coverage and
    therefore it is possible to identify control
  • In more established programs, the coverage is
    total and the most appropriate controls are the
    subjects prior to the evaluation.

Outcome Evaluation
  • When randomization is not possible the usual
    options in forming control groups are
  • matched controls reflexive controls (subjects are
    their own baseline)
  • generic controls (from established norms)
  • shadow controls (constructed from the opinions of
    knowledgeable stakeholders and experts about what
    would be expected without an intervention).
  • Use gt one control, since converging data will
    increase confidence

Outcome Evaluation
  • Direct Effectiveness Measures
  • Indirect Effectiveness Measures Client
  • Indirect Effectiveness Measures Social Validity
  • Assessing costs

Design Selection
  • Good enough" rule while randomization provides
    the best control, the evaluator needs to take
    into account practicality, feasibility, the
    resources available, and the expertise of the
  • In the context of the many constraints imposed in
    "real world' evaluation, it is most appropriate
    to choose the best achievable design

4.3 Evaluation of Recent Programs
  • The act of fine tuning focuses attention on
    program modifications that increase the magnitude
    of impact or decrease the cost per unit of impact
    while maintaining the efficacy of service
  • Rossi and Freeman suggest that program evaluation
  • re appraising objectives
  • identifying program modifications
  • undertaking reputability assessments
  • program re-planning and redesign
  • planning and implementing evaluation designs to
    monitor program changes and their impacts

4.4 Evaluation of Established Programs
  • Program conceptualization shifts in focus to
    assessing what already exists. Therefore,
    conceptualization may involve
  • preparing a program description
  • interviewing the relevant personnel
  • scouting the program
  • developing an evaluable program model
  • identifying evaluation users
  • achieving agreement to proceed

4.4 Evaluation of Established Programs (Cont.)
  • Monitoring will involve an assessment of what
    inputs, activities, and processes occur in a new
    version of the program to improve the fidelity to
    the program's intent, and to provide a clear base
    for relating causal processes to outcomes
  • An established program will be monitored to
    ensure accountability. Use
  • Management information system, which constantly
    monitors the program, but this is expensive in
    the long term
  • Cross- sectional survey, which although it is
    cheaper in the long-term, has expensive start-up
    costs and may be resisted by staff
  • The main focus in an established program is to
    ensure that there are minimal discrepancies
    between what how the program is formally seen to
    be operating and how it is operating in reality
  • Independence of evaluators
  • Should a program undertake its own evaluation, or
    should external evaluators be used?

4.4 Evaluation of Established Programs (Cont.)
  • Cost-benefit analysis-gt computes all program
    costs, and then compares them to program outcomes
    expressed in a common unit (e.g., dollars). The
    main problems facing such analyses is that all
    program costs and benefits need to be identified
    and measured, and they need to be expressible in
    a common metric.
  • Necessary that the program meets five conditions
  • 1. The program must have independent and
    separable funding.
  • 2. The program must be certain that its net
    effects are significant.
  • 3. The program's impact and magnitude must be
    known of calculable.
  • 4. The benefits can be expressed in monetary
  • 5. Decision makers are considering alternative
  • Usually, the fourth condition is not met, and
    Rossi and Freeman suggest that under these
    circumstances a cost-effectiveness analysis be
  • Cost-effectiveness analysis -gt only requires the
    identification in a common unit the program
    costs. The benefits remain expressed in outcome

5. Using Evaluation Findings
  • At the end of the day, what counts is not the
    program evaluation, but the extent to which the
    program evaluation modifies policies, programs,
    and practices
  • Consider instrumental use - that is, ensuring the
    use of the findings by decision- makers and other
  • understand the work of the decision-makers
    (therefore, present the results briefly and
  • ensure that the evaluation results are timely and
    are available when they are needed respect
  • be sensitive to stakeholders' program commitments
    plans for utilization and dissemination of
    results should be part of the evaluation design
  • Evaluation programs should assess the extent to
    which the results of an evaluation are utilized

6. Checklist for Evaluation
  • Pounds The cost of the evaluation has been
  • Politics The relevant people have been
  • Procedures The procedures are practical given
    the purpose of the evaluation

  • Berk, R. A., Rossi, P. H. (1990). Thinking
    about program evaluation. Newbury Park Sage.
  • Binkerhoff, R. O., Brethower, D. M., Hluchyj, T.,
    Nowakowski, J. R. (1983). Program evaluation A
    practitioner's guide for trainers and educators
    Design manual. Hingham, Ma. Kluwer-Nijhoff.
  • Binkerhoff, R. O., Brethower, D. M., Hluchyj, T.,
    Nowakowski, J. R. (1983). Program evaluation A
    practitioner's guidefor trainers and educators
    Sourcebook and casebook. Hingham, Ma.
  • Cronbach, L. J. (1982). Designing evaluations of
    educational and social programs. San Francisco.
  • Eisner, E. W. (1985). The art of educational
    evaluation. London Falmer.
  • Fitz-Gibbon, C. T., Morris, L. L. (1987). How
    to design a program evaluation. Newbury Park,
    Ca. Sage.
  • Hayes, S. C., Barlow, D. H., Nelson-Gray, R. O.
    (1999). The scientist-practitioner Research and
    accountability in the age of managed care (2nd
    ed.). Boston Allyn Bacon. (Especially chapters
    2 10).

References (Cont.)
  • Krosnick, J. A. (1999). Survey research. Annual
    Review of Psychology, 50, 537-567.
  • Maruish, M. E. (1999). The use of psychological
    testing for treatment planning and outcomes
    assessment (second edition). Mahwah, NJ.
    Lawrence Erlbaum.
  • Robson, C. (1993). Real world research. Oxford
    Blackwell. (Esp. Chapter 7)
  • Rossi, P. H., Freeman, H. E. (1989). Evaluation
    A systematic approach (4th ed.). Newbury Park,
    Calif. Sage Publications.
  • Shaddish, W. R., Cook, T. D., Leviton, L. C.
    (1993). Foundations of program evaluation
    Theories of practice. Newbury Park, Ca. Sage.
  • Stufflebeam, D. L., McCormick, C. H., Binkerhoff,
    R. O., Nelson, C. 0. (1985). Conducting
    educational needs assessment. Boson