Program Evaluation

1
Program Evaluation

• What is it and
• how do you do it?

2
Evaluation a purpose not a strategy

• The purpose of an evaluation is to assess the
  effects or effectiveness of an innovation or
  intervention policy, practice, or service
• Attempt to assess the worth of value of some
  innovation or intervention
• Evaluation highlights change past and future

3
Introduction

• Hayes, S. C., Barlow, D. H., Nelson-Gray, R. O.
  (1999). The scientist-practitioner Research and
  accountability in the age of managed care (2nd
  ed.).
• Rossi, P. H., Freeman, H. E. (1989). Evaluation
  A systematic approach (4th ed.). Newbury Park,
  Calif. Sage Publications.
• Shaddish, W. R., Cook, T. D., Leviton, L. C.
  (1993). Foundations of program evaluation
  Theories of practice. Newbury Park, Ca. Sage.

4
Rossi Freemans Model

• (i) a comprehensive evaluation,
• (ii) a tailored evaluation, and
• (iii) a theory-driven evaluation.

5
Summary
6
Novel (Innovative) Programs or Formative
Evaluation

• Conceptualization
• problem description
• operationalising objectives
• development of intervention model
• define extent and distribution of target
  population
• specify delivery systems
• Implementation
• research and development
• implementation monitoring
• Utility
• impact studies (determine program results and
  effectiveness)
• efficiency analyses

7
Recent Programs (Requiring Fine Tuning)

• Conceptualization
• identifying needed program
• redefining objectives
• designing program modifications
• Implementation
• research and development
• program refinements
• Utility
• monitoring program changes
• impact studies
• efficiency analyses

8
Established Programs or Summative Evaluation

• Conceptualization
• determining evaluability
• developing evaluation model
• identifying potential modification opportunities
• determining accountability requirements
• Implementation
• program monitoring and accountability studies
• Utility
• impact studies
• efficiency analyses

9
Program Evaluation

• (Rossi Freeman)

10
Features of an evaluation

• Utility must be useful to an audience
• Feasibility practical in political, practical,
  and cost-effectiveness terms
• Propriety must be ethical and fair
• Technical Adequacy carry it out with technical
  skill and sensitivity

11
Evaluation Models (House, 1978)

• Systems analysis inputs, outputs to look at
  effectiveness and efficiency
• Needs-based evaluation examines extent to which
  clients needs are being met
• Accreditation external accreditors determine
  the extent to which a program meets professional
  standards
• Decision-making evaluation structured to assist
  decision making
• Illuminative inductive, naturalistic enquiry
• Responsive responds to all stakeholders

12
1. Comprehensive Evaluation

• Purposes
• conceptualization and design of evaluations,
• monitoring of program implementation, and
• assessment of program usefulness.

13
1. Comprehensive Evaluation Questions

• is a new program cheaper?
• does a new training improve outcomes?
• is the new program getting sufficient
  administrative support?
• how do the trainees and trainers feel about the
  new program?
• are the contents of the program valid?
• are some parts of the program more effective than
  others?

14
1.1 Conceptualization and Design

• Program need be documented
• It needs to be assured that the intervention
• is targeted appropriately,
• is reaching the target population, and
• is ameliorating the need

15
1.2 Monitoring of Implementation

• is the program reaching its target population?
• is the structure of services consistent with the
  program specifications?
• why is site X having poorer success with a
  program than expected?
• what modifications is the program robust against
  and what modifications always lead to changes in
  outcome?

16
1.3 Assessment of usefulness

• If the program has not demonstrable effect it
  should not be continued
• Political and social forces may be more important
  in assuring the continuation of a program than
  its overall efficacy (e.g., sexual assault
  treatment)
• In this context cost-benefit analyses often carry
  more weight than clear experimental
  demonstrations of effectiveness
• Program evaluation is not the dispassionate
  investigation of the efficacy of treatment
  effectiveness, but a political activity that aims
  to ensure that programs improve the human
  condition but achieves this end by influencing
  policy and politics

17
1.3 Assessment of usefulness Purposes

• Clear
• Accessible
• Useful
• Relevant
• Humane
• Compatible
• Worthwhile

18
2. Tailored Evaluation

• Comprehensive evaluations are often impractical
  gt tailored evaluation
• still comprehensive, as the evaluator continues
  to ask questions about conceptualization,
  implementation, and utility, but asks them in
  different ways or with different procedures
  depending on the stage of the program
• Divide programs into
• novel (or innovative programs),
• recent (requiring fine-tuning), and
• established programs

19
Aside Stuffelbeams approach

• 1. Context evaluation Defines the situational
  context, identifies the target population, needs,
  solutions, and possible problems to implementing
  the solutions
• 2. Input evaluation Aims to identify the
  system capabilities, alternative program
  strategies, and procedural designs and strategies
• 3. Process evaluation Aims to identify or
  predict defects in the process of the program
• 4. Product evaluation Collect descriptions and
  judgments of outcome and to relate them to the
  program's objectives

20
Aside Activities of Evaluation

• Awareness evaluation who knows about the
  program? What do they know?
• Cost-benefit
• Cost-effectiveness
• Criterion-referenced are specific objectives
  being met?
• Quality assurance Minimum standards met? How can
  quality be monitored and demonstrated?

21
3. Theory-Driven Evaluation Why?

• Social change is brought about by political
  decisions (expert witness)
• Outcome of a program evaluation will rarely bring
  about change is that programs can do little to
  modify the structure of society and therefore the
  ultimate effects are always limited
• Allows us to interpret a failed outcome
• Without a theory to guide program evaluation, we
  are left to rely on trial-and-error-learning
• Internal vs external validity causal modeling
  derived from a theory enables one to steer a path
  that maximizes internal and external validity
• Addresses internal validity by ensuring adequate
  model specification within the context of a piece
  of evaluation research.
• By entering all of variables that may lead one to
  misinterpret the effect of a confounding variable
  as a treatment effect, influence of these can be
  modeled

22
4. Conducting a Program Evaluation

• Cycle of questions to refine the evaluation
• What are the evaluation questions?
• What will be reported to target audiences?
• What information will be gathered?
• How will the information be analyzed?

23
4.1 Considerations in Program Evaluation

• Who's The Boss?
• Each stakeholder will have an investment in a
  certain outcome, which places the evaluator in a
  difficult position
• Whose side to take? (e.g., the choice of measures
  will influence the outcomes observed)
• Sponsors of the evaluation may turn on the
  evaluators if the outcomes are not as expected.
  Future funding may be withdrawn if results are
  not as desired or if policy decisions will be
  made regardless of the data.

24
4.1 Considerations in Program Evaluation

• Evaluator can take side of one stakeholder or
  play an even-handed game
• Impartial evaluators are likely to have a greater
  impact on the stakeholder and are more likely to
  be co-opted by the stakeholder again
• Narrow loyalty is associated with being within an
  organization in best position to ensure
  organizational cooperation
• Rossi and Freeman suggest that an evaluator must
  make a moral judgment.
• Who are the relevant audiences?
• sponsors or commissioners of the evaluation
  decision makers
• target group from whom the information is
  gathered
• program planners/creators people running program
  interested parties
• people with a right to the information
• people whose roles may be affected by evaluation

25
4.1 Considerations in Program Evaluation

• How long is too long?
• Conflict between the time taken by evaluation and
  the time before policy decisions are made
• Three principles to guide practice
• Do not begin a long-term study if the outcome is
  needed before the study will be finished. It may
  be better to rely on expert opinion if the time
  for the study is not possible.
• Avoid technically complex evaluations at these
  times (better in pilot research)
• Attempt to second-guess the direction of programs
  and policies and conduct evaluations in advance.

26
4.1 Considerations in Program Evaluation

• Quantitative Versus Qualitative Methods
• There is a debate about whether the best measures
  are quantitative or qualitative

27
4.2 Evaluation of Novel Programs

• Ensure that the program is well implemented and
  well evaluated
• Evaluator need not be involved in the planning of
  the programming or issues to with targeting, etc.
  The evaluator is needed to provided refinement to
  permit evaluation
• One important role of the evaluator is turning
  program goals into program objectives. Goals are
  abstract, idealized statements about the desired
  purpose of a program. Programs cannot be designed
  around goals.
• Objectives are more detailed operational
  statements about measurable criteria of success
• Discrepancies - that the aim of the program is to
  reduce a discrepancy between an undesired and a
  desired state of affairs
• In order to do this, a needs assessment needs to
  be conducted

28
Needs Assessment.

• Stufflebeam, D. L., McCormick, C. H., Binkerhoff,
  R. O., Nelson, C. 0. (1985). Conducting
  educational needs assessment. Boson
  Kluwer-Nijhoff.

29
Needs Assessment

• 1. Preparation.
• Identification of questions to ask,
• the method of asking,
• the target participants,
• the methods of information collection and
  analysis,
• and the possible uses to which the results will
  be put
• By the end of the preparation phase you should be
  able to answer
• who will address what questions?
• with what methods?
• using what resources?
• according to what schedule?
• for what purposes?

30
Needs Assessment (Cont.)

• 2. Information Gathering.
• How
• 3. Analysis.
• The results must be accurate, timely,
  understandable, and relevant to target audiences.
• 4. Reporting.
• 5. Liaising and Applying.
• The difficulty in program evaluation is
  identifying appropriate goals, which are relevant
  to the needs of the target sample but also
  address the goals of stakeholders in as unbiased
  manner as possible.

31
Steps in Carrying Out a Needs Assessment (Robson,
1993)

• Identify possible objectives
• Be comprehensive
• Use literature, experts, and those involved in
  the program
• Decide on important objectives
• Get stakeholders to rate importance of potential
  objectives

32
Steps in Carrying Out a Needs Assessment (Cont.)

• Assess what is currently available to meet the
  important objectives
• Survey, records, compare meeting of needs now
  with before
• Select final set of objectives
• Must be high in importance and low in current
  availability

33
Delivery System Design

• For novel programs this will involve assessment
  of
• target problem and population service
• implementation qualifications and competencies of
  staff
• mechanisms of recruiting staff
• means of optimizing access to the intervention
• location and characteristics of delivery sites
• referral and follow-up procedures
• Therefore, the evaluator will need to consider
• the characteristics of the organization,
• methods of obtaining cooperation from relevant
  groups,
• relationships with other existing programs,
• the political and social context of the program,
• and the availability of required resources.

34
Delivery System Design (Cont.)

• Specifically, the program should consider
• Overcoverage (comparing individuals who
  participate in the program with those who drop
  out),
• Undercoverage (comparing individuals who
  participate in the program with those who are
  eligible but do not participate),
• Biased coverage (comparing individuals who
  participate in the program with those who drop
  out and those eligible non- participants)

35
Outcome Evaluation

• Outcome evaluation is inappropriate without
  evaluation of targeting and implementation of a
  program
• The strengths of novel evaluations are that they
  are often involve only partial coverage and
  therefore it is possible to identify control
  groups
• In more established programs, the coverage is
  total and the most appropriate controls are the
  subjects prior to the evaluation.

36
Outcome Evaluation

• When randomization is not possible the usual
  options in forming control groups are
• matched controls reflexive controls (subjects are
  their own baseline)
• generic controls (from established norms)
• shadow controls (constructed from the opinions of
  knowledgeable stakeholders and experts about what
  would be expected without an intervention).
• Use gt one control, since converging data will
  increase confidence

37
Outcome Evaluation

• Direct Effectiveness Measures
• Indirect Effectiveness Measures Client
  Satisfaction
• Indirect Effectiveness Measures Social Validity
• Assessing costs

38
Design Selection

• Good enough" rule while randomization provides
  the best control, the evaluator needs to take
  into account practicality, feasibility, the
  resources available, and the expertise of the
  evaluator.
• In the context of the many constraints imposed in
  "real world' evaluation, it is most appropriate
  to choose the best achievable design

39
4.3 Evaluation of Recent Programs

• The act of fine tuning focuses attention on
  program modifications that increase the magnitude
  of impact or decrease the cost per unit of impact
  while maintaining the efficacy of service
  delivery
• Rossi and Freeman suggest that program evaluation
  involves
• re appraising objectives
• identifying program modifications
• undertaking reputability assessments
• program re-planning and redesign
• planning and implementing evaluation designs to
  monitor program changes and their impacts

40
4.4 Evaluation of Established Programs

• Program conceptualization shifts in focus to
  assessing what already exists. Therefore,
  conceptualization may involve
• preparing a program description
• interviewing the relevant personnel
• scouting the program
• developing an evaluable program model
• identifying evaluation users
• achieving agreement to proceed

41
4.4 Evaluation of Established Programs (Cont.)

• Monitoring will involve an assessment of what
  inputs, activities, and processes occur in a new
  version of the program to improve the fidelity to
  the program's intent, and to provide a clear base
  for relating causal processes to outcomes
• An established program will be monitored to
  ensure accountability. Use
• Management information system, which constantly
  monitors the program, but this is expensive in
  the long term
• Cross- sectional survey, which although it is
  cheaper in the long-term, has expensive start-up
  costs and may be resisted by staff
• The main focus in an established program is to
  ensure that there are minimal discrepancies
  between what how the program is formally seen to
  be operating and how it is operating in reality
• Independence of evaluators
• Should a program undertake its own evaluation, or
  should external evaluators be used?

42
4.4 Evaluation of Established Programs (Cont.)

• Cost-benefit analysis-gt computes all program
  costs, and then compares them to program outcomes
  expressed in a common unit (e.g., dollars). The
  main problems facing such analyses is that all
  program costs and benefits need to be identified
  and measured, and they need to be expressible in
  a common metric.
• Necessary that the program meets five conditions
• 1. The program must have independent and
  separable funding.
• 2. The program must be certain that its net
  effects are significant.
• 3. The program's impact and magnitude must be
  known of calculable.
• 4. The benefits can be expressed in monetary
  terms.
• 5. Decision makers are considering alternative
  programs.
• Usually, the fourth condition is not met, and
  Rossi and Freeman suggest that under these
  circumstances a cost-effectiveness analysis be
  conducted
• Cost-effectiveness analysis -gt only requires the
  identification in a common unit the program
  costs. The benefits remain expressed in outcome
  units

43
5. Using Evaluation Findings

• At the end of the day, what counts is not the
  program evaluation, but the extent to which the
  program evaluation modifies policies, programs,
  and practices
• Consider instrumental use - that is, ensuring the
  use of the findings by decision- makers and other
  stakeholders
• understand the work of the decision-makers
  (therefore, present the results briefly and
  simply)
• ensure that the evaluation results are timely and
  are available when they are needed respect
• be sensitive to stakeholders' program commitments
  plans for utilization and dissemination of
  results should be part of the evaluation design
• Evaluation programs should assess the extent to
  which the results of an evaluation are utilized

44
6. Checklist for Evaluation

• Pounds The cost of the evaluation has been
  anticipated.
• Politics The relevant people have been
  considered
• Procedures The procedures are practical given
  the purpose of the evaluation

45
References

• Berk, R. A., Rossi, P. H. (1990). Thinking
  about program evaluation. Newbury Park Sage.
• Binkerhoff, R. O., Brethower, D. M., Hluchyj, T.,
  Nowakowski, J. R. (1983). Program evaluation A
  practitioner's guide for trainers and educators
  Design manual. Hingham, Ma. Kluwer-Nijhoff.
• Binkerhoff, R. O., Brethower, D. M., Hluchyj, T.,
  Nowakowski, J. R. (1983). Program evaluation A
  practitioner's guidefor trainers and educators
  Sourcebook and casebook. Hingham, Ma.
  Kluwer-Nijhoff.
• Cronbach, L. J. (1982). Designing evaluations of
  educational and social programs. San Francisco.
  Jossey-Boss.
• Eisner, E. W. (1985). The art of educational
  evaluation. London Falmer.
• Fitz-Gibbon, C. T., Morris, L. L. (1987). How
  to design a program evaluation. Newbury Park,
  Ca. Sage.
• Hayes, S. C., Barlow, D. H., Nelson-Gray, R. O.
  (1999). The scientist-practitioner Research and
  accountability in the age of managed care (2nd
  ed.). Boston Allyn Bacon. (Especially chapters
  2 10).

46
References (Cont.)

• Krosnick, J. A. (1999). Survey research. Annual
  Review of Psychology, 50, 537-567.
• Maruish, M. E. (1999). The use of psychological
  testing for treatment planning and outcomes
  assessment (second edition). Mahwah, NJ.
  Lawrence Erlbaum.
• Robson, C. (1993). Real world research. Oxford
  Blackwell. (Esp. Chapter 7)
• Rossi, P. H., Freeman, H. E. (1989). Evaluation
  A systematic approach (4th ed.). Newbury Park,
  Calif. Sage Publications.
• Shaddish, W. R., Cook, T. D., Leviton, L. C.
  (1993). Foundations of program evaluation
  Theories of practice. Newbury Park, Ca. Sage.
• Stufflebeam, D. L., McCormick, C. H., Binkerhoff,
  R. O., Nelson, C. 0. (1985). Conducting
  educational needs assessment. Boson
  Kluwer-Niihoff.