Adversary-Oriented Approaches to Evaluation - PowerPoint PPT Presentation


PPT – Adversary-Oriented Approaches to Evaluation PowerPoint presentation | free to download - id: 62b8a0-ZDA1O


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Adversary-Oriented Approaches to Evaluation


Adversary-Oriented Approaches to Evaluation EDF 5461: Introduction to Program Evaluation Week 7 Anthony Artino William Lightfoot Robin Smith Lesson Overview ... – PowerPoint PPT presentation

Number of Views:840
Avg rating:3.0/5.0
Slides: 23
Provided by: spUconnE
Learn more at:


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Adversary-Oriented Approaches to Evaluation

Adversary-Oriented Approaches to Evaluation
  • EDF 5461
  • Introduction to Program Evaluation
  • Week 7
  • Anthony Artino
  • William Lightfoot
  • Robin Smith

Lesson Overview
  • Introduction
  • History and Development
  • Model
  • Strengths and Limits
  • Summary

  • An approach that encourages and controls bias
    instead of trying to eliminate it.
  • Treats impartiality as naïve.
  • Opposition and conflict are planned.
  • The truth emerges for a hard but fair fight.
  • Can be used for contentious issues where unbiased
    parties dont exist.

History and Development
  • The first documented use of a judge and jury
    format to examine government programs was that of
    J. M. Rice in 1915.
  • His intent was to reduce corruption and increase
    government efficiency by exposing waste and

  • No further developments appeared until 1965 when
    E. G. Guba proposed evaluation trials, making
    use of hearings, human testimony, and other
    components of legal proceedings.
  • He contended that both sides of issues were
    fairly examined using the concept of advocacy.

  • In 1970, T. R. Owens set out to test Gubas
    assertions using a modified judicial model on a
    hypothetical school curriculum.
  • His results caught the attention of the
    evaluation community and resulted in dozens of
    studies over the next few decades.

  • Three general approaches emerged
  • 1) Two-view judicial models adapted from the
    legal paradigm.
  • 2) More that two views considered in quasi-legal
    adversary hearings.
  • 3) Debate or forensic structured evaluations.

  • Characteristics of an Adversary Approach
  • 1) Flexible rules.
  • 2) Wider definition of relevant information.
  • 3) Pretrial discovery of opposing case.
  • 4) Pretrial stipulations and challenges.
  • 5) Freer rules for testimony relevancy.
  • 6) Can use pretrial testimony of experts.
  • 7) Less focus on technical maneuvering.
  • 8) Other interested parties may participate.
  • (Owens)

  • Judicial Evaluation Model (Wolf)
  • 1) Issues generation.
  • 2) Issue selection.
  • 3) Preparation of arguments.
  • 4) The hearing.

  • General Aspects of Judicial Models
  • 1) Existing controversy.
  • 2) Formal case presentations.
  • 3) Fact heard by an impartial arbiter.
  • 4) Decisions based on arguments presented.
  • (Levine and Rosenberg)

  • Adversary Hearings Model
  • Multiple viewpoints need to be examined.
  • Characteristics
  • Multiple stakeholders (multi-adversarial).
  • Public testimony, question and cross examine.
  • Face to face interactions.
  • Real time evaluation as the process unfolds.
  • Examples
  • Congressional hearings.
  • Blue ribbon panels.
  • Presidential commissions.

  • Debate and Forensic Model
  • Usually a team approach.
  • Extensive preparation by each team.
  • Presentation and critiquing.
  • Rebuttal and discussion.
  • Reconciliation where possible.
  • Allows for alternative values and conceptions.

  • Building opposing views illuminates both positive
    and negative aspects of a program better than
    most other evaluation approaches.
  • The range of information collected is very broad.
  • The audiences informational needs are met
    through an interesting and informative manner.

Strengths continued
  • Its sufficiently broad and pluralistic, and it
    can be combined with other approaches.
  • Example - An expertise-oriented approach could be
    used by both teams.
  • It avoids excluding stakeholders and viewpoints
    that might otherwise be left out of most other
  • Its hard to argue that an adversary-oriented
    evaluation is unfair when both sides of an issue
    are presented and examined.

Strengths continued
  • Because both sides of an issue are presented, it
    diverts much subsequent criticism of the
    evaluation itself.
  • Most adversary-oriented evaluations are
    rigorously planned and implemented.
  • Nobody likes to look bad by being beaten
  • It has a built-in meta-evaluation, with the
    opposing side painstakingly reviewing all data
    collection, analysis, and interpretation of the
    opposite team.

Strengths continued
  • The use of direct, holistic human testimony as
    well as cross-examination of that testimony is
    normally seen as a strength of adversary-oriented
  • When any data, interpretation, or conclusion is
    agreed upon by two adversaries, it lends great
    credence to that aspect of the evaluation.

  • Adversary-oriented approaches are not yet
    sufficiently well developed to serve as a
    standard or model for future efforts.
  • Preoccupation with the respective paraphernalia
    of the various approaches could cause evaluators
    to overlook the benefits that might accrue from
    their use.
  • Rigid adherence to a legal model and courtroom
    rituals may be unnecessary and, in some cases,

Limitations continued
  • Use of the legal paradigm can also result in a
    seductive slide into an indictment mentality
    which can lead to a preoccupation with
    determining guilt or innocence.
  • Use of the legal model can result in the idea
    that an evaluation is only used when there is a
    problem to be solved.
  • This negative perception is something that is
    already all too common with any evaluation.

Limitations continued
  • Presentation of strong pro or con positions might
    increase the probability of an extreme decision
    on the part of the stakeholders.
  • Although stakeholders may gain a broader spectrum
    of data, they may compromise impartiality, a
    quality essential to rational decision making.
  • Most adversary approaches have a competitive
    element when competition is high, cooperation
    tends to be lower.
  • Shared conclusions are not easily come by.

Limitations continued
  • A skilled orator without solid supportive data
    could sway stakeholders by eloquence alone.
  • A skilled debater can often build a remarkably
    strong case on a very flimsy foundation and
    confusing logic.
  • One side of the argument could feel compelled to
    keep up with the oppositions arguments, even if
    it meant straining or ignoring the data.

Limitations continued
  • Adversary-oriented evaluations are very
    time-consuming and expensive.
  • They require extensive preparation and
    considerable investment of human and fiscal
  • The lack of a higher court of appeals in
    evaluation precludes rectifying improper
  • Some argue that although the adversary model may
    resolve conflicts, it has limited potential for
    getting at the truth of a matter.

Lesson Summary
  • The Adversarial Approach is useful when
  • The evaluation object affects many people
  • Controversy has created wide interest
  • Decisions are summative
  • Evaluators are external
  • The issues are clear
  • Administrators understand the intensity of the
    approach and
  • Resources are available for additional expenses.