The Ethical and Legal Basis for Evidence-based Education: Implications for the Profession

1
The Ethical and Legal Basis for Evidence-based
Education Implications for the Profession

• Ronnie Detrich
• Wing Institute
• TED Conference
• November, 2008
• Dallas, Texas

2
Overview

• Discuss the recent legal and regulatory
  requirements to base educational interventions on
  scientific research.
• Describe ethical requirements to rely on
  scientific knowledge.
• Describe what it means to be evidence-based
• Definitions
• Issues
• Controversies
• Appraisal of current status and future
  directions.

3
The Legal Basis

• No Child Left Behind (NCLB) Interventions used
  to improve educational performance are based on
  scientific research.
• Over 100 references to scientific research in
  NCLB.
• Individuals with Disabilities Education
  Improvement Act IDEIA (2004) Interventions are
  scientifically based instructional practices.

4
The Legal Basis

• Specific requirements of IDEIA include
• Pre-service and professional development to
  improve the academic achievement and functional
  performance of children with disabilities,
  including the use of scientifically based
  instructional practices, to the maximum extent
  possible.

5
The Legal Basis

• Scientifically based early reading programs,
  positive behavioral interventions and supports,
  and early intervention services to reduce the
  need to label children as disabled in order to
  address the learning and behavioral needs of such
  children.

6
The Legal Basis

• The Individualized Education Program (IEP)
  services based on peer-reviewed research to the
  extent practicable.

7
The Legal Basis

• In determining if a child has a specific learning
  disability, a local education agency may use a
  process that determines if a child responds to a
  scientific, research-based intervention as part
  of the evaluation procedures.

8
The Ethical Basis

• Most national psychological and educational
  organizations have ethical standards requiring
  science-based practices to address problems.
• American Psychological Association Ethical
  Standard 2.04
• Psychologists work is based on the established
  scientific and professional knowledge of the
  discipline.

9
The Ethical Basis

• National Association of School Psychologists
• Standard III F 4.
• School psychology faculty members and clinical or
  field supervisors uphold recognized standards of
  the profession by providing training related to
  high quality, responsible, and research-based
  school psychology services.

10
The Ethical Basis

• National Association of School Psychologists
• Standard IV 4.
• School psychologists use assessment techniques,
  counseling and therapy procedures, consultation
  techniques, and other direct and indirect service
  methods that the profession considers to be
  responsible, research-based practice.

11
The Ethical Basis

• Behavior Analyst Certification Board
• Standard 2.09a
• The behavior analyst always has the
  responsibility to recommend scientifically
  supported, most effective treatment procedures.
  Effective treatment procedures have been
  validated as having both long-term and short-term
  benefits to clients and society.
• Standard 2.09b
• Clients have a right to effective treatment
  (i.e., based on the research literature and
  adapted to the individual client).

12
Another Ethical Responsibility

• Education services are largely funded through
  public dollars (taxpayers dollars).
• There is an implicit assumption that the money
  will be spent for the public good.
• A fiduciary responsibility exists when one person
  or organization is charged with managing another
  persons money.

13
Another Ethical Responsibility
Fiduciary
Assure that the taxpayers are receiving the
greatest possible return on their investment.
Act solely for the benefit of the other party
Carries the weight of ethical conduct
14
How Do We Meet Our Fiduciary Responsibility?

• Interventions that have an evidence base are more
  likely to produce positive effects for students.
• Does not assure positive outcomes but increases
  the probability.
• The impact of a non-evidence based intervention
  is unknown.

15
How Do We Meet our Fiduciary Responsibility?

• Implications
• Using a non-evidence-based intervention when
  there are evidence-based interventions available
  constitutes unethical practice.
• Use of a non-evidence based intervention should
  be considered research.
• All of the safe-guards afforded research
  participants and their families should be in
  place.
• Conducting research with tax-dollars provided for
  education services may constitute a violation of
  our fiduciary responsibility.

16
Becoming Evidence-based

• Clearly, the intent of Congress, the U.S.
  Department of Education, and Office of Special
  Education Programs is to rely on interventions
  that have a scientific basis.
• Professional organizations place great value on
  scientific knowledge.
• What does it mean to be evidence-based?

17
What is Evidence-based Practice?

• At its core the EBP movement is a consumer
  protection movement.
• It is not about science per se.
• It is a policy to use science for the benefit of
  consumers.
• The ultimate goal of the evidence-based
  movement is to make better use of research
  findings in typical service settings, to benefit
  consumers and society. (Fixsen, 2008)

18
What is Evidence-based Practice?

• Evidence-based practice has its roots in
  medicine.
• Movement has spread across major disciplines in
  human services
• Psychology
• School Psychology
• Social Work
• Speech Pathology
• Occupational Therapy

19
What Is Evidence-based Practice?
Professional Judgment
Best available evidence
Client Values
Sackett et al (2000)
Client Values
Professional Judgment
Best Available Evidence

• EBP is a decision-making approach that places
  emphasis on evidence to
• guide decisions about which interventions to use
• evaluate the effects of an intervention.

20
Phases of Evidence-based Intervention
Identify
Identify
Evidence-based
Intervention
Implement
Evaluate
Implement
Evaluate
21
What is Evidence-based Education?
Identify

• The term evidence-based has become ubiquitous
  in last decade.
• There is no consensus about what it means.
• At issue is what counts as evidence.
• Federal definition emphasizes experimental
  methods.
• Preference for randomized trials.
• Definition has been criticized as being
  positivistic.

22
What Counts as Evidence?
Identify

• Ultimately, this depends on the question being
  asked.
• Qualitative methods are best for some questions.
• In EBP the goal is to identify causal relations
  between interventions and outcomes.
• Experimental methods do this best.

23
What Counts as Evidence?
Identify

• Even if we accept causal demonstrations to be
  evidence, we have no consensus.
• Randomized Clinical Trials (RCT) have become the
  gold standard.
• There is controversy about the status of single
  subject designs.
• Most frequently criticized on the basis of
  external validity.

24
How Are Evidence-based Interventions Identified?
Identify

• Identification is more than finding a study to
  support an intervention.
• Identification involves distilling a body of
  knowledge to determine the strength of evidence.

25
How Are Evidence-based Interventions Identified?
Identify

• Distillation requires standards of evidence for
  reviewing the literature.
• Standards specify
• the quantity of evidence
• the quality of evidence

26
Continua of Evidence
Identify
Quality of the Evidence
Quantity of the Evidence
Janet Twyman, 2007
27
How Are Evidence-based Interventions Identified?
Identify

• Two approaches to validating interventions
• Threshold approach
• Evidence must be of a specific quantity and
  quality before an intervention is considered
  evidence-based.
• Hierarchy of evidence approach
• Strength of evidence falls along a continuum with
  each level having differential standards.

28
How Are Evidence-based Interventions Identified?
Identify

• There are no agreed upon standards.
• It is possible for an intervention to be
  evidence-based using one set of standards and to
  fail to meet evidence standards using an
  alternative set.
• Difficult for consumers and decision makers to
  sort out the competing claims about what is
  evidence-based.

29
Identify
30
Identify
31
Identify
Actual Effectiveness
Assessed Effectiveness
Effective
Ineffective
Effective
Ineffective
Effective


Effective
Most likely with hierarchy approach
True
False
Positive
Positive
Most likely with threshold approach
True
False
Negative
Negative
Ineffective
Ineffective
32
Choosing Between False Positives and False
Negatives
Identify

• At this stage, it is better to have more false
  positives than false negatives.

• False Negatives
• Effective interventions will not be selected for
  implementation.
• As a consequence, less likely to determine that
  they are actually effective.

False Positives Progress monitoring will
identify interventions that are not effective.
33
Why Do We Need Evidence-based Education?
550 named interventions for children and
adolescents
Kazdin (2000)
Empirically evaluated
Cognitive-behavioral
Behavioral
Evidence-based interventions are less likely to
be used than interventions for which there is no
evidence or there is evidence about lack of
impact.
34
Are We Training Educators to be Evidence-based?
Survey of School Psychology Directors of Training
(Shernoff, Kratochwill, Stoiber, 2003)
Evidence-based interventions
29
Knowledge
41
training
programs
35
(No Transcript)
36
Implementing Evidence-based InterventionsWhere
Good Interventions Go to Die
Implement

• Intent of both legal and ethical guidelines is to
  have positive impact.
• Evidence-based interventions are assumed to give
  to give us that chance.
• Identification is necessary but not sufficient to
  assure that intervention will be effective.
• Must address complex issues associated with
  implementation.

37
Implementing Evidence-based Interventions
Implement

• Implementation is where the research to practice
  gap is most evident.
• Many innovations in education have very short
  life spans.
• 18 mos. (Latham)
• Most often a result of poor implementation.

38
Implementation is Fundamental
80 of initiatives ended within 2 years
90 of initiatives ended within 4 years
Data from Center for Comprehensive School Reform
39
Well Tested Programs Often Fail Large Scale
Implementation
Implement

• Elliott Mihalic (2004) review Blueprint Model
  Programs (violence prevention and drug prevention
  programs) replication in community settings.
• Programs reviewed across 5 dimensions
• Site selection
• Training
• Technical assistance
• Fidelity
• Sustainability

40
Keys to Implementation
Implement

• Critical elements in site readiness
• Well connected local champion
• Strong administrative support
• Formal organizational commitments
• Formal organizational staffing stability
• Up front commitment of necessary resources
• Program credibility within the community
• Program sustained by the existing operational
  budget

41
Keys to Implementation
Implement

• Critical elements of training
• Adhere to requirements for training, skills, and
  education.
• Hire all staff before scheduling training.
• Encourage administrators to attend training.
• Plan and budget for staff turnover.
• Implement program immediately after training.

42
Keys to Implementation
Implement

• Critical elements of Technical Assistance
• Proactive plan for technical assistance.
• Critical elements of Fidelity
• Monitor fidelity.
• Critical elements of Sustainability
• Function of how well other dimensions are
  implemented.

43
Implementing Evidence-based InterventionsDimensio
ns of Implementation
Implement

• Contextual fit
• Complexity of intervention

44
Implementing Evidence-based InterventionsContextu
al Fit
Implement

• Contextual Fit the degree to which an
  intervention matches the culture, training, and
  resources of a particular setting.
• These characteristics of a setting can be
  measured.
• Degree of contextual fit may moderate the impact
  of an intervention.

45
Implementing Evidence-based InterventionsContextu
al Fit
Implement

• Adoption or Accommodation
• Adoption Implementing intervention as evaluated
  to be effective.
• Assures intervention is evidence-based.
• Does not assure implementation.
• Accommodation adjusting intervention to meet
  local circumstances.
• May result in intervention no longer being
  evidence-based.
• May increase implementation with integrity.

46
Implementing Evidence-based InterventionsContextu
al Fit
Implement

• Logically it would seem to make sense to always
  implement intervention that produces greatest
  impact.
• There may be exceptions
• If high impact intervention requires great
  resources, specialized training, and is very
  different from current practices it may not be
  implemented with integrity.
• May be better to implement effective but lower
  impact intervention that is better contextual fit
  and will be implemented with greater integrity.

47
Implementing Evidence-based InterventionsComplexi
ty
Implement

• Level of precision may increase complexity.
• Be as precise as necessary but no more.

Individualized intervention plan
Good Behavior Game
Complexity
Catchem being good
Precision
48
Evaluating Evidence-based InterventionsProgress
Monitoring
Evaluate

• Implementation of evidence-based intervention
  does not assure success.
• Necessary to evaluate impact in local context.
• No intervention will be effective for all
  students.
• Cannot predict who will benefit.
• Progress monitoring is practice-based evidence
  about evidence-based practices.
• Consistent with legal requirements and ethical
  standards.

49
Ethical Standards and Progress Monitoring
Evaluate

• National Association of School Psychologists
• Standard IV C 1b.
• Decision-making related to assessment and
  subsequent interventions is primarily data-based.
• Standard IV 6.
• School psychologists develop interventions that
  are appropriate to the presenting problems and
  are consistent with the data collected. They
  modify or terminate the treatment plan when the
  data indicate the plan is not achieving the
  desired goals.

50
Ethical Standards and Progress Monitoring
Evaluate

• Behavior Analyst Certification Board
• Standard 4.04
• The behavior analyst collects data or asks the
  client, client-surrogate, or designated other to
  collect data needed to assess progress within the
  program.
• Standard 4.05
• The behavior analyst modifies the program on the
  basis of data.

51
Legal Requirements for Progress Monitoring
Evaluate

• Fundamental to IEP process.
• Response to Intervention is accepted as
  alternative means for determining eligibility for
  Learning Disability classification.
• Progress monitoring is the heart of RTI.
• All students routinely and systematically
  monitored to assure adequate progress is
  occurring.

52
Evaluating Evidence-based Interventions
Evaluate

• Progress monitoring is a systems level
  intervention.
• Systems must be in place to assure
• Data are collected
• Data are reviewed
• Decisions are based on the data
• If systems are not in place, response effort
  associated with data collection will compromise
  data-based decision making.

53
Progress Monitoring as an Intervention
Evaluate

• Progress monitoring 2-5/week in math and reading
• 4 times as effective as 10 increase in per pupil
  spending
• 6 times as effective as voucher programs
• 64 times as effective as charter schools
• 6 times as effective as increased accountability.
• (Yeh, 2007)

54
Evidence-based Education and Treatment Integrity

• Progress monitoring allows data based decision
  making about effects of an intervention.
• It is impossible to make informed decisions
  without knowing how well the intervention was
  implemented.

55
Outcome
Negative
Positive
Negative
Positive


High
Continue Intervention
Change Intervention
High
Integrity
Low
Low
Unknown reason
Unknown reason

• Intervention problem?
• Implementation problem?

• Other life changes?
• Unknown intervention?
• Intervention is effective?

56
Where Are We?
From a university in the U.S.
57
Where are We?

• Being evidence-based is more than a good idea, it
  is the law and it is ethical conduct however, it
  is not as easy as it might seem.
• Lack of consensus about evidence may do harm to
  consumers.
• The research to practice gap limits the impact of
  evidence-based education.
• The science of implementation and sustainability
  is in its infancy.

58
Where are We?

• Pre-service training should change to reflect
  current policy.
• Changes in both method of training and content.

59
Effects of Training
   
Joyce and Showers, 2002
60
Basis for Choosing Treatment Szatmari (2004)
Treatment
Evidence
Values
61
Do Nothing
Unethical
None
Clinical Paralysis
62
Do Nothing
Unethical
None
Clinical Paralysis
63
Toss a Coin
Unethical in light of evidence
None
64
Do Nothing
Toss a Coin
Unethical
Unethical in light of evidence
None
Clinical Paralysis
None
65
Training
None
Outdated
Perhaps some
Current
66
None
67
Etiology
Limited
Difficult
68
None
69
ABA
Not very humane
Robust
Effective
70
None
ABA
ABA
Not very humane
Not very humane
Robust
71
Developmental sociocognitive
None Yet
Highly preferred
72
To be Ethical
Inform Parents of Options
None
Developmental sociocognitive
None yet
Highly preferred
73
Thank you

• Copy of this presentation may be downloaded at
• www.winginstitute.org