Scientific Based Research: Doing What Works

1
Scientific Based ResearchDoing What Works

• Wendy Russell and Emily Crandall
• Region III Comprehensive Center
• May 8, 2003

2
Todays Outcomes

• To raise awareness of the importance and
  relevance of research for student achievement
• To develop skills in using research to identify
  what works.
• Define SBR
• Assess research quality
• Use research in your decision making

3
NCLB Shifts Emphasis

• The push towards scientifically-based research is
  the federal governments most visible effort to
  shift education practice in a different direction.

4
How do you make decisions about programs and
practices?
External evidence
Professional Wisdom
5
Where Are You On the Research Road?
6
Why SBR?

• Schools have largely based their practice on
  tradition, superstition, and anecdotes.
• Too much money has been invested in education and
  wasted in programs that do not get the intended
  results.
• Valerie Reyna, Deputy Director
• Office of Educational Research and Improvement

7
Cargo Cult Science

• We really ought to look into the theories that
  dont work, and science that isnt science.
  Richard P. Feynman, 1985

8
Primary Goal of SBR

• Ensure that programs have been proven effective
  and are more likely to benefit others when used.
• For students, afterschool, professional
  development, school improvement, etc

9
NCLB Legislation Goals

• One hundred percent proficiency for ALL students
  in 12 years.
• The method for achieving this goal is reasoned
  decision-making and transforming education into
  an evidence-based profession that utilizes SBR.

10
NCLB Defines Scientific Inquiry

• Use scientific method with an emphasis on
  experimental control (or comparison) groups
• Replication of results, using multiple studies by
  different investigators
• Ability to generalize results from one sample to
  others in the general population
• Fulfillment of rigorous standards with an
  emphasis on peer review
• Convergence (or consistency) of results between
  studies.

What makes research scientific is not the motive
for carrying it out, but the manner in which it
is carried out.
11
Research 101

• Two types of research
• Collecting numbers quantitative
• Collecting observations qualitative
• Three purposes for research
• To assess the implementation and replicability of
  the reform practice or program.
• To test a theory behind a practice or program.
• To measure impact, effectiveness of the practice
  or program.
• The abstract will tell you the type and purpose
  of the study.

12
Different Research Designs for Different Purposes

• Implementation and replicability
• Type Quantitative or Qualitative
• Theoretical Base
• Type -- Quantitative or Qualitative
• Evidence of Effectiveness
• Type -- Quantitative

13
Implementation and Replicability

• Asks
• How was practice or program implemented?
• In what setting?
• Under what conditions?

14
Theoretical Base

• Asks
• What is the theory behind this practice or
  program?
• What are the theorys guiding principles?
• What does the theory explain?

15
Evidence of Effectiveness

• Is there evidence showing that this practice or
  program improves student achievement?
• Core of SBR

16
Hierarchy of Evidence
Randomized Trial
Quasi- experimental
Correlational study with statistical controls
Correlational study without statistical controls
Case Studies
Valerie Reyna, OERI
17
Systematic Rigorous Methodology

• Gold Standard
• Random Assignment (Experimental)
• Silver Standard
• Control Group-Non Random (Quasi-experimental)
• Bronze Standard
• Case Study
• Lead Standard
• Testimonial, opinion, Intuition, Small sample,
  Selective criteria

18
Scientifically Based ResearchGold Standard

• Experimental study
• Research Type Quantitative
• Determines potential between practice or program
  and student achievement
• Controls all of the following
• Environment
• Intervention (practice or program)
• Subject selection (random sampling)
• Shows change in outcome as a result of treatment

19
Promising ResearchSilver Standard

• Quasi-experimental study
• When it is not possible to control for at least
  one of the three elements of experimental design
  
• Environment
• Intervention
• Assignment to experimental or control groups
• Cannot determine causality

20
Bronze StandardSupplemental Research

• Reports the way things are
• Includes both quantitative and qualitative data
• Descriptive data summarized and simplified.
• Effects on student achievement cannot be
  attributed directly to the intervention.

21
Whose Research Findings Should I Use?

• Findings/Conclusions Based on
• Multiple studies
• Independent studies
• Peer Reviewed
• Published
• Strong criteria for SBR

22
Available Evidence
Become a savvy consumer of research. Only 5 or
less of available research evidence is at Gold
SBR Standard
23
Quality and Quantity Challenge

• Where you go to find research.
• Debate, discussion and the review process are
  important.
• Need for accumulation of studies in order to
  establish scientific certainty.
• Find quality of works while using the research
  currently available.
• Context matters.

24
Components of all Research Studies

• Abstract
• Objective (problem/purpose
• Research procedure/methodology
• Findings
• Discussion Section
• Conclusion

25
Professional Wisdom

• school leaders will need to rely on the best
  available empirical evidence and some degree of
  professional judgment in creating their programs.

CSR Program Guidelines
26
Using Professional Judgment

• Being a critical consumer
• Examining research to determine whether it is
  relevant to your context.
• Synthesizing findings across studies
• Incorporating the evidence into the
  decision-making process.

27
Research-Based Decision Process

• Step One Gather relevant, quality research
• Identify research that is
• Of high quality
• Current (within 5 years
• Focused on your interest
• Researched in schools/programs like yours
• Make sure search is extensive to capture all
  evidence and analysis

28
Research-Based Decision Process

• Step Two Interpret Research for your Context
• Synthesize findings gathered
• Revisit needs
• Revisit and refine focus questions
• Narrow the range of possible solutions/interventio
  ns
• Make a judgment of the effectiveness of an
  intervention from the studies.

29
Research-Based Decision Process

• Step Three Make your case
• Make links between
• Identified needs
• Programs and practices that can address these
  needs directly
• Evidence from the highest quality research
  available to support the effectiveness of these
  interventions on student achievement.

30
More information

• National Clearinghouse for Comprehensive School
  Reform
• www.goodschools.gwu.edu
• What Works Clearinghouse
• w-w-c.org
• e-mail updates wwcinfo_at_w-w-c.org

31
Final Point

• Scientific research in education is about
  accumulating knowledge.
• Scientific understanding coalesces as it
  progresses, to make sense of systems, experiences
  and phenomena.
• Science advances understanding of various
  phenomena through sustained inquiry and debate.

NRC, Scientific Research in Education