A stepbystep guide to assessing the evidence for a new reading treatment - PowerPoint PPT Presentation

1 / 18
About This Presentation
Title:

A stepbystep guide to assessing the evidence for a new reading treatment

Description:

A step-by-step guide to assessing the evidence. for a new (reading) ... Subjective data no good (versus objective data) in treatment studies. If no, IGNORE ... – PowerPoint PPT presentation

Number of Views:28
Avg rating:3.0/5.0
Slides: 19
Provided by: McA130
Category:

less

Transcript and Presenter's Notes

Title: A stepbystep guide to assessing the evidence for a new reading treatment


1
A step-by-step guide to assessing the evidence
for a new (reading) treatment
2
What would you do?
  • The Miracle Belt
  • http//www.youtube.com/watch?vWwrAQlrTmKM
    (0017)
  • Dore
  • http//www.youtube.com/watch?vEMHliPbnmRYfeature
    related
  • Fish oil
  • http//www.youtube.com/watch?v1MwdPVIAk08
  • Brain Gym
  • http//www.youtube.com/watch?vM5rH7kDcFpc
    (0058)
  • Fast Forword
  • http//www.youtube.com/watch?vQsSUamFekwI

3
Step 1 Review theory offered by the company
  • Look up the companys www site and find the
    theory behind the treatment
  • Does it make sense to you logically
  • http//www.youtube.com/watch?vWwrAQlrTmKM
    (0130)?
  • Yes or no?
  • Why or why not?
  • Does it stem from an expert (e.g., scientific
    field or long-standing medical findings) or does
    it stem from a baseball player a paint
    manufacturer?
  • Yes or no?
  • Whom?

4
Step 2 Review research quoted by company
  • Look up the company www site and search for any
    group studies
  • Does it look like the treatment might work for
    some children?
  • Write it down
  • What type of children appear to benefit?
  • Write it down
  • Does this match the type of child who wants to do
    the training?
  • Write it down

5
Step 3 Anecdotes reported by company
  • If yes, IGNORE
  • Anecdotes do not represent average experience
  • Self-selected sample
  • Cognitive dissonance
  • Subjective data no good (versus objective data)
    in treatment studies
  • If no, IGNORE
  • Absence of anecdotes is not a bad thing since
    anecdotes are not valid data
  • Do not write any anecdotes down

6
Step 4 Reading and spelling exercises
  • To date, scientific evidence suggests that only
    treatments that include reading or spelling
    exercises help poor readers or poor spellers
  • Note This could change in the future
  • What reading or spelling exercises does the
    treatment include?
  • Write it down
  • Do these exercises treat the type of problem that
    your child has?
  • Write it down
  • Does the treatment include no reading or spelling
    exercises?
  • If so, write this down

7
Step 5 Screening tests for treatment
  • Some treatments include assessments prior to
    training and only treat children with correct
    profile
  • Does the treatment do this?
  • Note if this is the case or not
  • If so, what is the profile for the treatment?
  • Write it down
  • What is the profile of your child?
  • Write it down
  • Is there a good match?
  • Write it down

8
Step 6 Reviews of peer-reviewed studies
  • Done by independent expert who understands the
    scientific method
  • not a journalist without a scientific degree
  • not a private company
  • not a scientist with any personal relationship
    with a treatment or the people who make the
    treatment (e.g., Rod Nicolson Dore)
  • MUSEC briefings
  • http//www.musec.mq.edu.au/co_brief.aspx
  • Cochrane Institute
  • http//www.cochrane.org/
  • What Works Clearinghouse (view with some caution)
  • http//ies.ed.gov/ncee/wwc/
  • Bad Science (Ben Goldacre)
  • http//www.badscience.net/
  • Write down any verdicts and source of verdicts

9
Step 7 Collect peer-reviewed studies
  • Go to Google Scholar
  • http//scholar.google.com.au/
  • Find any studies in peer-reviewed journals that
    test the treatment
  • http//www.springerlink.com/content/r0704438528134
    91/
  • Download pdfs (you may have to pay for some)
  • http//www.springerlink.com/content/r0704438528134
    91/
  • Contact authors for free pdfs (find email on
    abstract)
  • http//www.springerlink.com/content/r0704438528134
    91/

10
Step 8 Review scientific background
  • Is there a good scientific basis for the theory
    behind treatment?
  • Has the theory behind the treatment been tested
    previously?
  • Were the results generally positive?
  • Were the results generally negative?
  • Were the results mixed/controversial?
  • Has the treatment has been tested in other
    studies?
  • If yes, what are the findings of previous studies?

11
Step 9 Review treatment group
  • All studies will include a group of children who
    are given the treatment
  • What version of the treatment were they given?
  • What type of children are included in the
    treatment group?
  • How many children are in the treatment group?
  • How much training did they do?
  • What tests are they given before and after
    treatment?
  • Do these tests include tests for reading or
    spelling?
  • Do these children look anything like your child?
  • Does the treated group show a statistically
    significant improvement on any of the tests?

12
Step 10 Review control data
  • Any improvement in the treatment group could be
    due to
  • A genuine effect of treatment
  • Practice doing the test (from pre-treatment
    session)
  • Getting older (and hence smarter)
  • The belief they will get better (placebo effect)
  • http//www.youtube.com/watch?vEMHliPbnmRYfeature
    related (0146)
  • An effect of other simultaneous training (e.g.,
    reading classes at school)
  • Regression to the mean (extreme scores get less
    extreme)

13
Step 10 Review control data
  • How do you decide which is true?
  • You look to see if the study includes one of
    three types of control data
  • Untrained control group
  • Tested on same tests before and after training
    period but do no training
  • Any improvements due to getting older, practice
    on test, simultaneous training at school,
    regression to mean
  • Compare this degree of improvement to degree of
    improvement after training
  • Any degree of improvement in treatment group
    greater than in untrained control group is
    probably due to treatment

14
Step 10 Review control data
  • Double-baseline in treatment group
  • Prior to training, treated group does tests
    before and after the same period of time as the
    training (e.G., 4 weeks) but without doing any
    training
  • Any improvements due to getting older, practice
    on test, simultaneous training at school,
    regression to mean
  • Compare this degree of improvement to degree of
    improvement after training
  • Any degree of improvement in after treatment
    period that is greater than after no-treatment
    period is probably due to genuine treatment effect

15
Step 10 Review control data
  • A placebo trained group
  • Tested on same tests before and after training
    period but do training with something not
    expected to help reading
  • Any improvements due to getting older, practice
    on test, simultaneous training at school,
    regression to mean, and placebo
  • Compare this degree of improvement to degree of
    improvement after training
  • Any degree of improvement in treatment group
    greater than in placebo group is probably due to
    treatment

16
Step 10 Review control data
  • Does the study include an untrained control
    group?
  • If so, did the treatment group improve
    significantly more than the untrained controls on
    any tests?
  • Does the study include a double-baseline in the
    treatment group?
  • If so, did the treatment group improve
    significantly more after the training phase than
    after the double-baseline phase on any tests?
  • Does the study include a placebo-trained group?
  • If so, did the treatment group improve
    significantly more than the placebo controls on
    any tests?
  • If the study does not include a control group
    then conclude that you cannot tell if there was a
    genuine treatment effect

17
Step 11 Review your data
  • Sit down and review all the information that you
    have written down
  • Consider the weight of evidence for and against
  • Consider the quality the sources of each piece of
    evidence
  • company website (biased) versus review of
    scientific studies (good if considered control
    trials) versus scientific study (good if includes
    proper control)
  • Consider the cost
  • Cost of treatment versus how much money the
    parents can afford
  • Sleep on it
  • Decide

18
More information
  • http//www.dystalk.com/talks/60-evaluating-alterna
    tive-solutions-for-dyslexia
  • Bad Science by Ben Goldacre (2008, Fourth Estate,
    London).
Write a Comment
User Comments (0)
About PowerShow.com