Title: A stepbystep guide to assessing the evidence for a new reading treatment
1A step-by-step guide to assessing the evidence
for a new (reading) treatment
2What would you do?
- The Miracle Belt
- http//www.youtube.com/watch?vWwrAQlrTmKM
(0017) - Dore
- http//www.youtube.com/watch?vEMHliPbnmRYfeature
related - Fish oil
- http//www.youtube.com/watch?v1MwdPVIAk08
- Brain Gym
- http//www.youtube.com/watch?vM5rH7kDcFpc
(0058) - Fast Forword
- http//www.youtube.com/watch?vQsSUamFekwI
3Step 1 Review theory offered by the company
- Look up the companys www site and find the
theory behind the treatment - Does it make sense to you logically
- http//www.youtube.com/watch?vWwrAQlrTmKM
(0130)? - Yes or no?
- Why or why not?
- Does it stem from an expert (e.g., scientific
field or long-standing medical findings) or does
it stem from a baseball player a paint
manufacturer? - Yes or no?
- Whom?
4Step 2 Review research quoted by company
- Look up the company www site and search for any
group studies - Does it look like the treatment might work for
some children? - Write it down
- What type of children appear to benefit?
- Write it down
- Does this match the type of child who wants to do
the training? - Write it down
5Step 3 Anecdotes reported by company
- If yes, IGNORE
- Anecdotes do not represent average experience
- Self-selected sample
- Cognitive dissonance
- Subjective data no good (versus objective data)
in treatment studies - If no, IGNORE
- Absence of anecdotes is not a bad thing since
anecdotes are not valid data - Do not write any anecdotes down
6Step 4 Reading and spelling exercises
- To date, scientific evidence suggests that only
treatments that include reading or spelling
exercises help poor readers or poor spellers - Note This could change in the future
- What reading or spelling exercises does the
treatment include? - Write it down
- Do these exercises treat the type of problem that
your child has? - Write it down
- Does the treatment include no reading or spelling
exercises? - If so, write this down
7Step 5 Screening tests for treatment
- Some treatments include assessments prior to
training and only treat children with correct
profile - Does the treatment do this?
- Note if this is the case or not
- If so, what is the profile for the treatment?
- Write it down
- What is the profile of your child?
- Write it down
- Is there a good match?
- Write it down
8Step 6 Reviews of peer-reviewed studies
- Done by independent expert who understands the
scientific method - not a journalist without a scientific degree
- not a private company
- not a scientist with any personal relationship
with a treatment or the people who make the
treatment (e.g., Rod Nicolson Dore) - MUSEC briefings
- http//www.musec.mq.edu.au/co_brief.aspx
- Cochrane Institute
- http//www.cochrane.org/
- What Works Clearinghouse (view with some caution)
- http//ies.ed.gov/ncee/wwc/
- Bad Science (Ben Goldacre)
- http//www.badscience.net/
- Write down any verdicts and source of verdicts
9Step 7 Collect peer-reviewed studies
- Go to Google Scholar
- http//scholar.google.com.au/
- Find any studies in peer-reviewed journals that
test the treatment - http//www.springerlink.com/content/r0704438528134
91/ - Download pdfs (you may have to pay for some)
- http//www.springerlink.com/content/r0704438528134
91/ - Contact authors for free pdfs (find email on
abstract) - http//www.springerlink.com/content/r0704438528134
91/
10Step 8 Review scientific background
- Is there a good scientific basis for the theory
behind treatment? - Has the theory behind the treatment been tested
previously? - Were the results generally positive?
- Were the results generally negative?
- Were the results mixed/controversial?
- Has the treatment has been tested in other
studies? - If yes, what are the findings of previous studies?
11Step 9 Review treatment group
- All studies will include a group of children who
are given the treatment - What version of the treatment were they given?
- What type of children are included in the
treatment group? - How many children are in the treatment group?
- How much training did they do?
- What tests are they given before and after
treatment? - Do these tests include tests for reading or
spelling? - Do these children look anything like your child?
- Does the treated group show a statistically
significant improvement on any of the tests?
12Step 10 Review control data
- Any improvement in the treatment group could be
due to - A genuine effect of treatment
- Practice doing the test (from pre-treatment
session) - Getting older (and hence smarter)
- The belief they will get better (placebo effect)
- http//www.youtube.com/watch?vEMHliPbnmRYfeature
related (0146) - An effect of other simultaneous training (e.g.,
reading classes at school) - Regression to the mean (extreme scores get less
extreme)
13Step 10 Review control data
- How do you decide which is true?
- You look to see if the study includes one of
three types of control data - Untrained control group
- Tested on same tests before and after training
period but do no training - Any improvements due to getting older, practice
on test, simultaneous training at school,
regression to mean - Compare this degree of improvement to degree of
improvement after training - Any degree of improvement in treatment group
greater than in untrained control group is
probably due to treatment
14Step 10 Review control data
- Double-baseline in treatment group
- Prior to training, treated group does tests
before and after the same period of time as the
training (e.G., 4 weeks) but without doing any
training - Any improvements due to getting older, practice
on test, simultaneous training at school,
regression to mean - Compare this degree of improvement to degree of
improvement after training - Any degree of improvement in after treatment
period that is greater than after no-treatment
period is probably due to genuine treatment effect
15Step 10 Review control data
- A placebo trained group
- Tested on same tests before and after training
period but do training with something not
expected to help reading - Any improvements due to getting older, practice
on test, simultaneous training at school,
regression to mean, and placebo - Compare this degree of improvement to degree of
improvement after training - Any degree of improvement in treatment group
greater than in placebo group is probably due to
treatment
16Step 10 Review control data
- Does the study include an untrained control
group? - If so, did the treatment group improve
significantly more than the untrained controls on
any tests? - Does the study include a double-baseline in the
treatment group? - If so, did the treatment group improve
significantly more after the training phase than
after the double-baseline phase on any tests? - Does the study include a placebo-trained group?
- If so, did the treatment group improve
significantly more than the placebo controls on
any tests? - If the study does not include a control group
then conclude that you cannot tell if there was a
genuine treatment effect
17Step 11 Review your data
- Sit down and review all the information that you
have written down - Consider the weight of evidence for and against
- Consider the quality the sources of each piece of
evidence - company website (biased) versus review of
scientific studies (good if considered control
trials) versus scientific study (good if includes
proper control) - Consider the cost
- Cost of treatment versus how much money the
parents can afford - Sleep on it
- Decide
18More information
- http//www.dystalk.com/talks/60-evaluating-alterna
tive-solutions-for-dyslexia - Bad Science by Ben Goldacre (2008, Fourth Estate,
London).