Using metaanalyses in your literature review - PowerPoint PPT Presentation

1 / 56
About This Presentation
Title:

Using metaanalyses in your literature review

Description:

... aggregated the findings of 375 psychotherapy outcome studies ... (and colleague Smith) concluded that psychotherapy did indeed work - 'the typical ... – PowerPoint PPT presentation

Number of Views:61
Avg rating:3.0/5.0
Slides: 57
Provided by: SteveH148
Category:

less

Transcript and Presenter's Notes

Title: Using metaanalyses in your literature review


1
Using meta-analyses in your literature review
  • BERA Doctoral Workshop
  • 3rd September 2008
  • Professor Steven Higgins
  • Durham University
  • s.e.higgins_at_durham.ac.uk

2
Aims
  • To support understanding of meta-analysis of
    intervention research findings in education
  • To extend understanding of reviewing quantitative
    research literature
  • To describe the techniques and principles of
    meta-analysis involved to support understanding
    of its benefits and limitations
  • To provide references and examples to support
    further work.

3
ESRC Researcher Development Initiative
  • Quantitative synthesis of intervention research
    findings in education
  • Collaboration between
  • Durham University
  • York University
  • Institute of Education, London

4
Why review?
  • Ask the person next to you what the purpose of
    the literature review is in their thesis
  • See how many different purposes you can think of
  • Join another pair and identify which are the 3
    you think are the most important

5
Why review?
  • Summarise existing knowledge
  • What we know, and how we know it
  • For what purpose?
  • Expectation
  • Scenery
  • State of the art (summary)
  • Positioning (conceptual)
  • Progressing knowledge (logic)

6
The PhD literature review
  • Narrative summary of the area
  • Grand tour of the concepts and terminology
  • Synthesis of empirical findings
  • Background to the study

7
A systematic review
  • is usually more comprehensive
  • is normally less biased, being the work of more
    than one reviewer
  • is transparent and replicable
  • (Andrews, 2005)

8
Examples of systematic reviews
  • EPPI Centre
  • UK based - wide range of educational topics
  • The Campbell Collaboration
  • 5 education reviews
  • Best Evidence Encyclopedia
  • Johns Hopkins - aimed at practice

9
Systematic reviewing
  • Key question
  • Search protocol
  • Inclusion/exclusion criteria
  • Coding and Mapping
  • In-depth review (sub-question)
  • Techniques for systematic synthesis

10
Systematic reviews
  • Research and policy
  • Specific reviews to answer particular questions
  • What works? - impact and effectiveness research
    with a tendency to focus on quantitative and
    experimental designs

11
Literature reviewing - conceptual relations
Narrative review
Systematic review
Meta-analysis
12
Meta-analysis
  • Synthesis of quantitative data
  • Cumulative
  • Comparative
  • Correlational
  • Surveys educational research (Lipsey and
    Wilson, 2001)

13
Origins
  • 1952 Hans J. Eysenck concluded that there were
    no favorable effects of psychotherapy, starting a
    raging debate which 25 years of evaluation
    research and hundreds of studies failed to
    resolve
  • 1978 To proved Eysenck wrong, Gene V. Glass
    statistically aggregated the findings of 375
    psychotherapy outcome studies
  • Glass (and colleague Smith) concluded that
    psychotherapy did indeed work - the typical
    therapy trial raised the treatment group to a
    level about two-thirds of a standard deviation on
    average above untreated controls the average
    person received therapy finished the experiment
    in a position that exceeded the 75th percentile
    in the control group on whatever outcome measure
    happened to be taken (Glass, 2000). Glass called
    the method meta-analysis
  • ( adapted from Lipsey Wilson, 2001)

14
Historical background
  • Underpinning ideas can be identified earlier
  • K. Pearson (1904)
  • Averaged correlations for typhoid mortality after
    inoculation across 5 samples
  • R. A. Fisher (1944)
  • When a number of quite independent tests of
    significance have been made although few or
    none can be claimed individually as significant,
    yet the aggregate gives an impression that the
    probabilities are on the whole lower than would
    often have been obtained by chance (p. 99).
  • Source of the idea of cumulating probability
    values
  • W. G. Cochran (1953)
  • Discusses a method of averaging means across
    independent studies
  • Set out much of the statistical foundation for
    meta-analysis (e.g., Inverse variance weighting
    and homogeneity testing)
  • ( adapted from Lipsey Wilson, 2001)

15
Significance versus effect size
  • Traditional test is of statistical significance
  • The difference is unlikely to have occurred by
    chance
  • However it may not be
  • Large
  • Important, or even
  • Educationally significant

16
The rationale for using effect sizes
  • Traditional reviews focus on statistical
    significance testing
  • Highly dependent on sample size
  • Null finding does not carry the same weight as
    a significant finding
  • Meta-analysis focuses on the direction and
    magnitude of the effects across studies
  • From Is there a difference? to How big is the
    difference?
  • Direction and magnitude represented by effect
    size

17
Effect size
  • Comparison of impact
  • Same AND different measures
  • Significance vs effect size
  • Does it work? vs How well does it work?

18
Effect size
  • Standardised way of looking at gain scores
  • Different methods for calculation
  • Experimental group mean - Control mean/ Standard
    deviation

19
What is effect size?
  • Standardised way of looking at difference
  • Different methods for calculation
  • Odds Ratio
  • Correlational (Pearsons r)
  • Standardised mean difference
  • Difference between control and intervention group
    as proportion of the dispersion of scores

20
Calculating effect size
  • Control group gain minus experimental group gain
    divided by the standard deviation of the groups

21
Effect size and impact
22
Interpreting effect sizes
  • Relative effects - average is about 0.37 - 0.4
    (Sipe and Curlette, 1997 Hattie, Biggs and
    Purdie, 1996)
  • Doing something different makes a difference
  • Visualising the difference

23
How much is the impact?
  • 0.1 percentile gain of 6 points
  • ie a class ranked 50th in a league table of 100
    schools would move from 50th to about 44th place
  • 0.5 percentile gain of 20 points
  • ie move from 50th to 30th place
  • 1.0 percentile gain of 34 points
  • ie move from 50th to 16th place

24
Other interpretations
  • 0.2 small difference in height between 15-16
    year olds
  • 0.5 medium difference in height between 14
    and 18 year olds
  • 0.8 large difference in height between 13 and
    18 year olds

Cohen 1969
25
Meta-analysis
  • Key question
  • Search protocol
  • Inclusion/exclusion criteria
  • Coding
  • Statistical exploration of findings
  • Mean
  • Distribution
  • Sources of variance

26
(No Transcript)
27
Some findings from meta-analysis
  • Pearson et al. 2005
  • 20 research articles, 89 effects related to
    digital tools and learning environments to
    enhance literacy acquisition. Weighted effect
    size of 0.489 indicating technology can have a
    positive impact on reading comprehension
  • Bernard et al. 2004
  • Distance education and classroom instruction -
    232 studies, 688 effects - wide range of effects
    (heterogeneity) asynchronous DE more effective
    than synchronous

28
More findings
  • Hattie and Timperley, 2007
  • The Power of Feedback, synthesis of other
    meta-analyses on feedback to provide a conceptual
    review 196 studies, 6972 effects - average effect
    of feedback on learning 0.79

29
Rank (or guess) some effect sizes
  • Formative assessment
  • CASE (Cognitive Acceleration Through Science
    Education)
  • Individualised instruction
  • ICT
  • Homework
  • Direct instruction

30
Rank order of effect sizes
  • 1. 04 CASE (Cognitive Acceleration Through
    Science Education) (Boys science GCSE - Adey
    Shayer, 1991)
  • 0.6 Direct instruction (Sipe Curlette, 1997)
  • 0.43 Homework (Hattie, 1999)
  • 0.32 Formative assessment (KMOFAP)
  • 0.31 ICT (Hattie, 1999)
  • 0.1 Individualised instruction (Hattie, 1999)

31
Super-syntheses
  • Syntheses of meta-analyses
  • Relative effects of different interventions
  • Assumes variation evens out across studies with a
    large enough dataset (Marzano/Hattie) or attempts
    to control for the variation statistically (Sipe
    Curlette)

32
Hattie Biggs and Purdie, 1996
  • Synthesis of study skills interventions
  • Meta-analysis of 51 studies of study skills
    interventions. Categorised the inverventions
    using the SOLO model (Biggs Collis, 1982),
    classified studies into four hierarchical levels
    of structural complexity and as either near or
    far transfer. The results support situated
    cognition, and that training for other than
    simple mnemonic tasks should be in context, use
    tasks within the same domain as the target
    content, and promote a high degree of learner
    activity and metacognitive awareness.
  • (average effect 0.4)

33
Sipe and Curlette, 1997
  • A metasynthesis of factors relating to
    educational achievement - testing Walbergs
    educational productivity model - synthesis of
    103 meta-analyses

34
Marzano, 1998
  • Theory driven
  • Self system - metacognition - cognition/
    knowledge
  • Self - 0.74
  • Metacogntive 0.72
  • Cognitive 0.55

35
Discussion
  • Work with a colleague to put the statements in
    order of how comparable you think the research
    findings are
  • Join another pair (or pairs) and decide how
    comfortable would you be with comparing the
    findings

36
Issues and challenges in meta-analysis
  • Conceptual
  • Reductionist - the answer is 42
  • Comparability - apples and oranges
  • Atheoretical - flat-earth
  • Technical
  • Heterogeneity
  • Publication bias
  • Methodological quality

37
Reductionist or flat earth critique
  • The flat earth criticism is based on Lee
    Cronbachs assertion that a meta-analysis looks
    at the big picture and provides only a crude
    average. According to Cronbach,
  • some of our colleagues are beginning to sound
    like a Flat Earth Society. They tell us that the
    world is essentially simple most social
    phenomena are adequately described by linear
    relations one-parameter scaling can discover
    coherent variables independent of culture and
    population and inconsistencies among studies of
    the same kind will vanish if we but amalgamate a
    sufficient number of studiesThe Flat Earth folk
    seek to bury any complex hypothesis with an
    empirical bulldozer (Cronbach, 1982, in Glass,
    2000).

38
Comparability
  • Apples and oranges
  • Same test
  • Different measures of the same construct
  • Different measures of different constructs
  • What question are you trying to answer?
  • How strong is the evidence for this?

Of course it mixes apples and oranges in the
study of fruit, nothing else is sensible
comparing apples and oranges is the only endeavor
worthy of true scientists comparing apples to
apples is trivial (Glass, 2000).
39
Empirical not theoretical?
  • What is your starting point?
  • Conceptual/ theoretical critique
  • Marzano
  • Hattie
  • Sipe and Curlette

40
Technical issues
  • Interventions
  • Publication bias
  • Methodological quality
  • Sample size
  • Homogeneity/ heterogeneity

41
Interventions
  • Super-realisation bias (Cronbach al. 1980)
  • Small-scale interventions tend to get larger
    effects
  • Enthusiasm, attention to detail, quality of
    personal relationships

42
Publication bias
  • Statistically significant (positive) findings
  • Smaller studies need larger effect size to reach
    significance
  • Larger effects
  • Funnel plot sometimes used to explore this
  • Scatterplot of the effects from individual
    studies (horizontal axis) against a study size
    (vertical axis)

43
(No Transcript)
44
Methodological quality
  • Traditional reviews privilege methodological
    rigour
  • Low quality studies higher effect sizes (Hattie
    Biggs Purdie, 1996)
  • No difference (Marzano, 1998)
  • High quality studies, higher effect sizes (Lipsey
    Wilson, 1993)
  • Depends on your definition of quality

45
(No Transcript)
46
Sample size
  • Median effect sizes for studies with sample
    sizes less than 250 were two to three times as
    large as those of larger studies. (Slavin
    Smith, 2008)

47
Heterogeneity
  • Variation in effect sizes
  • Investigate to find clusters (moderator
    variables)
  • Assumption that the effect will be consistent

48
Questions and reactions
  • With a colleague see if you can identify a
    question arising from the presentation so far
  • What is your reaction to the technique
  • How useful is it
  • Generally
  • To your own work?

49
Strengths of Meta-Analysis
  • Uses explicit rules to synthesise research
    findings
  • Can find relationships across studies which may
    not emerge in qualitative reviews
  • Does not (usually) exclude studies for
    methodological quality to the same degree as
    traditional methods
  • Statistical data used to determine whether
    relationships between constructs need clarifying
  • Can cope with large numbers of studies which
    would overwhelm traditional methods of review

50
Summary
  • Replicable and defensible method for
    synthesizing findings across studies (Lipsey
    Wilson, 2001)
  • Identifies gaps in the literature, providing a
    sound basis for further research
  • Indicates the need for replication in education
  • Facilitates identification of patterns in the
    accumulating results of individual evaluations
  • Provides a frame for theoretical critique

51
Other approaches to synthesis
  • Narrative
  • Quantitative (meta-analysis)
  • Best-evidence synthesis (Slavin)
  • Realist (Pawson)
  • Meta-ethnography (Noblitt Hare)
  • Thematic synthesis (Thomas Harden)
  • Grounded theory

52
Suggestions
  • Be explicit about your rationale
  • Be systematic (or at least methodical)
  • Be transparent
  • Describe
  • Analyse (content and methodology)
  • Synthesise

53
A (narrative) metaphor
  • Literature review as rhetoric
  • An act of persuasion
  • Introduce your study

54
Some useful websites
  • EPPI, Institute of Education, London
  • http//eppi.ioe.ac.uk/
  • The Campbell Collaboration
  • http//www.campbellcollaboration.org/
  • Best Evidence Encyclopedia, Johns Hopkins
  • http//www.bestevidence.org/
  • Best Evidence Synthesis (BES), NZ
  • http//www.educationcounts.govt.nz/themes/BES
  • Institute for Effective Education (York)
  • http//www.york.ac.uk/iee/research/reviews

55
Further training
  • ESRC RDI in quantitative synthesis
  • One day training sessions
  • Introduction (for interpretation)
  • Methods Training (for application)
  • Issues Seminars (methodological issues)
  • Durham, London, Edinburgh, Bristol, Belfast, York
  • s.e.higgins_at_durham.ac.uk

56
References
  • Bernard, R.M., Abrami, P.C., Lou, Y.,
    Borokhovski, E., Wade, A., Wozney, L., Wallet,
    P.A., Fiset, M., Huang, B. (2004) How Does
    Distance Education Compare with Classroom
    Instruction? A Meta-Analysis of the Empirical
    Literature Review of Educational Research, 74. 3,
    (Autumn, 2004), pp. 379-439.
  • Chambers, E.A. (2004). An introduction to
    meta-analysis with articles from the Journal of
    Educational Research (1992-2002). Journal of
    Educational Research, 98, pp 35-44.
  • Cronbach, L. J., Ambron, S. R., Dornbusch, S. M.,
    Hess, R.O., Hornik, R. C., Phillips, D. C.,
    Walker, D. F., Weiner, S. S. (1980). Toward
    reform of program evaluation Aims, methods, and
    institutional arrangements. San Francisco, Ca.
    Jossey-Bass.
  • Glass, G.V. (2000). Meta-analysis at 25.
    Available at http//glass.ed.asu.edu/gene/papers/
    meta25.html (accessed 9/9/08)
  • Hattie, J. A. (1992). Measuring the effects of
    schooling. Journal of Education, 36, pp 5-13
  • Hattie, J., Biggs, J. and Purdie, N. (1996)
    Effects of Learning Skills Interventions on
    Student Learning A Meta-analysis Review of
    Educational Research 66.2 pp 99-136.
  • Hattie, J.A. (1987) Identifying the salient
    facets of a model of student learning a
    synthesis of meta-analyses International Journal
    of Educational Research, 11 pp 187- 212.
  • Hattie, J. Timperley, H. (2007) The Power of
    Feedback Review of Educational Research 77. 1,
    pp. 81112.
  • Lipsey, Mark W., and Wilson, David B. (2001).
    Practical Meta-Analysis. Applied Social Research
    Methods Series (Vol. 49). Thousand Oaks, CA SAGE
    Publications.
  • Marzano, R. J. (1998) A Theory-Based
    Meta-Analysis of Research on Instruction. Aurora,
    Colorado, Mid-continent Regional Educational
    Laboratory. Available at http//www.mcrel.org80/
    topics/products/83/ (accessed 2/9/08).
  • Pearson, D.P., Ferdig, R.E., Blomeyer, R.L.
    Moran, J. (2005) The Effects of Technology on
    Reading Performance in the Middle-School Grades
    A Meta-Analysis With Recommendations for Policy
    Naperville, Il University of Illinois/North
    Central Regional Educational Laboratory .
  • Sipe, T. Curlette, W.L. (1997) A Meta-Synthesis
    Of Factors Related To Educational Achievement A
    Methodological Approach To Summarizing And
    Synthesizing Meta-Analyses International Journal
    of Educational Research 25. 7. pp. 583-698.
  • Slavin, R.E. and Smith, D. (2008) Effects of
    Sample Size on Effect Size in Systematic Reviews
    in Education Paper presented at the annual
    meetings of the Society for Research on Effective
    Education, Crystal City, Virginia, March 3-4,
    2008.
Write a Comment
User Comments (0)
About PowerShow.com