Systematic Review and MetaAnalysis: Lecture 2

1
Systematic Review and Meta-Analysis Lecture 2

• Dejana Braithwaite PhD MSc
• Assistant Professor
• Department of Epidemiology and Biostatistics

2

• Meta-analysis begins with scientific
  studies usually performed by academics or
  government agencies and sometimes incomplete or
  disputed. The data from the studies are then run
  through computer models of bewildering
  complexity which produce results of implausible
  precision.

Davis B. Wall Street Journal 1992
3
Learning objectives

• Understand the principles of combining studies
  quantitatively
• Understand key meta-analytic models
• - fixed effects models
• - random effects models
• 3. Understand methods for examining reasons for
  heterogeneity among studies

4
Historical note on the importance of research
synthesis

• Karl Pearson is probably the first medical
  researcher to use formal techniques to combine
  data from different studies (1904)
• He synthesized data from several studies on
  efficacy of typhoid vaccination
• His rationale for pooling data
• Many of the groups are far too small to allow
  of any definite opinion being formed at all
  having regard to the size of the probable error
  involved.

Egger et al. Systematic reviews in health care.
London BMJ Publications 2001.
5
Prof Archibald Cochrane CBE (1909 - 1988)

• The Cochrane Collaboration is named in honour of
  Archie Cochrane a British researcher.
• In 1979 he wrote It is surely a great criticism
  of our profession that we have not organised a
  critical summary by specialty or subspecialty
  adapted periodically of all relevant randomized
  controlled trials

Source http//www.cochrane.org/cochrane/archieco.
htm
6
The Cochrane Collaboration

• Archie Cochranes challenge led to the
  establishment during the 1980s of an
  international collaboration to develop the Oxford
  Database of Perinatal Trials.
• His encouragement and the endorsement of his
  views by others led to the opening of the first
  Cochrane centre (in Oxford UK) in 1992 and the
  founding of The Cochrane Collaboration in 1993.

Source http//www.cochrane.org/cochrane/archieco.
htm
7
Systematic reviews/meta-analyses indexed in
PubMed 10 years
Search meta-analysis(MeSH) OR meta-analysis(tw)
OR systematic review(tw)
8
http//www.medepi.net/meta/
9
Systematic Review in context
10
(Systematic review) (Metaanalysis)

• Meta-analysis is the statistical part of a
  systematic review
• Meta-analysis is always part of a systematic
  review
• Systematic review does not always have a
  meta-analytic part

11
Meta-analyses
IPD
Systematic reviews
IPD individual participant data
12
Systematic reviews key points

• Systematic review(s) when properly done
• are transparent
• are NOT synonymous with meta-analysis
• are replicable by others
• are exploding in frequency in numerous health
  disciplines
• Are valuable even when number of studies 0

13
Elements of a study protocol for a systematic
review and meta-analysis

• Objectives
• Background
• Information retrieval
• Data collection
• Data analysis

14
Steps in data analysis
Today

• Create summary data
• Examine heterogeneity
• Examine publication bias
• Consider conducting subgroup analysis (according
  to a priori hypotheses)

15
Central questions of interest
Are the results of the studies fairly similar
(consistent)
Yes
No
What is the common summary effect
What factors can explain the dissimilarities
(heterogeneity) in the study results
How precise is the common summary effect
16
Opening example of a meta-analysis BCG and the
risk of TB

• JAMA. 1994 Mar 2271(9)698-702.
• Efficacy of BCG vaccine in the prevention of
  tuberculosis. Meta-analysis of the published
  literature.
• Colditz GA Brewer TF Berkey CS Wilson ME
  Burdick E Fineberg HV Mosteller F. Technology
  Assessment Group Harvard School of Public
  Health Boston MA 02115.

17

• OBJECTIVE--To quantify the efficacy of BCG
  vaccine against tuberculosis (TB)
• DATA SOURCES-- MEDLINE experts from the Centers
  for Disease Control and Prevention and the World
  Health Organization
• STUDY SELECTION --1264 articles or abstracts
  considered 70 articles reviewed in depth
• --14 prospective trials and 12 case-control
  studies included
• DATA EXTRACTION --Recorded study design age
  range of study population number of patients
  enrolled efficacy of vaccine and items to
  assess the potential for bias in study design and
  diagnosis.

18

• DATA SYNTHESIS-- The relative risk (RR) or odds
  ratio (OR) of TB used to provide the measure of
  vaccine efficacy that we analyzed. The protective
  effect was then computed by 1-RR or 1-OR.
• A random-effects model estimated a weighted
  average RR or OR from those provided by the
  trials or case-control studies.
• In the trials the RR of TB was 0.49 (95
  confidence interval CI 0.34 to 0.70) for
  vaccine recipients compared with nonrecipients
  (protective effect of 51).
• In the case-control studies the OR for TB was
  0.50 (95 CI 0.39 to 0.64) or a 50 protective
  effect.
• 7 trials reporting tuberculous deaths showed a
  protective effect from BCG vaccine of 71 (RR
  0.29 95 CI 0.16 to 0.53)
• 5 studies reporting on meningitis showed a
  protective effect from BCG vaccine of 64 (OR
  0.36 95 CI 0.18 to 0.70).

19

• Geographic latitude of the study site and study
  validity score explained 66 of the heterogeneity
  among trials in a random-effects regression
  model.

20
(No Transcript)
21

• CONCLUSION--On average BCG vaccine significantly
  reduces the risk of TB by 50.
• Protection observed across many populations
  study designs and forms of TB.
• Age at vaccination did not enhance predictiveness
  of BCG efficacy.
• Protection against tuberculous death meningitis
  and disseminated disease is higher than for total
  TB cases although this result may reflect
  reduced error in disease classification rather
  than greater BCG efficacy.

22
Steps in data analysis

• Create summary data
• Examine heterogeneity
• Consider examining study quality and publication
  bias
• Consider conducting subgroup analysis (according
  to a priori hypotheses)

23
Step 1. Create summary data

• Prepare tables showing characteristics of primary
  studies
• Year
• Setting
• Patients
• Design
• Outcome (results)
• Gives a first hand feel for the data
• Can make some assessment of quality and
  heterogeneity

24

• Decide what data to combine
• Data types
• Continuous
• Dichotomous
• Examples of measures that can be combined
• Risk ratio
• Odds ratio
• Risk difference
• Effect size (Z statistic standardized mean
  difference)
• P-values
• Correlation coefficient (R)

25
Summary dataExample Cochrane albumin review
Cochrane Injuries Group Albumin Reviewers. Human
albumin administration in critically ill
patients systematic review of randomised
controlled trials. BMJ 1998317235-40.
26

• Efficient way of presenting summary results
• Forest plot
• Presents the point estimate and CI of each trial
• Also presents the overall summary estimate
• Allows visual appraisal of heterogeneity
• Other graphs
• Cumulative meta-analysis

27
Example 1 Meta-analysis
28
Example 2 Meta-analysis
29
Forest Plot
Cochrane albumin review
BMJ 1998317235-240
30
Braithwaite et al. JNCI 2004
31
Cumulative Meta-analysis Plot
Passive smoking and lung cancer review
Hackshaw AK et al. BMJ 1997315980-88.
32
(No Transcript)
33
Road Map to Generating Summary Effect Estimates

• Fixed effects models
• Mantel-Haenszel (ratio measures)
• Peto (ratio measures)
• General variance based (ratio and difference)
• Random effects models
• DerSimonian Laird (ratio and difference measures)

34
Fixed versus random effects models

• Fixed effects model assumes that the true effect
  of treatment is the same value in each study
  (fixed) the differences between studies is
  solely due to random error
• In random effects models the treatment effects
  for the individual studies are assumed to vary
  around some overall average treatment effect
• Allows for random error plus inter-study
  variability
• Results in wider confidence intervals
  (conservative)
• Studies tend to be weighted more equally
  (relatively more weight is given to smaller
  studies)

35
Steps in data analysis

• Create summary data
• Examine heterogeneity
• Consider examining study quality and publication
  bias
• Consider conducting subgroup analysis (according
  to a priori hypotheses)

36
Step 2. Examine heterogeneity

• Indicates that effect sizes vary considerably
  across studies
• If heterogeneity is present a common summary
  measure is hard to interpret
• Can be due to differences in
• Patient populations studied
• Interventions used
• Co-interventions
• Outcomes measured
• Study design features (eg. length of follow-up)
• Study quality
• Random error

37

• How to look for heterogeneity
• Visually
• Forest plot do confidence intervals of studies
  overlap with each other and the summary effect
• Statistically
• Chi-square test for heterogeneity
  (Mantel-Haenszel test or Cochran Q test)
• Tests whether the individual effects are farther
  away from the common effect beyond what is
  expected by chance
• Has poor power
• P-value

38
Visual appraisal of heterogeneity
Zinc for common cold Summary and incidence odds
ratios for the incidence of any cold symptom at 1
wk
Jackson JL et al. Zinc and the common cold a
meta-analysis revisited. J of Nutrition.
20001301512S-1515S
39
Dealing with Heterogeneity

• If significant heterogeneity is found
• Find out what factors might explain the
  heterogeneity
• Can decide not to combine the data

• If no significant heterogeneity
• Can perform meta-analysis and generate a common
  summary effect measure

40
Examples for calculating summary statistics and
assessing heterogeneity
41
Road Map to Generating Summary Effect Estimates

• Fixed effects models
• Mantel-Haenszel (ratio measures)
• Peto (ratio measures)
• General variance based (ratio and difference)
• Random effects models
• DerSimonian Laird (ratio and difference measures)

42
Example 1

• Count data (cohort or case-control studies)
• Outcome dichotomous (disease disease-)
• Goal is either an average RR (CIR or OR) over all
  of the studies or an average RD
• Issuehow to weight the studies in the overall
  average
• Issuehow to develop inference about the overall
  result (i.e. is it significant what is the CI
  etc.)

43
(No Transcript)
44
(No Transcript)
45
(No Transcript)
46
Example 2

• Outcome is continuous
• Goal is an average difference (in means) over all
  of the studies
• Issuehow to weight the studies in the overall
  average difference
• Issuehow to develop inference about the overall
  result (i.e. is the difference significant what
  is the CI etc.)

47
(No Transcript)
48
(No Transcript)
49
(No Transcript)
50
Pooled estimates (summary ratios or differences)
51
(No Transcript)
52
(No Transcript)
53
(No Transcript)
54
(No Transcript)
55
Mantel-Haenszel Test of Homogeneity
Reminder Comment on use of inverse variance
(IV) weights for wi but MH for point estimate
(i)
56
Dersimonian and Laird Random Effects Model

• (for use in meta-analysis)

57
(No Transcript)
58
DerSimonian and Laird Random Effects Model
Mean and variance
Weight of each study
59
(No Transcript)
60
(No Transcript)
61
Average men having an average meal
62
Combination of heterogeneous studies should
be avoided in favor of searching for sources of
heterogeneity.
63
4. Perform meta-analysis
Moher D et al. Arch Pediatr Adolesc Med
1998152915-20
64
Steps in data analysis

• Create summary data
• Examine heterogeneity
• Consider examining study quality and publication
  bias
• Consider conducting subgroup analysis (according
  to a priori hypotheses)

65
Step 3 Evaluate impact of study quality on
results

• Narrative discussion of impact of quality on
  results
• Display study quality and results in a tabular
  format
• Weight the data by quality (not recommended)
• Subgroup analysis by quality
• Include quality as a covariate in meta-regression

66
Step 3 Explore publication bias

• Studies with significant results are more likely
• to be published
• to be published in English
• to be cited by others
• to produce multiple publications
• Including only published studies can introduce
  publication bias
• Most reviews do not look for publication bias
• Methods for detecting publication bias
• Graphical funnel plot asymmetry
• Tests Egger test Rosenthals Fail-safe N

67
Funnel plot to detect publication bias
Egger et al. Systematic reviews in health care.
London BMJ books 2001.
68
Approaches to publication bias

• Fail-safe N
• Publications are filled with the 5 of studies
  with Type I error but file drawers are filled
  with the 95 with non-significant effects
• Funnel plots (nexts)
• Trim and fill method
• Small studies are removed from funnel plot until
  it is symmetric
• Then replace the small studies and balance them
  with studies on the opposite side of the funnel
• Statistical analogues of funnel plot
• Egger test
• bo b1s
• Where s standard error
• And each study is weighted by 1/var

69
(No Transcript)
70
(No Transcript)
71
(No Transcript)
72
(No Transcript)
73
(No Transcript)
74
(No Transcript)
75
Steps in data analysis

• Create summary data
• Examine heterogeneity
• Consider examining study quality and publication
  bias
• Consider conducting subgroup analysis (according
  to a priori hypotheses)

76
Subgroup analysis example
Egger et al. Systematic reviews in health care.
London BMJ books 2001.
77
Important controversies in the field
78
Important controversy Goal of meta-analysis

• The analytic goal Or should it be the
  identification and estimation of differences
  among study-specific effects (i.e. focused on the
  causes of heterogeneity)
• A purely analytic meta-analysis can leave a
  disappointing sense of not reaching a
  conclusion

The synthetic goal The estimation of a summary
or average effect across studies A purely
synthetic meta-analysis can give a false sense of
consistency across studies (despite the
possibility of a consistent error across many
studies)
OR
79
Meta-analysis versus mega trial

• Meta-analysis
• can test hypotheses about sources of differences
  and magnitudes of biases

• Mega trial
• Inference contingent on the study design

80
Meta-analysis versus single large trial
81
The use of meta-analysis in clinical trials

• Can help to define pre-trial expected effect
  sizes (thus permitting a more realistic sample
  size estimation)
• Identify (through meta-regression) the sources of
  heterogeneity in prior studiesand address these
  sources in the design phase of a new trial
• Determine effect estimates in key subgroups (e.g.
  based on gender or race/ethnicity or age etc.).
  Often in subgroups no single trial has
  sufficient data.

82
The use of meta-analysis in study designs that
are not clinical trials

• Although meta-analysis is most widely known in
  biomedical settings for its application to
  clinical trials the technique can also be used
  to synthesize/analyze other study designs
• Observational studies (e.g. case control
  non-randomized cohorts cross-sectional
  prevalence studies etc.)
• Studies evaluating diagnostic tests (sensitivity
  specificity predictive value)
• IPD individual patient data studies

83
Meta-analysis Software

• Free
• RevMan Review Manager
• Meta-Analyst
• Epi Meta
• Easy MA
• Meta-Test
• Meta-Stat
• Commercial
• Comprehensive Meta-analysis
• Meta-Win
• WEasy MA
• General stats packages
• Stata
• SAS
• S-Plus

http//www.prw.le.ac.uk/epidemio/personal/ajs22/me
ta/
84
Meta-analysis in Stata
85
(No Transcript)
86
(No Transcript)
87
(No Transcript)
88
(No Transcript)
89
(No Transcript)
90
(No Transcript)
91
Guidelines
92
QUORUM

• Moher et al. Improving the quality of reports of
  meta-analyses of randomised controlled trials
  The QUORUM statement. Lancet 19993541896-1900

93
(No Transcript)
94
MOOSE

• Stroup et al. Meta-analysis of observational
  studies in epidemiology. JAMA 20002832008-12

95
Section 2 Meta-analysis in STATA

• 2-3 hours to complete problem set
• May take longer if conducting own meta-analysis
• 4 things to learn
• Compute summary estimates
• Compute test of heterogeneity
• Test of publication bias
• Subgroup analysis