Title: Systematic Review and MetaAnalysis: Lecture 2
1Systematic Review and Meta-Analysis Lecture 2
- Dejana Braithwaite PhD MSc
- Assistant Professor
- Department of Epidemiology and Biostatistics
2- Meta-analysis begins with scientific
studies usually performed by academics or
government agencies and sometimes incomplete or
disputed. The data from the studies are then run
through computer models of bewildering
complexity which produce results of implausible
precision.
Davis B. Wall Street Journal 1992
3Learning objectives
- Understand the principles of combining studies
quantitatively - Understand key meta-analytic models
- - fixed effects models
- - random effects models
- 3. Understand methods for examining reasons for
heterogeneity among studies -
4Historical note on the importance of research
synthesis
- Karl Pearson is probably the first medical
researcher to use formal techniques to combine
data from different studies (1904) - He synthesized data from several studies on
efficacy of typhoid vaccination - His rationale for pooling data
- Many of the groups are far too small to allow
of any definite opinion being formed at all
having regard to the size of the probable error
involved.
Egger et al. Systematic reviews in health care.
London BMJ Publications 2001.
5Prof Archibald Cochrane CBE (1909 - 1988)
- The Cochrane Collaboration is named in honour of
Archie Cochrane a British researcher. - In 1979 he wrote It is surely a great criticism
of our profession that we have not organised a
critical summary by specialty or subspecialty
adapted periodically of all relevant randomized
controlled trials
Source http//www.cochrane.org/cochrane/archieco.
htm
6The Cochrane Collaboration
- Archie Cochranes challenge led to the
establishment during the 1980s of an
international collaboration to develop the Oxford
Database of Perinatal Trials. - His encouragement and the endorsement of his
views by others led to the opening of the first
Cochrane centre (in Oxford UK) in 1992 and the
founding of The Cochrane Collaboration in 1993.
Source http//www.cochrane.org/cochrane/archieco.
htm
7Systematic reviews/meta-analyses indexed in
PubMed 10 years
Search meta-analysis(MeSH) OR meta-analysis(tw)
OR systematic review(tw)
8http//www.medepi.net/meta/
9Systematic Review in context
10(Systematic review) (Metaanalysis)
- Meta-analysis is the statistical part of a
systematic review - Meta-analysis is always part of a systematic
review - Systematic review does not always have a
meta-analytic part
11Meta-analyses
IPD
Systematic reviews
IPD individual participant data
12Systematic reviews key points
- Systematic review(s) when properly done
- are transparent
- are NOT synonymous with meta-analysis
- are replicable by others
- are exploding in frequency in numerous health
disciplines - Are valuable even when number of studies 0
13Elements of a study protocol for a systematic
review and meta-analysis
- Objectives
- Background
- Information retrieval
- Data collection
- Data analysis
14Steps in data analysis
Today
- Create summary data
- Examine heterogeneity
- Examine publication bias
- Consider conducting subgroup analysis (according
to a priori hypotheses)
15Central questions of interest
Are the results of the studies fairly similar
(consistent)
Yes
No
What is the common summary effect
What factors can explain the dissimilarities
(heterogeneity) in the study results
How precise is the common summary effect
16Opening example of a meta-analysis BCG and the
risk of TB
- JAMA. 1994 Mar 2271(9)698-702.
- Efficacy of BCG vaccine in the prevention of
tuberculosis. Meta-analysis of the published
literature. - Colditz GA Brewer TF Berkey CS Wilson ME
Burdick E Fineberg HV Mosteller F. Technology
Assessment Group Harvard School of Public
Health Boston MA 02115.
17- OBJECTIVE--To quantify the efficacy of BCG
vaccine against tuberculosis (TB) - DATA SOURCES-- MEDLINE experts from the Centers
for Disease Control and Prevention and the World
Health Organization - STUDY SELECTION --1264 articles or abstracts
considered 70 articles reviewed in depth - --14 prospective trials and 12 case-control
studies included - DATA EXTRACTION --Recorded study design age
range of study population number of patients
enrolled efficacy of vaccine and items to
assess the potential for bias in study design and
diagnosis.
18- DATA SYNTHESIS-- The relative risk (RR) or odds
ratio (OR) of TB used to provide the measure of
vaccine efficacy that we analyzed. The protective
effect was then computed by 1-RR or 1-OR. - A random-effects model estimated a weighted
average RR or OR from those provided by the
trials or case-control studies. - In the trials the RR of TB was 0.49 (95
confidence interval CI 0.34 to 0.70) for
vaccine recipients compared with nonrecipients
(protective effect of 51). - In the case-control studies the OR for TB was
0.50 (95 CI 0.39 to 0.64) or a 50 protective
effect. - 7 trials reporting tuberculous deaths showed a
protective effect from BCG vaccine of 71 (RR
0.29 95 CI 0.16 to 0.53) - 5 studies reporting on meningitis showed a
protective effect from BCG vaccine of 64 (OR
0.36 95 CI 0.18 to 0.70).
19- Geographic latitude of the study site and study
validity score explained 66 of the heterogeneity
among trials in a random-effects regression
model.
20(No Transcript)
21- CONCLUSION--On average BCG vaccine significantly
reduces the risk of TB by 50. - Protection observed across many populations
study designs and forms of TB. - Age at vaccination did not enhance predictiveness
of BCG efficacy. - Protection against tuberculous death meningitis
and disseminated disease is higher than for total
TB cases although this result may reflect
reduced error in disease classification rather
than greater BCG efficacy.
22Steps in data analysis
- Create summary data
- Examine heterogeneity
- Consider examining study quality and publication
bias - Consider conducting subgroup analysis (according
to a priori hypotheses)
23Step 1. Create summary data
- Prepare tables showing characteristics of primary
studies - Year
- Setting
- Patients
- Design
- Outcome (results)
- Gives a first hand feel for the data
- Can make some assessment of quality and
heterogeneity
24- Decide what data to combine
- Data types
- Continuous
- Dichotomous
- Examples of measures that can be combined
- Risk ratio
- Odds ratio
- Risk difference
- Effect size (Z statistic standardized mean
difference) - P-values
- Correlation coefficient (R)
25Summary dataExample Cochrane albumin review
Cochrane Injuries Group Albumin Reviewers. Human
albumin administration in critically ill
patients systematic review of randomised
controlled trials. BMJ 1998317235-40.
26- Efficient way of presenting summary results
- Forest plot
- Presents the point estimate and CI of each trial
- Also presents the overall summary estimate
- Allows visual appraisal of heterogeneity
- Other graphs
- Cumulative meta-analysis
27Example 1 Meta-analysis
28Example 2 Meta-analysis
29Forest Plot
Cochrane albumin review
BMJ 1998317235-240
30Braithwaite et al. JNCI 2004
31Cumulative Meta-analysis Plot
Passive smoking and lung cancer review
Hackshaw AK et al. BMJ 1997315980-88.
32(No Transcript)
33Road Map to Generating Summary Effect Estimates
- Fixed effects models
- Mantel-Haenszel (ratio measures)
- Peto (ratio measures)
- General variance based (ratio and difference)
- Random effects models
- DerSimonian Laird (ratio and difference measures)
34Fixed versus random effects models
- Fixed effects model assumes that the true effect
of treatment is the same value in each study
(fixed) the differences between studies is
solely due to random error - In random effects models the treatment effects
for the individual studies are assumed to vary
around some overall average treatment effect - Allows for random error plus inter-study
variability - Results in wider confidence intervals
(conservative) - Studies tend to be weighted more equally
(relatively more weight is given to smaller
studies)
35Steps in data analysis
- Create summary data
- Examine heterogeneity
- Consider examining study quality and publication
bias - Consider conducting subgroup analysis (according
to a priori hypotheses)
36Step 2. Examine heterogeneity
- Indicates that effect sizes vary considerably
across studies - If heterogeneity is present a common summary
measure is hard to interpret - Can be due to differences in
- Patient populations studied
- Interventions used
- Co-interventions
- Outcomes measured
- Study design features (eg. length of follow-up)
- Study quality
- Random error
37- How to look for heterogeneity
- Visually
- Forest plot do confidence intervals of studies
overlap with each other and the summary effect - Statistically
- Chi-square test for heterogeneity
(Mantel-Haenszel test or Cochran Q test) - Tests whether the individual effects are farther
away from the common effect beyond what is
expected by chance - Has poor power
- P-value
38Visual appraisal of heterogeneity
Zinc for common cold Summary and incidence odds
ratios for the incidence of any cold symptom at 1
wk
Jackson JL et al. Zinc and the common cold a
meta-analysis revisited. J of Nutrition.
20001301512S-1515S
39Dealing with Heterogeneity
- If significant heterogeneity is found
- Find out what factors might explain the
heterogeneity - Can decide not to combine the data
- If no significant heterogeneity
- Can perform meta-analysis and generate a common
summary effect measure
40Examples for calculating summary statistics and
assessing heterogeneity
41Road Map to Generating Summary Effect Estimates
- Fixed effects models
- Mantel-Haenszel (ratio measures)
- Peto (ratio measures)
- General variance based (ratio and difference)
- Random effects models
- DerSimonian Laird (ratio and difference measures)
42Example 1
- Count data (cohort or case-control studies)
- Outcome dichotomous (disease disease-)
- Goal is either an average RR (CIR or OR) over all
of the studies or an average RD - Issuehow to weight the studies in the overall
average - Issuehow to develop inference about the overall
result (i.e. is it significant what is the CI
etc.)
43(No Transcript)
44(No Transcript)
45(No Transcript)
46Example 2
- Outcome is continuous
- Goal is an average difference (in means) over all
of the studies - Issuehow to weight the studies in the overall
average difference - Issuehow to develop inference about the overall
result (i.e. is the difference significant what
is the CI etc.)
47(No Transcript)
48(No Transcript)
49(No Transcript)
50Pooled estimates (summary ratios or differences)
51(No Transcript)
52(No Transcript)
53(No Transcript)
54(No Transcript)
55Mantel-Haenszel Test of Homogeneity
Reminder Comment on use of inverse variance
(IV) weights for wi but MH for point estimate
(i)
56Dersimonian and Laird Random Effects Model
- (for use in meta-analysis)
57(No Transcript)
58DerSimonian and Laird Random Effects Model
Mean and variance
Weight of each study
59(No Transcript)
60(No Transcript)
61Average men having an average meal
62 Combination of heterogeneous studies should
be avoided in favor of searching for sources of
heterogeneity.
634. Perform meta-analysis
Moher D et al. Arch Pediatr Adolesc Med
1998152915-20
64Steps in data analysis
- Create summary data
- Examine heterogeneity
- Consider examining study quality and publication
bias - Consider conducting subgroup analysis (according
to a priori hypotheses)
65Step 3 Evaluate impact of study quality on
results
- Narrative discussion of impact of quality on
results - Display study quality and results in a tabular
format - Weight the data by quality (not recommended)
- Subgroup analysis by quality
- Include quality as a covariate in meta-regression
66Step 3 Explore publication bias
- Studies with significant results are more likely
- to be published
- to be published in English
- to be cited by others
- to produce multiple publications
- Including only published studies can introduce
publication bias - Most reviews do not look for publication bias
- Methods for detecting publication bias
- Graphical funnel plot asymmetry
- Tests Egger test Rosenthals Fail-safe N
67Funnel plot to detect publication bias
Egger et al. Systematic reviews in health care.
London BMJ books 2001.
68Approaches to publication bias
- Fail-safe N
- Publications are filled with the 5 of studies
with Type I error but file drawers are filled
with the 95 with non-significant effects - Funnel plots (nexts)
- Trim and fill method
- Small studies are removed from funnel plot until
it is symmetric - Then replace the small studies and balance them
with studies on the opposite side of the funnel - Statistical analogues of funnel plot
- Egger test
- bo b1s
- Where s standard error
- And each study is weighted by 1/var
69(No Transcript)
70(No Transcript)
71(No Transcript)
72(No Transcript)
73(No Transcript)
74(No Transcript)
75Steps in data analysis
- Create summary data
- Examine heterogeneity
- Consider examining study quality and publication
bias - Consider conducting subgroup analysis (according
to a priori hypotheses)
76Subgroup analysis example
Egger et al. Systematic reviews in health care.
London BMJ books 2001.
77Important controversies in the field
78Important controversy Goal of meta-analysis
- The analytic goal Or should it be the
identification and estimation of differences
among study-specific effects (i.e. focused on the
causes of heterogeneity) - A purely analytic meta-analysis can leave a
disappointing sense of not reaching a
conclusion
The synthetic goal The estimation of a summary
or average effect across studies A purely
synthetic meta-analysis can give a false sense of
consistency across studies (despite the
possibility of a consistent error across many
studies)
OR
79Meta-analysis versus mega trial
- Meta-analysis
- can test hypotheses about sources of differences
and magnitudes of biases
- Mega trial
- Inference contingent on the study design
80Meta-analysis versus single large trial
81The use of meta-analysis in clinical trials
- Can help to define pre-trial expected effect
sizes (thus permitting a more realistic sample
size estimation) - Identify (through meta-regression) the sources of
heterogeneity in prior studiesand address these
sources in the design phase of a new trial - Determine effect estimates in key subgroups (e.g.
based on gender or race/ethnicity or age etc.).
Often in subgroups no single trial has
sufficient data.
82The use of meta-analysis in study designs that
are not clinical trials
- Although meta-analysis is most widely known in
biomedical settings for its application to
clinical trials the technique can also be used
to synthesize/analyze other study designs - Observational studies (e.g. case control
non-randomized cohorts cross-sectional
prevalence studies etc.) - Studies evaluating diagnostic tests (sensitivity
specificity predictive value) - IPD individual patient data studies
83Meta-analysis Software
- Free
- RevMan Review Manager
- Meta-Analyst
- Epi Meta
- Easy MA
- Meta-Test
- Meta-Stat
- Commercial
- Comprehensive Meta-analysis
- Meta-Win
- WEasy MA
- General stats packages
- Stata
- SAS
- S-Plus
http//www.prw.le.ac.uk/epidemio/personal/ajs22/me
ta/
84Meta-analysis in Stata
85(No Transcript)
86(No Transcript)
87(No Transcript)
88(No Transcript)
89(No Transcript)
90(No Transcript)
91Guidelines
92QUORUM
- Moher et al. Improving the quality of reports of
meta-analyses of randomised controlled trials
The QUORUM statement. Lancet 19993541896-1900
93(No Transcript)
94MOOSE
- Stroup et al. Meta-analysis of observational
studies in epidemiology. JAMA 20002832008-12
95Section 2 Meta-analysis in STATA
- 2-3 hours to complete problem set
- May take longer if conducting own meta-analysis
- 4 things to learn
- Compute summary estimates
- Compute test of heterogeneity
- Test of publication bias
- Subgroup analysis