The following slides were presented at a meeting of potential editors and methods advisors for the p

1

• The following slides were presented at a meeting
  of potential editors and methods advisors for the
  proposed Cochrane review group in February 2008.
  The slides were designed to promote discussion
  rather than represent the views and directions of
  this group.

2

• Approaches to Synthesis of
• Heterogeneous Evidence

Randy Elder, PhD, MEd Scientific Director for
Systematic Reviews Guide to Community Preventive
Services National Center for Health
Marketing Centers for Disease Control and
Prevention (CDC)
3
It is the mark of an educated man to look for
precision in each class of things just so far as
the nature of the subject admits.
Aristotle, c. 350 BC
4
Goals

• Review of tools for addressing heterogeneity
• Discuss utility of descriptive statistical
  approaches when MA is not feasible
• Raise conceptual and practical issues related to
  heterogeneity
• Substantive sources of variance
• Methodological sources of variance

5
Methods for SynthesizingHeterogeneous Data
6
Inferential Statistical Approaches Meta-analysis

• Requires sufficient homogeneity for estimate of
  central tendency to be useful
• Likely to be relatively uncommon in HPPH reviews
• Less complex interventions are most likely
  candidates
• E.g., safety belt laws
• Subgroup analysis can be used to account for
  some heterogeneity

7
Inferential Statistical Approaches
Meta-regression

• Able to account for sources of heterogeneity in
  more complex interventions
• Partially addresses colinearity issues that
  bedevil univariate subgroup analyses
• Potentially useful for selected interventions
  with large evidence base
• E.g., some school-based interventions
• Pitfalls include
• Poor reporting/measurement of effect modifiers
• Underpowered analyses of effect modification
• Potential for false positives with multiple
  comparisons
• Susceptibility to ecological fallacy

8
Descriptive ApproachesNarrative Synthesis

• Likely to be the most common approach for complex
  HPPH reviews
• ESRS guidance on narrative synthesis is a
  valuable tool for editors and authors
• Pro
• Can be applied to any data
• Often only option given heterogeneous
  interventions, populations, and outcomes
• Allows thoughtful synthesis of small bodies of
  evidence
• Con
• Challenging for larger bodies of evidence
• Tabular and graphical techniques can be helpful
  additions
• More prone to biased interpretation
• E.g., temptation to engage in vote-counting
• More difficult to evaluate effect modification

9
Use of Descriptive Statisticswith Narrative
Synthesis

• Descriptive summary statistics can provide a
  useful supplement to tabular and graphical
  methods
• Facilitate simple, concise text summaries of
  distribution of study results
• What is the central tendency? (e.g., median)
• How much variation in results can be expected?
  (e.g., range, interquartile interval)

10
Price Elasticities by Beverage Type
Design Suitability
11
Conceptual and Practical Issuesin Addressing
Heterogeneity
12
Accounting for Heterogeneity Effect Modification
and Subgroup Analysis

• HPPH and ECRS guidance on subgroup analysis
• Do it (within reason and with theoretical
  justification)
• Report results
• Interpret them cautiously
• This has the practical benefits of providing
  end-users with information they need
• Decisions re when, where, how, and with whom to
  implement interventions need to be made
• Any information is preferable to none
• An a priori assumption of homogeneity is a far
  less conservative approach
• Analyses done from a hypothesis-testing
  perspective face issues of confounding and tend
  to be underpowered (substantial risk of Type II
  error)
• Not doing analyses effectively guarantees Type II
  error (of uncertain magnitude)
• As I understand it

13
Incorporating Non-randomized Studies Cochrane
NRS Guidance

• Cochrane NRS Group guidance
• Dont use NRSs to supplement RCT data on
  effectiveness
• Few RCTs provide imprecise, unbiased estimate
• Including NRSs increases precision, but at the
  unacceptable cost of accuracy
• This position has some merit, but ignores some
  important characteristics of HPPH interventions
  and reviews

14
Sources of Variance in HPPH Reviews

• Meta-synthesis of psychological, behavioral and
  educational interventions (Wilson Lipsey ,
  2001)
• Reasonable generalizability to HP interventions
• Substantive variance (25 of total)
• Methodological Variance (21 of total)
• Study design (4)
• Operationalization of outcome (8)
• EPPI meta-synthesis on policy studies will be
  useful

15
Rationale for Including NRSs in Complex
Population-level Interventions (1)

• Bias needs to be considered at two levels
• The study (internal validity)
• The systematic review (generalizability)
• Distinction between systematic and non-systematic
  biases is also important

16
Rationale for Including NRSs in Complex
Population-level Interventions (2)

• Non-systematic sources of bias appear to
  contribute variance within an acceptable range of
  noise
• Selection bias is the major systematic threat in
  NRSs
• Self-selection
• Researcher-selection
• Threats of self-selection bias are not identical
  across interventions
• Most likely with individual-level interventions
• Less likely with population-level interventions
• Complicated causal pathway to implementation
  reduces risk of confounding
• Availability of data on comparability of groups
  pre-intervention
• Registry of PH interventions would help address
  researcher-driven selection biases

17
Rationale for Including NRSs in Complex
Population-level Interventions (3)

• Limiting reviews to RCTs may introduce more bias
  than it prevents
• Biassystematic error in the population effect
  estimate
• RCTs may provide biased effect estimates for
  complex interventions due to
• ITT analysis (difference between the
  effectiveness of the intervention and of
  randomization to the intervention condition)
• Resources
• Population selection
• Adherence to protocol
• Benefits of including NRSs
• Power

18
Rationale for Including NRSs in Complex
Population-level Interventions (4)

• Generalizability
• Power
• Increases potential to provide useful guidance on
  lumped effects
• Dramatically increases potential to provide
  useful guidance on effect modification issues
  (but only when there is no firewall between
  RCTs and other studies)

19
A Judgment Call

• Is study design such a unique and important
  source of variance that it should be singled out
  from among all other potential sources of bias
  and effect modification?
• Or do the harms of treating study design as
  qualitatively different from all other potential
  modifiers of effect estimates outweigh the
  benefits?

20
If the Latter..

• Guidance re addressing quantitative
  differences in study quality should apply
• Consider limiting review to studies above a
  threshold design quality
• Considering plausible systematic sources of
  variance for given subject matter
• Use sensitivity analysis to evaluate robustness
  of findings (giving up on the quest for
  precision)
• Avoid or cautiously apply quality weighting by
  design alone

21
Beware of the Outlier Randomized Trial

• Shatterproof glassware
• Students Against Drunk Driving
• Any multi-million dollar trial that cant
  feasibly be brought to scale

22
Thank You!

• Randy Elder
• rfe3_at_cdc.gov
• www.thecommunityguide.org