Principles of Research Synthesis - PowerPoint PPT Presentation

Loading...

PPT – Principles of Research Synthesis PowerPoint presentation | free to download - id: f188-OGE1Y



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Principles of Research Synthesis

Description:

... POSITIVE mortality effect in a subgroup defined only by astrological 'birth sign' Astrological 'birth sign' Atenolol effect on day 0-1 mortality ... – PowerPoint PPT presentation

Number of Views:143
Avg rating:3.0/5.0
Slides: 31
Provided by: pitt99
Learn more at: http://www.pitt.edu
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Principles of Research Synthesis


1
Principles of Research Synthesis
San Francisco Radiation Oncology Conference
February 28 to March 2, 2003
Benjamin Djulbegovic, M.D.,PhD. H. Lee Moffitt
Cancer Center University of South
Florida djulbebm_at_moffitt.usf.edu
2
I The need for research synthesis
3
The need for research synthesis
  • Health care decision makers need to access
    research evidence to make informed decisions on
    diagnosis, treatment and health care management
    for both individual patients and populations.
  • There are few important questions in health care
    which can be informed by consulting the result of
    a single empirical study.

4
II The problems with traditional review articles
5
The need for research synthesis
  • Importance of review articles
  • Review articles in medical journals summarize
    large amounts of information on a particular
    topic
  • and therefore are useful and popular source of
    information for health care professionals
  • review articles have the highest impact factor
  • which means, that research, practice and
    policy-decisions are significantly influenced by
    review articles

6
Science of research synthesis problems with
traditional review articles
  • Personal views on the available body of evidence
  • Selection bias and selective citations monster
  • Has been pervasive in medicine, economics and
    social sciences
  • Can obscure up to 40-60 of true interventions
    effect
  • In 2000, Nobel prize in Economic Science was
    awarded to James Heckman of the University of
    Chicago for his analysis of selection bias, which
    in turn profoundly affected applied research in
    economics as well as in other social sciences
  • Lack of reproducibility
  • that is, the lack of the key scientific criterion

7
Selective citation bias blind men and elephant
8
Critique of reviews of chemotherapy for ovarian
cancer
  • Crx superior
  • (by Qualitative analysis) 48/53
  • Search strategy 3/53
  • Inclusion/exclusion criteria 2/53
  • Validity assessment 1/53
  • Quantitative Assessment 3/53

Courtesy of Dr. C. Williams
9
Quality of Review Article158 articles only 2 met
all 10 criteria.
Ann Intern Med 1999131947-951
10
Research Synthesis terminology
  • Systematic review. The application of strategies
    that limit bias in the assembly, critical
    appraisal, and synthesis of all relevant studies
    on a specific topic. Meta-analysis may be, but is
    not necessary, used as part of this process.
  • Meta-analysis. The statistical synthesis of the
    data from separate but similar, i.e. comparable
    studies, leading to a quantitativ summary of the
    pooled results.

11
Key Distinctions Between Narrative and Systematic
Reviews, by Core Features of Such Reviews
Systematic Review
Core Feature
Narrative Review
Study question
Often broad in scope
Often a focused clinical question.
Which databases
Data sources and
Comperehensive search of many
were searched and
search strategy
databases as well as so-called gray
search strategy are not typically
literature. Explicit search strategy
provided.
Not usually specified, potentially
Selection of articles for
Criterion-based
Selection,
biased.
study
uniformly
applied.
Article review or
Variable, depending on
who is
Rigorous critical appraisal, typically
appraisal
conducting the review.
using
a data extraction form.
Some assessment of quality is almost
Study quality
If assessed, may not use formal
always included as part of the data
quality assessment.
extraction process.
Quantitative summary (
meta-analysis)

Often a qualitative summary.
Synthesis
if the data can be appropriately pooled
qualitative otherwise.
Inferences
Sometimes evidence-based.
Usually evidence-based
12
Principles of reliable detection of the effects
of health care interventions
  • Methods to reduce bias
  • Methods to reduce statistical imprecision

13
III Principles of systematic reviews and
meta-analysis
14
Principle 1the need to consider the totality of
evidence
  • The world can be only considered as the totality
    of facts…for the totality of facts determines
    what is the case, and also whatever is not the
    case
  • L. Wittgenstein
    (Tractus logico-philosophicus), 1921

15
Principle 2 requirement for reproducibility
  • Transparent, explicit and systematic approach in
    identifying and synthesizing evidence
  • Methods for search for evidence
  • Inclusion and exclusion criteria
  • Quality assessment

16
Steps of a Systematic Review
Search of
Consultation
personal
Computerized
with experts
files
Databases
Review of
Systematic
reference lists
manual searches
of articles
of key journals
Identify studies
Relevant
Not Relevant
Review for
relevance
Evaluate
methodological quality
Reject
Extract data
Analyze data
Draw Conclusions
17
the QUOROM statement
18
Principles of reliable detection of the effects
of health care interventions
  • Systematic bias must be intervention which we are trying to detect
  • The need for the totality of evidence (published
    and unpublished)
  • Random errors (play of chance) must be effect of intervention which we are trying to
    detect
  • uncertainty/imprecision reduced by pooling all
    available data
  • Need for large number of patients/events

19
Rationale for (quantitative) synthesis of all
available evidence
  • The rationale for pooling data is clinical and
    not statistical
  • Similar interventions for similar conditions will
    produce the similar effects (i.e. in the same
    direction)
  • While the effect size may not be the same, it
    will rarely be in the opposite directions
  • Meta-analysis attempts to show direction of the
    effect (i.e. help establish generalisability of
    the effect)

20
RR (95 CI Fixed)
A)
Disease population
RCT2
RCT3
RCT1
Test for heterogeneity chi square df 2, p0.1
Test for overall effect Z p0.02
0.8
1
10
Relative risk
Favors new treatment
Favors control
21
RR (95 CI Fixed)
B)
Disease population
RCT2
RCT3
RCT1
Test for heterogeneity chi square df 2, p0.02
Test for overall effect Z p0.05
0.8
1
10
Relative risk
Favors new treatment
Favors control
22
Calculate observed minus expected for each trial
Treated
Control
Obs10 Exp12.5
25
Dead
Obs15
o-e -2.5 v 5.5 odds ratio
0.64 Conf.Int. 0.28-1.46 P 0.29
Alive
175
100
100
200
Courtesy of Dr. K. Wheatley
23
Compare only patients in one trial with patients
in the same trial
  • Statistics
  • Obsd expd Variance
  • Trial 1 (o e)1 V1
  • Trial 2 (o e)2 V2
  • Trial 3 (o e)3 V3
  • All Trials (o e)T VT

Courtesy of Dr. K. Wheatley
24
Compare only patients in one trial with patients
in the same trial
  • Statistics
  • Obsd expd Variance
  • Trial 1 - 2.5 5.5
  • Trial 2 - 2.5 5.5
  • ……
  • Trial 10 - 2.5 5.5
  • All Trials - 25.0
    55.0
  • Odds ratio 0.63
  • 95 confidence interval 0.49 to 0.83
  • P

25
Rationale for (quantitative) synthesis of all
available evidence
  • Reduction of bias Comparison of alike with alike
  • Use of randomized comparison whenever possible
  • Always within the same trial
  • Pooling is done by adding trials (not patients)
  • Reduction of imprecision and uncertainty
  • Particularly important when the effects of
    interventions are of small to moderate size (e.g.
    RRR5-10 or 15-25)
  • 20 of reduction in a 50 risk of deathavoidance
    of death in 1 in 10 patients

26
Effect of random errors
  • Function of the size of the trial
  • Subgroup analysis

27
No evidence or no evidence of an effect?
  • Absence of evidence of benefit is not
  • evidence of absence of benefit
  • Truly negative trial (evidence of no effect) vs.
    false-negative trial (no evidence of an effect)

28
Size of randomized trials in myeloma
29
Effect of chance data-dependent subgroup
analysis vs. indirect extrapolation of overall
analysis
  • Data-dependent subgroup analyses may result to
    importantly biased conclusions… and should be
    avoided…
  • Paradoxically, even effects among specific
    categories of patients may be best assessed
    indirectly by approximation of overall treatment
    effect to the patients into a specific category
  • As long as the effect the effect in the specific
    subgroup is not qualitatively different from the
    overall effect

30
Real trial (ISIS-2) EXAGGERATEDLY POSITIVE
mortality effect in a subgroup defined only by
astrological birth sign
Astrological birth sign Atenolol effect on day
0-1 mortality in acute myocardial
infarction Mortality reduction
Statistical comparing
Atenlol significance with control
group (2P) Leo (I.e. born beween 71
23
signs Mean 24 Each 0.1
(NS) (taken separately) Any birth sign 30
10
About PowerShow.com