A mid flight correction: setting priorities

1
A mid flight correction setting priorities

• David Moher
• Canadian Cochrane Centre and Network
• Cochrane Collaborations Bias Methods Group
• University of Ottawas Evidence based Practice
  Center
• CIHR and AHRQ

Acknowledge members of the Chalmers Research Group
2
Outline of talk

• Quality of reports of systematic reviews
• Systematic review non-registration
• Systematic review non-publication
• Systematic review selective outcome reporting
• Updating systematic reviews

3
Epidemiology and reporting characteristics of
systematic reviews

• Searched MEDLINE to identify systematic reviews
• If the authors stated objective was to summarize
  evidence from multiple studies and the article
  described explicit methods, regardless of the
  details provided
• Included reports of Cochrane reviews
• November 2004
• Extracted information about epidemiology and
  reporting characteristics

Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman
DG. Epidemiology and reporting characteristics of
systematic reviews. PLoS Medicine 2007
(3)e78.doi10.1371/journal.pmed.0040078
4
Epidemiology and reporting characteristics of
systematic reviews

• Identified 300 reviews
• extrapolate to 2500 new English language
  systematic reviews published annually
• 20 are Cochrane 80 or non-Cochrane reviews
• Reviews are getting bigger More informative
• median 16 (iqi 7-30) non-cochrane 23
• number of participants median 1112 (iqi
  322-3750)

5
Epidemiology and reporting characteristics of
systematic reviews

• Most (91) reports of systematic reviews are
  published in small specialty journals
• Dont sample the usual 5 journals!
• Some improvements over time
• reports of use of risk of bias assessments
  (66.8)

6
Epidemiology and reporting characteristics of
systematic reviews

• Reports of Cochrane reviews are appreciably
  better than non-Cochrane ones
• Reports of non-Cochrane reviews are particularly
  troubling
• most reviews (17.7) are not reports of updates
• some suggestion of outcome reporting bias
• sub-optimal reporting of publication bias
  assessment (23.1)
• No report is registered in the conventional sense
• none had a formal registration number

7
More recent data

• Similar study
• Looking at systematic reviews of RCTs
• Sample (2000-2005)
• 35 published in 2005
• 51 published in 2004 and 2005
• Similar results
• Cochrane reviews better reported than
  non-Cochrane reviews
• Still room for improvement in reporting all
  reviews

Wen J, Ren Y, Wang L, Li Y, Liu Y, Zhou M, Liu P,
Ye L, Li Y, Tian W. The reporting quality of
meta-analyes improves a random sampling study.
Journal of Clinical Epidemiology 200861770-775
8

• Do not have data on how well evidence reports and
  comparative effectiveness reviews perform
  compared to Cochrane and other groups conducting
  systematic reviews

9
Methodological quality of systematic reviews of
animal studies a survey of reviews of basic
research

• Electronic search (1996-2004)
• medline, embase
• Contact experts searched reference lists
• Searched for systematic reviews
• terminology ambiguity
• Screened searched results
• Included 30 animal studies administered to live
  animals, laboratory measures
• Compared 30 animal studies to 45 randomly
  selected bench studies

Mignini LE, Khan KS. Methodological quality of
systematic reviews of animal studies a survey of
basic research. BMC Medical Research Methodology.
2006
10
Methodological quality of systematic reviews of
animal studies a survey of reviews of basic
research
11
Push (priority setting) at the expense of quality

• Little uptake, and adherence of reporting
  guidelines for systematic reviews
• PRISMA
• Inconsistent agreement (terminology) on what a
  systematic review is!
• Pushing reviews out the door at what expense?

12
Setting priorities

• NLM index term systematic reviews
• Fund quality feedback loop
• Funders need to work with publishers and editors
  to improve uptake and adherence of guidelines for
  reporting systematic reviews
• It improves quality
• Need to develop template for conducting
  systematic reviews
• It improves quality

13
Accurate and transparent reporting is like
turning on the light before you clean up a room
it doesnt clean it for you but does tell you
where the problems are

• (Frank Davidoff, editor of Annals of Internal
  Medicine, 2000).

14
Reporting guidelines

• Reporting Guideline specify minimum information
  necessary for clear accounting of research
  methodology and findings
• A good return on funder investment
• Number of reporting guidelines
• n 80
• As yet most guidelines have had limited impact
• Passive dissemination through publication only -
  not widely known
• Complience not required by journals

15
EQUATOR Enhancing the QUAlity and Transparency
Of health Research

• EQUATOR Network (set up in 2006) grew out of the
  work of CONSORT and other guidelines groups
• Umbrella organization promoting transparent and
  accurate reporting of health research through the
  implementation of reporting guidelines
• Key stakeholders
• Developers of reporting guidelines
• Editors peer reviewers of general and specialty
  journals
• Researchers (authors), medical writers,
• Research funders
• Everyone interested in improving the quality of
  research publications and of research itself

16
(No Transcript)
17
EQUATOR Core program

• Main objectives for the next 5 years
• Provide resources and training enabling the
  improvement of health research reporting
• Monitor progress in the improvement of health
  research reporting

www.equator-network.org
18
Improving quality

• Shared responsibility
• Moral obligation
• All stakeholders need to develop and fund
  feedback loop for quality and systematic reviews

19
Registering systematic reviews

• Moher and colleagues call for the registration
  of systematic reviews in the same manner as is
  now required for clinical trials it would then,
  at least, be possible to track those that are
  performed.

• reduce redundant research and publications
• reduce systematic review non-publication bias
• minimize selective outcome reporting
• Improve priority setting activities

The PLoS Medicine Editors. Many reviews are
systematic but some are more transparent and
completely reported than others. PLoS Medicine
2007
20

• Compliance with QUOROM and quality of reporting
  of overlapping meta-analyses on the role of
  acetylcysteine in the prevention of contrast
  associated nephropathy case study
• 10 systematic reviews published between 2003 and
  2005

• Acetylcysteine in the prevention of
  contrast-induced nephropathy a case study of the
  pitfalls in the evolution of evidence
• 11 meta-analyses

Biondi-Zoccai GGL, Lotrionte M, Abbate A, Teta L,
Remigi E, Burzotta F, Valgimigli M, Romagnoli E,
Crea F, Agostoni P. Compliance with QUOROM and
quality of reporting of overlapping meta-analyses
on the role of acetylcysteine in the prevention
of contrast associated nephropathy case study.
BMJ 2006 332(7535)202-9
Bagshaw SM, McAlister FA, Manns BJ, Ghali WA.
Acetylcysteine in the prevention of
contrast-induced nephropathy a case study of the
pitfalls in the evolution of evidence. Arch
Intern Med 2006166161-166
21
Registration and priority setting

• Need to know whats out there in order to set
  priorities

22
It is surely a great criticism of our profession
that we have not organised a critical summary, by
speciality and subspeciality, adapted
periodically, of all relevant randomised
controlled trials
It is surely a great criticism that we have not
organised a critical summary, by speciality and
subspeciality, adapted periodically, of all
relevant systematic reviews
23
Selective outcome bias

• selective reporting of outcomes
• typically statistically positive
• selected by investigators (post hoc)

24
Outcomes reporting bias

• Methods
• compared the contents of 102 trial protocols,
  approved by the scientific-ethics committees for
  copenhagen and frederiksberg, denmark, during
  1994 and 1995, with 122 subsequent publications

Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC,
Altman DG. Empirical evidence for selective
reporting of outcomes in randomized trials
comparison of protocols to published articles.
JAMA 20042912457-2465.
25
Some salient results

• Nearly two-thirds had a change in at least one
  primary outcome between the protocol and
  publication
• Statistically significant outcomes had a higher
  likelihood of being reported compared to
  non-significant ones

26
Does selective outcome reporting exist within
systematic reviews

• Back to the 300 reports of systematic reviews
• 80 of published literature is non-Cochrane
• 161 systematic reviews with meta-analysis
• Statistical significance of primary outcome (if
  stated)
• Cochrane 75 reported primary outcomes 14.4
  statistically significant
• Non-Cochrane 25 reported primary outcome 50
  statistically significance

Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman
DG. Epidemiology and reporting characteristics of
systematic reviews. PLoS Medicine 2007
4(3)e78.doi10.1371/journal.pmed.0040078
27
Does systematic review non-publication bias exist?

• Surveyed 625 authors of systematic reviews
• random sample of systematic reviews published in
  2005
• 1st author or corresponding author
• Developed 26-item survey to tap into
• publication process
• identification of unpublished reviews
• Pilot tested
• Completed using Dillman method
• Completed on Survey Monkey

1Tricco AC, Pham B, Brehaut J, Tetroe J, Cappelli
M, Hopewell S, Lavis J, Berlin JA, Moher D.
Systematic review publication bias an
international survey
28
Does systematic review non-publication bias exist?

• 64 survey response rate
• 93 survey completion rate
• 6.3 reported that they were never the lead on
  the review!
• Respondents reported a total of 1385 published
  systematic reviews
• median 2
• 52 respondents reported a total of 200
  unpublished systematic reviews
• median 2
• Most unpublished reviews (73) completed after
  2005

29
What is updating

• A discrete event aiming to search for and
  identify new evidence to incorporate into a
  previously completed systematic review with new
  evidence taken to mean any evidence not included
  in the previously completed review irrespective
  of its chronological appearance in the
  literature
• Most international organizations agree with this
  definition

Moher D, Tsertsvadze A. When is an update an
update. Lancet 2006 Garritty C, Tsertsvadze A,
Tricco AC, Sampson M, Moher D. Updating
systematic reviews an international survey
30
Why is updating important

• Keeps systematic reviews informative
• What happens if reviews are out-of-date
• Impact on clinical practice
• Policy recommendations
• Future primary research
• Funding agencies
• Most reports of systematic reviews do not appear
  to be updates

31
Updating systematic reviews

• To estimate the average time to changes in
  evidence sufficiently important to warrant
  updating systematic reviews
• To determine the performance characteristics of
  various surveillance protocols to identify
  important new evidence
• To assess the utility of rates and patterns of
  growth for evidence within clinical areas as
  predictors of updating needs
• To establish typical timeframes for the
  production and publication of systematic reviews
  in order to assess the extent to which they
  impact shelf-life
• Identify the updating experiences of healthcare
  organizations that fund and/or conduct systematic
  reviews

Shojania K, Sampson M, Ji J, Ansari M, Garritty
C, ORourke K, Rader, Moher D. Updating
Systematic Reviews. Technical Review No. 16.
(Prepared by the University of Ottawa
Evidence-based Practice Center under Contract No.
290-02-0017). AHRQ Publication No 07-0087
Rockville, MD Agency for Healthcare Research and
Quality. September 2007
32
Updating systematic reviews

• To estimate the average time to changes in
  evidence sufficiently important to warrant
  updating systematic reviews
• To determine the performance characteristics of
  various surveillance protocols to identify
  important new evidence
• To assess the utility of rates and patterns of
  growth for evidence within clinical areas as
  predictors of updating needs
• To establish typical timeframes for the
  production and publication of systematic reviews
  in order to assess the extent to which they
  impact shelf-life
• Identify the updating experiences of healthcare
  organizations that fund and/or conduct systematic
  reviews

Shojania K, Sampson M, Ji J, Ansari MT, Doucette
S, Moher D. How quickly do systematic reviews go
out of date A survival analysis. Annals of
Internal Medicine 2007147224-233.
33
How quickly do systematic reviews become out of
date a survival analysis

• Systematic reviews evaluating benefit or harm
• drug, device, or procedure
• published between 1995 and 2005
• included at least 1 meta-analysis

34
Signals for updating

• Quantitative signal to update
• change in statistical significance
• for primary outcome or any mortality outcome a
  relative change in effect estimate of at least
  50
• Qualitative signal to update
• substantial difference in characterization of
  effectiveness
• new information about harms
• superior alternative treatments
• important caveats about previous findings that
  would influence clinical decision making

35
Structured Search Protocols

• Identification of new SRs on the same topic
• Clinical Queries filters in PubMed
• Applied Related Articles function in PubMed to 3
  largest and 3 most recent trials in the original
  review
• Using a citing references search engine
  (Scopus) to identify new RCTs that cited original
  review
• Relevant sections of Clinical Evidence, UpToDate

36
Results

• Screened 325 reports to obtain cohort of 100
• 13 (median) included studies and 2663
  participants
• 85 evaluated drug therapies
• Five most common clinical areas
• cardiovascular medicine, 20
• gastroenterology, 13
• neurology, 11
• infectious diseases, 9
• respiratory system, 9

37
Results

• identified at least one new eligible trial for 85
  of the reviews
• 4 new trials (median increase in evidence base
  25)
• 1160 participants (median increase in evidence
  base 47)

38
Results

• A quantitative or qualitative signal occurred in
  57 of the reviews
• median event-free survival 5.5 years (95,
  confidence interval 4.6, 7.6)
• 23 of reviews needed updating within 2 years
• 15 within 1 year
• 7 already needed updating at time of publication
• Quantitative signal
• occurred in 20 of systematic reviews
• Qualitative signal
• occurred in 54 of systematic reviews

39
International survey

• Describe the updating practices and policies of
  agencies that sponsor or conduct systematic
  reviews

Garritty C, Tsertsvadze A, Tricco AC, Sampson M,
Moher D. Updating systematic reviews an
international survey
40
International survey

• Stratified purposeful sampling approach
• Key international groups commonly involved in
  undertaking/funding reviews
• 195 organizations
• 48 questions
• Pilot tested
• Administered through Survey Monkey

41

• 65 response rate from 26 countries
• 79 considered importance of updating high to
  very high
• 57 have formal policy
• although only 29 have written policy
• About half (51) respondents reported that more
  than 50 of their systematic reviews are
  estimated to be out-of-date

42
Setting priorities

• Before embarking on setting priorities for new
  reviews ensure that existing reviews are kept
  up-to-date
• Encourage journals to have an explicit policy on
  publishing systematic review updates
• Centralize aspects of updating
• Searching the literature

43
(No Transcript)