Outcomes measure reporting bias how much does it influence

1
Outcomes measure reporting bias- how much does it
influence?
Kristian Thorlund and Tahany Awad The Cochrane
Hepato-Biliary Group Copenhagen Trial Unit
Centre for Clinical Intervention
Research Rigshospitalet
2
Definition

• Outcome reporting bias is the selection of a
• subset of the original variables recorded, on
• the basis of the results, for inclusion in
• publication of trials
• Hutton et al. 2000 J Roy Stat Soc.(C) 49 359-70
  

3
Forms of outcome reporting

• Exclusion of outcomes from reports
• Selective choice of data for an outcome
• Selective reporting of analyses using the same
  data
• Selective reporting of subsets of the data
• Selective under-reporting of data

4
Example The Cantekin affair

• Two separate analyses (Mandel 1987 Cantekin
• 1991) of a RCT of antibiotics for ear infection
• in children reached opposite conclusions
• mainly due to different weight given to the
• various outcome measures

Rennie D. JAMA 1991 266 3333-37
5
Existence of reporting bias?

• Two studies provides empirical evidence
• Chan 2004
• Chan 2005

6
Empirical evidence

• Comparison of protocols to 102 published RCTs
• 38 of efficacy and 50 of safety outcomes
• were incompletely reported
• Statistically/clinically significant outcomes
  had
• higher odds ratio of being fully reported
• 62 of trials had at least one primary outcome
• that was changed, introduced or omitted
• Chan et al. JAMA 2004 291 2457-65

7
Empirical evidence

• Outcome reporting bias in 519 RCTs
• - 20 of the outcomes were incompletely reported
• Statistically/clinically significant outcomes
  had
• higher odds ratio of being fully reported
• 62 of trials had at least one primary outcome
• that was changed, introduced or omitted
• Chan et al. BMJ 2005 330 753

8
Estimated effect sizes
Association with of RCT contributing to
outcome 156 randomly selected Cochrane reviews
4222 RCTs, 227 binary, 167 continuous
meta-analyses Furukawa et al. JAMA
2007 297 468-70
9
Compulsory trial registration

• In 2004 a number of major medical journals
  belonging to the ICMJE announced they would no
  longer publish trials that were not registered at
  inception
• Abbasi K. BMJ 329 637-638

10
What is acceptable?

• The ICMJE described acceptable registers
• electronically searchable
• freely accessible to the public
• open to all registrants
• managed by a non-profit organisation
• adhering to a minimum dataset proposed by
• the WHO

11
The Cochrane Handbook recommends

• Compare protocols and published report
• Compare outcomes in the methods section with
  outcomes with results
• Non-significant results mentioned inadequately
  are likely to introduce bias in a meta-analysis
• Construct a matrix with outcomes reported across
  studies
• Contact authors (unreliable)

12
Sum up
Outcome measure reporting bias is among the most
problematic sources of bias Likely to have a
strong impact on the conclusions in systematic
reviews
13
Sum up
Trial registration now compulsory Newer trials
may readily be assessed for outcome reporting
bias (comparison of protocols and published
articles) May be difficult to assess outcome
reporting bias in older trials
14
Thank you!