Are We Living in an Evidence Biased World

1
Are We Living in an Evidence B(i)ased World?

• Joel Lexchin MD
• School of Health Policy and Management
• York University
• Department of Emergency Medicine
• University Health Network

2
My Disclosure

• 1995 I had a piece of
• stale pizza at a drug
• company lunch

3
Outline

• Funding of medical research
• Biases in clinical research
• Direction of research
• Sharing of information
• Outcome of clinical research
• Biases in medical journals
• Biases in guidelines
• Biases in CME

4
Funding of Medical Research
5
Funding for Clinical Research, United States, 2002
Billions
CenterWatch 2003
6
How Close is the Relationship?

• 50,000 U.S. clinical investigators received
  funding for one or more trial in 2002
• 1.5 billion from industry to academia annually
• 25 of academic investigators receive industry
  money 1/3 have personal financial ties with
  industry
• AMA, Council on Scientific Affairs, 2004

7
Health RD in Canada, 2004
Spending in millions of dollars, total 5.75
billion
Statistics Canada. Science Statistics. Catalogue
88-001-XIE
8
Clinical RD in Canada, 2004
9
Decline of Support for Non-Commercial Trials, UK
Chalmers et al. BMJ 2003 3271017-20
10
Biases in Medical Research
11
Source of Support and Choice of Research Topic
Blumenthal et al. Science 19862321361-6 Blumenth
al et al. NEJM 19963351734-9
12
Drug Companies Directing Research Female Sexual
Dysfunction
Moynihan. BMJ 200332645-7
13
Drug Companies Directing Research OA of the Knee
Tallon et al. Lancet 20003552037-40
14
Drug Companies Directing Research OA of the Knee

• Rheumatologist Focus Group
• NSAIDs over-researched especially through
  commercial sponsored drug trials
• Patient Focus Group
• Favoured conservative treatments such as
  physiotherapy
• More research on education and self-help
• GP Focus Group
• Research on surgical success rates and
  conservative treatments
• Oral drugs over-researched
• Physiotherapist Focus Group
• Absence of research on physiotherapy exercise
• Dominance of drug trials

15
Sharing of Information
16
Keeping Research Results Confidential
Out of 181 firms 58 reported company
typically required academic investigators to
keep information confidential for gt6 months in
order to file patent application NIH guideline
30-60 days reasonable period to delay
Blumenthal et al. NEJM 1996334 368-73
17
Sharing of Research Results

• Faculty with industrial support
• 11.1 refused to share research results or
  biomaterials
• Faculty without industrial support
• 5.8 refused to share research results or
  biomaterials
• Blumenthal et al. NEJM 19963351734-9

18
Outcome of Industry Funded Research
19
Outcome of Industry Funded Research -I

• Meta-analysis
• studies done over 2
• decades
• wide range of
• disease states, drugs and
• drug classes
• pharmacoeconomic
• studies, clinical trials,
• meta-analyses

Lexchin et al. BMJ 2003326 1167-70
20
Outcome of Industry Funded Research - II

• 370 drug trials from 25 Cochrane reviews
• Trials funded by for-profit organizations
  significantly more likely to recommend
  experimental drug as treatment of choice compared
  to trials funded by nonprofit organizations OR
  5.3 (95 CI 2.0, 14.4) (After adjusting for
  treatment effect and double blinding)
• Als-Nielsen et al. JAMA 2003290921-8

21
Outcome of Industry Funded Research - III

• Moncrieff. Br J Psych 2003163161-6
• Industry funded clinical trials on clozapine
  showed greater positive effect than trials with
  other sources of funding
• Baker et al. Br J Psych 2003183498-506
• Pharmacoeconomic studies of antidepressants
  revealed clear associations of study sponsorship
  with quantitative outcome
• Bhandari et al. CMAJ 2004170477-80
• 158 RCTs of drug products from 5 high-impact
  general medical journals
• Industry trials more likely to be favourable to
  product OR 1.6 (95 CI 1.1, 2.8)
• Procyshyn et al. Can J Psych 200449601-606
• 372 clinical trials on 3 atypical antipsychotics
  124 industry sponsored
• No industry sponsored trial reported negative
  results (findings favouring comparator serious
  concerns about safety or efficacy recommended
  comparator)

22
Possible Explanations

• Use of inappropriate comparators or wrong doses
  of comparators (e.g., COMET study)
• Non-publication of trials with negative results
• Misinterpretation of results due to financial
  conflicts-of-interest
• Violations of uncertainty principle (sponsoring
  trials likely to produce positive results)
• Discontinuing trials likely to produce negative
  results
• Emphasis on secondary outcomes or subgroup
  analyses or using per-protocol analysis
• Run-in bias
• Only partial publication of results (e.g., CLASS
  study)

23
Acceptability of Restrictions in Clinical Trial
Agreements - 107 US Medical Schools
Mello et al. NEJM 20053522202-10
24
Whats It Like in the Community Setting?
Corporate funding for clinical research is
increasingly moving into the community S
teinbrook. NEJM 20053522160- 2
25
Biases in Medical Journals
26
Publication of Research Results

• Suppression of publication
• Biases in published work

27
Suppression of Publication

• Have you personally conducted any research
• at your university the results of which are the
• property of the sponsor and cannot be
• published without their consent?
• Faculty with university-industry relationships
  24
• Faculty without university-industry
  relationships 5
• Blumenthal et al. Science 19862321361-6

28
One Possible Reason for Nonpublication

• Trade secrets
• 14.5 of faculty with industrial support produced
  research
• 4.7 of faculty without industrial support
• Blumenthal et al. NEJM 19963351734-9

29
Delaying Publication

• Publication delayed for gt 6 months
• Faculty with academic-industry research
  relationships 27
• Faculty without academic-industry research
  relationships 17
• Blumenthal et al. JAMA 19972771224-8

30
Delaying Publication Publication Source

• Industry-funded research takes longer to be
  published than research with other sources of
  sponsorship
• Industry-funded research more likely to be
  published in symposium proceedings
• Lexchin et al. BMJ 2003 3261167-70

31
Prepublication Review

• One-third of faculty who received gifts
• (biomaterials, research equipment,
• payment for trips to meetings, support for
• students, other research-related items)
• thought donors expected prepublication
• review of articles
• Unknown if donor actually did expect
  prepublication review
• Campbell et al. JAMA 1998279995-9

32
FINANCIAL RELATIONSHIPS BETWEEN AUTHORS AND
COMPANIES

• Conclusions of authors
• about value of calcium
• channel blockers as a
• function of financial
• relationship with
• company making
• product (p value for trend
• lt0.001)

Stelfox NEJM 1998338101-6
33
Declaration of Conflict of Interest
Higher the score, the more strongly the
treatment is recommended
Kjaergard et al. BMJ 2002 325249
34
Declaration of Conflict of Interest
Friedman, JGIM 20041951-6
35
Publication Bias in Industry Funded Trials

• Publication of trials submitted to Swedish
• regulatory authorities
• Studies showing significant differences between
  efficacy of drug and placebo 3 x more likely to
  appear as stand alone publications
• Authors names frequently different despite same
  data being published
• Intention to treat and per protocol analyses used
  in submissions to regulatory authorities but only
  24 of stand alone journal publications used
  intention to treat analysis
• High frequency of duplicate publication due to
  inclusion of different subsets of studies in
  several pooled publications
• Melander et al. BMJ 20033261171-3

36
Conclusions Not Supported by Data

• Manufacturer supported trials of NSAIDs in
  arthritis
• Several studies reporting superior efficacy used
  multiple outcomes without correction of threshold
  value for statistical significance
• In 10 of 22 studies concluding that
  manufacturer-associated drug was less toxic no
  supporting statistical test
• Rochon et al. Archives of Internal Medicine
  1994154157-63

37
Ghostwriting - Articles on Paroxetine
Healy et al. British Journal of Psychiatry
200318322-7
38
Biases in Clinical Practice Guidelines
39
Conflicts of Interest in Clinical Practice
Guidelines
Over 200 guidelines examined only 90 contained
details about individual conflicts of interest
of those only 31 free of industry influence
Nature 20054371070-1
40
Biases in CME
41
Income and Expenses, CME in US, 1998-2003
42
CME Funding, U.S. 2003
25th percentile
50th percentile
75th percentile
Total schools
43
Analysis of CME Funded by Academia and Industry,
2000-01
Katz et al. J Contin Ed Hlth Professions
20022243-54
44
Effects of CME On Prescribing Patterns
Course 1 Course 2
Bowman. J Cont Educ Hlth Profess 1988813-20
45
Conclusions

• Biases mean that evidence only exists for certain
  types of therapy
• Biases are in favour of commercial interests
• Biases (real or perceived) exist at many levels
• Choices research topic, CME topic
• Outcomes clinical research
• Content journal articles, guidelines, CME
• If unrecognized and unchallenged biases can
  result in suboptimal medicine