Title: 1. The Tribulations of Not DoingTrials 2. The Credibility of RCTs in the Eyes of Canadian Internists
11. The Tribulations of Not DoingTrials2. The
Credibility of RCTs in the Eyes of Canadian
Internists U of Alberta Grand Rounds
- David Sackett
- Trout Research Education Center at Irish Lake,
Canada
21.The Tribulations of Not DoingTrials
3Our predecessors werent stupid
- Anyone here think they are smarter than
- Hippocrates ?
- Dr. Benjamin Rush ?
- Dr. Sir William Osler ?
- Leading Professors of Cardiology ?
4Hippocrates
- Founder of Medicine
- Rejected mysticism and argued that disease had a
rational, physical explanation - Yet he insisted Strangury and dysuria are cured
by drinking pure wine and venesection
5Dr. Benjamin Rush
- Declaration of Independence
- Acknowledged Yellow fever expert
- Had 2 med schools named after him
- Yet he was the leading US advocate and teacher of
blood-letting - One of Rushs students followed his advice, let 5
pints out of George Washington in half a day, and
killed him
6Dr. Sir William Osler
- The most influential physician in modern history
- Chairs at McGill, Hopkins, and Oxford
- His Rx for diabetes Opium alone stands the test
of experience as a remedy - His second choice arsenic
7Why did such brilliant physicians say such stupid
things and advocate such lethal treatments?
- Most illnesses get better (so that any Rx they
receive appears efficacious) - Severe illnesses die (so that any lethal Rx they
receive escapes blame) - Todays patients fare better than Historical
Controls, even when neither gets treated at all
8Why did such brilliant physicians say such stupid
things and advocate such lethal treatments?
- Patients who comply with new treatments fare
better (even on placebos !) - The therapeutic recommendations of Experts
carry more weight than they deserve - Dead patients tell no tales
95 Scientific requirements for the evaluation of
any treatment
- Similar groups of treated and untreated patients
- Formed by drawing lots (nowadays by
randomization) - With treatment assignments totally outside the
control of their clinicians (concealed assignment)
105 Scientific requirements for the evaluation of
any treatment
- In drug trials Cared for by clinicians who are
blind to which treatment (active or control)
they are receiving - Avoiding contamination of the control group by
experimental Rx - Avoiding unequal co-intervention in either group
with some other Rx
115 Scientific requirements for the evaluation of
any treatment
- But blinding is impossible in surgical (and some
other) trials - So clinicians agree on all the other elements of
patient care . . . - . . . and monitor their execution throughout the
trial
125 Scientific requirements for the evaluation of
any treatment
- Assessment and adjudication of outcomes by
blind observers
13Note Placebos are not given for their placebo
effect
- They are given in order to blind both patients
and their clinicians, thereby - Preventing contamination
- Preventing co-intervention
- Preventing biased outcome assessment/adjudication
14The first (and most important) of these
requirements had been met centuries before Osler,
and even before Rush!
- Assigning treatments by drawing lots began
- 80 years before Rush was born !
- 190 years before Osler was born !
15Van Helmont in 1662 Before Rush!
- 'Let us take out of the hospitals...200 or 500
poor people, that have fevers, pleurisies. - Let us divide them into halves, let us cast lots,
that one halfe of them may fall to my share, and
the other to yours - I will cure them without bloodletting and
sensible evacuation but you do, as ye know... - We shall see how many funerals both of us shall
have.'
16Balfour in 1854 When Osler was 5!
- During a scarlet fever epidemic, Balfour assigned
treatment by lots to prevent the imputation of
selection - Showed that atropine (belladonna) failed to
protect against scarlet fever. - Reported "but the observation is good, because it
shows how apt we are to be misled by imperfect
observation. . . - . . . Had I given the remedy to all the boys, I
should probably have attributed to it the
cessation of the epidemic"
17Semmelweiss did his natural controlled trial in
1861
- The assignment of women to be delivered by
midwives or doctors was based on the date and
time of their admission. - Midwives washed their hands
- Doctors didnt (and started doing post mortems on
septic mothers) - Mothers 3 times more likely to die if delivered
by doctors
18Loius Pasteur in 1881
- He drew lots to assess the effects of vaccination
against anthrax - He inoculated alternate animals before exposing
them to a virulent culture of the bacterium - And he established that the vaccine prevented
anthrax
19In 1907, when Osler was prescribing opium and
arsenic
- William Fletcher, a Kuala Lumpur doctor, counted
off the lunatics in an asylum (drawing lots) - Even numbered ones got hulled rice
- Odd numbered ones got unhulled rice
- And he established that unhulled rice prevented
beri-beri
(Lancet 1907ii1776-9)
20First modern RCT - 1948
The hero was Austin Bradford Hill
21Hill did NOT randomize in order to generate
similar patient groups !
- He randomized in order to achieve concealment (so
that the clinicians could not know which
treatment the next eligible patient would
receive).
22The RCT has fluorished since
23So have we put our tribulations behind us?
- Not by a long shot
- By the mid 1980s, US cardiologists had been
doing RCTs for 20 years - Their leaders recommended encainide and
flecainide for dangerous arrhythmias following
heart attacks
24The basis for this recommendation
- Patients with these arrhythmias had very high
mortality - These drugs corrected the arrhythmias.
- It was logical to extrapolate this arrhythmia
effect to a life-saving effect - Tens of thousands of patients were treated with
these drugs
25Only later did they do the RCT to find out if
they saved lives
- Consenting patients with these arrhythmias were
randomized to - Encainide or flecainide
- or
- Placebo
- Lots of doctors considered the trial unethical
and refused to take part.
26Monitors kept track
27Monitors stopped the trial
- X reduced mortality by 61 !!!
28Monitors stopped the trial
- EF more than doubled mortality !!!
29In the interim . . . . .
- What about all those patients being treated on
the basis of that earlier extrapolation and
recommendation? - One critic calculated that more Americans were
killed by encainide and flecainide than by the
Viet Cong
30A modern ethical perspective
- When the efficacy of a treatment is unknown, it
is unethical NOT to test it in an RCT - Comment from an earlier version of this talk
(HRT advocates please note) - But wont that make patients guinea pigs,
sacrificed for the benefit of future generations?
31How do the outcomes of patients treated inside
RCTs compare with those of similar patients
treated outside these trials?
- In 53 studies found so far, patients inside
trials fared better (lower mortality, better
outcomes) - Cochrane review found patients receiving
identical Rx inside and outside trials have
similar outcomes
32The Cochrane Collaboration
- A world-wide collaboration between clinicians,
methodologists and patients (14 national centres) - Have tracked down 350,000 trials
- Have published 1500 systematic reviews of the
effects of health care - http//www.cochrane.org/
33- Our patients suffer the tribulations when we fail
to do the trials. - http//www.cochrane.org/
342. The Credibility of RCTs in the Eyes
of Canadian Internists
35A Survey of the Credibility of Identical Trial
Results When Generated by Independent
Investigators or by Drug ManufacturersTrout
Research Education Centre at Irish Lake, Canada
36Three developments are threatening the
credibility of controlled trials when they are
controlled by the drug industry
371. Growing skepticism among Editors of clinical
journals
- We have witnessed the unprecedented simultaneous
intervention by the editors of the major journals
demanding disclosure about sponsorship,
authorship, accountability, and the role of the
sponsor in trial reports.
382. Increased concerns about the bounties
industry pays to participating clinicians
- The moving of RCTs out of academic centres has
increased the exposure of community physicians to
cash payments for every patient they recruit. - The size of these payments has alarmed some
professional associations.
392. Increased concerns about industry bounties
Canadian Medical Association rewrote its Ethics
regulations to discourage bounties
- They cannot exceed lost income.
- They must be approved by the local ethics
committee. - Patients must be told that their doctor is
being paid for putting them in a trial.
403. Studies of the fairness of RCT reports
- (Copenhagen Trial Unit JAMA 2003290921-8)
- They examined 370 RCTs from a random sample of
167 Cochrane reviews - Compared Industry-funded and otherwise-funded
RCTs of the same treatments.
413. Studies of the fairness of RCT reports
- (Copenhagen Trial Unit JAMA 2003290921-8)
- No difference in treatment effects (based on
z-scores for ORs) or adverse events - Industry-funded trials did not generate larger
treatment effects !
42Studies of fairness
- Despite no difference in treatment effects,
industry-funded trials were 5 times as likely to
recommend the experimental therapy. - Might this be due to reliance on surrogate,
secondary, or subgroup analyses? - Might this be due to over-interpretation of
trends in trial results?
43But is the credibility of industry sponsored
trials really falling?
- BMJ studied the effects of authors disclosing
their competing interests on the believability
of their reports. . . (BMJ 20023251391-2) - . . . by sending BMA members a real RCT report
whose authors posed as industry employees or
had no competing interests.
44BMJ Study results
- The industry-employees paper was not only less
believable than the identical paper authored by
individuals with no competing interests. - The industry paper was also judged
- Less interesting
- Less important
- Less relevant
452. Credibility surveys among Canadian
Internists
- Survey-2001
- Survey-2004 (now nearing completion)
46Why Canadian Internists ?
- Tend to consult for, rather than compete with,
GPs and FPs. - They are opinion leaders in their communities.
- As a result, they set the pace for treatment in
their communities.
472001 Survey
- Finlay McAlister in Edmonton
- Roman Jaeschke in Hamilton
- Sharon Straus in Toronto
- Jack McDonald in London
482001 Survey
- Among grab (convenience) samples (n132) of
volunteer internists . . . - who were attending Grand Rounds or Clinical
Conferences . . . - in Edmonton, Hamilton, Toronto or London
492001 Survey
- Presented with two methodologically identical
randomized trials, - that asked identical questions (new drug for
unstable angina pectoris), - and generated identical, positive results (NNT to
prevent one more non-fatal MI or death 31), - and were both published in a general journal with
a high impact factor.
50But these identical trials originated from
different sources
- One was an Industry-Based, Industry-Initiated
Trial (abbrev Industry) - The other was an identically designed
University-based, Investigator-Initiated Trial
(abbrev University)
51In the Industry-Based, Industry-Initiated Trial
(Industry)
- All field data were sent to the Industry.
- The interim and primary analyses were carried out
by Industry. - Industry sponsors sat on its TMC/DSMB.
52In the University-based, Investigator-Initiated
Trial (University)
- Field data (except adverse events) sent only to
the University - Interim and primary analyses carried out by the
University only. - No Industry sponsors were on its TMC/DSMB.
53Internists were asked to rate the credibility of
the results of each trial
- 1 Not at all credible2 Only low
credibility3 Questionable credibility4
Moderate credibility5 High credibility6
Extremely high credibility
542001 SURVEY
Univ.
Industry
Credibility
55 2001 Survey
- 90 of clinicians found the Industry trial
results to be less credible than those of the
identical University trial.
56 2001 Survey
- The Industry trial results were10 times as
likely to be judged not at all credible, of
only low credibility, or of questionable
credibility as those of the University trial
(61 vs. 6).
57 2001 Survey
- The University trial results were 10 times as
likely to be judged of high credibility or of
extremely high credibility as those of the
Industry trial (86 vs. 8.6) - P-values for these differences range from 2-4 to
1.5-17
58But is the 2001 Survey credible?
- Are its grab samples representative of Canadian
internists? (maybe cynics were
over-represented) - Might asking each participant to consider both
trials constitute juxtaposition bias and
exaggerate true differences in credibility?
59So weve conducted another survey in 2004.
- For representativeness it is being performed
on a random sample of 150 Canadian Internists
(CSIM). - To avoid juxtaposition bias internists have
been randomized to receive 1 of 3 questionnaires.
602004 Survey To eliminate juxtaposition bias
- Group 1 Just the Industry Example
- Group 2 Just the University Example
- Group 3 Both examples
- (Group 3 also provides a replication of the 2001
Survey, to test for that surveys
representativness)
61Response rate for the 2004 Survey 76
- Juxtaposition bias was not found
- Internists credibility scores for single
examples (Groups 1 2) are closely similar to
their credibility scores when both examples are
provided (group 3).
622004 Survey - preliminary
- The earlier 2001 Survey did not appear to
over-represent cynics. - If anything, the credibility of the Industry
trial in 2004 is lower than it was in 2001.
632001 SURVEY reprise
Univ.
Industry
Credibility
642004 SURVEY
Univ.
Industry
Credibility
652004 Survey
- The of Industry trial results judged not at
all credible, of only low credibility, or of
questionable credibility rose from 60 in 2001
to 80 in 2004.
662004 Survey
- The Industry trial increased from 10x to 20x as
likely to be judged not at all credible, of
only low credibility, or of questionable
credibility as the results of the University
trial (now 81 vs. 4).
67Conclusion The credibility of a trial result, in
the eyes of Canadian internists,
depends on whose eyes were on the emerging trial
data
68(No Transcript)