1. The Tribulations of Not DoingTrials 2. The Credibility of RCTs in the Eyes of Canadian Internists

1
1. The Tribulations of Not DoingTrials2. The
Credibility of RCTs in the Eyes of Canadian
Internists U of Alberta Grand Rounds

• David Sackett
• Trout Research Education Center at Irish Lake,
  Canada

2
1.The Tribulations of Not DoingTrials
3
Our predecessors werent stupid

• Anyone here think they are smarter than
• Hippocrates ?
• Dr. Benjamin Rush ?
• Dr. Sir William Osler ?
• Leading Professors of Cardiology ?

4
Hippocrates

• Founder of Medicine
• Rejected mysticism and argued that disease had a
  rational, physical explanation
• Yet he insisted Strangury and dysuria are cured
  by drinking pure wine and venesection

5
Dr. Benjamin Rush

• Declaration of Independence
• Acknowledged Yellow fever expert
• Had 2 med schools named after him
• Yet he was the leading US advocate and teacher of
  blood-letting
• One of Rushs students followed his advice, let 5
  pints out of George Washington in half a day, and
  killed him

6
Dr. Sir William Osler

• The most influential physician in modern history
• Chairs at McGill, Hopkins, and Oxford
• His Rx for diabetes Opium alone stands the test
  of experience as a remedy
• His second choice arsenic

7
Why did such brilliant physicians say such stupid
things and advocate such lethal treatments?

• Most illnesses get better (so that any Rx they
  receive appears efficacious)
• Severe illnesses die (so that any lethal Rx they
  receive escapes blame)
• Todays patients fare better than Historical
  Controls, even when neither gets treated at all

8
Why did such brilliant physicians say such stupid
things and advocate such lethal treatments?

• Patients who comply with new treatments fare
  better (even on placebos !)
• The therapeutic recommendations of Experts
  carry more weight than they deserve
• Dead patients tell no tales

9
5 Scientific requirements for the evaluation of
any treatment

• Similar groups of treated and untreated patients
• Formed by drawing lots (nowadays by
  randomization)
• With treatment assignments totally outside the
  control of their clinicians (concealed assignment)

10
5 Scientific requirements for the evaluation of
any treatment

• In drug trials Cared for by clinicians who are
  blind to which treatment (active or control)
  they are receiving
• Avoiding contamination of the control group by
  experimental Rx
• Avoiding unequal co-intervention in either group
  with some other Rx

11
5 Scientific requirements for the evaluation of
any treatment

• But blinding is impossible in surgical (and some
  other) trials
• So clinicians agree on all the other elements of
  patient care . . .
• . . . and monitor their execution throughout the
  trial

12
5 Scientific requirements for the evaluation of
any treatment

• Assessment and adjudication of outcomes by
  blind observers

13
Note Placebos are not given for their placebo
effect

• They are given in order to blind both patients
  and their clinicians, thereby
• Preventing contamination
• Preventing co-intervention
• Preventing biased outcome assessment/adjudication

14
The first (and most important) of these
requirements had been met centuries before Osler,
and even before Rush!

• Assigning treatments by drawing lots began
• 80 years before Rush was born !
• 190 years before Osler was born !

15
Van Helmont in 1662 Before Rush!

• 'Let us take out of the hospitals...200 or 500
  poor people, that have fevers, pleurisies.
• Let us divide them into halves, let us cast lots,
  that one halfe of them may fall to my share, and
  the other to yours
• I will cure them without bloodletting and
  sensible evacuation but you do, as ye know...
• We shall see how many funerals both of us shall
  have.'

16
Balfour in 1854 When Osler was 5!

• During a scarlet fever epidemic, Balfour assigned
  treatment by lots to prevent the imputation of
  selection
• Showed that atropine (belladonna) failed to
  protect against scarlet fever.
• Reported "but the observation is good, because it
  shows how apt we are to be misled by imperfect
  observation. . .
• . . . Had I given the remedy to all the boys, I
  should probably have attributed to it the
  cessation of the epidemic"

17
Semmelweiss did his natural controlled trial in
1861

• The assignment of women to be delivered by
  midwives or doctors was based on the date and
  time of their admission.
• Midwives washed their hands
• Doctors didnt (and started doing post mortems on
  septic mothers)
• Mothers 3 times more likely to die if delivered
  by doctors

18
Loius Pasteur in 1881

• He drew lots to assess the effects of vaccination
  against anthrax
• He inoculated alternate animals before exposing
  them to a virulent culture of the bacterium
• And he established that the vaccine prevented
  anthrax

19
In 1907, when Osler was prescribing opium and
arsenic

• William Fletcher, a Kuala Lumpur doctor, counted
  off the lunatics in an asylum (drawing lots)
• Even numbered ones got hulled rice
• Odd numbered ones got unhulled rice
• And he established that unhulled rice prevented
  beri-beri
  (Lancet 1907ii1776-9)

20
First modern RCT - 1948
The hero was Austin Bradford Hill
21
Hill did NOT randomize in order to generate
similar patient groups !

• He randomized in order to achieve concealment (so
  that the clinicians could not know which
  treatment the next eligible patient would
  receive).

22
The RCT has fluorished since
23
So have we put our tribulations behind us?

• Not by a long shot
• By the mid 1980s, US cardiologists had been
  doing RCTs for 20 years
• Their leaders recommended encainide and
  flecainide for dangerous arrhythmias following
  heart attacks

24
The basis for this recommendation

• Patients with these arrhythmias had very high
  mortality
• These drugs corrected the arrhythmias.
• It was logical to extrapolate this arrhythmia
  effect to a life-saving effect
• Tens of thousands of patients were treated with
  these drugs

25
Only later did they do the RCT to find out if
they saved lives

• Consenting patients with these arrhythmias were
  randomized to
• Encainide or flecainide
• or
• Placebo
• Lots of doctors considered the trial unethical
  and refused to take part.

26
Monitors kept track
27
Monitors stopped the trial

• X reduced mortality by 61 !!!

28
Monitors stopped the trial

• EF more than doubled mortality !!!

29
In the interim . . . . .

• What about all those patients being treated on
  the basis of that earlier extrapolation and
  recommendation?
• One critic calculated that more Americans were
  killed by encainide and flecainide than by the
  Viet Cong

30
A modern ethical perspective

• When the efficacy of a treatment is unknown, it
  is unethical NOT to test it in an RCT
• Comment from an earlier version of this talk
  (HRT advocates please note)
• But wont that make patients guinea pigs,
  sacrificed for the benefit of future generations?

31
How do the outcomes of patients treated inside
RCTs compare with those of similar patients
treated outside these trials?

• In 53 studies found so far, patients inside
  trials fared better (lower mortality, better
  outcomes)
• Cochrane review found patients receiving
  identical Rx inside and outside trials have
  similar outcomes

32
The Cochrane Collaboration

• A world-wide collaboration between clinicians,
  methodologists and patients (14 national centres)
• Have tracked down 350,000 trials
• Have published 1500 systematic reviews of the
  effects of health care
• http//www.cochrane.org/

33

• Our patients suffer the tribulations when we fail
  to do the trials.
• http//www.cochrane.org/

34
2. The Credibility of RCTs in the Eyes
of Canadian Internists
35
A Survey of the Credibility of Identical Trial
Results When Generated by Independent
Investigators or by Drug ManufacturersTrout
Research Education Centre at Irish Lake, Canada
36
Three developments are threatening the
credibility of controlled trials when they are
controlled by the drug industry
37
1. Growing skepticism among Editors of clinical
journals

• We have witnessed the unprecedented simultaneous
  intervention by the editors of the major journals
  demanding disclosure about sponsorship,
  authorship, accountability, and the role of the
  sponsor in trial reports.

38
2. Increased concerns about the bounties
industry pays to participating clinicians

• The moving of RCTs out of academic centres has
  increased the exposure of community physicians to
  cash payments for every patient they recruit.
• The size of these payments has alarmed some
  professional associations.

39
2. Increased concerns about industry bounties
Canadian Medical Association rewrote its Ethics
regulations to discourage bounties

• They cannot exceed lost income.
• They must be approved by the local ethics
  committee.
• Patients must be told that their doctor is
  being paid for putting them in a trial.

40
3. Studies of the fairness of RCT reports

• (Copenhagen Trial Unit JAMA 2003290921-8)
• They examined 370 RCTs from a random sample of
  167 Cochrane reviews
• Compared Industry-funded and otherwise-funded
  RCTs of the same treatments.

41
3. Studies of the fairness of RCT reports

• (Copenhagen Trial Unit JAMA 2003290921-8)
• No difference in treatment effects (based on
  z-scores for ORs) or adverse events
• Industry-funded trials did not generate larger
  treatment effects !

42
Studies of fairness

• Despite no difference in treatment effects,
  industry-funded trials were 5 times as likely to
  recommend the experimental therapy.
• Might this be due to reliance on surrogate,
  secondary, or subgroup analyses?
• Might this be due to over-interpretation of
  trends in trial results?

43
But is the credibility of industry sponsored
trials really falling?

• BMJ studied the effects of authors disclosing
  their competing interests on the believability
  of their reports. . . (BMJ 20023251391-2)
• . . . by sending BMA members a real RCT report
  whose authors posed as industry employees or
  had no competing interests.

44
BMJ Study results

• The industry-employees paper was not only less
  believable than the identical paper authored by
  individuals with no competing interests.
• The industry paper was also judged
• Less interesting
• Less important
• Less relevant

45
2. Credibility surveys among Canadian
Internists

• Survey-2001
• Survey-2004 (now nearing completion)

46
Why Canadian Internists ?

• Tend to consult for, rather than compete with,
  GPs and FPs.
• They are opinion leaders in their communities.
• As a result, they set the pace for treatment in
  their communities.

47
2001 Survey

• Finlay McAlister in Edmonton
• Roman Jaeschke in Hamilton
• Sharon Straus in Toronto
• Jack McDonald in London

48
2001 Survey

• Among grab (convenience) samples (n132) of
  volunteer internists . . .
• who were attending Grand Rounds or Clinical
  Conferences . . .
• in Edmonton, Hamilton, Toronto or London

49
2001 Survey

• Presented with two methodologically identical
  randomized trials,
• that asked identical questions (new drug for
  unstable angina pectoris),
• and generated identical, positive results (NNT to
  prevent one more non-fatal MI or death 31),
• and were both published in a general journal with
  a high impact factor.

50
But these identical trials originated from
different sources

• One was an Industry-Based, Industry-Initiated
  Trial (abbrev Industry)
• The other was an identically designed
  University-based, Investigator-Initiated Trial
  (abbrev University)

51
In the Industry-Based, Industry-Initiated Trial
(Industry)

• All field data were sent to the Industry.
• The interim and primary analyses were carried out
  by Industry.
• Industry sponsors sat on its TMC/DSMB.

52
In the University-based, Investigator-Initiated
Trial (University)

• Field data (except adverse events) sent only to
  the University
• Interim and primary analyses carried out by the
  University only.
• No Industry sponsors were on its TMC/DSMB.

53
Internists were asked to rate the credibility of
the results of each trial

• 1 Not at all credible2 Only low
  credibility3 Questionable credibility4
  Moderate credibility5 High credibility6
  Extremely high credibility

54
2001 SURVEY
Univ.

Industry
Credibility
55
2001 Survey

• 90 of clinicians found the Industry trial
  results to be less credible than those of the
  identical University trial.

56
2001 Survey

• The Industry trial results were10 times as
  likely to be judged not at all credible, of
  only low credibility, or of questionable
  credibility as those of the University trial
  (61 vs. 6).

57
2001 Survey

• The University trial results were 10 times as
  likely to be judged of high credibility or of
  extremely high credibility as those of the
  Industry trial (86 vs. 8.6)
• P-values for these differences range from 2-4 to
  1.5-17

58
But is the 2001 Survey credible?

• Are its grab samples representative of Canadian
  internists? (maybe cynics were
  over-represented)
• Might asking each participant to consider both
  trials constitute juxtaposition bias and
  exaggerate true differences in credibility?
  

59
So weve conducted another survey in 2004.

• For representativeness it is being performed
  on a random sample of 150 Canadian Internists
  (CSIM).
• To avoid juxtaposition bias internists have
  been randomized to receive 1 of 3 questionnaires.

60
2004 Survey To eliminate juxtaposition bias

• Group 1 Just the Industry Example
• Group 2 Just the University Example
• Group 3 Both examples
• (Group 3 also provides a replication of the 2001
  Survey, to test for that surveys
  representativness)

61
Response rate for the 2004 Survey 76

• Juxtaposition bias was not found
• Internists credibility scores for single
  examples (Groups 1 2) are closely similar to
  their credibility scores when both examples are
  provided (group 3).

62
2004 Survey - preliminary

• The earlier 2001 Survey did not appear to
  over-represent cynics.
• If anything, the credibility of the Industry
  trial in 2004 is lower than it was in 2001.

63
2001 SURVEY reprise
Univ.

Industry
Credibility
64
2004 SURVEY

Univ.
Industry
Credibility
65
2004 Survey

• The of Industry trial results judged not at
  all credible, of only low credibility, or of
  questionable credibility rose from 60 in 2001
  to 80 in 2004.

66
2004 Survey

• The Industry trial increased from 10x to 20x as
  likely to be judged not at all credible, of
  only low credibility, or of questionable
  credibility as the results of the University
  trial (now 81 vs. 4).

67
Conclusion The credibility of a trial result, in
the eyes of Canadian internists,
depends on whose eyes were on the emerging trial
data
68
(No Transcript)