Randomisation vs' observation: an unnecessary opposition

1
Randomisation vs. observation an unnecessary
opposition

• Jan P Vandenbroucke
• Leiden University Medical Center

2
The case I want to make

• There is no opposition between randomised and
  observational research each has its own merits
  and fields of application

3
Discovery and Explanation

• Discoveries see things in new light
• Odd course of disease in a patient
• Strange result of lab experiment
• Peculiar subgroup in data
• Juxtaposition of ideas from literature
• Enthusiasm about idea get hold of whatever data,
  submit paper
• Next wave new variants, subgroups, definitions
  (mostly existing data)
• If no resolution brand new studies

4
Evaluation

• Is the patients lot improved by new diagnostics
  and new treatments?
• Most developed branch randomised trials of drug
  therapy
• Credibility depends on complete preplanning and
  registration
• no straying into promising side alleys!

5
What they think about each other

• Evaluation about DE
• Dangerously biased
• Explanations dreamed up
• Irresponsible origin of hypes and scares
• Unnecessary research loops

• DE about Evaluation
• Stifles imagination because pre-planned
• One-sided views chance needed for progress of
  science
• Only quality control
• Numbers are not explanations

6
Even within the mind of an individual scientist

• Interaction of oral contraceptives and Factor V
  Leiden mutation in causing venous thrombosis
• FVL - -
• OC - -
• Odds Ratio 1 4 7 35
• (Lancet 1994)
• Cry beware when (sponsored) RCTs highlight a
  particular subgroup

7
Outline

• Different hierarchies
• Different needs for randomisation
• Subgroup analysis, multiplicity, data dredging
• Hierarchies revisited
• The HRT debacle
• Conclusions

8
Hierarchy of study designs intended effects of
therapy

• Randomised controlled trial
• Prospective follow-up
• Retrospective follow-up
• Case-control
• Anecdotal case report and series

9
Hierarchy of study designsdiscovery and
explanation

• Anecdotal cases, lab result, subgroups
• Case-control
• Retrospective follow-up
• Prospective follow-up
• Randomised controlled trial

10
Hierarchy of study designs

• Evaluation of therapy
• RCT
• Prospect follow-up
• Retrospect follow-up
• Case-control
• Case report series

• Discovery Explanation
• Anecdotal Case, etc.
• Case-control
• Retrospect follow-up
• Prospect follow-up
• RCT

11

• Why randomisation is almost always necessary for
  evaluation of intended effects of therapy, and
  almost never for aetiology
• The example of adverse effects of treatment,
• RCTs mostly uninformative
• Follow-up too short
• Numbers not large enough
• Selected populations

12
Breaking the link between prognosis and
prescription

• Usual practice therapy guided by prognosis. For
  evaluation concealed randomization needed.
  Doctor knows prognosis, but as a result of
  concealed randomization cannot predict therapy.
• (Chalmers, JRSM 1997 Schulz Grimes, Lancet
  2002)
• Mirror image adverse effects are unintended,
  often unexpected, and are different diseases with
  different risk factors their prognosis is not
  known. Doctor knows therapy, but not risk for the
  adverse effect. Data from usual practice can be
  used.
• Example skin rash after prescription of
  antibiotic
• (Vandenbroucke, Lancet 2004)

13
Adverse effects selection of groups where events
are unpredictable

• Example type of oral contraceptives (OC) and
  venous thrombosis
• restrict to first thrombosis in OC-using women
  without any known risk factor
• use case-control design to compare with equally
  healthy OC-using women with no venous thrombosis
• Background theory Jick Vessey, Miettinen,
  Rubin Holland (refs in Vdb, Lancet 2004)

14
Adverse effectsEmpirical demonstration that
observational studies suffice

• For same therapy same adverse effect large
  meta-analyses of RCTs (4000) compared to large
  observational studies (exceptional) 15 topics
• No systematic difference - If anything
  observational more conservative!
• (Papanikolaou et al. CMAJ 2006174635 Vdb,
  Editorial 645)

15
A generalisation observational studies on
potential causes of disease

• Causes of disease are mostly unintended,
  unexpected effects.
• Epidemiologic classics smoking and lung cancer,
  asbestos and mesothelioma, lead in paint and
  child development, intrauterine irradiation and
  leukaemia, etc.
• Aetiology in general randomisation not needed
• Further examples genetics, outbreak
  investigations,

16
Not all observational research is equally
acceptable

• Axis of haphazardness of exposure
• Vegetarians Genetic
• mortality effects

Vandenbroucke, PLos Med 2008
17
Subgroups and multiplicity of analysis
18
Subgroups and multiplicity of analysis

• Axis of multiplicity
• Single Nucleotide Randomised
• Polymorfisms trials

19
Subgroups and multiplicity of analysis

• Axis of multiplicity prior belief
• Single Nucleotide Randomised
• Polymorphisms trials
• 1 in 100,000 50 - 50

20
Subgroups and multiplicity

• Many PhD students looking at data is NOT like
  tens of thousands of SNPs. Subgroups suddenly
  explain previous findings.
• PhD students hover over axis of multiplicity
• Is a subgroup specified beforehand more credible?
• RCTs unlikely that worthwhile subgroup was not
  thought about (Rothwell Lancet 2005)
• Observational research prior evidence may exist
  without investigators being aware. Data very
  often used for new purposes studies change aims
  of research.

21
Example of change in priors

• Case series of autopsies on patients with
  idiopathic fatal pulmonary emboli
• Aim presence of Factor V Leiden in paraffin
  blocks
• Leiden University, 1970-1994 30 cases
• Surprise 11 of 30 were psychiatric patients,
  treated with neuroleptic drugs
• Literature found
• German literature 1960-1980
• Recent (1997) study on new atypical
  antipsychotic high risk for pulmonary emboli as
  an unexplained finding
• (Thrombosis and Haemostasis 1998)

22
Subgroups and multiplicity

• Necessary for observational research that aims
  at discovery and explanations
• Solution for multiplicity with low priors
  replication
• RCTs meta-analysis of subgroups
• Genomics consortia for immediate checks
• Discovery in observational research
• Original report tell candidly how and why
• Thoughtful replication not the same over again,
  but trying to tackle potential bias and
  confounding
• Registration of observational research is no
  solution

23
Rethinking the hierarchy of evidence

• Randomised controlled trial
• Prospective follow-up
• Retrospective follow-up
• Case-control
• Case report and series
• Is this a hierarchy of prior odds?

24
Imagine an upside down world

• Randomised trials start with same prior odds as
  individual SNPs 1 in 100,000
• Observational studies start with 50-50 priors
• We would readily find explanations why
  observational studies perform so much better than
  RCTs

25
Hierarchies of study design

• Inverse for discovery and explanation vs.
  evaluation of therapy
• Confounded by prior odds are we deluding
  ourselves?

26
What about the Hormone Replacement Therapy
debacle?

• Observational vs. Randomised trials
• Myocardial infarction effects opposite
• Breast cancer effects stronger in observational
• Venous thrombosis, colon ca, fractures similar
• Solved! Not that much a matter of confounding,
  but about time windows (overview Vandenbroucke,
  Lancet 2009)
• Myocardial infarction excess occurred early not
  seen in observational studies on current users
  if reanalysis from time to start HRT results the
  same!
• Breast cancer excess in women treated early
  after menopause if reanalysis from time to
  menopause results the same!

a matter of bias or confounding More about time
window
27
Proposed conclusions

• The case I tried to make There is no opposition
  between randomized and observational research
  each has its own merits and fields of
  applicability
• We enjoy multiplicity with low priors for
  observational research, not for randomised trials
  of therapy consequences of being wrong are
  different
• We need both hierarchies, or perhaps no
  hierarchies at all we need discovery and
  explanation as well as evaluation of therapy

28
To reread, or search references

• Vandenbroucke JP. When are observational studies
  as credible as randomised trials? Lancet 2004
• Vandenbroucke JP. Observational research,
  randomised trials and two views of Medical
  Science. PloS Med 2008
• Supplementary material longer text with more
  examples and more topics
• Vandenbroucke JP, Psaty BM. Benefits and risks of
  drug treatments how to combine the best evidence
  on benefits with the best data about adverse
  effects. JAMA 2008
• Vandenbroucke JP. The HRT controversy
  observational studies and RCTs fall in line.
  Lancet 2009

29
(No Transcript)
30

• Haphazard is not random
• Still, haphazard or ostensibly irrelevant
  assignments are to be preferred to assignments
  which are known to be biased in ways that cannot
  be measured and removed analytically.
• Rosenbaum, Observational Studies, 2nd Ed
• Springer 2002.

31
Observational research for regulation?

• Credibility if unplanned discovery? e.g.
  adverse effect
• Hinges on
• If discovered by anecdotal report, in data etc,
  but quickly confirmed embedded in other
  scientific knowledge immediately acceptable
• Strength of association if weak ancillary
  evidence?
• If continuing controversy plan new observational
  studies with same mind-set as RCT completely
  pre-planned for single purpose (with existing or
  new data)

32
The progress of science

• Sir William Osler Truth may suffer all the
  hazards incident to generation and gestation
  and all scientific truth is conditioned by
  the state of knowledge at the time of its
  announcement (Harveian Oration, BMJ 1906)
• Stephen Jay Gould Science makes progress
• in a fitful and meandering way (Science 2000)

33
A difference in loss function (2)?

• The loss function of scientific research cannot
  be specified, in contrast to research that leads
  to practical decisions (RA Fisher)
• The loss function from evaluation research is
  about people cured or harmed the loss function
  of discovery and explanation is about future
  insight

34
(No Transcript)
35
Positive unexpected effects

• Oral contraceptives and cancer ovaries,
• aspirin and myocardial infarction
• Expected to be rare Richard Peto
• Hormonal replacement therapy randomisation vs.
  observation?
• - Myocardial infarction, effects opposite
• - Venous thrombosis, breast cancer, colon ca,
  fractures effects in similar direction
• Statins, HRT and NSAIDs protect from dementia
  never confirmed in RCTs.

36
Sir Austin Bradford Hill, the observational
researcher

• Original methodologic insights in case-control
  and cohort, set-up and analysis
• First study on smoking and lung cancer
  case-control
• (Doll, Cohort studies history of the method -
  2001)

37
Doll and Hill, case-control study, discussion
(BMJ 1950)

• If it can be assumed that the patients without
  carcinoma of the lung who lived in Greater London
  at the time of their interview are typical of the
  inhabitants of Greater London in regard to their
  smoking habits, then the number of people in
  London smoking different amounts of tobacco can
  be estimated. Ratios can be obtained between the
  number of patients seen with carcinoma of the
  lung and the populations at risk who have smoked
  comparable amounts of tobacco.

38
Ioannidis Why most research findings are false

• Argument based on Bayesian reasoning RCTs start
  with highest prior odds 50-50
• Observational research starts with much lower
  prior odds
• Because most literature is observational, most
  priors are way below 50-50 with an additional
  dose of bias and confounding, most research
  findings will have less than 50 chance of being
  true
• (PloS Medicine 2005)

39
The case-control study Work-horse of
observational research

• Second place in hierarchy of discovery and
  explanation first analytic design after initial
  idea
• In principle case-control, same information as
  follow-up study much more convenient
• Very often suffices no higher designs
  necessary
• Many methodologic explanations sought for
  failures simply inevitable because of
  discovery situation?

40
Relative position of observational designs

• Case-control
• First choice for aetiologic research
• Same relative risk as follow-up
• Often suffices
• Bad press, biases, failures inevitable?
• Prospective follow-up
• Started sparingly
• Only if worthwhile (strong prior) and necessary

41
Proposed conclusions (2)

• academics and commentators care more about
  whether ideas are interesting than whether they
  are true. Politicians live by ideas just as much
  as professional thinkers do, but they can't
  afford the luxury of entertaining ideas that are
  merely interesting. They have to work with the
  small number of ideas that happen to be true and
  the even smaller number that happen to be
  applicable to real life. In academic life, false
  ideas are merely false and useless ones can be
  fun to play with. In political life, false ideas
  can ruin the lives of millions and useless ones
  can waste precious resources. An intellectual's
  responsibility for his ideas is to follow their
  consequences wherever they may lead. A
  politician's responsibility is to master those
  consequences"
• Ignatieff, NYT 2007