Training 'translational researchers at the interface of clinical epidemiology and molecular science

1
Training 'translational researchers at the
interface of clinical epidemiology and molecular
science

• David F. Ransohoff, MDDirector, K30
  ProgramProfessor of Medicine and
  EpidemiologyUniversity of North Carolina at
  Chapel Hill

2
Translational Researchfrom RFA-RM-06-002
Institutional Clinical and Translational Science
Award

• Two broad areas1. Apply lab discoveries to
  studies in humans (bench to bedside)2. Adopt
  practices in community
• There is distinct discipline of translational
  and clinical science.
• Work is interdisciplinary, across fields.

3
Purpose of talk

• Identify challenges and opportunities in
  conducting translational research.
• Then consider larger lessons conducting
  translational research who will lead how to
  train

4
ExampleMolecular markers for cancer
5
New York Times, 2.3.04
6
Nature, 6.3.04
7
Science, 10.22.04
8
Organization Markers for cancer

• Past History Threats to validity of
  clinical research
• Present RNA expression genomics for cancer
  prognosis Serum proteomics for cancer diagnosis
• Future What lessons for translational
  research? Who leads? What training?

9
History Validation of cancer markersis
disappointing (not reproducible)

• Non-invasive markers Holy Grail of cancer
  diagnosis carcinoembryonic antigen (CEA)
  CA125 magnetic resonance imaging of blood

10
History Validation of cancer markersis
disappointing (not reproducible)

• Non-invasive markers Holy Grail of cancer
  diagnosis carcinoembryonic antigen (CEA)
  CA125 magnetic resonance imaging of blood
• 2. Lessons from CEA initial results (PNAS)
  100 sensitivity, specificity for colon
  cancer high expectationsfollowed by
  disappointment

11
History Validation of cancer markersis
disappointing (not reproducible)

• Non-invasive markers Holy Grail of cancer
  diagnosis carcinoembryonic antigen (CEA)
  CA125 magnetic resonance imaging of blood
• 2. Lessons from CEA initial results (PNAS)
  100 sensitivity, specificity for colon
  cancer high expectationsfollowed by
  disappointment experience led to lessons,
  rules of evidence to evaluate diagnostic
  tests (Ransohoff and Feinstein. NEJM 1978)

12
Now, cancer markers are promising

• Knowledge of molecular biology provides targets
  to measure past knew little about what to
  target now know DNA path from normal..
  adenoma.. cancer
• Assays can measure targets past assays one
  dimensional, like CEA, fecal occult blood
  testing (FOBT) prostate-specific antigen (PSA)
  now assays multi-dimensional can measure any
  target -DNA - primers and probes, PCR -protein
  - mass spectroscopy

13
Now, cancer markers promising, but..

• Mother Nature guards her secrets closely.
• New reductionist methods mean more data, but not
  necessarily more knowledge.
• Rules of evidence have not changed.
• Our job
• to explore new technologies/fields
  efficiently to avoid predictable mistakes,
  inflated expectations make effort
  interdisciplinary, translational molecular
  biology, clinical epidemiology, biostatistics.
• Culture clash hinders exploration.

14
Organization Markers for cancer

• Past History Threats to validity of
  clinical research
• Present RNA expression genomics for cancer
  prognosis Serum proteomics for cancer diagnosis
• Future What lessons for translational
  research? Who leads? What training?

15
Validity

• Meaning of validity is so broad (Lat
  strong) it can be confusing.
• (Rules of evidence for cancer molecular-marker
  discovery and validation. Nat Rev Cancer 2004
  4309-14)

16
Two critical threats to validity

• ChanceDoes chance explain discrimination?
• BiasDoes bias explain discrimination?
• Nat Rev Cancer 20055142-9

17
Two critical threats to validity

• ChanceDoes chance explain discrimination?
• Example genomicsThe study of all ...
  nucleotide sequences, including structural...
  regulatory... noncoding DNA segments, in the
  chromosomes of an organism.
• American Heritage
  Dictionary

18
(No Transcript)
19

20

21
Strong discrimination led to interpretation as
definitive

• for clinical practice
• ... gene-expression patterns of primary tumours
  are better than available clinicopathological
  methods for determining the prognosis of
  individual patients.6,10,11
• Ramaswamy and Perou, Lancet 20033611576-7
• for biological research
• ... compelling evidence... genetic program of a
  cancer cell at diagnosis defines its biologic
  behavior many years later, refuting a competing
  hypothesis.... Wooster and Weber, NEJM
  20033482339-47

22
Can chance explain results?

• Definition In multivariable predictive models,
  overfitting (a problem of chance) occurs when
  large N of predictor variables is fit to a small
  N of subjects. A model may fit perfectly by
  chance, even if no real relationship.
  (Simon, JNCI 2003)
• Consequence Results not reproducible in
  different subjects
• Method to check for Assess reproducibility in
  totally different subjects.

23
Can chance explain results?

• to the editor
• In research to validate a prognostic system, the
  inclusion of 61 patients of the 295 in the
  validation group from the training group
  means the validation group is not
  independent.... and the degree of prognostic
  discrimination may have been inflated....
• (Ransohoff. NEJM 20033481716)

24
How much discrimination when different,
independent subjects are assessed?

• Purpose to look at... the Amsterdam 70-gene
  prognostic signature (Van de Vijver et al, N
  Engl. J Med, 2002)... external, independent
  validation.
• (Piccart MJ, Loi S, VantVeer L, et
  al. abstract at SABCS 12.8.04)

25
How much discrimination when different,
independent subjects are assessed?
10-year OS 88 (81-95)
10-year OS 71 (63-78)
Piccart MJ, Loi S, VantVeer L, et al. SABCS
12.8.04 (with permission from M. Piccart)
26
If less discrimination, would interpretation be
so strong?

• for clinical practice
• ... gene-expression patterns of primary tumours
  are better than available clinicopathological
  methods for determining the prognosis of
  individual patients.6,10,11
• Ramaswamy and Perou, Lancet 20033611576-7
• for biological research
• ... compelling evidence... genetic program of a
  cancer cell at diagnosis defines its biologic
  behavior many years later, refuting a competing
  hypothesis.... Wooster and Weber, NEJM
  20033482339-47

27
To check for overfitting, assess reproducibility
in independent group
Ransohoff. Nat
Rev Cancer 2004
28
Overfitting is not addressed in many studies of
RNA expression
Lancet. Feb 5, 2005

• Michiels et al.When studies of RNA expression
  and prognosis of cancer were reanalyzed, using
  original data, in 5 of 7, results were
  no better than chance.
• Ioannidis, in editorial ( Microarrays and
  molecular research noise discovery?), suggests
  validation groups were not independent.
  Problem is readily avoidable.

29
N Engl J Med
20043512817-26.
Chance/overfitting is addressed in study
of RNA expression
30
N Engl J Med
20043512817-26.
Chance/overfitting is addressed in study
of RNA expression

• as demonstrated in MethodsThe prospectively
  defined assay methods and end points were
  finalized in a protocol signed on August 27,
  2003. RT-PCR analysis was initiated on September
  5, 2003, and... data were transferred... for
  analysis on September 29, 2003.

31
Chance as a threat to validity
Nat Rev Cancer 20044309-14
32
Two critical threats to validity

• ChanceDoes chance explain discrimination?
• BiasDoes bias explain discrimination?
• Nat Rev Cancer 20055142-9

33
Bias

• DefinitionSystematic difference between compared
  groups, so that comparison is erroneous.
• Bias is Serious Biases are common in
  non-experimental research. Even one bias can
  be fatal.

34
Strong claims that serum proteomics can diagnose
cancer

• Claims
• for multiple cancers (ovary, prostate, breast)
  sensitivity 95-100 specificity 95-100
• appeared in Lancet, Clin Chem, WSJ, NBC, PBS,
  etc.
• led to plans for commercial test, Ovacheck, in
  2003 but plans delayed by FDA
• led researchers to redirect effort, grant
  proposals.

35

36
Proteomics Petricoin, Lancet 2.02

• Purposeto diagnose ovarian cancer vs no cancer
• Methods ovarian cancer, controls serum
  assessed by mass spectroscopy (SELDI-TOF)
  spectra analyzed by genetic algorithm
  (Correlogic)

37
A mass analyzer
(Glish, Nat Rev 2003)
38

39
Proteomics Petricoin, Lancet 2.02

• ResultsThe discriminatory pattern correctly
  identified all 50 ovarian cancer cases. for a
  sensitivity of 100 specificity of 95

40
Does bias explain some serum proteomics results
for ovarian cancer?

• (Keith Baggerlys proposal,
  as reported in Nature news 2004)
• Was bias introduced by run order
  of specimens?
• If cancers and non-cancers are run on
  different days and if the mass spec drifts over
  time, then non-biologic signal, associated with
  Ca vs no-Ca, is hard-wired into results. Bias
  (signal) is not removed - or even detected - by
  splitting sample into training and validation
  groups.

41
Bias is the challenge in observational
(non-experimental) research

• Bias is not icing on the cake it is the cake.
• Bias is a large topic, difficult multiple
  biases require different methods to
  address (e.g., randomization, blinding, uniform
  handling, etc) some methods not available in
  observational research some biases may be
  impossible to identify even 1 bias may be fatal
• The process to deal with bias is routinely
  ignored by authors, reviewers, editors in omics
  research.

42
Bias is so serious that results are guilty (of
bias) until proven innocent.

• Innocence is proven byDoing process design
  - to avoid bias conduct - to measure if bias
  occurred interpretation - to determine if
  importantReporting process Methods, Results,
  Discussion
• (Bias as a threat to the validity of cancer
  molecular-marker research. Nat Rev Cancer
  20055142-9)

43
Process to deal with bias of baseline
inequality in RCT Report results of
randomization, in Table 1
44
Process to deal with bias of baseline
inequality in typical -omics study
45
All samples were stored at -80ºC until use.

• OK but were specimens handled equally in all
  steps, e.g., - time from blood draw to
  spin/freeze - number of thaw-freeze cycles -
  duration of storage - type of blood collection
  tube (red/purple) - time from thawing to
  assay - etc.
• Any step is possible source of fatal bias in a
  proteomics study.

46
Bias as a threat to validity
Nat Rev Cancer 2005 5142-9
47
Does serum proteomics diagnose cancer?

• Question Where is the landmark study in NEJM,
  or Science, that shows proof of
  principle? (I.e., convincingly avoids chance,
  bias)
• Answer March 2006 None exists.gt March
  2006 Who knows?

48
Expectations strong in 2006
49
New proteomics cancer test center stage Wall
Street Journal, January 1, 2006 page A01
50
New proteomics cancer test center stage Wall
Street Journal, January 1, 2006 page A01

• More biomarkers are likely to be found and
  validated in 2006.

51
J Clin Invest 2006116271
Serum proteomics can diagnose
prostate cancer
52

J Clin Invest 200611626.

53
(No Transcript)
54
J
Clin Invest 2006116271
Test is 100 sensitive and specific for PrCa
55
Serum proteomics can diagnose prostate cancer

• Authors sayexoprotease activities should be
  focus of future peptide biomarker discovery
  efforts
• Editorialists saylow molecular weight
  biomarker pipeline is surging with potential

56

• Can bias explain results?

57
J
Clin Invest 2006116271
Prostate Cancer age (mean) 67 yrs
58
J
Clin Invest 2006116271
Prostate Cancer age (mean) 67 yrs
Controls age (mean) 35 yrs gender 58
female
59
Organization Markers for cancer

• Past History Threats to validity of
  clinical research
• Present RNA expression genomics for cancer
  prognosis Serum proteomics for cancer diagnosis
• Future What lessons for translational
  research? Who leads? What training?

60
Translational ResearchLessons

• Translational research by definition,
  interdisciplinary
• Effective interdisciplinary communication is
  needed to conduct reliable research (avoids
  chance, bias).
• Clinical epidemiology key to 2, when research
  is observational (non-experimental), about
  diagnosis, prognosis, and etiology.

61
Who leads? What training?

62
Who leads? What training?

• First, we need leaders. teams may be created,
  but teams need someone to be interdisciplinary,
  to help make teamwork effective
• Leaders can happen in different ways e.g., a.
  laboratory scientist learns clinical
  epidemiologyb. clinical epidemiologist learns
  laboratory science

63
What training? Example Case report

• From birth overarching goal practice with
  father, grandfather
• 1975-7 trained in clinical epidemiology
  (Feinstein Yale) Problemsevaluating
  diagnostic tests NEJM 1978
• 1977-9 trained in gastroenterology (U Chicago)
• gt1979 clinical research (clin epi/HSR)
  gallstones natural history and management
  cancer risk in ulcerative colitis colon cancer
  screening

64
Case report

• 1998 Opportunity knocksExact Sciences asks for
  help to assess stool DNA molecular markers to
  diagnose CRC.

65
Vogelstein model molecular basis of stool DNA
test for CRC
Altered DNA Methylation
Other Genetic Alterations? (e.g. TGF-ß type II
receptor)
Modified from Fearon and Vogelstein Cell 1990
61759-767
66
(No Transcript)
67
NEJM 20043512704
68
Case report

• Extend my work to molecular markers for
  cancer
• Sabbatical scientific meetings NCI/EDRN
  (Early Detection Research Network) Gordon
  conferences etc (what questions, what
  methods)
• course on molecular technology (technical
  details) begin collaborations write on role
  of c.e. methods
• Learning molecular methods easier than
  expected.

69
Case report

• As a clinical epidemiologist, I see current
  -omics research as 80 routine clinical epi
  ....where marker happens to be molecule.

70
Lessons for other translational fields

• 
  (from CTSA RFA)
• Two translational research areas1. Apply lab
  discoveries to studies in humans (bench to
  bedside)2. Adopt practices in community

71
ConclusionChallenge, opportunity

• 1. An exciting era, because we know so much
  biology have powerful tools to measure biology
• 2. But rules of evidence, about validity, have
  not changed such rules (clinical epidemiology)
  may be helpful.
• 3. Expectations in 2006 are greater than results
  will support.
• 4. Disappointment and wasted effort may occur
  that will have been predictable related to
  culture clash.
• Our job Go back to 1 avoid predictable
  disappointment and wasted effort Be efficient
  in generating reliable data about markers.