A methodology for the comparison of ROC umbrella volumes applied to the assessment of lung cancer di

1
A methodology for the comparison of ROC umbrella
volumes applied to the assessment of lung cancer
diagnostic markers

• Christos T. Nakas, PhD
• University of Thessaly
• Todd A. Alonzo, PhD
• University of Southern California
• August 2, 2007

2
Outline

• Motivation
• ROC surface
• Umbrella ROC graph, Umbrella Volume
• Lung Cancer Analysis
• Summary

3
Lung Cancer

• Most common cancer worldwide
• Leading cause of cancer-related death in U.S. and
  worldwide
• Early detection usually results in better
  prognosis
• Goal Find new markers for early cancer detection

4
DNA Methylation Marker

• Changes in DNA methylation occur early in
  carcinogenesis (Laird 1997)
• New technology for measuring DNA methylation
  allows the search for new cancer markers
• We consider quantitative DNA methylation which is
  measured on a continuous scale

5
Lung Cancer Data

• 131 lung specimens
• 54 squamous cell carcinoma (SQ)
• 26 large cell carcinoma (LC)
• 51 non-tumor lung (NTL)
• DNA methylation measured using MethyLight, a
  high-throughput quantitative methylation assay
  that utilizes fluorescence-based real time PCR
• of methylated reference sample for two markers
• tumor necrosis factor receptor superfamily,
  member 25 (TNFRSF25)
• proenkephalin (PENK)

6
Extending ROC to 3 disease states ROC Surface

• Y1, Y2, Y3 marker values for 3 disease states
• For two ordered thresholds c1ltc2
• TCR1P(Y1ltc1)
• TCR2P(c1ltY2ltc2)
• TCR3P(c2ltY3)
• ROC surface is plot of TCRs for all possible c1,
  c2 (Scurfield 1996)

7
Volume Under the ROC Surface (VUS)

• VUS P(Y1 lt Y2 lt Y3)
• VUS 1 when 3 classes are perfectly
  discriminated in the correct order
• VUS 1/6 when 3 distributions completely overlap
• VUS 0 when 3 classes are perfectly
  discriminated in a wrong order (Y3 lt Y2 lt Y1)

8
VUS Estimator

• Non-parametric estimator (Dreiseitl et al 2000)
• where
• Var(VUS) can be estimated using U-statistics
  theory (Dreiseitl et al 2000) or bootstrap

9
Markers for Lung Cancer (1/2)
10
Markers for Lung Cancer (2/2)
11
Umbrella Ordering

• Dominance of two disease classes over a third
  (i.e. Y2 gt Y1 lt Y3)
• Dominance of one class over the other two (i.e.
  Y1 lt Y3 gt Y2)
• Generalization of the ROC surface to accommodate
  umbrella orderings (Nakas, Alonzo 2007)

12
Umbrella ROC graph

• Consider the ordering Y2 gt Y1 lt Y3
• Key observation
• P(Y2 gt Y1 lt Y3)P(Y1 lt Y2 lt Y3)P(Y1 lt Y3 lt
  Y2)
• Can construct two ROC surfaces (A and B)
  corresponding to the 2 components
• These can be viewed on a single graph (umbrella
  ROC graph) by plotting TCR for A and B

13
Umbrella ROC graphs
(a) Y1N(0,1), Y2N(0.5,1), Y3N(0.8,1) (b)
Y1Y2Y3
14
Volume of umbrella ROC graph

• UV P(Y1ltY2ltY3) P(Y1ltY3ltY2) , equivalently
  volume under surface A plus volume above surface
  B
• where IU 1 if Y2gtY1ltY3 0 otherwise
• UV 1 when classes perfectly discriminated in
  order Y2gtY1ltY3
• UV 1/3 when 3 distributions completely overlap
• UV 0 when classes perfectly discriminated in
  order Y2ltY1gtY3
• Var(UV) can be estimated using U-statistics
  theory or bootstrap (Nakas, Alonzo 2007)

15
ROC UV comparison (1/2)
16
ROC UV comparison (2/2)
17
Umbrella ROC graphs for PENK, TNFRSF25
18
Umbrella volumes (UVs)

• TNFRSF25 UV (SQ gt NTL lt LC)
• VUS (NTL lt LC lt SQ) VUS (NTL lt SQ lt LC)
• 0.33 0.37 0.70 (0.55, 0.85)
• PENK UV (SQ gt NTL lt LC) 0.50 (0.35, 0.65)
• Z2.212 (p0.027). TNFRSF25 is significantly
  better than PENK discriminating NTL from LC, SQ
  specimens, alpha0.05, without specifying
  relationship between LC, SQ specimens

19
Alternative approach

• Convert the 3 disease classes into 2 classes by
  collapsing SQ and LC into 1 class then construct
  ROC curve
• It has been shown that this approach can conceal
  important relationships and can lead to biased
  estimates of accuracy

20
Another alternative approach

• Pairwise ROC analysis
• Doesnt test hypothesis of interest
• Interpretation difficult as of disease states
  increase because of comparisons increase too

21
Discussion

• Developed approach to compare ROC umbrella
  volumes
• Approach applied to diagnostic markers. Methods
  apply more generally to any classifier
• Other examples on the importance of the study of
  different restricted class orderings given in
  Hollander, Wolfe (1999), Silvapulle, Sen (2005),
  Lee et al (2006).

22
Application to clinical trials

• By considering treatment arms as the classes, the
  methods can be used to assess efficacy of 3 arms
  of a clinical trial where umbrella ordering is of
  interest
• E.g., compare two treatment arms to a placebo arm