New Developments in Confidence Intervals That Improve Result Reporting: Confidence Levels, Clinical Significance Curves and Risk-Benefit Contours.

1
New Developments in Confidence Intervals That
Improve Result Reporting Confidence Levels,
Clinical Significance Curves and Risk-Benefit
Contours.

• Dr. Thomas P. Shakespeare
• MBBS, FRANZCR, FAMS, MPH, GradDipMed(ClinEpi)

2
Critical References

• For a full discussion please refer to our
  original article in The Lancet
• Shakespeare TP, Gebski VJ, Veness MJ, Simes J.
  Improving interpretation of clinical studies by
  use of confidence levels, clinical significance
  curves, and risk-benefit contours. Lancet. 2001
  357 134953.
• Also you can download a free Confidence
  Calculator for our methods www.theshakespeares.co
  m/confidence_calculator.html

3
Objectives

• To understand the limitations and potential
  misinterpretation of p-values and 95 confidence
  intervals.
• To understand how new methods can improve
  statistical analysis and result reporting.
• To understand how to calculate confidence levels,
  clinical significance curves and risk-benefit
  contours.
• To understand when it is appropriate to use these
  new methods for analyzing and reporting study
  results.

4
Problems when reporting results

• 1. P values and confidence intervals are often
  misinterpreted.
• 2. They do not answer our basic clinical
  questions
• How likely is it that a clinically relevant
  benefit or detriment is present?
• How confident are we that a benefit is not
  outweighed by unacceptable toxicity?

5
The solution

• Develop tools that improve result reporting
• Confidence levels
• Clinical significance curves
• Risk-Benefit contours.
• Their advantages
• Prevent misinterpretation
• Answer our clinical questions
• Improve the decision-making process.

6
An exampleof methods to report results

• WHO Melanoma Study (Cascinelli et al, Lancet
  1998 351 793-96)
• 252 patients with truncal melanoma ?1.5mm thick
• Randomized to immediate nodal dissection or
  observation (and delayed dissection if required).

7
WHO Melanoma Study

• Results
• 5 year survival favoured immediate nodal
  dissection 61.7 vs 51.3 , HR 0.72
• 95 CI 0.49-1.04, not significant p0.07
• Authorsconclusion
• Immediate nodal dissection had no impact on
  survival, and should not be used. The results
  have been misinterpreted!

8
What information is in the 95 CI?

• 95 CI for the hazard ratio is 0.49-1.04
• Thus we can be 95 confident that the true hazard
  ratio lies within these limits, based on this
  study.
• How likely is it that a survival benefit exists,
  or does not exist, based on this data?
• Confidence intervals cant tell us, however a
  confidence level can.

9
95 Confidence intervalWHO melanoma study. Point
estimate 0.72, SE 0.192
detrimental
beneficial
1.04
0.49
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
2.5
!
!
!
!
!
!
!
!
!
2.5
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
95
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
0.4
0.5
0.6
0.7
0.8
0.9
1
1.1
1.2
Relative survival benefit (hazard ratio)
10
Confidence level for any benefit

• We need to determine how much confidence lies
  below 1.00 (HR lt 1 indicates a survival benefit).
  
• From the WHO study, the point estimate for
  survival was 0.72, with a standard error of 0.192
  (extrapolated from the original publication).

11
Confidence level for any benefit

• 1. Calculate the confidence interval around the
  hazard ratio with an upper limit of 1.00 (93 CI
  in this example)
• 2. Calculate how much confidence lies below this
  interval (half of 7 3.5)
• 3. Add the two percentages (93 3.5 96.5)
• Thus there is 96.5 confidence that a
  survival benefit exists.
• This is very high despite the lack of
  significance!

12
Confidence level for any benefit(Point estimate
0.72)
Detriment
Benefit
96.5
1.00
0.52
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
3.5
!
!
!
!
!
!
!
!
!
!
!
!
!
3.5
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
93
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
0.4
0.5
0.6
0.7
0.8
0.9
1
1.1
1.2
Relative survival benefit (hazard ratio)
13
Other magnitudes of benefit

• Confidence levels can be determined for any
  benefit or detriment of interest.
• What if my patient is only interested in a 3
  benefit or more?
• We use the same methods but set an upper value of
  0.97
• We are 94 certain that dissection results in a
  survival benefit of 3 or more.

14
Confidence level for minimum 3 benefit(Point
estimate 0.72)
Clinically
No Relevant
relevant benefit
94
benefit/detriment
0.53
0.97
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
6
!
!
6
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
88
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
0.4
0.5
0.6
0.8
0.9
1
1.1
1.2
0.7
Relative survival benefit (hazard ratio)
15
The WHO results revisited

• HR for survival 0.72, not significant (95 CI
  0.49-1.04, p0.07)
• However
• 96.5 confidence that a benefit exists
• 94 confidence that the benefit is 3 or more
• Thus a clinically relevant benefit is probable,
  and further studies are required to confirm it.

16
The WHO results revisited

• 96.5 confidence that a benefit exists
• 94 confidence that the benefit is 3 or more
• Thus a clinically relevant benefit is probable,
  and further studies are required to confirm it.
• This is in contradistinction to the authors
  conclusion.
• Confidence levels may have avoided
  misinterpretation, and provided more clinically
  relevant information.

17
Confidence levels improve result interpretation

• Confidence levels give us the level of
  confidence, likelihood or probability that a
  benefit exists, and tell us whether the benefit
  is clinically relevant.
• They are more useful than P values and confidence
  intervals.
• Confidence levels have been used to analyze
  meta-analyses and clinical studies.

18
Clinical Significance Curves

• Individuals may accept different benefit
  thresholds before using a new therapy.
• We can provide confidence levels for any
  threshold of benefit or detriment.
• These can be combined to produce a Clinical
  Significance Curve (CSC).

19
CSC for survival in WHO study

• Individuals can select an acceptable benefit
  threshold and determine the level of confidence
  associated with it.
• For example if a clinician is only interested in
  a benefit of 15 or more, we can see that there
  is only 81 confidence that such a benefit
  exists.
• CSCs provide clinically relevant information to
  individual clinicians.

20
Clinical significance curve for WHO study
100
X
!
!
!
X
90
significance level (97.5)
X
80
Any benefit 96.5 confidence ? 3 benefit 94
confidence ? 15 benefit 81 confidence ? 28
benefit 50 confidence
70
Confidence level ()
60
50
X
40
30
20
10
0
0
5
10
15
20
25
30
35
40
45
50
55
60
Relative survival benefit ()
21
Risk-Benefit Contours

• CSCs can be constructed for any comparison eg
  OS, local control, toxicity. Alternatively, the
  information can be displayed in tabular form, at
  say 5 increments of difference.
• We can combine CSCs (eg survival with toxicity),
  to form Risk-Benefit Contours (RBCs).
• RBCs allow us to calculate the confidence
  associated with acceptable risk-benefit scenarios.

22
An example

• Intergroup study 0099 (Al-Sarraf et al, JCO
  1998).
• Chemoradiotherapy vs radiotherapy for Stage III
  and IV NPC
• Results
• 3 year survival 78 v 47 (p0.005)
• Grade 3 or 4 acute toxicity 76 v 50.

23
CSC for survival

• High levels of confidence for large survival
  benefits (eg up to 20 or so).
• Statistically significant survival benefits.

24
Clinical significance curve for Intergroup
study(Absolute 3 year survival 78 v 47)
100
X
90
significance level (97.5)
80
Any benefit 99.997 confidence Min 16 benefit
97.5 confidence Min 18 benefit 95 confidence
70
Confidence level ()
60
50
40
30
20
10
0
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
0
10
20
30
40
50
60
Absolute survival benefit ()
25
CSC for toxicity

• High level of confidence that
  chemoradiotherapy causes excess acute grade 3/4
  toxicity.

26
Clinical significance curve for Intergroup
study(Acute G3/4 toxicity 76 v 50)
100
X
90
significance level (97.5)
80
Any toxicity ? 99.95 confidence Min 10.4 ?
97.5 confidence Min 13 ? 95 confidence
70
60
Confidence level ()
50
40
30
27
Risk-Benefit Contours

• Formed by combining the 2 curves.
• To use RBCs, the individual clinician first
  determines an acceptable risk-benefit scenario,
  then reads off the corresponding confidence
  associated with it.

28
60
99.95
99.99
50
99.90
99.00
97.00
95.00
85.00
40
Maximum toxicity detriment ()
90.00
80.00
70.00
60.00
50.00
30
25.00
20
0
5
10
15
20
25
30
35
Minimum survival benefit ()
29
Doctor As risk-benefit scenario

• Doctor A believes the new treatment (chemoRT)
  is more worthwhile than RT alone if
• at least an extra 5 out of his next 100 patients
  will be cured due to the new treatment
• AND
• at most an extra 50 of these 100 patients will
  experience G3/4 toxicity due to the new
  treatment.
• We can see that there is 99.9 confidence that
  the above scenario exists, based on the study.

30
60
99.99
99.90
99.95
Doctor As scenario
50
99.90
99.00
97.00
95.00
Maximum toxicity detriment ()
40
85.00
90.00
80.00
70.00
60.00
30
50.00
25.00
20
0
5
10
15
20
25
30
35
Minimum survival benefit ()
31
Doctor Bs risk-benefit scenario

• Doctor B will only accept a scenario in which
• at least an extra 5 out of his next 100 patients
  will be cured due to the new treatment
• AND
• at most an extra 30 of these 100 patients will
  experience G3/4 toxicity due to the new
  treatment.
• We can see that there is only 70 confidence that
  this scenario exists.
• Doctor B is not very confident. Risk-benefit
  contours have aided the decision-making process.

32
60
99.99
99.95
Doctor As scenario
50
99.90
99.90
97.00
95.00
40
90.00
Maximum toxicity detriment ()
85.00
80.00
70.00
60.00
50.00
30
Doctor Bs scenario
25.00
20
10
5
15
20
25
30
35
0
Minimum survival benefit ()
33
Conclusion

• P values and confidence intervals are too easily
  misinterpreted, and do not answer our simple
  clinical questions.
• By using Confidence Levels, Clinical Significance
  Curves and Risk-Benefit Contours, we can
• Gain a better understanding of our results
• Avoid misinterpretation
• Aid the decision-making process.

34
Confidence Calculator

• To calculate confidence levels, only takes a hand
  calculator and z table of normal values.
• Clinical significance curves and risk-benefit
  contours can be calculated with standard
  statistical software.
• Alternatively a dedicated Confidence calculator
  can be used.
• A free calculator is available from
  www.theshakespeares.com/confidence_calculator.html

35
References

• Please see attached notes page for a full list
  of references used in this lecture.