Performance of a diagnostic test

1
Performance of a diagnostic test

• Manuel Dehnert
• 15th EPIET Introductory Course
• Lazareto, Menorca, Spain
• October 2009
• Source
• Thierry Ancelle, Marta Valenciano, 2007

2
What affects the performance of a testapplied to
a given population ?

• the quality of the test itself
• the frequency of the disease in the population

3
Outline

• Performance of a test in an experimental
  setting(intrinsic characteristics)
• sensitivity
• specificity
• choice of a threshold
• Performance of a test in a population
• predictive value of a positive test (PVP)
• predictive value of a negative test (PVN)
• impact of disease prevalence, sensitivity, and
  specificity on predictive values

4
1. Performance of a test in an experimental
setting
5
Sensitivity of a test

• Ability of a test to identify correctly affected
  individuals
• proportion of people testing positive among
  affected individuals

True patients (gold standard)
-
True positive (TP)
Test
False negative (FN)
Sensitivity (Se) TP / ( TP FN )
6
Sensitivity of a PCR for congenital
toxoplasmosis
Sensitivity 54 / 58 0.931 93.1
7
Specificity of a test

• Ability of test to identify correctly
  non-affected individuals
• - proportion of people testing negative among
  non-affected individuals

Non-affected people
-
False positive (FP)
Test
True negative (TN)
Specificity (Sp) TN / ( TN FP )
8
Specificity of a PCR for congenital
toxoplasmosis
Specificity 114 / 125 0.912 91.2
9
Performance of a test
Disease
No
Yes

FP
TP
Test

TN
FN
TN Sp TN FP
TP Se TP FN
10
Distribution of quantitative test results among
affected and non-affected people(ideal case)
Non affected
Threshold for positive result
Affected
Number of people tested
TN
TP
0 5 10
15 20
Quantitative result of the test
11
Distribution of quantitative results among
affected and non-affected people(realistic case)
Non-affected
Threshold for positive result
Affected
TN
TP
Number of people tested
FN
FP
0 5 10
15 20
Quantitative result of the test
12
Effect of Decreasing the Threshold
Non affected
Threshold for positive result
Affected
FP
Number of people tested
TP
TN
FN
0 5 10
15 20
Quantitative result of the test
13
Effect of Decreasing the Threshold
Disease
No
Yes

FP
TP
Test

TN
FN
TN Sp TN FP
TP Se TP FN
14
Effect of Increasing the Threshold
Non-affected
Threshold for positive result
Affected
TN
Number of people tested
TP
FN
FP
0 5 10
15 20
Quantitative result of the test
15
Effect of Increasing the Threshold
Disease
No
Yes

FP
TP
Test

TN
FN
TP Se TP FN
TN Sp TN FP
16
Performance of a Test and Threshold

• Sensitivity and specificity vary in opposite
  directions when changing the threshold
• The choice of a threshold is a compromise to
  best reach the objectives of the test
• consequences of having false positives?
• consequences of having false negatives?

17
When false diagnosis (FP) is worse than missed
diagnosis (FN)

• Example Screening for congenital toxoplasmosis
• One should minimise false positives
• Prioritise SPECIFICITY

18
When missed diagnosis (FN) is worse than false
diagnosis (FP)

• Example Screening of phenylketonuria at birth
• One should minimise the false negatives
• Prioritise SENSITIVITY

19
Receiver Operating Characteristics curve(ROC
curve)

• Representation of relationship between
  sensitivity and specificity for a test
• Simple tool to
• define best cut-off value of a test
• compare performance of two tests

20
Prevention of Blood Transfusion Malaria Choice
of an Indirect IF Threshold
Sensitivity
100
1/10
1/20
1/40
80
1/80
1/160
60
IIF Dilutions
1/320
40
1/640
20
0
0
20
40
60
80
100
1- Specificity
21
Comparison of Performance of ELISA and CATT Test
for Screening of Human Trypanosomiasis
Sensitivity
100
80
ELISA CATT
60
40
20
0
0
25
50
75
100
1- Specificity
22
Comparison of Performance of ELISA and CATT Test
for Screening of Human Trypanosomiasis
Sensitivity
100
80
ELISA CATT
60
40
20
0
0
25
50
75
100
1- Specificity
23
Comparison of Performance of ELISA and CATT Test
for Screening of Human Trypanosomiasis
Sensitivity
100
80
ELISA CATT
60
Area under the ROC curve (AUC)
40
20
0
0
25
50
75
100
1- Specificity
24
Outline

• Performance of a test in an experimental
  setting(intrinsic characteristics)
• sensitivity
• specificity
• choice of a threshold
• Performance of a test in a population
• predictive value of a positive test (PVP)
• predictive value of a negative test (PVN)
• impact of disease prevalence, specificity, and
  sensitivity on predictive values

25
2. Performance of a test in a population
26
Rationale

• The status healthy / sick of a patient is not
  known
• Tests are not perfect

27
Rationale

• Questions to be addressed by the clinician
• probability that a subject with a positive test
  is really sick?
• probability that a subject with a negative test
  is really healthy?
• Question to be addressed by the epidemiologist
• proportion of positive tests corresponding to
  true patients?
• proportion of negative tests corresponding to
  healthy subjects?

28
Predictive Value of a Positive test(PVP)

• Probability that an individual testing positive
  is truly affected
• proportion of affected people among
• those testing positive

Disease
No
Yes

Test
FP
TP
PVP TP/(TPFP)
29
Predictive Value of a Negative test(PVN)

• Probability that an individual testing negative
  is truly non-affected
• proportion of non affected among
• those testing negative

Disease
No
Yes

Test
TN
FN
PVN TN/(TNFN)
30
Predictive Value of a Positive and a Negative
test
Disease
No
Yes

PVP TP/(TPFP)
Test

PVN TN/(TNFN)
TN
FN
31
Problem ?

• The predicted values depend on the sensitivity
• and on the specificity of the test as well as on
  the
• prevalence of the disease

32
Relation between predictive values and
sensitivity / specificity
Disease
No
Yes

PVP TP/(TPFP)
Test

PVN TN/(TNFN)
TN
FN
33
Step 1 Specify the prevalence (Pr) of disease
Disease
No
Yes


Test


Pr
1-Pr
34
Step 2 Use sensitivity (Se) to distribute test
results among the diseased
Disease
No
Yes


Se Pr

Test


(1-Se)Pr

Pr
1-Pr
35
Step 3 Use specificity (Sp) to distribute test
results among the non-diseased
Disease
No
Yes

(1-Sp)(1-Pr)
Se Pr

Test

Sp(1-Pr)
(1-Se)Pr

Pr
1-Pr
36
Step 4 Determine the proportion testing positive
and the proportion testing negative
Disease
No
Yes

(1-Sp)(1-Pr)
Se Pr (1-Sp)(1-Pr)
Se Pr
Test

Sp(1-Pr)
(1-Se)Pr
(1-Se)Pr Sp(1-Pr)
Pr
1-Pr
37
Step 5 Calculate PPV and NPV with appropriate
expressions from Step 4
38
Relation between predictive values and
sensitivity / specificity
The PVP of a test is affected by its specificity
The PVN of a test is affected by its sensitivity
39
Relation between predictive values and prevalence

• High prevalence
• test will pick up
• more true positives (increasing PVP)
• more false negatives
• Low prevalence
• test will pick up
• more false positives
• more true negatives (increasing PVN)

40
Se 90
Sp 90
PVP 90
Prevalence 50
Not ill
Ill

Test
PVP 50

Prevalence 10
41
Predictive value of a positive (PVP) and negative
(PNV) test according to the prevalence (80
sensitivity and specificity)
100
80
PVN
60
Predictive value ()
40
20
PVP
0
0
25
50
75
100
Prevalence ()
42
Example Screening for human trypanosomiasisin
two settings

• CATT test
• Sensitivity 95
• Specificity 75
• Endemic area
• Prevalence 20
• Low endemic area
• Prevalence 0.5
• 100,000 tests performed in each area

43
Example Screening for human trypanosomiasisin
two settings
CATT test sensitivity 95 CATT test
specificity 75
Prevalence 20
PVP 48.7 PVN 98.4
44
Example Screening for human trypanosomiasisin
two settings
CATT test sensitivity 95 CATT test
specificity 75
Prevalence 0.5
PVP 1.90 PVN 98.97
45
Summary of Predictive Values

• Predictive values affected by disease prevalence
• high prevalence, test will pick up
• more true positives (increasing PVP)
• more false negatives
• low prevalence, test will pick up
• more false positives
• more true negatives (increasing PVN)
• The PVP of a test is affected by its specificity
• The PVN of a test is affected by its sensitivity

46
Conclusions

• Sensitivity and specificity
• intrinsic characteristics of a test
• capacity to identify the affected
• capacity to identify the non-affected
• matter to laboratory specialists
• independent from the disease prevalence
• Predictive values
• performance of a test in real life
• how to interpret a positive test
• how to interpret a negative test
• matter to clinicians and epidemiologists
• dependent on the disease prevalence

47
References

• Ancelle T. Statistique épidémiologique. Maloine.
  2002
• Case study Toxoplamosis