Performance of a diagnostic test

1
Performance of a diagnostic test
Steen Ethelberg 17th EPIET Introductory
Course Lazareto, Menorca, Spain September 2011

• Thierry Ancelle
• Marta Valenciano

2
Outline

• Performance characteristics of a test
• Sensitivity
• Specificity
• Choice of a threshold.
• Performance of a test in a population
• Positive predictive value of a test (PPV)
• Negative predictive value of a test (NPV)
• Impact of disease prevalence, sensitivity and
  specificity on predictive values.

3
1. Performance of a test in an experimental
setting
4
Population with ill and non-ill individuals
5
Test should identify the ill only
6
In reality, tests are not perfect
7
Sensitivity specificity
8
Sensitivity of a test

• Ability of a test to correctly identify affected
  individuals
• Proportion of people testing positive among
  affected individuals

Patients
-
True positive (TP)
Test
False negative (FN)
Sensitivity (Se) TP / ( TP FN )
9
Sensitivity of a PCR for congenital
toxoplasmosis
Patients with toxoplasmosis

Rapid test True positive 54
Rapid test False negative 4
58
Sensitivity 54 / 58 0.931 93.1
10
Specificity of a test

• Ability of test to identify correctly
  non-affected individuals
• Proportion of people testing negative among
  non-affected individuals

Non-affected people
-
False positive (FP)
Test
True negative (TN)
Specificity (Sp) TN / ( TN FP )
11
Specificity of a PCR for congenital
toxoplasmosis
Individuals without toxoplasmosis

Rapid test False positive 11
Rapid test True negative 114
125
Specificity 114 / 125 0.912 91.2
12
Performance of a test
Disease
No
Yes

FP
TP
Test

TN
FN
TN Sp TN FP
TP Se TP FN
13
Distribution of quantitative test results among
affected and non-affected people
Ideal situation
Non affected
Affected
Number of people tested
TN
TP
0 5 10
15 20
Quantitative result of the test
14
Distribution of quantitative results among
affected and non-affected people
More realistic situation
Non-affected
Affected
TN
TP
Number of people tested
FN
FP
0 5 10
15 20
Quantitative result of the test
15
Effect of Decreasing the Threshold
Non affected
Threshold for positive result
Affected
FP
Number of people tested
TP
TN
FN
0 5 10
15 20
Quantitative result of the test
16
Effect of Decreasing the Threshold
Disease
No
Yes

FP
TP
Test

TN
FN
TN Sp TN FP
TP Se TP FN
17
Effect of Increasing the Threshold
Non-affected
Threshold for positive result
Affected
TN
Number of people tested
TP
FN
FP
0 5 10
15 20
Quantitative result of the test
18
Effect of Increasing the Threshold
Disease
No
Yes

FP
TP
Test

TN
FN
TN Sp TN FP
TP Se TP FN
19
Performance of a test and threshold

• Sensitivity and specificity vary in opposite
  directions when changing the threshold (e.g. the
  cut-off in an ELISA)
• The choice of a threshold is a compromise to best
  reach the objectives of the test
• consequences of having false positives?
• consequences of having false negatives?

20
When false diagnosis is worse than missed
diagnosis

• Example Screening for congenital toxoplasmosis
• One should minimise false positives
• Prioritise SPECIFICITY

21
When missed diagnosis is worse than false
diagnosis

• Example Testing for Helicobacter pylori
  infection
• One should minimise the false negatives
• Prioritise SENSITIVITY

22
Using several tests

• One way out of the dilemma is to use several
  tests that complement each other
• First use test with a high sensitivity
• Second use test with a high specificity

23
ROC curves

• Representation of relationship between
  sensitivity and specificity for a test
• Receiver Operating Characteristics curve
• Simple tool to
• Help define best cut-off value of a test
• Compare performance of two tests.

24
Prevention of Blood Transfusion MalariaChoice
of an Indirect IF Threshold
Sensitivity ()
100
1/10
1/20
1/40
80
1/80
1/160
60
IIF Dilutions
1/320
40
1/640
20
0
0
20
40
60
80
100
100 - Specificity ()
25
Comparison of Performance of ELISA and CATT Test
for Screening of Human Trypanosomiasis
Sensitivity ()
100
80
ELISA CATT
60
40
20
0
0
25
50
75
100
100 - Specificity ()
26
Comparison of Performance of ELISA and CATT Test
for Screening of Human Trypanosomiasis
Sensitivity ()
100
80
ELISA CATT
60
Area under the ROC curve (AUC)
40
20
0
0
25
50
75
100
100 - Specificity ()
27
Performance of a test

• Validity
• Sensitivity
• Specificity
• Reproducibility
• Concepts may also used more broadly
• Exposure status
• Case definitions

28
2. Performance of a test in a population
29
Would also like to know

• As a clinician
• probability that a individual with a positive
  test is really sick?
• probability that a individual with a negative
  test is really healthy?
• As an epidemiologist
• proportion of positive tests corresponding to
  true patients?
• proportion of negative tests corresponding to
  healthy subjects?

30
Predictive values
31
Positive Predictive Value

• Probability that an individual testing positive
  is truly affected
• proportion of affected people among
• those testing positive

Disease
No
Yes

Test
FP
TP
PPV TP/(TPFP)
32
Negative Predictive Value

• Probability that an individual testing negative
  is truly non-affected
• proportion of non affected among
• those testing negative

Disease
No
Yes

Test
NPV TN/(TNFN)
TN
FN
33
Predictive value of a positive and a negative
test
Disease
No
Yes

PPV TP/(TPFP)
Test

NPV TN/(TNFN)
TN
FN
PPV VPP PV NPV VPN
PV-
34
Predicted values are not constants

• The predicted values depend on the sensitivity
  and on the specificity of the test as well as on
  the prevalence of the disease
• Will be different in different populations.

35
Relation between predictive values and
sensitivity / specificity
Disease
No
Yes

PPV TP/(TPFP)
Test

NPV TN/(TNFN)
TN
FN
36
Step 1 Specify the prevalence (Pr) of disease
Disease
No
Yes


Test


Pr
1-Pr
37
Step 2 Use sensitivity (Se) to distribute test
results among the diseased
Disease
No
Yes


Se Pr

Test


(1-Se)Pr

Pr
1-Pr
38
Step 3 Use specificity (Sp) to distribute test
results among the non-diseased
Disease
No
Yes

(1-Sp)(1-Pr)
Se Pr

Test

Sp(1-Pr)
(1-Se)Pr

Pr
1-Pr
39
Step 4 Determine the proportion testing positive
and the proportion testing negative
Disease
No
Yes

(1-Sp)(1-Pr)
Se Pr (1-Sp)(1-Pr)
Se Pr
Test

Sp(1-Pr)
(1-Se)Pr
(1-Se)Pr Sp(1-Pr)
Pr
1-Pr
40
Step 5 Calculate PPV and NPV with appropriate
expressions from Step 4
41
Relation between predictive values and
sensitivity / specificity
Increasing specificity ? increasing PPV
Increasing sensitivity ? increasing NPV
42
Relation between predictive values and prevalence
Increasing prevalence ? increasing PPV
Decreasing prevalence ? increasing NPV
43
PPV and NPV of a test according to the prevalence
(80 sensitivity and specificity)
100
80
NPV
60
Predictive value ()
40
20
PPV
0
0
25
50
75
100
Prevalence ()
44
Example Two different populations, SeSp90
Prevalence 50
? PPV 90
Not ill
Ill
TP
FP

Test
FN
TN

Prevalence 10
? PPV 50
45
Example Screening for human trypanosomiasis in
two settings

• CATT test
• Sensitivity 95
• Specificity 75
• Endemic area
• Prevalence 20
• Low endemic area
• Prevalence 0.5
• 100,000 tests performed in each area

46
Example Screening for human trypanosomiasis in
two settings
CATT test sensitivity 95 CATT test
specificity 75
Prevalence 20
Trypanosomiasis Trypanosomiasis Trypanosomiasis Trypanosomiasis Trypanosomiasis
Yes No Total
CATT 19,000 20,000 39,000
CATT 1,000 60,000 61,000
20,000 80,000 100,000
PPV 48.7 NPV 98.4
47
Example Screening for human trypanosomiasis in
two settings
CATT test sensitivity 95 CATT test
specificity 75
Prevalence 0.5
Trypanosomiasis Trypanosomiasis Trypanosomiasis Trypanosomiasis Trypanosomiasis
Yes No Total
CATT 475 24,875 25,350
CATT 25 74,625 74,650
500 99,500 100,000
PPV 1.90 NPV 98.97
48
To sum up

• Sensitivity and specificity
• intrinsic characteristics of a test
• capacity to identify the affected
• capacity to identify the non-affected
• independent from the disease prevalence
• Predictive values
• performance of a test in real life
• how to interpret a positive test
• how to interpret a negative test
• dependent on the disease prevalence

49
Thank you!QUESTIONS?