The clinical value of diagnostic tests A well-explored but underdeveloped continent - PowerPoint PPT Presentation

About This Presentation
Title:

The clinical value of diagnostic tests A well-explored but underdeveloped continent

Description:

The clinical value of diagnostic tests A well-explored but underdeveloped continent J. Hilden : March 2006 The clinical value of diagnostic tests The diagnostic test ... – PowerPoint PPT presentation

Number of Views:135
Avg rating:3.0/5.0
Slides: 66
Provided by: jh488
Category:

less

Transcript and Presenter's Notes

Title: The clinical value of diagnostic tests A well-explored but underdeveloped continent


1
The clinical value of diagnostic tests A
well-explored but underdeveloped continent

  • J. Hilden March 2006

2
The clinical value of diagnostic tests The
diagnostic test and some neglected aspects of its
statistical evaluation.
  • Some aspects were covered in my
    seminar spring 2003

3
Plan of my talk
  • Historical sociological observations
  • Clinicometric framework
  • Displays and measures of diagnostic power
  • Appendix math. peculiarities

4
Plan of my talk
  • Historical sociological observations
  • Skud vildskud
  • - Diagnostic vs. therapeutic research
  • 3 key innovations some pitfalls
  • Clinicometric framework
  • Displays and measures of diagnostic power
  • Appendix math. peculiarities

5
A quantitative framework for diagnostics is much
harder to devise than for therapeutic trials.
  • Trials concern what happens observably
  • concern 1st order entities (mean effects)
  • Diagnostic activities aim at changing the docs
    mind
  • concern 2nd order entities (uncertainty /
    entropy change)

CONSORT gtgt 10yrs gtgt STARD CC 1993
CC 2003
6
  • In the 1970s
  • medical decision theory established itself
  • but few first-rate statisticians took notice.
  • Were they preoccupied with other topics, Cox,
    prognosis, trial follow-up ?
  • Sophisticated models became available for
    describing courses of disease conditionally on
    diagnostic data.
  • Fair to say that they themselves remained
  • a vector of covariates ?

7
Early history
  • Yerushalmy 1947
  • studies of observer variation
  • Vecchio
  • /BLACKWHITE/ Model 1966
  • - simplistic but indispensable
  • - simple yet often misunderstood?!
  • Warner 1960
  • congenital heart dis. via BFCI

important but not part of my topic today
8
Other topics not mentioned
  • Location (anatomical diagnoses)
  • and multiple lesions
  • Monitoring, repeated events, prognosis
  • Systematic reviews meta-analyses
  • Interplay between diagnostic test data
    knowledge from e.g. physiology
  • Tests with a therapeutic potential
  • Non-existence of prevalence-free
  • figures of merit
  • Patient involvement, consent

9
BFCI (Bayes Formula w. Conditional
Independence)
  • based on the assumption of CI
  • what does that mean?
  • Do you see why it was misunderstood?

Indicant variables independent condlly on
pts true condition
10
BFCI (Bayes Formula w. Conditional Independence)
  • Bayes based on the assumption of CI
  • - what does that mean?
  • There is no Bayes Theorem without CI
  • The BFCI formulae presuppose CI
  • (CI is a necessary condition for correctness)

No, CI is a sufficient condition whether it is
also necessary is a matter to be determined
and the answer is No.
Counterexample next picture !
11
Joint conditional distribution of two tests in
two diseases (red, green)
.0625 .0375 .1
.25 .15 .4
.4375 .0625 .5
.75 .25 1
.125 .075 .2
.1875 .1125 .3
.4375 .0625 .5
.75 .25 1
with 3 and 2 test qualitative outcomes
12
Vecchios /BLACKWHITE/ Model 1966
  • Common misunderstandings
  • The sensitivity and specificity are properties
    of the diagnostic test rather than of the
    patient population
  • They are closely connected with the ability of
    the test to rule out in

True only when the prevalence is intermediate
13
Plan of my talk
  • Historical sociological observations
  • Clinicometric framework
  • Displays and measures of diagnostic power
  • Appendix math. peculiarities

14
You cannot discuss Diagnostic Tests without
Some conceptual framework
  • A Case, the unit of experience in the clinical
    disciplines,
  • is a case of a Clinical Problem, defined by the
    who-how-where-why-what of a clinical encounter
  • or Decision Task.
  • We have a case population or
  • case stream (composition!) with a case flow
    (rate, intensity).

Clinicometrics, rationel klinik,
15
Examples
  • Each time the doc sees the patient we have a new
    encounter / case, to be compared with suitable
    statistical precedents and physio-
    pharmacology.
  • Prognostic outlook at discharge from hospital a
    population of cases discharges, not patients
    (CPR Nos.).

Danish Citizen No.
16
Diagnosis?
  • Serious diagnostic endeavours are always
    action-oriented
  • or at least
    counselling-oriented
  • i.e., towards what should be done so as to
    influence the future (action-conditional
    prognosis).
  • The truth is either
  • a gold standard test (facitliste), or
  • syndromatic (when tests define the disease,
    e.g. rheum. syndromes, diabetes)

in clinimetrics there is little need for that
word!
17
Example
  • The acute abdomen
  • there is no need to discriminate between
    appendicitis and non-app. (though it is fun to
    run an appendicitis contest)
  • What is actionwise relevant is the decision open
    up or wait-and-see?
  • ltThis is frequently not recognized in the
    literaturegt

18
In clinical studies the choice of sample, and
of the variables on which to base
one's prediction, must match the
clinical problem as it presents itself
at the time of decision making. In
particular, one mustn't discard subgroups
(impurities?) that did not become
identifiable until later prospective
recognizability !
Data collection
19
Purity vs. representativenessA meticulously
filtered case stream ('proven
infarctions') may be needed for patho- and
pharmaco-physiological research, but is
inappropriate as a basis for
clinical decision rules incl. cost studies.
Data collection
20
Consecutivity as a safeguard against selection
bias.Standardization (Who examines the
patient? Where? When? With access to clin.
data?)Gold standard the big problem !! w.
blinding, etc.Safeguards against change of data
after the fact.
Data collection
STARD !
21

Discrepant analysis
  • If the outcome is FALSE negative or positive,
  • you apply an arbiter test
  • in order to resolve the discrepant finding,
  • i.e. a 2nd, 3rd, reference test.
  • If TRUE negative or positive, accept !
  • The defendant decides who shall be allowed to
    give testimony and when

22
Randomized trials of diagn. tests
Digression
  • theory under development
  • Purpose design many variants
  • Sub(-set-)randomization, depending on the pt.s
    data so far collected.
  • Non-disclosure some data are kept under seal
    until analysis. No parallel in therapeutic
    trials!
  • Main purposes

23
Randomized trials of diagn. tests
  • when the diagnostic intervention is itself
    potentially therapeutic
  • when the new test is likely to redefine the
    disease(s) ( cutting the cake in a completely new
    way )
  • when there is no obvious rule of translation from
    the outcomes of the new test to existing
    treatment guidelines
  • 4) when clinician behaviour is part of the
    research question

end of digression
24
Plan of my talk
  • Historical sociological observations
  • Clinicometric framework
  • Displays and measures of diagnostic power
  • Appendix math. peculiarities

25
Displays measures of diagnostic power
  1. The Schism between
  2. ROCography
  3. VOIography

26
ROCography
  • classical discriminant analysis / pattern
    recognition
  • Focus on disease-conditional distribution of test
    results (e.g., ROC)
  • AuROC (the area under the ROC) is popular
    despite 1991 paper

27
VOI (value of information)
  • decision theory.
  • VOI increase in expected utility afforded by an
    information source such as a diagnostic test
  • Focus on posttest conditional distribution of
    disorders, range of actions and the associated
    expected utility and
  • its preposterior quantification.
  • Less concerned with math structure, more with
    medical realism.

28
VOI
  • Do we have a canonical guideline?
  • 1) UTILITY
  • 2) UTILITY / COST
  • Even if we don't have the utilities
  • as actual numbers, we can use this
  • paradigm as a filter
  • evaluation methods that violate it are
  • wasteful of lives or resources.
  • Stylized utility (pseudo-regret functions) as a
    (math. convenient) substitute.

29
VOI
  • Def. diagnostic uncertainty as expected regret
  • (utility loss, relative to if you knew what ailed
    the pt.)
  • Diagnosticity measures (DMs)
  • Diagnostic tests
  • should be evaluated in terms of
  • pretest-posttest difference
  • in diagnostic uncertainty.
  • Auxiliary quantities like sens and spec
  • go into the above.

so much as to VOI principles
30
Diagnosticity measures and auxiliary quantities
NOT

BLACKWHITE
31
Diagnosticity measures and auxiliary quantities
  • Sens (TP), spec (TN) nosografic distrib.
  • PVpos, Pvneg diagnostic distr.test result
  • Youdens Index Y sens spec 1
  • 1 (FN) (FP)
  • det(nosog. 2X2)
  • (TP)(TN)(FP)(FN)
  • 2(AuROC ½)
  • AuROC
  • sensspec / 2

ROC
Y 1
FN
TP
Y 0
BLACKWHITE
FP
TN
32
Diagnosticity measures and auxiliary quantities
  • Sens, spec nosografic distribution
  • LRpos, LRneg slopes of segments

The Likelihood ratio term is o.k. when
diagnostic hypotheses are likened to scientific
hypotheses
ROC
Y 1
FN
neg
pos
TP
Y 0
BLACKWHITE
FP
TN
33
Diagnosticity measures and auxiliary quantities
  • Utility index (sens) x Y.
  • ... is nonensense

ROC
Y 1
FN
TP
Y 0
BLACKWHITE
FP
TN
34
Diagnosticity measures and auxiliary quantities
  • DOR (diagnostic odds ratio)
  • (TP)(TN) / (FP)(FN)
  • infinity in this example
  • even if TP is only 0.0001. ... careful!

ROC
Y 1
FN
Y 0
BLACKWHITE
TP
FP 0
TN
35

Ideal test

Three test outcomes
36

FREQUENCY-WEIGHTED ROC

37

implies constant misclassification

Continuous test Cutoff at x c minimizes
misclassification
38

39

Parallelogram

Two binary tests and their 6 most important
joint rules of interpretation
40
Overhull implies superiority

slope f(x) / g(x)
41


Essence of the proof that overhull implies
superiority



42
Utility-based evaluation in general

43
Utility-based evaluation in general


44
Utility-based evaluation in general
?(pdy qdx )mina (LaD pdy La,nonD qdx)/(pdy
qdx) is how it looks when applied to the ROC
(which contains the required information about
the disease-conditional distributions).


45
The area under the ROC (AuROC) is misleading
  • You have probably seen my
  • counterexample before.
  • Assume D and non-D
  • equally frequent and also
  • utilitywise symmetric

Medical Decision Making 1991 11 95-101
46

Two Investigations
47

48

Expected regret (utility drop relative to perfect
diagnoses)

Bxsens
The tent graph
Cxspec
pretest
49

50

51

52

53

54

55

Good bad pseudoregret functions

Shannon-like
Brier-like
56
Plan of my talk
  • Historical sociological observations
  • Clinicometric framework
  • Displays and measures of diagnostic power
  • Appendix math. peculiarities

57

58

59
(No Transcript)
60
(No Transcript)
61

62

63

64

LRpos LRneg 1
65
End of my talk
  • Thank you !
  • Tak for i dag !
Write a Comment
User Comments (0)
About PowerShow.com