Better understanding diversity: achievements of DIRC research theme

1
Some Difficult Decisions are Easier without
Computer Support / TA Mammography, RT Diversity
/

Andrey A. Povyakalo (work together with E
Alberdi, L Strigini and P Ayton) andrey_at_csr.city
.ac.uk
DIRC workshop, Edinburgh, 16 March 2005
2
Computer Aided Detection for Mammography
Computer Aided Detection (CAD) Tool aims to
mark Regions of Interest (ROI) on a digitised
mammogram image to prevent overlooking by the
human reader applies a pattern recognition
algorithm claimed not to be a diagnostic tool
claimed that the potential for missed lesions
is not increased over routine screening
mammography when used as labelled
Prescribed Procedure Reader looks at original
mammogram and interpret it as usual, then
activates CAD and looks at a small low resolution
image of the mammogram with marked ROIs on it ,
then checks whether or not some ROIs have been
overlooked and... ... revises her/his
assessment, if necessary
3
Controversy

• US FDA (1998) use of the device improved the
  radiologist's detection rate from approximately
  80 out of 100 cancers to almost 88 out of 100
• Warren Burhenne, LJ et al. (2000) ...CAD
  prompting could have potentially helped reduce
  this false-negative rate by 77 (89 of 115)
  without an increase in the recall rate.
• Brem, RF et al. (2003) for every 100,000 women
  with breast cancer identified without the use of
  computer-aided detection, an estimated additional
  21,200 cancers would be found with the use of
  computer-aided detection. ...
• Freer, TW Ulissey MJ (2001) The use of CAD
  ... can increase the detection of early-stage
  malignancies without undue effect on the recall
  rate or positive predictive value for biopsy. (8
  more cancers of 49 found with CAD)
• Taylor, PM et al. (2004) this version of the
  ImageChecker would not have a significant impact
  on the UK screening programme...
• Gur, D et al. (2004) The introduction of
  computer-aided detection was not associated
  with statistically significant changes in recall
  and ... detection rates
• Alberdi, E et al (2004) Possible automation
  bias effects in CAD use ... may degrade human
  decision-making for some categories of cases
  under certain conditions...

4
HTA trial (University College London)

• 50 readers looked at 180 cases
• 60 cancers
• 120 normal cases (normals)
• in two conditions
• without computer support (unprompted session)
• with computer support (prompted session)
• to make a recall decision
• Rate of cancers much higher than in real working
  conditions
• CADT printout used instead of using real system
• Results
• the trial administrators found NO statistically
  significant impact of CAD on human performance

5
Trial data for cancers

• Sensitivity fraction of cases recalled by the
  reader without CAD
• Case difficulty fraction of readers missing
  the case without CAD
• Blue points mark ltcase, readergt pairs
  where the unaided decision was wrong and the
  decision supported by the CAD was correct
• Red points mark ltcase, readergt pairs
  where the unaided decision was correct but the
  decision supported by CAD was wrong

Readers ranked by their sensitivity
Cases ranked by their difficulty
6
Regression Estimates

• Difficulty of case i d(i)
• Sensitivity of reader j f(j)
• Probability that reader j recalling case i
• in the unprompted condition
• Pun (i, j) F( d(i), f(j) )
• in the prompted condition
• Ppr (i, j) G( d(i), f(j) )
• F, G -some functions found via logit regression
• Impact
• Imp( d(i), f(j) ) G( d(i), f(j) ) - F( d(I),
  f(j) )

7
Effect of CAD on probability of recalling cancer
(all cases)
The more sensitive readers hindered
Maximum effect
0.85
0.80
Fraction of cases recalled by the reader without
CAD (sensitivity)
0.75
0.70
The less sensitive readers benefit
0.65
Maximum damage
0.60
0.0
0.2
0.4
0.6
0.8
More of the difficult cases missed
More of the easy cases recalled
Fraction of readers missing the case without
CAD (case difficulty)
8
Effect of CAD on probability of recalling cancer
(cases with correct prompts)
The more sensitive readers hindered
Maximum effect
Fraction of cases recalled by the reader without
CAD (sensitivity)
The less sensitive readers benefit
Maximum damage
More of the easy cases recalled
Fraction of readers missing the case without
CAD (case difficulty)
9
Effect of CAD on probability of recalling cancer
(cases without correct prompts)
The more sensitive readers hindered
Fraction of cases recalled by the reader without
CAD (sensitivity)
Maximum damage
More of the difficult cases missed
Fraction of readers missing the case without
CAD (case difficulty)
10
Concordance of decisions

• More precisely Probability that two randomly
  selected readers both recall or not recall
  randomly selected case
• significantly greater in the prompted condition
  for
• all cases by
• 0.812 - 0.789 0.022 (95 CI 0.018, 0.027)
• correctly prompted cases by
• 0.849 - 0.834 0.015 (95 CI 0.010, 0.019)
• cases without correct prompts by
• 0.701 - 0.655 0.046 (95 CI 0.036, 0.056)
• Does the technology reduce the human diversity?

11
Conclusions

• Exploratory analyses to generate hypotheses
• Generated hypotheses to be tested with
  independent data
• Conjecture CAD helps the less sensitive
  radiologists
• The use of CAD by more sensitive radiologists
  is questionable
• use of CAD leads to more concordance between
  decisions of different radiologists
• Generalisation of results from studies with
  small number of participating radiologists is
  questionable
• Mechanisms?
• MIRCAD proposal submitted to EPSRC
• City, Edinburgh and UCL involved
• ___________
• similar to those by Warren Burhenne, LJ et al.
  (2000) - 5 readers, Brem, RF et al. (2003) - 7
  readers, Freer, TW Ulissey MJ (2001) - 2
  readers (Freer Ulissey),