INS Investigators Workshop: Methods for SingleCase Studies in Neuropsychology

1
INS Investigators WorkshopMethods for
Single-Case Studies in Neuropsychology

• John R. Crawford
• School of Psychology
• College of Life Sciences and Medicine
• Kings College
• University of Aberdeen
• and
• School of Psychology
• Flinders University of South Australia
• j.crawford_at_abdn.ac.uk

www.abdn.ac.uk/psy086/dept/
Cognitive Neuroscience Research Group
2
Collaborators

• Prof Paul H Garthwaite The Open University
• also
• Prof David C Howell University of Vermont
• Prof Keith R Laws University of Hertfordshire
• Prof Annalena Venneri University of Hull
• Dr Colin D Gray University of Aberdeen
• Prof Addelchi Azzalini University of Padova
• (Dr Sytse Knypstra
  University of Groningen)

3
The Importance of Dissociations
Dissociation is the key word of
neuropsychology. (Rossetti Revonsuo, 2000, p.
2)
4
The Case for Single Cases
Studies in groups of patients which aim at
elucidating the neurological and functional
architecture of mental processes are useless and
harmful, since they provide misleading results.
The only appropriate method is to study
individual patients (Vallar, 2000, p. 334)
5
The need for methodological rigour in single-case
studies
If advances in theory are to be sustainable they
must be based on unimpeachable methodological
foundations. (Caramazza McCloskey, 1988,
p.619).
6
Evaluating Tests for Deficits in Single-Case
Studies

• Massive revival of interest in single-case
  studies in neuropsychology and neurology
• The arguments for single-case studies over group
  studies are viewed by many as compelling
• However, it is clear that they present
  difficulties when it comes to statistical
  analysis
• This aspect of single-case studies has been
  relatively neglected

7
Single-case research The three basic approaches
to drawing inferences concerning a patients
performance

• Patient is administered fully standardized
  neuropsychological tests and performance is
  compared to large sample normative data
• At other extreme, patients performance is not
  referenced to normative data or control
  performance i.e., analysis is limited to
  intra-individual comparisons
• Patient is compared to a (modestly sized) matched
  control sample

8
Limitations of the fully standardized approach

• Can only be used in fairly circumscribed
  situations because
• New constructs are constantly emerging in
  neuropsychology
• In contrast, collection of norms is a long and
  arduous process
• Even where norms are available, may not be
  applicable to patient

9
Dangers of the intra-individual approach

• Results can be very misleading as performance is
  not referenced to normal performance
• Category specificity literature provides a good
  example of dangers
• Reports of apparently striking dissociations
  between naming of living versus non-living things
  and even within these categories (Broccolis
  area?)
• In vast majority of these studies inferences are
  drawn on the basis of chi-square tests comparing
  a patients living and non-living naming

10
Laws, Gale, Leeson Crawfords (2005) study on
living / non-living naming

• Laws et al. examined cases of AD who had or had
  not been classified as exhibiting a dissociation
  using intra-individual approach (chi-square test)
• It was found that the performance of some
  patients with dissociations was not unusual
  when referenced to control performance
• Moreover, patients who had not been identified as
  exhibiting dissociations were identified as such
  when performance was referenced to controls
• In one case a patient classified as exhibiting a
  dissociation in favour of non-living things was
  found to exhibit a dissociation in the opposite
  direction

11
To recap

• Three basic methods of forming inferences
• Referring patients performance to large scale
  normative data will often not be possible
• Intra-individual approach is very problematic
• Therefore, third approach is commonly employed
  i.e. patients performance is referred to that of
  a matched control sample
• We will now turn to how such data are analysed
  starting with the basic question of how we detect
  a deficit

12
Testing for a deficit in single-case studies the
standard method

• Patients performance is converted to a standard
  score based on mean and SD of control sample and
  referred to table of areas under the normal curve
• The statistics of the control sample are treated
  as population parameters
• When sample size is large this is not too much of
  a problem as the statistics provide sufficiently
  accurate estimates of the parameters
• However, large sample sizes are rare in the
  single-case literature

13
Testing for a deficit in single-case studies
using Crawford Howells (1998) proposed method

• Uses formula set out by Sokal and Rohlf (1995)
• Modified t-test tests hypothesis that patient
  did not come from the control population (under
  null hypothesis patient is an observation from
  this population)
• Control sample statistics are treated as
  statistics
• Crawford Garthwaite (2002) also developed
  method of setting confidence limits on
  abnormality of score (using non-central t
  distributions)

14
Comparison of two methods for testing for a
deficit Type I errors (Crawford Garthwaite,
Neuropsychology,2005)

• Monte Carlo simulation study
• 5 control sample sizes (N) were examined 5, 10,
  20, 50 and 100
• For each value of N one million observations of N
  1 were drawn from a normal distribution
• The first N observations were taken as the
  control sample data and the N1th observation as
  the control case
• The alternative tests for deficits were applied
  to these data and the percentage of Type I errors
  compared to the specified rate of 5

15
Monte Carlo simulation Sampling from the control
population
Step (2) Get machine to repeat this one million
times
Step (3) Meanwhile go and get yourself a
16
(No Transcript)
17
Comparison of two methods for testing for a
deficit Type I errors (Crawford Garthwaite,
2005)
18
Conclusion

• Type I error rate is under control when Crawford
  Howells test is used to test for a deficit at
  all values of N
• Type I error rate is markedly inflated when z is
  used to detect a deficit with small control
  samples
• z will also exaggerate the abnormality of the
  patients score

19
Departures from normality

• Both z and modified t-test assume control data
  are drawn from normal distribution
• However, in single-case studies there is often
  evidence of negative skew in scores of the
  control samples (ie control mean50, SD10, but
  max score 55)
• We have run Monte Carlo simulations to examine
  control of Type I error rate when control data
  are non-normal
• Same method as in previous study except that the
  N1 observations were sampled from distributions
  that were skew and /or leptokurtic

20
Sampling from negatively skewed and / or
leptokurtic distributions
Step (2) Repeat this one million times
Step (3) Meanwhile go and get yourself a
21
Results of a Monte Carlo study Robustness in
face of moderate skew
22
Effects of departures from normality on methods
for testing for a deficit Conclusions

• Negative skew and / or leptokurtosis can inflate
  the Type I error rate for both z and Crawford
  Howells test but the effects are not severe
  (Crawford, Garthwaite, Azzalini, Howell, Laws,
  Neuropsychologia, in press)
• The Type I error rate is markedly inflated for z
  for small N but this is due more to treating
  statistics as parameters

23
The Internet

• Most of calculations involved with these methods
  are simple (exception being CLs)
• However, still tedious and error prone
• Therefore, we have written computer programs that
  implement these methods
• Freely available on the web www.abdn.ac.uk/psy086
  /dept/SingleCaseMethodsComputerPrograms.HTM
• Calculations can be performed literally in seconds

24
(No Transcript)
25
In this example, the patients score is
significantly below controls and so we conclude
he/she has a deficit. Also, it is estimated that
only 1.13 of the control population would
exhibit this poor a score the 95 CI on this
estimate of abnormality is from 0.05 to 4.68
26
Modifed T-Test Versus Modified ANOVA

• Mitchell and colleagues (Mycroft et al, 200
  Mitchell et al, 2004) have criticised the
  foregoing method
• They argue that (a) a notional patient population
  will have markedly increased variance relative to
  the control population, and (b) our method will
  therefore produce inflated Type I errors
• Mitchell and colleagues propose an ANOVA that
  employs more conservative critical values to
  overcome this perceived problem

27
Modifed T-Test Versus Modified ANOVA

• We believe there are two major problems with
  Mitchell et als position
• The argument over Type I errors is untenable
  (Crawford et al, Cognitive Neuropsychology, 2004)
  
• Statistical power to detect a deficit is very low
  for Mitchell et als method (Crawford
  Garthwaite, Cognitive Neuropsychology, in press)

28
A graphic illustrating Mitchell et als.
scenario A notional patient population (gray
line) has same mean as controls (dark line) but
has greater variability
This scenario is not realistic If the means do
not differ but patients are more variable, then
scores below control mean must be exactly
balanced by scores above control mean
29
If the patient mean is lower than control mean
(even marginally) then issue of Type I error does
NOT arise a deficit is present and the question
is whether it can be detected (i.e., it is a
power issue)
30
Power to detect a (2 SD) deficit comparison of
three methods (Crawford Garthwaite, Cognitive
Neuropsychology, in press)
31
Power to detect a (2 SD) deficit comparison of
three methods (Crawford Garthwaite, Cognitive
Neuropsychology, in press)
32
PART 2Dissociations in Neuropsychology and
Statistical Tests on Differences
33
DISSOCIATIONS

• In neuropsychology, deficits are of limited
  theoretical interest unless they are accompanied
  by preserved or less impaired performance on
  other tasks i.e. the aim of many single-case
  studies is to demonstrate dissociations of
  function

34
Conventional Definition of a Classical
Dissociation
If patient X is impaired on task 1 but performs
normally on task 2, then we may claim to have a
dissociation between tasks (Ellis and Young,
1996, p. 5)
35
A Classical Dissociation (based on Shallice,
1988)
Performance
Task X
Task Y
36
The Importance of Dissociations
Dissociations play an increasingly crucial role
in the methodology of cognitive neuropsychology
they have provided critical support for several
influential, almost paradigmatic, models in the
field. (Dunn Kirsner, 2003, p. 2)
37
Criteria for Dissociations Three Problems

• What constitutes a deficit and being within
  normal limits is very poorly specified
• One half of the typical definition essentially
  involves an attempt to prove the null hypothesis
• A patients score on the impaired task could
  lie just below the critical value for defining
  impairment and the performance on the other test
  lie just above it (see Caramazza Shelton, 1998
  for similar point)

38
Problems with Conventional Criteria for a
Classical Dissociation
Performance
Task X
Task Y
39
Crawford, Garthwaite Gray ( 2003)
40
Potential Solutions to the Three Problems

• Crawford et al. (2003) provided fully explicit
  criteria for a deficit (using Crawford Howells
  test)
• They also introduced a requirement that the
  patients performance on Task X should be
  significantly poorer than performance on Task Y
• This criterion deals with the problem of trivial
  differences
• It also provides us with a positive test for a
  dissociation (thereby lessening reliance on what
  boils down to an attempt to prove the null
  hypothesis of no deficit or impairment on Task Y)

41
Crawford et als. criteria for a classical
dissociation
Y not significantly different from controls on
Crawford Howells test (one-tailed)
Performance
X significantly different from Y
X significantly below controls on Crawford
Howells test (one-tailed)
Task X
Task Y
42
Testing for a difference between a patients
performance on Tasks X and Y

• How should we test for significant difference
  between a patients score on Tasks X and Y ?
• In most single-case studies the two tasks of
  interest will have different means and SDs
• For example a patients performance on a ToM task
  with (mean35, SD12) is to be compared with
  performance on an executive task (mean22 SD6)
• In order to meaningfully compare performance it
  is necessary to standardize scores on the two
  tasks

43
The Payne and Jones method

• A long established method is that of Payne
  Jones (1957)

44
Testing for a difference between a patients
performance on Tasks X and Y Crawford, Howell
Garthwaite (1998) method
45
Revised Standardized Difference Test (Crawford
Garthwaite, 2005b Garthwaite Crawford, 2004)

• Looks nasty but is essentially of familiar form

46
Monte Carlo Evaluation of tests for differences
between tasks

• 5 control sample sizes (N) were examined 5, 10,
  20, 50 and 100
• For each value of N and for each of 4 values of r
  (the correlation between tasks), one million
  pairs of observations of N 1 were drawn from a
  bivariate normal distribution
• The first N pairs were taken as the control
  sample data and the N1th pair was as the control
  case
• The alternative tests for differences were
  applied to these data and the percentage of Type
  I errors compared to the specified rate of 5

47
Simulation study of Type I errors for tests on
differences
Meanwhile, have a noodle about on the
48
Monte Carlo simulation Type I error rate for
Revised Standardized Difference Test (rxy 0.5 in
this example)
49
Monte Carlo simulation Type I error rate for
Revised Standardized Difference Test (rxy 0.5 in
this example)
50
Conclusions

• Payne and Jones method very poor control with
  modest Ns
• Although Crawford et als. (1998) test is a
  marked improvement it is clear that it does not
  follow a t-distribution and yields inflated Type
  I error rates
• In contrast, RSDT controls the Type I error rate

51
Evaluating criteria for classical dissociations

• To recap we have considered two sets of criteria
  for detecting dissociations the conventional
  criteria and Crawford Garthwaites (2005b)
  criteria
• We now have suitable test for Crawford
  Garthwaites third criterion (i.e. it requires a
  significant difference between a patients scores
  on X and Y)
• We (Crawford Garthwaite, 2005a,
  Neuropsychology) have examined performance of
  these two sets of criteria
• Same approach as used for evaluating foregoing
  tests i.e. sample from bivariate distributions
  but apply the sets of criteria rather than
  individual tests for components of these criteria

52
Type I error rate for Crawford Garthwaites
(2003 2005b) criteria and conventional criteria
for a classical dissociation (in this example rxy
0.5
53
Type I error rate for Crawford Garthwaites
(2003 2005b) criteria and conventional criteria
for a classical dissociation (in this example rxy
0.5
54
Evaluating criteria for a classical dissociation
Conclusions

• Conventional criteria for a classical
  dissociation will misclassify a worryingly high
  percentage of healthy controls as exhibiting a
  classical dissociation regardless of the size of
  the control sample (rate was as high as 18.6 in
  one of the scenarios)
• In contrast, Crawford Garthwaites (2003
  2005b) criteria are conservative i.e. very low
  percentage of controls misclassified
• Results underline importance of testing the
  difference between patients X and Y scores
• Crawford Garthwaites criteria were relatively
  robust in face of skewed control data

55
(No Transcript)
56
In this example, the patient is classified as
exhibiting a dissociation by Crawford
Garthwaites (2005b) criteria. (Crucially) the
RSDT shows that there is a significant difference
between the patients scores on the two tasks.
Task X is significantly below controls but Task Y
is not therefore it is a classical dissociation.
57
Further evaluation of criteria for a classical
dissociation

• Up to this point Type I errors have been defined
  as incorrectly identifying a control as
  exhibiting a dissociation
• However, there is another form of Type I error
  that should be considered
• Alternative definition of Type I error
  misclassifying a patient who has equivalent
  deficits on X and Y as exhibiting a classical
  dissociation

58
Lesion study Type I errors now defined as
misclassifying a patient
59
Type I error rate for the competing criteria
Type I errors defined as identifying a patient as
exhibiting a classical dissociation
60
Type I error rate for the competing criteria
Type I errors defined as identifying a patient as
exhibiting a classical dissociation
61
Evaluating criteria for a classical dissociation
Further conclusions

• Performance of conventional criteria for a
  classical dissociation is extremely poor very
  high percentages of patients will be incorrectly
  classified as exhibiting a classical dissociation
  (close to 50 in some scenarios)
• In contrast, Crawford Garthwaites (2005)
  criteria are much more conservative i.e.
  percentage is below 5 in most scenarios
• These results further underline importance of
  testing the difference between patients X and Y
  scores

62
Power to detect a classical dissociation

• Up to this point concern has been with Type I
  errors, i.e. false positives
• However, we should also be concerned with the
  statistical power of criteria for dissociations
  i.e. the ability of these criteria to avoid false
  negatives
• This issue has not previously been examined
  empirically
• It does not make sense to examine power unless
  the Type I error rate is under reasonable control
• Therefore, power only examined for Crawford
  Garthwaites (2005b) criteria

63
Why power to detect a dissociation will be low

• Crawford and colleagues have argued that power
  is almost inevitably low-to-moderate in
  single-case studies
• An individual patient rather than a sample of
  patients is compared to a control sample
• The control sample itself is usually modest in
  size
• The existence of substantial individual
  differences in premorbid competencies is a
  further factor

64
Why power to detect a dissociation will be low
1 SD
Mean
-1 SD
Task X
Task Y
65
Lesion study Power to detect a dissociation
66
Simulation results Power to detect a classical
dissociation(for control sample N of 20)
67
Crawford Garthwaite (2005a,
Neuropsychology)Power to detect a dissociation
Conclusions

• Results confirm that, if the Type I error rate is
  to be controlled, power is low in single-case
  studies
• Not surprisingly, power is higher with large
  control samples therefore, control sample Ns
  should be larger than is typical currently
• This is not unreasonable if a researcher
  believes single-case studies are more useful than
  group studies then she/he should be willing to
  expend the effort
• More encouragingly, power is higher when tasks
  are moderately to highly correlated

68
Single-Case Methods Overall Summary

• Massive revival of interest in single-case
  studies in neuropsychology and neurology
• The arguments for single-case studies over group
  studies are viewed by many as compelling
• However, it is clear that they present
  difficulties when it comes to statistical
  analysis
• The methods and criteria commonly used in
  single-case studies are problematic
• However, many of the problems can be overcome
  using the methods outlined
• These latter methods are rigorous but easy to
  apply using the accompanying computer programs

69
Finally

• Regression equations can play a useful role in
  single-case research and clinical practice
• For example, attempting to detect a deficit by
  comparing an obtained score with a predicted
  score based on demographic variables or a measure
  of premorbid ability
• In neuropsychology, inferences concerning
  discrepancies between predicted scores and
  obtained scores are typically made using the
  standard error of estimate
• This method produces inflated Type I error rates
  (Crawford Garthwaite, Neuropsychology, in press)

70
Finally

• Building on earlier work by Crawford Howell
  (1998b) we (Crawford Garthwaite, in press) have
  recently proposed and evaluated an alternative
  method that controls Type I error rate
• The method also produces confidence limits on the
  abnormality of the discrepancy
• Methods implemented in accompanying computer
  program
• Poster on this topic later today

71
THE END
Web page summarizing our work on single-case
methods www.abdn.ac.uk/psy086/dept/SingleCaseMet
hodology.htm
Web page containing the computer programs
referred to in this presentation www.abdn.ac.uk/
psy086/dept/SingleCaseMethodsComputerPrograms.HTM
A reference list for the methods is provided in
your handout following the copy of this slide
end
72
References
Crawford, J. R. (2004). Psychometric foundations
of neuropsychological assessment. In L. H.
Goldstein J. E. McNeil (Eds.), Clinical
neuropsychology A practical guide to assessment
and management for clinicians (pp. 121-140).
Chichester Wiley. Crawford, J. R.,
Garthwaite, P. H. (2002). Investigation of the
single case in neuropsychology Confidence limits
on the abnormality of test scores and test score
differences. Neuropsychologia, 40,
1196-1208. Crawford, J. R., Garthwaite, P. H.
(2004). Statistical methods for single-case
research Comparing the slope of a patient's
regression line with those of a control sample.
Cortex, 40, 533-548. Crawford, J. R.,
Garthwaite, P. H. (2005a). Evaluation of criteria
for classical dissociations in single-case
studies by Monte Carlo simulation.
Neuropsychology, 19, 664-678. Crawford, J. R.,
Garthwaite, P. H. (2005b). Testing for
suspected impairments and dissociations in
single-case studies in neuropsychology
Evaluation of alternatives using Monte Carlo
simulations and revised tests for dissociations.
Neuropsychology, 19, 318-331.
73
References contd
Crawford, J. R., Garthwaite, P. H. (in
press-a). Comparing an individual's predicted
test score from a regression equation with an
obtained score a significance test and point
estimate of abnormality with accompanying
confidence limits. Neuropsychology. Crawford,
J. R., Garthwaite, P. H. (in press-b). Methods
of testing for a deficit in single case studies
Evaluation of statistical power by Monte Carlo
simulation. Cognitive Neuropsychology. Crawford,
J. R., Garthwaite, P. H., Azzalini, A., Howell,
D. C., Laws, K. R. (in press). Testing for a
deficit in single case studies Effects of
departures from normality. Neuropsychologia. Cra
wford, J. R., Garthwaite, P. H., Gray, C. D.
(2003). Wanted Fully operational definitions of
dissociations in single-case studies. Cortex, 39,
357-370. Crawford, J. R., Garthwaite, P. H.,
Howell, D. C., Gray, C. D. (2004). Inferential
methods for comparing a single case with a
control sample Modified t- tests versus Mycroft
et al's. (2002) modified ANOVA. Cognitive
Neuropsychology, 21, 750-755.
74
References contd
Crawford, J. R., Garthwaite, P. H., Howell, D.
C., Venneri, A. (2003). Intra-individual
measures of association in neuropsychology
Inferential methods for comparing a single case
with a control or normative sample. Journal of
the International Neuropsychological Society, 9,
989-1000. Crawford, J. R., Howell, D. C.
(1998a). Comparing an individuals test score
against norms derived from small samples. The
Clinical Neuropsychologist, 12,
482-486. Crawford, J. R., Howell, D. C.
(1998b). Regression equations in clinical
neuropsychology An evaluation of statistical
methods for comparing predicted and obtained
scores. Journal of Clinical and Experimental
Neuropsychology, 20, 755-762. Crawford, J. R.,
Howell, D. C., Garthwaite, P. H. (1998). Payne
and Jones revisited Estimating the abnormality
of test score differences using a modified paired
samples t-test. Journal of Clinical and
Experimental Neuropsychology, 20,
898-905. Garthwaite, P. H., Crawford, J. R.
(2004). The distribution of the difference
between two t-variates. Biometrika, 91,
987-994. Laws, K. R., Gale, T. M., Leeson, V.
C., Crawford, J. R. (2005). When is category
specific in Alzheimer's disease? Cortex, 44,
452-463.