Rohlings Interpretive Method: Use of MetaAnalytic Procedures for Single Case Data Analysis

1
Rohlings Interpretive Method Use of
Meta-Analytic Procedures for Single Case Data
Analysis

• Martin L. Rohling, Ph.D. University of South
  Alabama L. Stephen Miller, Ph.D. University of
  Georgia

2
Introduction to the RIM Process

• Flexible battery (multiple measure) use
• Is the most frequently cited model of assessment
  among neuropsychologists.
• Only 15 of neuropsychologists use a fixed
  battery (Sweet, et al, 2000, TCN)
• Regarding the suitability, practicality, and
  usefulness of any fixed battery
• We know of no batteries that fully satisfy these
  criteria.
• (Lezak, Howieson Loring 2004,
  Neuropsychological Assessment, 4th Edition, p
  648.)

3
Advantages of Flexible Battery Approaches

• Dynamic.
• Cover 1 or many domains.
• Flexible, can be adapted for each patient.
• Can oversample domains.
• Well suited as a hypothesis-driven approach.

4
Potential Problems - Flexible Battery Approaches

• Inflated error rates.
• Multicollinearity.
• Weighting decision problems.
• Unknown veracity/reliability of sets of tasks.
• Human judgment errors.

5
Human Judgment Errors(Wedding Faust, 1989, ACN)

• Hindsight bias.
• Confirmatory bias.
• Overreliance on salient data.
• Under-utilization of base rates.
• Failure to take into account co-variation.

6
RIM Potential

• Judgment errors can threaten reliability and
  validity of multiple measure test batteries.
• RIM was designed to reduce these effects.
• Based on meta-analytic techniques.
• Uses a linear combination of scores placed on a
  common metric.

7
RIM Potential

• A strategy that produces summary results
  analogous to those generated in a fixed-battery
  approach (e.g., HII, GNDS, AIR).
• Takes advantage of psychometric properties of
  same metric data e.g. T-Scores.

8
Todays Intent

• Present a set of procedures that allows for a
  quantitatively-based comparison of an overall
  battery of measures.
• Non-specific to battery measures themselves.
• Can be used for any individual patient.
• Demonstrate importance and practicality of use
  of established statistical indices.
• (e.g., alpha, beta, effect size).

9
Todays Intent (contd)

• Present a data format for any set of measures to
  be inspected at
• Global level (OTBM)
• Domain level (DTBM)
• Test measure level (ITBM)
• Present a series of calculations to assist in the
  generation of these indices.
• Present Steps in conjunction with clinical
  judgment from an informed position.

10
RIM Categories

• Symptom Validity (SV) Tests.
• Emotional / Personality (EP) Measures.
• Estimated Premorbid General Ability (EPGA).
• Test Battery Means.
• Overall (OTBM), Domain (DTBM), Instrument
  (ITBM).
• Cognitive Domains
• VC, PO, EF, AML, VML, AW, PS
• Non-Cognitive Domains.
• PM, LA, SP

11
Sample RIM Summary Table
12
Sample RIM Graphic Display
13
Brief of RIM Steps

• There are 24 steps to the RIM process
• 17 calculation steps
• Advice on design of the battery
• Calculation of summary statistics
• Generation of graphic displays
• 7 interpretative steps.
• Detail a systematic procedure for use of the
  statistical summary table and graphic displays
  to
• Assess and verify summary data.
• Identify strengths/limitations of current data.
• Obtain a reliable diagnosis.
• Develop tx plans based on sound judgments.
• We briefly review each step in just a moment.

14
Support for the RIM Process

• Rational support/reasoning Reduce clinical
  judgment errors.
• The RIM is a Process, not a program.
• A way of formulizing your thinking and
  interpretation of your data.
• This is operationalizing what you already do.

15
Support for the RIM ProcessSpecific Advantages

• Psychometric properties at level with fixed,
  co-normed batteries, without their limitations.
• Flexibility of test selection.
• Flexibility of theoretical view of cognition
  (domain structure)

16
Support for the RIM ProcessSpecific Advantages

• Quantitatively support your conclusions and
  interpretations
• Statistical evaluation
• Measure of confidence in findings
• Measure of limitations of findings
• Ability to present data at different levels of
  interpretation
• Greater defensibility

17
The RIM as a Procedure of Specific Steps
18
RIM Steps 1-4 Summary Data

• Design administer battery.
• Use well standardized recently normed tests.
• Estimate premorbid general ability.
• Use Reading (WTAR), Regression (OPIE-III),
  academic records (rank, SAT, ACT).
• Convert test scores to a common metric.
• We recommend T scores, but z or SS OK too.
• Assign scores to domains.
• Factor analysis to support assignment (Tulsky et
  al., 2003)

19
RIM Steps 5-8 Summary Data

• Calculate domain M, sd, n.
• Calculate test battery means (TBM).
• Overall TBM All scores, large N high power.
• Domain TBM Avoids domain over weighting.
• (e.g., attention memory).
• Instrument TBM One score per norm sample.
• Calculate p for heterogeneity.
• Have you put apples oranges together?
• Determine categories of impairment.
• Recommend using of Heaton et al. (2003).

20
RIM Steps 9-12 Summary Data

• Determine of test impaired.
• Analogous to Halstead Impairment Index
• scores below cutoff / total of scores
• Calculate ES for all summary stats.
• Use Cohens d (Me Mc) / SD pooled
• Calculate confidence interval for stats.
• 90 CI 1.645 x SEM
• Upper limit of performance for impair.
• Look for overlap between 90 CI of EPGA (lower)
  Summary Stats (upper)

21
RIM Steps 13-17 Summary Data

• Conduct one-sample t tests.
• Use EPGA as reference point
• Conduct a between-subjects ANOVA.
• Looking for strengths weaknesses
• Conduct power analyses.
• Only needed for those NS differences
• Sort scores for visual inspection.
• Graphically display summary statistics.

22
RIM Steps 18-20 Interpretation

• Assess battery validity.
• Examine the Symptom Validity scores.
• Caution in accepting low power results.
• Look at heterogeneity of summary stats.
• Normative sample unrepresentative of patient.
• Scores assigned to wrong domain.
• Inconsistent performance on construct measures.
• Examine influence of psychopathology.
• Examine scores for heterogeneity.
• Check OTBM, DTBM, ITBM impairment.

23
RIM Steps 21-24 Interpretation

• Examine strengths/weaknesses looking at
• Confidence intervals overlap.
• Results from one-sample t tests.
• Results of ANOVA.
• TI show differences otherwise not evident.
• Determine if pattern existed premorbidly.
• Examine non-cognitive domains.
• Psychomotor, Lang/Aphasia, Sensory Percept
• Explore Type II errors need more tests?
• Examine sorted T-scores
• Look for patterns missed by summary stats.

24
RIM Sample Case 1 Obvious TBI

• Age 37
• Handed Left
• Race Euro-American
• Sex Female
• Ed 14 years
• Occup Nursing
• Marital Sep. 10 yrs
• Living Camper in parents backyard

• Reason for Referral
• TBI in head-on boat accident. Propeller hit pt in
  right parietal-occipital lobe (LOC 7 days GCS
  3). Eval. to determine capacity for medical
  financial decisions, parenting skills,
  occupational prognosis, disability status.
  Significant emotional, behavioral, occupational,
  and social problems pre-TBI.

25
RIM Sample Case 1 Obvious TBI
26
RIM Sample Case 1 Obvious TBI
27
TBI Dose Response CurvesDikmen ESs Meyers T
Scores
28
Return to Work Study OTBMs for 4 Groups of TBI
Survivors
29
RIM Sample Case 2 Obvious TBI Normal
Distribution of T Scores
30
RIM Sample Case 2 Subtle Diabetes

• Reason for Referral
• 2 yrs dangerous work habits. Eval to see if
  atrial fib Type II diabetes impairs cognition.
  Hospitalized TIA-like Sx. Admitted to problems
  for 20 yrs, cardiac dysrhythmia bradycardia,
  pacemaker, blood sugar difficult to manage,
  family Hx of heart disease diabetes.

• Age 55
• Handed Right
• Race Euro-American
• Sex Male
• Ed 13 years
• Occup Mechanic
• Marital Married 20 yr
• Living at home w/wife

31
RIM Sample Case 2 Subtle Diabetes
32
RIM Sample Case 2 Subtle Diabetes
33
RIM Sample Case 2 Subtle Diabetes Normal
Distribution of T Scores
34
RIM Critiques Concern 1

• The method of calculating the standard deviations
  (SDs) for summary statistics and domain scores is
  incorrect.
• Since many of the remaining steps of the RIM
  depend on the use of these SDs, this error is
  magnified in the subsequent steps.
• SDs statistically can not exceed 9.99 and are
  more likely to be around 6.4

35
Response 1 RIM Means 4 Large Datasets
36
Response 1 Inter-Individual Ms SDs
37
Response 1 RIM SDs 4 Large Datasets
38
Response 1 Intra-Individual Ms SDs
39
RIM Critiques Concern 2

• More false-positives then clinical judgment.
• Palmer et al. (2004) expressed concern that
• We failed to distinguish statistical from
  clinical significance.
• This failure is a critical error that precludes
  the prudent clinician from using the RIM.

40
Response 2 RIM vs. Manual Detecting Differences
Overall
41
Response 2 RIM vs. Manual Detecting
Differences ES
42
Response 2 RIM vs. ManualDetecting Differences
Scaled Scores
43
RIM Critiques Concern 3

• Clinicians who use the RIM will
• Idiosyncratically assign scores to cognitive
  domains.
• This will result in low inter-rater reliability
  in analysis diagnosis.

44
RIM Critiques Concern 4

• Scores on domains are unit weighted, which
  introduces error.
• Willson Reynolds (2004) said scores load on
  multiple domains. Assignment to domains weights
  depend on
• Battery of tests administered.
• Patients whose test scores are being examined.

45
Response 4 Cross-Validation Unit Wts

• Conducted 4 multiple reg. on 457 pts WAIS-R.
• Split sample in ½ - assess shrinkage.
• Regressed patients verbal subtests onto PIQ.
• Generated ideal weights for the 1st ½ of sample.
• Used wts to predict PIQs in the 2nd ½ of sample.
• Pre-PIQs regressed on actual PIQs 2nd ½ sample.
• Also, generated weights for the 2nd ½ of sample.
• Use Pre-PIQs regress on actual PIQs 1st ½
  sample.
• Repeated, except performance subtests predict VIQ
• split sample ½ generate same statistics as
  before.

46
Response 4 Cross-Validation Unit Wts

• Purpose of these procedures
• How much variance in wts. is sample specific.
• Amount of shrinkage using cross-validated wts.
• Shrinkage error compared to error introduced by
  using unit wts vs. ideal wts.
• Results 98 of the variance accounted for with
  unit wts. Compared to ideal weights.
• Support use of unit wts. Rather than ideal wts.
• Also see, Dawes, R. M. (1979). The robust beauty
  of improper linear models in decision making.
  American Psychologist, 34, 571-582.

47
RIM Critiques Concern 5

• Multiple measures used to generate composite
  scores
• Results in less accurate estimates of the
  cognitive domains.

48
Response 5 Estimate FSIQ Using Scaled Score
Meanss
49
RIM Critiques Concern 6

• A general ability factor is used to represent
  premorbid functioning for all domains.
• This not supported by the literature.
• This results in inaccurate conclusions regarding
  degree of impairment suffered by a patient in
  each cognitive domains assessed.

50
RIM Critiques Concern 7

• Norms used come from samples that are of
  undocumented comparability.
• Furthermore, even when norms used were generated
  from different but comparable samples, their
  format prohibits ready comparisons.

51
Response 7 Split-Half Reliability

• Analyze Dataset 2 - OTBMs from 42 DVs.
• Individuals data split into two sets
• 21 test variables for each OTBM (1 2).
• 2 independent OTBMs created for patient.
• Split DVs intentionally - separated so that no
  normative sample was included in both OTBMs.

52
Response 7 Split-Half Reliability

• Results r .81, 66 of variance accounted
• Slope of the regression line was .82 (SE .027)
• Intercept 9.2 (SE 1.20).
• Mean OTBM-1 45.0 (SD 7.3).
• Mean OTBM-2 43.6 (SD 7.2).
• Results simulate worse case scenario.
• used an entirely different set of norms.
• Est. test-retest r for OTBM 42 DVs increased r
  .82 - .88 (Spearman-Brown correction).

53
Response 7 Split-Half Reliability

• No overlap in normative samples.
• Worst-case condition, generally administer
  instruments (e.g., WAIS-III) with OTBMs generated
  from co-normed variables.
• Meyers Rohling test-retest reliability of .86.
• When different norms used, often gave same
  instruments (e.g., AVLT or RCFT).
• Our simulation, no instruments included in OTBM-1
  included in OTBM-2.
• Heaton et al.s (2001) - schizophrenic pts.
• Obtained a test-retest reliability of .97.
• Comparing 2 identical batteries, not worst-case.

54
RIM Critiques Concern 8

• The RIM will result in an undue inflation of
  clinicians confidence.
• Such overconfidence results in more error in a
  interpretation, not less.

55
RIM vs. Tulsky et al. (2003) Case 1
56
RIM vs. Tulsky et al. (2003) Case 2
57
Summary of the Rohling Interpretive Method of
Statistical Analysis of Neuropsychological Data
58
Summary of RIM Steps

• 24 total steps to the RIM process
• 17 calculation steps
• Battery Design
• Calculation of summary statistics
• Generation of graphic displays
• 7 interpretative steps.
• Use of summary table and graphic displays to
• Assess and verify summary data
• Identify strengths/limitations of current data
• Obtain a reliable diagnosis
• Develop tx plans based on clinical judgments.

59
Summary of RIM Advantages

• Formulize thinking interpretation of data
• Operationalize what you already do.
• Reduce judgment errors thru RIM Process.
• Take advantage of psychometric properties at
  level with fixed, co-normed batteries.
• Allows flexibility of test selection.
• Allows flexibility of theoretical view of
  cognition (e.g., domain structure)

60
Summary of RIM Advantages contd

• Gives Quantitative support for your conclusions
  and interpretations
• Statistical evaluation
• Measure of confidence in findings
• Measure of limitations of findings
• Ability to present data at different levels of
  interpretation
• Equals greater defensibility

61
Our RIM Cautions/Concerns

• Does not replace clinical judgment, rather,
  informs clinical judgment.
• This still means CJ errors are possible.
• Susceptibility T-Scores to distrib. deviance
• Process, not program
• Pre-morbid ability estimates
• Domain selection, test placement

62
Published Research Findings Using the RIM

• 1) RIM vs. HRB
• 2) Variance Accounted for by SVT
• 3) Effect of Depression on NP Results
• 4) Prediction of Employment after TBI

63
RIM of HRB OTBM vs. HII

• Heaton et al.s (1991) HRB norms for OTBM
• T Score (M50, sd10)
• OTBM r with HII -.79
• (p lt .0001)
• 62 variance account.
• Over predicts low
• Under predicts high

64
RIM of HRB OTBM vs. GNDS

• OTBM r with GNDS -.87
• 76 variance acc.
• OTBM neither under nor over predicts across range
  of GNDS
• Intercept impairment is T Score 46.0
• Reitan Wolfson (GNDS 29)

65
RIM of HRB OTBMs Relationship to Global Indices
66
RIM of HRB Diagnostic Classification Using the
HII
67
RIM of HRB Cross-Validation of RIM using HRB in
2 Samples

• Regressed Dikmen Meyers TBI data
• Generated a predicted HII for pts in OK dataset.
• Correlation actual predicted HII .95
• Sense .60, Spec .77, PPV .78, NPV .59
• Overall Correct Classification 71
• Predicted HII from MSBs OTBM more accurate
  indicator of impairment than actual HII.

68
Factor Loadings of Domain Scores
69
Means SDs of Composite Scores
70
Mean z Score on Objective Tests

• Small differences between Gen. Normal Gen.
  Neuro on NPT.
• No differences between Exag Normal Exag Neuro
  on NPT.
• Deficits for Exag Neuro were more modest than for
  Exag Normals on SVT.
• Interaction between Validity Neuro Status.

71
Mean z Score Self-Report

• No differences between Gen Neuro Exag Neuro on
  Memory Complaints.
• No differences between Gen Exag Neuro on
  Psychiatric Sx.
• Deficits for Exag Normal on the Psych Sx Memory
  Complaints latter is larger.
• Interaction between Validity Neuro Status.

72
Depression Study Reference

• Rohling, M. L., Green, P., Allen, L. M.,
  Iverson, G. L. (2002). Depressive symptoms and
  neurocognitive test scores in patients passing
  symptom validity tests. Archives of Clinical
  Neuropsychology, 17, 205-222.

73
Mood Group Assignment

• Patients classified into 2 subgroups
• From entire sample, 420 passed all SVTs
• Sample split based on BDI
• Low-Depressed 25ile on BDI (lt 10)
• n 178, M 6 (3)
• High-Depressed 75ile on BDI (gt 25)
• n 187, M 31 (6)

74
Depression Study Participants

• All 365 patients referred for evaluation for
  compensation-related purposes
• All diagnostic groups included
• 53 Head injury referrals
• 22 Medical referrals
• 14 Psychiatric referrals
• 11 Other neurological
• Age 42 (11) Ed 13 (3) Sex 64 males
  Non-English 18 Handedness 9 Left

75
Results Mood Validity Status
SVT Status
Mood BDI
Genuine
Exaggerating
175 (48)
Depress 75ile
186 (52)
NonDep 25ile
266 (74)
95 (26)
76
Results Sample Split by Validity
77
Effect of Mood Depends on Effort

• Exaggerating patients accounted for
• 39 of High-Dep group
• 14 of Low-Dep group
• Mood Effort used as IVs and Cognition DV
• Effect for effort, no effect for mood
• However, when Memory Complaints DV
• Effects for both effort and mood
• Also, when other Emotion/Personality DV
• Effects for both effort and mood

78
Effect of Mood Depends on Effort

• When both mood groups were included in
  regression analysis, as predicted
• Memory ratings related to mood
• (r .60 p lt .0001)
• Mood not correlated with cognition
• (r .10 p gt .10)
• Memory ratings not related to cognition
• (r .13, p .06)

79
Mood Replication

• Gervais pain sample findings (n 177)
• Exaggerating patients accounted for
• 55 of High-Dep 33 of Low-Dep group
• Memory ratings related to mood (r .55)
• Mood not correlated with cognition (r .06)
• Memory ratings related to cognition (r .15)
• Group means correlated with Greens .94
• all patient (High-D, Low-D, Gen, Exag).

80
Effect if Pain on OTBM
81
Effect if Pain on OTBM
82
Return to Work after Injury

• Three main hypotheses using MSB-RIM
• OTBM will predict return to work level
• Cognitive domain that will be most predictive
  will be executive function
• Adding the Patient Competency Rating Scale will
  improve work prediction
• PCRS is by Prigatano (1985)

83
Return to Work ANOVA of OTBM
84
Logistic Regression Using OTBM
85
Return to Work Summary

• OTBM differences between groups.
• Disabled /Unemployed not able to separate.
• Below/At Previous not able to separate.
• Collapsed groups result in 71 correct
• above base rate of 52 correct.

86
Return to Work Domain Analysis

• Executive function not the most predictive.
• Most of variance carried by Perceptual
  Organization Working Memory
• Using Cognitive Domains
• OTBM increases Correct from 71 to 74
• Incremental validity of PCRS very low.
• 7 of the variance

87
Return to Work Domain Analysis

• By including premorbid variables, increases
  diagnostic accuracy most helpful being
• Premorbid IQ, level of occupation, education
• Including acute measures increases accuracy most
  helpful being
• LOC group
• Time since injury

88
Depression Study Conclusions

• Memory complaints not synonymous with impairment
  in compensation sample
• Findings replicated
• Effort accounts for more variance in self-ratings
  of cognition objective performance than mood
• Findings replicated

89
Rohlings Interpretive Method Use of
Meta-Analytic Procedures for Single Case Data
Analysis

• Martin L. Rohling
• L. Stephen Miller
• Questions Comments Welcome!