A MultiAspect Approach to the Evaluation of Outcome Quality in Psychosomatic Treatment

1
A Multi-Aspect Approach to the Evaluation of
Outcome Quality in Psychosomatic Treatment
Werner W. WittmannUniversität Mannheim,
Lehrstuhl Psychologie II Jürgen
SchmidtPrivatinstitut für Evaluation und
Qualitätssicherung im Gesundheits- und
Sozialwesen (eqs.)mbH, Karlsruhe
Workshop Quality Management and Outcome
MonitoringStuttgart, 14.03. - 16.03.2002
2
Jürgen SchmidtPrivatinstitut für Evaluation und
Qualitätssicherung im Gesundheits- und
Sozialwesen (eqs.)mbH, Karlsruhe
3
A stately mansion of evaluation research and
quality control
Evaluation research
Program evaluation
Research design and data analysis
Assessment
Decision and evaluation aids
Donald T Campbell
Lee J Cronbach
Eleonor Chelimsky
Len Bickman
4
DATA BOX (CATTELL, 1957)
5
The Five Data-Box Conceptualization
6
Success through symmetry A really magic concept
7
The beauty of Brunswik symmetry
ETR - Experimental Treatment BoxPR - Predictor
BoxCR - Criteria BoxNTR - Nonexperimental
Treatment Box
8
Full asymmetry, the case of nothing works!All
correlations between predictors and criteria are
zero!
Hierarchy of predictors
Hierarchy of criteria
9
Asymmetry due to a broad higher level prediction
and a narrower lower level criterion
Hierarchy of predictors
Hierarchy of criteria
10
Asymmetry due to a narrower lower level predictor
and a broad higher level criterion
Hierarchy of predictors
Hierarchy of criteria
11
The hybrid case of asymmetry, mismatch at the
same level of generality!
Hierarchy of predictors
Hierarchy of criteria
12
The Brunswik-lens-equationfor relating
experimental treatment (ETR) to criteria (CR)
There are 6 dangers to underestimateagainst 2
dangers to overestimatea true effect size! So do
you still wonder??
13
Southwestern path
14
The structure of the follow-up research design in
the Magic-Mountain study
15
Multiple-act criterion
16
Variants of aggregating assessment information
Single multiple acts (SMAC)
17
A taxonomy of assessment variants
18
Mact 27
1. How do you feel at follow-up (oneyear after)
compared to intake? For indicators 2. to
27.change in ... 2. quality of life 3. state of
health 4. state of mind 5. general
condition 6. competitiveness 7. complaints/trouble
s/disturbances 8. state of health 9. dealing with
problems/daily pressure 10. life-style concerning
health 11. drug consumption 12. relationship to
most closely related people 13. relationship to
partner
14. family life 15. ability to work 16. number of
visits to physician 17. time of inability/absence
from work 18. days in hospital 19. well-being 20.
dealing with problems/coping 21. capability for
self-help 22. endurance of disappointment 23. copi
ng with work 24. ability to take
stress 25. getting along with people 26. getting
along with problems/limitations/
impairments 27. balanced mood
19
All 27 components had to be rated according to
differences in change comparing the pre year
period after treatment in relation to the one
year period after treatment. One point was given
for improvement or positive reduction, a zero
point for no change, reduction or worsening. So
the worst case is a MACT_27 score of zero,the
best case a MACT_27 score of 27.
20
Distribution of MAC-27-Scores 1 year after
treatmentN 367 (Magic-Mountain II-Study)
21
Regression analysis of MACT_27 on quality of
treatment and amount of demoralization
22
Another multiple-act criterion
23

• MACT-MO
• Change in ...
• Drug consumption
• ability to work
• number of visits to physician
• time of inability to work
• days in hospital

24
Regression analysis of MACT_MO on quality of
treatment and amount of demoralization
25
Regression analysis of demoralization on quality
of treatment plus the missing-data-dummy
26
Path-analytic (causal) modelling of treatment
effects in the magic mountain study causally
investigating the southwestern path of the
five-data-box conceptualization
27
Regression analysis of demoralization change
scores on amount of demoralization at intake and
hours of therapeutic interventions demonstrating
the validity of change scores
Interpretation The lower the amount of
demoralization at intake the lower the amount of
reduction of demoralization. The higher the
number of hours of interventions the higher the
reduction.That means those with high
demoralization at intake had a higher amount of
reduction than those already low on
demoralization at intake! Alhough the bivariate
correlations show a small significant effect of
selection into treatment, the higher dosage for
those already lower on demoralization did not pay
off in higher reduction. The values in
parentheses correspond to an analysis with
corrections for attenuation.
28
The pattern of correlation/regression parameters
between intake (PR-Box), nonexperimental
treatment factors (NTR-Box) and change processes
(CR-Box)
CR-Box Reduction of demoralization at
discharge (Post-pretest change scores, high
scores, high reduction). r tt .70
R 2 .373adjusted R 2 .353 all
beta-wights are significant N 137
29
Attenuation corrected correlation/regression
pattern between intake (PR-Box), nonexperimental
treatment dimensions (NTR-Box) and change
processes (CR-Box)The analysis is based on true
scores of all variables only
CR-Box Reduction of demoralization at
discharge (Post-pretest change scores, high
scores, high reduction). r tt 1.00
R 2 .768adjusted R 2 .761 all
beta-weights are significant N 137