Leonard Bickman

1
Reforming Childrens Mental Health Services
Lessons Learned from the Past and Suggestions for
the Future
Leonard Bickman and Manuel Riemer Center for
Mental Health Policy Vanderbilt University
2
Presentation Overview
Lessons Learned from the Past Childrens
Mental Health Reform in the US
Professionalism
System Level
Evidence Based Practice
Suggestions for the Future CQI and the Role of
Measurement and Feedback
Measurement
Feedback Intervention
3
Learning from History - A Critical Evaluation
of Three Major Approaches to Childrens Mental
Health Reform
4
Defining the Problem
Two reports, almost 20 years past by, but
J. Knitzer, 1982 Unclaimed Children

• Limited access
• Little funding
• Little coordination across agencies
• Very few staff
• Inappropriate services
• Provides little information about effectiveness
  of services

Surgeon General, 1999 Report on Mental Health

• More funding
• More coordination
• More staff
• More and different services
• Limited access
• Still, we know little about effectiveness of
  services

5
Purpose of this Analysis
Analyze 3 main reformmodels to find
leveragepoints for improvement
Reform
6
Three Approaches to Reform
Clinical Outcome
SYSTEM
PROFESSION
TREATMENT
7
Professionalism
Clinical Outcome
Professionalism The conduct, aims, or qualities
that characterize or
mark a profession or a professional
person (Merriam Webster Dictionary, 2001)
8
Professionalism Approaches
1) Instruction
3) Selection
Accreditation of providers
Preservice training
Inservice training
Licensing of clinicians
Supervision
2) Experience
Requirement of practice experience
9
Is Professionalism Effective in Improving
Clinical Outcomes?
?
Clinical Outcome
10
Finding the Evidence

• Instruction
• Preservice Training
• Christensen Jacobson (1994)
• Stein Lambert (1995)
• Shadish et al. (1993)
• Inservice Training
• Beaudry (1989)
• Davis (1998) Davis et al. (1992) Davis et al.
  (1995)
• Bickman (1999)
• Supervision
• Holloway Neufeldt (1995)
• Steinhelber et al. (1984)
• Ellis et al. (1996)

11
Finding the Evidence (cont.)

• Experience
• Sechrest, Gallimore, Hersch (1967)
• Dawes (1994)
• Luborsky et al. (1980)
• Smith Glass (1977)
• Stein Lambert (1984)
• Strupp Hadley (1979)
• Dush, Hirt, Schroeder (1989)
• Lyons Woods (1991)
• Stein Lambert (1995)

12
Finding the Evidence (cont.)

• Selection
• Accreditation
• Hadley McGurrin (1988)
• Bravo et al. (1999)
• Licensing
• 173 publications on licensing (19671999) but
  no investigation of association between
  licensure and quality of professional services

13
Results

• Currently there is not enough empirical evidence
  to make reliable conclusion whether
  professionalism is effective or not

?
Clinical Outcome
14
What are the Logic and the Underlying Assumptions
of Professionalism?
15
Instruction
Transportability
Assessment Diagnosis
Clinician receives instructions about
best practice and how to apply it
Clinician uses best practice
Clinician is motivated to use best practice
Clinician Is capable of using best practice
Clinician can match best practice to
client problem
Clinician Is in the position to learn best
practice
Improved clinical outcome
Best practice has been empirically identified to
be effective as well as efficacious
Professionals and organizational context
Evidence base of treatment
16
Experience
Clinician works in a clinical setting
Clinician obtains experience
Clinician learns
Clinician positively changes his/her behavior
Improved clinical outcome
Professionals andorganizational context and
conditions
Process of learning through experience
17
Selection
Authority makes selection
Ineffective or bad organizations or
clinicians are sorted out
The remaining group of organizations and
clinicians are more effective or better
Improved clinical outcome
Process of change
18
Three Approaches to Reform
Clinical Outcome
System of care Delivering coordinated but
diverse services on an individualized basis using
case management and interdisciplinary treatment
teams to integrate and facilitate transition
between services.
19
System Level Approaches

• Increasing access
• Community-based services
• Adding new services
• Using the existing services more effectively
• Organizing and systematizing
• Cross-collaboration and coordination of services
• Tailoring
• Individualized services wrapped around the
  child and his/her family
• Community-based services
• Continuum of care
• Diversification of services

20
Is System Level Reform Effective in Improving
Clinical Outcome?
?
Clinical Outcome
21
Finding the Evidence

1. Fort Bragg Study (1995 Continuum of Care)
2. Stark County Study (1998 System of Care)
3. Wraparound Project Study (2001 Wraparound)
4. Comprehensive Community Mental Health System
   Reform (1993 present System of Care)

22
Results

• System changes have been shown to
• Increase satisfaction
• Increase access
• Increase costs
• But not improve clinical outcomes

?
Clinical Outcome
23
What are the Logic and the Underlying Assumptions
of System Level Reform?
24
Increasing Access
More children will receive services
Services are used by children In need
Increase access to mental health services
Improved clinical outcome
Client Context
Effective treatments are offered and
implemented correctly by services
Effective treatments have been identified
Evidence base of treatment
Transportability Dissemination
25
Organizing and Systematizing
More children can be served
Increased access
Services are organized and systematized
Services are used more efficiently
Improved clinical outcome
Costs are reduced
Saved resources are invested into
additional services
Policy context
26
Tailoring
Assessment Diagnosis
Clinician is able to assess childs need correctly
Clinician is able to match the
correct treatment with childs need
Services are tailored to child needs
Child receives more appropriate care
Improved clinical outcome
Evidence base of treatment
27
Three Approaches to Reform
Clinical Outcome
EBP Use of scientifically evaluated treatments
that havebeen shown to be efficacious and/or
effective
28
EBP Approaches

• Setting standards
• Finding and publishing empirically supported
  treatments (EST)
• Efficacious studies
• Effectiveness studies
• Reviews and meta-analyses
• Creating a (web-based) data base with ESTs
• Developing manuals, guidelines, treatment
  algorithms

29
Is EBP Effective in Improving Clinical Outcome?
?
30
Finding the Evidence

• Standards
• APAs Division 12 Task Force, Committee on
  Science Practice (1995)
• Interdisciplinary committee on Evidence-Based
  Youth Mental Health Care
• FDA
• International Psychopharmacology Algorithm
  Project

31
Finding the Evidence (cont.)

• Reviews Meta-Analyses
• APAs Division 12 Review of evidence-based
  therapies (1998)
• Kazdin, Psychotherapy for children and
  adolescents, Oxford (2000)
• Weisz Jensen, Mental Health Services Research
  (1999)
• JAACAP special issue on psychopharmacology
  (1999)
• Rones Hoagwood, School-based mental health
  services, Clinical Child and Family Psychology
  Review (2000)
• Burns, Hoagwood, Mrazek Child Clinical and
  Family Psychology Review (2000)
• Chorpita et al., Hawaii Emperical Basis to
  Services Task Force (2001)
• Surgeon Generals Mental Health Report (1999)
• Surgeon Generals Youth Violence Report (2001)
• Surgeon Generals Supplement on Culture, Race,
  Ethnicity (2001)

32
Finding the Evidence (cont.)

• Web-Based Archives
• Cochrane Collaboration
• Campbell Collaboration
• Center for Evidence-Based Medicine
• Center for Evidence-Based Mental Health
• FOCUS
• Society of Clinical Psychology

33
Results
?
?

• Limited number of efficacious treatments
• Very small number of effective treatments
• Almost no research findings on
  transportability, implementation,
  dissemination

?
Clinical Outcome
?
TREATMENT
34
What are the Logic and the Underlying Assumptions
of EBP?
35
EBP
Evidence base of ESTs is developed
ESTs are implemented in practice
Clinicians provide more efficacious treatments
Improved clinical outcome
Measurement
36
What have We Learned?
We need to pay more attention to the
underlyingassumptions of our reform models

1. What are the real world conditions and context
   (client, practitioner, organization, policy)
2. Is it enough to intervene at one level only (e.g.
   professionals, treatment, system, organization)?
3. What are the actual processes of change?
4. How do we improve transportability and
   dissemination? (e.g. involve practitioners and
   providers)
5. What are the processes of practitioners behavior
   change (such as learning through experience)
6. How can we match the right treatments to
   individual clients?
7. How can we assure correct assessment?
8. How can we develop valid measures for processes
   and outcomes?

37
Continuous Quality Improvement As a Reform
The Roles of Measurementand Feedback
38
Previous Reform Efforts Had Limitations

• They did not make all their assumptions explicit
• They did not involve all levels of an
  organization
• Professionalism deals with clinician
• System level reform deals with system
• EBP focuses mainly on treatment
• They are not focused on improving the quality of
  treatment
• They are not sensitive to the real world
• They are externally implemented and not
  internally innovative

39
Another Suggested Reform
Implement continuous quality improvement (CQI)
with an integrative concurrent consumer
measurement system
40
Continuous Quality Improvement (CQI)

• Originally developed for industrial and
  manufacturing applications
• CQI involves the use of assessment, feedback, and
  application of information to improve services
• CQI relies on a continuous evaluation of
  processes and outcomes
• CQI involves a dynamic interplay of assessment,
  feedback, and application of information
• Typically requires changes in the organization
  and thus is multilevel
• CQI can empower organizations, clinicians and
  clients

41
A CQI Change Process Model
System
Organization
DATA
SERVICES

• process
• outcome

INTERPRETATION
ACTION

• guidelines
• training
• feedback to practitioner
• system modification

42
CQI Was Developed for Simple Processes

• Highly sensitive process measures
• Highly stabilized and replicable processes
• Good ability to refine and tweak production
  processes
• Relatively closed system - few external
  influences on processes and outcome
• Few unexpected effects
• Simple descriptive statistics
• Strong causal linkage of inputs, process, outcome

43
CQI Is More Difficult in Human Services

• Few sensitive process and outcome measures
• No systematic feedback process
• Highly variable and ill defined processes
• Low service replicability
• Clinician behavior difficult to re-program
• Open system - multiple external influences on
  structure, process, and outcome
• May be detrimental to program goals
• Gaming when a basis for rewards or accountability
  
• Goal displacement

44
Little Experience With CQI

• Limited experience in mental health
• Exception is the Heidelberg-Stuttgart Model
• Not a sufficient number of validated measures of
  quality exist
• Clinician education and resistance are major
  factors
• Resources seen as barrier to implementation
• Few if any evaluations of CQI

45
What Does CQI Require?

• That we understand the links between process of
  care and outcome
• That we systematically collect data on these
  links and its elements
• That we feedback how processes affect outcomes
• That we use information to change clinician
  behavior
• That we create an atmosphere of change that
  supports treatment based on data

46
A CQI Change Process Model
System
Organization
Measurement
DATA
SERVICES

• process
• outcome

INTERPRETATION
ACTION

• guidelines
• training
• feedback to practitioner
• system modification

• Feedback intervention

47
Measurementof Processes and Outcomes
48
Mental Health Measurement Needs For CQI

• Sensitive, accurate, real time estimates of
  meaningful clinical change over time
• A comprehensive and integrated system that uses
  consumer outcomes e.g. symptoms, functioning
• Sensitive and informative characterization of
  process indicators (therapeutic alliance,
  treatment modality, readiness to change, etc.)
• Reduction in the paperwork burden for
  clinicians, youth and families

49
Components of the Child Adolescent Measurement
System (CAMS)

• Measures concurrent and baseline
• Functioning - strengths and impairment
• Symptom severity
• Hopefulness/satisfaction with life
• Satisfaction with services
• Victimization
• Acuity
• Substance use
• Violent/aggressive behavior

50
Feedback Intervention Model to Change Clinician
Behavior
X
Y
M
51
What is Feedback?

• General FeedbackThe American Heritage
  DictionaryThe return of a portion of the
  output of a process or system to the input,
  especially when used to maintain performance or
  to control a system or process.
• Feedback InterventionKluger DeNesi
  (1996)Actions taken by (an) external agent(s)
  to provide information regarding some aspect(s)
  of ones task performance.

52
Feedback Interventions can Improve Performance

• Possible to intervene so that professionals can
  learn more effectively from experience
• Feedback intervention in several settings have
  been shown to have a positive effect on
  performance and client outcomes
• Meta analysis show medium size effects in
  non-mental health settings

53
Feedback Interventions in Psychotherapy

• Little research on the effects of feedback to
  clinicians about client progress
• Lambert has several studies showing feedback
  increased length of time in treatment as well as
  improved clinical outcomes
• Feedback was simple colors red, yellow, green
  on progress
• Cant tell how feedback operated but provides
  support for CQI

54
The Effects of Feedback Interventions often Vary

• Feedback is not consistently an effective
  learning technique
• In 1/3 of the studies the feedback decreased the
  quality of the performance
• Effect sizes from differentstudies were very
  different
• Feedback is not a simplephenomenon that is
  easilyimplemented

55
Previous Research on Feedback is Limited

• Most research is applied and mainly atheoretical
• The theoretically grounded research is artificial

56
Four Step Approach to Developing a Research
Program
Theory application
Theory refinement
Theory testing
Theory construction
57
Step 1 Theory Construction

• Used an interdisciplinary team of experts
• Research synthesis through traditional literature
  reviews, meta-analytic procedures, etc.
• Findings from cognitive psychology, learning
  theory, social psychology, organizational
  psychology, management, and other related fields
• Work has already been started on this phase
• Need to drill down into details
• Move from macro perspective to micro perspective.

58
Our Preliminary Theory
59
1) Existing Moderating Conditions

• Task characteristics
• Complexity
• Level of task generated feedback

• Workplace atmosphere
• Leadership structure
• Learning structure
• Organization goal standard
• Competitiveness among peers

Setting specific
Client Characteristics
Practice behavior
Organizational characteristics
Practitioner Characteristics
Implicit theory / working model

• Personality
• Regulatory focus (prevention or promotion)
• Cultural background (e.g collectivistic or
  individualistic)
• Values
• Knowledge
• Competence

• Personal goal standard

Competing demands
Organizational goal standard
60
2) Practice Process Chain
FEEDBACK INTERVENTION (FI)
PROCESS OUTCOME
Practice behavior
Process mediators
Concurrent outcome
Proximal outcome
Ultimate outcome
61
3) Feedback Evaluation Part I
Feedback is disregarded
NO
Feedback is considered valid
FEEDBACK INTERVENTION (FI)
YES
Feedback information
Perceived important feedback standard discrepancy
Feedback- goal standard comparison
Goal is clear and important to practitioner
Personal goal standard
YES
NO
Not attend to Feedback
62
4) Feedback Evaluation Part II
Motivation to overachieve goal standard
YES
Motivation to keep or increase effort
YES (feedback gt standard)
Motivation to reduce effort
Perceived important feedback standard discrepancy
NO
Current behavior is considered as successful
NO
YES (feedback lt standard)
Disregard feedback
Choosing discrepancy reduction strategy
Motivation to reduce discrepancy
Motivation to change behavior
Abandon or change goal standard
Abandon or change goal standard
Practice behavior
63
5) Behavior Change Processes
Feedback initiated learning
YES
Attribution of failure to external causes
Believe in success by learning
Attribution of failureto self
NO
?
NO
Believe in success by increasing effort or
shift attention
YES
Increase effort or shift attention
Motivation to change behavior is eliminated
64
6) Feedback Initiated Learning
New behavior, skill, knowledge effective?
Acquisition of new behavior, skill, knowledge suc
cessful?
Motivation to learn new behavior / skill/ to
gain new knowledge
Effort to learn new behavior / skill/ to gain
new knowledge
YES
YES
NO
NO
Belief in success by another trial of learning
YES
Feedback initiated learning
NO
Attribution of failure to external causes
Application of new skill, behavior, knowledge
Attribution of failure to self
?
65
7) Attribution to Self
Increased learned helplessness
Decreased job-satisfaction
Barriers to leave job?
NO
Decreased self-efficacy
Drop-out
Increase in work-related stress
Increased burn-out symptoms
YES
Changes to the self
Decreased work motivation
66
Hypothesized Permanent Learning through Feedback
1. Feedback loop
2. Feedback loop
3. Feedback loop
4. Feedback loop
67
Step 2 Theory Testing

• Whole theory cannot be tested
• Select key elements to test
• Primarily experimental and field based tests
• Will need large number of cooperating
  professionals

68
Example of a Test of the Model

• Evaluate the impact of a feedback intervention,
  dissonance arousing academic detailing approach
  on assessment, diagnosis and treatment of 80
  pediatricians for children with ADHD
• Assess the extent of adoption of the American
  Academy of Pediatrician guidelines and its affect
  on children and families

69
Studies Can Take Place in Several Settings

• Settings including mental health, health, higher
  education and primary and secondary education
• Enhance the generalizability of findings across
  types of practitioners, settings, and
  organizations

70
Step 3 Theory Refinement

• Based on several studies fromseveral fields the
  theory is modified

Advantage Findings can easily be integrated in
dynamic model
71
Step 4 Theory Application

• Ultimate goal is the utilization of the theory in
  the real world to improve client outcomes
• The theory generating process will uncover many
  potential leverage points for interventions that
  can be used individually or simultaneously across
  many different settings and professional fields

72
Conclusions

• Previous reform efforts have been based on
  assumptions that were not explicit
• There is not sufficient evidence to support these
  assumptions
• We need to try alternative approaches to
  reforming services so as to improve outcomes for
  children
• CQI is an attractive alternative to previous
  reform efforts
• However we do not have sufficient experience with
  CQI in the human services
• Practical measurement is available - technology
  for real time measurement is advancing

73
Conclusions (cont.)

• Need investment in system infrastructure for
  measurement
• Need to establish links between interventions and
  outcomes in real world (Quality)
• Feedback intervention is potentially an effective
  approach but it is complex and requires more
  empirical research