Using CANS Data to Assist in the Creation of a Learning Collaborative to Understand Program Effectiveness

1

Using CANS Data to Assist in the Creation of a
Learning Collaborative to Understand Program
Effectiveness
Representatives of the Maryland Treatment Foster
Care Learning Collaborative Chris Kelly, LCSW-C,
Child and Family Services Division, Catholic
Charities Steve Herr, PhD, LCSW-C, University of
Maryland, Department of Psychiatry, Division of
Service Research Paul Brylske, LCSW-C, Kennedy
Krieger Family Center Amy Meyerl, LCSW-C, Kennedy
Krieger Family Center Maurice Williams, LCSW,
Williams Life Center Maisha Davis, LGSW,
Progressive Life Center Debbie Marini, LCSW-C,
Baptist Family Services
7th Annual National CANS Conference, May 13-15,
2011
2
Outline and Takeaways

• Introduction and Formation of the MD TFC
  Learning Collaborative (MDTFCLC)
• Implementation Research Parallels
• The Learning Collaborative at Work
• Using Data for Decision Making The Innovation
  Stage----Panel Review of Sample Data
• Future Capacity

3
MD TFC Learning Collaborative

• Baptist Family and Children Services
• Catholic Charities -Center for Family Services
• Kennedy Krieger Institute
• Kennedy Krieger Family Center
• Progressive Life Center
• Williams Life Center
• Comparable programs for comparison

4
MDTFCLC Demographic Profile (N) 326
5
MD TFC Learning Collaborative

• Forged around a commitment to outcomes management
  - Maryland TFC Coalition
• Belief that outcomes can guide and inform
  client, program and system level effectiveness
• July 2009--- five private Treatment Foster Care
  providers in partnership with the University
  Marylands Childrens Outcome Management Center
  (COMC) began focus on the use of data for program
  and practice improvement in a learning
  collaborative
• Use of KIDnet and CANS

6
A Brief Retrospective

• A Commitment to Outcomes with Sumone for Kids
  (MARFY Initiative - 17 years ago)
• Success and challenges of Sumone for Kids and
  CAFAS
• Federal Science to Service Grant- partnership
  with local universities to bring research and
  outcomes to fields like child welfare (pilot
  project)- forged relationship with COMC
• Increased emphasis on accountability through
  outcomes
• Strategic Vision (program level and state wide)
• State initiatives with CANS in Group Homes
• MARFY TFC Outcomes Committee partnering with
  Innovations to foster discussions on Outcomes
• State initiatives with CANS in TFC
• Formation of MD TFC Learning Collaborative as TFC
  programs began adopting the COMC system

7
Implementation Research

• MDTFCLC found parallels in the implementation
  process and work of Fixen, Naoom, Blase, Friedman
  and Wallace in their 2005 publication,
  Implementation Research A Synthesis of the
  Literature
• Implementation - a specified set of activities
  designed to put into practice an activity or
  program
• EBPs and programs are not effective unless they
  are fully implemented
• Performance implementation---that which is
  effective is often elusive
• Goal is to get beyond paper implementation---
  to create actual change in practices that
  produces effects benefitting the clients
• The MDTFCLC serves as the vehicle for programs to
  share, challenge and vet best practices in
  implementing KIDnet

8
Stages of Implementation

• Exploration and Adoption assess match examine
  resources make determination to adopt (or not)
• Program Installation putting policies in
  place/hiring
• Initial Implementation process begins (often the
  stage where inertia is lost)
• Full Implementation completely operational with
  all clients
• Innovation experience is gained desirable
  changes that improve program effectiveness are
  innovations and are adopted as standard
  practice fidelity adaptations are made to
  improve implementation.
• Sustainability goal is continued implementation
  in the face of an ever changing system
• Fixen et al

9
Implementation The Bear Den Challenge
10
Implementation of Health IT
11
The Learning Collaborative and Implementation

• Parallels seen in provider experience
• Overcoming the Bear Den Challenge Together,
  through joint experience, we were able to
  anticipate and work through barriers to counter
  resistance
• MDTFCLC became the vehicle to review and vet
  opportunities and challenges in implementation
• Through a series of regular monthly meetings, the
  MDTFCLC began to discuss the challenges and
  barriers in implementation
• Solutions and best practices emerged as the group
  partnered with COMC
• Changes and priorities to the system were made
  using a priority and consensus matrix

12
MDTFC Best Practices

• Initial implementation Champions User Groups
  Creating buy-in with Leadership and Development
  of Training for TFC parents on the CANS and
  KIDnet tools---created a remarkable culture shift
  in many organizations
• Full Implementation Webinars and use of the
  priority and consensus matrix became critical in
  creating sustainable buy-in
• MDTFCLC is currently in the Innovation
  Stage---making fidelity adaptations and
  examining data for client and program level
  improvement opportunities

13
The Context of TFC in MD

• Focus for youth and families least restrictive
  and permanent environment access to the familys
  community limited movement in placement
  reduction of youth entering care and reduction
  of youth in residential care.
• Implications for TFC average age of placement
  increases increase in range and acuity of
  behavior shorter lengths of stay greater
  potential for disruption of placement.
• Set the stage for examining those youth who exit
  our service to higher levels of care

14
Review of Aggregate Data The Starting Point
---ROLES LOR
15
Review of Aggregate Data
16
(No Transcript)
17
(No Transcript)
18
Average Length of Stay
19
EXPLORATORY PROCESS

• Ok---I have good data---now what? ---common
  question among providers
• Our starting point in the exploration
  process--(1) What about those youth we are
  struggling to serve---the 25 that go to more
  restrictive settings? (2) How do we fair in
  relationship to state mandated outcome areas?
• First question --- What can we learn when
  looking at CANS domain level discharge scores?
  --- led to further discussions and drill down

20
Drilling Down CANS Domains
21
Life Functioning Domain

• Family
• Living Situation
• Social Development
• Recreational
• Developmental
• Job Functioning
• Legal

• Medical
• Physical
• Sexuality
• School Behavior
• School Achievement
• School Attendance

22
(No Transcript)
23
Check SCHOOL BEHAVIOR Please rate the highest level from the past 30 days
0 Child is behaving well in school.
1 Child is behaving adequately in school although some behavior problems exist.
2 Child is having moderate behavioral problems at school. He/she is disruptive and may have received sanctions including suspensions.
3 Child is having severe problems with behavior in school. He/she is frequently or severely disruptive. School placement may be in jeopardy due to behavior.
Check SCHOOL BEHAVIOR Please rate the highest level from the past 30 days
0 Child is behaving well in school.
1 Child is behaving adequately in school although some behavior problems exist.
2 Child is having moderate behavioral problems at school. He/she is disruptive and may have received sanctions including suspensions.
3 Child is having severe problems with behavior in school. He/she is frequently or severely disruptive. School placement may be in jeopardy due to behavior.
24
Observations at item level Externalizing
behaviors trending toward area of need---item
level score of 2 or 3.
25
Check FAMILY Please rate the highest level from the past 30 days
0 Child is doing well in relationships with family members.
1 Child is doing adequately in relationships with family members although some problems may exist. For example, some family members may have some problems in their relationships with child.
2 Child is having moderate problems with parents, siblings and/or other family members. Frequent arguing, difficulties in maintaining any positive relationship may be observed.
3 Child is having severe problems with parents, siblings, and/or other family members. This would include problems of domestic violence, constant arguing, etc.
26
(No Transcript)
27
Child Risk Behavior

• Suicide Risk
• Self Mutilation
• Other Self Harm
• Danger to Others
• Sexual Aggression

• Runaway
• Delinquency
• Judgement
• Fire Setting
• Social Behavior

28
(No Transcript)
29
Check DANGER TO OTHERS Please rate the highest level from the past 30 days
0 No evidence
1 History of homicidal ideation, physically harmful aggression or fire setting that has put self or others in danger of harm.
2 Recent homicidal ideation, physically harmful aggression, or dangerous fire setting but not in past 24 hours.
3 Acute homicidal ideation with a plan or physically harmful aggression OR command hallucinations that involve the harm of others. Or, child set a fire that placed others at significant risk of harm.
Check DANGER TO OTHERS Please rate the highest level from the past 30 days
0 No evidence
1 History of homicidal ideation, physically harmful aggression or fire setting that has put self or others in danger of harm.
2 Recent homicidal ideation, physically harmful aggression, or dangerous fire setting but not in past 24 hours.
3 Acute homicidal ideation with a plan or physically harmful aggression OR command hallucinations that involve the harm of others. Or, child set a fire that placed others at significant risk of harm.
30
Observations

• Both groups tend to be admitted with similar
  scoring profiles
• What is happening in the first few updates
  cycles? Why do scores tend to be the same for
  both groups until later in service, e.g. 4th
  update?
• Externalizing behaviors school behavior,
  oppositional behavior, impulsivity, risk activity
  are demonstrating areas of need with this More
  Restrictive cohort.

31
Observations continued

• ALOS data suggest program decision making may be
  unique at certain points in the service delivery
  process. In other words, there is no defined
  approach to working with MR youth.

32
Whats Next?

• Focus on the 25 of youth being discharged to
  more restrictive settings
• More refined benchmarking MDTFCLC beginning to
  look at those programs that have lower values in
  this area---what are they doing differently?
• T-Test analysis to identify means differences
  that are truly statistically significant between
  both groups.
• Consider regression analysis to understand the
  impact of key variables on discharge disposition,
  e.g. age, gender, race, diagnosis.

33
Whats Next? continued

• Consider longitudinal trajectory analysis to
  understand the expected trajectory of youth with
  specific conditions.
• Is this data a reflection of what is happening at
  the state level---youth traditionally served in
  residential setting transitioning to TFC?
• Programs with significant wraparound services
  like PRP, crisis stabilization, mentoring---are
  they more successful?
• Are fundamental changes in program preparation
  (for example pre-service training, trauma
  treatment, substance abuse treatment and targeted
  recruitment) enough to promote stabilization?

34
Future Capacity and QA

• MDTFCLC meets the third Thursday of every month
  from 1030am - 1230pm
• To join the mailing list please contact
  meyerl_at_kennedykrieger.org