Taking Research to Practice: Rethinking Outcomes and Performance Measures for the Child and Family Service Reviews

1
Taking Research to Practice Rethinking Outcomes
and Performance Measures for the Child and Family
Service Reviews

• John D. Fluke, Child Protection Research Center
  American Humane Association
• Erin Dalton, Allegheny County Department of Human
  Services
• Barbara Needell, University of California,
  Berkeley
• Fred H. Wulczyn, University of Chicago
• Mark E. Courtney, University of Chicago
• Donald Baumann, Texas Department of Family and
  Protective Services
• Society for Social Work Research, January 15,
  2011

2
The Child and Family Service Review Outcomes Do
Research Principles Matter?

• We will restore science to its rightful place
  Barack Obama, January 21, 2009
• Framework for Impact
• Mission Driven Framework for Outcomes Safety,
  Permanency, Well Being
• Embedded within systems
• Entries and Exits
• States and Processes
• Quality of Care
• Focus on Improvement
• Research
• Rigorous Design/Sample
• Validity
• Reliability
• Parsimony
• Risk Adjustment

3
How do the Basic Principles of Outcome Based
Research Line up With the Child and Family
Services Review Process?

• Strengths
• Effort to focus on outcomes
• Use of Data Driven Framework
• Development of Infrastructure
• Focus on system improvement
• Issues - Outcome Research Principles
• Measures are not necessarily
• Valid Parsimonious
• Rigorously Designed or Implemented Risk
  Adjusted
• Interpretation of Review Findings are not
  necessarily informed by research
• Remedies (Program Improvement Plans) are not
  demonstrably tied to research
• Measures of change are focused on unadjusted
  normative standards
• False Choices Along the continuum of Rigor and
  Feasibility
• Risk of Type I and Type II errors

4
The Way Forward What Needs Attention in the CFSR
Outcomes and Evaluation?

• Measures
• State and National Data Infrastructure
• Focus on Validity and Parsimony
• Range of Data
• Adequate Samples
• Standards
• Based on Risk Adjustment
• Based on Changes
• Improvement
• Performance monitoring
• Improving the scientific rigor
• Reducing the Type I and Type II errors (same as
  administrators)

5
Examples of Measurement
6
A Local Perspective

• Child welfare -- leaders in this area
• Unnecessary complexity
• Competition amongst measures
• Different, inconsistent definitions
• Data quality
• Error
• Serious Penalties
• Doing the wrong things
• Financial

7
Unnecessary Complexity

• Permanency Composite 1
• Measures Timeliness and Permanency of
  Reunification, and consists of four measures
• National standard 122.6
• National median 113.7
• Pennsylvania score 85.2

8
Inappropriate Complexity Competition Amongst
Measures Within Composites

• Permanency Composite 1
• Measures Timeliness and Permanency of
  Reunification, and consists of four measures
• Exits to reunification Of those reunified
  during the report period , what percent went home
  within 12 months of removal?
• Median length of stay Of those reunified, what
  was the median months in care?
• Of those entering care for first time in the 6
  months prior to the period, what percent were
  reunified within 12 months of removal?
• Of those exiting care in the 12 months prior to
  the report period, what percent returned to care
  within 12 months of being discharged?

9
Different Populations - Lack of Risk Adjustment

• Recurrence of Maltreatment- Of the children who
• were victims of substantiated abuse during first
  six months of the report
• period, what percentage were not victims of
  another substantiated report
• within six months.
• National standard 94.6
• PA 97.0
• Why are we doing so well?
• Are we doing well?

10
Even within State
11
But are we doing better?
CFSR case review suggests otherwise
Our analysis suggests
12
Different Populations - Lack of Risk Adjustment

• Of those exiting care in the 12 months prior to
  the report period, what percent returned to care
  within 12 months of being discharged?
• PA 28.5 Median 15.0 75th 9.9
• Why are we performing so differently than the
  nation?
• Could be serving a more difficult population?
• An older population?
• More likely its because juvenile justice
  population included in our counts, not in others.

13
Returns to Care

• Insert returns to care w/in 12 mos from perm
  exits
• Look at age composition over time

14
Data Quality

• Placement Stability Composite
• Consists of three measures
• National standard 101.5
• National median 93.3
• Pennsylvania score 102.4

15
Opportunity for Reform

• Interest at the federal level
• APHSA obtaining suggestions from the states and
  experts
• Casey Family Programs convening states to discuss
• No reason to sacrifice science in developing
  measures and recommending approaches to improving
  services