Using Data to Drive Evidence-Informed Practice and Outcomes: Performance Based Contracting and Quality Assurance Systems

1
Using Data to Drive Evidence-Informed Practice
and Outcomes Performance Based Contracting and
Quality Assurance Systems
17th National Conference on Child Abuse and
Neglect
2
Workshop Objectives

• Nature and scope of public/private partnerships
  in child welfare service provision
• Performance Based Contracting and Quality
  Assurance Systems in promoting outcome
  achievement
• Florida
• Illinois
• Missouri
• Opportunities for participating in a national
  dialogue
• Presenters Crystal Collins-Camargo, Cynthia
  Schuler, Erwin McEwen Paul Sundet

3
The Quality Improvement Center Concept

• The QIC PCW continues the experiment by the
  Childrens Bureau to utilize QICs as a method of
  research and demonstration
• evidence-based topic selection
• rigorous evaluation
• targeted TA
• broad dissemination
• This is a knowledge development initiativethe
  goal is to move the child welfare field forward

4
Funded by the Childrens Bureau, QIC PCW has the
Following Goals

• To promote and support an evidence-based
  and outcomes-focused approach to child
  welfare system development and organizational
  improvement.
• To facilitate a collaborative information-sharing
  and problem-solving national network among
  subgrantees, the Childrens Bureaus training and
  technical assistance network, public child
  welfare agencies, private service providers, and
  other stakeholders.
• To build consensus on appropriate models of
  reform, the respective roles and responsibilities
  of public and private agencies, and to provide
  input on areas on which the child welfare policy
  and evaluation fields should focus.

5
Four Components Over Two Distinct Phases

• Phase I (Year 1)
• Conducting a national needs assessment to
  identify knowledge gaps and assess the current
  status of privatization efforts
• Identification of research questions to be
  answered through demonstration subgrants in
  subsequent years
• Phase II (Years 2-5)
• Administering subgrants and providing TA
• Evaluating process and outcomes
• Disseminating knowledge and facilitating an
  information-sharing consortium

6
Phase I Needs Assessment and Knowledge Gap
Analysis
Analysis of Data Literature Review Stakeholder
Group and Individual Discussions Targeted
Forums National Advisory Board Key Informant
Discussions
Iterative Process
Data-driven
Refinement of Topical Focus Area
Examination of the State of Privatization
7
Public/Private Partnership in Child Welfare

• Privatization Lack of consensus on
  definition
• Three predominant features distinguish it from
  traditional subcontracting
• Shifting more, and more core child welfare
  services
• Transferring case management and decision-making
  authority
• Delivering results
• Language is important
• QIC focuses its work on the broad continuum of
  public/private partnership in CW as it is
  relevant to all states

8
QIC PCWs On-going Approach to Information
Dissemination

• To join the QIC PCW listserv and
  participate in quarterly cross-state dialogue on
  topics related to partnership, email
  jghall2_at_uky.edu
• Check out the website http//uky.edu/socialwork/
  qicpcw
• Findings from knowledge gaps analysis and
  literature review, Proceedings Documents and
  other resources
• Regularly updated annotated bibliography
• Summit on Public/Private Partnership

9
Privatization of Case Management 2008 Key
Informant Discussions with 47 PCW Administrators

• Purpose Follow up to our 2006 study to
  determine the extent states are
• Transferring primary case management to private
  providers
• Using performance based contracts
• 17 out of 47 states report having contracts for
  direct child welfare services where the private
  agency had primary case management authority
• 6 States reported large scale efforts
• 11 States reported more limited efforts

10
Year States without Privatized Case Management States without Privatized Case Management States without Privatized Case Management States implementing Limited Privatized Case Management States implementing Limited Privatized Case Management States implementing Limited Privatized Case Management States implementing Large Scale Case Management States implementing Large Scale Case Management States implementing Large Scale Case Management Total
States N States N States N
2006 AL AK AR CA CT DE GA ID IN KY LA MA ME MN MS MT NC ND NH NM NV OK OR PA RI SC TX UT VA VT WA WY 32 71 AZ CO MI MO OH SC TN WI 8 18 DC FL IL KS NY 5 11 45
2008 AK AL AR AZ CA CT DE GA, HI, IA, IN KY LA MA MS MT OK OR PA ND NM NV RI SC UT VA VT WA WV WY 30 64 CO ID MD MI MN MO NE OH SD TN WI 11 23 DC FL IL KS ME NY 6 13 47
11
Privatized Services

• All services, except investigations
• Florida
• Milwaukee, Wisc.
• Alternative Response
• Maine
• Hennepin County, MN
• Prevention
• Michigan
• New York
• In-home Services
• Franklin County, Ohio
• Areas in Idaho
• Kansas

• Foster Care
• Majority
• Illinois
• Kansas
• New York
• Limited
• Colorado
• DC
• Michigan
• Missouri
• Nebraska
• Ohio
• South Dakota
• Tennessee

• Other Services
• Adoptions
• Kansas
• Michigan
• Maryland
• Post-adoptions
• New York
• Independent Living
• Franklin County, Ohio
• Residential Serv.
• Michigan

Inclusivity ? Clarity ?
Transparency ? Objectivity
12
Performance Based Contracting

• Definition Supplied These contracts
  specify expected levels of performance, most
  commonly in the way of service or client outcomes
  and may tie at least a portion of the
  contractors payment, as well as any contract
  extension or renewal, to their achievement
• 25 States reported some form of PBC but
  definitions and approaches varied
• 11 States not currently using PBC are considering
  moving toward this type of contracting

Inclusivity ? Clarity ?
Transparency ? Objectivity
13
Examples of Services with PBC

• All Services ID, WI-BMCW
• Prevention NY, CO, WY
• Counseling/ Therapy IN
• Foster Care IL, OH, TN, NE
• Independent Living KY, WA
• Foster Care Recruitment AZ, MN, ND
• Home Studies MN
• Residential Services/ Treatment IN, WY
• Adoption/ Subsidized Adoption WI, ND

• Challenges Noted
• Difficulties capturing data to monitor outcomes
• Resistance of private providers
• Need for TA on how to implement PBC

Inclusivity ? Clarity ?
Transparency ? Objectivity
14
Triangulation of Data Led to Selection of Topical
Focus Area for Subgrants
Initial NAB/CB Discussions Key Informant
Discussions with PCW Administrators Discussions
with Stakeholder Groups Targeted Forums with
Experienced States Literature Review

• Test innovative performance based
  contracting and quality assurance systems
  ability to promote
• CW outcomes
• Quality service delivery
• Accountability
• Collaboration

15
Promising Practices Worthy of Evaluation re
Contribution to Outcome Achievement

• Shared vision
• Inclusive planning and contract components
  development process
• Engagement of key external entities
• Contract monitoring on both case and systemic
  levels
• Quality assurance and positive outcome-seeking
  systems of utilization management
• Ongoing communication and management of the true
  partnership

16
Research Questions Guiding Cross-Site
Site-Specific Evaluations

• Does an inclusive and comprehensive planning
  process produce broad-scale buy-in to
  clearly defined performance based contracting
  goals and ongoing quality assurance?
• What are the components of performance based
  contracts and quality assurance systems that are
  perceived to have the greatest impact on
  practice, service, and staffing changes, and
  ultimately client outcomes?
• When operating under a performance based
  contract, are the child, family and system
  outcomes produced better than those produced
  under the previous contracting system employed?
• Are there essential contextual variables that
  independently appear to promote contract and
  system performance?
• Once implemented initially, how do program
  features and contract monitoring systems evolve
  over time to ensure continued success?

17
Focus on Use of Data to Improve Practice and
Client Outcomes

• Framed based on drivers of successful
  implementation identified by Fixsen et al. at the
  National Implementation Research Network
• http//www.fpg.unc.edu/nirn/
• Perceived impact of array of QA/QI
  activities
• Impact of selected incentives/disincentives and
  performance indicators on outcome achievement
• Use of data on performance to adjust policy and
  practice
• Frontline supervision promoting evidence informed
  practice

18
The Funded PBCQA Projects

• Florida
• Department of Children and Families Judicial
  Circuit 5,
• Kids Central, Inc. and Jean K. Elder
  Associates
• Illinois
• Department of Children and Family Services
• Child Care Association of Illinois and the
  University of Illinois at Urbana-Champaign
• Missouri
• Childrens Division, Seven consortia of
  private childrens agencies and the University
  of Missouri- Columbia

Inclusivity ? Clarity ?
Transparency ? Objectivity
19

• Florida
• Improving Outcomes for Children in Out-Of-Home
  Care Through Performance-Based Contracting and
  Enhanced Quality Assurance Processes
• Cynthia Schuler, JD
• CEO, Kids Central

20
Focus of the FL Project

• To demonstrate the effect of
• the use of an inclusive and comprehensive
  planning process in the development of
  a performance-based contract for case management
  services which includes performance incentives
  and disincentives (shared risk among service
  provision partners) and
• the enhancement and alignment of the quality
  assurance process with the performance-based
  contract expectations on child welfare outcomes
• Is a collaboration between Kids Central Inc. and
  Florida Department of Children and Families (DCF)
  Circuit 5
• Create shared vision of practice drivers and
  outcomes
• Implement inclusive planning process and contract
  negotiation
• Create a comprehensive monitoring process
• Integration of an inclusive continuous quality
  improvement process that involves all levels of
  staff

21
Project Mission and Vision

• The contract process will so clarify required
  outcomes that
• Staff will understand how critical providing
  state of the art practice is to families in the
  Circuit
• Staff will understand how what they do everyday
  relates to a childs overall safety, permanency
  and well being
• The collaboration between Kids Central, DCF and
  Case Management Agencies (CMAs) will be one where
  information sharing and best practice will
  flourish

22
Strategy for Collaborative Planning

• Development of a collaborative environment
  through the use of a neutral, third-party
  facilitator to assist with the implementation of
• a shared-vision of practice drivers and outcomes,
• contractual incentive measures designed to
  promote best practices,
• performance objectives with appropriate financial
  incentives,
• a performance-based shared-risk concept, and
• a comprehensive monitoring process.

23
Shared Risk

• Shared Risk refers to the concept that CMAs
  not only share in the rewards
  for good performance (fiscal incentives), but are
  accountable when performance based contractual
  outcomes are not met.
• This is accomplished through an agreement
  stating that if performance on the incentivized
  measures is below expectations, CMAs will be
  afforded one quarter (3 months) to attempt to
  correct any problems and performance issues.
  During this period, Kids Central will provide
  requested technical assistance, advice, or
  support to sustain the efforts of the CMA.
  After one quarter of below-par performance, CMAs
  will pay for technical assistance from Kids
  Central (at a rate equal to the daily staff rate,
  250 , that Kids Central pays to the CMAs).

24
1. Face to Face Supervision within 4 Days of Case
Receipt and at 30 to 45 Days
Contractual Incentive Measures

• Status
• Supervisory Review Tool designed to guide and
  focus supervisory sessions with front line staff
  implemented
• Tool has been greatly discussed as to whether its
  intent is compliance or guidance
• Extensive effort expended by Kids Central QA
  staff to review completed tool to assess quality
  and outcome of face-to-face supervision
• Did supervision occur as intended
• Was supervision meaningful and of quality
• Continued improvement in quality supervision of
  staff as evidenced by data trends

• Measure At between 2-4 working days all new
  cases transferred for services from PI
  investigation will receive a supervisory
  screening with worker, and again between 30-45
  days quarterly there after.

25
1. Face to Face Supervision within 4 Days of Case
Receipt and at 30 to 45 Days
Contractual Incentive Measures
26
2. Case Information Entered within 2 Days
Contractual Incentive Measures

• Status
• Change to the SACWIS system has impacted ability
  of CMAs to meet the objective of this measure
• Initial discussions surrounding intent of measure
  and methodology to capture data have been
  resolved via consensus between CMAs and Kids
  Central
• Consensus surrounding the evaluation of the
  measure has been reached
• Additional research and technical assistance
  related to mitigating barriers to performance is
  required
• Group level incentive offered to individual case
  management units that can achieve this goal for
  two consecutive months
• Multiple groups achieved goal (Announced March,
  2009) and will be asked to provide technical
  advice to case management teams as to how they
  were able to accomplish this (sharing of best
  practice)

• Measure All case information will be entered
  into Florida Safe Families Network accurately and
  in a timely (within 2 working days) manner
• The provider shall input and update all required
  case management information into the Florida Safe
  Families Network data information system.
  Furthermore, Provider shall correct all errors
  indicated on the AFCARS Error Report minimally on
  a monthly basis and also by request from Kids
  Central.

27
2. Case Information Entered within 2 Days
Contractual Incentive Measures

• Outcome The data trend for this measure
  indicates that CMAs are making slow progress
  towards the objective. Units earning incentive
  award are sharing their techniques for meeting
  the data entry goal.

28
3. Contact with Biological Parents
Contractual Incentive Measures

• Status
• This incentive is a modification of a Statewide
  performance / outcome measure
• This measure represented a significant shift in
  the case work paradigm for the CMAs
• Compliance with required contact with biological
  parents was extremely low across the entire State
• Extensive discussion during Roundtable meetings
  as to reasonability of requiring face-to-face
  contact with parents that are incarcerated,
  out-of-state, or unresponsive
• Consensus regarding giving credit for attempted
  contacts or alternative methods of contacts was
  reached

• Measure Case managers of children in out of home
  care will have contact with biological parents.
• Contacts with biological parents will increase by
  25. And tracked on an ongoing basis utilizing an
  agreed upon set of questions (ie describe your
  involvement with your case planning process what
  is the hardest thing for you to achieve in the
  case plan easiest? etc.)

29
3. Contact with Biological Parents
Contractual Incentive Measures

• Outcome Case Management Agencies have made
  tremendous progress in their overall percentage
  of contact with birth parents. CMAs within the
  Circuit lead the state for this measure and the
  trend line indicates continued improvement to
  performance over time. Over 91 of available
  incentive money available for this measure has
  been earned.

30
4. Permanency
Contractual Incentive Measures

• Status
• Initially attempted to establish baseline
  expectations for CMAs
• Proved difficult to establish
• Discussion surrounding whether the Level 3 Option
  is reasonable as Independent Living is not a
  recognized permanency outcome
• Option removed as incentivized outcome during
  second year of contracts
• Incentive Payment
• 1500 Reunification
• 1000 Guardianship
• 68,500 in Incentives Paid Over Course of the
  Project

• Measure Case management agency will work to
  achieve one of the three mentioned permanency
  options for youth aged 13-17 and then maintain
  the permanency for 6 mos.
• Level1 Youth Return to Parent
• Level Youth has legal guardianship/kinship care
  situation.
• Level 3 Youth has independent living services

31
Changes in Quality Assurance System

• Internal Modification
• Additional quality assurance processes added to
  assess compliance with
  individual incentive measures
• Changes to the State System
• Move from a multi-tiered review process to a
  regional-based QA review
• Supervisor Reviews
• Case management supervisor will review 100 of
  the cases in their unit each quarter.
• A guide is used so supervisors think about
  quality of casework, and systematically document
  their review for QA purposes. The review may be
  completed face to face or as a file review.
• CBC Base Reviews
• CBC QA staff will review a sample of 25 case
  management cases per quarter.
• Collaborative Side-by-Side Reviews
• CBC QA staff and Regional QA staff reviews
  subsample of the 25 cases reviewed during the
  quarter. Calls for an objective monitor or
  facilitator who guides and coordinates the review
  of each case file.
• Collaborative In-Depth Reviews
• CBC QA and Regional QA conduct a more in-depth
  review of a subsample of the cases reviewed in
  the side-by-side. These reviews will include
  interviews with case managers, parents, children,
  providers and other stakeholders.
• Continuous improvement is promoted through shared
  learning and the identification of best practice

32
Using Data to Promote Outcomes

• Data generated through multiple sources and/or
  tracking processes is
  utilized to establish monthly performance
  outcomes for each contracting
  entity (case management agency)
• State SACWIS System Data
• State Digital Dash Board
• Local Data Tracking Related to Incentive Measures
• Quality Assurance Assessment Processes and
  Results
• Data analysis drives technical assistance and
  quality improvement processes
• Assessment of whether performance measure and
  incentives need to be changed
• Impact of evidence-informed, best-practices on
  child-welfare outcomes assessed through data
• Are performance trends improving
• Are the incentivized measures impacting overall
  outcomes
• Improvement to child safety, permanency, and
  well-being as assessed by specific AFCARS
  measures.

33
Project Outcomes

• Communication and Trust Between Department
  of Children and Families, Kids Central and
  Case Management Agencies Greatly Improved
• Using Collaborative Forums to Discuss Practice
  Modifications that will Support and Enhance
  Outcomes
• Improved Family Engagement
• Quality Supervisory Support and Discussions
• Timely Data Entry
• Creating New Initiatives Promoting Systemic
  Improvement Using Successful Aspects of the QIC
  PCW Collaboration

34
Illinois

• Striving for Excellence
• Extending Performance Based Contracting to
  Residential, Independent and Transitional Living
  Programs
• Erwin McEwen, Director
• Illinois Department of Children
• and Family Services

35
Overarching Goals of Striving for Excellence

• Improve outcomes for children and youth
• Build on success in foster care and kinship case
  management using performance based contracting
• Enhance existing public-private partnerships
• Address CFSR deficiencies in Permanency and Well
  Being
• Inform the field through documentation and
  rigorous evaluation of the process used

36
(No Transcript)
37
Changing Characteristics of the Residential
Population

• Behavioral health challenges and
  concomitant medical complications
• Chronic mental illness which likely will require
  transition to the adult mental health system upon
  emancipation
• Increased number of pregnant and parenting teens
  with behavioral health challenges
• Dual mental illness and developmental
  disabilities
• Severe conduct

38
Pre-PBC Status of Residential Treatment in
Illinois

• Individualized cost based rate methodology
• Compliance monitoring by outsourced
  university-based monitors
• Capacity challenges assuring availability of
  appropriate level of treatment based upon client
  needs
• Cumbersome admission process

39
Collaborative Planning
40
2 Effectively and Efficiently Reduce
Symptoms/ Increase Functionality
1 Improve Safety/Stability During Treatment
3 Improve Outcomes At And Following Discharge
Indicator Rate of Treatment Opportunity Days
Indicator Sustained Favorable Discharge Rate
41
Treatment Opportunity Days Rate

• Percentage of total days of care in a fiscal year
  residents are not on the run, hospitalized, or
  incarcerated
• Calculated based on treatment spells from
  admission to discharge
• Agencies fiscally penalized if they exceed
  performance benchmarks at rate of 25 of per diem
  payment for each day they exceed benchmark

42
Sustained Favorable Discharge Rate

• Favorable Discharge
• Positive - stepdown to less restrictive setting,
  including residential or group home settings by
  program classification (within or between
  agencies)
• Neutral - placement in chronic mental health
  setting
• Unfavorable Discharge
• Negative - lateral residential/group home move
  between agencies, step up to more restrictive
  setting, disruption from placement via runaway,
  hospital, detention/DOC

43
Sustained Favorable Discharge Rate

• Percentage of all spells from which youth that
  were favorably discharged were able to sustain
  their discharge placement for 180 days.
• Agencies can earn a fiscal bonus if they exceed
  performance benchmarks

44
Risk Adjustment

• Each provider serves a population of children
  with a different mix of characteristics
• To measure performance fairly, that case mix
  needs to be taken into account
• Risk adjustment attempts to level the playing
  field
• Since Illinois has adopted a no decline policy
  for youth referred for residential treatment,
  risk adjustment is critical for providers

45
System Changes

• Centralized Matching Team established to
  streamline and enhance admission process
• Discharge and Transitional Protocol developed to
  clarify roles and responsibilities at discharge
  and post-discharge
• Residential Treatment Outcomes System (RTOS)
  deployed to provide reporting and outcome
  tracking capability
• Targeted initiatives designed to enhance success

46
For more information

• Brice Bloom-Ellis
• Residential Quality Assurance Director
• Illinois Department of Children and Family
  Services
• Brice.Bloom-Ellis_at_illinois.gov
• Judge Kathleen A. Kearney
• Project Evaluator
• Children and Family Research Center
• University of Illinois at Urbana-Champaign
• kkearney_at_illinois.edu

47

• MAINTENANCE FACTORS IN FOSTER CARE CASE
  MANAGEMENTTHE MISSOURI PROJECT ON
  PERFORMANCE-BASED OUT-OF-HOME CARE
• Paul Sundet
• Project Evaluator

48
PROJECT BACKGROUND

• 1988 2005 contracted case management in
  metro areas
• 2003 2005 detailed collaborative planning and
  design for PBC
• 2005 2008 initial competitive independent PBC
  contract executed
• 2008 -2011 second competitive contracts awarded
  

49
MISSOURI PBC DEMOGRAPHICS

• Three Year Contract Period (2005-2008)
• Urban Concentration
• 2548 Base Caseload In FCCM Agencies
• 7 Consortia Encompassing 25 Private Agencies (32
  locations)
• Random Assignment Of Cases

50
Research Sites
51
Project Emphasis

• Elements Contributing To Or Impeding
  Sustainability
• Indicators For Contract, Policy, Procedure And
  Practice Modification
• Comparative Case Outcomes
• Improving Clinical Case Processes

52
Research Design Elements

• Pre-intervention Baseline (2003-2005)
• Concurrent Public/Private Site Performance
  Analysis
• Mirror Sites Within Public Agency
• Decision Rule On Locales Studied
• Research Advisory Board

53
GOALS FOR CASE OUTCOMES

• Re-entry
• Target of children in custody or supervision
  of the state must not re-enter within 12 months
  of exit
• Stability
• Target of children shall experience 2 or less
  placement settings
• Permanency
• Target of out-of-home-children must achieve
  permanency
• Safety
• Target of out-of-home children must not have a
  substantiated CAN report with the alternative
  caregiver as the perpetrator

54
VARIABLE CONTROL

• Demographic changes
• Control group modifications
• Judicial system interventions
• Budget fluctuations

55
PBC IMPACTS TO DATE

1. COMPARATIVE CASE OUTCOMES
2. SYSTEM MODIFICATIONS
3. QA INITIATIVES
4. OVERALL RESULTS

56
ST. LOUIS REGION
         
             
  Base Yr 1 Base Yr 2 PY-1 CD PY-1 Pvt PY-2 CD PY-2 Pvt
             
             
Reentries            
gt91.4 98 98.20 98.17 96.20 98.68 95.40
             
Stability            
gt82 86 85.51 83.00 92 85 80
             
Permanency            
gt32 32.08 28.16 32.53 26 29.63 30
             
Safety            
gt99.43 99.24 99.73 99.72 100 99.82 100.00
57
SPRINGFIELD REGION

           
                   
  Base Yr 1 Base Yr 2 PY-1 CD PY-1 Mirror PY-1 Mirror PY-1 Pvt PY-2 CD PY-2 Mirror PY-2 Pvt
                   
                   
Reentries                  
gt91.4 99 98.80 99.40 100 100 95.40 99.20 94.00 96.80
                   
Stability                  
gt82 87 89.50 88.30 95 95 97 85 86 87
                   
Permanency                  
gt24 21.90 24.30 37.20 20 20 23 26.10 34 27
                   
Safety                  
gt99.43 99.80 99.90 99.70 99.61 99.61 100 99.61 99.56 99.80
58
KANSAS CITY REGION
           
                 
  Base Yr 1 Base Yr 2 PY-1 CD PY-1 Mirror PY-1 Pvt PY-2 CD PY-2 Mirror PY-2 Pvt
                 
                 
Reentries                
gt91.4 98 97.10 96.00 81 94.90 97.80 85.20 90.90
                 
Stability                
gt82 94 89.40 84.10 89 94 87 79 86
                 
Permanency                
gt30 28.80 29.50 37.90 40 37 43.20 34 33
                 
Safety                
gt99.43 99.32 99.53 99.63 100.00 100 99.64 100.00 99.64
59
SYSTEM MODIFICATIONS

• Reduced public agency caseloads
• State-wide accreditation almost achieved
• Greater rigor in case reporting and
  accountability
• Professional competition to achieve goals
• Upgrading of public worker training and
  supervision

60
QA INITIATIVES

• Joint public/private QA/QI best practices
  training
• Public/private practice summit
• Joint CFSR preparation
• Peer Record Review exchange teams
• Practice Development Reviews
• Multi-level CQI process
• Increase in Oversight Specialist Staff

61
OVERALL RESULTS

• Case outcomes are improving in both contract and
  public agency settings
• Financial incentives do not appear to produce
  statistically significant better results
• Lower caseload sizes and close supervision appear
  to be critical to improved results
• Professionally directed competition is beneficial

62
The Collaborative Research Process Is Designed to
Promote Usable Evidence in the Real World of
Public/Private Partnership
Stay Tuned!
Source Gredig, D. and Summerfeld, P. (2008).
New Proposals for Generating and Exploiting
Solution-Oriented Knowledge. Research on Social
Work Practice, 18.