Findings from the National Evaluation Of Child Welfare Training Grants: Lessons and Implications

1
Findings from the National Evaluation Of Child
Welfare Training GrantsLessons and Implications
Mary Elizabeth Collins, MSW, Ph.D. Maryann
Amodeo, MSW, Ph.D. Cassandra Clay, MSW,
EdM. Boston University School of Social
Work May 2007 Grant 9OCT0124 funded by U.S.
DHHS, Childrens Bureau
2
National Evaluation of Child Welfare Training
Grants

• Funded by the Childrens Bureau (Oct 2003- Sep
  2006)
• Large project multiple data collection tasks
  and several research questions
• Overall goal Greater understanding of
  federally-funded CWT and how to use information
  from the evaluation to improve our training
  strategies.

3
National Evaluation of Child Welfare Training
Four Main Components

• Case studies of 9 previously federally funded
  training projects Independent Living
• Comprehensive review of child welfare training
  literature
• Survey of faculty of (mostly) schools of social
  work
• Survey of state (and some county) child welfare
  training administrators

4
Update on Four Components

• Multiple case study of IL projects
• DONE, report available
• Review of child welfare training literature
• DONE, report available
• Phone interview of state child welfare training
  directors 90 done, response rate48/51, great
  data!
• Mail survey of faculty 60 done, N97 response
  rate54

5
Conceptual Model
6
Recommendations from Case Study Report
Childrens Bureau, Future Grantees, Public
Agencies

• Youth/Consumer Involvement
• e.g., Public agencies have been making progress
  in emphasizing more partnership with services
  users training initiatives and strategies are
  another mechanism by which agencies can engage
  consumers in their work.
• Evaluation
• e.g., Public child welfare agencies need to
  cooperate with the evaluation task. This will
  include allowing evaluators to conduct follow-up
  with trainees from the state agency. Examination
  of the transfer of learning to the agency setting
  and mastering skills in the work setting is
  sorely needed. This will nearly always require
  follow-up in the practice setting which will
  include data collection involving interviews,
  observations, case record reviews, and other
  methods. Additionally, access to comparison
  groups may be needed.
• Collaboration
• e.g., Grantees should aim to establish
  collaborations with organizations rather than
  individuals to provide more stability to the
  collaborations. Collaborators should be chosen
  in part to facilitate long term
  institutionalization. Thus, at the start of
  projects, active advocacy should occur to secure
  organizational collaborators who will share the
  work and responsibility for outcomes

7
Recommendations from Case Study Report
Childrens Bureau, Future Grantees, Public
Agencies

• Institutionalization
• e.g. The Childrens Bureau should aim to fund
  the type of training projects that are of
  critical need to public child welfare agencies.
  If the funding priorities of the Childrens
  Bureau are not aligned with the needs of the
  field, public child welfare agencies have no
  reason to engage in long-term institutionalization
  of training programs. After projects are
  completed, the Childrens Bureau is the only
  entity with appropriate infrastructure to keep
  the products that have been developed at the
  forefront of child welfare practice.
• Knowledge Development
• e.g., Grantees should recognize that their
  projects are opportunities to develop learning
  about the field of child welfare training that
  can and should be shared with wider
  constituencies. Although lessons learned are
  typically requested as a part of project
  reporting, the content is generally thin and
  lessons are not shared. Grantees should think
  conceptually about the core lesson of their
  project and disseminate the contributions via
  conference presentations and journal articles.
  These should be less focused on promoting
  projects and more focused on linking project
  innovations to the wider field of child welfare
  training.

8
Findings from the Literature Review

• Adult Learning Theory
• e.g., Organizational realities that are barriers
  to training based on these theories need to be
  confronted head-on. These barriers include time
  and space for sophisticated training
  professional orientation toward workers that
  recognizes their expertise and development of a
  learning culture in which organizational learning
  and development is the norm. Without addressing
  these barriers, training programs will have
  limited impact.
• Training Implementation
• e.g., The field should move toward the
  development of training systems rather than
  training courses. Training systems infuse
  training with an organizational context and have
  greater potential for viewing training
  holistically, with connection to achievement of
  organizational outcomes. This will add to better
  conceptualization of the purposes of training and
  the anticipated linkage of training content and
  training activities with expected outcomes

9
Findings from the Literature Review

• Agency/University Partnerships
• e.g., As with other forms of training, better
  evaluation is needed to assess the results of
  partnership efforts. Most of the literature
  emphasizes the benefits of partnerships, but
  there are limitations. Sometimes these are
  discussed as logistical problems or the melding
  of academic and practice cultures. Partnerships
  may limit the voice of schools of social work to
  critique child welfare practice and policy, and
  may cause schools to shift toward vocational
  education rather than professional education.
• Evaluation of Training
• e.g., The methodology for evaluating outcomes
  related to knowledge and attitude change are
  fairly well developed and easily implemented.
  The measurement of skill development is more
  complicated and requires more methodological
  sophistication. Measures such as case plans and
  action plans should be more fully developed for
  use in training evaluation. Also observational
  methods and measures should receive attention so
  that the field can begin to measure the
  interaction of worker and client.

10
Survey of State and County Child Welfare Training
Administrators -- Content

• Main challenges agency faces in providing
  effective CWT
• Experiences with federally-funded CWT
• Description
• Strengths/Weaknesses
• Impact
• Involvement with IV-E partnerships
• Description
• Strengths/Weaknesses
• Impact
• State/county funded training programs

11
Survey of State and County Child Welfare Training
Administrators -- Content

• Administrative supports for training
• Impact of CFSRs on training
• Existing mechanisms to institutionalize training
• Relevant historical factors that have influenced
  CWT
• Future initiatives in CWT
• Training evaluation
• What could federal government do to help states
  and counties deliver effective CWT?
• What should be federal priorities (topics) of CWT
  in the next few years?

12
Survey of State and County Child Welfare Training
Administrators Training Evaluation

• Many (n18) refer to levels of evaluation
• Level 1/satisfaction, n 8 Would like to do
  more
• Level 2/pre-post change, n13 with some moving
  toward more complex evaluation for some training.
• Additional evaluation steps e.g., longitudinal
  follow-up, n20
• Most developed training systems, n4

13
Survey of State and County Child Welfare Training
Administrators Training Evaluation

• The success of training is first measured when
  trainees take a test. followed by a field
  based assessment piece, which is left up to
  various supervisors. The supervisors evaluation
  includes an evaluation form, document review and
  observation of workers in various case management
  activities. Also an online tracking system,
  which tells them who passes competencies and
  other information.

14
Survey of State and County Child Welfare Training
Administrators Training Evaluation

• Evaluation method varies based on the kind of
  program. Some Level 1 (happy sheets) that are
  standardized. Available electronicallywhen a
  course is offered, evaluation information is put
  in immediately, can access it. Use pre/post
  knowledge test for basic training and all
  specialty programs. Occasional attempts at level
  3 4 for very special situations its expensive
  and hard. Also have a system to do field
  evaluations conduct an on-site field review,
  either scheduled randomly or a project manager
  requests it.

15
Survey of State and County Child Welfare Training
Administrators Training Evaluation

• Utilization
• Generic statement about reviewing results (n9)
• Primarily to review courses and trainers (n23)
• With information system to tract participation,
  link with retention, overall quality assurance
  (n12)

16
Survey of State and County Child Welfare Training
Administrators Training Evaluation

• The information is compiled and examined
  quarterly. It is then utilized for the
  following 1. to look at trainers 2. are
  workers getting what they want? (i.e., if they
  need to change curriculum) 3. how is the
  environment? (are environmental changes needed).
  Future plans include a practice reform initiative
  with the University which will have its own
  formal evaluation and they are working towards
  developing a model that will include behavioral
  anchors for new workers.

17
Survey of Colleges/Universities Relationship
with Child Welfare Agency

• N97
• Your Relationship Your Schools
  Relationship
• Quality 3.46 3.16
• Amount 3.22 2.97
• Mutuality/Reciprocity 3.09 2.83
• Productivity 3.13 2.82
• Desire for Future Collaboration 3.73 3.44
• Scale 1 (low) 4 (high)

18
Survey of Colleges/Universities Perceived
Impact of Federally-Funded Projects (N30)

• Moderate/Strong
• n percent
• Recruitment of new CW staff 15 50
• Development of current CW staff 26 87
• CW worker knowledge 30 100
• CW worker attitude 23 85
• CW worker skills 27 90
• CW supervisor knowledge 22 76
• CW supervisor attitude 16 59
• CW supervisor skill 18 69
• Macro level CW policy/practice 11 47

19
Triangulation Common Findings

• Reluctance of state agency to fully support
  training and its evaluation
• Potential and pitfalls of partnerships between
  agencies and universities
• Training knowledge dominated by curriculum
  development and training delivery expansion
  needed in evaluation, organizations, and systems
• Training knowledge dispersed in variety of fields
  and ranges from highly micro (e.g.,
  instrumentation) to macro (e.g., policy
  implementation)

20
Discussion Topics

• Relative roles of faculty, state/county training
  agencies, and the federal government in advancing
  the state of training.
• Enhancing the knowledge base of child welfare
  training.
• Addressing conceptual and methodological
  challenges to conducting training.
• Assessing what is known and unknown in child
  welfare training.
• Challenges of linking training outcomes to
  performance outcomes.

21
For more information, contact

• Mary Elizabeth Collins, Ph.D.
• Boston University School of Social Work
• 264 Bay State Road
• Boston, MA 02215
• 617-353-4612
• mcollins_at_bu.edu