Early Childhood Mental Health Consultation ECMHC Evaluation Protocol

1
Early Childhood Mental Health Consultation
(ECMH-C) Evaluation Protocol

• A Partnership between
• University of Maryland, Baltimore
• Georgetown University
• CKD Communications, LLC
• April 6, 2009

2
Project Details

• Principal Investigators
• David Pruitt, MD UMD
• Bruno Anthony, PhD Deborah Perry, PhD
  Georgetown University
• Project Coordinator
• Sarah James, UMD
• Funding
• Maryland State Department of Education (MSDE),
  Division of Early Childhood
• Involves 12 sites across Maryland
• Three year evaluation (2008-2011)

3
Early Childhood Mental Health Consultation (ECMHC)

• Promotes social emotional development and seeks
  to reduce childrens challenging behavior
• Primarily indirect services that build the
  capacity of child care providers and family
  members
• The impact can be measured in the domains of
  child, parent, staff, and family

4
Stakeholders

• MSDE
• Linda Zang
• Tresa Hanna
• 12 ECMH Program Directors
• Mental Health Consultants
• Early care and education providers (ECE
  providers)
• Parents (legal guardians of referred children)
• Maryland Committee for Children

• Evaluation Liaisons
• Bruno Anthony
• Deborah Perry
• Barbara Baumgardner
• Nikki Darling-Kuria
• Karen Pell
• Jennifer Mettrick
• Sarah James

5
Early Care and Education (ECE) providers

• Center-based and family child care providers who
  receive consultation services from the funded
  projects.
• Classroom staff in center-based ECE programs
• ECE Directors in charge of center-based ECE
  programs
• Family child care providers.

6
Funded Projects and Evaluation Liaisons
Group II Sites Abilities Network (Harford and
Cecil Counties), Child Care Choices, Prince
Georges Child Resource Center
7
Evaluation Logic Model
8
Approach to Evaluation

• Collaborative
• Participatory
• Comprehensive
• Sensitive to respondent burden
• No site comparisons
• Technical Assistance related to evaluation and
  data collection will be provided

9
Evaluation Activities and Timelines 2008-2009
Pilot data collection (all components listed on
the Data Collection Plan Summary
First quarterly report to MSDE
Finalize data collection plans
Submit IRB proposal
Begin full evaluation
Feb. March April May
June July Aug. Sep.
Train ECMHC programs on evaluation protocol
webinar and liaison visits
Obtain feedback on evaluation activities from
programs
Adjust evaluation protocol as needed
Technical assistance by evaluation liaisons
provided on a monthly basis (phone/email/in
person) in addition to quarterly in-depth site
visits
10
Evaluation Components
Implementation Factors
Core Outcomes
Expulsions
ECE Providers SDQ TOS RQS DECA
ECE Providers Qualitative Interviews
Program Directors Model Description Quarterly
Interview
Parents Qualitative Interviews
Stakeholder Reports
Parents PSI PBI DECA
Consultants Knowledge Skills Service Log
ECE Directors Qualitative Interviews
ECE Directors GAS
Consultants Climate RQS
Program Directors Monthly Log
Note instruments in red will only be collected
for child-specific referrals
11
Levels of Data Collection

• Changes in program-level outcomes
• For all EMCHC models, measures are designed to
  assess changes in ECE provider behaviors and
  beliefs structural and interpersonal
  characteristics of the ECE setting and the
  quality of the relationship between the MHC and
  ECE providers.
• Child-/Family level outcomes
• For child-specific referrals where an individual
  childs social, emotional and/or behavioral
  problems are the focus of the MHC in the ECE
  setting and/or the MHC is working closely with
  the family on addressing such behaviors at home
• Specific (additional) measures are designed to
  assess changes in the childs behavior, parenting
  stress and behaviors

12
Evaluation Instruments
13
Model Description

• This evaluation tool is completed by the 12
  Program Directors to document information about
  each ECMHC funded project being evaluated. More
  specifically, it gathers information at the
  beginning of the evaluation such as
• Project name
• Number of years in operation
• Ages of children receiving services
• Setting in which services are provided
• Quality of early child care and education
  programs
• Duration of services
• Intensity of services

14
Quarterly Site Interviews

• This semi-structured interview will be conducted
  by the evaluation liaisons with each Program
  Director to track any/all changes in program
  implementation, challenges, successes, staffing,
  and other important issues related to operating
  the funded project.
• Quarterly interviews will be used to update model
  descriptions.

15
Monthly Tracking of Expulsions

• Each Program Director will provide the evaluation
  team with a summary of the number of children
  expelled from ECE settings that are actively
  receiving child-specific or programmatic ECMHC.
• Families of children who have been expelled will
  be invited to participate in a semi-structured
  interview about their experiences
• ECE providers who expelled a child will be
  invited to participate in a semi-structured
  interview about their experiences

16
Knowledge Skills (KS) Inventory

• This tool will be completed by the Mental Health
  Consultants each time a new consultant is hired
  and updated annually. This inventory collects
  background and demographic information on the
  about their professional and educational
  experiences, and knowledge of early childhood
  mental health. For example, it documents
  information such as
• Highest degree
• Years of experience
• Race and ethnicity
• Specific knowledge of and skills in early
  childhood development

17
Service Log

• This evaluation tool will be completed by the
  Mental Health Consultants to document the
  frequency, intensity and type of consultation
  services being provided.
• A service log form will be provided by the
  evaluation liaison but each funded project will
  be encouraged to use data already collected by
  their consultants to document services provided.
  This process will be customized for each ECMHC
  site.

18
Strengths Difficulties Questionnaire (SDQ)

• ECE providers will complete this instrument for
  each child in their care at the onset of ECMHC
  services. The purpose of this tool is to assess
  the level of problem behaviors in the classroom
  or family child care setting.
• SDQ Impact Form There are 5 yes or no questions
  on this form. If the ECE provider indicates that
  an individual child has no problem behavior, they
  only need to answer one-question, and then move
  on to the next child.
• Additional data from the SDQ will be gathered at
  3 months and 6 months after consultation services
  began

19
Teacher Opinion Survey (TOS)

• This 13-item scale will be completed by each ECE
  provider when they begin ECMHC services. It is
  designed to assess teachers attitudes about
  their
• confidence and competence in managing challenging
  behaviors
• ability to make a positive difference in
  childrens lives
• Additional data on the TOS will be gathered at 3
  months and 6 months after consultation began.

20
Relationship Quality Scales (RQS)

• These surveys will be completed by the MHC, ECE
  provider and family members who participated in
  ECMHC once the consultation is completed. They
  are 20 items and the tool assesses the quality of
  the relationship between the consultants,
  teachers and the parents.
• There are three parallel versions
• Teacher Final Perceptions Form
• Parent Final Perceptions Form
• Consultant Final Perceptions Form

21
Devereux Early Children Assessment (DECA)

• Parents and ECE providers who are participating
  in child-specific ECMHC will complete this
  standardized, norm-referenced behavior rating
  scale at a minimum of 3 time points when the
  child is initially referred, 3 months later and 6
  months later. Some families may consent to
  longer-term follow-up for up to 2-years.
• The DECA is already used by most of the funded
  ECMHC projects. The DECA features 27 positive
  behaviors and 10 item behavioral screen for
  preschool children ages 2-5 there is also an
  infant form for children under age 2.

22
Goal Achievement Scale (GAS)

• This tool will be completed by the ECE Director
  where consultation is being provided. This
  evaluation tool is used to assess the directors
  opinions and attitudes when consultation is
  initiated and completed.
  
• Family child care providers may complete this
  question if they also see themselves as the
  directors of their programs.
• Additional data on the GAS will be gathered at 3
  months and 6 months after consultation began.

23
Parent Behavior Inventory (PBI)

• The Parent Behavior Inventory is a 20-item
  assessment used to gain insight into parenting
  practices and how they interact with their
  children.
• This tool will only be completed by parents whose
  children were recipients of child-specific
  consultation services.
• Parents will be asked to complete the form when
  services are initiated, after 3 months, and again
  when consultation services are completed.

24
Parenting Stress Index (PSI)

• The Parenting Stress Index is a 37-item tool that
  assesses parental distress and parent-child
  interaction
• This tool will only be completed by parents whose
  children were recipients of child-specific
  consultation services.
• Parents will be asked to complete the form when
  services are initiated, after 3 months, and again
  when consultation services are completed.

25
Preschool Climate Scale

• This tool will be completed by the mental health
  consultants for every child-specific and
  programmatic consultation they initiate.
• The instrument assesses multiple dimensions of
  the classroom environment and ECE providers
  behaviors.
• They will complete this scale at baseline, 3
  months and 6 months after initiation of ECMHC.

26

• Which Evaluation Tools will ECE Providers Be
  Asked to Complete?

27
Instrument Completion-ECE Providers
28

• Which Evaluation Tools will Child Care Directors
  Be Asked to Complete?

29
Goal Achievement Scale (GAS)

• This tool will be completed by the ECE Director
  where consultation is being provided. This
  evaluation tool is used to assess the directors
  opinions and attitudes when consultation is
  initiated and completed.
  
• Family child care providers may complete this
  question if they also see themselves as the
  directors of their programs.
• Additional data on the GAS will be gathered at 3
  months and 6 months after consultation began.

30
Monthly Expulsion Tracking

• Who completes the tool?
• This evaluation tool will be completed by ECE
  Directors and Family Child Care Providers
  receiving ECMHC services. It will complement
  in-depth interviews, a dimension of the
  qualitative research.
• When should the tool be completed?
• ECE Directors and Family Child Care Providers
  will track expulsions on a continuous basis.
• How will the data be provided to the evaluation
  team?
• This tool will be collected by the MH consultants
  at least monthly and submitted to their
  evaluation liaisons on a continuous basis.
• Level of data collection
• This tool will be used for both child and
  program referrals.

31

• Which Evaluation Tools will Parents Be Asked To
  Complete?

32
Instrument Completion - Parents
For those who consent to long-term follow along
33

• Which Evaluation Tools will MH Consultants
  Complete?

34
Instrument Completion - Consultants
35

• ECMHC Program Director
• Participation

36
Instrument Completion Program Directors
37
Model Description

• Who completes the tool?
• This evaluation tool will be completed by the
  ECMHC Program Directors of each participatory
  site.
• When should the tool be completed?
• ECMHC Program Directors have already completed
  this tool at baseline. It will continue to be
  completed on a annual basis.
• How will the data be provided to the evaluation
  team?
• This evaluation tool will be collected by the
  evaluation liaison.

38
Quarterly Site Interviews

• Who completes the tool?
• This evaluation tool will be completed by the
  ECMHC Program Directors at each participatory
  site.
• When should to tool be completed?
• ECMHC Program Directors are required to complete
  this tool with their liaisons during their
  quarterly site visits.
• How will the data be provided to the evaluation
  team?
• This evaluation tool will be collected by the
  evaluation liaison on a quarterly basis.

39
Flow of Data Completion and Collection
Program Directors Quarterly Interviews
w/ Evaluation Liaisons
Consultant ECE Provider Parent Child Care
Director Liaison Data
Completion
Evaluation Liaisons
Collection
Consultants
Data Summaries
Quarterly Reports
MSDE and Stakeholders
40
IRB Consent Procedures

• Evaluation Liaisons will review and obtain
  consent forms from ECMHC consultants and agency
  directors
• This will occur at the start of the evaluation
• The consent form will cover all forms of
  participation in the evaluation
• At any time however, if there is a particular
  tool or question from a tool that you do not wish
  to complete, it is your right to refuse
• Participation in the evaluation is voluntary and
  will not be associated with your job performance

41
IRB Consent Procedures (Cont.)

• Consent forms for ECE providers, Child Care
  Directors and Parents
• We will ask ECMHC consultants to deliver consent
  forms and survey instruments to each of the above
  respondents
• Consent forms and Instruments will be delivered
  in sealed envelopes. A self-addressed stamped
  envelope to the ECMHC Evaluation team will be
  provided for respondents
• ECMHC consultants will not be asked to review
  consent forms with ECE providers, Child Care
  Directors or Parents. Per IRB mandate only those
  on the Evaluation team can review consent
  procedures and answer questions regarding the
  evaluation.

42
What to Expect from Your Evaluation Liaison

• Provide technical assistance and tools for
  program evaluation processes
• Set up the necessary data collection procedures
• Inform Program Director of all evaluation
  developments and share site specific findings
• Assist Program Director with local evaluation
  issues as relevant
• Refer Program Director, MH consultants and others
  to community resources for technical assistance
  when needed
• Facilitate trainings for all participants in the
  evaluation on tools, as needed

43
For More Information

• Website
• http//medschool.umaryland.edu/innovations/ECMHC_e
  valuations.asp
• Contact
• Sarah James
• 410-706-5764
• sjames_at_psych.umaryland.edu