Framework for Developing Evidence-Based Early Literacy Learning Practices

1
Measuring Fidelity in Early Childhood Scaling-Up
Initiatives A Framework and Examples Carl
J. Dunst, Ph.D. Orelena Hawks Puckett
Institute Asheville, North Carolina Presentat
ion made at the Fidelity How Much Devotion is
Enough? (and Other Methodological Virtues)
session at the Office of Special Education
Programs Project Directors Conference, Washington
, DC, July 2008
2
Purposes of the Presentation

• Describe a framework for measuring fidelity as
  part
• of scaling-up early literacy learning
  practices.
• Define different types of fidelity at different
  levels of
• training and practice.
• Illustrate how variations in fidelity can be
  related to
• variations in practice adoption and outcomes.

3
Center for Early Literacy Learning

• The main aims of CELL are to (1) Synthesize
  research
• evidence on effective early literacy practices,
  (2) develop
• evidence-based practices from the research
  syntheses,
• (3) evaluate the validity and use of the
  practices, and (4)
• conduct both generalized and specialized
  technical
• assistance promoting adoption and use of CELL
  practices.

4
From Evidence-Based Practices to Fidelity to
Outcomes
Developing Evidence-Based Early Literacy Learning
Practices
Scaling-Up the Use of the Practices
Measuring the Fidelity of Use of the Practices
Child Outcomes
5
Early Literacy Learning
6
Specialized Technical Assistance
7
Framework for Measuring Fidelity
8
Types of Fidelity
Implementation
Diffusion
Intervention
9
Implementation Fidelity

• Implementation fidelity is defined as the degree
  to which
• CELL training activities of known characteristics
  are
• implemented as planned and promote participant
• understanding of the known characteristics of the
  evidence-
• based practices constituting the focus of
  training.

10
Diffusion Fidelity

• Diffusion fidelity is defined as the degree to
  which second
• and third generation technical assistance
  providers use
• CELL training methods and procedures of known
• characteristics to promote end-user understanding
  of the
• key characteristics of evidence-based practices.

11
Intervention Fidelity

• Intervention fidelity is defined as the extent to
  which end
• users adopt and use instructional practices of
  known
• characteristics for implementing evidence-based
  practices
• of known characteristics.

12
Elements of Fidelity

• Each type of fidelity includes two elements

• Fidelity of the training processes (how)
• Fidelity of practice understanding and adoption
  (what)

13
Elements of Fidelity

• Training processes are HOW we promote adoption
  and use
• of WHAT we want implemented by end-users. As
  state by
• Fixsen et al. (2005), it is important to be aware
  of the
• difference between the practices constituting the
  focus of
• training and the training methods used to promote
  adoption
• of the practices.

14
Framework for Structuring Fidelity Assessments
15
CELL Fidelity Checklist for MeasuringImplementati
on and Diffusion Fidelity

• Fidelity of both implementation and diffusion
  training is
• being measured by a CELL fidelity checklist that
  assesses
• the degree to which CELL training processes and
  CELL
• practices are accomplished with (1) sufficient
  dose (2)
• and with ample quality exposure to (3) key
  training and
• practice characteristics, and (4) are viewed by
  participants
• as socially valid.

16
(No Transcript)
17
Measuring Fidelity of Intervention

• Fidelity of intervention is measured in terms of
  practitioner
• or parent, or both, use of instructional
  practices and
• evidence-based practices of known characteristics
  as part
• of the provision of everyday early childhood
  intervention.
• Fidelity is measured by the frequency and number
  of
• practices that are used and the
  development-enhancing
• qualities (active ingredients, core features,
  etc.) of the
• intervention practices. The social validity of
  both the
• practices and the child outcomes from using the
  practices
• are also assessed.

18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
Relationship Between Fidelity and Child Outcomes
Low Users
High Users
a
a
d
Practice Measures
Mean
Mean
F-Test
Cohens
Number of Weeks -.21
.16 1.26 .37
Number of Learning Games -.32
.32 5.70 .67
Game Characteristics -.61
.51 5.73 1.32
a
Mean standardized score.
p lt .05.
22
Further Evaluation of the Effectivenessof the
Practice

• Additional information about the effectiveness of
  the
• practices was determined by a post-test only
  quasi-
• experimental between group design that included
  two
• outcomes, one of which was a dependant variable
  that a
• treatment should affect and a second dependant
• measure that was not expected to be related to
  the use of
• the practice (Shadish et al., 2002, p.184).
  Obtaining an
• expected group by outcome measure interaction was
  used
• to ascertain effectiveness.

23
(No Transcript)
24
Conclusions

• Fidelity is important as part of ascertaining the
  adoption and use of evidence-based practices.
• A fidelity framework can help structure the
  development and collection of fidelity measures.
• It is important to measure both implementation
  and intervention fidelity.
• Relating variation in fidelity to variation in
  outcomes can help identify what matters most in
  terms of targeted practices.

25
For More Information Visit

• www.puckett.org for methods and procedures for
  identifying evidence-based practices
• www.earlyliteracylearning.org for information on
  the Center for Early Literacy Learning.