Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings

1
Early Learning in Mathematics (ELM)The Efficacy
of a Kindergarten Curriculum Implemented in Whole
Classroom Settings

• Scott K. Baker, PhD
• Pacific Institutes for Research / University of
  Oregon
• Ben Clarke, PhD
• Pacific Institutes for Research
• Hank Fien, PhD
• University of Oregon
• Keith Smolkowski, PhD
• Oregon Research Institute
• Chris Doabler, PhD
• Pacific Institutes for Research
• David Chard, PhD
• Southern Methodist University
• IES Conference June 2010

2
Acknowledgements

• Institute of Education Sciences
• The research reported here was supported by the
  Institute of Education Sciences, U.S. Department
  of Education, through Goal 2 development grant,
  R305K040081, and a Goal 3 efficacy grant,
  R305A080114, to Pacific Institutes for Research.
  The opinions expressed are those of the authors
  and do not represent views of the Institute or
  the U.S. Department of Education.
• Additional Oregon Project Staff
• Kathy Jungjohann / Karen Davis Curriculum
  development
• Mari Strand Cary / Rhonda Griffiths Coordination
  and research
• Chris Doabler Observation measurement and
  research

3
Early Learning in Mathematics (ELM)

• 4-year randomized efficacy control trial
• Measuring the efficacy of a kindergarten
  mathematics curriculum in Oregon and Texas.
• Research Design
• Randomized Controlled Block Design
• Classrooms within school matched on full / half
  day and randomly assigned to treatment (ELM) or
  control conditions

4
Purpose of the 4-year project

• Study 1 Efficacy trial of the whole group
  curriculum (ELM) on kindergarten students
  mathematics achievement
• Study 2 Efficacy trial of small group ELM
  component Roots on the achievement of at-risk
  students
• Examine potential mediation variables,
  dose-response variables, and moderation factors

5
Structure of the Curriculum

• Daily Calendar Lessons / Activities
• 15 minutes daily, whole class circle time
• Monthly booklets with objectives and application
  activities
• 120 Core Lessons divided into 4 quarters
• 30 minutes whole class instruction
• 15 minutes teacher directed written work
• End of quarter assessment of progress

6
ELM Instructional Content

• National Math Advisory Panel (2008) recommends a
  focused, coherent progression of mathematics
  learning with emphasis on proficiency with key
  topics
• ELM focuses on key strands rather than a broad
  array of mathematical content
• Numbers and Operations
• Geometry
• Measurement
• Vocabulary (NCTM Process Standard, 2000)

NCTM Curriculum Focal Points for K (2006)
7
ELM Conceptual Framework
8
Research questions

• What is the immediate impact of ELM taught in
  general education kindergarten classrooms on
  mathematics achievement compared to standard
  district practice?
• Is impact moderated by student level of risk for
  mathematics difficulties?
• Does rate of teacher models or student practice
  opportunities mediate a condition effect?
• Is there evidence of an interaction between
  condition and student practice on student
  outcomes?

9
Study Sample

• Assignment at the classroom level blocked on
  school
• Districts 3
• Schools 24
• Intervention classrooms 34
• Control classrooms 30
• Students nested in classrooms
• Whole class instruction
• N 1,349

10
Student Demographics

• 56.3 eligible for free or reduced lunch
• 38.4 English Learners
• 8.4 special education
• 49.5 White
• 36.4 Latino
• 4.8 Asian / Pacific Islander
• 2.3 African American

11
Hypothesized Model of ELM Impact
Intervention
Mediators
Proximal Outcomes
Distal Outcomes
12
Descriptive data on implementation

• Classroom fidelity observations treatment and
  control
• Classroom observations focusing on instructional
  interactions treatment and control
• Teacher logs addressing content coverage
  treatment and control classrooms

13
Implementation Fidelity

• General ratings (8 items)
• Models skills/concepts appropriately and with
  ease
• Engages students in learning throughout the
  lesson
• Uses ELM / completes all lesson activities
  (dichotomous)
• For each ELM activity (range 1-7 per lesson)
  Full (2) / Partial (1) / Not Taught (0)

14
Implementation Fidelity Data ELM Lesson
Activities

• 81 ELM (fidelity) observations during the year
• Fall mean 1.71 (SD .19)
• Winter mean 1.65 (SD .33)
• Spring mean 1.62 (SD .43)
• Overall mean 1.65 (SD .36) (83 of Full)
• 2 of 81 lessons had a mean below Partial (1)
  level of implementation

15
Student measures of impact

• Test of Early Mathematics Ability (TEMA)
• Early Numeracy CBM
• Oral Counting
• Number Identification
• Quantity Discrimination
• Missing Number

16
Method and Analysis Framework

• Competing curricula
• All students received instruction
• Time balanced across conditions
• Sample
• At risk (some or high risk)
• lt 40th percentile
• 66 of student sample
• No risk
• 40th percentile
• 34 of student sample

17
Nested Time Condition Analysis

• Outcome net differences from pre to post
• Nested students within classrooms
• Control for nonindependence (e.g., ICCs)
• Controls for teacher effects
• Maximum likelihood (restricted)
• Includes all cases with data at either T1 or T2
• Reduces bias from missing data
• Moderation added Time Risk Condition
  interaction
• Effect sizes Hedges g

18
Sample Means, SDs, and Ns
T No Risk No Risk Some Risk Some Risk
ELM Control ELM Control
TEMA Raw T1 29.0 (6.8) 28.8 (7.3) 14.1 (6.0) 14.6 (6.2)
T2 39.6 (6.3) 39.1 (7.9) 28.6 (8.4) 26.9 (8.0)
TEMA Percentile T1 66.5 (16.4) 65.1 (17.4) 14.5 (11.4) 15.4 (11.7)
T2 70.4 (18.3) 68.1 (20.8) 35.5 (23.6) 31.0 (22.4)
CBM Total T1 120.3 (44.0) 116.7 (43.5) 45.0 (32.6) 45.5 (35.5)
T2 193.1 (35.8) 187.9 (38.1) 138.4 (50.3) 126.6 (50.5)
Sample Size T1 203 181 397 343
T2 190 174 341 312
Note. Standard deviations (SDs) presented in parentheses. For students with some risk at T1, we collected TEMAs from 53 fewer students in ELM classrooms and 48 fewer students in control classrooms. Note. Standard deviations (SDs) presented in parentheses. For students with some risk at T1, we collected TEMAs from 53 fewer students in ELM classrooms and 48 fewer students in control classrooms. Note. Standard deviations (SDs) presented in parentheses. For students with some risk at T1, we collected TEMAs from 53 fewer students in ELM classrooms and 48 fewer students in control classrooms. Note. Standard deviations (SDs) presented in parentheses. For students with some risk at T1, we collected TEMAs from 53 fewer students in ELM classrooms and 48 fewer students in control classrooms. Note. Standard deviations (SDs) presented in parentheses. For students with some risk at T1, we collected TEMAs from 53 fewer students in ELM classrooms and 48 fewer students in control classrooms. Note. Standard deviations (SDs) presented in parentheses. For students with some risk at T1, we collected TEMAs from 53 fewer students in ELM classrooms and 48 fewer students in control classrooms.
19
TEMA Percentile Scores Gains by Condition

• Gains
• Control 10.94
• ELM 14.73
• Difference 3.79
• Test of Condition
• t 2.10
• df 61
• p .0396
• ES 0.14
• T1 differences were not statistically significant
  (t 0.57)

20
CBM Total Scores Gains by Condition

• Gains
• Control 77.20
• ELM 84.87
• Difference 7.67
• Test of Condition
• t 1.99
• df 61
• p .0509
• ES 0.14
• T1 differences were not statistically significant
  (t 0.60)

21
TEMA Raw ScoresCondition by Risk Status

• Main Effects
• Difference in gains 1.32
• t 2.41, df 61, p .0190
• Condition by Risk Status
• t 2.47, df 61, p .0162
• No Risk 40th tile
• Difference in gains 0.04
• t 0.51, df 61, p .9586
• Risk lt 40th tile
• Difference in gains 1.98
• t 3.29, df 61, p .0017

22
CBM Total ScoresCondition by Risk Status

• Main Effects
• Difference in gains 7.67
• t 1.99, df 61, p .0509
• Condition by Risk Status
• t 2.24, df 61, p .0289
• No Risk 40th tile
• Difference in gains -0.27
• t -0.05, df 61, p .9570
• Risk lt 40th tile
• Difference in gains 10.81
• t 2.54, df 61, p .0138

23
Effect Sizes (Hedges g)
Measure Not At Risk At Risk
Tema Raw Score 0.006 0.242
EN-CBM Total 0.014 0.215
p lt .05 p lt .01
24
Summary

• ELM classrooms outperformed controls
• TEMA raw and percentile scores
• EN-CBM Total
• Students at risk
• Improve on all measures more than no-risk
  students
• Control students at risk catching up on no-risk
  students
• TEMA 14.0 percentile gain on no-risk students
• EN-CBM 9.6 point gain on no-risk students
• ELM students at risk catching no-risk students
  faster
• TEMA 18.6 percentile gain on no-risk students
• EN-CBM 20.63 point gain on no-risk students
• No condition effects for students with no risk

25
Preliminary Analysis of Association between
Observation Data and Student Outcomes
Intervention
Mediators
Proximal Outcomes
Distal Outcomes
26
Coding of Academic Teacher-Student Interactions
(CATS) Observation Instrument

• CATS uses a frequency count approach to measure
  teacher-student instructional interactions
• Observers code behavior occurrences in a
  continual, serial fashion.
• Utilizes a strict coding structure
• CATS based on evidence of effective instruction
  in early literacy and beginning mathematics, and
  adapted from the STICO observation instrument
  (Smolkowski Gunn, 2010)

STUDENT BEHAVIORS TEACHER BEHAVIORS
Individual responses Group responses Covert responses Mistakes Teacher models Academic feedback
27
Hypothetical Case of a Instructional Interaction
28
(No Transcript)
29
The Role of Teacher Modeling and Student Practice
in Student Outcomes
30
Preliminary Mediation Analysis

• Does rate of teacher models or student practice
  opportunities mediate condition effect?
• Rates of (a) teacher models, (b) student group
  practice opportunity and (c) individual student
  practice opportunities entered as mediators to
  determine if they decreased condition effect
• Condition effect was still significant
• No evidence to support this mediation hypothesis

31
Secondary Analysis

• If student practice overall is not mediating
  impact, perhaps the value (quality) of practice
  differs by classroom and is related to condition

32
Interaction between Rate of Practice and Condition

• By condition do rates of practice opportunities
  show the same pattern of impact on student
  outcomes?
• Are treatment control differences on student
  outcomes similar in classrooms with high rates of
  practice vs. low rates of practice?

33
Interaction between Rate of Practice and Condition

• Number of Classrooms by Treatment Condition and
  Median Rate of Individual and Group Practice
  Opportunities
• Practice Opportunity Quartiles in Rate per Minute

Control ELM Total
Above Median 6 26 32
Below Median 24 8 32
Total 30 34 64

Control ELM Total
Minimum 0.4 1.0 0.4
25th ile 0.8 1.9 1.2
Median 1.3 2.3 1.9
75th ile 1.7 3.0 2.4
Maximum 4.1 4.1 4.1
34
TEMA Scores by Rate of Practice

• Within ELM condition
• High-practice classrooms outperform low-practice
  classrooms
• Difference 2.59, t 2.58, df 29, p 0.0151
• Within control condition
• No difference between high- and low-practice
  classrooms
• Difference 0.39, t 0.31, df 29, p .7575
• Within classrooms with an above-median practice
  rate
• ELM classrooms (might) outperform control
  classrooms
• Difference 2.33, t 1.85, df 29, p .0747
• Within classrooms with a below-median practice
  rate
• No difference between ELM and control classrooms
• Difference 0.14, t 0.14, df 29, p .8935

35
EN-CBM Scores by Rate of Practice

• Within ELM condition
• No difference between high- and low-practice
  classrooms
• Difference 9.83, t 1.46, df 29, p 0.1538
• Within control condition
• No difference between high- and low-practice
  classrooms
• Difference -7.59, t -0.92, df 29, p .3635
• Within classrooms with an above-median practice
  rate
• ELM classrooms outperform control classrooms
• Difference 19.37, t 2.35, df 29, p .0257
• Within classrooms with a below-median practice
  rate
• No difference between ELM and control classrooms
• Difference 1.95, t 0.29, df 29, p .7731

36
Next steps

• Have just completed implementation of Study 1 in
  Dallas, Texas
• Have just completed implementation of Study 2 in
  Oregon
• Will implement Study 2 in Dallas, Texas in
  2010-11
• Ongoing analysis to investigate impact of
  condition and mediation and moderation variables
  associated with impact