Title: Response to Intervention: Using Data to Enhance Outcomes for all Students
1Response to Intervention Using Data to Enhance
Outcomes for all Students
- Amanda VanDerHeyden
- Education Research and Consulting, Inc.
2Objectives Today
- Overview of RTI, RTI decision making, and
expected outcomes - Specific How-To for RTI
- Interpreting Assessment Data to determine need
for Tiers 1, 2, and 3 Interventions - Selection and implementation of Tier 2
Interventions - Selection and implementation of Tier 3
Interventions - Implementing intervention for sustenance and
system change
316 x 3 48 hours
4What is RTI?
- A science of decision making and way of thinking
about how educational resources can be allocated
(or reallocated) to best help all children learn - Major premium on child outcomes
5RTI is Not
- A program, a curriculum, an intervention, a
particular model
6Data allow us to
- Provide faster, more effective services for ALL
children - Work smarter not harder, better utilize the
talents of the school psychologist and
school-based assessment and intervention teams. - Make implementation SIMPLE and EASY for teachers
(low cost, few errors) - Prevent diagnosis
7Early Screening Identifies Children At Risk of
Reading Difficulty
J
5
4
Low Risk on Early Screening
Reading grade level
3
2
At Risk on Early Screening
1
1 2 3 4
Grade level corresponding to age
This Slide from Reading First Experts
From Reading First
8Early Intervention Changes Reading Outcomes
J
5.2
5
4
Low Risk on Early Screening
Reading grade level
3
2.5
2
At Risk on Early Screening
1
1 2 3 4
Grade level corresponding to age
This Slide from Reading First Experts
From Reading First
9Evolution
- Wait to Fail
- Lets provide services early!
- Costly sp ed programs not improving learning
- Lets shift resources to provide services in less
restrictive setting! - Increasing numbers of children struggling in
general ed - Lets provide help in general education!
- Traditional measures are de-contextualized and
the constructs are problematic - Lets help children who struggle academically by
measuring performance in response to certain
intervention strategies and then deliver what
works!
10Rationale for System Change
200-300 increase in SLD
- Level and Rate of Performance
- Return to General Education
- Lack of Certified Teachers
- No demonstrated instructional techniques that
differentially benefit SLD - Drop-out
- Disproportionate Representation by Ethnicity
11History of RTI
- Effective Instruction Lit, CBA/M Lit
- Lab-Quality Intervention Programs
- Progress Monitoring Data and Problem-Solving
Models - Reading First data
12Consistent with
- NCLB
- Reauthorized IDEA
- Recommendations of panel reports on Minority
students in special education, National Reading
Panel, Science and Math Initiative
13Impetus
- Faster, more effective services for ALL children
- Work smarter not harder, better utilize the
talents of the school psychologist and
school-based assessment and intervention teams. - Make implementation SIMPLE and EASY for teachers
(low cost, few errors)
14Why RTI?
- Viable alternative to traditional diagnosis of
high-incidence disabilities, particularly
Learning Disability (LD) - Reauthorized IDEA guidelines for identifying LD
state that - a) A severe discrepancy between achievement and
intellectual ability shall not be required - b) A response to intervention (RTI) may be
considered
15Why RTI?
- RTI can address the problem of disproportionate
identification of children with LD by race and
gender - The utility of curriculum-based measures (CBM)
for - Identifying children not likely to benefit from
the general education curriculum without
assistance, - Predicting important long-term outcomes
- Tracking individual student growth and informing
instructional programming changes has been
established
16Considerations
- There has been some consensus concerning the need
for change however, there has not been consensus
on how this change can best be achieved. - Whereas RTI has considerable promise as a tool
within special and general education, it is a
vulnerable construct if misapplied.
17Improved Treatment Validity
- Direct link to treatment or consequential
validity - Efforts are focused to
- Properly articulate a concern
- Develop targeted intervention to resolve the
concern - Collect information to determine whether or not
the concern has been adequately addressed or
whether different solution efforts need to be
implemented - This approach changes the goal of assessment from
what some have described as admiration of the
problem to problem-solving.
18Contextualized Decision Making
- RTI emphasizes the pre-referral conditions (child
and environment) and this context becomes part of
the decision-making equation. - Allows practitioners to quantify
- The state of instructional affairs in the childs
regular education environment - Potential learning given optimal instructional
conditions - RTI may enable improvement in general education
programming leading to children receiving
assistance in a more efficient manner.
19Improved Identification Accuracy for LD
- Under RTI models intervention becomes a
specified, operationalized variable, thus false
positive identification errors should be reduced
dramatically. - Removing the current reliance on teacher
identification and requiring direct measures of
child performance in context will enhance
identification accuracy.
20More Effective Intervention
- RTI is likely to facilitate less restrictive
interventions and placements for children. - RTI allows school psychologists to bring their
expertise to bear on assessment strategies at the
classroom level and assist teachers to use data
formatively to enhance their instructional
programming.
21- Possible Challenges of RTI
22Decision-making Criteria
- Must be operationalized and validated through
research - The purpose of RTI will be critical to
determining how implementation should proceed
23New Challenges for Teams
- Effective intervention delivery will depend on
relevant intervention variables - To be effective the intervention must have been
- Properly identified
- Implemented with integrity and with sufficient
frequency, intensity, and duration
24STEEP Model
- Screening to Enhance Educational Progress
25Tier 1 Math Screening
- Math Probe
- Group administered.
- Materials Worksheet consisting of a series of
problems sampling the target skill(s) (e.g., sums
to 5, double digit multiplication with
regrouping). - Timing 2 minutes
- Information obtained digits correct in two
minutes.
26Math Probe Example
- Total Digits 38
- Errors 5
- Digits Correct 33
27Tier 1 Writing Screening
- Writing Probe
- Group administered.
- Materials story starter (e.g., If I had a
million dollars) printed at the top of a blank
page. - Timing 1 minute to think, 3 minutes to write.
- Scoring words written or correct word sequences
in three minutes.
28Writing Example
29Tier 1 Reading Screening
- Reading Probe
- Individually administered
- Materials A content-controlled reading passage.
- Procedure The student reads aloud as the teacher
listens and records errors. - Timing 1 minute
- Information obtained words read correctly in one
minute.
30CBM Reading Sample Scoring
31Class-wide Screening
32Feedback to Teachers
33Tier 2 Class-wide Intervention
34No Class-wide Problem Detected
35Tier 2 Cant Do/Wont Do Assessment
3-7 minutes per child
- Cant Do/Wont Do
- Individually-administered
- Materials
- Academic material that student performed poorly
during class assessment. - Treasure chest plastic box filled with tangible
items.
36Cant Do/Wont Do Assessment
37Decision Rule Following Cant Do/Wont Do
Assessment
38Tier 3 Individual Intervention
39Response to Intervention
Before Intervention
During Intervention
Correct
Avg. for his Class
Each Dot is one Day of Intervention
Intervention Sessions
Intervention in Reading
40Response to Intervention
Before Intervention
During Intervention
Correct
Avg. for his Class
41Vehicle for System ChangeSystem-wide Math
Problem
Instructional range
Frustrational range
Each bar is a students performance
42Re-screening Indicates No Systemic Problem
Fourth Grade
43Rest of Grade at Standard
A
B
C
D
E
F
Classroom
44Spring 2003 Classroom F
F
45Teacher moved to lower grade in Fall 2003
46Class-wide Intervention
Teacher F Mult 0-12
120
100
80
Digits Correct Two Minutes
60
40
20
0
11/7/2003
10/24/2003
10/31/2003
11/14/2003
11/18/2003
Weeks
47Increased Difficulty- Intervention Continues
48Mixed Mult/Div/Fractions Probe Classroom F
49Growth Obtained
actual growth
aimline
50Effect on High-Stakes Scores
VanDerHeyden, in prep
51Effect on High-Stakes Scores
VanDerHeyden, in prep
52District-wide Implementation Data
- Vail Unified School District
- www.vail.k12.az.us
- Three years, system-wide implementation of STEEP
grades 1-8
53System Outcomes
- Referrals reduced greater than half
- who qualify from 50 stable baseline over three
years to nearly 100 - SLD down from 6 of children in district in
2001-2002 (with baseline upward trend) to 3.5 in
2003-2004 school year - Corresponding gains on high-stakes tests
(VanDerHeyden Burns, 2005) - Intervention successful for about 95 to 98 of
children screened
VanDerHeyden, Witt, Gilbertson, 2007
54Cost Reduction
VanDerHeyden, Witt, Gilbertson, 2007
55Findings
- Number of Evaluations dramatically reduced 70
at highest referral school - Diverse settings, psychologists of diverse
backgrounds and no prior experience with CBM or
functional academic assessment - Percentage qualify increased at 4 of 5 schools
- Disproportionate representation of males
positively affected - Number of children placed dramatically reduced
VanDerHeyden, Witt, Gilbertson, 2007
56Team Decision-Making Agreement
RTI and Evaluated RTI- and Did Not Evaluate
2003-2004 (3 schools) 100 41
2004-2005 (5 schools) 100 87
VanDerHeyden, Witt, Gilbertson, 2007
57Team Decision-Making
VanDerHeyden, Witt, Gilbertson, 2007
58Fall to Spring Reading Growth
VanDerHeyden Witt, 2005
59What Proportion of Ethnicity Represented Before
and After Intervention in Risk Category?
VanDerHeyden Witt, 2005
60Criterion
Criterion -
13 9
4 75
PVS
PVS -
.76/.116.9 children with learning problem about
7 times more likely to have a failed RTI than
children without a learning problem .24/.89.27
children without a learning problem are 3.7 times
more likely to have a successful RTI than those
children with a learning problem
Sens- .76 Spec- .89 PPP- .59 NPP- .95
LR- sens/(1-spec) - LR- (1-sens)/spec
61Criterion
Criterion -
16 39
1 45
Probe
Probe -
.94/.462.0 children with learning problem about
2 times more likely to perform below screening
benchmark than children without a learning
problem .06/.54.11 children without a learning
problem are 9 times more likely to have a above
criterion screening score than those children
with a learning problem
Sens- .94 Spec- .54 PPP- .29 NPP- .98
62Identification Accuracy
- High-achieving classrooms (lt20)
- Procedures paired with RTI criterion were more
accurate than other commonly used screening
devices - Low-achieving classrooms (gt50)
- Procedures paired with RTI criterion were more
accurate than other commonly used screening
devices
VanDerHeyden Witt, 2005
63Questions
- Is there a classwide problem?
- Is there a gradewide problem?
- Whats the most efficient way to deliver
intervention?
64Screening tells you
- How is the core instruction working?
- What problems might exist that could be
addressed? - Most bang-for-the-buck activity
- Next most high-yield activity is classwide
intervention at Tier 2.
65Screening Guidelines
- Efforts at Tier 1 pay off with fewer children
needing individual intervention - 3 times per year, single probe
- Use small team of trained coaches
- Prepare all needed materials in a packet for each
teacher - Score and return within 1 week on graph
- Use data to generate aimlines, can be used to set
benchmarks
66Here is 3rd grade at Cottonwood. Each circle
corresponds to a students score on a reading CBM
probe in March and the AIMS reading score the
same month. So the circle near the blue lines is
a child who read 158 wc/min and scored 486 on the
AIMS Reading. The diagonal line represents the
best fit line or the line closest to all the
circles. This shows there is a strong positive
correlation between CBM and AIMS reading scores
with this group.
67How to Set a Benchmark
This is words read correctly per minute. You move
this line up and down to catch as many of those
who will not pass as possible.
431 pass for AIMS. This line does not move
68Setting 95 wc/min as the pass standard for CBM
These are children who were predicted to pass
AIMS based on CBM and did pass. Hits
These are the children predicted to pass AIMS who
actually failed. False Negative Errors. The
worst kind of error.
These are the children predicted to fail AIMS who
actually passed. False Positive Errors.
These are the children who were predicted to fail
AIMS based on CBM who did fail. Hits
69Moving the horizontal line up will catch two
more cases who failed the AIMS, but will result
in many more false positive errors.
Thus, 95 wc/min at 3rd grade at Cottonwood is a
good standard that will tell you which children
are likely to fail the AIMS reading section.
70ROC printout
71Mastery Model Measurement (CBA)
Letter naming fluency
Isolated sound fluency
Beginning sound fluency
Ending sound fluency
72General Outcome Measurement (CBM)
Beginning sound fluency
Letter naming fluency
Isolated sound fluency
Ending sound fluency
Words read correctly per minute
73Tracking Year-Long Growth
mastery
aimline
instructional
74Digits Correct Two Minutes
12
1
Weeks
75Any Curriculum Area
1-5
1-5
5-10
5-10
Students
80-90
80-90
Dave Tilly, 2005
76Weighing a cow doesnt make it fatter.
77Any Curriculum Area
1-5
1-5
5-10
5-10
Students
80-90
80-90
Dave Tilly, 2005
78Class-wide Intervention
- Use pair-peered practice (classwide peer
tutoring, PALS) - Model, Guided Practice, Independent timed
practice with delayed error correction, reward
contingency
79Instructional Hierarchy
Generalization
Fluency
Acquisition
80(No Transcript)
81Reading classwide intervention
82Select a Few Good Interventions to Keep it Simple
Classwide Individual
Math Flash card Practice Cover copy compare Cue Cards Highlighted errors
Reading Listening Preview Repeated Readings Error Correction Key Words
83(No Transcript)
84(No Transcript)
85Intervention Plan- 15 Min per Day
- Protocol-based classwide peer tutoring,
randomized integrity checks by direct observation - Model, Guide Practice, Independent Timed Practice
with delayed error correction - Group performance contingency
- Teachers encouraged to
- Scan papers for high error rates
- Do 5-min re-teach for those with high-error rates
- Provide applied practice using mastery-level
computational skill
86(No Transcript)
87Sample Sequence
88Intervention Plan
- Class Median reaches mastery range for skill,
next skill is introduced - Following promising results at one site in
2002-2003, lead to implementation district-wide
grades 1-8 for all children by 2004-2005.
89Class-wide Math Intervention
90With teacher support
- Consider time, resources, materials
- Remove skill barriers with
- classroom training for students
- classroom coaching for teachers
- Remove implementation barriers after use new
steps - follow-up supportive meetings to problem solve.
- frequent acknowledgment of a teachers efforts
-
91Training Package
Tell Rational Step by step
protocol Show Model Do Train
students Implement with guided practice
Implement independently with support
92-
- 88 of interventions are not used
- without support
- Decisions do not always correspond to data
(someone must check)
93 Tier 2 Intervention Effects
94Class 1 at Screening
95Class 1 Following 10 Days Intervention
96Class 1 Following 15 Days Intervention
97Class 2 at Screening
98Class 2 Following 5 Days Intervention
99Class 2 Following 10 days Intervention
100Class 3 at Screening
101Class 3 Following 5 days Intervention
102Following 10 Days Intervention
103Tier 1 Screening Indicates Class-wide Problem
104Tier 2 Class-wide Intervention
Teacher F Mult 0-12
120
100
80
Digits Correct Two Minutes
60
40
20
0
11/7/2003
10/24/2003
10/31/2003
11/14/2003
11/18/2003
Weeks
105Increased Difficulty- Intervention Continues
106Contextually-Relevant Comparisons and Use of
Trend Data
1075th Grade Math Intervention
108Pre-post changes to performance detected by CBM
Instructional range
Frustrational range
Each bar is a students performance
109Fourth Grade
110Effect on SAT-9 Performance
111Effect on CBM Scores
112Computation Gains Generalized to High Stakes
TestImprovements (Gains within Multiple
Baselineshown as pre-post data)
113Gains within Multiple Baseline (shown as
pre-post data)
114Identification Accuracy
CBA RTI Criterion ITBS WJ-R
STEEP Sensitivity .76 1 .58
STEEP Specificity .89 .99 .77
STEEP Positive Predictive Power .59 .67 .44
STEEP Negative Predictive Power .95 1 .86
Teacher Referral Sensitivity .46 .33 .42
Teacher Referral Specificity .69 .94 .85
Teacher Referral Positive Predictive Power .19 .17 .45
Teacher Referral Negative Predictive Power .89 .97 .83
VanDerHeyden, et al., 2003
115Percent Identified at each Tier
Identified
CBM (Classwide Assessment) 55 (15)
CBM Reward (Performance/skill Deficit Assessment) 40 (11)
CBM Reward Instruction (STEEP ) 22 (6)
Teacher Referral 32 (19)
CIBS-R 64 (18)
DRA 17 (9)
RTI Criterion Assessment 17 (5)
WJ-R 12
ITBS deficit 3 (4)
VanDerHeyden, et al., 2003
116Any Curriculum Area
1-5
1-5
5-10
5-10
Students
80-90
80-90
Dave Tilly, 2005
117Tier 3
- Assessment Data
- Instructional level performance
- Error analysis (high errors, low errors, pattern)
- Effect of incentives, practice, easier task
- Verify intervention effect
- Same implementation support as Tier 2
- Instructional-level materials Criterion-level
materials
118Assessment Data
119Assessment Data
120Tier 3
- Implement for 5-15 consecutive sessions with 100
integrity - Link to referral decision
- Weekly graphs to teacher and weekly
generalization probes outside of classroom,
supply new materials - Troubleshoot implementation weekly
121Lessons Learned
- Most individual interventions for reading
- Standard protocols with slight modifications are
best - Most interventions are successful (should be
successful) - Generalization must be attended to
- Team will not follow data without support and
training to do so - Coordinate intervention start times with
principal and stagger start dates (10-15 at a
time plus Tier 2s). - Organize master schedule for data collection and
Tier times
122Tier 3 Intervention
- gt5 of children screened (total population) IF
solid Tier 1 - Possibly as low as 2 IF solid Tier 1 and Tier 2
- About 1-2 failed RTI 10 of most at-risk
123Successful RTI
1242004- 3rd Grade
125Successful RTI
126Successful Writing
127Successful RTI
128Measurement Error
129Successful Math
130Unsuccessful Math
131Tips for Effective Implementation
132Our Recipe for Intervention Success
- PREPARE
- Identify and Use standard protocols for
intervention - Develop all needed materials
- Develop packets or put on a central web site
- Determine graphing program
133Our Recipe for Intervention Success
- TRAIN
- Explain
- Watch the teacher do it with the actual child
before you leave - Call or meet teacher after first day to problem
solve
134Our Recipe for Intervention Success
- COLLECT DATA AND SUPPORT
- Each week, graph intervention performance and do
a generalization check with the child. - Graphed feedback to teachers with generalization
checks for individual intervention once per week - Response-dependent performance feedback to
sustain implementation accuracy - Monthly CBM to track growth and enhance existing
Tier 1 Programs or advise new Tier 1 - Data to principal weekly. Summarize effects and
integrity of procedures.
135Tracking Title Progress
136Our Recipe for Intervention Success
- USE DATA TO MAKE DECISIONS
- RTI successful if child performs criterion-level
probe (from screening) in the instructional
range. RTI unsuccessful if 15 consecutive
intervention sessions and criterion probe is not
in the instructional range. - Increase task difficulty for intervention if
child scores at mastery on task during
intervention sessions
137Infrastructure for Implementation
- Grade-level planning periods can be utilized
- Special education team at school can be
utilized - School Psych must be on-site 1 day/week
- Developing master schedule for Tier 1, 2, and 3
intervention times is useful - Integrate efforts with evaluation referral team
efforts (consider major reduction in meeting time
and shift to intervention efforts!)
138Materials needed
- Computer and software to organize data
- Student data imported. Clerical person to enter
data on-site for tier 1 screen only. - Color printer to print graphs extra color
cartridges - Probe materials, digital count-down timers
- Intervention protocols, intervention materials
(e.g., flashcard sets, reading materials) - Access to copier and some assistance with copying
- Reinforcers for treasure chest (no more than 500
per school)
139Guidelines for Implementers
- Use single trial scores for screening
- Following screening, grade-wide graphs to
principal - Return data to teachers within 48 hours with
personal interpretation at grade-level team
meeting - Include principal in critical meetings
- Involve teachers at all stages
140Guidelines for Implementers
- Learn about curriculum and instruction.
- Integrate RTI with ongoing school and system
reform efforts - Thoughtfully merge to subtract duplicate
activities and to enhance more comprehensive
supplemental and core instructional support
activities that may be in place - Use RTI data to evaluate the value of ALL
instructional programs or resource allocation
decisions. Quantify bang for the buck using
student performance data.
141Lessons Learned from Vail USD
- Infrastructure for education being a
results-based enterprise - Accountability
- Principal is the Instructional Leader of a school
- Principal as change agent
- School psychologist as change agent
- Replace resources/substitute dont add
- Minimize meetings
- Track outcomes that matter
142Upset parent
Tantruming child
Bus 11 is late
Police on site for child abuse report
Principal
Visit from health department, bathroom out of
paper towels again
Teacher out sick for rest of year
Meeting at district office/items due
143Principal
FILTER-- How much time allocated to instruction?
Children actively engaged? Standards introduced?
Effective instruction occurring?
Upset parent
DATA on Learning
Check on health dept
Goal Setting
Check on police interview
Teacher Evaluation
Etc.
Allocation of Instructional Resources
144Great Instructional Leaders
- Have a filter
- Allocate time and resources according to their
filter - Use an AIMLINE
- Have a framework for making data-driven decisions
(know how to access the data they need to reach
timely decisions) - Hold teachers, staff, students accountable
- Research findings on effective schooling
145Great School Psychologists
- Hand the principal the data the principal did not
know to ask for but cant live without - Follow the aimline and attend to implementation
integrity - Understand the variables of effective instruction
and engage in contextualized assessment that is
technically valid for the purposes needed AND has
treatment utility - Minimizes meeting time and avoids the science of
strange behavior
146Great Districts
- Minimize time away from school, but use time
together to review school improvement
implementation efforts and ongoing results - Have the will to proactively chart the course of
a district - Provide adequate resources and space for
principals to be effective instructional leaders
and hold them accountable for results - Respect the role of parents and actively engage
them - Have a framework for evaluating results (know how
to access data for decision making) - Evaluate quality of all programs locally and make
decisions about continued use based on DATA.
147Great Teachers
- Use data to identify where more/different/less
instruction is needed - Have as a goal to accelerate all learning of all
children - Proactively address barriers to learning
- Take responsibility for learning that occurs in
the classroom - Are confident and ready to collaborate in the
classroom - Appreciate childhood and children (a little
humor, lots of patience, enthusiasm)
148For More Information
- amandavande_at_gmail.com
- www.isteep.com
- Thank you to the US Dept of Education for
providing all film clips shown in this
presentation