Title: Improving Outcomes for ALL Students Through the Flexible Student Services Model FSSM : Data Collecti
1Improving Outcomes for ALL Students Through the
Flexible Student Services Model (FSSM) Data
Collection, Analysis, and Evaluation and
Introduction to School-wide Assessment Strategies
- Gary L. Cates, Ph.D.
- Mark E. Swerdlik, Ph.D
- Illinois State University
Kirkwood, Mehlville, Special School District,
Webster Groves (KMSW) Cooperative Expect the
Best
2Opening activity
- Work in teams on identifying strengths of your
team, challenges for team, and a point of pride
(one very successful action they are taking as a
team that they would like to share with their
colleagues).
3Steps of Problem-Solving
2. Problem Analysis
1. Problem Identification
3. Plan Development
5. Plan Evaluation
4. Plan Implementation
4 Determine Situations in Which Behavior is Most
Likely and Least Likely to Occur
- Review all RIOT data to find convergent evidence
about when, with whom, where, and how a student
may succeed.
5Data Collection Analysis
- Linking Assessment to Effective Intervention from
the Beginning
6Why Collect Data?
- Support hypotheses for why socially inappropriate
behavior is occurring - Support hypotheses for why academic and other
socially acceptable behaviors are not occurring - Determine current level of target behaviors
7(No Transcript)
8Types of Data Collection
- Verbal Reports (e.g. interviews)
- Rating Scales (e.g. BASC)
- Record Review (e.g. Cumulative file, homework
permanent products) - Observation Systems (e.g. BOSS)
- Direct Systematic Behavioral Observation (e.g.
- Interval recording, frequency counts)
9Verbal Reports
- Unreliable
- DO NOT TRUST THEM!
- Get direct data (i.e. independent observation) to
corroborate
10Rating Scales
- More reliable than verbal report
- Used only as a screener
- DO NOT USE ALONE FOR INTERVENTION OR DIAGNOSIS!
11Observations
- This is not an anecdotalreport of what
someoneobserved for a class period
12- General guidelines for observations
- Dont be intrusive.
- Agree upon a clearly defined and observable
behavior first. - Observe across days/times/settings to increase
reliability. - Use with other forms of assessment to increase
validity. - Carefully consider the goal of the observation
before selecting an observation tool. - Always note the environmental context of the
behavior. - Observe students in their natural environments.
- Always observe peers for a comparison.
13Observation systems
- Save your money
- Very limited
- Use direct behavioral systematic observation
methods
14Direct Behavioral Observations
- ABC Logs
- Frequency Tabulation Logs
- Systematic Interval Recording
15Examples of Direct Observations
- ABC Recording
- Antecedents - what occurs right before the
behavior. - Behavior - problem behavior (observable and
defined) - Consequences - what happens right after the
behavior
16(No Transcript)
17Practice Analyzing an ABC Log
- See handout
- Why do you think the behavior is occurring?
- What might you do for an intervention?
- What is an acceptable alternative behavior?
- How would you monitor progress?
18Examples of Direct Observations
- Frequency Count (RATE MEASURE!)
- A measure of how often a clearly defined behavior
occurs within a given period of time. - Examine the frequency of the behavior by tallying
or counting the behavior as it occurs. - Use this when the behavior is discrete (has an
obvious beginning and ending) and does not occur
at very high rates. - This information is helpful at ALL steps of the
problem solving process - ALWAYS MEASURE AS RATE WHEN POSSIBLE!!!!
19(No Transcript)
20Practice Using A Frequency Count/Rate Measure Log
- See Handout
- Determine the rate of behavior
- Determine Discrepancy Ratio
- The average child does this on average 1.8 times
per day. - Write a hypothesis ICEL
- Develop a method for hypothesis testing RIOT
21Examples of Direct Observations
- Systematic Data Recording
- Examine percentage of target behavior by
- Recording when the selected student is engaging
in target behavior during 10-second intervals for
15 minutes. - Peers are observed in the same way as a
comparison. - Requires more training than the other observation
tools. - This information is helpful at all steps of the
problem solving process
22Systematic Direct Behavioral Observations
Interval Recording
- Partial Interval Recording Occurs anytime within
interval - Whole Interval Recording Occurs majority of
Interval - Momentary Time Sampling Within 3 seconds
- Duration Recording How long behavior occurs
23Target Child
Composite Child
24Lets Collect Some Data
- ABC Log
- Duration
- Frequency/Rate
- Momentary Time Sampling
- Partial Interval Recording
- Whole Interval Recording
25What did you come up with?
- Definition of Behavior is important!
- Corroborating Reports is important!
- This Stuff takes some practice!
- You now have baseline data!
- Does it lead to intervention?
- Must analyze the problem after definition and
data collection!
2610 Minute Break
27Data Analysis Part 1
28What you must keep in mind
- Behavior has a function
- You are trying to identify the function
- You cannot be circular in your logic (e.g., ADHD).
294 Functions of BehaviorHypotheses for why
Inappropriate Behavior is Occurring
- Attention Peer, teacher, peer and teacher
- Escape Demands, tasks, environments, other
pressures - Tangibles Food, games, items that are
reinforcing - Sensory Stimulation Some habits, more common
among lower functioning students
30What do you think the function of disruption
Was?
31Functional Assessment/Analysis
- What are conditions that maintain the behavior?
32How to determine function
- Descriptive Functional Assessment
- No manipulation of environment (Tier II)
- Experimental Functional Analysis
- Manipulate Environment (Tier III)
33Descriptive Functional Assessment
34Review (R)
- Review existing information on instruction,
curriculum, and environment. - One way to gather existing information about the
student is to review the cumulative folder for - Health/Medical records
- Attendance
- Educational history
- Onset duration of problem
- Past interventions
35Interviews (I)
- Interview multiple people to get multiple
perspectives and increase the reliability of the
interview. - Use with other assessment procedures to increase
the validity of the interview. - Keep interviews as short as possible (e.g., 15-20
minutes).
36Specific Questions to Teachers Behavior Problems
- What does the behavior look like?
- How often does it occur
- What happens immediately before the behavior?
- What happens immediately after?
- What have you tried so far?
- What behavior would you rather see?
37- Functional Assessment Interview
- Examine the identified behavior of concern in
terms of - when the behavior is most likely and least likely
to occur. - antecedents consequences to problem behavior.
- whether it is a skill problem or a performance
problem. - potential function(s) of the behavior
38ABC Logs Revisited
- See Handout ABC-Log
- What is a hypothesized function for the
behaviors? - Write a Hypothesis
- How would you test this hypothesis? (RIOT)
39Experimental Analysis of Behavior The T in RIOT
- Test the 4 hypothesized functions of behavior
- Attention Condition
- Demand Condition
- Tangible Condition
- Alone Condition
- Control Condition No demands, No Tangibles and
No attention available
40(No Transcript)
41Applied Functional Analysis Example
42(No Transcript)
43Data Analysis Part 2
44Hypotheses For why Academic Behavior is not
occurring
- Cant do
- Acquisition Consistently accurate, not yet fast.
- Increase through Modeling, Demonstration,
Prompts - Fluency Consistently accurate and fast
- Increase with Drill
- Generalization Can do it under one set of
conditions but not another - Increase by slowly changing conditions and
practicing - Wont do Reinforcement issue
45Review (R)
- Review existing information on instruction,
curriculum, and environment. - One way to gather existing information about the
student is to review the cumulative folder for - Health/Medical records
- Attendance
- Educational history
- Onset duration of problem
- Past interventions
46Interviews (I)
- Instructional Planning Form (IPF)
- TIES See Shapiro
- Information gathered is used to develop
hypotheses and effective interventions.
47Specific Questions to Teachers Academic Problems
- How are instructional assignments presented?
- What is expected?
- Where is the student currently?
- How are opportunities for practice presented?
- How is feedback provided?
- What has or has not worked?
48Instructional Planning Form
Student Name_________________ ___
Teacher Name________________ School Year
____________ Goal ________________________________
__________________________________________________
_
10/03 Adapted from the U of Oregon
49Instructional Planning Form
Student Name_______________________ Teacher
Name________________ School Year
____________ Goal ________________________________
__________________________________________________
_
50Team Problem Solving Activity
51 Plan for the Collection of Additional Data
Needed to Support Hypotheses
- Your hypotheses should be supported by at least 2
convergent sources of RIOT data with at least one
piece being objective. - If you develop a hypothesis that you dont have
enough data to support, plan for the collection
of additional data you need validate or refute
the hypothesis. - Data collection should be planned not random!
52Team Problem Solving Activity
53Problem Analysis Complete What is your
Hypothesis?
54Writing a Hypothesis
- Provide the discrepancy statement
- Add because at the end of the discrepancy
statement and insert your hypothesis. - The hypothesis should be specific, observable,
and measurable. - Example
- Beth is on-task for 35 of intervals while
peers are on-task 87 of intervals during a
20-minute observation during direct instruction
in Math class, because she is escaping the Math
work which is above her instructional level.
55Test (T)
- Tests are used to assess student skill.
- Guidelines
- Tests should be reliable, valid, linked to
intervention, efficient, and repeatable. - Obtain training on administration scoring
procedures to maintain standardization. - Use testing that is outside the box!
- Examples of tests include
- CBM/CBE
- Functional Analysis
- Placement Tests
- Informal Reading Inventory
56Functional Analysis of Academic Responding
- A method of testing hypotheses and Interventions
57(No Transcript)
58(No Transcript)
59(No Transcript)
60(No Transcript)
61(No Transcript)
62(No Transcript)
63Step 3 Plan Development
- Question What is the goal?
- A. Write the goal, a measurable statement of
expected outcomes. - Question What is the intervention plan to
address the goal? - B. Define logistics (e.g., what
strategies/procedures will be used, when and how
often the intervention will occur, who will
implement the intervention and where it will be
implemented, and when it will begin). - Question How will progress be monitored?
- C. Define logistics (e.g., what materials are
used, when and how often data will be collected,
where data will be collected, and who is
responsible). - D. Decide on decision-making rules for plan
evaluation.
64Writing goals
- Given a set of materials and conditions, student
will perform at a given level by a specific date - Given a worksheet of 15 2 digit by three digit
multiplication problems with carrying Mark will
be able to complete 100 of the problems with
98 accuracy in 7 minutes within 3 weeks. - Consider discrepancy ratios as well as
conditions/criteria
65Step 4 Plan Implementation
- Question How will implementation integrity be
ensured? - Provide support to those implementing
interventions. -
- B. Observe intervention in action.
- C. Make adjustments to intervention plan if
needed. - D. Collect and graph data on student
performance/behavior. -
-
-
66Intervention Integrity
- Strategies that improve Integrity of Intervention
Implementation - Follow-up by a consultant/support staff
- Presentation of student data illustrating
response to intervention - Review of treatment implementation
- Frequency--range from daily to weekly initially
-
- Teacher responsiveness to implementing
interventions - Understands the need for intervention
- Perceives self as possessing skills to implement
OR has the social support to implement while
acquiring skills
67Step 5 Plan Evaluation
- Question Is the intervention plan effective?
- A. Is the student making progress toward the
goal? - Is the student decreasing the discrepancy between
him/her and the general education peers? - Is the plan able to be maintained in the general
education setting?
68 Is the student making progress toward the goal?
69 Is the student decreasing the discrepancy
between him/her and the general education peers?
70 Is the plan able to be maintained in the
general education setting ?
71Case Practice
- Read case study-Hannah and
- 1. identify the problems, prioritize them and
choose one or two target behaviors including a
replacement behavior(s). - 2. Select Baseline Measure for target behavior
- 3. Construct hypotheses and determine how you
will test them - 4. Develop an intervention for you confirmed
hypothesis - 5. Develop an evaluation plan for the intervention
72Analyzing Graphical Data
73Review Components of the Graph
- X axis
- Y axis
- Letters (A-B-C etc.)
- Numbers (1, 2, 3, etc.)
74What Do the Letters and Numbers Mean?
- A Baseline
- B Intervention
- C Intervention 2 etc.
- 1 first level of Intervention
- 2 second level of Intervention etc.
75Three Important Criteria
- Trend The slope of the line
- Stability Amount of fluctuation
- Level Performance level
76Increasing
Trend?
Stability?
Stable
Level?
Above baseline
77Trend?
Decreasing
Variable
Stability?
Immediate Increase/Subsequent decrease
Level?
78Determining Trend Lines
79Practice Interpreting Graphs
- See handout for interpreting graphs
80Trend lines
- A trend line is a straight line drawn through a
series of continuous frequency plots on a chart.
- A trend line represents the students actual rate
of progress over time
81(No Transcript)
82(No Transcript)
83Trend Line Construction Rules
- When you have an even number of data points
choose the datum furthest away from the median
datum - This is an estimate Always use stability and
level in addition to trend lines
84What Decision Rules Should be Used to Analyze
Trends?
- Decide if the decision rule is based on an
ascending goal line or if the decision is based
on a descending goal line - Decisions are different depending on the type of
goal line used (ascending or descending)
85Comparison of Slope Rules
- If slope is flatter than the slope of the goal
line, make instructional change - If slope is steeper than the slope of the goal
line, adjust the long term goal and draw a new
line - If the slope of the trend line is the same as the
slope of the goal line, make no change
86Practice Constructing Trend Line
- See handout on Trend lines
87Data Point Rules
- If the student is at or above goal for 3
consecutive data points, raise the goal - If the student is below the line, for 3
consecutive data points make a change - If neither of the above rules apply, make no
change
88Data Point Rules - Non Academic
- If the student is above the goal for 3
consecutive data points, do nothing (reward) - If the student is below the goal for 3
consecutive data points, raise the goal
89Practice Making Intervention Decisions
- See handout on Intervention Decisions
90Trend Outcome Decisions
- Student performance in comparison to self
- Student performance in comparison to another
standard (e.g. Benchmarks, typical peer behavior) - Change in independence
- Effort and resources required for the measured
change (team teacher decisions)
91(No Transcript)
92How to determine Aimline
- Depends on Goal
- 25th percentile?
- lt 2x discrepant?
- Meeting predicted level of success on state test?
93(No Transcript)
94(No Transcript)
95(No Transcript)
96(No Transcript)
97(No Transcript)
98(No Transcript)
99(No Transcript)
100Response To Intervention
101Tungsten VS. CBM
- Not sensitive to small changes in improvement
- Tied to state standards
- Is not early prevention oriented
- Focus on early prevention
- Is used nationwide
- Can be sensitive to small changes in performance
102The Reading Process 5 Big Ideas
- Phonemic Awareness
- Alphabetic Principle
- Fluency
- Vocabulary
- Comprehension
103DIBELS
104DIBELS Administration Timeline
Oral Reading Fluency
Retell Fluency
Nonsense Word Fluency
Phoneme Segmentation Fluency
Letter Naming Fluency
Initial Sound Fluency
DIBELS Word Use Fluency
F W S Preschool
F W S Kindergarten
F W S First Grade
F W S Second Grade
105What is DIBELS Missing From The Big 5?
- Vocabulary
- Comprehension
- Should consider methods for providing progress
monitoring for such variables - (e.g. maze, cloze, grade level definitions?)
106What DIBLES can be used for
- Screening for potential at risk.
- Progress Monitoring
- Measurement of reading vital signs
- Benchmarking
- Understanding idiosyncratic reading behavior that
may be diagnostic (e.g. doesnt know letter
sounds, substitutes, omits, commits, pauses,
attempts to sound out etc.)
107What DIBELS should not be used for
- Diagnosing LD
- Measuring reading achievement
- Measuring reading ability
- Determining how to teach reading
- Gauging Reading Comprehension
108TIER IHow Can DIBELS Help?
- Establish benchmarks at various points in time
for various skills
109(No Transcript)
110Tier IIHow can DIBLES Help?
- Determine if small group instruction is helpful
from fall benchmark to winter benchmark.
111(No Transcript)
112Lets Drop in An Intervention
- My Breakfast Reading Program
113(No Transcript)
114Tier IIIHow can DIBELS Help?
- Progress monitoring
- Aim Lines
- Goal Setting
115Lets Continue With the Intervention
- My Breakfast Reading Program X 2
116(No Transcript)
117Questions?
- Next we will learn how to graph Benchmarks
118Excel Demonstration
- Cliffs Notes version
- Used for Teacher Conferences Evaluating
intervention, etc. (new slide) - More accurate/reliable than hand graphing
119Closing Activity