Improving Outcomes for ALL Students Through the Flexible Student Services Model FSSM : Data Collecti

1
Improving Outcomes for ALL Students Through the
Flexible Student Services Model (FSSM) Data
Collection, Analysis, and Evaluation and
Introduction to School-wide Assessment Strategies

• Gary L. Cates, Ph.D.
• Mark E. Swerdlik, Ph.D
• Illinois State University

Kirkwood, Mehlville, Special School District,
Webster Groves (KMSW) Cooperative Expect the
Best
2
Opening activity

• Work in teams on identifying strengths of your
  team, challenges for team, and a point of pride
  (one very successful action they are taking as a
  team that they would like to share with their
  colleagues).

3
Steps of Problem-Solving
2. Problem Analysis
1. Problem Identification
3. Plan Development
5. Plan Evaluation
4. Plan Implementation
4
Determine Situations in Which Behavior is Most
Likely and Least Likely to Occur

• Review all RIOT data to find convergent evidence
  about when, with whom, where, and how a student
  may succeed.

5
Data Collection Analysis

• Linking Assessment to Effective Intervention from
  the Beginning

6
Why Collect Data?

• Support hypotheses for why socially inappropriate
  behavior is occurring
• Support hypotheses for why academic and other
  socially acceptable behaviors are not occurring
• Determine current level of target behaviors

7
(No Transcript)
8
Types of Data Collection

• Verbal Reports (e.g. interviews)
• Rating Scales (e.g. BASC)
• Record Review (e.g. Cumulative file, homework
  permanent products)
• Observation Systems (e.g. BOSS)
• Direct Systematic Behavioral Observation (e.g.
• Interval recording, frequency counts)

9
Verbal Reports

• Unreliable
• DO NOT TRUST THEM!
• Get direct data (i.e. independent observation) to
  corroborate

10
Rating Scales

• More reliable than verbal report
• Used only as a screener
• DO NOT USE ALONE FOR INTERVENTION OR DIAGNOSIS!

11
Observations

• This is not an anecdotalreport of what
  someoneobserved for a class period

12

• General guidelines for observations
• Dont be intrusive.
• Agree upon a clearly defined and observable
  behavior first.
• Observe across days/times/settings to increase
  reliability.
• Use with other forms of assessment to increase
  validity.
• Carefully consider the goal of the observation
  before selecting an observation tool.
• Always note the environmental context of the
  behavior.
• Observe students in their natural environments.
• Always observe peers for a comparison.

13
Observation systems

• Save your money
• Very limited
• Use direct behavioral systematic observation
  methods

14
Direct Behavioral Observations

• ABC Logs
• Frequency Tabulation Logs
• Systematic Interval Recording

15
Examples of Direct Observations

• ABC Recording
• Antecedents - what occurs right before the
  behavior.
• Behavior - problem behavior (observable and
  defined)
• Consequences - what happens right after the
  behavior

16
(No Transcript)
17
Practice Analyzing an ABC Log

• See handout
• Why do you think the behavior is occurring?
• What might you do for an intervention?
• What is an acceptable alternative behavior?
• How would you monitor progress?

18
Examples of Direct Observations

• Frequency Count (RATE MEASURE!)
• A measure of how often a clearly defined behavior
  occurs within a given period of time.
• Examine the frequency of the behavior by tallying
  or counting the behavior as it occurs.
• Use this when the behavior is discrete (has an
  obvious beginning and ending) and does not occur
  at very high rates.
• This information is helpful at ALL steps of the
  problem solving process
• ALWAYS MEASURE AS RATE WHEN POSSIBLE!!!!

19
(No Transcript)
20
Practice Using A Frequency Count/Rate Measure Log

• See Handout
• Determine the rate of behavior
• Determine Discrepancy Ratio
• The average child does this on average 1.8 times
  per day.
• Write a hypothesis ICEL
• Develop a method for hypothesis testing RIOT

21
Examples of Direct Observations

• Systematic Data Recording
• Examine percentage of target behavior by
• Recording when the selected student is engaging
  in target behavior during 10-second intervals for
  15 minutes.
• Peers are observed in the same way as a
  comparison.
• Requires more training than the other observation
  tools.
• This information is helpful at all steps of the
  problem solving process

22
Systematic Direct Behavioral Observations
Interval Recording

• Partial Interval Recording Occurs anytime within
  interval
• Whole Interval Recording Occurs majority of
  Interval
• Momentary Time Sampling Within 3 seconds
• Duration Recording How long behavior occurs

23
Target Child
Composite Child
24
Lets Collect Some Data

• ABC Log
• Duration
• Frequency/Rate
• Momentary Time Sampling
• Partial Interval Recording
• Whole Interval Recording

25
What did you come up with?

• Definition of Behavior is important!
• Corroborating Reports is important!
• This Stuff takes some practice!
• You now have baseline data!
• Does it lead to intervention?
• Must analyze the problem after definition and
  data collection!

26
10 Minute Break
27
Data Analysis Part 1

• Behavior Problems

28
What you must keep in mind

• Behavior has a function
• You are trying to identify the function
• You cannot be circular in your logic (e.g., ADHD).

29
4 Functions of BehaviorHypotheses for why
Inappropriate Behavior is Occurring

• Attention Peer, teacher, peer and teacher
• Escape Demands, tasks, environments, other
  pressures
• Tangibles Food, games, items that are
  reinforcing
• Sensory Stimulation Some habits, more common
  among lower functioning students

30
What do you think the function of disruption
Was?
31
Functional Assessment/Analysis

• What are conditions that maintain the behavior?

32
How to determine function

• Descriptive Functional Assessment
• No manipulation of environment (Tier II)
• Experimental Functional Analysis
• Manipulate Environment (Tier III)

33
Descriptive Functional Assessment

• Do R-I-O of RIOT

34
Review (R)

• Review existing information on instruction,
  curriculum, and environment.
• One way to gather existing information about the
  student is to review the cumulative folder for
• Health/Medical records
• Attendance
• Educational history
• Onset duration of problem
• Past interventions

35
Interviews (I)

• Interview multiple people to get multiple
  perspectives and increase the reliability of the
  interview.
• Use with other assessment procedures to increase
  the validity of the interview.
• Keep interviews as short as possible (e.g., 15-20
  minutes).

36
Specific Questions to Teachers Behavior Problems

• What does the behavior look like?
• How often does it occur
• What happens immediately before the behavior?
• What happens immediately after?
• What have you tried so far?
• What behavior would you rather see?

37

• Functional Assessment Interview
• Examine the identified behavior of concern in
  terms of
• when the behavior is most likely and least likely
  to occur.
• antecedents consequences to problem behavior.
• whether it is a skill problem or a performance
  problem.
• potential function(s) of the behavior

38
ABC Logs Revisited

• See Handout ABC-Log
• What is a hypothesized function for the
  behaviors?
• Write a Hypothesis
• How would you test this hypothesis? (RIOT)

39
Experimental Analysis of Behavior The T in RIOT

• Test the 4 hypothesized functions of behavior
• Attention Condition
• Demand Condition
• Tangible Condition
• Alone Condition
• Control Condition No demands, No Tangibles and
  No attention available

40
(No Transcript)
41
Applied Functional Analysis Example
42
(No Transcript)
43
Data Analysis Part 2

• Academic Problems

44
Hypotheses For why Academic Behavior is not
occurring

• Cant do
• Acquisition Consistently accurate, not yet fast.
• Increase through Modeling, Demonstration,
  Prompts
• Fluency Consistently accurate and fast
• Increase with Drill
• Generalization Can do it under one set of
  conditions but not another
• Increase by slowly changing conditions and
  practicing
• Wont do Reinforcement issue

45
Review (R)

• Review existing information on instruction,
  curriculum, and environment.
• One way to gather existing information about the
  student is to review the cumulative folder for
• Health/Medical records
• Attendance
• Educational history
• Onset duration of problem
• Past interventions

46
Interviews (I)

• Instructional Planning Form (IPF)
• TIES See Shapiro
• Information gathered is used to develop
  hypotheses and effective interventions.

47
Specific Questions to Teachers Academic Problems

• How are instructional assignments presented?
• What is expected?
• Where is the student currently?
• How are opportunities for practice presented?
• How is feedback provided?
• What has or has not worked?

48
Instructional Planning Form
Student Name_________________ ___
Teacher Name________________ School Year
____________ Goal ________________________________
__________________________________________________
_
10/03 Adapted from the U of Oregon
49
Instructional Planning Form
Student Name_______________________ Teacher
Name________________ School Year
____________ Goal ________________________________
__________________________________________________
_
50
Team Problem Solving Activity
51
Plan for the Collection of Additional Data
Needed to Support Hypotheses

• Your hypotheses should be supported by at least 2
  convergent sources of RIOT data with at least one
  piece being objective.
• If you develop a hypothesis that you dont have
  enough data to support, plan for the collection
  of additional data you need validate or refute
  the hypothesis.
• Data collection should be planned not random!

52
Team Problem Solving Activity
53
Problem Analysis Complete What is your
Hypothesis?
54
Writing a Hypothesis

• Provide the discrepancy statement
• Add because at the end of the discrepancy
  statement and insert your hypothesis.
• The hypothesis should be specific, observable,
  and measurable.
• Example
• Beth is on-task for 35 of intervals while
  peers are on-task 87 of intervals during a
  20-minute observation during direct instruction
  in Math class, because she is escaping the Math
  work which is above her instructional level.

55
Test (T)

• Tests are used to assess student skill.
• Guidelines
• Tests should be reliable, valid, linked to
  intervention, efficient, and repeatable.
• Obtain training on administration scoring
  procedures to maintain standardization.
• Use testing that is outside the box!
• Examples of tests include
• CBM/CBE
• Functional Analysis
• Placement Tests
• Informal Reading Inventory

56
Functional Analysis of Academic Responding

• A method of testing hypotheses and Interventions

57
(No Transcript)
58
(No Transcript)
59
(No Transcript)
60
(No Transcript)
61
(No Transcript)
62
(No Transcript)
63
Step 3 Plan Development

• Question What is the goal?
• A. Write the goal, a measurable statement of
  expected outcomes.
• Question What is the intervention plan to
  address the goal?
• B. Define logistics (e.g., what
  strategies/procedures will be used, when and how
  often the intervention will occur, who will
  implement the intervention and where it will be
  implemented, and when it will begin).
• Question How will progress be monitored?
• C. Define logistics (e.g., what materials are
  used, when and how often data will be collected,
  where data will be collected, and who is
  responsible).
• D. Decide on decision-making rules for plan
  evaluation.

64
Writing goals

• Given a set of materials and conditions, student
  will perform at a given level by a specific date
• Given a worksheet of 15 2 digit by three digit
  multiplication problems with carrying Mark will
  be able to complete 100 of the problems with
  98 accuracy in 7 minutes within 3 weeks.
• Consider discrepancy ratios as well as
  conditions/criteria

65
Step 4 Plan Implementation

• Question How will implementation integrity be
  ensured?
• Provide support to those implementing
  interventions.
• B. Observe intervention in action.
• C. Make adjustments to intervention plan if
  needed.
• D. Collect and graph data on student
  performance/behavior.

66
Intervention Integrity

• Strategies that improve Integrity of Intervention
  Implementation
• Follow-up by a consultant/support staff
• Presentation of student data illustrating
  response to intervention
• Review of treatment implementation
• Frequency--range from daily to weekly initially
• Teacher responsiveness to implementing
  interventions
• Understands the need for intervention
• Perceives self as possessing skills to implement
  OR has the social support to implement while
  acquiring skills

67
Step 5 Plan Evaluation

• Question Is the intervention plan effective?
• A. Is the student making progress toward the
  goal?
• Is the student decreasing the discrepancy between
  him/her and the general education peers?
• Is the plan able to be maintained in the general
  education setting?

68
Is the student making progress toward the goal?
69
Is the student decreasing the discrepancy
between him/her and the general education peers?
70
Is the plan able to be maintained in the
general education setting ?
71
Case Practice

• Read case study-Hannah and
• 1. identify the problems, prioritize them and
  choose one or two target behaviors including a
  replacement behavior(s).
• 2. Select Baseline Measure for target behavior
• 3. Construct hypotheses and determine how you
  will test them
• 4. Develop an intervention for you confirmed
  hypothesis
• 5. Develop an evaluation plan for the intervention

72
Analyzing Graphical Data
73
Review Components of the Graph

• X axis
• Y axis
• Letters (A-B-C etc.)
• Numbers (1, 2, 3, etc.)

74
What Do the Letters and Numbers Mean?

• A Baseline
• B Intervention
• C Intervention 2 etc.
• 1 first level of Intervention
• 2 second level of Intervention etc.

75
Three Important Criteria

• Trend The slope of the line
• Stability Amount of fluctuation
• Level Performance level

76
Increasing
Trend?
Stability?
Stable
Level?
Above baseline
77
Trend?
Decreasing
Variable
Stability?
Immediate Increase/Subsequent decrease
Level?
78
Determining Trend Lines
79
Practice Interpreting Graphs

• See handout for interpreting graphs

80
Trend lines

• A trend line is a straight line drawn through a
  series of continuous frequency plots on a chart.
  
• A trend line represents the students actual rate
  of progress over time

81
(No Transcript)
82
(No Transcript)
83
Trend Line Construction Rules

• When you have an even number of data points
  choose the datum furthest away from the median
  datum
• This is an estimate Always use stability and
  level in addition to trend lines

84
What Decision Rules Should be Used to Analyze
Trends?

• Decide if the decision rule is based on an
  ascending goal line or if the decision is based
  on a descending goal line
• Decisions are different depending on the type of
  goal line used (ascending or descending)

85
Comparison of Slope Rules

• If slope is flatter than the slope of the goal
  line, make instructional change
• If slope is steeper than the slope of the goal
  line, adjust the long term goal and draw a new
  line
• If the slope of the trend line is the same as the
  slope of the goal line, make no change

86
Practice Constructing Trend Line

• See handout on Trend lines

87
Data Point Rules

• If the student is at or above goal for 3
  consecutive data points, raise the goal
• If the student is below the line, for 3
  consecutive data points make a change
• If neither of the above rules apply, make no
  change

88
Data Point Rules - Non Academic

• If the student is above the goal for 3
  consecutive data points, do nothing (reward)
• If the student is below the goal for 3
  consecutive data points, raise the goal

89
Practice Making Intervention Decisions

• See handout on Intervention Decisions

90
Trend Outcome Decisions

• Student performance in comparison to self
• Student performance in comparison to another
  standard (e.g. Benchmarks, typical peer behavior)
• Change in independence
• Effort and resources required for the measured
  change (team teacher decisions)

91
(No Transcript)
92
How to determine Aimline

• Depends on Goal
• 25th percentile?
• lt 2x discrepant?
• Meeting predicted level of success on state test?

93
(No Transcript)
94
(No Transcript)
95
(No Transcript)
96
(No Transcript)
97
(No Transcript)
98
(No Transcript)
99
(No Transcript)
100
Response To Intervention

• A Focus on Reading

101
Tungsten VS. CBM

• Not sensitive to small changes in improvement
• Tied to state standards
• Is not early prevention oriented

• Focus on early prevention
• Is used nationwide
• Can be sensitive to small changes in performance

102
The Reading Process 5 Big Ideas

• Phonemic Awareness
• Alphabetic Principle
• Fluency
• Vocabulary
• Comprehension

103
DIBELS

• Reading Assessment Tool

104
DIBELS Administration Timeline
Oral Reading Fluency
Retell Fluency
Nonsense Word Fluency
Phoneme Segmentation Fluency
Letter Naming Fluency
Initial Sound Fluency
DIBELS Word Use Fluency
F W S Preschool
F W S Kindergarten
F W S First Grade
F W S Second Grade
105
What is DIBELS Missing From The Big 5?

• Vocabulary
• Comprehension
• Should consider methods for providing progress
  monitoring for such variables
• (e.g. maze, cloze, grade level definitions?)

106
What DIBLES can be used for

• Screening for potential at risk.
• Progress Monitoring
• Measurement of reading vital signs
• Benchmarking
• Understanding idiosyncratic reading behavior that
  may be diagnostic (e.g. doesnt know letter
  sounds, substitutes, omits, commits, pauses,
  attempts to sound out etc.)

107
What DIBELS should not be used for

• Diagnosing LD
• Measuring reading achievement
• Measuring reading ability
• Determining how to teach reading
• Gauging Reading Comprehension

108
TIER IHow Can DIBELS Help?

• Establish benchmarks at various points in time
  for various skills

109
(No Transcript)
110
Tier IIHow can DIBLES Help?

• Determine if small group instruction is helpful
  from fall benchmark to winter benchmark.

111
(No Transcript)
112
Lets Drop in An Intervention

• My Breakfast Reading Program

113
(No Transcript)
114
Tier IIIHow can DIBELS Help?

• Progress monitoring
• Aim Lines
• Goal Setting

115
Lets Continue With the Intervention

• My Breakfast Reading Program X 2

116
(No Transcript)
117
Questions?

• Next we will learn how to graph Benchmarks

118
Excel Demonstration

• Cliffs Notes version
• Used for Teacher Conferences Evaluating
  intervention, etc. (new slide)
• More accurate/reliable than hand graphing

119
Closing Activity