A Multistage Adaptive and Accessible Reading Assessment for Accountability

1
A Multistage Adaptive and Accessible Reading
Assessment for Accountability Cara Cahalan
Laitusis ETS
2
ETS Contributors

• Linda Cook
• Kelly Bruce
• Jennifer Dean
• Dan Eignor
• Lois Frankel
• Gena Gourley
• Eric Hansen

• Branden Hart
• Teresa King
• Skip Livingston
• Pavan Pillarisetti
• Kitty Sheehan
• Elizabeth Stone
• Klaus Zechner

3
DARA Goal 4

• Field test a multi-stage component-based reading
  assessment.
• Reduce number of students performing at chance
  level
• Allow students to show what they know
• Push instructional to include both comprehension
  and reading fluently for students with
  reading-based LD

4
DARA Test Design

5
Accessibility Elements

• Students with disabilities included in pilot test
• Higher interest passages selected based on
  student ratings
• Single column question format (increased white
  space and reduced wrapping of text)
• Included context sentence
• Panel of disability experts reviewed items and
  made suggested revisions (simplified language)

6
(No Transcript)
7
(No Transcript)
8
(No Transcript)
9
(No Transcript)
10
Data Collection Design

11
Primary Research Questions

• For accountability purposes, is it possible to
  combine scores from the two different routes on
  the component test (i.e., average scores from
  Test 1 and Test 2)?
• Is the Component test more accessible than the
  state assessment
• Do RLD students do better on the Component test
  than the state assessment while students without
  disabilities (NLD) perform similarly on both
  assessments?

12
Other Research Questions

• Can we reduce the number of students scoring at
  chance level?
• Can we use automated scoring technology
  (SpeechRater) to score oral reading fluency
  measure?
• Can we accurately route students based on 7, 14,
  21, and 28 items?
• What is the best measure of oral reading fluency?
  
• How do we combine fluency and comprehension test
  scores (50/50, 25/75, 75/25)?

13
Sample

• 8th Grade Students
• 26 Middle Schools
• 294 RLD (final sample275)
• 194 LP (not include in this presentation)
• 500 Non-Disabled (final sample486)

14
Description of Sample by NLD/RLD

• Race, Gender, and cut score impact

Group SEX SEX RACE RACE RACE RACE RACE RACE RACE
Group M F A.I. A B H M P.I. W
NLD 1 46.18 53.82 0.00 2.82 2.42 8.47 2.02 3.63 80.65
NLD 2 46.84 53.16 0.42 2.97 4.24 8.05 2.97 3.81 77.54
RLD 1 63.12 36.88 0.00 1.42 4.96 22.70 6.38 2.84 61.70
RLD 2 62.69 37.31 0.75 0.75 5.22 8.96 5.22 4.48 74.63
15
(No Transcript)
16
(No Transcript)
17
Test Score Summaries Route 1
N Mean (Std Dev) Criterion (48 items) Component Comprehension (42 items) Component Fluency (obs max222.75) Component Total Scaled (max48)
RLD (Route 1) 141 14.92 (3.96) 141 19.14 (6.16) 141 71.44 (32.37) 141 20.61 (5.60)
NLD (Route 1) 249 36.34 (7.95) 249 33.71 (6.12) 249 145.49 (30.18) 249 37.34 (6.27)
18
Test Score Summaries Route 2
N Mean (Std Dev) Criterion (48 items) Component Comprehension (42 items) Component TotalScaled (max48)
RLD (Route 2) 134 27.21 (6.01) 134 22.66 (7.43) 134 25.90 (8.49)
NLD (Route 2) 237 34.29 (8.05) 237 30.65 (7.41) 237 35.03 (8.47)
19
Primary Research Questions

• For accountability purposes, is it possible to
  combine scores from the two different routes on
  the component test (i.e., average scores from
  Test 1 and Test 2)? YES
• Is the Component test more accessible than the
  state assessment
• Do RLD students do better on the Component test
  than the state assessment while students without
  disabilities (NLD) perform similarly on both
  assessments? YES, for Route 1

20
Can we reduce the number of students scoring at
chance level?
21
Routing decision
Can we accurately route students based on 7, 14,
21, and 28 items?
  Best passage (2) Passage 1 Passages 1, 2 Passages 1, 2, 3 Full routing
  (8 items Route 1 lt 3) (8 items Route 1 lt 3) (16 items Route 1 lt 6) (24 items Route 1 lt 10) (32 items Route 1 lt 13)
Reliability 0.63 0.60 0.74 0.77 0.79
(N) students assigned to Route 1 40.73 (112) 41.45 (114) 34.55 (95) 43.64 (120) 51.27 (141)
(N) students assigned to Route 1 on this test and full routing test 70.21 (99) 72.34 (102) 65.96 (93) 82.98 (117) 100 (141)
22
Fluency TestHuman vs. Automated Scoring
Can we use automated scoring technology
(SpeechRater) to score oral reading fluency
measure?
ALL Passage 1 Passage 2 Passage 3 Passage 4
N 547 126 151 148 122
Pearson r 0.68 0.81 0.71 0.60 0.76
23
Future Questions for Study and Policy

• Q What is the best measure of oral reading
  fluency?
• Corrected words per minute in 1st minute
• Words per minute, corrected words per minute,
  percent correct, rating
• Q How do we combine comprehension and fluency
  scores
• 25 fluency 75 comprehension
• 50/50, 75/25

24
Contact information

• Cara Cahalan Laitusis
• Senior Research Scientist
• Educational Testing Service
• Mailstop 09R
• Princeton, NJ 08541
• CLaitusis_at_ETS.org

25
Extra Slides
26
Test Score Correlations Route 1
  Criterion Component Comprehension Component Fluency Component Total (Scaled)
Criterion 1.00 0.80 0.55 0.83
Component Comprehension 0.30 1.00 0.46 0.97
Component Fluency 0.27 -0.02 1.00 0.67
Component Total (Scaled) 0.38 0.94 0.31 1.00
NLD
RLD
27
Test Score Correlations Route 2
  Criterion Component Total (Scaled)
Criterion 1.00 0.82
Component Total (Scaled) 0.76 1.00
NLD
RLD