Enhancing State Assessments

1

• Enhancing State Assessments
• Research on the Presentation Effect of Passage
  Based Computer Tests
• Michael RussellTechnology and Assessment Study
  CollaborativeBoston College

2
Reading Comprehension/Text Presentation Study

• Research Questions
• When 4th grade reading comprehension assessments
  are transitioned to computers, what is the impact
  on students test scores?
• Are there differences in test scores when reading
  passages are presented on the computer using
  scrolling text versus whole page text?
• How is student performance (modal effect) related
  to their computer literacy/use or fluidity?

3
Text Presentation Study - Participating Schools
and Design

• 219 Vermont public school students were assessed
• Participating Students
• 37 students from urban schools
• 83 students from suburban schools
• 99 students from rural schools
• Students were randomly assigned to take the
  assessment on paper, on laptop using scrolling
  text, or on laptop using whole page text

4
Demographics of Sample
5
Text Presentation Instruments

• Assessment
• 4 reading passages (2 NAEP, 1 PIRLS, 1 NH)
• 19 multiple choice items
• Computer Fluidity Exercises
• Keyboarding
• Click on moving object
• Drag and drop
• Arrow keys
• Technology Survey
• Computer Literacy
• Eleven released VA and NC technology multiple
  choice items
• Demo

6
Text Presentation Study Analyses

• Group score averages show a six point difference
  across groups

7
Comparison of Scores Among the three Presentation
Modes
8
Item Level differences
9
Computer Fluidity
10
Computer Fluidity
11
Computer Literacy
12
Computer Fluidity
13
Home Use
14
Computer Fluidity
15
Gender
16
IEP Status
17
Race/Ethnicity Status
18
Multilingual Status
19
Open-ended Survey

• 82 easier on computer
• Twenty-six students indicated that they had a
  problem while taking the test on computer.
• not understanding the directions (1), problems
  scrolling (2), having problems using the mouse
  (2), not understanding how to end the test (2),
  and having the computer freeze (2)
• 16.8 reported using the highlighting feature
• 17.4 said that they did mark items for review

20
Reading Comprehension/Text Presentation Study
Analyses

• There were no significant differences in reading
  comprehension scores across testing modes (paper
  58.1, scrolling 52.2, whole page 56.9)
• There were no statistical differences in reading
  comprehension scores based on computer fluidity
  and computer literacy, but a pattern in
  performance may indicate that students are
  disadvantaged by the scrolling text mode,
  particularly those lower computer skills.
• The majority of students who took the reading
  test on a computer indicated that they would
  prefer to take the test on computer.
• Providing highlighters and review markers is
  useful for some students.