How a Theoretical and Data-based Modification Process Can Help Students Eligible for an AA-MAS The Consortium for Alternate Assessment Validity and Experimental Studies (CAAVES Project)

1
How a Theoretical and Data-based Modification
Process Can Help Students Eligible for an
AA-MASThe Consortium for Alternate Assessment
Validity and Experimental Studies (CAAVES
Project)

• Presented by Ryan J. Kettler on April 15, 2009 at
  the annual meeting for the
• National Center for Measurement in Education San
  Diego, CA
• The CAAVES project is funded by the
• US Department of Education, Office of Elementary
  and Secondary Education

2
CAAVES Project Research Questions

• We sought to answer the following questions about
  
• item modifications
• Will modifications in testing conditions change
  the reliability of measurement?
• Will taking the test under modified conditions
  change the comparability of scores for eligible
  students?
• What specific modifications are the most helpful
  for making the scores of eligible students
  comparable to the scores of ineligible students?

2
3
Multi-State Sample
Students without Disabilities, Ineligible Students with Disabilities, Ineligible Students with Disabilities, Eligible
Arizona 28 25 21
Hawaii 20 16 18
Idaho 62 43 59
Indiana 159 152 152
4
Typical Item in Original Condition
5
Reading Item in Original Condition
6
Item Modification Strategies

• Universal Design Principles
• Research on item answer choices
• Cognitive Load Theory
• Cognitive lab results

7
CAAVES Item Modification Strategies

• For all items
• Removed least effective distractor
• Increased white space
• For many items
• Bolded key vocabulary terms
• Simplified language in the directions, stimulus,
  stem, and answer choices
• Reorganization of layout
• Added graphic support

8
Reading Item in Modified Condition
9
Mathematics Item in Modified Condition
10
Reliability and Concurrent Validity with a
Proficiency Test
11
Mean Item Difficulty by Group and Condition in
Reading
Student abilities were equated using a Rasch
model.
12
Mean Item Difficulty by Group and Condition in
Math
Student abilities were equated using a Rasch
model.
13
Upcoming Analyses Difficulty Change on
Individual Items
Students with Disabilities, Eligible
Students without Disabilities, Ineligible
14
Individual Modification Analysis

• Reviewed items based on differential boost
• Nine well-modified items were identified
• Six poorly-modified items were identified
• Patterns were revealed for two modifications
• One modification was used in 7 of 9 well-modified
  items, and only 1 of 6 poorly-modified items
• Another modification was used on all 3
  poorly-modified reading items, and 0 of 4
  well-modified items

15
Thank you!

• Any questions? Please email
• r.j.kettler_at_vanderbilt.edu