Title: How a Theoretical and Data-based Modification Process Can Help Students Eligible for an AA-MAS The Consortium for Alternate Assessment Validity and Experimental Studies (CAAVES Project)
1How a Theoretical and Data-based Modification
Process Can Help Students Eligible for an
AA-MASThe Consortium for Alternate Assessment
Validity and Experimental Studies (CAAVES
Project)
-
- Presented by Ryan J. Kettler on April 15, 2009 at
the annual meeting for the - National Center for Measurement in Education San
Diego, CA - The CAAVES project is funded by the
- US Department of Education, Office of Elementary
and Secondary Education
2CAAVES Project Research Questions
- We sought to answer the following questions about
- item modifications
- Will modifications in testing conditions change
the reliability of measurement? - Will taking the test under modified conditions
change the comparability of scores for eligible
students? - What specific modifications are the most helpful
for making the scores of eligible students
comparable to the scores of ineligible students?
2
3Multi-State Sample
Students without Disabilities, Ineligible Students with Disabilities, Ineligible Students with Disabilities, Eligible
Arizona 28 25 21
Hawaii 20 16 18
Idaho 62 43 59
Indiana 159 152 152
4Typical Item in Original Condition
5Reading Item in Original Condition
6Item Modification Strategies
- Universal Design Principles
- Research on item answer choices
- Cognitive Load Theory
- Cognitive lab results
7CAAVES Item Modification Strategies
- For all items
- Removed least effective distractor
- Increased white space
- For many items
- Bolded key vocabulary terms
- Simplified language in the directions, stimulus,
stem, and answer choices - Reorganization of layout
- Added graphic support
8Reading Item in Modified Condition
9Mathematics Item in Modified Condition
10Reliability and Concurrent Validity with a
Proficiency Test
11Mean Item Difficulty by Group and Condition in
Reading
Student abilities were equated using a Rasch
model.
12Mean Item Difficulty by Group and Condition in
Math
Student abilities were equated using a Rasch
model.
13Upcoming Analyses Difficulty Change on
Individual Items
Students with Disabilities, Eligible
Students without Disabilities, Ineligible
14Individual Modification Analysis
- Reviewed items based on differential boost
- Nine well-modified items were identified
- Six poorly-modified items were identified
- Patterns were revealed for two modifications
- One modification was used in 7 of 9 well-modified
items, and only 1 of 6 poorly-modified items - Another modification was used on all 3
poorly-modified reading items, and 0 of 4
well-modified items
15Thank you!
- Any questions? Please email
- r.j.kettler_at_vanderbilt.edu