MiniWorkshop on Developing AccessBased Assessment Items for English Language Learners and Students w

1
Mini-Workshop on Developing Access-Based
Assessment Items for English Language Learners
and Students with Disabilities2006 National
Conference on Large-Scale Assessment, San
Francisco, CA

• Rebecca KoprivaMichigan State University
• Therese Gleason Carr
• Amelia Brailsford
• Elizabeth Jones

South Carolina Department of Education
2
Acknowledgments
Slides taken from Emick, J., Kopriva, R.,
Chen, C., Mislevy, R., and Carr, T. (2006).
Achieving accurate results for diverse learners
Access-enhanced item development. Paper, 2006
American Educational Research Association Annual
Meeting, San Francisco.
3
Achieving Accurate Results for Diverse Learners
(AARDL)

• 2004 Enhanced Assessment Instruments Grant
• Project Partners
• South Carolina Department of Education, lead
  state
• Maryland, Minnesota, and Pennsylvania, partner
  sates
• Center for the Study of Assessment Validity and
  Evaluation (University of Maryland/Michigan
  State)
• University of Oregon, University of Nebraska,
  University of South Carolina
• SERVE and DRC

4
Need for Access-Based Item Development

• New strategies for test development are critical
  to the valid, reliable, and accurate assessment
  of English language learners and students with
  disabilities.
• Having accurate results for all students is
  essential to ensure the accountability of
  educational systems, determine how to best meet
  students educational needs, and track student
  progress over time.

5
What is Access-Based Item Development?

• Minimizes some language challenges while
  providing compensatory avenues to access meaning,
  solve problems, and demonstrate solutions
• Goal is to design parallel forms that provide
  comparable scores for students who access
  assessments in ways that are most appropriate for
  them

6
The Access-Based Item Format

• Produces test items specifically designed to
  increase access for ELLs and students with
  language-based disabilities,
• Uses item templates that focus on specific
  structural and contextual factors that impact
  item access, and
• Retains the comparability of scores across forms
  and the integrity of the target construct.

7
Using Templates to Develop Access-Based Items
8
Using Access-Based Item Templates

• Goal is to keep construct constant across forms
• Defining construct too broadly leads to different
  items/forms measuring different aspects of
  constructs
• Templates allow the identification of central
  construct
• Templates assist in identifying
  construct-irrelevant factors in items

9
Access-Based Item Templates

• Identify both the measurement intent of an item
  and what the item measures that is NOT intended
• When completed, provide information about
  components that specify the conceptual parameters
  of an item

10
Information in Completed Access-Based Item
Templates

• How items fit relative to content standards
• Precise information about the measurement targets
  of each item
• Target-irrelevant knowledge and skills

11
Access-Based Item Templates

• Parallel items should assess the same
  measurement intent but may vary in terms of
  target/construct-irrelevant characteristics.

12
Access-Based Item Templates

• By identifying what must remain constant in
  order for parallel items to measure the same
  construct AND what is construct-irrelevant, it
  becomes clear which aspects of the item may be
  altered in order to increase access.

13
Access-Based Item Template Example
14
Objective

• Targeted content knowledge and skills
• Targeted facets are defined within a specific
  Indicator and are usually the same across more
  than one item (depending on how detailed test
  inferences are expected to be)

15
Item-specific subject matter (ISSM)

• Content knowledge and skills that define only an
  instance of the Objective
• Varies by core and should include an explanation
  of the content as well as the targeted complexity
  of the items skills

16
Item-specific constraints (ISC)

• Additional constraints specified here
• Test-level constraints that would be invariant
  over all items (e.g., not changing the item type,
  or retaining the same numbers over items that
  share the same core)
• Core-specific constraints that may or may not be
  content-related
• Example of a content-related constraint
  requiring knowledge of prerequisite skills (such
  as multiplication or addition) when target is
  asking students to compute an algebraic algorithm
• If prerequisite skills not specified as a
  constraint, then lack of prerequisite skills can
  be compensated for in an item

17
Access-Based Item Template Example

• Grade 3 Mathematics Item
•   Maria is going to spend her allowance on
  stuffed animals. Each stuffed animal costs 5.
  What is the largest number of stuffed animals she
  can buy if she has 28?
• A. 6
• B. 5
• C. 4
• D. 3

18
Completed Template
19
Structural and Contextual Factors in Access-Based
Item Development
20
Contextual Factors

• Culturally broad experiences
• Clear and explicit instructions
• Prior learning expectations

21
Culturally Broad Experiences

• Prior knowledge that assumes mainstream U.S.
  experiences
• Expectations that assume a common U.S. value
  system

22
Structural Factors

• Simple Language Structures
• Vocabulary
• Effective Visuals
• Effective Item Formats
• Text Amount
• Text Support
• Content-Based Resources
• Activities
• Manipulatives
• Impact of Home Language

23
Simple Language Structures

• Use simple sentences
• Use similar paragraph organization
• Use present tense and active voice
• Minimize use of prepositional phrases, dependent
  clauses, and conditionals
• Minimize use of rephrasing
• Avoid idiomatic expressions

24
Vocabulary

• Use familiar, high-frequency language
• Limit use of substitute words/pronouns
• Carefully use multi-meaning words

25
Multi-Meaning Words Examples

• The farm was used to produce produce.
• The dump was so full that it had to refuse more
  refuse.
• I did not object to the object.
• The soldier decided to desert his dessert in the
  desert.
• Since there is no time like the present, he
  thought it was time to present the present.
• A seamstress and a sewer fell down into a sewer
  line.

26
Effective Visuals

• Use relevant visuals
• Use effective format
• Use illustrations to mirror text
• Use illustrations to replace text
• Use first person visuals
• Use visuals to organize events in time
• Use visuals to clarify textual meaning

27
Effective Item Format

• Separate key ideas
• Clearly identify item questions
• Use titles
• Mix symbols and text
• Use examples
• Highlight key words or phrases
• Use boxes or lines

28
Text Amount

• Retain complexity of target while using
  contextual cues as possible

29
Text Support

• Bilingual glossaries or dictionaries
• Monolingual glossaries
• Picture-word dictionaries
• Side-by-side forms (a.k.a. dual language test
  booklet)

30
Content-Based Resources

• Primary source documents
• Prior experience information

31
Activities

• Brief interactive discussion
• Brief collection of data

32
Manipulatives

• Concrete materials
• Computer simulations with drag and click options
  and/or graphic drawing opportunities

33
Impact of Home Language

• Use cognates
• Reduce use of linguistically confusing words
• Use symbols consistently
• Reviews of text by those familiar with the
  culture and language

34
Examples of Access-Based Items
35
Standard Item
At Jefferson Midlands Middle School, the sixth
grade students and their teacher are planning a
field trip to the state capital at the end of the
year. In the morning they will visit the state
legislature, and in the afternoon they will go to
the zoo.   There are 33 students in sixth grade.
Five parents and two teachers will be coming with
the students on the trip. Each of the adults has
a car that can hold four students. One of the
teachers says There are not enough cars to take
all of us! Do you agree with the teacher?
Explain your answer.  
36
Access-Based Item

• 33 students are going on a class trip.
• 5 parents and 2 teachers are going with the
  students.
• Each adult has a car. Each car takes 4
  students.
• A student says

There are not enough cars to take all of us!
Is the student right? (circle one) Yes
No Explain Tactile Support
This symbol is introduced before the test and is
common across all tests this state uses. It means
students need to provide an answer and they can
do so using words, algorithms, pictures, or other
diagrams. This symbol is introduced before the
test and is common across all tests this state
uses. It means that there is a tool that
students can tactilely manipulate to help them
solve the problem.
37
2. Plain language principles have been applied to
the item to reduce the semantic and syntactic
complexity of the item. The sentences are
shorter and straightforward, using present tense
and active voice and reducing the use of
prepositional phrases and dependent clauses. A
visual is used to illustrate the item. Note that
numerals have been used consistently throughout.
The translation between a verbal and symbolic
representation of a number was considered
construct-irrelevant mathematics.
1.Information that is not needed to set the
context for the problem has been eliminated,
reducing the amount of text.

• 33 students are going on a class trip.
• 5 parents and 2 teachers are going with the
  students.
• Each adult has a car. Each car takes 4
  students.
• A student says

4.While both the base and the variation assume
students are familiar with class trips, which may
not be the case in all schools, potential
cultural schooling bias has been reduced in the
variation by having a students statement be the
focus of the question. In some cultures,
children are not used to questioning teacher
judgments and decisions1.
3.The formatting has been arranged to provide
maximum access to the problem requirements. Each
complete piece of information is presented
separately, since, for this item, selecting the
appropriate information from among relevant and
irrelevant pieces of information was not part of
the measurement target. The question is clearly
separated from the rest of the text, and the
two-stage character of the item, answering the
question and explaining the response, is evident.
There are not enough cars to take all of us!
Is the student right? (circle one) Yes
No Explain Tactile Support
5. Students are given options for how they
represent their response.
6. Students are allowed to use counters to help
them represent and solve the problem. The
targeted content knowledge and skills do not
preclude allowing various methods of
representation or solution, as noted in the ISSM.
The manipulatives provide students who are ELLs
a way to represent the text that may help them
understand the problem situation.
38
Standard
Access-Based

• Ms. Kopriva is a band teacher. She had 9
  old recorders. She bought 6 new recorders. Then
  4 of the recorders had to be thrown away. Which
  number phrase can be used to find how many
  recorders were left?
• A. 9 4
• B. 9 6 4
• C. 6 4
• D. 9 - 6 4

I have 9 old shirts.
I buy 6 new shirts.
Then, I give my sister 4 shirts.
How many shirts do I have left? Choose the
correct number phrase.
A. 9 4 B. 9 6 4 C. 6 4 D. 9 - 6 4
39
Standard
Access-Based

• Patty just received a letter in the mail telling
  about a new promotion with stuffed animals. When
  Patty has collected and shown proof of owning 125
  stuffed animals she will receive the new Million
  Dollar Bear free. Patty has 79 animals right now.
  Which of the following equations show how many
  more animals Patty will need to collect to get
  her free Million Dollar Bear?
• A. ? - 125 79
• B. 79 ? 125
• C. 79 - ? 125
• D. 125 79 ?

• A class has 79 stars.
• They need 125 stars.
• How many more stars do they need?
• Choose the correct equation.
• A. ? - 125 79
• B. 79 ? 125
• C. 79 - ? 125
• D. 125 79 ?

40
Access-Based
Standard
41
Standard
Access-Based

• What is the difference?
• A. 517
• B. 527
• C. 627
• D. 911

42
Access-Based
Standard
43
Standard
Access-Based
44
Standard
Access-Based
45
Standard
Access-Based

• Andrea had two paper-clip chains. The first
  chain had a total length of 15 centimeters
  (cm), and the second chain had a total length of
  12 centimeters (cm). What was the difference
  in the lengths of the two chains?
• A. 3 cm
• B. 3 cm
• C. 3 cm
• D. 3 cm

• Lily drew two lines.
• The first line was 15 1/3 cm long.
• The second line was 12 1/8 cm long.
• 15 1/3 cm
• 12 1/8 cm
• What was the difference in the length of
  these two lines?
•        A. 3 1/5 cm
•        B. 3 1/24 cm
•   C. 3 1/11 cm
• D. 3 5/24 cm

46
Standard
Access-Based
47
Workshop Practice Items
48
Practice Item 1 Grade 4 Mathematics

• In a gumball machine there are 100 red, 75 blue,
  50 green, and 125 yellow gumballs. These 350
  gumballs are mixed up. Sam puts money in and one
  gumball comes out. Which color is most likely to
  come out?
• A) Red
• B) Blue
• C) Green
• D) Yellow

SOURCE U.S. Department of Education, Institute
of Education Sciences, National Center for
Education Statistics, National Assessment of
Educational Progress (NAEP), 2005 Mathematics
Assessment.
49
Practice Item 2 Grade 8 Mathematics

• Ms. Thierry and 3 friends ate dinner at a
  restaurant. The bill was 67. In addition, they
  left a 13 tip. Approximately what percent of the
  total bill did they leave as a tip?
• A) 10
• B) 13
• C) 15
• D) 20
• E) 25

SOURCE U.S. Department of Education, Institute
of Education Sciences, National Center for
Education Statistics, National Assessment of
Educational Progress (NAEP), 2005 Mathematics
Assessment.
50
Practice Item 3 Grade 8 Mathematics
The map above gives the distances, in miles,
between various locations in a state park.
Traveling the shortest possible total distance
along the paths shown on the map, from the
visitor center Teresa visits the cave, waterfall,
and monument, but not necessarily in that order,
and then returns to the visitor center. If she
does not retrace her steps along any path and the
total distance that Teresa travels is 14.7
miles, what is the distance between the cave and
the monument? A) 2.2 miles B) 2.5 miles C) 2.7
miles D) 3.0 miles E) 3.2 miles
SOURCE U.S. Department of Education, Institute
of Education Sciences, National Center for
Education Statistics, National Assessment of
Educational Progress (NAEP), 2005 Mathematics
Assessment.
51
Contact Information

• Rebecca Kopriva, rkopriva_at_umd.edu
• Therese Gleason Carr, tcarr_at_sde.state.sc.us
• Amelia Brailsford, abrailsf_at_sde.state.sc.us
• Elizabeth Jones, ejones_at_sde.state.sc.us