Title: Assessment and Accountability Issues for English Language Learners and Students With Disabilities
1Assessment and Accountability Issues for English
Language Learners and Students With Disabilities
- Oregon Department of Education
- October 4, 2007
- Jamal Abedi
- University of California, Davis
- National Center for Research on Evaluation,
Standards and Student Testing (UCLA/CRESST)
2Accountability Questions
- Are there specific ELL/SWD subgroup features that
affect the accountability system? - Yes No
- Could the current accountability system for
ELLs/SWDs be improved? - Yes No
- Do research findings help inform assessment
accountability systems for these students? - Yes No
3Should Schools Test Children with Disabilities?
? Yes
Students with disabilities (SWD) can be placed at
a disadvantage because
- Assessment outcomes may not be valid because
their disabilities interfere with content
knowledge performance - Test results affect decisions regarding promotion
or graduation - They may be inappropriately placed into special
educational programs where they receive
inappropriate instruction - SWD students may not have received the same
curriculum which is assumed for the test
4- Should Schools Test English Language Learners?
? Yes
English language learners (ELLs) can be placed at
a disadvantage because
- Assessment outcomes may not be valid because
their low level of English proficiency
interferes with content knowledge
performance - Test results affect decisions regarding promotion
or graduation - They may be inappropriately placed into special
educational programs where they receive
inappropriate instruction - ELL students may not have received the same
curriculum which is assumed for the test
5- Should Schools Test English Language Learners and
Students with Disabilities?
? No
- Problems
- Due to the powerful impact of assessment on
instruction, ELL and SWD students quality of
instruction may be affected - If excluded, they will be dropped out of the
accountability picture - Institutions will not be held responsible for
their performance in school - They will not be included in state or federal
policy decision - Their academic progress, skills, and needs may
not be appropriately assessed
6Problems in AYP Reporting for ELL Students
- Problems in classification/ reclassification of
ELL students (moving target subgroup) - Measurement quality
- Low baseline
- Instability of the ELL subgroup
- Sparse ELL population
- ELL cutoff points (Conjunctive vs. Compensatory
model)
7Percent of ELL student in 2000-2001 (Kindler,
2002)
- California 1,511,646 25.0
- New Mexico 63,755 19.9
- Arizona 135,248 15.4
- Texas 570,022 14.0
- Nevada 40,131 11.8
- Florida 254,517 10.7
- Utah 44,030 9.3
- Oregon 47,382 8.7
8Composition of SWD PopulationNational Statistics
- Grade 8
- 10 Nationally
- Ranging between 6 to 14 by different states
- Grade 4
- 11 Nationally
- Ranging between 4 to 17 by different states
9Some recent statistics
- In 2005-2006 there were a total of 559,215
students were enrolled in the K-12 public schools
in Oregon - Of the total, 11.7 (65,239) were ELLs (as
compared with 8.7 in 2000-2001) and 12.8
(71,517) were Special Education students. - Of the 65,239 ELL students, (76.9) had Spanish
as their language of origin which is about 9 of
the total students enrolled
10Population Change
11Special Education
12Language Diversity
13A Point for Discussion
- This large majority of ELL students speaking the
same language makes Oregon one of the most
eligible state to use native language testing (in
Spanish). - However, the decision is not that simple. Why
not, what are the issues? - Alignment of language of assessment and language
of instruction - Technical issues in translation or
trans-adaptation - Comparability Issues
14Students with Disabilities and Statewide Testing
in Oregon
- Students with Disabilities being served with
active Individual Education Plans (IEP) or 504
plans have a set of choices for participation in
Oregons Statewide Assessment System. - Implementing accommodations and/or modifications
- Out of level testing (no longer an option in
Oregon) - Alternative Assessments for those with profound
cognitive disabilities
15Alternative Assessments(Oregon)
- The Alternate Assessment is comprised of tasks
designed to measure basic skills which are
anchored to the Oregon standards. - The Alternate Assessment is administered for the
same age groups (in the Spring). - Tasks can be administered in a variety of ways
and students can respond with considerable
latitude (e.g., pointing, sign language).
16Challenges in Statewide Testing of ELLs
Students with Disabilities
- Providing effective and valid accommodations
- Developing reliable and valid alternate
assessment - Most importantly, comparability issues
17Why Should English Language Learners be
Accommodated?
- Their possible English language limitations may
interfere with their content knowledge
performance. - Assessment tools may be culturally and
linguistically biased for these students. - Linguistic complexity of the assessment tools may
be a source of measurement error. - Language factors may be a source of
construct-irrelevant variance.
18Why Should Students with Disabilities be
Accommodated?
- Their disabilities put them at disadvantage
- Accommodations must be provided to level the
playing field
19Performance Difference Between SWD and Non-SWD
20Site 4 Grade 8 Descriptive Statistics for the
SAT 9 Test Scores by Strands
21 Reading Science
Math M SD M SD M SDGrade
10 SWD only 16.4 12.7 25.5 13.3 22.5 11.7
ELL only 24.0 16.4 32.9 15.3 36.8 16.0 ELL
SWD 16.3 11.2 24.8 9.3 23.6 9.8 Non-ELL/SWD
38.0 16.0 42.6 17.2 39.6 16.9 All
students 36.0 16.9 41.3 17.5 38.5 17.0 Grade
11 SWD Only 14.9 13.2 21.5 12.3 24.3 13.2
ELL Only 22.5 16.1 28.4 14.4 45.5 18.2 ELL
SWD 15.5 12.7 26.1 20.1 25.1 13.0 Non-ELL/SWD
38.4 18.3 39.6 18.8 45.2 21.1 All
Students 36.2 19.0 38.2 18.9 44.0 21.2
Normal Curve Equivalent Means Standard
Deviations for Students in Grades 10 and 11, Site
3 School District
22Stanford 9 Sub-scale Reliabilities (Alpha), Grade
9
23Grade 11 Stanford 9 Reading and Science
Structural Modeling Results, Site 3
24Fundamental Questions
- Why should ELLs and SWDs be accommodated? (to
level the playing field) - What are the major characteristics of
accommodations that would level the playing
field? - Do the most commonly used accommodations possess
those characteristics
25The major characteristics of accommodations that
would level the playing field
- Effective
- Valid
- Consistent with students background
- Feasible
- Relevant
- From now on our focus will be on ELL students
26Problems with Accommodation Usage
- Accommodations for English language learners are
often selected based on feedback from teachers
and bilingual coordinators without enough
influence from research findings - Several studies have identified some
accommodations that may provide unfair advantage
to the recipients of the accommodations and may
render invalid results
27Examples of Accommodations for ELL Students that
May Alter the Construct
- Providing an English dictionary (Abedi, Courtney,
Leon, 2003 Abedi, Lord, Boscardin, Miyoshi,
2000) - Providing extra or extended time (Abedi, Lord,
Hofstetter, Baker, 2000 Hafner, 2001 Thurlow,
2001) - Translating tests into students native language
(Abedi, Lord, Hofstetter, Baker, 2000) - By gaining access to definition of
content-related terms, recipients of a dictionary
may be advantaged over those who did not have
access to the dictionaries. This may compromise
the validity of assessment (Abedi, Courtney,
Mirocha, Leon, Goldberg, 2005)
28Native language testing
- Problems in creating parallel forms of the test
- Translation issue
- Alignment of language of assessment and language
of instruction
29Glossary Plus Extra Time
Glossary with extra time raised the performance
of both ELL and non-ELL students (Abedi,
Hofstetter, Lord, and Baker, 1998, 2000)
- ELL students performance increased by 13 when
they were tested under glossary with extra time
accommodation. - While this looks promising, it does not present
the entire picture. - Non-ELL students also benefited from this
accommodation, with an increase of 16. - English and bilingual dictionaries recipients may
be advantaged over those without access to
dictionaries. This may jeopardize the validity of
assessment.
30How can accommodations be examined for validity?
Only through experimentally-controlled research
where
- ELL and non-ELL students are randomly assigned to
experimental and control groups - Both ELL and non-ELL students are observed under
accommodated and non-accommodated assessments - Using existing data?
31How the validity of accommodations can be tested
in an experimentally controlled condition?
32Characteristics of ELL students
- ELL students constitute a very diverse and
heterogeneous population (SES, cultural and
linguistic backgrounds). - They can be vastly different in their level of
proficiency in their native or home language - They are quite different in their level of
proficiency in English - Studies show that the level of English language
proficiency of these students range from high
(even higher than some native English speakers)
to very low
33The Most Commonly Used Accommodations for ELL
Students Include(Rivera, 2003)
- Extended time (42 of the 48 states)
- Use of glossary (26 states)
- Use of an English dictionary (33 states)
- Use of a bilingual dictionary (22 states)
- Linguistically-simplified test items (12 states)
34The Most Commonly Used Accommodations for
Students With Disabilities Include (Thurlow, et
al, 2001 Tindal et al, 2000)
- Braille (allowed by 33 of the 48 states studied)
- Computerized assessment (34 states)
- Dictation of responses to a scribe (32 states)
- Extended time (37 states)
- Translation of instructions (40 states)
- Allowance for marking answers in the test
booklets (33 states) - Test items read aloud (34 states)
- Simplified test directions (31 states)
- Test breaks (33 states).
35Do we have enough evidence on the following
characteristics of these most commonly used
accommodation?
- Effective
- Valid
- Consistent with students background
- Feasible
- Relevant
36Accommodations for ELLs and Students with
Disabilities
- Can the same accommodations used for students
with disabilities be used for ELLs? - Can the same accommodations used for ELLs be used
for students with disabilities?
37SY 2000-2001 Accommodations Designated for ELLs
Cited in States Policies
There are 73 accommodations listed N Not
Related R Remotely Related M Moderately
Related H Highly Related
From Rivera (2003) State assessment policies for
English language learners. Presented at the 2003
Large-Scale Assessment Conference
38There are 73 Accommodations Listed
- 7 or 10 are remotely related
- 8 or 11 are moderately related
- 11 or 15 are highly related
39Samples accommodations used for ELL students
- Test-taker marks answers in test booklet
- Copying assistance provided between drafts
- Test-taker indicates answers by pointing or other
method - Paper secured to work area with tape/magnet
- Physical assistance provided
40Samples accommodations used for ELL students
- Enlarged answer sheets provided
- Breaks provided
- Test individually administered
- Test administered in small group
- Test administered in location with minimal
distraction
41Assessment Options for all ELLs in Oregon
- Students may take the test under standard
administration with or without accommodations - Students may take side-by-side English/Spanish,
English/Russian versions where provided. - Eligible students may respond on the Writing
Assessment in Spanish. - Students may take the test under modified
conditions. - Students in grade 3 may take the Aprenda (Spanish
reading test at grade 3). - Students may use the Juried Assessment process
for reading or writing in another language.
42Test Decisions for ELLs(Oregon)
- A teacher and instructional team who know the
student make the decision to test under standard
conditions or modify test - Consult parent/guardian
- Each student must be considered individually for
each assessment - Best interest of student
- Not on participation in a program nor
identification as an ELL - How about research evidence?
43Accommodations TablesChanges in Timing or
Scheduling
44Accommodations TablesChanges in Test Directions
45Accommodations TablesChanges in How the Test
Questions are Presented
46Accommodation TablesChanges in How the Student
Responds
47Accommodation TablesChanges in Test Setting
48How are we doing in practice nationally?
- Are states and districts across the nation
cognizant of this important principle of using
accommodations that are appropriate for ELLs? - Are there any objective national criteria to help
states to select appropriate accommodations for
ELL students? - Or, is the assignment of accommodations to these
students based on temporary and subjective
decisions?
49Comparability Issues
- If accommodated assessment is not valid then the
outcome may not be comparable with the
non-accommodated assessment (a major peer-review
concern) - To report AYP for ELL students, it is imperative
to establish the validity of accommodated
assessment - Construct-irrelevant sources such as linguistic
and cultural biases should be controlled before
reporting AYP based on accommodated assessments
50Are there accommodations that would benefit both
ELLs and SWDs?
- Assessment can be designed in a way to be
accessible to both groups. - For example, long tests, crowded pages, tables
and texts would create frustration and anxiety
for everyone, particularly for ELLs and SWDs. - Complex linguistic structure of assessment would
be a major nuisance variable for ELLs and SWDs
(Tindal, G., Anderson, L., Helwig, R., Miller,
S., Glasgow, A. (2000). Accommodating students
with learning disabilities on math tests using
language simplification. Eugene University of
Oregon, RCTP)
51A Sample Representing a Subgroup of Students with
Disabilities
52A Sample Representing a Subgroup of Students with
Disabilities
53A Clear Language of Instruction and Assessment
Works for ELLs, SWDs, and Everyone
- What is language modification of test items?
54Learning Disability Language of Assessment
- Students in the Learning Disability category may
have difficulty processing complex language in
assessment - Simplifying the language of test items will help
students with disabilities, particularly those
with learning disabilities - As the sample page suggests, a large majority of
students with disabilities are in the Learning
Disability category
55Examining Complex Linguistic Features in
Content-Based Test Items
56Linguistic Modification Concerns
- Familiarity/frequency of non-math vocabulary
unfamiliar or infrequent words changed - census gt video game
- A certain reference file gt Macks company
- Length of nominals long nominals shortened
- last years class vice president gt vice
president - the pattern of puppys weight gain gt the pattern
above
- Question phrases complex question phrases
changed to simple question words - At which of the following times gt When
- which is best approximation of the number gt
approximately how many
57Linguistic Modification cont.
- Voice of verb phrase passive verb forms
changed to active The weights of 3 objects were
compared gt Sandra compared the weights of 3
rabbits - If a marble is taken from the bag gt if you take
a marble from the bag
- Conditional clauses conditionals either
replaced with separate sentences or order of
conditional and main clause changed
If Lee delivers x newspapers gt Lee delivers x
newspapers - If two batteries in the sample were found to be
dead gt he found three broken pencils in the
sample
- Relative clauses relative clauses either
removed or re-cast - A report that contains 64 sheets of paper gt He
needs 64 sheets of paper for each report
58 Harriet, Jim, Roberto, Maria, and Willie are
in the same eighth grade class. One of them is
this years class president. Based on the
following information, who is the class
president?The class president was last years
vice president and lives on Vince
Street.Willie is this years class vice
president.Jim and Maria live on Cypress
Street.Roberto was not last years vice
president. A. JimB. HarrietC. RobertoD.
MariaE. Willie
Original Item
59Modified Item
Harriet, Jim, Roberto, Maria, and Willie ran
for president of their eight-grade class. One of
them won. Who is president? The president now
was vice president last year and lives on
Vince Street.Willie is vice president now.Jim
and Maria live on Cypress Street.Roberto was not
vice president last year. A. JimB. HarrietC.
RobertoD. MariaE. Willie
60OriginalThe census showed that three hundred
fifty-six thousand, ninety-seven people
lived in Middletown. Written as a number,
that isA. 350,697B. 356,097C. 356,907D.
356,970
ModifiedJanet played a video game. Her score
was three hundred fifty-six thousand,
ninety-seven. Written as number, that
is A. 350,697B. 356,097C. 356,907D. 356,970
61Interview Study
- Table 1. Student Perceptions Study First Set
(N19) - Item Original item chosen Revised item chosen
- 1 3 16
- 2 4 15
- 3 10 9
- 4 11 8
- Table 2. Student Perceptions Study Second Set
(N17) - Item Original item chosen Revised item
chosen5 3 14 - 6 4.5a 12.5
- 7 2 15
- 8 2 15
62Many students indicated that the language in the
revised item was easier
- Well, it makes more sense.
- It explains better.
- Because that ones more confusing.
- It seems simpler. You get a clear idea of
what they want you to do.
63The revised items need less time for response
- Its easier to read, and it gets to the point,
so you wont have to waste time. - I might have a faster time completing that one
cause theres less reading. - Less reading then I might be able to get to the
other one in time to finish both of them. - Cause its, like, a little bit less writing.
64Conclusions and Recommendations
- Accommodations
- Must be relevant in addressing assessment issues
for ELL students - Must be effective in reducing the performance gap
between accommodated and non-accommodated
students - Should not alter the construct being measured
- The accommodated results can be aggregated with
the assessments under standard conditions - Must be feasible in national and state assessments
65Conclusions and Recommendations
Examples of research-supported accommodations
- Providing a customized dictionary is a viable
alternative to providing traditional
dictionaries. - The linguistic modification of test items that
reduce unnecessary linguistic burdens on students
is among the accommodations that help ELL
students without affecting the validity of
assessments. - Computer testing with added extra time and
glossary was shown to be a very effective, yet
valid accommodation (Abedi, Courtney, Leon, and
Goldberg, 2003)
66Conclusions and Recommendations
- It is thus imperative to examine different forms
of accommodations before using them in state
and/or national assessments.
- Without information on important aspects of
accommodations such as validity, it would be
extremely difficult to make an informed decision
on what accommodations to use and how to report
the accommodated and non-accommodated results.
67Accountability Questions
- Are there specific ELL/SWD subgroup features that
affect the accountability system? - Yes No
- Could the current accountability system for
ELLs/SWDs be improved? - Yes No
- Do research findings help inform assessment
accountability systems for these students? - Yes No
68For more information, please contact Jamal Abedi
at UC Davis/CRESST
- (530) 754-9150
- or
- jabedi_at_ucdavis.edu