Assessment and Accountability Issues for English Language Learners and Students With Disabilities - PowerPoint PPT Presentation

1 / 68
About This Presentation
Title:

Assessment and Accountability Issues for English Language Learners and Students With Disabilities

Description:

instruction ... instruction. ELL students may not have received the same curriculum ... Alignment of language of assessment and language of instruction ... – PowerPoint PPT presentation

Number of Views:361
Avg rating:3.0/5.0
Slides: 69
Provided by: jamal
Category:

less

Transcript and Presenter's Notes

Title: Assessment and Accountability Issues for English Language Learners and Students With Disabilities


1
Assessment and Accountability Issues for English
Language Learners and Students With Disabilities
  • Oregon Department of Education
  • October 4, 2007
  • Jamal Abedi
  • University of California, Davis
  • National Center for Research on Evaluation,
    Standards and Student Testing (UCLA/CRESST)

2
Accountability Questions
  • Are there specific ELL/SWD subgroup features that
    affect the accountability system?
  • Yes No
  • Could the current accountability system for
    ELLs/SWDs be improved?
  • Yes No
  • Do research findings help inform assessment
    accountability systems for these students?
  • Yes No

3
Should Schools Test Children with Disabilities?
? Yes
Students with disabilities (SWD) can be placed at
a disadvantage because
  • Assessment outcomes may not be valid because
    their disabilities interfere with content
    knowledge performance
  • Test results affect decisions regarding promotion
    or graduation
  • They may be inappropriately placed into special
    educational programs where they receive
    inappropriate instruction
  • SWD students may not have received the same
    curriculum which is assumed for the test

4
  • Should Schools Test English Language Learners?

? Yes
English language learners (ELLs) can be placed at
a disadvantage because
  • Assessment outcomes may not be valid because
    their low level of English proficiency
    interferes with content knowledge
    performance
  • Test results affect decisions regarding promotion
    or graduation
  • They may be inappropriately placed into special
    educational programs where they receive
    inappropriate instruction
  • ELL students may not have received the same
    curriculum which is assumed for the test

5
  • Should Schools Test English Language Learners and
    Students with Disabilities?

? No
  • Problems
  • Due to the powerful impact of assessment on
    instruction, ELL and SWD students quality of
    instruction may be affected
  • If excluded, they will be dropped out of the
    accountability picture
  • Institutions will not be held responsible for
    their performance in school
  • They will not be included in state or federal
    policy decision
  • Their academic progress, skills, and needs may
    not be appropriately assessed

6
Problems in AYP Reporting for ELL Students
  • Problems in classification/ reclassification of
    ELL students (moving target subgroup)
  • Measurement quality
  • Low baseline
  • Instability of the ELL subgroup
  • Sparse ELL population
  • ELL cutoff points (Conjunctive vs. Compensatory
    model)

7
Percent of ELL student in 2000-2001 (Kindler,
2002)
  • California 1,511,646 25.0
  • New Mexico 63,755 19.9
  • Arizona 135,248 15.4
  • Texas 570,022 14.0
  • Nevada 40,131 11.8
  • Florida 254,517 10.7
  • Utah 44,030 9.3
  • Oregon 47,382 8.7

8
Composition of SWD PopulationNational Statistics
  • Grade 8
  • 10 Nationally
  • Ranging between 6 to 14 by different states
  • Grade 4
  • 11 Nationally
  • Ranging between 4 to 17 by different states

9
Some recent statistics
  • In 2005-2006 there were a total of 559,215
    students were enrolled in the K-12 public schools
    in Oregon
  • Of the total, 11.7 (65,239) were ELLs (as
    compared with 8.7 in 2000-2001) and 12.8
    (71,517) were Special Education students.
  • Of the 65,239 ELL students, (76.9) had Spanish
    as their language of origin which is about 9 of
    the total students enrolled

10
Population Change
11
Special Education
12
Language Diversity
13
A Point for Discussion
  • This large majority of ELL students speaking the
    same language makes Oregon one of the most
    eligible state to use native language testing (in
    Spanish).
  • However, the decision is not that simple. Why
    not, what are the issues?
  • Alignment of language of assessment and language
    of instruction
  • Technical issues in translation or
    trans-adaptation
  • Comparability Issues

14
Students with Disabilities and Statewide Testing
in Oregon
  • Students with Disabilities being served with
    active Individual Education Plans (IEP) or 504
    plans have a set of choices for participation in
    Oregons Statewide Assessment System.
  • Implementing accommodations and/or modifications
  • Out of level testing (no longer an option in
    Oregon)
  • Alternative Assessments for those with profound
    cognitive disabilities

15
Alternative Assessments(Oregon)
  • The Alternate Assessment is comprised of tasks
    designed to measure basic skills which are
    anchored to the Oregon standards.
  • The Alternate Assessment is administered for the
    same age groups (in the Spring).
  • Tasks can be administered in a variety of ways
    and students can respond with considerable
    latitude (e.g., pointing, sign language).

16
Challenges in Statewide Testing of ELLs
Students with Disabilities
  • Providing effective and valid accommodations
  • Developing reliable and valid alternate
    assessment
  • Most importantly, comparability issues

17
Why Should English Language Learners be
Accommodated?
  • Their possible English language limitations may
    interfere with their content knowledge
    performance.
  • Assessment tools may be culturally and
    linguistically biased for these students.
  • Linguistic complexity of the assessment tools may
    be a source of measurement error.
  • Language factors may be a source of
    construct-irrelevant variance.

18
Why Should Students with Disabilities be
Accommodated?
  • Their disabilities put them at disadvantage
  • Accommodations must be provided to level the
    playing field

19
Performance Difference Between SWD and Non-SWD
20
Site 4 Grade 8 Descriptive Statistics for the
SAT 9 Test Scores by Strands
21
Reading Science
Math M SD M SD M SDGrade
10 SWD only 16.4 12.7 25.5 13.3 22.5 11.7
ELL only 24.0 16.4 32.9 15.3 36.8 16.0 ELL
SWD 16.3 11.2 24.8 9.3 23.6 9.8 Non-ELL/SWD
38.0 16.0 42.6 17.2 39.6 16.9 All
students 36.0 16.9 41.3 17.5 38.5 17.0 Grade
11 SWD Only 14.9 13.2 21.5 12.3 24.3 13.2
ELL Only 22.5 16.1 28.4 14.4 45.5 18.2 ELL
SWD 15.5 12.7 26.1 20.1 25.1 13.0 Non-ELL/SWD
38.4 18.3 39.6 18.8 45.2 21.1 All
Students 36.2 19.0 38.2 18.9 44.0 21.2
Normal Curve Equivalent Means Standard
Deviations for Students in Grades 10 and 11, Site
3 School District
22
Stanford 9 Sub-scale Reliabilities (Alpha), Grade
9
23
Grade 11 Stanford 9 Reading and Science
Structural Modeling Results, Site 3
24
Fundamental Questions
  • Why should ELLs and SWDs be accommodated? (to
    level the playing field)
  • What are the major characteristics of
    accommodations that would level the playing
    field?
  • Do the most commonly used accommodations possess
    those characteristics

25
The major characteristics of accommodations that
would level the playing field
  • Effective
  • Valid
  • Consistent with students background
  • Feasible
  • Relevant
  • From now on our focus will be on ELL students

26
Problems with Accommodation Usage
  • Accommodations for English language learners are
    often selected based on feedback from teachers
    and bilingual coordinators without enough
    influence from research findings
  • Several studies have identified some
    accommodations that may provide unfair advantage
    to the recipients of the accommodations and may
    render invalid results

27
Examples of Accommodations for ELL Students that
May Alter the Construct
  • Providing an English dictionary (Abedi, Courtney,
    Leon, 2003 Abedi, Lord, Boscardin, Miyoshi,
    2000)
  • Providing extra or extended time (Abedi, Lord,
    Hofstetter, Baker, 2000 Hafner, 2001 Thurlow,
    2001)
  • Translating tests into students native language
    (Abedi, Lord, Hofstetter, Baker, 2000)
  • By gaining access to definition of
    content-related terms, recipients of a dictionary
    may be advantaged over those who did not have
    access to the dictionaries. This may compromise
    the validity of assessment (Abedi, Courtney,
    Mirocha, Leon, Goldberg, 2005)

28
Native language testing
  • Problems in creating parallel forms of the test
  • Translation issue
  • Alignment of language of assessment and language
    of instruction

29
Glossary Plus Extra Time
Glossary with extra time raised the performance
of both ELL and non-ELL students (Abedi,
Hofstetter, Lord, and Baker, 1998, 2000)
  • ELL students performance increased by 13 when
    they were tested under glossary with extra time
    accommodation.
  • While this looks promising, it does not present
    the entire picture.
  • Non-ELL students also benefited from this
    accommodation, with an increase of 16.
  • English and bilingual dictionaries recipients may
    be advantaged over those without access to
    dictionaries. This may jeopardize the validity of
    assessment.

30
How can accommodations be examined for validity?
Only through experimentally-controlled research
where
  • ELL and non-ELL students are randomly assigned to
    experimental and control groups
  • Both ELL and non-ELL students are observed under
    accommodated and non-accommodated assessments
  • Using existing data?

31
How the validity of accommodations can be tested
in an experimentally controlled condition?
32
Characteristics of ELL students
  • ELL students constitute a very diverse and
    heterogeneous population (SES, cultural and
    linguistic backgrounds).
  • They can be vastly different in their level of
    proficiency in their native or home language
  • They are quite different in their level of
    proficiency in English
  • Studies show that the level of English language
    proficiency of these students range from high
    (even higher than some native English speakers)
    to very low

33
The Most Commonly Used Accommodations for ELL
Students Include(Rivera, 2003)
  • Extended time (42 of the 48 states)
  • Use of glossary (26 states)
  • Use of an English dictionary (33 states)
  • Use of a bilingual dictionary (22 states)
  • Linguistically-simplified test items (12 states)

34
The Most Commonly Used Accommodations for
Students With Disabilities Include (Thurlow, et
al, 2001 Tindal et al, 2000)
  • Braille (allowed by 33 of the 48 states studied)
  • Computerized assessment (34 states)
  • Dictation of responses to a scribe (32 states)
  • Extended time (37 states)
  • Translation of instructions (40 states)
  • Allowance for marking answers in the test
    booklets (33 states)
  • Test items read aloud (34 states)
  • Simplified test directions (31 states)
  • Test breaks (33 states).

35
Do we have enough evidence on the following
characteristics of these most commonly used
accommodation?
  • Effective
  • Valid
  • Consistent with students background
  • Feasible
  • Relevant

36
Accommodations for ELLs and Students with
Disabilities
  • Can the same accommodations used for students
    with disabilities be used for ELLs?
  • Can the same accommodations used for ELLs be used
    for students with disabilities?

37
SY 2000-2001 Accommodations Designated for ELLs
Cited in States Policies
There are 73 accommodations listed N Not
Related R Remotely Related M Moderately
Related H Highly Related
From Rivera (2003) State assessment policies for
English language learners. Presented at the 2003
Large-Scale Assessment Conference
38
There are 73 Accommodations Listed
  • 47 or 64 are not related
  • 7 or 10 are remotely related
  • 8 or 11 are moderately related
  • 11 or 15 are highly related

39
Samples accommodations used for ELL students
  • Test-taker marks answers in test booklet
  • Copying assistance provided between drafts
  • Test-taker indicates answers by pointing or other
    method
  • Paper secured to work area with tape/magnet
  • Physical assistance provided

40
Samples accommodations used for ELL students
  • Enlarged answer sheets provided
  • Breaks provided
  • Test individually administered
  • Test administered in small group
  • Test administered in location with minimal
    distraction

41
Assessment Options for all ELLs in Oregon
  • Students may take the test under standard
    administration with or without accommodations
  • Students may take side-by-side English/Spanish,
    English/Russian versions where provided.
  • Eligible students may respond on the Writing
    Assessment in Spanish.
  • Students may take the test under modified
    conditions.
  • Students in grade 3 may take the Aprenda (Spanish
    reading test at grade 3).
  • Students may use the Juried Assessment process
    for reading or writing in another language.

42
Test Decisions for ELLs(Oregon)
  • A teacher and instructional team who know the
    student make the decision to test under standard
    conditions or modify test
  • Consult parent/guardian
  • Each student must be considered individually for
    each assessment
  • Best interest of student
  • Not on participation in a program nor
    identification as an ELL
  • How about research evidence?

43
Accommodations TablesChanges in Timing or
Scheduling
44
Accommodations TablesChanges in Test Directions
45
Accommodations TablesChanges in How the Test
Questions are Presented
46
Accommodation TablesChanges in How the Student
Responds
47
Accommodation TablesChanges in Test Setting
48
How are we doing in practice nationally?
  • Are states and districts across the nation
    cognizant of this important principle of using
    accommodations that are appropriate for ELLs?
  • Are there any objective national criteria to help
    states to select appropriate accommodations for
    ELL students?
  • Or, is the assignment of accommodations to these
    students based on temporary and subjective
    decisions?

49
Comparability Issues
  • If accommodated assessment is not valid then the
    outcome may not be comparable with the
    non-accommodated assessment (a major peer-review
    concern)
  • To report AYP for ELL students, it is imperative
    to establish the validity of accommodated
    assessment
  • Construct-irrelevant sources such as linguistic
    and cultural biases should be controlled before
    reporting AYP based on accommodated assessments

50
Are there accommodations that would benefit both
ELLs and SWDs?
  • Assessment can be designed in a way to be
    accessible to both groups.
  • For example, long tests, crowded pages, tables
    and texts would create frustration and anxiety
    for everyone, particularly for ELLs and SWDs.
  • Complex linguistic structure of assessment would
    be a major nuisance variable for ELLs and SWDs
    (Tindal, G., Anderson, L., Helwig, R., Miller,
    S., Glasgow, A. (2000). Accommodating students
    with learning disabilities on math tests using
    language simplification. Eugene University of
    Oregon, RCTP)

51
A Sample Representing a Subgroup of Students with
Disabilities
52
A Sample Representing a Subgroup of Students with
Disabilities
53
A Clear Language of Instruction and Assessment
Works for ELLs, SWDs, and Everyone
  • What is language modification of test items?

54
Learning Disability Language of Assessment
  • Students in the Learning Disability category may
    have difficulty processing complex language in
    assessment
  • Simplifying the language of test items will help
    students with disabilities, particularly those
    with learning disabilities
  • As the sample page suggests, a large majority of
    students with disabilities are in the Learning
    Disability category

55
Examining Complex Linguistic Features in
Content-Based Test Items
56
Linguistic Modification Concerns
  • Familiarity/frequency of non-math vocabulary
    unfamiliar or infrequent words changed
  • census gt video game
  • A certain reference file gt Macks company
  • Length of nominals long nominals shortened
  • last years class vice president gt vice
    president
  • the pattern of puppys weight gain gt the pattern
    above
  • Question phrases complex question phrases
    changed to simple question words
  • At which of the following times gt When
  • which is best approximation of the number gt
    approximately how many

57
Linguistic Modification cont.
  • Voice of verb phrase passive verb forms
    changed to active The weights of 3 objects were
    compared gt Sandra compared the weights of 3
    rabbits
  • If a marble is taken from the bag gt if you take
    a marble from the bag
  • Conditional clauses conditionals either
    replaced with separate sentences or order of
    conditional and main clause changed
    If Lee delivers x newspapers gt Lee delivers x
    newspapers
  • If two batteries in the sample were found to be
    dead gt he found three broken pencils in the
    sample
  • Relative clauses relative clauses either
    removed or re-cast
  • A report that contains 64 sheets of paper gt He
    needs 64 sheets of paper for each report

58
Harriet, Jim, Roberto, Maria, and Willie are
in the same eighth grade class. One of them is
this years class president. Based on the
following information, who is the class
president?The class president was last years
vice president and lives on Vince
Street.Willie is this years class vice
president.Jim and Maria live on Cypress
Street.Roberto was not last years vice
president. A. JimB. HarrietC. RobertoD.
MariaE. Willie 
Original Item
59
Modified Item
Harriet, Jim, Roberto, Maria, and Willie ran
for president of their eight-grade class. One of
them won. Who is president? The president now
was vice president last year and lives on
Vince Street.Willie is vice president now.Jim
and Maria live on Cypress Street.Roberto was not
vice president last year. A. JimB. HarrietC.
RobertoD. MariaE. Willie
60
OriginalThe census showed that three hundred
fifty-six thousand, ninety-seven people
lived in Middletown. Written as a number,
that isA. 350,697B. 356,097C. 356,907D.
356,970
ModifiedJanet played a video game. Her score
was three hundred fifty-six thousand,
ninety-seven. Written as number, that
is A. 350,697B. 356,097C. 356,907D. 356,970
61
Interview Study
  • Table 1. Student Perceptions Study First Set
    (N19)
  • Item Original item chosen Revised item chosen
  • 1 3 16
  • 2 4 15
  • 3 10 9
  • 4 11 8
  • Table 2. Student Perceptions Study Second Set
    (N17)
  • Item Original item chosen Revised item
    chosen5 3 14
  • 6 4.5a 12.5
  • 7 2 15
  • 8 2 15

62
Many students indicated that the language in the
revised item was easier
  • Well, it makes more sense.
  • It explains better.
  • Because that ones more confusing.
  • It seems simpler. You get a clear idea of
    what they want you to do.

63
The revised items need less time for response
  • Its easier to read, and it gets to the point,
    so you wont have to waste time.
  • I might have a faster time completing that one
    cause theres less reading.
  • Less reading then I might be able to get to the
    other one in time to finish both of them.
  • Cause its, like, a little bit less writing.

64
Conclusions and Recommendations
  • Accommodations
  • Must be relevant in addressing assessment issues
    for ELL students
  • Must be effective in reducing the performance gap
    between accommodated and non-accommodated
    students
  • Should not alter the construct being measured
  • The accommodated results can be aggregated with
    the assessments under standard conditions
  • Must be feasible in national and state assessments

65
Conclusions and Recommendations
Examples of research-supported accommodations
  • Providing a customized dictionary is a viable
    alternative to providing traditional
    dictionaries.
  • The linguistic modification of test items that
    reduce unnecessary linguistic burdens on students
    is among the accommodations that help ELL
    students without affecting the validity of
    assessments.
  • Computer testing with added extra time and
    glossary was shown to be a very effective, yet
    valid accommodation (Abedi, Courtney, Leon, and
    Goldberg, 2003)

66
Conclusions and Recommendations
  • It is thus imperative to examine different forms
    of accommodations before using them in state
    and/or national assessments.
  • Without information on important aspects of
    accommodations such as validity, it would be
    extremely difficult to make an informed decision
    on what accommodations to use and how to report
    the accommodated and non-accommodated results.

67
Accountability Questions
  • Are there specific ELL/SWD subgroup features that
    affect the accountability system?
  • Yes No
  • Could the current accountability system for
    ELLs/SWDs be improved?
  • Yes No
  • Do research findings help inform assessment
    accountability systems for these students?
  • Yes No

68
For more information, please contact Jamal Abedi
at UC Davis/CRESST
  • (530) 754-9150
  • or
  • jabedi_at_ucdavis.edu
Write a Comment
User Comments (0)
About PowerShow.com