STELLA A Computerized System for Individually Assigning Test Accommodations to ELLs

1
STELLA A Computerized System for Individually
Assigning Test Accommodations to ELLs

• Rebecca Kopriva, University of Maryland
• Therese Gleason Carr, South Carolina Department
  of Education
• rkopriva_at_umd.edu
  tcarr_at_sde.state.sc.us

2

• STELLA stands for
• The
• Selection Taxonomy
• for
• English Language Learner Accommodations

3
Purpose of STELLA

• While appropriate accommodations are being
  identified and integrated into large-scale
  academic achievement systems, how do the proper
  accommodations get to the correct students?
• This is the question that STELLA addresses.

4
One Size Doesnt Fit All

• Blanket Accommodations All students receive the
  same suite of accommodations.
• Sometimes based on expediency.
• Sometimes an attempt to give students all the
  support possible, e.g., all allowable
  accommodations, whether effective/appropriate or
  not.
• For example an oral administration may
  negatively impact a student who has higher
  reading proficiency than oral language
  proficiency.

5
Making the case for differential accommodations

• There is a need for differential accommodations
  decision making based on individual student
  characteristics and variables.
• Currently, there is no set of standard,
  research-based guidelines for making consistent
  decisions about which accommodations are
  appropriate for individual English language
  learners.
• STELLA allows for the triangulation of data from
  Records, Parents/Guardians, and Teachers.

6
STELLA Overview

• Data Collection
• Teacher Form
• Parent Form
• Records Form
• Preloaded in System
• Student Information Variables
• Test Accommodations
• Conversion and Consolidation Rules
• Decision-Making Rules
• Output
• Student Profile
• Accommodation Decisions for Each Student
• Associated Materials
• Data Collection Manual
• Output Interpretation Manual
• Technical Manual

7
STELLA Development Initial Questions

• What are the salient student background
  variables that should be taken into account in
  accommodations decision making?
• What are the active characteristics in
  accommodations that make them effective for
  individual students?

8
Relevant Characteristics of Students
9
Relevant Characteristics of Students cont.
10
Relevant Characteristics of Students cont.
11
Relevant Characteristics of Students cont.
12
Current Domain of Accommodations

• Pre-Test Best Practice Accommodations
• Family Assessment Night
• Tailored Classroom Support
• Forms
• Standard or some Universal Design Forms
• Access-based Form
• L1 or Side-by-Side forms as available
• Tools
• Bilingual word list, general or test specific
• Picture-word dictionary
• Problem solving tools

13
Current Domain of Accommodations

• Administration
• Small group
• Individual Administration
• Oral English
• Oral L1
• Language Liaison
• Extra time
• More frequent breaks
• Response
• Written L1 or code switching
• Oral English
• Oral L1 or code switching
• Demonstrated or modeled response

14
Records Form
Language of instruction
English language proficiency test
L1 proficiency test
15
Records Form
Type of ELL program
16
Parent/Guardian Interview Form

• Interview protocol with rating scales
• L1 information, 4 domains
• Full-time academic programs in U.S.
• Length of time in U.S. schools
• Consistency
• School atmosphere in native country if applicable
• Time (months, days/week, hours/day)
• Number of students in classroom
• Describing the school (e.g. chalkboards, desks,
  textbooks per student, other books, supplies for
  math or science, additional comments

17

L1 Proficiency in four domains
18
Attendance in U.S. schools
Schooling in native country
19
Classroom equipment, books, and supplies in
native country
Types of assessments in native country
20
Grading and assignments
Test scores and experiences with testing
21

• Teacher Form
• English and L1 proficiency
• (L1 includes dont know option)
• Standardized score accuracy and judgments
  about reasons for inaccuracy
• Students experience with standard test formats
• Students perceived purpose of standardized
  testing
• Classroom test condition options
• Condition options that help student on classroom
  tests, evaluations

22
Experience with standardized testing Testin
g conditions used in the classroom
23
Beneficial testing conditions in the classroom
24
System Output

• Individual Student Profile
• English language proficiency, 4 domains
• L1 proficiency, 4 domains
• Cultural proximity, 3 variables
• Time in U.S.
• Experience with testing procedures
• Schooling proximity, native country
• to U.S. experience

25
System Output STELLA, Teacher Recommended and
SEA/LEA Allowed
Since different states allow different
accommodations, STELLA program output is set up
to identify the subset of allowable
accommodations for each student.
Non allowable accommodations selected for each
student based on research and other best
practice literature are also displayed in gray.
This provides guidance to teachers and others
about additional accommodations that could be
useful for ELL students with specified needs.
26
STELLA Formative Development

• State and District Survey
• Teacher Focus Groups
• Literature Review
• Parent Interviews
• Teacher Interviews
• Expert Panel Reviews

27
State and District Surveys (summer/fall 2003)

• Survey designed to gather comprehensive
  information about policies and policy development
  of accommodations selection and use.
• Policy makers from grant partner SEAs and LEAs
  completed detailed questionnaires about
  accommodations for ELL and SWD.
• States provided state policy information and
  information from four representative districts.

28
Teacher Focus Groups(fall 2003)

• Outcomes
• Teachers had clear opinions about the sorts of
  accommodations that work.
• While teachers clearly used some sort of
  decision-making process to assign accommodations,
  they struggled to articulate the process.
• Teachers provided a rich list of classroom
  accommodations, but not information on how they
  matched these accommodations to specific
  students.

29
Literature Review

• Literature reviews on large-scale and classroom
  accommodations were compiled.
• Results were combined with the survey and focus
  groups to identify salient student variables, and
  promising accommodations.

30
Parent Interviews (fall 2004)

• In Maryland, 12 parents of ELL students reviewed
  the Parent/Guardian Form/Interview Protocol.
• Parents were interviewed using the form and asked
  to rate the difficulty and clarity of each
  question and their ability to answer it on a
  scale of 1 (easy to understand) to 5 (difficult
  to understand).
• Most questions were rated as fairly easy to
  understand a few were rated as difficult.
  Suggestions from parents were incorporated into
  the form.

31
Teacher Interviews(fall 2004)

• Teachers in Washington, D.C. completed each
  STELLA data collection form and were interviewed
  to provide feedback .
• Teachers indicated that overall, language in
  forms was clear, appropriate.
• Teachers agreed that critical information was
  collected on the forms, but identified tension
  between need to collect this much information and
  the time necessary to complete the forms.

32
Expert Panel Reviews

• In winter and spring 2005, three experts reviewed
  the algorithms/decision rules underpinning the
  STELLA system.
• Experts had extensive knowledge in the
  instruction and assessment of English language
  learners.
• Purpose of expert panel was to review and revise
  the conversion, consolidation, and
  decision-making rules of STELLA.

33
Test Conversion Rules

• Currently, output from 4 tests are preloaded.
  There is also the opportunity for personnel to
  add results from another test.
• In all cases score conversion rules place
  students on a common scale with four levels
• Beginner ELL
• Low intermediate ELL
• High intermediate ELL
• Grade level competitive ELL

34
Student Information ConsolidationRules

• In several cases, more than one piece of student
  information is used to make judgments about the
  his or her level on relevant variables. Two types
  of consolidation rules are part of STELLA. These
  are
• Consolidation of data from related items
• Consolidation of information from more than one
  source

35
Combining Data from More Than One Question An
Example

• Time in School LOW
• If the student has been in the US less than 1
  academic year
• OR
• If the student has been in US between 1 and 2
  years AND has missed more than two months of
  school per year for 1 or more years.
• OR
• If the student has been in US between 2 and 3
  years AND has missed more than two months of
  school per year for more than one year.

36
Combining Data from More Than One Source An
Example

• Consolidation rules are also used to determine
  L1 proficiency and English language proficiency .
  
• For L1 proficiency consolidation rules are
  applied to information from teacher, parent and
  records.

37
What Trumps What? Consolidation Rules Example

• Parents describe student as performing above
  grade level in L1
• Teacher describes student as performing below
  grade level in L1
• Parents trump teachers knowledge of L1, but test
  scores of L1 proficiency trump teachers.

38
Test Accommodation Decision-Making Rules

• A beginning set of decision-making rules were
  developed and tested. These rules take relevant
  student information and pair it with relevant
  accommodation factors for individual students.
  The accommodation factors identify which student
  needs the specific accommodation was designed to
  remediate.
• In this way, individual students are matched
  with accommodations appropriate to their
  particular needs.

39
Example of Decision-Making Rules
40
Decision-Making Rules cont.

• At this time, the rules explicitly use
  information from English language reading and
  oral proficiency and L1 reading proficiency to
  make broad decisions.
• Cultural proximity and US schooling variables
  are informally used to make the final decisions
  about the selection of accommodations. Future
  decision trees will be developed which formally
  identify how these latter factors are used.

41
STELLA Adaptation

• 1. The STELLA platform allows the developer to
  add additional questions to the forms, refine, or
  delete as necessary.
• The platform also allows the developer to upgrade
  the systems conversion, consolidation, and
  decision-making algorithms.
• New accommodations can be added to the system as
  well.

42
Example of New Accommodations
43
Initial Verification Studies

• Cut-score Study
• Independent Raters Study

44
1. Computer-based Cut-Score Study

• Method
• 276 3rd and 4th grade Spanish-speaking ELL
  students administered mathematics test (30
  multiple choice and 3 constructed response
  items).
• Based on information from teachers, schools,
  appropriate accommodations were identified for
  each student.
• Accommodations (picture-word, Spanish-English,
  oral English) were randomly assigned. Each
  student received none, 1, 2 or 3.

45
1. Cut-Score Study cont.

• After completing test
• Students were assigned to 3 groups Appropriate
  accommodation group, random accommodation group,
  no accommodation group (IV) scores (DV).
• Findings
• ANOVA results indicate a significant difference
  (F3.2, p.04).

46
1. Cut-Score Study cont.

• Findings cont.
• Appropriate accommodations group scored
  significantly higher than other two groups
• t (no/app) 2.24, p.03
• t (random/app) 2.33, p.02
• No significant difference between random and no
  accommodations groups
• t (random/no) 1.67, p .49
• Regression results found accommodations addressed
  ELP reading and listening, L1 reading needs

47
1. Cut-Score Study cont.

• Implications
• For accommodations assignment
• Variables utilized appear to be among the most
  salient.
• For validity of scores
• When accommodations were non-appropriate, scores
  did not increase over those receiving no
  accommodations. With appropriate accommodations,
  scores did increase. This pattern suggests
  improved validity.

48
2. Independent Raters Study

• Method
• 5 sets of accommodations (STELLA, 3 teacher
  recommendations, 1 randomly generated)
• 4 raters 3 teachers/ELL specialists, 1
  researcher with experience in ELL testing
• Reviewed completed forms for each student
• Blindly rated 5 sets of accommodations for each
  student from most appropriate to least
  appropriate

49
2. Raters Study cont.

• Findings
• ANOVA results of ratings found a significant
  difference by accommodation sets (F 76.789, p
  lt.001).
• Significant difference between STELLA findings
  and all other findings, best fit for STELLA (p
  .000).
• No significant difference between any of the
  other findings.
• Rater by source interaction was not significant
  (F 1.184, p .288), suggesting that raters did
  not differentially assign accommodation sets

50
2. Raters Study cont.

• Implications
• Findings suggest that teacher ratings are not
  significantly different from random, no matter
  how much targeted information they collect.
• Even when teachers know what variables they are
  asked to focus upon, their results are not
  significantly different from when they are asked
  to assign accommodations based on only their
  understanding of the students.
• STELLA results consistently and significantly
  provide best fit.

51
Next Steps

• Continual refinement and customization necessary.
• Additional or more explicit cut points need to be
  validated.
• Data needs to be collected on the impact of the
  system for different ages and content areas.