Redesign of Alternate Assessments for Students with Significant Disabilities: National and State Perspectives

1
Redesign of Alternate Assessments for Students
with Significant Disabilities National and State
Perspectives

• CCSSO
• June 16, 2008

2
Presentation Team

• Moderator Patty McDivitt, Data Recognition
  Corporation
• Aran Felix, Alaska Department of Education and
  Early Development
• Melissa Fincher, Georgia Department of Education
• Claudia Davis, Louisiana Department of Education
• Discussant Rachael Quenemoen, National Center
  for Educational Outcomes

3
Presentation Focus

• Purpose and Rationale
• Goals
• Development
• Challenges
• Lessons Learned

4
Alaska

• Aran Felix
• Alaska Department of Education and Early
  Development

5
Purpose and Rationale

• Alaskas previous Alternate Assessment
• Portfolio assessment
• Housed in the Special Education Unit
• Assessed English/Language Arts, Math, and Skills
  for Healthy Living (only E/LA and Math used for
  AYP)
• Intended use for grades 3, 6, 8, 11 only
• Stressed academic content
• Alternate Performance Standards (content
  standards)
• Used as Goals on student IEPs
• Stressed inclusion (Generalization dimension of
  scoring)
• Department of Education organized scoring
  sessions
• Single set of achievement standards
• Scoring dimensions Skill (student achievement),
  Generalization, Appropriateness

6
Purpose and Rationale

• Steps to explore a solution
• Alternate Assessment moved to Assessment Unit
• Teacher and Parent Survey conducted
• Reliability-Validity Study conducted
• Considered removing Generalization
  Appropriateness dimensions from scoring for AYP
  report only to districts.
• Needed a new standard setting and better overall
  technical quality
• OSEP Condition drove development of a Data-folio
  assessment for grades 4, 5, 7, 9, 10, Portfolio
  for all grades overwhelming teachers.
• Accomplished the original intent of inclusion and
  teaching/assessing content standards
• Moved scoring to test vendor
• Passed Peer Review requirements

7
Goals

• Assess a broader range of content standards
• Manage test size and testing window
• Increase standardization
• content assessed (apples to apples)
• training of teachers to administer the assessment
• scoring of the assessment
• training of the protégés
• Establish a higher technical quality
• Create an online assessment (training, scoring,
  reporting)

8
Development

• 1) Funding Issues
• Small population of alternate-eligible students
• Request for Information
• 2) What type of assessment to develop?
• Research other states approaches
• Research using another states item bank
• Decision needed Retrofit portfolio with
  performance tasks or move to a performance task
  system?

9
Development

• Look of new assessment
• Used another states item bank while examining
  alignment the first year
• Performance Tasks in content areas
• 4-8 items per task
• Paper/pencil materials allowing for some
  adaptation
• Assessors administer one-on-one to student
• Online test materials (scoring protocols, student
  materials, training manuals)
• Online training for test administrators plus
  proficiency modules to maintain reliability
• Online scoring, reporting, unofficial report
• Original 3-year plan compressed to 2 years (peer
  review)

10
Development

• Year 1 2005-2006 (Portfolio still in place)
• Developed Request for Proposals
• Contracted with new vendor
• Convened committees of educators
• Developed Proficiency level descriptors and
• Developed Extended Grade Level Expectations
• Piloted online system with technology
  coordinators
• Created security access levels
• Developed online security rules
• Trained Qualified Assessors for Pilot
• Pilot tested the new assessment
• Created a crosswalk of test items to Alaska
  content standards (phase 1 blueprint)

11
Development

• Year 2 2006-2007
• Developed cousin items
• Conducted bias review
• Conducted statewide trainings and orientations
• Developed a qualified assessor and qualified
  mentor-trainer path and materials
• Included mentor pre-test and debrief audios
• Administered assessment (one test)
• Piloted science assessment
• Conducted standard setting for RWM
• Received Technical Report
• Submitted new assessment to Peer Review (and
  received approval)
• AYP and student reports to parents/districts/websi
  te

12
Development

• Year 3 2007-2008
• Developed cousin items and conducted bias review
• Repackage all content tests into grade clusters
  vs. grade level within content areas
• New look for website
• Trained Mentors on science assessment
• Administered all assessments including science
• Teacher Survey of Consequential Validity, Mentor
  Audios
• Conducted standard setting for science
• Conducted standard validation for
• Received Technical Report
• Submitted new assessment to Peer Review (and
  received approval)
• Piloted science assessment
• AYP and student reports to parents/districts/websi
  te

13
Development

• Future Plans for years 2008-2011
• Develop larger item bank, conduct reviews
• Field test new items
• Construct new operational test forms, A B
• Conduct a new Standard Setting on Forms A and B
  of each content-area assessment
• Consider a program review by external evaluator

14
Challenges

• Compressing the schedule from 3-4 years to 2
  years
• Possibility of administering two Alternates
  simultaneously
• Using another states items
• Alaskanizing the assessment
• Moving from a one-size model to a grade-cluster
  model

15
Lessons Learned

• The importance of flexibility and responsiveness
• Understanding capacity when huge demands (such as
  compressed time schedule) are imposed by second
  peer review deadlines
• The role of an outside consultant as EED advisor
  for the first years of the project
• Involvement of TAC prior to issuing RFP

16
Recommendations

• RFP get help from other states or hire a
  consultant include any customizing details
  require experienced project manager include
  right to own test items
• Link pattern of test development to general
  education cycle and include content experts in
  all aspects of test development (standards to
  test items)
• Communicate with the field regularly through the
  process of developing extended content standards.
  Provide these standards to the field early.
• If using any online pieces conduct pilot of
  online system to explore platform issues and get
  sign-off from someone in charge at district level
• Exercise caution when updating online system
  during test window
• Include an errata page on online system for
  notifying field of updates
• Have a backup server if you have online
  components
• Remain cheerful!

17
Georgia

• Melissa Fincher
• Georgia Department of Education

18
Purpose and Rationale

• Why did Georgia redesign our alternate?
• In a word COMPLIANCE
• In a few words OPPORTUNITY TO LEARN

19
Goals

• The GAA is designed to ensure that students with
  significant cognitive disabilities are
• Provided access to the state-mandated curriculum.
• Given the opportunity to demonstrate progress
  toward achievement of curriculum knowledge,
  concepts, and skills.

20
Development

• The GAA is a portfolio of student work provided
  as evidence that a student is making progress
  toward grade-level academic standards.
• Evidence provided must show instructional
  activities and student work that is aligned to
  specific grade-level standards.

21
Development

• Georgia elected to go with a portfolio format to
  allow flexibility for a very diverse group of
  students.
• The portfolio format allows the teacher, who
  knows the student best, to design and document
  instructional tasks that are meaningful and
  purposeful for the individual student.

22
Development

• Collaboration between Assessment, Special
  Education, and Curriculum
• Significant investment in training of educators
  surrounding curriculum access
• Ongoing documentation of decisions surrounding
  development and implementation
• Involvement of Georgias Technical Advisory
  Committee
• Augmented with an AA-AAS expert

23
Challenges

• Documenting technical quality
• Traditional indices do not lend themselves easily
  to alternate assessments
• Federal Peer Review
• Complicated assessment program
• Teacher buy-in

24
Lessons Learned

• Investing in teacher training pays off, but it
  needs to be continual as the assessment evolves.
• Technical documentation of alternate assessments
  looks different, but it is worth pursuing.
• Validity is an ongoing journey.
• Students are doing things we never thought
  possible.

25
Louisiana

• Claudia Davis
• Louisiana Department of Education

26
Purpose and Rationale

• Purpose Louisianas alternate assessment did not
  receive approval in the USDOE peer review
  process.

27
Goals

• Louisianas alternate assessment (LEAP Alternate
  Assessment or LAA) lacked
• Academic focus (although linked to state
  standards)
• Alignment with grades or grade spans
• USDOE advised a redesign of the LAA to be
  implemented in spring of 2008
• LDE began the redesign in July of 2007

28
Development

• First, Louisiana developed Extended Standards
  (ES), i.e., extensions of state content
  standards.
• What should students with significant cognitive
  disabilities know and be able to do?
• How do we identify expectations regarding the
  breadth and depth of the standards, benchmarks,
  and grade-level expectations (GLEs)?
• How should students demonstrate knowledge and
  skills based on the GLEs?
• What is the appropriate range of tasks to be used
  to measure this knowledge and these skills?

29
Development

• LDE contracted with the testing vendor to develop
  Extended Standards for ELA, Mathematics, and
  Science.
• Recommendations from an initial committee of
  state special educators regarding selected
  standards and a template for the ESs, including
  Complexity Levels, were used to guide the
  development.
• Committees of Louisiana educators (general and
  special educators) reviewed the proposed ESs with
  Complexity Levels.
• Content-area groups across four grade spans
  (grades 3-4, 5-6, 7-8, and 9-11)
• Review was time intensive -- one intensive week

30
Development

• Complexity Levels of the Extended Standards
• Each ES has 3 levels of complexity (1 being least
  difficult) that provide access to general
  education concepts and skills.
• They serve as guidelines for the development of
  assessment tasks at 3 levels of complexity.
• Math Example
• 3. Add and/or subtract to solve simple problems.
• 2. Identify simple addition and subtraction
  concepts within daily living
  problems.
• 1. Count to solve simple problems.

31
SAMPLE PAGE AND KEY FOR ENGLISH LANGUAGE ARTS SAMPLE PAGE AND KEY FOR ENGLISH LANGUAGE ARTS SAMPLE PAGE AND KEY FOR ENGLISH LANGUAGE ARTS SAMPLE PAGE AND KEY FOR ENGLISH LANGUAGE ARTS
Standard One Students read, comprehend, and respond to a range of materials, using a variety of strategies for different purposes. Standard One Students read, comprehend, and respond to a range of materials, using a variety of strategies for different purposes. Standard One Students read, comprehend, and respond to a range of materials, using a variety of strategies for different purposes. Standard One Students read, comprehend, and respond to a range of materials, using a variety of strategies for different purposes.

Benchmarks Grade-Level Expectations Extended Standards Complexity Levels
ELA-1-E4 recognizing story elements (e.g., setting, plot, character, theme) and literary devices (e.g., simile, dialogue, personification) within a selection 8. Identify story elements, including theme conflict character traits, feelings, and motivation (ELA-1-E4) 5. Identify a variety of story elements, including the impact of setting on character multiple conflicts first- and third-person points of view development of theme (ELA-1-E4) ES-8-5 Identify story elements, including character 3. Identify the main character in a story 2. Identify two characters in a story 1. Identify one character in a story
3 is most complex
Extended Standard ES-8 /5 refers to GLE 8 from
grade 3 ES-8/ 5 refers to GLE from grade 4
O
O
GLE from grade 3 (top) and grade 4 (bottom)
32
Development

• Extended standards and complexity levels were
  finalized in late September.
• Extended Standards Handbook (draft) disseminated
  to school districts in October.
• Assessment development began.

33
Alternate Assessments Grade Span Content Area

• ELA, Math, Science
• ELA, Math
• ELA, Math, Science
• ELA, Math
• ELA, Math
• Science

• 3 and 4
• 5 and 6
• 7 and 8
• High school
• 9
• 10
• 11

34
Alternate Assessments

• 25 performance tasks per content and grade span
• Each task administered and scored by the teacher
• Tasks scored on a 0-1 point or 0-2 point scale
  using a rubric

35
Development

• Alignment Study
• Alignment of performance tasks to the Extended
  Standards
• Panel of eight expert independent reviewers (4
  state reviewers/4 national reviewers)
• Based on Webbs Depth-of-Knowledge alignment
  process for use in aligning ES to performance
  tasks (2007)
• Level 1 Recall of Information
• Level 2 Basic Reasoning
• Level 3 Complex Reasoning

36
Challenges

• Time was the biggest challenge.
• Short time frame 8 months between first
  development activity (July, 07) and
  administration window for the assessment
  (FebruaryMarch, 08)
• Lack of adequate time for development activities
• Lack of preparation time for teachers to provide
  instruction and to prepare themselves and their
  students for the assessment
• Lack of time for adequate delivery of
  professional development to district personnel
• Lack of time to field test the items (2008 scores
  will not be included in state accountability
  results for schools and districts)

37
Challenges

• Other challenges
• Scheduling collaborative meetings with colleagues
  in Special Populations on a short timeframe
• Student IEPs completed based on former alternate
  assessment
• Necessary changes in Participation Criteria for
  AA
• Decisions regarding
• Test format
• Accommodations for students
• Manipulatives for tasks
• Assistive technology

38
Lessons Learned

• Adequate preparation time for students and
  teachers is a MUST!
• Adequate professional development is a MUST!
• The assessment is not appropriate for all
  students with significant cognitive disabilities
  (SCD).
• Teachers of students with SCD are not so familiar
  with standardized testing and need more training
  on administration procedures and security.
• Opinions from the field of the new assessment
  ranged from loved it to hated it? based
  primarily on their students abilities to access
  the test.
• Assistive technology personnel need to be
  included at the beginning of the development
  phase.

39
To End on a Positive

• Some Teacher Comments
• Im so pleased to have a curriculum (Extended
  Standards) to guide my instruction.
• The new test is easier to administer.
• A good test for content.
• IEP goals will have to be rewritten to address
  the new assessment.
• Great improvement.
• Great design!
• Pleasantly positively surprised.
• The test reflected what I teach.

40
National and StatePerspectives. And Next Steps

• Rachel F. Quenemoen, Senior Research Fellow, NCEO
• National Center on Educational Outcomes

41
NCEO STATE SURVEY REPORTS

• 2005 State Special Education Outcomes Steps
  Forward in a Decade of Change
• 2003 State Special Education Outcomes Marching
  On
• 2001 State Special Education Outcomes A Report
  on State Activities at the Beginning of a New
  Decade
• 1999 State Special Education Outcomes A Report
  on State Activities at the End of the Century
• Thompson Thurlow (1999, 2001, 2003)
• Thompson, Johnstone, Thurlow, Altman (2005)

42
Content Addressed by Alternate Assessments
Change Over Time
Year Fnct. skill No link Stnds Fnct. skill Link Stnds Stnds Plus Fnct. skills Exp/ ext St stnd Grade level stnd IEP team deter cntnt Other Revising
1999 16 --- 1 19 --- --- 24 ---
2000 9 3 7 28 --- --- 3 ---
2001 4 15 9 19 --- --- 3 ---
2003 2 --- 4 36 --- 3 3 2
2005 --- --- 1 21 10 1 7 10
Category possibly included grade level standards
prior to 2005 Category introduced in 2005
43
2005 - Outcomes Measured by Rubrics on
Alternate Assessments
Number of Regular States
(Numbers in parentheses from 2001)
44
Alternate Assessment Approaches 2000-2005 (from
2005 Survey)
Year Portfolio or Body of Evidence Rating Scale or Checklist IEP Analysis Other In Develop- ment/ Revision
Regular States Regular States Regular States
1999 28 (56) 4 (8) 5 (10) 6 (12) 7 (14)
2001 24 (48) 9 (18) 3 (6) 12 (24) 2 (4)
2003 23 (46) 15 (30) 4 (8) 5 (10) 3 (6)
2005 25 (50) 7(14) 2 (4) 7 (14) 8 (16)
Unique States Unique States Unique States Unique States Unique States Unique States
2003 4 (44) 0 (0) 1 (11) 1 (11) 3 (33)
2005 1 (11) 1 (11) 1 (11) 0 (0) 1 (11)
Of these 25 states, 13 use a standardized set
of performance/events/tasks/skills. Of these 7
states, three require the submission of student
work.
45
Flexibility and Standardization

• Nominal categories are NOT often useful for
  characterizing the technical aspects of the
  assessment (see Gong Marion, 2006).
• The evaluation of technical adequacy interacts
  with the types of alternate assessments (i.e.,
  choices/ degree of flexibility-standardization)
  being employed.
• This does NOT mean that standardization is good
  and flexibility is badit all depends on
  purposes!

46
Where Are We Now?

• Content coverage National Alternate Assessment
  Center work University of Kentucky Is it
  reading? Is it math? Is it science? University
  of North Carolina Links for Academic Learning
  other methodologies for alignment.
• Peer Review suggests great variability, near and
  far linkages, but a steady trend is toward
  academic content. Key questions is it
  measureable bits or big ideas and concepts or
  both? What is grade level for these students?
• Scoring criteria and procedures What does
  student performance look like? Student vs.
  system? How do we measure independence? Who
  scores? Who checks? Trust but verify? Flexibility
  vs. standardization issue.
• Peer Review suggests great variability on this.
  Concerns related to support, prompts,
  communications issues, academics interaction.

47
Where Are We Now?Part 2

• Performance/achievement descriptors and standard
  setting
• Is there achievement on the content?
• Is the content clearly referenced?
• How have accessibility issues been factored in?
• What does independence mean? How good is good
  enough?
• What should these students know and be able to
  do? How well? This needs careful monitoring over
  time, consequential validity studies.
• Approach
• Degree and logic of flexibility and
  standardization choices
• Nominal categories are not particularly useful
  descriptors.
• Unfortunately, the naked eye is drawn to test
  format not
• educational soundness (Baker, 2007)

48
More or Less Than Meets the Eye?

• BECAUSE of the number of uncertainties still in
  play, we need
• Transparency
• Integrity
• Consequential validity studies
• Planned improvement over time

49
Questions