Considerations for the Alternate Assessment based on Modified Achievement Standards

1
Considerations for the Alternate Assessment based
on Modified Achievement Standards

• Briefing on a white paper commissioned by the New
  York Comprehensive Center in collaboration with
  the New York State Education Department
• January 11, 2010
• The contents of this publication were developed
  under cooperative agreement S283B050019 with the
  U. S. Department of Education. However, the
  contents do not necessarily represent the policy
  of the Department of Education, and you should
  not assume endorsement by the Federal Government.

2
Introductions and Project Overview

• NYCC received a letter from USED informing them
  that funds were available for a research project
• NYSED agreed that an expert panel meeting
  culminating in a white paper on issues
  surrounding the AA-MAS would be useful
• NYCC applied for the funds and contracted with
  NCIEA to support this work

3
Issues Facing NY and Most States

• New regulation without much discussion with
  states led to many questions
• Should states spend limited resources to develop
  an AA-MAS?
• Will it yield useful information to guide
  instruction?
• Is that the best way to support the learning of
  this population?
• Would this test only be helpful to schools for
  accountability purposes?
• Are there current best practices with this
  population in assessment development that should
  be followed?
• What does the research say?

4
NYSED Initial Questions Regarding the AA-MAS

• Which students are best served by this
  assessment?
• How different are they from the rest of the
  special education population?
• What is an appropriately challenging
  achievement standard?
• Which modifications make the most sense in the
  context of the AA-MAS?
• How do the modifications affect the validity and
  reliability of the interpretation?
• What is the credential that is most appropriate
  for students participating in the AA-MAS and what
  does it lead to in terms of post-secondary
  potential?

5
Overview of Project

• NYCC contracted with the Center for Assessment to
  assemble an expert panel and develop a white
  paper on issues concerning the development of an
  AA-MAS
• NYSED provided guidance on important policy
  questions and recommended addressing issues in
  Filbin (2008) report
• Expert panel met twice in person and several
  times over WebEx
• Developed a 10-chapter report exploring many
  issues surrounding the conceptualization and
  development of an AA-MAS

6
Project Participants

• Project Director
• Larry Hirsch, New York Comprehensive Center
• Project Manager/Editor
• Marianne Perie, Center for Assessment
• New York State Department of Education
• David Abrams, Assistant Commissioner for
  Standards, Assessment, and Reporting
• Candy Shyer, Bureau Chief of Test Development,
  Office of State Assessment
• Rebecca Cort, Deputy Commissioner, Office of
  Vocational and Educational Services for Students
  with Disabilities

• Expert Panel
• Jamal Abedi, University of California, Davis
• Chris Domaleski, Center for Assessment
• Steve Dunbar, University of Iowa
• Howard Everson, Fordham University
• Claudia Flowers, University of North Carolina,
  Charlotte
• Brian Gong, Center for Assessment
• Meagan Karvonen, Western Carolina University
• Suzanne Lane, University of Pittsburgh
• Scott Marion, Center for Assessment
• Jim Pellegrino, University of Illinois, Chicago
• Marianne Perie, Center for Assessment
• David Pugalee, University of North Carolina,
  Charlotte
• Rachel Quenemoen, National Center on Educational
  Outcomes
• Robert Rickelman, University of North Carolina,
  Charlotte
• Katherine Ryan, University of Illinois, Urbana
  Champagne
• Gerald Tindal, University of Oregon
• Cathy Welch, University of Iowa
• Phoebe Winter, Pacific Metrics

7
Authors and Reviewers

• NYSED and the Center for Assessment identified
  chapter authors and reviewers and sent requests
  prior to the first expert panel meeting
• Eleven expert panelists served as authors
• The remaining seven experts served as reviewers
• Discussions were organized by section each
  reviewer was assigned a section
• Three reviewers read all chapters

8
Organizing Structure of Report

• Report is organized into three sections
• Identifying and understanding the population
• Test Design Understanding content and
  achievement standards and incorporating
  appropriate item modifications
• Technical considerations and practical
  applications
• Or
• Should I develop an AA-MAS?
• How should I develop it?
• How should I evaluate it?

9
Organization (continued)

• 10 chapters within the three sections
• Introduction
• Identifying Understanding the Population
• Identifying the Population
• Standards-based IEPs
• Cognition of Low Achievers
• Test Design
• Reading and Math Content
• Test Development
• Standard Setting
• Technical Considerations
• Comparability
• Validity
• Accountability and operationalization

10
Section I Identifying and Understanding the
Population

• Intro and first chapter provide a policy context
  for the AA-MAS and a historical perspective on
  educating students with disabilities
• Includes a discussion framework for state
  policymakers on assessment options and on
  improving student access to grade-level
  curriculum

11
Section I (continued)

• Second chapter describes how a standards-based
  IEP supports an educational program that meets
  the needs of AA-MAS-eligible students, through
• access to grade-level curriculum using effective
  instruction
• supports to address learner characteristics and
• setting and monitoring goals to support a
  transition into grade-level achievement.
• Third chapter describes relationships among
  assessment, curriculum, and instruction, and
  conceptualizing assessment as a process of
  reasoning from evidence driven by theories and
  data on student cognition, and includes examples
  related to the AA-MAS.

12
Section II Test Design

• Starts with a chapter describing content domains
  for ELA and Math with a focus on ways to modify
  content so that we can measure greater depth with
  easier items
• Next chapter discusses item and test development
  and reviews approaches to modification of items
  and assembly of test forms with respect to
  psychometric consequences and standards-based
  interpretations of proficiency for the AA-MAS
  population.

13
Section II (continued)

• Final chapter in this section focuses on
  developing modified achievement standards
• Includes a discussion on how to define modified
  proficiency, synthesizing information from the
  previous sections on cognition and test design,
  and provides ideas for setting cut scores on
  assessments with fewer items and lower sample
  sizes

14
Section III Technical Considerations

• First chapter in this section discusses
  comparability between AA-MAS and general
  assessments from different perspectives including
  
• content and construct,
• psychometrics,
• scale and score,
• linguistic structure,
• text features, and
• depth of knowledge.
• The next chapter describes why and how to develop
  a validity argument using Kanes argument-based
  approach as a framework for considering validity
  issues related to AA-MAS.

15
Section III (continued)

• The final chapter in this section describes
  operational and accountability issues and is
  specifically geared towards policymakers
• It focuses on the interrelationship of the AA-MAS
  to the existing state assessment and
  accountability system considering several
  practical, technical, and policy issues that must
  be considered when implementing a new program.

16
Other Features

• Other resources developed at the request of NYSED
• List of additional web-based resources on
  instructing and assessing students with
  disabilities
• Tool for policymakers that includes guiding
  questions for state policymakers considering the
  development of an AA-MAS and references back to
  specific sections within the report
• Glossary of terms related to special education,
  assessment, and federal policy

17
Overarching Themes

• Clear link between assessment, instruction, and
  student cognition,
• Several of the chapters focus on the importance
  of bringing lessons learned in studying the
  students and designing the assessments into the
  classroom.
• Similar to accommodations, certain modifications
  will only be successful to the degree they are
  incorporated into a students daily instruction.
• Important to develop a validity argument for this
  assessment early in the process and test the
  various assumptions throughout.
• Testing the assumption that a new assessment is
  needed is one of the first important
  recommendations.
• Many chapters address ongoing collection and
  evaluation of validity evidence to ensure that
  the development is in line with the expected
  goals.
• Consider how to incorporate the recommendations
  into an existing assessment and accountability
  system
• Including how to work with current state content
  standards and grade-level achievement standards.

18
Suggested Uses

• State policymakers should find this report useful
  in determining whether or not developing an
  AA-MAS makes sense for their state, and if so,
  how.
• Even in states with no intention of designing an
  AA-MAS, this report could be a useful tool for
  designing professional development activities.
• Contains suggestions for several areas special
  education, curriculum instruction, assessment,
  and accountability

19
Lingering Concerns

• Still concerned about identifying right
  population
• Kids are classified differently from one school
  to the next
• We need to understand the need for such
  classifications and find ways to better classify
  them into instructionally useful categories
• If AA-MAS modifications work well for these kids,
  why not expand the approach for all assessments
• Need more focus on universal design, better
  accommodations, and aligning assessments with
  diagnostic approaches to learning.

20
Lessons Learned from the Report

• This report is applicable to multiple tests and
  types of students not just the 2 population.
• We need to consider how the AA-MAS fits in with
  the general assessment.
• It highlighted the importance of an integrated
  test design, meaning we must consider assessment,
  curriculum, and instruction together in
  consideration of how tests will be used and
  validated.

21
Bigger Lessons Learned

• Key decision points should be identified and
  informed by research and best practices.
• There is a value in triangulating
  federally-funded research with states
  operational work
• States need time to have thoughtful discussion to
  process issues.
• This type of work increases professional
  development of state department staff.
• It helps inform several areas like RTTT and
  allows measurement research to inform policy.

22
Lessons Learned on Next Steps

• We need to strengthen our understanding of
  learning progressions and integrate that
  understanding into classroom instruction and
  assessment development.
• Where do students with disabilities differ from
  general population?
• Need to understand more than scope and sequence
• Should provide information on next steps to help
  strengthen nuances of student learning

23
Next Steps for New York

• Will not develop an AA-MAS under current
  political context
• Need to see what happens with ESEA
  reauthorization and RTTT
• In the meantime
• This work is helping us think through formative
  and interim assessment strategies.
• We are also working with our colleagues in
  special education to rethink IEP design.