Setting Alternate Achievement Standards

1
Setting Alternate Achievement Standards

• Prepared by Sue Rigney
• U.S. Department of Education
• NCEO Teleconference
• March 21, 2005

2
Regulation on alternate achievement standards
200.1

• For students with the most significant
  cognitive disabilities, who take an alternate
  assessment, a State may, through a documented and
  validated standards-setting process, define
  alternate achievement standards

3
Content v Achievement Standards
Genl Assmt AA-GLAS AA-AAS
Content Standards taught and assessed (access and alignment targets) Grade level Grade level Grade level, w/ extension or expansion to entry points
Achievement Standards Grade level Grade level Alternate level
Participating Students Most students, including those with disabilities, (with or w/o accommoda-tions) Students with disabilities who need alternate way(s) to show what they know Students with significant cognitive disabilities
4
Achievement standard
4 Components

• Label
• Descriptor
• Student work
• Cut score(s)

5
Alternate assessment based on alternate
achievement standards

• Should have
• Clearly defined structure
• Guidelines for which students may participate
• Clearly defined scoring criteria and procedures
• Report format that clearly communicates student
  performance in terms of the academic achievement
  standards defined by the State

6
This is a real assessment

• Based on academic content
• Structured to permit observation of student
  achievement of a clearly defined construct
• Scoring criteria are consistent with the
  construct
• Aggregation of data is consistent with the
  construct

7
Essentials of the standard setting process

• Define goals purpose to be served by the
  cutscore
• Select participants qualifications depend on the
  decisions to be made using the cutscore,
  familiarity with population tested
• Train participants start with practice
  feedback
• Define the performance standard
• Data collection procedures

Kane, ed Cizek pp.64-69
8
Standard setting options

• Test-centered
• Angoff - estimates difficulty of items for
  hypothetical examinees
• Bookmark - sets cut points within set of ordered
  item booklet
• Examinee-centered
• Contrasting groups - based on judgements about
  performance of examinees
• Body of work
• Profile (policy capturing)
• Different procedures yield different results

9
Bookmark method

• Arrange items in order of difficulty from easiest
  to hardest
• 67 answer this item correctly
• Descriptors provide general notion of levels
• Judges identify boundaries between performance
  levels
• Judges get feedback data and confirm cut points

10
Body of work method

• Judges examine student responses to a variety of
  tasks
• Collections of student work prepared for
• Training
• Range finding
• Pinpointing
• Data analysis

11
Generic Steps

• Select large, representative panel of judges
• Choose method, prepare agenda and training
  materials
• Prepare achievement level descriptors
• Train participants to use the method
• Compile item ratings or other judgments, prepare
  summary information as feedback

12
More steps

• Participants discuss initial summary information
• Another round of ratings, compile information and
  discuss as in 5 6
• Final opportunity to review information and
  arrive at final recommended achievement standards
• Evaluate standard-setting process
• Assemble documentation of process and other
  evidence appropriate for validity

13
Document the procedures

• Number of participants, how selected
• Qualifications of participants
• Qualifications of those designing methodology
• Materials used
• Instructions to participants
• Frameworks developed by participants
• Timeline, schedule of events, actual agenda

14
More documentation

• Any deviations from intended procedures
• Evidence of consistency/inconsistency of
  judgments
• Between rates
• Between rounds

15
Checklist for performance standards

• Understandable and useful for stakeholders
• Clearly differentiate among levels
• Grounded in student work but not tied to status
  quo
• Built by consensus
• Focused on learning

Handbook, p. 16
16
Peer Review looks for

• Formal adoption by Board?
• How linked with grade-level content?
• Involvement of diverse stakeholders?
• Alternate achievement standard available for each
  grade level?
• Documentation of process used?
• Results reported in terms of the alternate
  achievement standards?

17
Questions

• What should alternate standards look like?
  Grade-by-grade?
• How can group data reflect individual student
  growth?
• What if the AA-AAS doesnt meet commonly
  accepted professional standards? Will our state
  pass peer review?