Title: Redesign of Alternate Assessments for Students with Significant Disabilities: National and State Perspectives
1Redesign of Alternate Assessments for Students
with Significant Disabilities National and State
Perspectives
2Presentation Team
- Moderator Patty McDivitt, Data Recognition
Corporation - Aran Felix, Alaska Department of Education and
Early Development - Melissa Fincher, Georgia Department of Education
- Claudia Davis, Louisiana Department of Education
- Discussant Rachael Quenemoen, National Center
for Educational Outcomes
3Presentation Focus
- Purpose and Rationale
- Goals
- Development
- Challenges
- Lessons Learned
4Alaska
- Aran Felix
- Alaska Department of Education and Early
Development
5Purpose and Rationale
- Alaskas previous Alternate Assessment
- Portfolio assessment
- Housed in the Special Education Unit
- Assessed English/Language Arts, Math, and Skills
for Healthy Living (only E/LA and Math used for
AYP) - Intended use for grades 3, 6, 8, 11 only
- Stressed academic content
- Alternate Performance Standards (content
standards) - Used as Goals on student IEPs
- Stressed inclusion (Generalization dimension of
scoring) - Department of Education organized scoring
sessions - Single set of achievement standards
- Scoring dimensions Skill (student achievement),
Generalization, Appropriateness
6Purpose and Rationale
- Steps to explore a solution
- Alternate Assessment moved to Assessment Unit
- Teacher and Parent Survey conducted
- Reliability-Validity Study conducted
- Considered removing Generalization
Appropriateness dimensions from scoring for AYP
report only to districts. - Needed a new standard setting and better overall
technical quality - OSEP Condition drove development of a Data-folio
assessment for grades 4, 5, 7, 9, 10, Portfolio
for all grades overwhelming teachers. - Accomplished the original intent of inclusion and
teaching/assessing content standards - Moved scoring to test vendor
- Passed Peer Review requirements
7Goals
- Assess a broader range of content standards
- Manage test size and testing window
- Increase standardization
- content assessed (apples to apples)
- training of teachers to administer the assessment
- scoring of the assessment
- training of the protégés
- Establish a higher technical quality
- Create an online assessment (training, scoring,
reporting)
8Development
- 1) Funding Issues
- Small population of alternate-eligible students
- Request for Information
- 2) What type of assessment to develop?
- Research other states approaches
- Research using another states item bank
- Decision needed Retrofit portfolio with
performance tasks or move to a performance task
system?
9Development
- Look of new assessment
- Used another states item bank while examining
alignment the first year - Performance Tasks in content areas
- 4-8 items per task
- Paper/pencil materials allowing for some
adaptation - Assessors administer one-on-one to student
- Online test materials (scoring protocols, student
materials, training manuals) - Online training for test administrators plus
proficiency modules to maintain reliability - Online scoring, reporting, unofficial report
- Original 3-year plan compressed to 2 years (peer
review)
10Development
- Year 1 2005-2006 (Portfolio still in place)
- Developed Request for Proposals
- Contracted with new vendor
- Convened committees of educators
- Developed Proficiency level descriptors and
- Developed Extended Grade Level Expectations
- Piloted online system with technology
coordinators - Created security access levels
- Developed online security rules
- Trained Qualified Assessors for Pilot
- Pilot tested the new assessment
- Created a crosswalk of test items to Alaska
content standards (phase 1 blueprint)
11Development
- Year 2 2006-2007
- Developed cousin items
- Conducted bias review
- Conducted statewide trainings and orientations
- Developed a qualified assessor and qualified
mentor-trainer path and materials - Included mentor pre-test and debrief audios
- Administered assessment (one test)
- Piloted science assessment
- Conducted standard setting for RWM
- Received Technical Report
- Submitted new assessment to Peer Review (and
received approval) - AYP and student reports to parents/districts/websi
te
12Development
- Year 3 2007-2008
- Developed cousin items and conducted bias review
- Repackage all content tests into grade clusters
vs. grade level within content areas - New look for website
- Trained Mentors on science assessment
- Administered all assessments including science
- Teacher Survey of Consequential Validity, Mentor
Audios - Conducted standard setting for science
- Conducted standard validation for
- Received Technical Report
- Submitted new assessment to Peer Review (and
received approval) - Piloted science assessment
- AYP and student reports to parents/districts/websi
te
13Development
- Future Plans for years 2008-2011
- Develop larger item bank, conduct reviews
- Field test new items
- Construct new operational test forms, A B
- Conduct a new Standard Setting on Forms A and B
of each content-area assessment - Consider a program review by external evaluator
14Challenges
- Compressing the schedule from 3-4 years to 2
years - Possibility of administering two Alternates
simultaneously - Using another states items
- Alaskanizing the assessment
- Moving from a one-size model to a grade-cluster
model
15Lessons Learned
- The importance of flexibility and responsiveness
- Understanding capacity when huge demands (such as
compressed time schedule) are imposed by second
peer review deadlines - The role of an outside consultant as EED advisor
for the first years of the project - Involvement of TAC prior to issuing RFP
16Recommendations
- RFP get help from other states or hire a
consultant include any customizing details
require experienced project manager include
right to own test items - Link pattern of test development to general
education cycle and include content experts in
all aspects of test development (standards to
test items) - Communicate with the field regularly through the
process of developing extended content standards.
Provide these standards to the field early. - If using any online pieces conduct pilot of
online system to explore platform issues and get
sign-off from someone in charge at district level - Exercise caution when updating online system
during test window - Include an errata page on online system for
notifying field of updates - Have a backup server if you have online
components - Remain cheerful!
17Georgia
- Melissa Fincher
- Georgia Department of Education
18Purpose and Rationale
- Why did Georgia redesign our alternate?
- In a word COMPLIANCE
- In a few words OPPORTUNITY TO LEARN
19Goals
- The GAA is designed to ensure that students with
significant cognitive disabilities are - Provided access to the state-mandated curriculum.
- Given the opportunity to demonstrate progress
toward achievement of curriculum knowledge,
concepts, and skills.
20Development
- The GAA is a portfolio of student work provided
as evidence that a student is making progress
toward grade-level academic standards. - Evidence provided must show instructional
activities and student work that is aligned to
specific grade-level standards.
21Development
- Georgia elected to go with a portfolio format to
allow flexibility for a very diverse group of
students. - The portfolio format allows the teacher, who
knows the student best, to design and document
instructional tasks that are meaningful and
purposeful for the individual student.
22Development
- Collaboration between Assessment, Special
Education, and Curriculum - Significant investment in training of educators
surrounding curriculum access - Ongoing documentation of decisions surrounding
development and implementation - Involvement of Georgias Technical Advisory
Committee - Augmented with an AA-AAS expert
23Challenges
- Documenting technical quality
- Traditional indices do not lend themselves easily
to alternate assessments - Federal Peer Review
- Complicated assessment program
- Teacher buy-in
24Lessons Learned
- Investing in teacher training pays off, but it
needs to be continual as the assessment evolves. - Technical documentation of alternate assessments
looks different, but it is worth pursuing. - Validity is an ongoing journey.
- Students are doing things we never thought
possible.
25Louisiana
- Claudia Davis
- Louisiana Department of Education
26Purpose and Rationale
- Purpose Louisianas alternate assessment did not
receive approval in the USDOE peer review
process.
27Goals
- Louisianas alternate assessment (LEAP Alternate
Assessment or LAA) lacked - Academic focus (although linked to state
standards) - Alignment with grades or grade spans
- USDOE advised a redesign of the LAA to be
implemented in spring of 2008 - LDE began the redesign in July of 2007
28Development
- First, Louisiana developed Extended Standards
(ES), i.e., extensions of state content
standards. - What should students with significant cognitive
disabilities know and be able to do? - How do we identify expectations regarding the
breadth and depth of the standards, benchmarks,
and grade-level expectations (GLEs)? - How should students demonstrate knowledge and
skills based on the GLEs? - What is the appropriate range of tasks to be used
to measure this knowledge and these skills? -
29Development
- LDE contracted with the testing vendor to develop
Extended Standards for ELA, Mathematics, and
Science. - Recommendations from an initial committee of
state special educators regarding selected
standards and a template for the ESs, including
Complexity Levels, were used to guide the
development. - Committees of Louisiana educators (general and
special educators) reviewed the proposed ESs with
Complexity Levels. - Content-area groups across four grade spans
(grades 3-4, 5-6, 7-8, and 9-11) - Review was time intensive -- one intensive week
-
30Development
- Complexity Levels of the Extended Standards
- Each ES has 3 levels of complexity (1 being least
difficult) that provide access to general
education concepts and skills. - They serve as guidelines for the development of
assessment tasks at 3 levels of complexity. - Math Example
- 3. Add and/or subtract to solve simple problems.
- 2. Identify simple addition and subtraction
concepts within daily living
problems. - 1. Count to solve simple problems.
-
31 SAMPLE PAGE AND KEY FOR ENGLISH LANGUAGE ARTS SAMPLE PAGE AND KEY FOR ENGLISH LANGUAGE ARTS SAMPLE PAGE AND KEY FOR ENGLISH LANGUAGE ARTS SAMPLE PAGE AND KEY FOR ENGLISH LANGUAGE ARTS
Standard One Students read, comprehend, and respond to a range of materials, using a variety of strategies for different purposes. Standard One Students read, comprehend, and respond to a range of materials, using a variety of strategies for different purposes. Standard One Students read, comprehend, and respond to a range of materials, using a variety of strategies for different purposes. Standard One Students read, comprehend, and respond to a range of materials, using a variety of strategies for different purposes.
Benchmarks Grade-Level Expectations Extended Standards Complexity Levels
ELA-1-E4 recognizing story elements (e.g., setting, plot, character, theme) and literary devices (e.g., simile, dialogue, personification) within a selection 8. Identify story elements, including theme conflict character traits, feelings, and motivation (ELA-1-E4) 5. Identify a variety of story elements, including the impact of setting on character multiple conflicts first- and third-person points of view development of theme (ELA-1-E4) ES-8-5 Identify story elements, including character 3. Identify the main character in a story 2. Identify two characters in a story 1. Identify one character in a story
3 is most complex
Extended Standard ES-8 /5 refers to GLE 8 from
grade 3 ES-8/ 5 refers to GLE from grade 4
O
O
GLE from grade 3 (top) and grade 4 (bottom)
32Development
- Extended standards and complexity levels were
finalized in late September. - Extended Standards Handbook (draft) disseminated
to school districts in October. - Assessment development began.
33 Alternate Assessments Grade Span Content Area
- ELA, Math, Science
- ELA, Math
- ELA, Math, Science
- ELA, Math
- ELA, Math
- Science
- 3 and 4
- 5 and 6
- 7 and 8
- High school
- 9
- 10
- 11
34Alternate Assessments
- 25 performance tasks per content and grade span
- Each task administered and scored by the teacher
- Tasks scored on a 0-1 point or 0-2 point scale
using a rubric
35Development
- Alignment Study
- Alignment of performance tasks to the Extended
Standards - Panel of eight expert independent reviewers (4
state reviewers/4 national reviewers) - Based on Webbs Depth-of-Knowledge alignment
process for use in aligning ES to performance
tasks (2007) - Level 1 Recall of Information
- Level 2 Basic Reasoning
- Level 3 Complex Reasoning
36Challenges
- Time was the biggest challenge.
- Short time frame 8 months between first
development activity (July, 07) and
administration window for the assessment
(FebruaryMarch, 08) - Lack of adequate time for development activities
- Lack of preparation time for teachers to provide
instruction and to prepare themselves and their
students for the assessment - Lack of time for adequate delivery of
professional development to district personnel - Lack of time to field test the items (2008 scores
will not be included in state accountability
results for schools and districts)
37Challenges
- Other challenges
- Scheduling collaborative meetings with colleagues
in Special Populations on a short timeframe - Student IEPs completed based on former alternate
assessment - Necessary changes in Participation Criteria for
AA - Decisions regarding
- Test format
- Accommodations for students
- Manipulatives for tasks
- Assistive technology
38Lessons Learned
- Adequate preparation time for students and
teachers is a MUST! - Adequate professional development is a MUST!
- The assessment is not appropriate for all
students with significant cognitive disabilities
(SCD). - Teachers of students with SCD are not so familiar
with standardized testing and need more training
on administration procedures and security. - Opinions from the field of the new assessment
ranged from loved it to hated it? based
primarily on their students abilities to access
the test. - Assistive technology personnel need to be
included at the beginning of the development
phase.
39To End on a Positive
- Some Teacher Comments
- Im so pleased to have a curriculum (Extended
Standards) to guide my instruction. - The new test is easier to administer.
- A good test for content.
- IEP goals will have to be rewritten to address
the new assessment. - Great improvement.
- Great design!
- Pleasantly positively surprised.
- The test reflected what I teach.
40National and StatePerspectives. And Next Steps
- Rachel F. Quenemoen, Senior Research Fellow, NCEO
- National Center on Educational Outcomes
41NCEO STATE SURVEY REPORTS
- 2005 State Special Education Outcomes Steps
Forward in a Decade of Change - 2003 State Special Education Outcomes Marching
On - 2001 State Special Education Outcomes A Report
on State Activities at the Beginning of a New
Decade - 1999 State Special Education Outcomes A Report
on State Activities at the End of the Century - Thompson Thurlow (1999, 2001, 2003)
- Thompson, Johnstone, Thurlow, Altman (2005)
42Content Addressed by Alternate Assessments
Change Over Time
Year Fnct. skill No link Stnds Fnct. skill Link Stnds Stnds Plus Fnct. skills Exp/ ext St stnd Grade level stnd IEP team deter cntnt Other Revising
1999 16 --- 1 19 --- --- 24 ---
2000 9 3 7 28 --- --- 3 ---
2001 4 15 9 19 --- --- 3 ---
2003 2 --- 4 36 --- 3 3 2
2005 --- --- 1 21 10 1 7 10
Category possibly included grade level standards
prior to 2005 Category introduced in 2005
432005 - Outcomes Measured by Rubrics on
Alternate Assessments
Number of Regular States
(Numbers in parentheses from 2001)
44Alternate Assessment Approaches 2000-2005 (from
2005 Survey)
Year Portfolio or Body of Evidence Rating Scale or Checklist IEP Analysis Other In Develop- ment/ Revision
Regular States Regular States Regular States
1999 28 (56) 4 (8) 5 (10) 6 (12) 7 (14)
2001 24 (48) 9 (18) 3 (6) 12 (24) 2 (4)
2003 23 (46) 15 (30) 4 (8) 5 (10) 3 (6)
2005 25 (50) 7(14) 2 (4) 7 (14) 8 (16)
Unique States Unique States Unique States Unique States Unique States Unique States
2003 4 (44) 0 (0) 1 (11) 1 (11) 3 (33)
2005 1 (11) 1 (11) 1 (11) 0 (0) 1 (11)
Of these 25 states, 13 use a standardized set
of performance/events/tasks/skills. Of these 7
states, three require the submission of student
work.
45Flexibility and Standardization
- Nominal categories are NOT often useful for
characterizing the technical aspects of the
assessment (see Gong Marion, 2006). - The evaluation of technical adequacy interacts
with the types of alternate assessments (i.e.,
choices/ degree of flexibility-standardization)
being employed. - This does NOT mean that standardization is good
and flexibility is badit all depends on
purposes!
46Where Are We Now?
- Content coverage National Alternate Assessment
Center work University of Kentucky Is it
reading? Is it math? Is it science? University
of North Carolina Links for Academic Learning
other methodologies for alignment. - Peer Review suggests great variability, near and
far linkages, but a steady trend is toward
academic content. Key questions is it
measureable bits or big ideas and concepts or
both? What is grade level for these students? - Scoring criteria and procedures What does
student performance look like? Student vs.
system? How do we measure independence? Who
scores? Who checks? Trust but verify? Flexibility
vs. standardization issue. - Peer Review suggests great variability on this.
Concerns related to support, prompts,
communications issues, academics interaction.
47Where Are We Now?Part 2
- Performance/achievement descriptors and standard
setting - Is there achievement on the content?
- Is the content clearly referenced?
- How have accessibility issues been factored in?
- What does independence mean? How good is good
enough? - What should these students know and be able to
do? How well? This needs careful monitoring over
time, consequential validity studies. - Approach
- Degree and logic of flexibility and
standardization choices - Nominal categories are not particularly useful
descriptors. - Unfortunately, the naked eye is drawn to test
format not - educational soundness (Baker, 2007)
48More or Less Than Meets the Eye?
- BECAUSE of the number of uncertainties still in
play, we need - Transparency
- Integrity
- Consequential validity studies
- Planned improvement over time
49Questions