Welcome to the Race to the Top Assessment Program Technical Assistance Public Meeting

1
Welcome to the Race to the Top Assessment
ProgramTechnical Assistance Public Meeting

• Creating Valid, Reliable, and Fair Assessments
  for Students with Disabilities English Learners
• Washington, DC
• August 10, 2011
• Please silence all cell phones and pagers.
• Thank you!

2
Race to the Top Assessment (RTTA) Program
Overview and Meeting Goals

• Joe Conaty
• Patrick Rooney
• U.S. Department of Education

3
RTTA Public Meetings

• This is the third in a series of public meetings
  on RTTA.
• Two prior meetings April 15 on State and Local
  Technology Infrastructure and June 10 on
  Automated Scoring of Assessments
• Details on additional meetings will be
  forthcoming
• Purpose of the meetings
• To provide technical assistance to and to support
  collaborative efforts of PARCC and SBAC as they
  develop new assessment systems
• To expand the knowledge and expertise of the
  Department and the public around key assessment
  issues
• To facilitate discussion of key components of the
  systems with experts and the public at large
• Funded in part by The William and Flora Hewlett
  Foundation

4
RTTA Program Goals

• Support states in delivering a system of more
  effective and instructionally useful assessments
  that
• Provide accurate information about what students
  know and can do by
• Eliciting complex student demonstrations or
  applications of knowledge and skills, as
  appropriate
• Accurately measuring student achievement across
  the full performance continuum
• Accurately measuring student growth over a full
  academic year or course
• Helping educators determine whether individual
  students are ready for college and careers by the
  time of high school graduation and, in previous
  grade levels, whether they are on-track for
  readiness
• Reflect good instructional practice and support a
  culture of continuous improvement
• Effectively assess all students, including
  students with disabilities and English learners

5
Looking Forward

• Assessment systems must include one or more
  summative assessment components that are fully
  implemented by every state in each consortium by
  SY 2014-15, and are administered at least once
  during the academic year in, at a minimum
• Reading/language arts and mathematics
• Grades 3-8 and high school
• Results used to inform
• Teaching, learning, and program improvement
• Determinations of school effectiveness
• Determinations of principal and teacher
  effectiveness for the purposes of evaluation and
  support
• Determinations of individual student college and
  career readiness

6
RTTA Grantees

• Nearly 360 million awarded in September 2010 to
  two consortia, which together represent 45 states
  and DC
• Partnership for Assessment of Readiness of
  College and Careers (PARCC)
• Project Management Partner Achieve
• SMARTER Balanced Assessment Consortium (SBAC)
• Project Management Partner WestEd
• PARCC and SBAC have demonstrated commitment from
  institutions of higher education (IHEs) in member
  states that they will use the results of these
  assessments to determine entry into
  credit-bearing courses
• The IHEs represent 90 (PARCC) and 74 (SBAC) of
  students who matriculate directly from K-12

7
Students with Disabilities English Learners

• The absolute priority required the consortia to
  create assessments for all students, including
  English learners and students with disabilities
• The consortia are required to develop tests
  accessible for these populations and to create
  and standardize accommodations policies
• Each consortium must develop a definition of
  English learner that is uniform across member
  states
• Additional consortia
• Alternate assessments for students with the most
  significant cognitive disabilities
• English language proficiency

8
Expectations for the Meeting

• We have invited a range of experts to this
  meeting to share their knowledge and experience
  with the consortia members, looking at both
  current state of research and promising
  approaches to improving accessibility for these
  students
• Format
• The morning will focus on key questions that need
  to be addressed regarding the needs of students
  with disabilities and English learners in the
  assessment system and possible methods to address
  those questions
• The afternoon will focus on two standards one
  in English language arts and one in mathematics
  in a practical application of the issues to
  creating valid, reliable, and fair assessment
  items for these populations

9
Meeting Agenda

• 900-935 Welcome/setting the stage
• 935-1015 Fishbowl discussion
• 1015-1030 Break
• 1030-Noon Fishbowl discussion continued
• Noon-100 Lunch
• 100-130 Fishbowl discussion of public
  comments
• 130-300 Table exercise
• 300-315 Public comments
• 315-330 Wrap-up
• 330 Adjourn

10
Invited Experts

• Jamal Abedi, University of California, Davis
• Lizanne DeStefano, University of Illinois
• Rebecca Kopriva, Wisconsin Center for Educational
  Research
• Mike Russell, Measured Progress
• Stephen Sireci, University of Massachusetts,
  Amherst
• Guillermo Solano-Flores, University of Colorado,
  Boulder

11
Public Comments

• ED wants to hear from the public on key
  considerations for creating valid, reliable, and
  fair assessments for students with disabilities
  and English learners
• In the morning
• Comment cards are available at the registration
  desk
• ED, the consortia, and the experts will discuss
  the comments/questions as time allows at the
  start of the afternoon session
• All input from comment cards will be posted on
  our website
• In the afternoon
• We have scheduled time for verbal public comment
  from 300-315 pm
• Sign up to speak during the lunch break at the
  registration desk
• Time limit Up to 3 minutes per
  person/organization
• Due to limited time, those not able to provide
  comments in person may email them to
  racetothetop.assessment_at_ed.gov

12
Reminders

• Please place all cell phones and other devices on
  vibrate
• Race to the Top Assessment resources
  Applications, FAQs, plus todays materials and
  transcription available at
• http//www2.ed.gov/programs/racetothetop-assessmen
  t
• The purpose of this event is to promote a full
  discussion and hear a wide range of viewpoints on
  creating valid, reliable, and fair assessments
  for English learners and students with
  disabilities, as well as the challenges and
  opportunities afforded by the Race to the Top
  Assessment program. Through this meeting, the
  U.S. Department of Education is not seeking to
  promote and/or endorse any particular program,
  project, methodology or approach to this work.

13
INTRODUCTIONS

• Patrick Rooney
• U.S. Department of Education
• Meeting Facilitator

14

• Accessibility and Accommodations

Deborah Matthews - Kansas Accessibility
Accommodations Workgroup dmatthews_at_ksde.org
15
The Purpose of the Consortium

• To develop a set of comprehensive and innovative
  assessments for grades 3-8 and high school in
  English language arts and mathematics aligned to
  the Common Core State Standards
• Students leave high school prepared for
  postsecondary success in college or a career
  through increased student learning and improved
  teaching
• The assessments shall be operational across
  Consortium states in the 2014-15 school year

16
29 Member States
17
Work Groups

• Transition to Common Core State Standards
• Technology Approach
• Assessment Design Item Development
• Assessment Design Performance Tasks
• Assessment Design Test Design
• Assessment Design Test Administration
• Reporting
• Formative Processes and Tools/Professional
  Development
• Accessibility and Accommodations
• Research and Evaluation

18
Accessibility And Accommodations Workgroup

• Purpose
• Ensure the SBAC Assessment System is maximally
  accessible to the broadest range of students
  through
• identifying, recommending, and evaluating
  strategies, tools, and technologies, thereby
• providing information and guidance that will
  positively impact critical aspects of assessment
  design and development

19
Accessibility and Accommodations Workgroup

• New paradigm that focuses on the student first,
  not the test items which addresses accessibility
  issues as part of item development, not as an
  afterthought.
• Computer based assessment allows technology to
  open many doors for students because
  accessibility is built into the assessments.
• The necessity of accommodations is reduced.
  Accommodations that are allowed are more
  targeted.

20
Accessibility and Accommodations Workgroup

• In both policy and practice, SBAC will
• include the broadest range of students
• by facilitating each students ability to
  demonstrate as fully as possible what they know
  and can do
• on the targeted constructs being measured
• in a manner that is equitable and reliable, and
  yields valid interpretations of results.

21
Five Goals Key Activities

• Create policies that reflect current research,
  best practices, and future possibilities related
  to accessibility and accommodations
• Create assessments that are free from bias and
  sensitivity issues leveraging new technologies,
  including interoperability while preserving test
  constructs
• Create accessible and accommodated assessments
  that will yield valid and reliable results

22
Five Goals Key Activities

• Ensure accessibility and accommodations practice
  and policy are implemented with fidelity
• Develop useful reporting and presentation
  guidelines that include information on
  accessibility and accommodations actions in the
  aggregate and at the individual student level

23
SBAC Representativeswww.smarterbalanced.org

• Michael Hock Vermont Accessibility
  Accommodations Workgroup Co-Chair
• Michael.Hock_at_state.vt.us
• Wendy Carver - Utah Accessibility
  Accommodations Workgroup
• Wendy.Carver_at_schools.utah.gov
• Shelbi Cole Connecticut Performance Tasks
  Workgroup
• Shelbi.Cole_at_ct.gov
• Gaye Fedorchak New Hampshire Accessibility
  Accommodations Workgroup
• gfedorchak_at_ed.state.nh.us
• Viji Somasundaram Wisconsin Item Development
  Workgroup
• visalakshi.somasundaram_at_dpi.wi.gov

24

• Accessibility for Students
• August 10, 2011
• www.PARCConline.org

25
The PARCC Vision

• Create high-quality assessments that measure the
  full range of the Common Core State Standards
• Build a pathway to college and career readiness
  for all students and make accurate and reliable
  determinations as to whether students are
  on track or ready for college and
  careers
• Provide information that supports various
  accountability uses (e.g., school, educator,
  student)
• Provide timely and actionable information that
  supports continuous improvements in curriculum
  and instruction that inform effective classroom
  instruction and assessment practices
• Leverage technology for a variety of uses
  innovative items, accommodations, administration,
  and scoring and reporting.
• Report results that allow for comparability
  across all PARCC states, across consortia, and to
  national and international assessments.

26
PARCC Accessibility Goals

• The Partnership will
• Work to minimize/eliminate features that are
  irrelevant to what is being measured and measure
  the range of complexity of the standards so that
  students can demonstrate their knowledge
• Design each component in a manner that allows ELL
  students and students with identified needs to
  demonstrate what they know and can do
• Apply principles of universal design for
  accessible assessments throughout every stage of
  developing assessment components, items, and
  performance tasks
• Leverage technology for delivering assessment
  components as widely accessible as possible and
• Establish a Committee on Accessibility and
  Accommodations comprised of knowledgeable testing
  officials from member states (OWG).

27
PARCC Governance Structure
Steering Committee
28
PARCC Technical Working Groups (TWG)

• Limited number of groups convened by the TAC to
  address high priority topics that would benefit
  from collective problem-solving by leading
  experts
• Comprised of domain-specific technical advisors
  who interact with leadership and working groups
  and report to the TAC
• Accessibility, Accommodations, and Fairness TWG
• Committee members represent a range of expertise
  in accessibility and accommodations
• Role is to help guide the efforts of working
  groups in designing accessible assessments that
  remain true to the intended constructs

29
Accessibility, Accommodations, and Fairness TWG
Invited Members

• Diane August
• Center for Applied Linguistics (ELL)
• David Edyburn
• University of Wisconsin-Milwaukee (SWD)
• Claudia Flowers
• University of North Carolina Charlotte (SWD)
• Dianne Piche
• Leadership Conference on Civil Rights

• Charlene Rivera
• George Washington University (ELL)
• Diane Spence
• Region 4 Education Service Center, Braille
  Services (Braille)
• Martha Thurlow
• National Center on Educational Outcomes (SWD)
• Dan Wiener, Chair
• Massachusetts Department of Elementary and
  Secondary Education (accommodations for state
  assessments)

• Gerunda Hughes will serve as the liaison to the
  PARCC Technical Advisory Committee (TAC).

30
PARCC Operational Working Groups (OWG)

• Who Comprised of state representatives, Achieve
  staff members, and eventually vendor
  representatives
• What Responsible for the day-to-day aspects of
  work of key components of work
• Why To ensure efficient and effective
  collaboration among PARCC members to meet PARCC
  goals.

31
Accessibility, Accommodations, and Fairness OWG
Members

• Roberta Alley (Chair / Leadership Team)
• Arizona Department of Education
• Trinell Bowman
• Maryland State Board of Education
• Mira Monroe
• Colorado Department of Education
• Melissa Fincher (Leadership Team)
• Georgia Department of Education
• Charity Flores
• Indiana Department of Education
• Andrew Hinkle
• Ohio Department of Education
• Leila Williams
• Arizona Department of Education

• Bambi Lockman
• Florida Department of Education
• Phyllis Lynch
• Rhode Island Department of Education
•  Michael Reid
• Oklahoma State Department of Education
• Lori Rodriguez
• Florida Department of Education
• Dan Wiener
• Massachusetts Department of Elementary and
• Secondary Education
• Jessica Tickle
• Achieve/PARCC
• Danielle Griswold
• Achieve/PARCC

32
Accessibility and AccommodationsWorking Groups

• The Working Groups will be responsible for
• Drafting a set of Partnership-wide policies in a
  Partnership Accommodations Manual to be adopted
  by each member state for identifying eligible
  students, selecting allowable accommodations, and
  administering accommodations. That process will
  include
• Analyzing extant state accommodation policies,
• Building a list of recommended standard
  accommodations,
• Identifying constructs and research
  accommodations,
• Recommending a set of proposed accommodation
  policies for the assessment,
• Drafting a common Partnership Accommodations
  Manual,
• Ensuring comparability in assessment
  administrations,
• Monitoring ongoing refinements of accommodations,
  and
• Developing training modules for IEP teams.

33
Accessibility and AccommodationsWorking Groups

• Adopting key policies and definitions that will
  include
• a common definition of English Learner
• a common set of policies and procedures for
  providing assessment accommodations for English
  learners and students with identified needs and
• a common set of policies and procedures for
  participation of English learners and students
  with identified needs in the assessment system.

34
Accessibility and AccommodationsWorking Groups

• Accessibility and Accommodations as a part of the
  development process
• Design review and feedback
• Test blueprint development
• Technology development and selection
• Passage and media review committee involvement
• Item review committee involvement
• Bias and sensitivity committee involvement
• Testing the efficacy of assessment items with
  accommodations with the intended groups of
  students in pilot and field testing
• Including sufficient number of students with
  identified needs (across sub-categories) in pilot
  and field testing
• Data review committee involvement

35
Accessibility and AccommodationsWorking Groups

• Build accessibility throughout the test itself
  with no trade-off between accessibility and
  validity
• Use a combination of accessible-authoring and
  accessible technologies from the inception of
  items and tasks
• Establish and maintain a close working connection
  with the Technology, Design, and Research Working
  Groups

36
UNDERSTANDING THE POPULATION

• Patrick Rooney
• U.S. Department of Education

37
Who are students with disabilities?
Source U.S. Department of Education, SY 2008-09
Annual Performance Reports. Figure courtesy of
the National Center on Educational Outcomes
38
Understanding the Population

• In 2008-09, some 6.5 million children ages 3-21
  received special education services (13 percent
  of the population)
• 95 percent were enrolled in regular public
  schools
• 57 percent spent most of their time in general
  classes
• The vast majority of students with disabilities
  take the general reading/language arts and
  mathematics assessments
• Students with significant cognitive disabilities
  Current law permits up to 1 percent of all
  students in the state (approximately 10 percent
  of students with disabilities) to take an
  alternate assessment based on alternate academic
  achievement standards
• All students are expected to have access to, and
  be assessed against, grade-level content standards

39
Trends in Growth of English Learners
Percentage of children ages 5-17 who spoke a
language other than English at home and
percentage who spoke a language other than
English at home and spoke English with
difficulty Selected years, 1980-2009
Source Census Bureau NCES Condition of
Education 2011
40
Where are English learners?
Percentage of children ages 5-17 who spoke a
language other than English at home and spoke
English with difficulty, by state or
jurisdiction 2009
Source Census Bureau NCES Condition of
Education 2011
41
Understanding the Population

• Over 300 different languages are spoken by
  students.
• Top 5 languages spoken by English learners
• Spanish (3.5 million)
• Vietnamese (93,000)
• Chinese (80,000)
• Arabic (71,000)
• Hmong (50,000)
• In 2000, some 64 percent of English learners were
  born in the United States
• 42 percent were 2nd generation
• 22 percent were 3rd generation
• The HHS Office of Refugee Resettlement identified
  18,500 refugee children ages 6-18 in 2010.

42
UNDERSTANDING THE POPULATION

• What are the challenges or key questions that
  need to be addressed by the consortia when
  developing their assessment systems?

43
BREAK

44

• Accessible Assessment APIP

• A Brief Overview

• Michael Russell

45

• Accessibility Assessment

• A Two-Way Street

46

• Accessibility Assessment

• A Two-Way Street

47

• Digital Assessment Delivery

What we can already do

• Match Content Form to Student Access Need
• Language
• Translation (Directions, Whole Item, Individual
  Words/Phrases)
• Simplified English
• Audio
• Text-to-Speech, Pre-recorded Voice
• Text-based content, Graphics Tables, Non-Visual
  Descriptions
• Braille
• Sign

48

• Digital Assessment Delivery

• What we can already do

• Match Interaction Response Modes to Need
• Alternate keyboards
• Tab-Enter control devices
• Touch screens
• Eye gaze
• Speech-to-text

49

• Current Challenges

• Interoperable Accessibility

• Standard Coding for Student Access
  Needs/Accommodations
• Standard Tagging System for Item Content
  Accessibility Information
• Standard File Exchange Format for Student Access
  Needs and Accessible Test Items

50

• Accessible Portable Item Profile Standard

• What is APIP?

File Exchange Format
51

• Access Needs Addressed by APIP

• Language-Related

• Translated Item
• Translated Words/Phrases
• Translated Directions
• Simplified Language
• Audio Representation
• Symbolic Representation
• Extended Time/Breaks

52

• Access Needs Addressed by APIP

• Visual

• Magnification
• Reverse Contrast
• Alternate Fore/Background Colors
• Color Tinting/Overlay
• Increased White Space

53

• Access Needs Addressed by APIP

• Executive Function/Maintaining Attention

• Auditory Calming
• Masking
• Line Reader
• Breaks
• Extended Time

54

• Access Needs Addressed by APIP

• Information Processing

• Flagging
• Keyword Highlight
• Alternate Representations
• Scaffolding
• Chunking
• Reduced Answer Options
• Negatives Removed

55

• Access Needs Addressed by APIP

• Representational Form

• Audio text-only, graphic-only, text graphic,
  non-visual
• Tactile
• Braille
• Sign (ASL, Signed English)

56

• Standards in the Education Sector

• Each Standard Addresses A Distinct Need

• Content Standards - CCSS
• Performance Standards Consortia Assessments
• Data Standards SIF, CEDS, Ed-Fi
• Interoperability Standards QTI-APIP

57

• Standards in the Education Sector

• How They Work Together in an Assessment Context

Common Core State Standards
58

• APIP

• Myths and Misunderstandings

• Proprietary/Industry Led Standard
• Competes/Clashes with SIF
• Does not support Innovative Items
• Requires High Bandwidth

59

• APIP

• Open Discussion

• Questions Comments

60
DISCUSSION

• What challenges and benefits does
  computer-administered testing create for
  accessibility?

61
What is the current state of research on
accommodationsand what future research is
necessary?ELL Students

• U.S. Department of Education
• RTTA Public Meeting on
• Creating Valid, Reliable, and Fair Assessments
  for Students with Disabilities and English
  Learners
• Jamal Abedi
• University of California, Davis
• August 10, 2011

62
What we need to know about accommodations before
using them for ELLs and SWDs

1. Effectiveness How effective are accommodations
   in making assessments more accessible to ELL
   students?
2. Validity How valid is the outcome of the
   accommodated assessment when compared to a
   non-accommodated assessment?
3. Differential Impact To what degree are these
   accommodations universally applicable to ELL
   students with different background
   characteristics?
4. Comparability Can accommodated and
   non-accommodated assessment outcomes be
   aggregated?
5. Relevance How appropriate are the accommodations
   used for these students?
6. Feasibility How feasible is it to implement
   these accommodations in large-scale assessments?

63
What we know based on current research

• Some accommodations may not be effective in
  making assessments more accessible to ELLs (e.g.
  one-on-one and small group testing)
  EFFECTIVENESS RELEVANCE
• Some accommodations may alter the construct by
  providing an unfair advantage to the recipients
  (e.g. dictionary or glossary with content-related
  terms) VALIDITY COMPARABILITY
• Some accommodations do no alter the focal
  construct and ensure comparability (e.g.
  linguistic modification) VALIDITY
• However we have no hard evidence on the VALIDITY
  and EFFECTIVENESS for the majority of
  accommodations currently in use.

64
What research needs to focus on next

• Research needs to
• Examine the validity of accommodations
• i.e. if accommodations impact the focal construct
• Determine the effectiveness of accommodations in
  making assessments more accessible to ELLs and
  SWDs
• What does this look like?
• Randomized Field Experiment in which all major
  sources of threat to internal and external
  validity of the experiment are controlled
• Existing data on accommodations may not be a good
  source for examining either the validity or
  effectiveness of accommodations

65
How to test the validity and effectiveness of an
accommodation in an experimentally controlled
field study
ELL Status/Accommodation Accommodated Non-Accommodated
ELL G1 G2
Non-ELL G3 G4
Effectiveness Comparing G1 with G2 Validity
Comparing G3 with G4 The main reason for
inconsistencies between existing research on
accommodations is the lack of control of
extraneous variables
66
What is the current state of research on
accommodationsand what future research is
necessary?Students with Disabilities

• U.S. Department of Education
• RTTA Public Meeting on
• Creating Valid, Reliable, and Fair Assessments
  for Students with Disabilities and English
  Learners
• Martha Thurlow
• University of Minnesota
• August 10, 2011

67
Current State of Research on Accommodations for
Students with Disabilities

• Evidence-based Accommodations
• Extended Time
• Oral Administration for Math Assessments
• Accommodations with Conflicting Evidence
• Oral Administration for Reading Assessments
• Segmented Text
• Scribe
• Calculator
• Accommodations without Research
• Engagement/motivation Accommodations
• Hundreds of other accommodations currently listed
  in state policies!

68
Needs for Research on Accommodations for
Students with Disabilities

• Clarification of the content!
• Strategies for determining, based on strong
  rationales, which accommodations do not
  compromise the content assessed and therefore
  need not be subjected to research
• Improved selection of students for participation
  in studies (only those who truly need the
  accommodation studied)
• Both extant data studies and experimental
  empirical? studies on targeted and
  controversial accommodations
• More attention to decision-making processes for
  who needs which accommodations

69
DISCUSSION

• How should the consortia focus on ensuring
  appropriate access to all students to minimize
  construct-irrelevance, etc. during item design
  and development?
• What methods or strategies do you need to
  determine whether the items are valid and fair
  for all populations?

70
LUNCH

71
DISCUSSION OF SELECTEDWRITTEN PUBLIC COMMENTS

72
TABLE EXERCISE ON PUTTING THEORY INTO PRACTICE

• Lizanne DeStefano, University of Illinois
• Rebecca Kopriva, Wisconsin Center for Educational
  Research

73
Questions to Consider

• What skills and knowledge are you trying to
  measure?
• How can students demonstrate whether they have
  the skills and knowledge?
• What are possible approaches to making accessible
  items for students with disabilities or English
  learners?
• What challenges arise with these approaches?
• How can these efforts improve assessment for all
  learners?

74
Lessons Learned from the NAEPAccessible Block
Study

• Lizanne DeStefano
• Jeremiah Johnson

75
Purpose

• To explore the use of modified NAEP blocks as a
  means of improving measurement of the abilities
  and skills of students who score at lower end of
  NAEP performance continuum (including SD and ELL)
  
• Develop a definition of what constitutes an
  accessible block of items
• Refine the process for developing accessible
  blocks that are aligned with the NAEP frameworks
• Develop two accessible blocks of math items per
  grade level
• Scale modified blocks/items with existing item
  pool

76
Item Modification

• A panel of ten education professionals, math
  content specialists, and individuals with ELL/SD
  experience was assembled
• Blocks of NAEP items were modified according to
  the Item Modification Guidelines and
  Procedures.
• All modified items maintained their original
  alignment with the math content areas defined by
  the NAEP framework.
• The item modification panel edited, revised, and
  updated the Item Modification Guidelines and
  Procedures to reflect their thoughts on best
  practice

77
Creating Accessible Items

• Clearly identify the construct(s) of interest for
  each item aligned with standard.
• Identify the range of knowledge and skills
  expressed within a single standard (level of
  proficiency).
• Compare the emphasis of knowledge and skills in
  standard and on assessment.
• Consider the characteristics of the target
  population

78
Guidelines for Increasing Accessibility

• Reduce language load.
• Carefully consider distracters.
• Provide consistent, simple formatting.
• Use supportive, complete graphics.
• Provide contextual information that enhances
  understanding.
• Eliminate extraneous information.
• Provide cues to aid understanding

79
Reducing cognitive demand

• Reduce the number of objectives assessed in a
  single item
• Limit the number of steps required to correctly
  answer an item or provide a template to structure
  the process
• Be transparent about how open ended items will be
  scored

80
Overall

• 4th Grade Average Percent Correct Difference
  32.27
• Accessible Blocks 81.64
• Source Blocks 49.37
• 8th Grade Average Percent Correct Difference
  26.41
• Accessible Blocks 73.87
• Source Blocks 47.46

81
Results By Block (4th Grade)
82
Percent Correct for SD(4th Grade)
83
Percent Correct for ELL (4th Grade)
84
Results By Block (8th Grade)
85
Percent Correct for SD(8th Grade)
86
Percent Correct for ELL(8th Grade)
87
Scaling Results
88
(No Transcript)
89
(No Transcript)
90
(No Transcript)
91
(No Transcript)
92
(No Transcript)
93
(No Transcript)
94
(No Transcript)
95
(No Transcript)
96
(No Transcript)
97
(No Transcript)
98
(No Transcript)
99
(No Transcript)
100
Summary of Findings

• Across groups and subgroups there were
• Substantial and similar average gains in percent
  correct by block
• Consistent declines in the number of students
  omitting various items
• Significant declines in the percentage of
  students not reaching items

101
Summary of Findings

• All items were scalable
• Modified items had similar discrimination and
  guessing characteristics (a and c parameters)
• There were significant reductions in item
  difficulty (b parameters)

102
Summary of Findings

• For the lowest performing students, the
  conditional standard error of measurement was
  significantly lower on the accessible blocks than
  the source blocks

103
Some Effective Uses of Technology Measuring
Content Knowledge and Skills of English Learners
and Students with Disabilities

• RTTA Meeting, August 10, 2011
• Rebecca Kopriva
• University of Wisconsin
• rkopriva_at_wisc.edu
• 
  

104
Think About What Technology Can Do

• Technology can fundamentally improve the
  measurement of valued knowledge and skills for
  Els and SwDs in several ways, including
• Making use of multi-semiotic representations to
  primarily convey meaning
• Establishing effective profiles so students can
  be provided the proper accommodations or
  adaptations.

105
1. Why Bother with Multi-Semiotic Representations?

• Students with literacy and language challenges
  ARE learning complex content.
• How?
• They and their teachers have learned to convey
  meaning using modes other than text as primary
  communication methods, supported by key language
  as needed.
• This means successful adaptations need to include
  ways to
• convey meaning to the student
• convey meaning from the student
• These adaptations may be useful for other
  students as well.

106
What Does This Mean for Assessment?

• Properly constructed, these methods can
• Broaden how students are allowed to respond.
• Broaden how we present the problems.
• Broaden our understanding of how students
  conceptualize knowledge and use skills.
• Most often it is best if multiple avenues of
  access are built into each of the tasks at each
  of these points.

107
Open Up Response Methods

108
Open Up Presentation Methods

109
SAMPLE ITEMS
110
Open Up Problem Solving WindowsBroadening our
understanding of how students conceptualize and
use skills

111
2. Identify Effective Student Profiles and Use
Them

• While the EL and SwD populations are
  heterogeneous, a reasonable number of student
  accommodation profiles can be assembled.
• The purpose of the profiles is to group students
  by similar characteristics that make a difference
  in how to best accommodate them on assessments.
• Students within the same profile share
• similar strengths and challenges
• the same suite of accommodations

112
Effective Student Profiles

• Effective profiles capture targeted student
  information that can successfully identify the
  most appropriate accommodations.
• Ineffective profiles categorize students by
  irrelevant or incomplete information and lead to
  inappropriate or incomplete accommodation suites.

113
Relevant Characteristics of EL Students
114
Effective Linking Procedures

• Effective profiles can be linked to appropriate
  accommodations.
• This is usually completed through a series of
  algorithms that are each keyed to specific
  profiles and specific accommodation choices and
  suites.
• These algorithms can be appropriate or
  incomplete.

115
Effective Adaptations

• Profiles can also be used to build effective
  access avenues into assessment items and tasks.
• Examples of these kinds of adaptions have been
  shown above.

116
Questions to Consider

• What skills and knowledge are you trying to
  measure?
• How can students demonstrate whether they have
  the skills and knowledge?
• What are possible approaches to making accessible
  items for students with disabilities or English
  learners?
• What challenges arise with these approaches?
• How can these efforts improve assessment for all
  learners?

117
Mathematics Common Core Standard

• Grade 7
• Equations and Expressions (7.EE.3)
• Solve multi-step real-life and mathematical
  problems posed with positive and negative
  rational numbers in any form (whole numbers,
  fractions, and decimals), using tools
  strategically. Apply properties of operations to
  calculate with numbers in any form convert
  between forms as appropriate and assess the
  reasonableness of answers using mental
  computation and estimation strategies.

118
Mathematics Common Core Standard

• Identify a particular student profile
• Specify what knowledge or skills  from this
  standard they intend to measure in a basic way,
  and in a more complex way
• Give an example of a particular item/task topic
  (outline of an item) that will measure the more
  basic knowledge and skills and an item/task topic
  that will measure the more complex
  knowledge/skills
• Explain how they might make tasks that would
  cover each of these targets accessible for their
  chosen student profile. Use the questions to
  consider as a guide in explaining

119
Mathematics Common Core Standard

• Grade 7
• Equations and Expressions (7.EE.3)
• Solve multi-step real-life and mathematical
  problems posed with positive and negative
  rational numbers in any form (whole numbers,
  fractions, and decimals), using tools
  strategically. Apply properties of operations to
  calculate with numbers in any form convert
  between forms as appropriate and assess the
  reasonableness of answers using mental
  computation and estimation strategies.

120
Reading/Language Arts Common Core Standard

• Grade 8
• Reading Informational Text (RI.8.8)
• Delineate and evaluate the argument and specific
  claims in a text, assessing whether the reasoning
  is sound and the evidence is relevant and
  sufficient recognize when irrelevant evidence is
  introduced.

121
Reading Language Arts Common Core Standard

• Identify a particular student profile
• Specify what knowledge or skills  from this
  standard they intend to measure in a basic way,
  and in a more complex way
• Give an example of a particular item/task topic
  (outline of an item) that will measure the more
  basic knowledge and skills and an item/task topic
  that will measure the more complex
  knowledge/skills
• Explain how they might make tasks that would
  cover each of these targets accessible for their
  chosen student profile. Use the questions to
  consider as a guide in explaining

122
Reading/Language Arts Common Core Standard

• Grade 8
• Reading Informational Text (RI.8.8)
• Delineate and evaluate the argument and specific
  claims in a text, assessing whether the reasoning
  is sound and the evidence is relevant and
  sufficient recognize when irrelevant evidence is
  introduced.

123
CONCLUDING COMMENTS

124
PUBLIC COMMENTS

125
Race to the Top Assessment ProgramTechnical
Assistance Public Meeting Closing Comments

• Joe Conaty
• U.S. Department of Education

126
Reminders

• Transcript and presentations from todays meeting
  will be available at
• www2.ed.gov/programs/racetothetop-assessment
• Additional written input may be submitted to
  racetothetop.assessment_at_ed.gov

127
Future Public Meetings

• Future meetings may focus on
• Interoperability and technology standards
• Selection of a uniform growth model consistent
  with test purpose, structure, and intended uses
• Setting achievement standards setting and
  performance level descriptors
• As details are finalized, information will be
  posted on ed.gov and shared with stakeholder
  groups and prior meeting participants