Welcome to the Race to the Top Assessment Program Technical Assistance Public Meeting - PowerPoint PPT Presentation

Loading...

PPT – Welcome to the Race to the Top Assessment Program Technical Assistance Public Meeting PowerPoint presentation | free to view - id: 4714dc-NmVmN



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Welcome to the Race to the Top Assessment Program Technical Assistance Public Meeting

Description:

Welcome to the Race to the Top Assessment Program Technical Assistance Public Meeting Creating Valid, Reliable, and Fair Assessments for Students with Disabilities ... – PowerPoint PPT presentation

Number of Views:602
Avg rating:3.0/5.0
Slides: 128
Provided by: NewbyD
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Welcome to the Race to the Top Assessment Program Technical Assistance Public Meeting


1
Welcome to the Race to the Top Assessment
ProgramTechnical Assistance Public Meeting
  • Creating Valid, Reliable, and Fair Assessments
    for Students with Disabilities English Learners
  • Washington, DC
  • August 10, 2011
  • Please silence all cell phones and pagers.
  • Thank you!

2
Race to the Top Assessment (RTTA) Program
Overview and Meeting Goals
  • Joe Conaty
  • Patrick Rooney
  • U.S. Department of Education

3
RTTA Public Meetings
  • This is the third in a series of public meetings
    on RTTA.
  • Two prior meetings April 15 on State and Local
    Technology Infrastructure and June 10 on
    Automated Scoring of Assessments
  • Details on additional meetings will be
    forthcoming
  • Purpose of the meetings
  • To provide technical assistance to and to support
    collaborative efforts of PARCC and SBAC as they
    develop new assessment systems
  • To expand the knowledge and expertise of the
    Department and the public around key assessment
    issues
  • To facilitate discussion of key components of the
    systems with experts and the public at large
  • Funded in part by The William and Flora Hewlett
    Foundation

4
RTTA Program Goals
  • Support states in delivering a system of more
    effective and instructionally useful assessments
    that
  • Provide accurate information about what students
    know and can do by
  • Eliciting complex student demonstrations or
    applications of knowledge and skills, as
    appropriate
  • Accurately measuring student achievement across
    the full performance continuum
  • Accurately measuring student growth over a full
    academic year or course
  • Helping educators determine whether individual
    students are ready for college and careers by the
    time of high school graduation and, in previous
    grade levels, whether they are on-track for
    readiness
  • Reflect good instructional practice and support a
    culture of continuous improvement
  • Effectively assess all students, including
    students with disabilities and English learners

5
Looking Forward
  • Assessment systems must include one or more
    summative assessment components that are fully
    implemented by every state in each consortium by
    SY 2014-15, and are administered at least once
    during the academic year in, at a minimum
  • Reading/language arts and mathematics
  • Grades 3-8 and high school
  • Results used to inform
  • Teaching, learning, and program improvement
  • Determinations of school effectiveness
  • Determinations of principal and teacher
    effectiveness for the purposes of evaluation and
    support
  • Determinations of individual student college and
    career readiness

6
RTTA Grantees
  • Nearly 360 million awarded in September 2010 to
    two consortia, which together represent 45 states
    and DC
  • Partnership for Assessment of Readiness of
    College and Careers (PARCC)
  • Project Management Partner Achieve
  • SMARTER Balanced Assessment Consortium (SBAC)
  • Project Management Partner WestEd
  • PARCC and SBAC have demonstrated commitment from
    institutions of higher education (IHEs) in member
    states that they will use the results of these
    assessments to determine entry into
    credit-bearing courses
  • The IHEs represent 90 (PARCC) and 74 (SBAC) of
    students who matriculate directly from K-12

7
Students with Disabilities English Learners
  • The absolute priority required the consortia to
    create assessments for all students, including
    English learners and students with disabilities
  • The consortia are required to develop tests
    accessible for these populations and to create
    and standardize accommodations policies
  • Each consortium must develop a definition of
    English learner that is uniform across member
    states
  • Additional consortia
  • Alternate assessments for students with the most
    significant cognitive disabilities
  • English language proficiency

8
Expectations for the Meeting
  • We have invited a range of experts to this
    meeting to share their knowledge and experience
    with the consortia members, looking at both
    current state of research and promising
    approaches to improving accessibility for these
    students
  • Format
  • The morning will focus on key questions that need
    to be addressed regarding the needs of students
    with disabilities and English learners in the
    assessment system and possible methods to address
    those questions
  • The afternoon will focus on two standards one
    in English language arts and one in mathematics
    in a practical application of the issues to
    creating valid, reliable, and fair assessment
    items for these populations

9
Meeting Agenda
  • 900-935 Welcome/setting the stage
  • 935-1015 Fishbowl discussion
  • 1015-1030 Break
  • 1030-Noon Fishbowl discussion continued
  • Noon-100 Lunch
  • 100-130 Fishbowl discussion of public
    comments
  • 130-300 Table exercise
  • 300-315 Public comments
  • 315-330 Wrap-up
  • 330 Adjourn

10
Invited Experts
  • Jamal Abedi, University of California, Davis
  • Lizanne DeStefano, University of Illinois
  • Rebecca Kopriva, Wisconsin Center for Educational
    Research
  • Mike Russell, Measured Progress
  • Stephen Sireci, University of Massachusetts,
    Amherst
  • Guillermo Solano-Flores, University of Colorado,
    Boulder

11
Public Comments
  • ED wants to hear from the public on key
    considerations for creating valid, reliable, and
    fair assessments for students with disabilities
    and English learners
  • In the morning
  • Comment cards are available at the registration
    desk
  • ED, the consortia, and the experts will discuss
    the comments/questions as time allows at the
    start of the afternoon session
  • All input from comment cards will be posted on
    our website
  • In the afternoon
  • We have scheduled time for verbal public comment
    from 300-315 pm
  • Sign up to speak during the lunch break at the
    registration desk
  • Time limit Up to 3 minutes per
    person/organization
  • Due to limited time, those not able to provide
    comments in person may email them to
    racetothetop.assessment_at_ed.gov

12
Reminders
  • Please place all cell phones and other devices on
    vibrate
  • Race to the Top Assessment resources
    Applications, FAQs, plus todays materials and
    transcription available at
  • http//www2.ed.gov/programs/racetothetop-assessmen
    t
  • The purpose of this event is to promote a full
    discussion and hear a wide range of viewpoints on
    creating valid, reliable, and fair assessments
    for English learners and students with
    disabilities, as well as the challenges and
    opportunities afforded by the Race to the Top
    Assessment program. Through this meeting, the
    U.S. Department of Education is not seeking to
    promote and/or endorse any particular program,
    project, methodology or approach to this work.

13
INTRODUCTIONS
  • Patrick Rooney
  • U.S. Department of Education
  • Meeting Facilitator

14
  • Accessibility and Accommodations

Deborah Matthews - Kansas Accessibility
Accommodations Workgroup dmatthews_at_ksde.org
15
The Purpose of the Consortium
  • To develop a set of comprehensive and innovative
    assessments for grades 3-8 and high school in
    English language arts and mathematics aligned to
    the Common Core State Standards
  • Students leave high school prepared for
    postsecondary success in college or a career
    through increased student learning and improved
    teaching
  • The assessments shall be operational across
    Consortium states in the 2014-15 school year

16
29 Member States
17
Work Groups
  • Transition to Common Core State Standards
  • Technology Approach
  • Assessment Design Item Development
  • Assessment Design Performance Tasks
  • Assessment Design Test Design
  • Assessment Design Test Administration
  • Reporting
  • Formative Processes and Tools/Professional
    Development
  • Accessibility and Accommodations
  • Research and Evaluation

18
Accessibility And Accommodations Workgroup
  • Purpose
  • Ensure the SBAC Assessment System is maximally
    accessible to the broadest range of students
    through
  • identifying, recommending, and evaluating
    strategies, tools, and technologies, thereby
  • providing information and guidance that will
    positively impact critical aspects of assessment
    design and development

19
Accessibility and Accommodations Workgroup
  • New paradigm that focuses on the student first,
    not the test items which addresses accessibility
    issues as part of item development, not as an
    afterthought.
  • Computer based assessment allows technology to
    open many doors for students because
    accessibility is built into the assessments.
  • The necessity of accommodations is reduced.
    Accommodations that are allowed are more
    targeted.

20
Accessibility and Accommodations Workgroup
  • In both policy and practice, SBAC will
  • include the broadest range of students
  • by facilitating each students ability to
    demonstrate as fully as possible what they know
    and can do
  • on the targeted constructs being measured
  • in a manner that is equitable and reliable, and
    yields valid interpretations of results.

21
Five Goals Key Activities
  • Create policies that reflect current research,
    best practices, and future possibilities related
    to accessibility and accommodations
  • Create assessments that are free from bias and
    sensitivity issues leveraging new technologies,
    including interoperability while preserving test
    constructs
  • Create accessible and accommodated assessments
    that will yield valid and reliable results

22
Five Goals Key Activities
  • Ensure accessibility and accommodations practice
    and policy are implemented with fidelity
  • Develop useful reporting and presentation
    guidelines that include information on
    accessibility and accommodations actions in the
    aggregate and at the individual student level

23
SBAC Representativeswww.smarterbalanced.org
  • Michael Hock Vermont Accessibility
    Accommodations Workgroup Co-Chair
  • Michael.Hock_at_state.vt.us
  • Wendy Carver - Utah Accessibility
    Accommodations Workgroup
  • Wendy.Carver_at_schools.utah.gov
  • Shelbi Cole Connecticut Performance Tasks
    Workgroup
  • Shelbi.Cole_at_ct.gov
  • Gaye Fedorchak New Hampshire Accessibility
    Accommodations Workgroup
  • gfedorchak_at_ed.state.nh.us
  • Viji Somasundaram Wisconsin Item Development
    Workgroup
  • visalakshi.somasundaram_at_dpi.wi.gov

24
  • Accessibility for Students
  • August 10, 2011
  • www.PARCConline.org

25
The PARCC Vision
  • Create high-quality assessments that measure the
    full range of the Common Core State Standards
  • Build a pathway to college and career readiness
    for all students and make accurate and reliable
    determinations as to whether students are
    on track or ready for college and
    careers
  • Provide information that supports various
    accountability uses (e.g., school, educator,
    student)
  • Provide timely and actionable information that
    supports continuous improvements in curriculum
    and instruction that inform effective classroom
    instruction and assessment practices
  • Leverage technology for a variety of uses
    innovative items, accommodations, administration,
    and scoring and reporting.
  • Report results that allow for comparability
    across all PARCC states, across consortia, and to
    national and international assessments.

26
PARCC Accessibility Goals
  • The Partnership will
  • Work to minimize/eliminate features that are
    irrelevant to what is being measured and measure
    the range of complexity of the standards so that
    students can demonstrate their knowledge
  • Design each component in a manner that allows ELL
    students and students with identified needs to
    demonstrate what they know and can do
  • Apply principles of universal design for
    accessible assessments throughout every stage of
    developing assessment components, items, and
    performance tasks
  • Leverage technology for delivering assessment
    components as widely accessible as possible and
  • Establish a Committee on Accessibility and
    Accommodations comprised of knowledgeable testing
    officials from member states (OWG).

27
PARCC Governance Structure
Steering Committee
28
PARCC Technical Working Groups (TWG)
  • Limited number of groups convened by the TAC to
    address high priority topics that would benefit
    from collective problem-solving by leading
    experts
  • Comprised of domain-specific technical advisors
    who interact with leadership and working groups
    and report to the TAC
  • Accessibility, Accommodations, and Fairness TWG
  • Committee members represent a range of expertise
    in accessibility and accommodations
  • Role is to help guide the efforts of working
    groups in designing accessible assessments that
    remain true to the intended constructs

29
Accessibility, Accommodations, and Fairness TWG
Invited Members
  • Diane August
  • Center for Applied Linguistics (ELL)
  • David Edyburn
  • University of Wisconsin-Milwaukee (SWD)
  • Claudia Flowers
  • University of North Carolina Charlotte (SWD)
  • Dianne Piche
  • Leadership Conference on Civil Rights
  • Charlene Rivera
  • George Washington University (ELL)
  • Diane Spence
  • Region 4 Education Service Center, Braille
    Services (Braille)
  • Martha Thurlow
  • National Center on Educational Outcomes (SWD)
  • Dan Wiener, Chair
  • Massachusetts Department of Elementary and
    Secondary Education (accommodations for state
    assessments)
  • Gerunda Hughes will serve as the liaison to the
    PARCC Technical Advisory Committee (TAC).

30
PARCC Operational Working Groups (OWG)
  • Who Comprised of state representatives, Achieve
    staff members, and eventually vendor
    representatives
  • What Responsible for the day-to-day aspects of
    work of key components of work
  • Why To ensure efficient and effective
    collaboration among PARCC members to meet PARCC
    goals.

31
Accessibility, Accommodations, and Fairness OWG
Members
  • Roberta Alley (Chair / Leadership Team)
  • Arizona Department of Education
  • Trinell Bowman
  • Maryland State Board of Education
  • Mira Monroe
  • Colorado Department of Education
  • Melissa Fincher (Leadership Team)
  • Georgia Department of Education
  • Charity Flores
  • Indiana Department of Education
  • Andrew Hinkle
  • Ohio Department of Education
  • Leila Williams
  • Arizona Department of Education
  • Bambi Lockman
  • Florida Department of Education
  • Phyllis Lynch
  • Rhode Island Department of Education
  •  Michael Reid
  • Oklahoma State Department of Education
  • Lori Rodriguez
  • Florida Department of Education
  • Dan Wiener
  • Massachusetts Department of Elementary and
  • Secondary Education
  • Jessica Tickle
  • Achieve/PARCC
  • Danielle Griswold
  • Achieve/PARCC

32
Accessibility and AccommodationsWorking Groups
  • The Working Groups will be responsible for
  • Drafting a set of Partnership-wide policies in a
    Partnership Accommodations Manual to be adopted
    by each member state for identifying eligible
    students, selecting allowable accommodations, and
    administering accommodations. That process will
    include
  • Analyzing extant state accommodation policies,
  • Building a list of recommended standard
    accommodations,
  • Identifying constructs and research
    accommodations,
  • Recommending a set of proposed accommodation
    policies for the assessment,
  • Drafting a common Partnership Accommodations
    Manual,
  • Ensuring comparability in assessment
    administrations,
  • Monitoring ongoing refinements of accommodations,
    and
  • Developing training modules for IEP teams.

33
Accessibility and AccommodationsWorking Groups
  • Adopting key policies and definitions that will
    include
  • a common definition of English Learner
  • a common set of policies and procedures for
    providing assessment accommodations for English
    learners and students with identified needs and
  • a common set of policies and procedures for
    participation of English learners and students
    with identified needs in the assessment system.

34
Accessibility and AccommodationsWorking Groups
  • Accessibility and Accommodations as a part of the
    development process
  • Design review and feedback
  • Test blueprint development
  • Technology development and selection
  • Passage and media review committee involvement
  • Item review committee involvement
  • Bias and sensitivity committee involvement
  • Testing the efficacy of assessment items with
    accommodations with the intended groups of
    students in pilot and field testing
  • Including sufficient number of students with
    identified needs (across sub-categories) in pilot
    and field testing
  • Data review committee involvement

35
Accessibility and AccommodationsWorking Groups
  • Build accessibility throughout the test itself
    with no trade-off between accessibility and
    validity
  • Use a combination of accessible-authoring and
    accessible technologies from the inception of
    items and tasks
  • Establish and maintain a close working connection
    with the Technology, Design, and Research Working
    Groups

36
UNDERSTANDING THE POPULATION
  • Patrick Rooney
  • U.S. Department of Education

37
Who are students with disabilities?
Source U.S. Department of Education, SY 2008-09
Annual Performance Reports. Figure courtesy of
the National Center on Educational Outcomes
38
Understanding the Population
  • In 2008-09, some 6.5 million children ages 3-21
    received special education services (13 percent
    of the population)
  • 95 percent were enrolled in regular public
    schools
  • 57 percent spent most of their time in general
    classes
  • The vast majority of students with disabilities
    take the general reading/language arts and
    mathematics assessments
  • Students with significant cognitive disabilities
    Current law permits up to 1 percent of all
    students in the state (approximately 10 percent
    of students with disabilities) to take an
    alternate assessment based on alternate academic
    achievement standards
  • All students are expected to have access to, and
    be assessed against, grade-level content standards

39
Trends in Growth of English Learners
Percentage of children ages 5-17 who spoke a
language other than English at home and
percentage who spoke a language other than
English at home and spoke English with
difficulty Selected years, 1980-2009
Source Census Bureau NCES Condition of
Education 2011
40
Where are English learners?
Percentage of children ages 5-17 who spoke a
language other than English at home and spoke
English with difficulty, by state or
jurisdiction 2009
Source Census Bureau NCES Condition of
Education 2011
41
Understanding the Population
  • Over 300 different languages are spoken by
    students.
  • Top 5 languages spoken by English learners
  • Spanish (3.5 million)
  • Vietnamese (93,000)
  • Chinese (80,000)
  • Arabic (71,000)
  • Hmong (50,000)
  • In 2000, some 64 percent of English learners were
    born in the United States
  • 42 percent were 2nd generation
  • 22 percent were 3rd generation
  • The HHS Office of Refugee Resettlement identified
    18,500 refugee children ages 6-18 in 2010.

42
UNDERSTANDING THE POPULATION
  • What are the challenges or key questions that
    need to be addressed by the consortia when
    developing their assessment systems?

43
BREAK

44
  • Accessible Assessment APIP
  • A Brief Overview
  • Michael Russell

45
  • Accessibility Assessment
  • A Two-Way Street

46
  • Accessibility Assessment
  • A Two-Way Street

47
  • Digital Assessment Delivery

What we can already do
  • Match Content Form to Student Access Need
  • Language
  • Translation (Directions, Whole Item, Individual
    Words/Phrases)
  • Simplified English
  • Audio
  • Text-to-Speech, Pre-recorded Voice
  • Text-based content, Graphics Tables, Non-Visual
    Descriptions
  • Braille
  • Sign

48
  • Digital Assessment Delivery
  • What we can already do
  • Match Interaction Response Modes to Need
  • Alternate keyboards
  • Tab-Enter control devices
  • Touch screens
  • Eye gaze
  • Speech-to-text

49
  • Current Challenges
  • Interoperable Accessibility
  • Standard Coding for Student Access
    Needs/Accommodations
  • Standard Tagging System for Item Content
    Accessibility Information
  • Standard File Exchange Format for Student Access
    Needs and Accessible Test Items

50
  • Accessible Portable Item Profile Standard
  • What is APIP?

File Exchange Format
51
  • Access Needs Addressed by APIP
  • Language-Related
  • Translated Item
  • Translated Words/Phrases
  • Translated Directions
  • Simplified Language
  • Audio Representation
  • Symbolic Representation
  • Extended Time/Breaks

52
  • Access Needs Addressed by APIP
  • Visual
  • Magnification
  • Reverse Contrast
  • Alternate Fore/Background Colors
  • Color Tinting/Overlay
  • Increased White Space

53
  • Access Needs Addressed by APIP
  • Executive Function/Maintaining Attention
  • Auditory Calming
  • Masking
  • Line Reader
  • Breaks
  • Extended Time

54
  • Access Needs Addressed by APIP
  • Information Processing
  • Flagging
  • Keyword Highlight
  • Alternate Representations
  • Scaffolding
  • Chunking
  • Reduced Answer Options
  • Negatives Removed

55
  • Access Needs Addressed by APIP
  • Representational Form
  • Audio text-only, graphic-only, text graphic,
    non-visual
  • Tactile
  • Braille
  • Sign (ASL, Signed English)

56
  • Standards in the Education Sector
  • Each Standard Addresses A Distinct Need
  • Content Standards - CCSS
  • Performance Standards Consortia Assessments
  • Data Standards SIF, CEDS, Ed-Fi
  • Interoperability Standards QTI-APIP

57
  • Standards in the Education Sector
  • How They Work Together in an Assessment Context

Common Core State Standards
58
  • APIP
  • Myths and Misunderstandings
  • Proprietary/Industry Led Standard
  • Competes/Clashes with SIF
  • Does not support Innovative Items
  • Requires High Bandwidth

59
  • APIP
  • Open Discussion
  • Questions Comments

60
DISCUSSION
  • What challenges and benefits does
    computer-administered testing create for
    accessibility?

61
What is the current state of research on
accommodationsand what future research is
necessary?ELL Students
  • U.S. Department of Education
  • RTTA Public Meeting on
  • Creating Valid, Reliable, and Fair Assessments
    for Students with Disabilities and English
    Learners
  • Jamal Abedi
  • University of California, Davis
  • August 10, 2011

62
What we need to know about accommodations before
using them for ELLs and SWDs
  1. Effectiveness How effective are accommodations
    in making assessments more accessible to ELL
    students?
  2. Validity How valid is the outcome of the
    accommodated assessment when compared to a
    non-accommodated assessment?
  3. Differential Impact To what degree are these
    accommodations universally applicable to ELL
    students with different background
    characteristics?
  4. Comparability Can accommodated and
    non-accommodated assessment outcomes be
    aggregated?
  5. Relevance How appropriate are the accommodations
    used for these students?
  6. Feasibility How feasible is it to implement
    these accommodations in large-scale assessments?

63
What we know based on current research
  • Some accommodations may not be effective in
    making assessments more accessible to ELLs (e.g.
    one-on-one and small group testing)
    EFFECTIVENESS RELEVANCE
  • Some accommodations may alter the construct by
    providing an unfair advantage to the recipients
    (e.g. dictionary or glossary with content-related
    terms) VALIDITY COMPARABILITY
  • Some accommodations do no alter the focal
    construct and ensure comparability (e.g.
    linguistic modification) VALIDITY
  • However we have no hard evidence on the VALIDITY
    and EFFECTIVENESS for the majority of
    accommodations currently in use.

64
What research needs to focus on next
  • Research needs to
  • Examine the validity of accommodations
  • i.e. if accommodations impact the focal construct
  • Determine the effectiveness of accommodations in
    making assessments more accessible to ELLs and
    SWDs
  • What does this look like?
  • Randomized Field Experiment in which all major
    sources of threat to internal and external
    validity of the experiment are controlled
  • Existing data on accommodations may not be a good
    source for examining either the validity or
    effectiveness of accommodations

65
How to test the validity and effectiveness of an
accommodation in an experimentally controlled
field study
ELL Status/Accommodation Accommodated Non-Accommodated
ELL G1 G2
Non-ELL G3 G4
Effectiveness Comparing G1 with G2 Validity
Comparing G3 with G4 The main reason for
inconsistencies between existing research on
accommodations is the lack of control of
extraneous variables
66
What is the current state of research on
accommodationsand what future research is
necessary?Students with Disabilities
  • U.S. Department of Education
  • RTTA Public Meeting on
  • Creating Valid, Reliable, and Fair Assessments
    for Students with Disabilities and English
    Learners
  • Martha Thurlow
  • University of Minnesota
  • August 10, 2011

67
Current State of Research on Accommodations for
Students with Disabilities
  • Evidence-based Accommodations
  • Extended Time
  • Oral Administration for Math Assessments
  • Accommodations with Conflicting Evidence
  • Oral Administration for Reading Assessments
  • Segmented Text
  • Scribe
  • Calculator
  • Accommodations without Research
  • Engagement/motivation Accommodations
  • Hundreds of other accommodations currently listed
    in state policies!

68
Needs for Research on Accommodations for
Students with Disabilities
  • Clarification of the content!
  • Strategies for determining, based on strong
    rationales, which accommodations do not
    compromise the content assessed and therefore
    need not be subjected to research
  • Improved selection of students for participation
    in studies (only those who truly need the
    accommodation studied)
  • Both extant data studies and experimental
    empirical? studies on targeted and
    controversial accommodations
  • More attention to decision-making processes for
    who needs which accommodations

69
DISCUSSION
  • How should the consortia focus on ensuring
    appropriate access to all students to minimize
    construct-irrelevance, etc. during item design
    and development?
  • What methods or strategies do you need to
    determine whether the items are valid and fair
    for all populations?

70
LUNCH

71
DISCUSSION OF SELECTEDWRITTEN PUBLIC COMMENTS

72
TABLE EXERCISE ON PUTTING THEORY INTO PRACTICE
  • Lizanne DeStefano, University of Illinois
  • Rebecca Kopriva, Wisconsin Center for Educational
    Research

73
Questions to Consider
  • What skills and knowledge are you trying to
    measure?
  • How can students demonstrate whether they have
    the skills and knowledge?
  • What are possible approaches to making accessible
    items for students with disabilities or English
    learners?
  • What challenges arise with these approaches?
  • How can these efforts improve assessment for all
    learners?

74
Lessons Learned from the NAEPAccessible Block
Study
  • Lizanne DeStefano
  • Jeremiah Johnson

75
Purpose
  • To explore the use of modified NAEP blocks as a
    means of improving measurement of the abilities
    and skills of students who score at lower end of
    NAEP performance continuum (including SD and ELL)
  • Develop a definition of what constitutes an
    accessible block of items
  • Refine the process for developing accessible
    blocks that are aligned with the NAEP frameworks
  • Develop two accessible blocks of math items per
    grade level
  • Scale modified blocks/items with existing item
    pool

76
Item Modification
  • A panel of ten education professionals, math
    content specialists, and individuals with ELL/SD
    experience was assembled
  • Blocks of NAEP items were modified according to
    the Item Modification Guidelines and
    Procedures.
  • All modified items maintained their original
    alignment with the math content areas defined by
    the NAEP framework.
  • The item modification panel edited, revised, and
    updated the Item Modification Guidelines and
    Procedures to reflect their thoughts on best
    practice

77
Creating Accessible Items
  • Clearly identify the construct(s) of interest for
    each item aligned with standard.
  • Identify the range of knowledge and skills
    expressed within a single standard (level of
    proficiency).
  • Compare the emphasis of knowledge and skills in
    standard and on assessment.
  • Consider the characteristics of the target
    population

78
Guidelines for Increasing Accessibility
  • Reduce language load.
  • Carefully consider distracters.
  • Provide consistent, simple formatting.
  • Use supportive, complete graphics.
  • Provide contextual information that enhances
    understanding.
  • Eliminate extraneous information.
  • Provide cues to aid understanding

79
Reducing cognitive demand
  • Reduce the number of objectives assessed in a
    single item
  • Limit the number of steps required to correctly
    answer an item or provide a template to structure
    the process
  • Be transparent about how open ended items will be
    scored

80
Overall
  • 4th Grade Average Percent Correct Difference
    32.27
  • Accessible Blocks 81.64
  • Source Blocks 49.37
  • 8th Grade Average Percent Correct Difference
    26.41
  • Accessible Blocks 73.87
  • Source Blocks 47.46

81
Results By Block (4th Grade)
82
Percent Correct for SD(4th Grade)
83
Percent Correct for ELL (4th Grade)
84
Results By Block (8th Grade)
85
Percent Correct for SD(8th Grade)
86
Percent Correct for ELL(8th Grade)
87
Scaling Results
88
(No Transcript)
89
(No Transcript)
90
(No Transcript)
91
(No Transcript)
92
(No Transcript)
93
(No Transcript)
94
(No Transcript)
95
(No Transcript)
96
(No Transcript)
97
(No Transcript)
98
(No Transcript)
99
(No Transcript)
100
Summary of Findings
  • Across groups and subgroups there were
  • Substantial and similar average gains in percent
    correct by block
  • Consistent declines in the number of students
    omitting various items
  • Significant declines in the percentage of
    students not reaching items

101
Summary of Findings
  • All items were scalable
  • Modified items had similar discrimination and
    guessing characteristics (a and c parameters)
  • There were significant reductions in item
    difficulty (b parameters)

102
Summary of Findings
  • For the lowest performing students, the
    conditional standard error of measurement was
    significantly lower on the accessible blocks than
    the source blocks

103
Some Effective Uses of Technology Measuring
Content Knowledge and Skills of English Learners
and Students with Disabilities
  • RTTA Meeting, August 10, 2011
  • Rebecca Kopriva
  • University of Wisconsin
  • rkopriva_at_wisc.edu


104
Think About What Technology Can Do
  • Technology can fundamentally improve the
    measurement of valued knowledge and skills for
    Els and SwDs in several ways, including
  • Making use of multi-semiotic representations to
    primarily convey meaning
  • Establishing effective profiles so students can
    be provided the proper accommodations or
    adaptations.

105
1. Why Bother with Multi-Semiotic Representations?
  • Students with literacy and language challenges
    ARE learning complex content.
  • How?
  • They and their teachers have learned to convey
    meaning using modes other than text as primary
    communication methods, supported by key language
    as needed.
  • This means successful adaptations need to include
    ways to
  • convey meaning to the student
  • convey meaning from the student
  • These adaptations may be useful for other
    students as well.

106
What Does This Mean for Assessment?
  • Properly constructed, these methods can
  • Broaden how students are allowed to respond.
  • Broaden how we present the problems.
  • Broaden our understanding of how students
    conceptualize knowledge and use skills.
  • Most often it is best if multiple avenues of
    access are built into each of the tasks at each
    of these points.

107
Open Up Response Methods

108
Open Up Presentation Methods

109
SAMPLE ITEMS
110
Open Up Problem Solving WindowsBroadening our
understanding of how students conceptualize and
use skills

111
2. Identify Effective Student Profiles and Use
Them
  • While the EL and SwD populations are
    heterogeneous, a reasonable number of student
    accommodation profiles can be assembled.
  • The purpose of the profiles is to group students
    by similar characteristics that make a difference
    in how to best accommodate them on assessments.
  • Students within the same profile share
  • similar strengths and challenges
  • the same suite of accommodations

112
Effective Student Profiles
  • Effective profiles capture targeted student
    information that can successfully identify the
    most appropriate accommodations.
  • Ineffective profiles categorize students by
    irrelevant or incomplete information and lead to
    inappropriate or incomplete accommodation suites.

113
Relevant Characteristics of EL Students
114
Effective Linking Procedures
  • Effective profiles can be linked to appropriate
    accommodations.
  • This is usually completed through a series of
    algorithms that are each keyed to specific
    profiles and specific accommodation choices and
    suites.
  • These algorithms can be appropriate or
    incomplete.

115
Effective Adaptations
  • Profiles can also be used to build effective
    access avenues into assessment items and tasks.
  • Examples of these kinds of adaptions have been
    shown above.

116
Questions to Consider
  • What skills and knowledge are you trying to
    measure?
  • How can students demonstrate whether they have
    the skills and knowledge?
  • What are possible approaches to making accessible
    items for students with disabilities or English
    learners?
  • What challenges arise with these approaches?
  • How can these efforts improve assessment for all
    learners?

117
Mathematics Common Core Standard
  • Grade 7
  • Equations and Expressions (7.EE.3)
  • Solve multi-step real-life and mathematical
    problems posed with positive and negative
    rational numbers in any form (whole numbers,
    fractions, and decimals), using tools
    strategically. Apply properties of operations to
    calculate with numbers in any form convert
    between forms as appropriate and assess the
    reasonableness of answers using mental
    computation and estimation strategies.

118
Mathematics Common Core Standard
  • Identify a particular student profile
  • Specify what knowledge or skills  from this
    standard they intend to measure in a basic way,
    and in a more complex way
  • Give an example of a particular item/task topic
    (outline of an item) that will measure the more
    basic knowledge and skills and an item/task topic
    that will measure the more complex
    knowledge/skills
  • Explain how they might make tasks that would
    cover each of these targets accessible for their
    chosen student profile. Use the questions to
    consider as a guide in explaining

119
Mathematics Common Core Standard
  • Grade 7
  • Equations and Expressions (7.EE.3)
  • Solve multi-step real-life and mathematical
    problems posed with positive and negative
    rational numbers in any form (whole numbers,
    fractions, and decimals), using tools
    strategically. Apply properties of operations to
    calculate with numbers in any form convert
    between forms as appropriate and assess the
    reasonableness of answers using mental
    computation and estimation strategies.

120
Reading/Language Arts Common Core Standard
  • Grade 8
  • Reading Informational Text (RI.8.8)
  • Delineate and evaluate the argument and specific
    claims in a text, assessing whether the reasoning
    is sound and the evidence is relevant and
    sufficient recognize when irrelevant evidence is
    introduced.

121
Reading Language Arts Common Core Standard
  • Identify a particular student profile
  • Specify what knowledge or skills  from this
    standard they intend to measure in a basic way,
    and in a more complex way
  • Give an example of a particular item/task topic
    (outline of an item) that will measure the more
    basic knowledge and skills and an item/task topic
    that will measure the more complex
    knowledge/skills
  • Explain how they might make tasks that would
    cover each of these targets accessible for their
    chosen student profile. Use the questions to
    consider as a guide in explaining

122
Reading/Language Arts Common Core Standard
  • Grade 8
  • Reading Informational Text (RI.8.8)
  • Delineate and evaluate the argument and specific
    claims in a text, assessing whether the reasoning
    is sound and the evidence is relevant and
    sufficient recognize when irrelevant evidence is
    introduced.

123
CONCLUDING COMMENTS

124
PUBLIC COMMENTS

125
Race to the Top Assessment ProgramTechnical
Assistance Public Meeting Closing Comments
  • Joe Conaty
  • U.S. Department of Education

126
Reminders
  • Transcript and presentations from todays meeting
    will be available at
  • www2.ed.gov/programs/racetothetop-assessment
  • Additional written input may be submitted to
    racetothetop.assessment_at_ed.gov

127
Future Public Meetings
  • Future meetings may focus on
  • Interoperability and technology standards
  • Selection of a uniform growth model consistent
    with test purpose, structure, and intended uses
  • Setting achievement standards setting and
    performance level descriptors
  • As details are finalized, information will be
    posted on ed.gov and shared with stakeholder
    groups and prior meeting participants
About PowerShow.com