Panel Discussion on Critical Issues in Assessment and Accountability for ESEA Reauthorization

1
Panel Discussion on Critical Issues in Assessment
and Accountability for ESEA Reauthorization
Moderator Rolf Blank, Director of Education
Indicators Programs, CCSSO
2
Presenters

• Mitchell Chester, Senior Associate Superintendent
  for Policy and Accountability, Ohio Department of
  Education
• Les Morse, Director, Assessment and
  Accountability, Alaska Department of Education
  and Early Development
• Ed Roeber, Director, Office of Educational
  Assessment and Accountability, Michigan
  Department of Education
• Jim Popham, Professor Emeritus, UCLA Graduate
  School of Education and Information Studies
• Martha Thurlow, Professor and Director, National
  Center on Educational Outcomes, University of
  Minnesota
• Linda Turner, South Dakota Department of
  Education, Special Education Programs
• Shelda Hale, Consultant, Kentucky Department of
  Education and Title III, LEP, and Immigrant
  Students
• Ellen Forte, President, edCount, LLC and
  Coordinator, CCSSO LEP SCASS

3
Mitchell Chester Senior Associate Superintendent
for Policy and Accountability Ohio Department of
Education
4
ESEA Reauthorization Contributions
Limitations of Core NCLB Accountability
Provisions
Mitchell D. Chester Ohio Department of
Education February 4, 2007
5
Systemic Validity
Education policies are systemically valid if they
result in decisions and actions that lead to
progress toward one or more intended goals
without causing regression with respect to other
goals. Henry Braun -- ETS -- 2006
6
Purpose of NCLB Accountability

• Improve student learning of important academic
  content
• Focus on achievement gaps -- students who are
  traditionally not well-served by schools

7
Mixed NCLB Message

• universal attainment of fundamental skills
• OR
• implementation of more challenging content
  standards
• Richard Hill -- 2006

8
Standards for Assessment Accountability Systems

• Standards for Educational Psychological Testing
  (AERA, APA, NCME -- 1999)
• High Stakes Testing for Tracking, Promotion,
  Graduation (NAS -- 1999)
• Position Statement on High-Stakes Testing (AERA
  -- 2000)
• Standards for Educational Accountability Systems
  (CRESST -- 2002)

9
Framework for High-Quality Assessment
Accountability Programs

• Good targets
• Aligned symmetrical
• Fair
• A. Porter M. Chester -- 2002

10
Good Targets

• Tests based on important outcomes
• Internal validity -- well articulated K-12
  expectations
• External validity -- skills needed beyond HS
• Assess a range of subjects
• Whole school responsibility
• Disaggregate results
• Performance standards that are attainable, but
  require schools to exceed current levels of
  attainment (Goldilocks standards)
• Size nature of rewards sanctions

11
Good Targets NCLB

• Contributions
• Disaggregation
• focus on gaps
• focus on SWD
• Promotes whole school responsibility

• Limitation
• Disincentive to
• raise standards
• benchmark to skills knowledge needed for
  success in higher education the workplace

12
Aligned Symmetrical

• Schools, teachers, students have overlapping
  stakes incentives
• Teachers of tested untested grades share
  responsibility
• Consistent signals
• state accountability programs incorporate federal
  criteria
• local accountability programs incorporate state
  criteria
• Evaluation criteria for teachers consider data
  used to assess students

13
Aligned/Symmetrical NCLB

• Contribution
• Distributes accountability widely across teachers

• Limitations
• AYP as a dichotomous rating
• Ability to subsume AYP within state system
  without overwhelming the non-AYP requirements

14
Fair

• Student opportunity to learn
• Adequate resources for schools
• Decisions based on reliable valid information
• State support commensurate with accountability
  standards
• Ongoing evaluation of assessment accountability
  program

15
Fairness NCLB

• Contributions
• Shifts focus from opportunity to results
• Options for students enrolled in schools that are
  not succeeding

• Limitation
• State capacity to support LEAs schools to be
  successful
• diagnostic capability
• intervention capability
• resource implications

16
Les Morse Director, Assessment and
Accountability Alaska Department of Education
and Early Development
17
ESEA Accountability

• Measuring change in achievement
• Groups of learners
• Applying consequences
• Measuring teacher quality
• Building capacity
• The NOT to do list

18
ESEA Accountability

• Consequences
• Differentiate miss AYP by a little versus a
  lot CCSSO Recommendations
• A little of what or a lot of what?
• Participation, a single content, graduation rate
• Does the consequence fit the problem
• Develop appropriate solutions - formative
  accountability!

19
ESEA Accountability

• How do we measure changes in achievement
• Status
• improvement
• Growth
• Different models that are not framed by
  restrictive rules, but rather based on the output
  of improving student achievement

20
ESEA Accountability

• Measuring Achievement
• Do the current targets make sense?
• AMO intervals
• 100
• Proficient
• Growth . . . How much is good enough?
• On track to what?
• Proficient
• Improvement individually
• Advanced achievement

21
ESEA Accountability

• Measuring changes in achievement for all learners
• Subgroups . . .
• Think differently . . . About evidence of
  achievement
• A district of 50,000 students
• A district of 10 students
• Standard is the constant, time is the variable
• different groups may need different variable, as
  well as a different level of sophistication

22
ESEA Accountability

• The right consequences
• Big law for a big land!
• Is what is right in New Orleans also right in
  Tuluksak?
• Will we continue to use consequences that may or
  may not work?
• How about unique consequences to address the
  problem
• Desk and live instructional audits may reveal
  need more clearly
• Do we know how to improve achievement? If so, why
  is that not a consequence?

23
ESEA Accountability

• Accountability for Teacher Quality
• Inputs versus outputs
• Reasonable . . . Big law, big country, what makes
  sense in NOLA may not make sense in Tuluksak!
• 20 of schools have 3 or fewer teachers and are
  k-12 in Alaska
• Core content subject knowledge have we over
  identified the core

24
ESEA Accountability

• Building Capacity
• To build and administer tests
• To support the RIGHT consequences
• To provide the appropriate tools and skills for
  analysis and diagnosis

25
ESEA Accountability

• What should we NOT do?
• Assume that NAEP and the state assessments
  measure the same things
• Assume our current consequences are the right
  ones
• Assume sciences education is not advancing unless
  it is included in the AYP calculation
• Assume that the challenge to improve the
  graduation rate is best addressed by further
  disaggregated graduation rate results at the
  school level (it may just create more arguments
  about n sizes)
• Assume that public schools will improve by adding
  the consequence of choice to a private school

26
Ed Roeber Director, Office of Educational
Assessment and Accountability Michigan Department
of Education
27
Introduction

• Assessment Issues
• Accountability Issues
• Assessment Suggestions
• Accountability Suggestions

28
Assessment Issues

• Peer Review issues
• Difficulties in understanding complex assessment
  systems by reviewers
• Difficulties of summarizing the status of the
  state across so many Critical Elements
• Inadequately trained Peer Reviewers
• Translation of the peer review into letters sent
  to the state
• Missed technical assistance opportunities

29
Assessment Issues

• How to determine the extent of alignment?
• How much alignment is enough?
• What types of alignment are important?
• How to determine the rigor of state standards?
• How to determine the rigor of the states
  assessments?

30
Assessment Issues

• Inadequate range of alternate assessments for
  students with disabilities
• 1 (alternate achievement standards)
• (Gap - modified achievement standards)
• 2 (as proposed)
• General assessment, with or without
  accommodations
• Requiring ELL students to participate in the
  states mathematics assessments in their first
  year in the U.S. is an exercise in frustration
  for the students (and their teachers).

31
Assessment Issues

• Innovative assessments and assessment systems not
  encouraged
• Multiple measures
• Formative assessments
• Assessment systems of formative, interim, and
  summative assessments
• Computer-adaptive testing
• Multiple end-of-course tests
• Will science assessment be expanded?
• Will social studies be added?

32
Accountability Issues

• Peer review issues
• Technical issues - e.g., studying the intended
  and unintended consequences of the assessments
• Political issues - suspicion that certain states
  received special favors
• Status model is used to determine adequate
  progress

33
Accountability Issues

• Inconsistencies across states
• Different minimum N across states
• Determination of AYP with sub-groups
• Measurement and sampling error
• Other differences
• Presumption of gaming the system versus
  technical accuracy
• One-size-fits-all school sanctions

34
Accountability Issues

• Students transition out of subgroups (ELL and
  SWD) and may not be counted in the subgroup
• Making accountability determinations when states
  changes their content standards and assessments
  may be challenging
• At the high school level, states may not be able
  to use student performance through the end of
  grade 12
• Little or no information on students in pre-K and
  K-2 programs

35
Accountability Issues

• What happens when restructuring schools doesnt
  work?
• Will science achievement be used for school or
  district accountability purposes?
• How will the Peer Review of states English
  language proficiency assessments be handled?

36
Assessment Suggestions

• Encourage innovative approaches to assessment
• Computer-adaptive
• Multiple end-of-courses
• Growth reporting
• Primary-level assessments
• Encourage balanced assessment systems to better
  balance instructional and accountability uses of
  assessment

37
Assessment Suggestions

• For ELLs, encourage states to develop a balanced
  assessment system - formative, interim, and
  summative assessments.
• Permit ELL students to be assessed only with
  formative and interim assessments in their first
  two years in the U.S.
• Better define student eligibility and exit from
  programs for ELL students.

38
Assessment Suggestions

• Provide meaningful assessments for each student
  with a disability so that these students can
  participate in assessments that they can
  accomplish - more than the 1 and proposed 2
  assessments
• Permit states that use a broader range of
  assessments for students with disabilities to
  count all of the students for accountability
  purposes.

39
Accountability Suggestions

• Differentiate consequences - a school in which
  the entire school and all subgroups fail to make
  AYP vs. one in which only one subgroup did not
  make AYP.
• Help states to determine the rigor of their state
  standards and assessments
• Examine how students do on both state and NAEP
  assessments.
• Conduct studies on how to determine the rigor of
  states assessment components
• Determine how to report this information in a
  helpful manner

40
Accountability Suggestions

• Suggest what states should do with schools who
  have restructured but still not made AYP.
• Permit states more flexibility to use student
  progress or growth as the central part of their
  accountability program.

41
Accountability Suggestions

• Condense the Peer Review technical criteria to
  focus on fewer criteria
• Improve Peer Review
• Better reviewer training prior to Peer Review
• More reviewer-state interactions during peer
  review
• Focus on state improvements

42
Jim Popham Professor Emeritus UCLA Graduate
School of Education and Information Studies
43
FIXABLE FLAW ONE

• Altogether Unrealistic Improvement Obligations
  for American Educators

44
FIXABLE FLAW TWO

• Federal Acquiescence in Allowing States to Use
  Instructionally Insensitive NCLB Tests

45
ESEA Reauthorization

• Discussion

46
Martha Thurlow Professor and Director National
Center on Educational Outcomes University of
Minnesota
47
Critical Issues in Assessment and Accountability
for Special Education Students
Martha L. Thurlow National Center on Educational
Outcomes www.nceo.info
48
Improvements in Clear Disaggregated
Participation Reporting to the Public
2000-2001 any content
2004-2005 Math
49
Assessments are doing what they are supposed to
be doing!
2004-2005 Middle School Math Assessments
50
Assessments are showing that performance of
special education students is changing in states!
1
2
3
4
5
6
7
8
9
10
51
National assessments are showing increases in
performance too!
NAEP Grade 4 Math Average Scale Scores, 1996 -
2005
From http//nces.ed.gov/nationsreportcard/nrc/rea
ding_math_2005/x0029.asp?printerver
52
If something is done for all students, it should
be done for special education students
Think very carefully about how (whether?)
something is done just for students with
disabilities consider all consequences
Stop giving lip service to All Means All Expect
it!
53
Linda Turner Special Education Programs South
Dakota Department of Education
54
Reauthorization

• IDEA 04 included
• Commitment of partnership and collaboration
• Expectations for high quality instruction
• Accountability
• ESEA reauthorization must
• Continue collaboration
• Consider unique needs of students with
  disabilities

55
Talking Points

• Flexibility
• 1 - Alternate Assessment
• 2 - Modified Assessment
• Growth Models
• Graduation

56
Regs. Are Important, But.
I need some help for a child that is blind, deaf,
and profoundly retarded.  She is 11 years old and
should be a fifth grader. She is not able to sit
erect, crawl or walk.  She has a buggy that she
lies in and has foam forms to help keep her
comfortable because there are problems with her
hips turning and being dislocated.  She is very
rigid and tense and there is a real danger of
breaking bones if you force her.  She only weighs
35 pounds.  She is about 36 inches tall. She
cannot talk.  She only cries.  She cannot grasp
or point.  The only senses she uses are smell,
taste and touch.  She will not respond when you
talk to her.  She will not nod or shake her head
yes or no.  She doesn't respond to light. She
does eat pureed food that is fed to her from a
spoon.  She is also fed through a G tube.  She is
darling and the grandmother takes excellent care
of her.  I am at a total loss on how to do this.
I would like to meet with someone after the
workshop and get some assistance. Any help you
or one of your associates could render would be
so appreciated.  I am anxious to hear from you.
57
2 - Modified Assessment

• Gray Area / Gap Students
• Group not clearly defined
• Will the flexibility/regulations be the solution?

58
Graduation Goals

• Chiefs Recommendations
• Every student a graduate
• prepared for postsecondary education, work and
  citizenship
• US Dept. of Ed. Building on Success
• Graduate ready to succeed
• graduating students prepared to enter college or
  the workforce with the skills to succeed

59
Graduation Goals

• IDEA and State Performance Plan
• Raise graduation rates
• Lower drop-out rates
• HS transition
• Post school outcomes

60
Taken from Education Week Quality Counts 2007
From Cradle to CareerConnecting American
Education from Birth to Adulthood,
http//www.edweek.org/ew/toc/2007/01/04/index.html
referenced in CCSSO readings and resources
61
The Big Picture
62
The Big Picture
63
Shelda Hale Consultant Kentucky Department of
Education and Title III, LEP, and Immigrant
Students
64
Ellen Forte LEP SCASS Coordinator and
President edCount, LLC
65
Every English Language Learner a Graduate

• MegaSCASS
• February 4, 2007
• Ellen ForteedCount, LLCCCSSO LEP SCASS

66
Orientation

• Our educational responsibility to ELLs is to
  support the development of their English language
  proficiency and of their academic competencies
  from the first day of their enrollment in our
  schools.
• At present, most states are unable to measure
  adequately ELLs academic knowledge and skills.
  High quality measures of academic English
  language proficiency are only in their infancy.

• Federal education policy should incentivize
  practices that are effective in achieving English
  language proficiency and academic competencies.
• Federal education policy should promote the
  development of better assessment tools and
  neither require nor encourage inappropriate
  assessment practices.

67
Suggestions in 3 Key Areas

• Standards
• Assessment
• Accountability

68
ESEA Reauthorization

• Discussion

69
UpcomingConcurrent Sessions

• Emerging Inclusion of the Early Grades in State
  Assessment, Accountability, and Data Systems
• Jana Martella, Poydras Room
• Assessment and Accountability for English
  Language Learners Supporting Linguistic and
  Academic Proficiency
• Ellen Forte, Pelican Room 2
• Policy Supports for Effective Formative
  Assessment for All Students
• Don Long, Jim Popham, Doug Rindone, La Salle B
• Technical Requirements Flexibility for
  Innovative Approaches
• Phoebe Winter, Pelican Room 1
• Accountability Requirements Supporting States to
  Improve the Validity of School Accountability
• J.P. Beaudoin, Marianne Perie, Fulton Room
• Use of Growth Models in School Accountability
  Systems
• Bill Auty, Paul Bielawski, La Salle C