Performance Accountability System: Key Considerations for Accountability System Design

1
Performance Accountability System Key
Considerations for Accountability System Design

• Scott Marion
• Center for Assessment
• August 25, 2009

2
Perie, Park, Klau (CCSSO, 2007)

• Seven core design principles
• Goals
• Performance Indicators
• Design Decisions
• Consequences
• Communication
• Support
• System Evaluation, Monitoring, and Support

3
GoalsWhat are we trying to do?

• It is critical to be explicit about the goals and
  purposes of the accountability system as well as
  the uses of the results
• These decisions will serve as touchstones during
  subsequent design decisions
• Thinking about the mechanisms for how the goals,
  purposes, and uses will be fulfilled is also very
  important.

4
Reports

• Reports make the assessment results
  actionableteachers and leaders do not have time
  to do in-depth analyses, therefore the reports
  must lead to valid decisions
• Designing the reports up front is a useful way to
  help think about accountability system design

5
Outcomes/Consequences

• How will results be used (e.g. public reporting,
  rewards) to create incentives for intended
  behavior?
• How can we avoid unintended outcomes?
• We need to be clear about the manner in which the
  results will be used.
• We are limited in this regard by the constraints
  in SB180

6
Utility

• We argue that utility is one of the most
  important considerations in accountability
  designs
• In other words, how will the system be designed
  so that it can most effectively fulfill its
  intended purposes?

7
Tradeoffs

• Like the old saying, there is no free lunch
  accountability system designers attempt to find
  the perfect balance point among multiple
  constraints, pressures, and goals because we can
  rarely (if ever) get exactly what we want
• We highlight just a few here because we think
  they are especially applicable to this work

8
Simplicity-Validity

• It has been said that for every complex problem,
  there is a simple solutionexcept it is wrong!
• We want to search for the simplest and most
  elegant solution possible, but the most simple is
  often not the most fair
• Often the most fair can be very complex, which
  makes it tough to understand and communicate
• We need to find the balance

9
Rich DataData Burden

• Many have justifiably complained that systems
  such as NCLB focused too much on once/year test
  data in two subject areas
• We want rich pictures of school quality, but we
  do not want to burden school personnel into
  becoming our full-time data collectors
• We also want data that we can trust
• Again, the search for balance!

10
The NH Performance Accountability System
11
SB 180 Task Force must

• (a) Define the performance-based accountability
  system to be used by schools that will ensure
  that the opportunity for an adequate education is
  maintained.
• (b) Identify performance criteria and
  measurements.
• (c) Establish performance goals and the relative
  weights assigned to those goals.
• (d) Establish the basis, taking into account the
  totality of the performance measurements, for
  determining whether the opportunity for an
  adequate education exists, which may include the
  assignment of a value for performance on each
  measurement.
• (e) Ensure the integrity, accuracy, and validity
  of the performance methodology as a means of
  establishing that a school provided the
  opportunity for an adequate education as defined
  in RSA 193-E2-a.

12
SB 180 Requirements

• The task force shall develop a performance-based
  scoring system using only the best available data
  and indicators which are already provided to the
  department and/or performance measures that
  schools are already required to provide the
  department under other state or federal law.

13
system may consider one or more of the
following data and indicators

• (a) Performance on state tests administered
  pursuant to RSA 193-C and, upon the prior
  approval of the department, other assessments
  administered at local option that are consistent
  with the states curriculum standards.
• (b) Number and percentage of pupils participating
  in an advanced placement course.
• (c) Number and percentage of graduating pupils
  going on to post-secondary education and military
  service.
• (d) Attendance rates
• More.

14
Potential indicators (continued)

• (e) Annual cumulative drop-out rates of high
  school pupils.
• (f) School environment indicators, such as safe
  schools data.
• (g) Expulsion and suspension rates, including
  in-school and out-of-school suspensions, which
  shall be reported for each school year.
• (h) Number and percentage of classes taught by
  highly qualified teachers.
• (i) Teacher and administrative turnover rates at
  the school and district levels.

15
Goals of the Performance System

• Provide another opportunity for schools to
  demonstrate adequacy
• Collect and report data to assist educators in
  improving student achievement
• Identify desirable educational practices and
  outcomes to
• Facilitate public reporting of school
  effectiveness to education stakeholders

16
Performance Indicators

• Inclusion Are students participating in the
  system?
• Achievement Are students learning and growing?
• Readiness Are students prepared for success in
  secondary and post-secondary?
• Development Are schools implementing effective
  practices to promote student achievement?

17
Performance Indicators Inclusion

• Participation number of test takers divided by
  enrollment.
• Attendance number of FAY students in all grades
  for that school with fewer than 15 absences for
  the full school year divided by the total number
  of students in all grades for that school
  enrolled for a full academic year.

18
Performance Indicators Achievement

• NECAP Status Percent proficient, mean scale
  score
• Reading/ELA
• Writing
• Math
• Science?
• NECAP Growth FTC growth metric, student growth
  percentiles, other?
• Other local assessment options?

19
Performance Indicators Readiness

• Postsecondary attendance
• Graduation rate
• Dropout rate
• Other
• AP courses and performance
• Concurrent enrollment
• SAT, PSAT, ACT, Plan,

20
Performance Indicators Development

• Based on the idea that assessment data alone
  (outputs) are not sufficient to gauge the quality
  of the school or the effectiveness of the pilot.
  
• Rather, a number of specific actions and
  interventions (inputs) must take place to support
  a process that improves student achievement and
  promote the theory of action.

21
Performance Indicators Development

• Highly qualified teachers
• Advanced degrees, etc
• Teacher/leader turnover rates
• School environment indicators
• School climate
• Safe and drug-free schools
• Expulsion/suspension rates
• We do not assume that one size fits all for
  each school. One alternative may be for schools
  to pick from menu of selections

22
Performance Indicators Development

• Two primary challenges
• Collecting the data
• Evaluating the data
• Important to minimize burden to state department
  and local systems.
• Solution must be efficient and scalable

23
Design Decisions

• What is the unit of analysis and/or
  accountability?
• Student
• Class
• Subgroup
• School
• District

24
Design Decisions

• What measure will be reported?
• Attainment of target score
• Scale score
• Domain score
• Norm-referenced score (e.g. percentile)

25
Design Decisions

• How should the measures be reported?
• Status performance reported at one point in time
  (e.g. percent proficient)
• Improvement performance reported over time for
  different students on the same measure (e.g.
  grade 3 performance in 2008 and grade 3
  performance in 2009)
• Growth evaluating change over time for the same
  student or cohort (e.g. change in percent
  proficient from grade 3 to grade 4)
• To whom should the reports be targeted?
• Differences in reports by stakeholder position
  (e.g., parents, teachers, leaders)?

26
Design Decisions

• What is good enough performance?
• What level of performance on each assessment
  should be regarded as meeting standard?
• What percent of students should meet target
  standard?
• How much improvement or growth should be
  expected?
• How do decisions impact different types of
  schools (e.g. high status/low growth vs. low
  status/ high growth)

27
Design Decisions

• How can multiple indicators be combined?
• Conjunctive schools must meet standard on all
  indicators
• Disjunctive schools must meet standard on any
  one element (not likely to meet SB 180)
• Compensatory system/ Index High performance in
  one area can offset lower performance in another
• Profile system Categories of acceptable outcomes
  are identified, where a value in one area may
  influence another

28
Index Example

• Four assessments in the model ELA, mathematics,
  science, and writing
• Decision is made to value ELA and Mathematics
  over science and writing
• Structured such that attaining mean index score
  of 100 meets expectations
• Lower performance in one area can be offset in
  another

29
Profile Example

• Four assessments in the model ELA, mathematics,
  science, and writing
• Three possible outcomes that are deemed
  acceptable are presented
• Model values mathematics and ELA performance no
  profile is acceptable without at least meeting
  standards in these areas
• Writing performance can be below standard if ELA
  exceeds standard
• Science can be below standard if math exceeds

30
Questions for the group

• What do we value and what would we like to
  measure other than the SB 180 elements?
• Which of these elements do we currently collect?
• In what form are the data (quantitative/qualitativ
  e)?
• Are the data trustworthy/reliable?
• Would the data be easily corruptible if
  included in accountability?
• For the elements that we do not collect yet, what
  would it take to collect the data?
• How do we envision the relationship between the
  input and performance system, i.e., can we offer
  an acceptable alternative to what is in SB180?