Cell Size and Reliability Strategies

1
Cell Size and Reliability Strategies

• CCSSO Mega-SCASSJanuary 22-23, 2005
• Facilitator Andra Williams
• Presenter Ellen Forte Fast

2
Overview

• Why worry about AYP reliability?
• What affects AYP reliability?
• How are states addressing reliability in AYP?
• Where can I get more information?

3
Why worry about AYP reliability?

• To reduce the likelihood of misclassifying
  schools and districts
• Most states are more concerned about incorrectly
  identifying schools for improvement than about
  failing to identify schools that really do need
  help
• To reduce the number of schools and districts
  identified for improvement

4
What affects AYP reliability and what can be done
about it?

• As the reporting statistic is reduced (e.g.,
  changed from the mean to percent proficient),
  reliability decreases
• What states can do nothing right now, since
  states must use percent proficient
• As the number of scores increases, total score
  reliability increases
• Increase the minimum n and/or account for at
  least some portion of the error

5
How are states addressing reliability in AYP?

• Minimum n the number of scores required for
  the cell to be considered for AYP
• Confidence Intervals

6
How are states addressing reliability in AYP?

• Minimum n the number of scores required for
  the cell to be considered for AYP
• Single year minimum
• Possible combination of data from 2-3 years
• Conjunctive total and by grade
• Proportion of total
• Different minimums for different groups
• Different minimums for different indicators

7
How are states addressing reliability in AYP?

• Single year minimum
• Status
• Range 5 to 60
• Mode 40
• Trend
• Many states increased the minimum n
• Oklahoma changed from 30 to 0 with a 99 CI

8
How are states addressing reliability in AYP?

• Possible combination of data from 2-3 years
• Status
• Some states allow combination as an option if the
  single year n is below the minimum
• Some states allow combination when AYP is missed
• Trend
• Shift from combination as a requirement to
  combination as an option
• California deleted the combination option

9
How are states addressing reliability in AYP?

• Conjunctive total and by grade
• Kentucky originally required10 students per
  grade and 30 students per school,now requires10
  students per grade and 60 students per school

10
How are states addressing reliability in AYP?

• Proportion of total
• Status
• A few states require the group to represent a
  specific proportion of the school
• Texas 50/10/200
• California 50/15/100
• A number of states require the group to represent
  a specific proportion of the district
• California 100/15/200
• Trend
• Increasing use of proportions for
  districtse.g., minimum 40 for 1-4000 scores,
  then 1 of scores

11
How are states addressing reliability in AYP?

• Different minimums for different groups
• Status
• Many states use a larger minimum n for students
  with disabilities
• Some states use a larger minimum n for English
  language learners
• Maryland counts a student in only one program
  category
• Trend
• Increasing use of this strategy
• Increasing the minimum n

12
How are states addressing reliability in AYP?

• Different minimums for different indicators
• Status
• A number of states use higher minimum n for
  participation rate than for percent proficient
  (range 30-50 mode 40)
• Nevada for the participation rate, minimum
  20,but if a school has fewer than 20 scores, n-1
• Trends
• Not much has changed other than taking advantage
  of the March 29, 2004 guidance regarding the use
  of data for 1, 2, or 3 years, whichever provides
  the best result

13
How are states addressing reliability in AYP?

• Confidence Intervals
• Statistical estimates of the range in which a
  school or districts true AYP score (e.g.,
  percent proficient) might fall
• Address the question Is the AYP score really
  different from the AYP target?
• Balance between
• Falsely identifying good schools as needing
  improvement
• Failing to identify needy schools

Increase the size of the confidence interval
Decrease the size of the confidence interval
14
How are states addressing reliability in AYP?

• Percent Proficient Many states use confidence
  intervals of 95 or 99
• Safe Harbor Originally not allowed now many
  states allowed to use a 75 CI for safe harbor

15
Where can I get more information?

• National Center for the Improvement of
  Educational Assessment
• Hill, R. and DePascale, C.A. (April 2003).
  Adequate yearly progress under NCLB Reliability
  considerations. http//www.nciea.org/publications/
  NCMEconf_RHCD03.pdf.
• Richard Hill. (2000). Issues related to the
  reliability of school accountability scores.
• Richard Hill Charles DePasquale. (2002).
  Determining the reliability of school scores.
• CRESST
• Linn, R. L., Baker, E. and Betebenner, D. (2002).
  Accountability systems Implications of
  requirements of the No Child Left Behind Act of
  2001. CSE Technical Report 567.
  http//www.aera.net/pubs/er/pdf/vol31_06/AERA31060
  2.pdf.
• CCSSO
• Marion, S., White, C., Carlson, D., Erpenbach, W.
  J., Rabinowitz, S. and Sheinker, J. (2002).
  Making valid and reliable decisions in
  determining adequate yearly progress.
  http//www.ccsso.org/content/pdfs/AYPpaper.pdf.
• Forte Fast, E. Hebbler, S. (2004). A Framework
  for Examining Validity in State Accountability
  Systems. http//www.ccsso.org/publications/details
  .cfm?PublicationID236