Interpreting PVAAS School Reports using a Growth Standard

1
PVAAS Growth Standard Methodology Statewide
Implementation
Pennsylvania Value-Added Assessment System (PVAAS)
Questions about these materials and the Growth
Standard Methodology can be directed to the PVAAS
Statewide Core Team. See contact info on Slide
31.
2
Important Understandings

• A Growth Standard is the minimal growth for
  experience from one year of schooling for each
  student.
• A Normal Curve Equivalent (NCE) Score is a score
  that indicates position of a scaled score from
  any distribution on a specific reference scale.

3
Growth Standard The Key Metric in PVAAS

• The Growth Standard specifies the minimum
  acceptable gain from grade to grade.
• This Standard can be set as a policy decision and
  is not relative to any group of students.
• The use of a Growth Standard creates the
  possibility that ALL schools can demonstrate
  appropriate growth.

4
The Benefit of NCE Scores

• NCE scores indicate the position of a scaled
  score on a reference scale (mean 50, sd
  21.06) so that the scaled scores from different
  distributions with different scales can be
  compared.
• The use of NCE scores does not impose a normal
  distribution on the data, nor does the use of NCE
  scores have any relationship to normed referenced
  tests.
• NCEs are excellent for looking at scores over
  time.
• Using Data to Improve Student Learning in High
  Schools
• Victoria L. Bernhardt

5
Mechanics of NCE Scores

• Click here to see how NCE scores are calculated.
• Click here to continue the presentation.

6
NCE Scores Are About Position

• To calculate an NCE score
• Calculate the z-score of the data value of
  interest, that is, the number of standard
  deviations the data value is from the mean of its
  distribution
• Equate the observed values z-score to the NCE
  z-score and solve

7
A Question

• George scores a 655 on the SAT mathematics exam.
• George also scores a 28 on the ACT mathematics
  exam.

• Which score should he report to his colleges if
  he wants to provide the better score?

8
A Matter of Comparison
The nature of each distribution is irrelevant to
the question of interest

• How do we compare Georges scores?

9
A Solution

• Conversion of both scores to NCE scores allows
  for the identification of the position of each
  score on the same scale.
• This identification of position provides the
  capability of comparison since the converted
  scores will be based on the same distribution
  parameters.

10
Which Score Should George Choose to Report?
Using a NCE scale with mean 50 and standard
deviation 21.06 SAT score of 655 ? NCE score
75.85 ACT score of 28 ? NCE score 80.74
ACT score
SAT score
Clearly, he should report his ACT score!
11
Why Change to NCE Scoresfor PSSA Scores?

• PSSA tests have different means and standard
  deviations at each grade and for different years.
  For example, in 8th grade

12
Consider Another Hypothetical Scenario
In 2006, Wilma was in 4th grade and scored as
follows on the 4th grade PSSA Mean for 4th Grade
2006 1303.24 Standard Deviation for 4th
Grade 2006 164.20 Wilmas scaled score
1425
In 2005, Wilma was in 3rd grade and scored as
follows on the 3rd grade PSSA Mean for 3rd Grade
2005 1356.75 Standard Deviation for 3rd
Grade 2005 126.20 Wilmas scaled score
1425
Do these scores indicated that Wilma progressed
during 4th grade?
13
Lets Look at it Graphically
Wilma
Wilma
Even though Wilmas scaled scores were the same
(both 1425), since the distributions were
different, we really cant compare the two scores
14
A Tentative Solution Conversion to Percentiles
Wilma
Wilma
In our example, Wilma score of 1425 was in the
66th percentile for 2005 but was in the 76th
percentile for 2006. These percentiles focus on
Wilmas position in each distribution.
15
But

• We cannot calculate Wilmas gain the difference
  of percentiles does not make sense
• Percentiles are not meaningful for calculating
  means for different years, gains, etc., since
  they are calculated from different distributions.

16
The Complete Solution Conversion to NCE Scores

• To establish a basis of comparison for different
  distributions from different schools in different
  years, we convert the scaled scores to units in
  the SAME scale.
• The scale we will use is from the NCE
  distribution with mean 50 and standard deviation
  approximately equal to 21.06.

Mean
17
The NCE Distribution and Wilma

• Wilmas NCE score for 2005 (3rd grade) is 61
• while her score for 2006 (4th grade) is 66.

Wilma 2006 4th
Wilma 2005 3rd
18
Wilmas gain

• Wilmas gain
• 2006 NCE score 2005 NCE score
• (4th Grade) (3rd
  Grade)
• 66 61
• 5
• The mean gain of all of the students in Wilmas
  grade can now be compared to the Growth Standard
  for growth for Wilmas grade.

19
PVAAS Statewide Methodology
Student A Base Year NCE Score (2006)
2009 Observed School Mean NCE Scores
20
The Problem with the Mean of the Observed Scores

• The mean of the observed NCE scores at best
  represents a single snapshot in time of student
  achievement of the PSSA Anchors
• Is it the most comprehensive assessment of the
  schools TRUE level of achievement?
• How about the Bad Day syndrome?

21
Observed vs. Composite EstimateWhich is better?

• What if we combined the new, observed data with
  all of the prior PSSA assessment information that
  we have for this cohort of students?
• Would not a longitudinal view of the cohorts
  performance yield a more precise and reliable
  estimate of the true level of achievement?
• This is the essence and power of the
• PVAAS methodology!

22
Consider an Example

• Determine the percent of candies that are blue

If you were to open only one bag and find that
13 of the candies are blue, how much confidence
would you have in your estimate of the true
percentage of blue candies for all candies?
23
Only One Sample? A Bit Risky

• Lets open 50 bags and look at the distribution
  of the percents of blue candies

Looking at these 50 bags, what would you estimate
the true percent of blue candies for all
candies?
24
What If?
Distribution with n 50

• Lets open 50 more bags and add them to the 50
  selected earlier

Distribution with n 100
With this additional data, we can make a better
estimate of the true percent of blue candies!
25
The Function of Estimates

• The PVAAS methodology provides estimates of
  current and previous achievement, and subsequent
  gain for the school entity using all information
  for each student, no matter how complete or
  sparse.
• This process yields fair estimates of the impact
  of schooling on the rates of progress of the
  student populations and mitigates the problem of
  student mobility.

26
PVAAS Statewide Methodology
2009 Observed School Mean NCE Scores
2009 Estimated School Mean NCE Scores
Computer
2008 Estimated School Mean NCE Score
Gain 2009 Estimate 2008 Estimate
2007 Estimated School Mean NCE Score
2006 Estimated School Mean NCE Score
Compare to Growth Standard ? School Rating
27
How to Measure Growth of a School?

• Using a Growth Standard
• Student scaled scores are converted to NCE
  scores.
• The mean NCE score for each school is calculated.
• PVAAS revises all earlier estimates based on the
  addition of the current data.
• PVAAS calculates an estimated NCE mean score.
• Estimated Mean NCE Gain
• Current Estimated NCE mean Previous Estimated
  NCE mean
• Gain is compared to Growth Standard for School
  Effect Rating.

28
Here is the Fall 2006 PVAAS District/School
Report
29
Gain Ratings
Mean NCE Gain for a single grade in a given year
represents the progress of students in that grade
relative to the Growth Standard of 0. Color
ratings Green students made at least one
years growth ? favorable indicator Yellow
students achieved somewhat less than expected
? warning sign Light Red students fell behind
their peers ? stronger caution Red students
made little progress ? most serious warning
30
Level of Evidence The Role of Standard Error

• The color-coded ratings on the mean gain of
  cohorts are based on the level of confidence we
  have that the gain of the cohort is truly below
  the Growth Standard

31
The Power of PVAAS

• The power of this methodology is that it
  produces
• Accurate estimates of the true level of
  achievement of the students in this school.
• Updated estimates of all prior mean performance
  estimates simultaneously as new data is input
  into the longitudinal data structure.
• Over time, more accurate and reliable estimates
  of the true level of understanding of the
  students in this grade or school.

32
Questions?

• For more information, contact
• pdepvaas_at_iu13.org
• 717-606-1911

33
www.pde.state.pa.us
Gerald L. Zahorchak, D.Ed.
Secretary of Education
Commonwealth of Pennsylvania