The Data-informed Decision-making Challenge

1
PPT Presentation TemplateThis PPT includes all
slides to present a district or building level
overview of PVAAS. This was used with a
district-wide K-12 faculty group in a small
school district, where one grade level
represented one building versus multiple
buildings on a district level report.
2
PVAASEvaluating Growth,Projecting Performance
3
Todays Session

• Concept of PVAAS
• Overview of Reporting
• Math and Reading, Grades 4-8
• Math and Reading, Grade 11
• Science, Grades 4, 8 and 11
• Writing, Grades 5, 8 and 11
• School District Data
• Questions and Answers

4
What Is PVAAS?
5
PVAAS IS

• Analysis of existing assessment data
• Longitudinal PSSA scores in available content
  areas
• Math, Reading, Science, Writing
• Provides reliable estimates
• A measure of a schools effectiveness
• Critical Relationship
• Growth Data vs. Achievement Data

6
PVAAS

• Is not another test
• Is not a comparison of single test scores
• Critical concept
• Does not provide a measure of growth for an
  individual student

7
Current Data Collection Approaches
High Achievement
Average Achievement
Low Achievement
Achievement is based on each students ending
point.
8
PVAAS Adds Another Variable To AchievementProgres
s
Below Growth Standard
At or Above Growth Standard
Progress is based on each students starting
point.
9
Viewing Achievement and Progress Together

• Scatter Plots
• Grades 4-8
• Reading and Math

10
The Scatter Plot Coordinate System
Vertical Axis PSSA Percent Proficient or Advanced
Growth Standard Boundary
One Year Growth
AYP Percent Proficient Target
Horizontal Axis PVAAS Growth Measure
11
Scatter Plot Quadrants and Their Meanings
Quad 2 Adequate Achievement Negative Growth
Quad 1 Adequate Achievement Positive Growth
Quad 4 Below Adequate Achievement Positive Growth
Quad 3 Below Adequate Achievement Negative Growth
12
SD Scatter PlotsGrade 4
Reading
Math
Insert District Scatter Plots Here
13
SD Scatter PlotsGrade 5
Reading
Math
Insert District Scatter Plots Here
14
SD Scatter PlotsGrade 6
Reading
Math
Insert District Scatter Plots Here
15
SD Scatter PlotsGrade 7
Reading
Math
Insert District Scatter Plots Here
16
SD Scatter PlotsGrade 8
Reading
Math
Insert District Scatter Plots Here
17
Viewing Achievement and Progress Together

• Scatter Plots
• Grade 11
• Reading and Math

18
Grade 11 Scatter Plots

• The PSSA-PVAAS Scatter Plots visually display the
  following data from the statewide distribution of
  PSSA results for the most current school year.
• Achievement
• The mean predicted performance of the cohort.
• The actual mean performance of all students in
  the state is displayed for reference
• Growth
• School Effect Observed Mean Predicted Mean
• Standardized School Effect School Effect
  relative to its error displays degree of
  confidence
• The following two slides explain the meaning of
  each of the regions of an 11th grade scatter
  plot.

19
The Scatter Plot Coordinate System for Grade 11
Vertical Axis Mean Predicted Scaled Score
Growth Descriptor Boundaries (2SEs)
Horizontal Axis Growth Measure in Standard Errors
2010 Mean Scaled Score
20
Scatter Plot Regions and Their Meanings for
Grade 11
21
SD Scatter PlotsGrade 11
Reading
Math
Insert District Scatter Plots Here
22
The TWO Types ofPVAAS Information
Looking Forward/Planning PVAAS Projection
Reports For Individual Students and Cohorts of
Students
Looking Back/Evaluation Value-added Growth
Reports For Cohorts of Students
Today
NO individual measure of student growth.
23
PVAAS Reporting
24
Student Projections

• Wouldnt it be great to know the likelihood that
  a student will be proficient on a future PSSA?

25
Student Projection Methodology

• A separate longitudinal modeling process focused
  on the performance scores of individual students.
• The model uses all of the available data (both
  math and reading) to project future PSSA
  performances in Math, Reading, Science and
  Writing.
• Projections can be calculated for an individual
  student or groups of students.

26
How do I know that the projectionis accurate?

• The projection is precise, and is created using a
  model that has been reviewed and approved by four
  different peer review panels and the GAO (US
  Government Accountability Office).
• The most recent 2008 growth model proposal to
  USDOE includes information regarding the
  statistical model and projection reliability
  study.
• (http//www.ed.gov/admins/lead/account/growthmodel
  /pa/index.html)
• Recent studies have confirmed that the PVAAS
  projections (even as far as 3 years into the
  future) are more reliable at looking at the
  future performance of a student than the most
  recent PSSA score.

27
PVAAS Student Projection Report
28
PVAAS Projections for Groups of Students Student
Search
29
Identified Students Meeting Search Requirements
30
PVAAS Reporting Projections to Basic,
Proficient, and Advanced
Math and Reading 3 ? 4 or 5 4 ? 5 or 6 5 ? 6 or 7 6 ? 7 or 8 7 ? 8 8 ? 11 Writing 3 ? 5 4 ? 5 5 ? 8 6 ? 8 7 ? 8 8 ? 11 Science 3 ? 4 4 ? NA 5 ? 8 6 ? 8 7 ? 8 8 ? 11
Grade Last Tested
Grade Last Tested
Grade Last Tested
31
Projection Summary Reports

• What is it?
• Report that summarizes the numbers and percents
  of students in likelihood ranges of performing at
  the proficient level on a future PSSA exam.
• How might a school use this report?
• Intervention Planning
• Resource Allocation
• Strategic Planning
• School Improvement Planning
• Cautions
• This report provides another indicator about
  likelihood of future performance.

32
School (Single Grade)Projection Summary Report
33
Value-Added

• Wouldnt it be great to know if your students
  made one years growth for the year of schooling
  they experienced last year?
• In Math and Reading (4-8 and 11)?
• In Science (4, 8 and 11)?
• In Writing (5, 8 and 11)?

34
PVAAS Growth Methods
Testing Subjects Grades Methodology
PSSA in consecutive years Math Reading 4-8 Growth Standard
PSSA not in consecutive years Writing 5, 8, 11 Science 4, 8, 11 Math Reading 11 Predictive
35
Value-added Measure of GrowthMath and Reading
(Grades 4-8)

• Comparison to a State Growth Standard
• The Growth Standard specifies the minimal
  acceptable academic gain from grade to grade for
  a cohort of students.
• The use of a Growth Standard creates the
  possibility that ALL schools can demonstrate
  appropriate growth.

36
PVAAS Value-Added Growth Descriptors Grade 4
8 Math and Reading
The PA Growth Standard specifies the minimal acceptable academic gain from grade to grade for a cohort of students.  The Growth Standard is met when the students in a school or district on average maintain their relative position when compared to the 2006 statewide distribution of student scores (2006 is the base year). The PA Growth Standard specifies the minimal acceptable academic gain from grade to grade for a cohort of students.  The Growth Standard is met when the students in a school or district on average maintain their relative position when compared to the 2006 statewide distribution of student scores (2006 is the base year).
Estimated mean NCE gain equal or greater than zero.  There is evidence that the school has met or exceeded the Growth Standard.  All schools can achieve this rating.
Estimated mean NCE gain less than zero by one standard error or less.  There is slight evidence that the school has not met the Growth Standard.
Estimated mean NCE gain less than zero by between one and two standard errors.  There is moderate evidence that the school has not met the Growth Standard.
Estimated mean NCE gain more than two standard errors less than zero.  There is significant evidence that the school has not met the Growth Standard.
37
PVAAS School ReportGrades 4 through 8 - Math and
Reading
38
What Value-added Gain Means!
Cohorts Achievement 7th grade 66.6
Cohorts Achievement 6th grade 60.7
0 10 20 30 40 50
60 70 80 90 100 NCE
Scale State Mean 50
PVAAS Gain Change in position from 6th to 7th
grade 66.6 60.7 5.9
NCE points
39
PVAAS Growth Methods
Testing Subjects Grades Methodology
PSSA in consecutive years Math Reading 4-8 Growth Standard
PSSA not in consecutive years Writing 5, 8, 11 Science 4, 8, 11 Math Reading 11 Predictive
40
Value-added Measure of Cohort Growth Math and
Reading (Grade 11)Science (Grades 4, 8, and
11)Writing (Grades 5, 8, and 11)

• Based on a predictive methodology
• A predicted score is calculated for each student
• Mean of predicted scores is calculated for
  students in a cohort
• Mean of observed scores of students in the
  cohort
• School Effect is the difference between the
  Observed and the Predicted Scores.

41
PVAAS Predictive Analysis

1. We collect all of the individual student data
   that is available for a minimum of two years.
2. All of these data are used to develop individual
   student profiles.
3. PVAAS then calculates a Predicted Performance
   Value based on the student profile for each
   student.

Year 1 Year 2 Year 3 PSSA
PSSA PSSA (11th) 7th 8th
Predicted Actual
Reading 1280 1300
Math 1330 1305
1290 ??
1320 ??
42
PVAAS Ratings of School Effect
Remember School Effect ? Mean of Student Scores
Mean of Predicted Scores So, if the students
perform as expected, School Effect ? 0. The
ratings are in terms of the expected value of 0.
Not Detectably Different
Above
Below
-2 SEs 0 2 SEs
Standard Error (SE) is a measure of variation in
the prediction.
43
Writing, Science Grade 11 Reading/Math Growth
Descriptors
Progress is defined by the district or school effect, which is a measure of the difference between observed performance and predicted performance. Progress is defined by the district or school effect, which is a measure of the difference between observed performance and predicted performance.
Progress is significantly above the progress of the average district in the state.
Progress is not detectably different from the progress of the average district in the state.
Progress is significantly below the progress of the average district in the state.
44
PVAAS School ReportGrades 11- Math and
Reading Grades 4, 8 11 Science Grades 5, 8
11 Writing
45
Summary ofPVAAS Growth Methods
Testing Subjects Grades Methodology
PSSA in consecutive years Math Reading 4-8 Growth Standard
PSSA not in consecutive years Writing 5, 8, 11 Science 4, 8, 11 Math Reading 11 Predictive
46
SummaryOne Years Growth
Growth Standard Method Math and Reading Grades 4 through 8 Cohort maintains its achievement position.
Predictive Method Math Reading Grade 11 Writing Grades 5, 8, 11 Science Grades 4, 8, 11 Cohorts actual performance is as expected.
47
PVAAS Performance Diagnostic ReportThe Key is
the RED Whisker!
48
Growth Standard MethodologyWhat the whiskers
tell us
Exceeded Growth Standard More than One Years
Growth Green
Met the Growth Standard Made One Years Growth
Yellow
Growth Standard (One Years Growth)
Below Growth Standard Less than One Years
Growth Rose
49
Predictive MethodologyWhat the whiskers tell us
ABOVE Expectations -- Observed Score Was Greater
Than Predicted Score
NDD No Detectable Difference Between Observed
Score and Predicted Score
Observed Predicted (One Years Growth)
BELOW Expectations -- Observed Score Was Less
Than Predicted Score
50
Performance Diagnostic Report
51
Greater Detail based on of students
52
Click on Hot Keys for lists of students.
53
Student Lists from Performance Diagnostic Reports
54
(No Transcript)
55
Your District and School Data
56
PAs Intent for PVAAS

• Another Tool for Continuous Improvement
• Measure of Growth to Add to Achievement/Status
  Measures
• Not to Be Used in Isolation

57
Questions and Feedback
58
QuestionsPVAAS Materials or Statewide
Implementationpdepvaas_at_iu13.org717-606-1911
PVAAS Report Web Sitehttps//pvaas.sas.com
59
www.pde.state.pa.us
333 Market Street
Harrisburg, PA 17126