Pennsylvania Value-Added Assessment System (PVAAS): PVAAS Public Data Release Update to PDE and PAIU - PowerPoint PPT Presentation

Loading...

PPT – Pennsylvania Value-Added Assessment System (PVAAS): PVAAS Public Data Release Update to PDE and PAIU PowerPoint presentation | free to download - id: 48500e-NDg0M



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Pennsylvania Value-Added Assessment System (PVAAS): PVAAS Public Data Release Update to PDE and PAIU

Description:

Pennsylvania Value-Added Assessment System (PVAAS): PVAAS Public Data Release Update to PDE and PAIU 2 Sessions March 9 and 10, 2011 Kristen Lewald, Ed.D. – PowerPoint PPT presentation

Number of Views:207
Avg rating:3.0/5.0
Slides: 79
Provided by: Jennif203
Learn more at: http://www.iu5.org
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Pennsylvania Value-Added Assessment System (PVAAS): PVAAS Public Data Release Update to PDE and PAIU


1
Pennsylvania Value-Added Assessment System
(PVAAS)PVAAS Public Data ReleaseUpdate to PDE
and PAIU
  • 2 Sessions
  • March 9 and 10, 2011
  • Kristen Lewald, Ed.D.
  • PVAAS Statewide Team for PDE
  • Lancaster-Lebanon IU 13

2
Todays Session
  • PDE and PAIU Executive Directors Meeting was
    held last Friday
  • Questions from public and districts/schools
  • Provide an update to PAIU
  • PDE joining us today as well

3
Agenda
  • Achievement and Progress
  • Available Data Crosswalk
  • Public Website Demonstration
  • Myths and Concerns Feedback
  • Suggested Resources/Supports for Communicating
    PVAAS
  • Questions

4
Achievement vs. Progress
  • Progress
  • Achievement
  • Is the concept underlying value-added analysis
    and reporting
  • Not correlated with demographic factors
  • Dependent upon what happens as a result of
    schooling
  • The final result of an academic experience
  • Highly correlated with demographic factors, such
    as socioeconomic status
  • Affected by factors outside the school

5
Value-Added is
  • A statistical analysis used to measure a
    districts or schools impact on the academic
    progress rates of groups of students from year to
    year.
  • Conceptually, a growth measure is approximately
    the difference between current achievement
    (current results) and prior achievement (prior
    results) with achievement being measured by an
    appropriate assessment, such as the PSSA.
  • However, PVAAS is NOT a simple comparison of two
    scores!

6
PVAAS Provides
Looking Forward/Planning PVAAS Projection
Reports For Individual Students and Cohorts of
Students
Public Site
Looking Back/Evaluation Value-added Growth
Reports For Cohorts of Students
Today
7
Key Concepts in Understanding PVAAS Reporting
  • One of the inherent benefits of the PVAAS
    analyses is that all scores for all students are
    included in the analyses. Simplistic approaches
    are less precise and sometimes biased.
  • While PVAAS does use individual student data to
    yield value-added reporting, there are NO
    individual student measures of growth yielded
    from PVAAS

8
Value-Added PVAAS
  • Pennsylvanias model for value-added is called
    PVAAS - the Pennsylvania Value-Added Assessment
    System.
  • PVAAS is based on the EVAAS Methodology the
    Education Value-Added Assessment System.
  • The EVAAS methodology has been nationally
    reviewed and published.

9
PVAAS System
  • Uses EVAAS statistical methodology
  • EVAAS Education Value-Added Assessment System
  • Mixed Model Multivariate Longitudinal Analyses
  • Lead Developer Dr. Bill Sanders, Univ. of TN
  • Now SAS, Inc.(Cary, NC) for EVAAS/PVAAS
  • Jim Goodnight, CEO

10
8 Year History of PVAAS
  • Pilot from 2002-2005 100 Districts
  • Fall 2006 Reporting Grades 4 and 6 Reporting to
    all 501 Districts
  • Fall 2007 Grades 4-8 Reporting to all Districts
  • Fall 2008 Grades 4-8, 11 Reporting to all
    Districts
  • Fall 2009 and 2010 Full Reporting to 500
    Districts
  • Math, Reading, Science, Writing
  • Grades 4-8, 11
  • Used as provision to meet AYP for NLCB AYP
    Growth Model
  • February 2011 Release of Public Reporting Site
    https//pvaas.sas.com

11
PVAAS Data Provides Information to
  • Raise Achievement
  • Close Achievement Gaps
  • Decrease Dropouts
  • Increase College Readiness

12
Key Concepts in Understanding PVAAS Reporting
  • PVAAS reporting reflects the effectiveness of
    your districts or schools Standards-Aligned
    System
  • PVAAS reporting reflects the districts or
    schools system regarding curriculum, assessment,
    and instruction

13
PVAAS Password-Protectedvs. Public Site
  • PVAAS yields data on Districts, Schools, Grades,
    Subgroups, Students
  • Math, Reading, Science, Writing
  • Public
  • Protected

14
Achievement Growth
  • Achievement results (PSSA) and growth results
    (PVAAS) must be used together to get a complete
    picture of student learning.
  • To view the achievement results of Pennsylvania's
    public districts/schools, go to
    http//paayp.emetric.net/

15
Example Achievement Growth
16
Example Achievement Growth Same Schools
17
Overview of PVAAS Public Reports Features
Screen Shots, THEN LIVE Web Demo
18
New PVAAS Login Page
  • https//pvaas.sas.com

19
PVAAS Public Reports
Use of Reports Tab to Select View Reports
20
PVAAS Public Reports
  • Value-Added Summary Reports
  • District/LEA and School Level data only
  • Math and Reading
  • Grades 4-8 and 9-11
  • School Search Capability
  • Allows users to find and view the progress of
    local schools, charter schools, and full-time
    CTCs across Pennsylvania.
  • Can search for similar schools based on grade
    levels tested, various demographics, Intermediate
    Unit (IU) region and/or county..

21
PVAAS Public Reports
  • Use of Tests Tab to View Reports at Different
    Grade Levels
  • Use of Subjects Tab to View Reports for Different
    Subjects

22
Reports on Public Site
  • They have a different format from the district
    password protected site
  • However, the measures on the PVAAS public site
    come directly from the reports on the
    district/school password protected site.
  • We will cross walk between these two sites in
    this session.

23
3 Key Resources for Public Reporting
  • Guide to Public Reporting
  • Crosswalk
  • Two pager on public reporting

24
Purpose of District SchoolValue-Added Data
  • Provides users with information to assist them
    in evaluating the overall effectiveness of a
    district/LEA or school on the academic progress
    of groups of students.
  • This report is NOT a report on teacher
    effectiveness!

25
Example of District Value-Added Summary
ReportGrades 4-8, Math Reading
26
Example of School Value-Added Summary
ReportGrades 4-8, Math Reading
27
Crosswalk Where can I find this information on
other reports?
Public Site School Value-Added Summary Report
Restricted Site School Value-Added Report
28
What is the Average Gain over Grades Relative to
the Growth Standard?
  • Represents the average gain across the grade
    levels served between 4 and 8 compared to the
    Growth Standard.
  • It is the average academic growth of the
    districts or schools students, compared to the
    Growth Standard.
  • Answers the question, How effective was the
    district/LEA/school in impacting the academic
    progress of its students compared to the Growth
    Standard?

29
Example Average Gain over Grades Relative to the
Growth Standard
  • Average Gain over Grades Relative to the Growth
    Standard (on the public report) is the SAME as
    the Mean NCE Gain over Grades Relative to the
    Growth Standard on the School Value-Added Report
    (password-protected site)!

30
What is the Average Gain over Grades Relative to
the State?
  • Represents the average gain across the grade
    levels served between 4 and 8 compared to the
    average progress of all students in Pennsylvania
    at the same grade levels.
  • It is the average academic growth of the
    districts or schools students, compared to the
    academic growth of students statewide.
  • Answers the question, How much did the
    district/LEA/school impact the academic progress
    of its students compared to the progress of other
    students in Pennsylvania in those same grade
    levels?

31
Example Average Gain over Grades Relative to the
State
  • Average Gain over Grades Relative to the State
    (on the public report) is the SAME as the Mean
    NCE Gain over Grades Relative to the State on the
    School Value-Added Report (password-protected
    site)!

32
What is Average Gain over Grades Relative to the
Growth Standard for the State?
33
Example Average Gain over Grades Relative to the
State
  • Average Gain over Grades Relative to the State
    (on the public report) is the SAME as the Mean
    NCE Gain over Grades Relative to the State on the
    School Value-Added Report (password-protected
    site)!

34
PVAAS Value-added Growth Descriptors Grades 4
8, Math and Reading
The Average Gain over Grades for grades 4-8 is expressed in Normal Curve Equivalent (NCE) units. The use of the NCEs allows PSSA scores in any school year and grade level to be compared across years. The Average Gain over Grades for grades 4-8 is expressed in Normal Curve Equivalent (NCE) units. The use of the NCEs allows PSSA scores in any school year and grade level to be compared across years.
Green (Favorable) The district/LEA/school was effective in supporting students to achieve one years worth of academic growth in a year.
Yellow (Caution) There was minimal evidence that the district/LEA/school was not effective in support students to achieve one years worth of academic growth in a year.
Rose (Concern) There was moderate evidence that the district/LEA/school was not effective in support students to achieve one years worth of academic growth in a year.
Red (Strong Concern) There was significant evidence that the district/LEA/school was not effective in support students to achieve one years worth of academic growth in a year.
35
Example of District Value-Added Summary
ReportGrades 9-11, Math Reading
36
Example of School Value-Added Summary
ReportGrades 9-11, Math Reading
37
Crosswalk Where can I find this information on
other reports?
Public Site School Value-Added Summary Report
Restricted Site School Value-Added Report
38
PVAAS Value-added Growth Descriptors Grades
9-11, Math and Reading
The District/School Effect for grades 9-11 is expressed in PSSA scaled score points. The District/School Effect for grades 9-11 is expressed in PSSA scaled score points.
Green (ABOVE Predicted Achievement) The district/LEA/school was highly effective. The district/LEA/school exceeded the expected progress with its students.
Yellow (MET Predicted Achievement) The district/LEA/school was effective. The district/LEA/school met the expected progress with its students.
Rose (BELOW Predicted Achievement) The district/LEA/school was not effective. The district/LEA/school did not meet the expected progress with its students.
39
What is the District/School Effect?
  • Provides an estimate of the districts or
    schools impact on students academic progress.
    Specifically, the District/School Effect is a
    function of the difference between the
    observed/actual PSSA achievement and the
    predicted PSSA achievement.
  • It is a measure of growth that the students
    tested in grade 11 have made over the past 3
    years since being tested in grade 8 and uses data
    from grades 3-8.
  • If students score as expected (i.e., students
    observed scores are equal to their predicted
    scores), then the District/School Effect would be
    0. A negative District/School Effect indicates
    students actual scores were lower than their
    predicted scores, while a positive
    District/School Effect indicates students actual
    scores were higher than their predicted scores.
  • Answers the question, How effective was the
    district/LEA/school in promoting student academic
    growth and supporting students to meet or exceed
    their expected progress?

40
Is it appropriate to compare the amount of
progress made by a district/school to another
district/school?
  • No! Not using the District and School
    Value-Added Summary Reports
  • Without taking the Standard Error into account,
    it is NOT possible to directly compare these gain
    values across districts/schools.
  • The color-coding of the growth measures (gain
    values) does in fact take into account the
    Standard Error.
  • Note the link to the color code legends on each
    report.
  • The Average Growth Index found on the School
    Search report feature takes the Standard Error
    into account and allows a more direct comparison
    across schools.

41
PVAAS Public Reports
Use of Reports Tab to Select View Reports
42
Purpose of School Search
  • Users can find and view the progress of public
    schools across Pennsylvania and search for
    similar schools based on grade levels tested,
    various demographics, Intermediate Unit (IU)
    region, and/or county.

43
School Search
44
School Search
45
Where can I find this information on other
reports?
Public Site School Search
Restricted Site School Search
46
Which schools are included when I use School
Search?
  • Schools with at least one tested grade in common
    as the reference school you selected AND any
    demographics selected, as well as IU or county
    region.
  • Example Your reference school is a grade 6-8
    school
  • Other schools included in the search may include
    grade 6-7 schools, grade 7-8 schools, K-6
    schools, etc.

47
How are schools compared?
  • PVAAS Average Growth Index
  • What is an Index?
  • A numerical scale used to compare variables with
    one another or with some reference number
  • Analogy Consumer Price Index
  • A measure of the average change over a period of
    time
  • Statistical Indicator
  • Reflects patterns
  • The PVAAS Average Growth Index allows viewers to
    compare growth across schools.

48
What is the Average Growth Index?
  • A measure of student progress across the tested
    grade levels in a school.
  • This index is a value based on the average
    growth across grade levels and its relationship
    to the standard error so that comparison among
    schools is meaningful. If the standard error is
    not accounted for, users might get a skewed
    picture of the relative effectiveness of
    different schools.
  • For grades 4 through 8, the Average Growth
    Index is calculated by dividing the Average Gain
    over Grades Relative to the Growth Standard by
    the corresponding Standard Error.
  • For grades 9 through 11, the Average Growth Index
    is calculated by dividing the School Effect by
    the corresponding Standard Error.

49
Example Average Growth Index, Grades 4-8
Average Growth Index Average Gain over Grades
Relative to the Growth Standard divided by the
Standard Error
50
Example Average Growth Index, Grades 4-8
Average Growth Index Average Gain over Grades
Relative to the Growth Standard divided by the
Standard Error 2.9 / 0.4 7.3 (due to rounding
this may NOT be exactly what is reported on
public site)
51
Example Average Growth Index, Grades 9-11
Average Growth Index School Effect divided by
the Standard Error
52
Example Average Growth Index, Grades 9-11
Average Growth Index School Effect divided by
the Standard Error -13.7 / 9.1 -1.5 (due to
rounding this may NOT be exactly what is reported
on public site)
53
What is the Average Growth Index?
  • Average Growth Index gt 0 ? On average, students
    in the school achieved a years worth of academic
    growth in a year. A large, positive Average
    Growth Index provides more evidence that more
    than a years worth of growth was experienced by
    the average student in the school.
  • Average Growth Index lt 0 ? On average, students
    in the school achieved less than a years worth
    of academic growth in a year. A large, negative
    Average Growth Index provides more evidence that
    less than a years worth of growth was
    experienced by the average student in the school.

54
Why I cannot find a district/school on the PVAAS
public site?
  • Districts/schools with less than 10 students have
    been suppressed
  • PA public reporting requirements
  • Districts/schools that do not receive PVAAS
    reporting are not included
  • Example K-3 school
  • PVAAS reporting is provided for reading and
    mathematics in grades 4-8 and 11

55
Why I cannot find a district/school on the PVAAS
public site?
  • Districts/schools that only receive growth
    reports reflecting the progress of ONE grade
    level are suppressed
  • Example Grade 11 only school
  • Example Grade 11-12 schools
  • Example Grade 6 only school
  • Example K-4 school
  • Act 104 legislative intent was district and
    building level data

56
Websitehttps//pvaas.sas.com
57
The Myths of PVAAS
58
Myth 1 PVAAS provides growth measures for an
individual student.
  • PVAAS does not estimate growth for one student
    because
  • The PSSA observed scores (and resulting NCE
    scores) are simply snapshots in time making
    comparisons of the observed scores as a measure
    of growth very unreliable.
  • The error in an estimate for a data set with only
    one record (one student) is too large to make the
    estimate meaningfuland
  • Error depends on variation in data AND the sample
    size (the number of student records in a
    dataset).
  • An estimate of progress, or growth, based on only
    one student would have a much larger error, and
    therefore be considerably less precise than when
    considering a group or cohort of students.

59
Myth 2 Growth is correlated with certain
demographic variables.
  • There is NO relationship between demographic
    variables, such as socioeconomic status, and
    growth.
  • There are high achieving schools making high
    growth
  • There are high achieving schools making low
    growth
  • There are low achieving schools making high
    growth
  • There are low achieving schools making low
    growth.
  • Growth reporting reflects what WE do with
    students in terms of academic growth in
    schools/districts.
  • -- VAAS can remove the effects of factors not
    under the control of the school. (McCaffrey,
    Lockwood, Koretz Hamilton, 2003 Ross, Wang,
    Sanders, Wright Stringfield, 1999a Wright,
    Horn Sanders, 1997).

60
Myth 3 Growth (grades 4-8) is calculated based
on how other schools perform.
  • Each year, a groups growth is calculated by
    comparing its position in the current grade
    distribution from 2006 to its former position in
    the previous grades distribution from 2006.
  • Performance of other groups in a given year does
    NOT affect the growth calculation of the cohort
    in question. Each group becomes its own control
    group!
  • Growth Analogy
  • For a child to get taller, another child does not
    have to get shorter!
  • A child can grow taller in a given year no matter
    how his/her peers grow.

61
Myth 4 Since PSSA distributions change each
year, growth (grades 4-8) is based on a moving
target. (We could never get a green.)
  • All grade 4-8 growth calculations and
    interpretations are based on the base year
    distributions from 2006.
  • The 2006 PSSA Math and Reading distributions
    provide typical demonstrations of achievement of
    cohorts as they progress through the grade
    levels.
  • PSSA scaled scores are converted to NCE units
    using the parameters from the 2006 distributions
    so they are relative to the same standard each
    year.
  • SAS, Inc. (the vendor for PVAAS) evaluates each
    years distributions to verify that using the
    base year of 2006 continues to be appropriate.

62
Myth 5 PSSA is not designed to discriminate
well at the extremes so growth cannot be
calculated using the PSSA.
  • PSSAs are designed to have sufficient stretch to
    discriminate between Below Basic, Basic,
    Proficient, and Advanced performance levels.
  • There is no ceiling on the PSSA! - PDE
  • Each year, scores are scaled to allow the high
    end to be scaled on the distribution of the data
    not on a fixed, pre-determined value.
  • The PSSA meets the three conditions to be used in
    PVAAS analyses.
  • Must be aligned to curriculum standards.
  • Must be reliable and valid.
  • Must demonstrate sufficient stretch at the
    extremes.

63
Myth 6 PVAAS is not reliable or valid since it
is only based on only one test, the PSSA.
  • PVAAS uses a multivariate, longitudinal mixed
    effect model in its analyses. It is not a simple
    comparison of two test scores!
  • All prior assessment scores are used.
  • Standard error is always reported.
  • PVAAS is an indicator of growth, or progress,
    of groups of students towards mastery of the
    Pennsylvania academic standards.

64
Myth 7 If students are already high achieving,
it is harder to show growth.
  • In PVAAS, one years growth is about maintaining
    achievement levels (grades 4-8) or meeting
    expected performance (grades 9-11) based on a
    specific groups prior academic performance.
  • For high-achieving groups, one years growth may
    be sufficient or acceptable.
  • For low-achieving schools, one years growth may
    not be sufficient or acceptable in order for
    students to meet long-term achievement goal of
    proficiency.

65
Myth 8 It is not possible to show progress with
all groups of students, such as students with
IEPs or gifted students.
  • If assessments have enough stretch to measure
    the achievement of both low- and high-achieving
    students, it is possible to measure all groups of
    students progress. The PSSA meets the criteria!
  • The value-added methodology used is sensitive
    to individual students achievement levels.
  • It measures growth from the end of one year to
    the end of the next year, regardless of whether a
    student performs below, at, or above grade
    level/proficiency.

66
Myth 9 PVAAS should always indicate growth if
our percent of students proficient/advanced
increased since last year.
  • When following the same grade level from one
    year to the next in determining the percent
    proficient/advanced, these are two different
    groups of students (i.e., 6th graders in 2010 are
    not the same group of students as 6th graders in
    2011). PVAAS on the other hand, is looking at
    the most recent group of students and evaluating
    their progress from the prior school year in the
    prior grade level (same group of students).
  • PVAAS is not measuring progress by students
    increasing or decreasing entire performance
    levels. PVAAS is sensitive to subtle changes in
    progress, even within performance levels.
  • Example Some students may have moved from a
    non-proficient to proficient status. However,
    students already proficient/advanced may be
    slipping in terms of their level of achievement
    compared to where they were the year prior. In
    other words, students may still be
    proficient/advanced, just not as high within
    those performance levels as they were in the
    prior year.

67
Myth 10 PVAAS cannot measure the progress of
districts and schools with high mobility rates.
  • Value-added analysis includes all students, for
    which there are sufficient test data, including
    highly-mobile students.
  • From a statistical perspective, it is important
    to include highly-mobile students in the analysis
    because their exclusion could bias the results.
  • From a philosophical perspective, all students
    must be included in the schools analysis to
    ensure that highly-mobile students receive the
    same level of attention as non-mobile students.
  • The EVAAS modeling approaches do take into
    account the quantity and quality of information
    available for each student!

68
Disclaimer for ANY Data Tool
  • NO data source should ever be considered in
    isolation.
  • ALL education decisions should be made on the
    basis of multiples sources of both quantitative
    and qualitative data.
  • ALL data provides indicators of phenomena.

69
When new data is gathered
  • The intelligent user of data should ask
  • Do these data provide insights that have not been
    available before?
  • Are these data consistent with data already
    collected?
  • Do these data confirm or conflict with our
    existing profile of students or programs?
  • What other data should be investigated in light
    of the new profile?

70
Suggested Resources for Communicating PVAAS
71
Communication Resource
  • See PVAAS Key Communication Messages NEW!
  • Key messages if you get a call from the
    press/media
  • Or, if your IU wants to proactively assist
    districts, charter schools, full-time CTCs
  • District Press Release Template
  • Formal Press Release in development by PDE/PVAAS
    Statewide Core Team

72
PDE PVAAS Website, New Resources Professional
Development Opportunities, Intermediate Unit
Supports
73
Suggested Resources
  • Guide to PVAAS Public Reporting
  • PVAAS Crosswalk
  • PVAAS Key Communication Messages
  • PVAAS Public Reporting Site Overview
  • PVAAS Evaluating Growth Projecting Performance

74
  • PDE PVAAS Webpage
  • Help Menus
  • PowerPoint Presentations
  • with Trainer Notes
  • Resource Materials
  • Video Clips
  • District Case Studies

75
Suggested Resources
  • Archived webinars and PowerPoint presentations
    detailing the public reporting site
  • Coming Soon! Podcasts
  • Introduction to PVAAS
  • Value-Added Reporting
  • School Search

76
PVAAS Help Menus onPublic Reporting
77
QuestionsPVAAS Materials or Statewide
Implementationpdepvaas_at_iu13.org717-606-1911
PVAAS Report Web Sitehttps//pvaas.sas.com
78
www.pde.state.pa.us
333 Market Street
Harrisburg, PA 17126
About PowerShow.com