What do international assessments measure: PISA

1
What do international assessments measure PISA
Raymond J. Adams
Washington DC, May 30 2008
This paper is intended to promote the exchange of
ideas among researchers and policy makers.  The
views expressed in it are part of ongoing
research and analysis and do not necessarily
reflect the position of the National Center for
Education Statistics, the Institute of Education
Sciences, or the U.S. Department of Education.
2
What is PISA?

• Programme for International Student Assessment
• Funded by the Organisation for Cooperation and
  Development (OECD)
• International data collection has been managed
  and led by ACER since its inception

3
Developing PISA

• 1995 OECD countries sought
• comparative assessment of yield of education
  systems
• a basis for defining educational standards
  cross-nationally
• 1997 OECD countries adopted strategy and
  financial framework to...
• monitor a broad range of curricular and
  cross-curricular outcomes...
• within a comparative framework established
  collaboratively by countries
• 1998 Contract let

4
Participation

• 2000
• OECD 28, non-OECD 4
• PISA, 11 non-OECD
• ? 170,000 students
• 2003
• OECD 30, non-OECD 12
• ? 250,000 students
• 2006
• OECD 30, non-OECD 27
• 14365 schools
• 398,750 students (representing a cohort of over
  22 million)
• 2009
• OECD 30, non-OECD 35 (plus Scotland)

5
Plus Dubai, Kazakhstan, Moldova
6
PISA survey cycle
7
PISA survey cycle
eg self-regulation, metacognitive
eg Reading Activities, motivation, interest,
engagement
8
PISA survey cycle
9
PISA survey cycle
10
PISA survey cycle
11
Separating policy and science
Two consortia commissioned to run PISA 2009 Core A
Australian Council for Educational Research
cApStAn Linguistic Quality Control
The German Institute for International
Educational Research
National Institute for Educational Research, Japan
Unit for the Analysis of Educational Systems and
Practices. University of Liege, Belgium
Westat, USA
12
Separating policy and science

• Core B
• Netherlands National Institute for Educational
  Measurement (CITO)
• University of Jyvaskyla
• University of Twente
• French Ministry of Education

13

• Sets policy objectives and priorities
• Sets budget
• Monitors adherence to policy objectives
• Guides analysis and reporting of results
• Establishes quality standards
• Adjudicates on breaches of quality standards

14
Consortia are commissioned to run PISA
15
Innovations in PISA

• International, intergovernmental ownership
• primary focus on public policy issues
• sustained commitment
• collaborative development
• Age not grade-based target population
• Substance
• not common denominator of national curricula
• not whether students have learned
• whether they can use what they have learned
• criterion-based reporting

16
PISAs Literacy Orientation

• Projective (rather than reflective) approach to
  instrument development
• Not specifically related to curriculum
• It is about authentic application of what is
  learned
• In contrast to assessments that are like
  instructional exercises
• A broadening, not narrowing notion

17
Literacy Orientation

• Domain definitions include two components
• each domain defined, in terms of knowledge and
  skills needed in adult life, not merely in terms
  of mastery of the school curriculum.
• emphasis is placed on the mastery of processes,
  the understanding of concepts and the ability to
  function in various situations related to real
  life.

18
Basic Methodology

• Target population 15-year-olds in school
• Students are spread over classes and grades
• Target sample sizes of 5250
• 150 schools of 35 students
• 50 schools required for national sub-entities
• Rotated two-hour written tests
• 13 booklets
• 3 domains (two minor and one major)
• About 200 items in total
• About 11 responses per item in each school
• Students respond to an average of 13 minor domain
  items and 30 major domain items
• Some students are not assessed in minor domains

19
Basic Methodology

• Student questionnaires
• School questionnaires
• Parent questionnaires
• No teacher questionnaires
• Students from multiple classrooms
• Focus is not limited to taught subjects

20
Focus of the Results

• Yield for well-defined populations and domains
• Reading, Mathematics, and Science every three
  years
• Sub-domains every nine years
• National sub-group performance
• Profiles encompassing broad domain definitions
• Literacy orientation, habits, behaviours,
  attitudes
• Trends
• Focus on comparisons of equity
• Eg Gender, minority status, socio-economic status
• Relations between national structures outcomes
• Eg Tracking, Assessment, funding and Decision
  making policies

21
Shall I slice the pizza into four or eight pieces?
Make it four ! I could never eat eight !