The future of Assessments Lessons learned internationally

1
The future of AssessmentsLessons learned
internationally
Washington, 9 March 2010 Andreas
SchleicherHead, Indicators and Analysis
DivisionOECD Directorate for Education
2
The future of assessments
Or the Alchemists Stone?
The Holy Grail?

• Know why you are looking
• You cannot improve what you cannot measure
• The yardstick for success is no longer just
  improvement by national standards but the best
  performing education systems globally
• Know what you are looking for
• A new assessment culture
• Responsive to changing skill requirements
• Capitalising on methodological advances
• Not sacrificing validity gains for efficiency
  gains
• Know how you will recognise it when you find it
• Gauging predictive validity
• Impact on improving learning and teaching
• Implications and lessons learned .

3
The old bureaucratic system The modern enabling system
Hit and miss ? Universal high standards
Uniformity ? Embracing diversity
Provision ? Outcomes
Bureaucratic look-up ? Devolved look outwards
Talk equity ? Deliver equity
Prescription ? Informed profession
Conformity ? Ingenious
Curriculum-centred ? Learner-centred
Interactive ? Participative
Individualised ? Community-centred
Delivered wisdom ? User-generated wisdom
Management ? Leadership
Public vs private ? Public with private
Culture as obstacle ? Culture as capital



4
A world of change higher education
Expenditure per student at tertiary level (USD)
Cost per student
Graduate supply
Tertiary-type A graduation rate
5
A world of change higher education
Expenditure per student at tertiary level (USD)
United States
Cost per student
Finland
Graduate supply
Tertiary-type A graduation rate
6
A world of change higher education
Expenditure per student at tertiary level (USD)
Australia
Finland
United Kingdom
Tertiary-type A graduation rate
7
A world of change higher education
Expenditure per student at tertiary level (USD)
Tertiary-type A graduation rate
8
A world of change higher education
Expenditure per student at tertiary level (USD)
Tertiary-type A graduation rate
9
A world of change higher education
Expenditure per student at tertiary level (USD)
Tertiary-type A graduation rate
10
A world of change higher education
Expenditure per student at tertiary level (USD)
Tertiary-type A graduation rate
11
A world of change higher education
Expenditure per student at tertiary level (USD)
Tertiary-type A graduation rate
12
A world of change higher education
Expenditure per student at tertiary level (USD)
United States
Australia
Finland
Tertiary-type A graduation rate
13
Latin America then
GDP/pop 1960 Years schooling
Asia 1891 4
Sub-Saharan Africa 2304 3.3
MENA 2599 2.7
Latin America 4152 4.7
Europe 7469 7.4
Orig. OECD 11252 9.5
Hanushek 2009
14
Latin America then and now
GDP/pop 1960 Years schooling
Asia 1891 4
Sub-Saharan Africa 2304 3.3
MENA 2599 2.7
Latin America 4152 4.7
Europe 7469 7.4
Orig. OECD 11252 9.5
GDP/pop 1960 Years schooling Growth 1960-2000 GDP/pop 2000
Asia 1891 4 4.5 13571
Sub-Saharan Africa 2304 3.3 1.4 3792
MENA 2599 2.7 2.7 8415
Latin America 4152 4.7 1.8 8063
Europe 7469 7.4 2.9 21752
Orig. OECD 11252 9.5 2.1 26147
Hanushek 2009
15
Latin America then and now
Why quality is the key
GDP/pop 1960 Years schooling Growth 1960-2000 GDP/pop 2000 PISAscore
Asia 1891 4 4.5 13571 480
Sub-Saharan Africa 2304 3.3 1.4 3792 360
MENA 2599 2.7 2.7 8415 412
Latin America 4152 4.7 1.8 8063 388
Europe 7469 7.4 2.9 21752 492
Orig. OECD 11252 9.5 2.1 26147 500
Hanushek 2009
16
Know what you are looking for
The Holy Grail was a well-described object, and
there was only one true grail
17
Schooling in the medieval age The school of the
church
18
Schooling in the industrial age Uniform
learning
19
Schooling in the industrial age Uniform
learning
The challenges today Universal quality Motivated
and self-reliant citizens Risk-taking
entrepreneurs, converging and continuously
emerging professions tied to globalising contexts
and technological advance
20
How the demand for skills has changedEconomy-wide
measures of routine and non-routine task input
(US)
Mean task input as percentiles of the 1960 task
distribution
The dilemma of assessments The skills that are
easiest to teach and test are also the ones that
are easiest to digitise, automate and outsource
(Levy and Murnane)
21
Changing skill demands

• The great collaborators and orchestrators
• The more complex the globalised world becomes,
  the more individuals and companies need various
  forms of co-ordination and management
• The great synthesisers
• Conventionally, our approach to problems was
  breaking them down into manageable bits and
  pieces, today we create value by synthesising
  disparate bits together
• The great explainers
• The more content we can search and access, the
  more important the filters and explainers become

22
Changing skill demands

• The great versatilists
• Specialists generally have deep skills and narrow
  scope, giving them expertise that is recognised
  by peers but not valued outside their domain
• Generalists have broad scope but shallow skills
• Versatilists apply depth of skill to a
  progressively widening scope of situations and
  experiences, gaining new competencies, building
  relationships, and assuming new roles.
• They are capable not only of constantly adapting
  but also of constantly learning and growing
• The great personalisers
• A revival of interpersonal skills, skills that
  have atrhophied to some degree because of the
  industrial age and the Internet
• The great localisers
• Localising the global

23

• Education today needs to prepare students
• to deal with more rapid change than ever
  before
• for jobs that have not yet been created
• using technologies that have not yet been
  invented
• to solve problems that we dont yet know will
  arise
• Its about new
• Ways of thinking
• involving creativity, critical thinking,
  problem-solving and decision-making
• Ways of working
• including communication and collaboration
• Tools for working
• including the capacity to recognise and exploit
  the potential of new technologies
• The capacity to live in a multi-faceted world as
  active and responsible citizens.

24
Mathematics in PISA
The real world
The mathematical World
Making the problem amenable to mathematical
treatment
A model of reality
A mathematical model
Understanding, structuring and simplifying the
situation
Using relevant mathematical tools to solve the
problem
A real situation
Validating the results
Mathematical results
Real results
Interpreting the mathematical results
25

• By whom?
• Evaluators
• Users of feedback
• Evaluation agencies

National assessment systems differ
Student assessment


Classroom

• How? Methods and procedures, mix of criteria and
  instruments
• Mapping of feedback to different units

Teacher appraisal

• For what? E.g.
• Accountability
• Improvement

Who is assessed
School
School evaluation
System
System assessment

• What?
• Inputs
• Processes
• Outcomes

• With whom?
• Agents involved

26
Assessment culturesPutting the pressure on top
of the education system is the easy part,
building capacity is harder
Participative/internal
Interactive, reflective, critical friend
Survey
Summative
Formative
Standardised assessment
Inspectorate
Administrative external
27
Participative / internal
Formative classroom-based assessments(e.g.
Europe, Asia)
Efficiency gains
Validity gains
Large scale and high-stakes summative
assessments, typically multiple-choice to
contain costs(US, England, Latin America)
Large scale and low-stakes assessments,
sample-based administration allows for complex
task types (e.g. Northern Europe, Scotland,
PISA)
Administrative / external
28
but there are global trends

• Understanding what the assessment reveals about
  students thinking to shape better opportunities
  for student learning
• Responding to assessments can enhance student
  learning if tasks are well crafted to incorporate
  principles of learning
• Capitalise on improved data handling tools and
  technology connectivity to combine formative and
  summative assessment interpretations for a more
  complete picture of student learning

• Multi-layered, coherent assessment systems from
  classrooms to schools to regional to national to
  international levels that
• Support improvement of learning at all levels of
  the education system
• Are largely performance-based
• Make students thinking visible and allow for
  divergent thinking
• Are adaptable and responsive to new developments
• Add value for teaching and learning by providing
  information that can be acted on by students,
  teachers, and administrators
• Are part of a comprehensive and well-aligned
  continuum, communicate what is expected and hold
  relevant stakeholders accountable .

• Integrate, synthesize and creatively apply
  content knowledge in novel situations
• Activate students as owners of their own learning
  and activate students as learning resources for
  one another

29
Know how you will recognise it when you find it
The Alchemists stone was to be recognised by
transforming ordinary metal into gold
30
Increased likelihood of postsec. particip. at age
19/21 associated with PISA reading proficiency at
age 15 (Canada)after accounting for school
engagement, gender, mother tongue, place of
residence, parental, education and family income
(reference group PISA Level 1)
Odds ratioCollege entry
School marks at age 15
PISA performance at age 15
31
Relationship between test performance and
economic outcomesAnnual improved GDP from
raising performance by 25 PISA points
Percent addition to GDP
32
Increase average performance by 25 PISA points
(Total 115 trillion )
bn
33
Implications and lessons learned
The medieval Alchemists followed the dictates of
a well-established science but that was built on
wrong foundations The search for the Holy Grail
was overburdened by false clues and cryptic
symbols
34
From assessment-inhibited practice towards
outcome driven reform
Strong focus on processes
Integrated quality management
Good willand trust
Weak outcome-based management
Strong outcome-based management
External control, uninformed prescription
Deprivation
Weak focus on processes
35
Some criteria used in the world

• Coherence
• Built on a well-structured conceptual basean
  expected learning progressionas the foundation
  both for large scale and classroom assessments
• Consistency and complementarity across
  administrative levels of the system and across
  grades
• Comprehensiveness
• Using a range of assessment methods to ensure
  adequate measurement of intended constructs and
  measures of different grain size to serve
  different decision-making needs
• Provide productive feedback, at appropriate
  levels of detail, to fuel accountability and
  improvement decisions at multiple levels
• Continuity
• A continuous stream of evidence that tracks the
  progress of both individual students .

36
Designing assessments

• Assessment frameworks
• A working definition of the domain and its
  underlying assumptions
• Organising the domain and identifying key task
  characteristics that guide task construction
• Operationalising task characteristics in terms of
  variables
• Validating the variables and assessing the
  contribution they each make to understanding task
  difficulty
• Establishing an interpretative scheme .

37
Understanding learning progressions

• Learning targets
• Defining what mastery means for a given skill
  level
• Progress variables
• Delineate a pathway that characterise the steps
  that learners typically follow as they become
  more proficient
• Evaluation of students reasoning in terms of the
  correctness of their solutions as well as in
  terms of their complexity, validity and precision
• Levels of achievement
• Describing the breadth and depth of the learners
  understanding of the domain at a particular level
  of advancement
• Learning performances
• The operational definitions of what students
  understanding would look like at each of the
  stages of progress .

Wilson, ATC21S
38

Interest science Indicate curiosity in science
and science-related issues and endeavours Demonstr
ate willingness to acquire additional scientific
knowledge and skills, using variety of resources
and methods Demonstrate willingness to seek
information and have an interest in science,
including consideration of science-related
careers Support for science Acknowledge the
importance of considering different scientific
perspectives and arguments Support the use of
factual information and rational
explanation Logical and careful processes in
drawing conclusions
Knowledge of science Physical systems (structure
of matter, properties of matter, chemical changes
of matter, motions and forces, energy and its
transformations, energy and matter) Living
systems (cells, humans, populations, ecosystems,
biosphere) Earth and space (structures of the
earth system, energy in the earth system, change
in the earth system, earths history,
space) Technology systems (Concepts and
principles, science and technology) Knowledge
about science Scientific enquiry (purpose,
experiments, data, measurement, characteristics
of results) Scientific explanations (types,
rules, outcomes)
Identifying Recognising issues that can be
investigated scientifically Identifying keywords
in a scientific investigation Recognising the key
features of a scientific investigation Explaining
Applying knowledge of science in a
situation Describing or interpreting phenomena
scientifically or predicting change Using
evidence Interpreting scientific evidence and
drawing conclusions Identifying the assumptions,
evidence and reasoning behind conclusions

• Context
• - Personal
• Social/public
• Global

• Competencies
• Identify scientific issues
• Explain phenomena scientifically
• Use scientific evidence

• Knowledge
• Knowledge of science
• Knowledge about science

Attitudes -Interest in science -Support for
scientific enquiry -Responsibility

39

OECD Level 2
OECD Level 6
Identifying Recognising issues that can be
investigated scientifically Identifying keywords
in a scientific investigation Recognising the key
features of a scientific investigation Explaining
Applying knowledge of science in a
situation Describing or interpreting phenomena
scientifically or predicting change Using
evidence Interpreting scientific evidence and
drawing conclusions Identifying the assumptions,
evidence and reasoning behind conclusions
Students can demonstrate ability to understand
and articulate the complex modelling inherent in
the design of an investigation.
Students can determine if scientific measurement
can be applied to a given variable in an
investigation. Students can appreciate the
relationship between a simple model and the
phenomenon it is modelling.

• Context
• - Personal
• Social/public
• Global

• Competencies
• Identify scientific issues
• Explain phenomena scientifically
• Use scientific evidence

Students can draw on a range of abstract
scientific knowledge and concepts and the
relationships between these in developing
explanations of processes
Students can recall an appropriate, tangible,
scientific fact applicable in a simple and
straightforward context and can use it to explain
or predict an outcome.

• Knowledge
• Knowledge of science
• Knowledge about science

Attitudes -Interest in science -Support for
scientific enquiry -Responsibility
Students demonstrate ability to compare and
differentiate among competing explanations
by examining supporting evidence. They can
formulate arguments by synthesising evidence from
multiple sources.
Students can point to an obvious feature in a
simple table in support of a given statement.
They are able to recognise if a set of given
characteristics apply to the function of
everyday artifacts.

40
Some methodological challenges

• Can we sufficiently distinguish the role of
  context from that of the underlying cognitive
  construct ?
• Do new types of items that are enabled by
  computers and networks change the constructs that
  are being measured ?
• Can we drink from the firehose of increasing data
  streams that arise from new assessment modes ?
• Can we utilise new technologies and new ways of
  thinking of assessments to gain more information
  from the classroom without overwhelming the
  classroom with more assessments ?
• What is the right mix of crowd wisdom and
  traditional validity information ?
• How can we create assessments that are activators
  of students own learning ?

Wilson, ATC21S
41
High policy value
A real-time assessment environment that bridges
the gap between formative and summative
assessment .
Quick wins
Must haves
Examine individual, institutional and systemic
factors associated with performance
Extending the range of competencies through which
quality is assessed
Monitor educational progress
Measuring growth in learning
Low feasibility
High feasibility
Establish the relative standing of students and
schools
Assuming that every new skill domain is
orthogonal to all others
Money pits
Low-hanging fruits
Low policy value
42
Getting the sequencing right
Adequate ? Good
Poor ? Adequate
Good ? Great
Main focus of assessment

• Tackling underperformance

• World class performance.
• Continuous learning and innovation .

• Transparency .
• Spreading best practice

Role of government

• Regulating .
• Capacity-building

• Enabling
• Incentivising .

• Prescribing .
• Justifying

• Accommodating
• Evidence-based
• Adopting best . practice

• Implementing
• Accepting evidence
• Adopting minimum standards

Role of professions

• Leading
• Evidence-driven
• Achieving high reliability and innovation
  .

• Principled
• Strategic partnership

• Negotiated
• Pragmatic .

• Top-down
• Antagonistic .

Nature of relationship between government and
professions
Main outcomes

• Improvement in outcomes
• Reduction of public anxiety.

• Steady improvement
• Growing public satisfaction .

• Consistent quality
• Public engagement and co-production .

43
Thank you !

• www.oecd.org www.pisa.oecd.org
• All national and international publications
• The complete micro-level database
• email Andreas.Schleicher_at_OECD.org
• Twitter _at_SchleicherEDU
• and remember
• Without data, you are just another person with
  an opinion