Computerbased Assessment of Science CBAS in the Programme for International Student Assessment PISA

1
Computer-based Assessment of Science (CBAS)in
theProgramme for International Student
Assessment (PISA)
Ron Martin Marten Koomen
2
Programme for International Student
AssessmentOverview (1)

• Science, Maths and Reading Assessment
• Target population 15-years-olds attending school
• 3 year cycle (2000, 2003, 2006 )
• Science is the major domain for PISA 2006
• 2 hour test, 30 minutes questionnaire
• 57 Countries Participating in PISA 2006

3
Programme for International Student
AssessmentOverview (2)

• Computer-based Assessment of Science
• 13 Countries, as an international option
• 1-hour paper-based plus 1-hour computer-based
• 30 min Student Questionnaire

Data allows investigation of differences in
performance on computer-based and paper-based
4
CBAS 2006 Introduction (1)

• Countries Participating in CBAS
• Australia
• Austria
• Bulgaria
• Chinese Taipei
• Denmark
• Iceland
• Ireland
• Japan
• Norway
• Portugal
• Republic of Korea
• Slovak Republic
• Russian Federation

5
Key StakeholdersandKey Processes
6
Computer-based Assessmentwhat is involved
All functions critically linked by platform
Test development
Sampling
platform design decisions depend on domain
Translation Adaptation
Less scope for Bricolage, Making do with what is
available
Test Material Preparation
Test Administration
Capabilities for National Centres not as
developed as for paper-based
Coding
Data Submission
Early design choices will affect future
flexibility
7
Key Stakeholders and Key Processes

• Test Development
• Data
• Hardware
• Translation and Verification
• How much should ICT literacy affect the
  assessment
• Test Administration

8
Framing the ItemThe CBAS solution
9
Title
Introduction to Stimulus Text
Stimulus Picture With some text
Question
Response
Coding Guide
10
Title
Introduction to Stimulus Text
Stimulus Picture With some text
Question (optional)
Response
Coding Guide
11
Title
Introduction to Stimulus Text
Stimulus Picture With some text And Dynamic
Response
Coding Guide
12
Title
Introduction to Stimulus Text

• CBAS Item Development Constraints
• - Omit Language Elements in Film
• Omit Narration
• (expensive to Translate)
• Remove signs from film
• (impossible to Translate)

Coding Guide
13

• Stimulus
• Video
• Flash (can have text)

14
Software Developers
Content Developers
Animators Film Crew
Flash Developers Action Script
15
A CBAS Item
Text
Media
Text
16
(No Transcript)
17
Item XML
Display Screen
ltItem xmlnsxs"http//www.w3.org/2001/XMLSchema-i
nstance"gt ltItemInfo code"SC002Q02"
title"Sound Detective" /gt ltTitlegt
ltTextElement style"title"gtlt!CDATASound
Detective (DM002Q02)gtlt/TextElementgt lt/Titlegt
ltStimulusgt ltTextElement height"20"
style"stimulus" width"600"gtlt!CDATAThe monitor
screen displays an image of an artery in
Anastasias neck. The colour changes show blood
moving at different speeds inside the
artery.gtlt/TextElementgt ltComponentgt
ltMovieElementgtSC002-movie-02.avilt/MovieElementgt
lt/Componentgt lt/Stimulusgt ltResponseAreagt
ltTextElement height"10" style"stimulus"
width"600"gtlt!CDATAWhich one of the following
statements explains the regular changes in the
blood flow in Anastasias artery?gtlt/TextElementgt
ltSimpleMultipleChoicegt
ltAlternativeGroup layout"VERTICAL"gt
ltAlternativegt ltTextElement height"10"
style"stimulus" width"600"gtlt!CDATAThe artery
is squeezed when she swallows.gtlt/TextElementgt
lt/Alternativegt ltTextElement
style"stimulus"gtlt!CDATAOption 4 is the
key. Acknowledgement Prof. R.N. Gibson,
Department of Radiology, Royal Melbourne
Hospital. Australiagtlt/TextElementgt
lt/CodingGuidegt lt/Itemgt
18
Display Screen
Item text and layout file(Small - 20kb)
Media Files(Large 2Meg)
19
Translating items
20
ACER (Melbourne)Releases English Source Version
21
Minimising ICT Literacy Requirement
22
Assessment of Sciencenot an assessment of ICT
literacy
Source PISA 2003 data www.oecd.org
23
Reducing ICT literacy requirement in CBAS

• minimise interactions types
• clicking radio buttons
• click navigation
• drag and drop
• Click, play, stop, pause buttons
• other considerations
• No keyboard responses
• No Scrolling
• No hyperlinks

Demonstrate Practice Session
24
Priorities for Test Development

• Adding value to science assessment
• (beyond that available in the paper-based test)
• Production of items consistent with the
  conceptual framework for PISA 2006
• Comparative demonstration of computer-based
  assessment with the paper-based test.

25
Adding Value

• By
• reducing reading and expression load
• motivating students for the assessment task
• linking dynamic contexts with data interpretation
  
• enabling student interaction with the media
• allowing assessment of aspects of science not
  available in paper-based forms

26
Constraints

• Unit context requirements (stimulus on each page)
• Avoiding assessing observational literacy instead
  of scientific literacy in easier items
• Limiting reading load in explanations of
  interactive behaviour
• Achieving PISA Framework requirements for
  various classifications of science literacy
  assessment

27
Reducing Reading Load

• By
• conveying some of the information graphically
• Resulting in
• lessening the dependency of assessment of science
  literacy on reading literacy
• Outcome
• Some change in gender performance compared to the
  paper-based trial
• SC001

28
Motivation

• By
• providing interesting dynamic stimulus
• Resulting in
• strongly focussed attention on the assessment
  task
• Outcome
• Survey data indicated students preferred a mixed
  computer-based, paper-based test over a solely
  paper-based test
• SC002 DM006

29
Enabling Data Collection by Observation

• By
• using dynamic scenarios
• Resulting in
• assessment of aspects of science not available in
  paper-based tests
• Outcome
• A difference between the construct(s) measured by
  computer-based and paper-based tests resulting in
  a different scale.
• DM003

30
Assessing Research Strategies

• By
• student interaction in dynamic scenarios
• Resulting in
• assessment of the strategies used to gather
  information or test an hypothesis
• Outcome
• Assessment of a broader range of scientific
  literacy but a difference between the
  construct(s) measured by computer-based and
  paper-based tests.
• DM004