Computerized Adaptive Testing: developments in theory and practice - PowerPoint PPT Presentation

1 / 26
About This Presentation
Title:

Computerized Adaptive Testing: developments in theory and practice

Description:

... Testing: Theory and Practice', Wim J. van der Linden & Cees A.W. Glas (2002). Linear models for optimal test design' Wim J. van der Linden (2005) ... – PowerPoint PPT presentation

Number of Views:311
Avg rating:3.0/5.0
Slides: 27
Provided by: TOL50
Category:

less

Transcript and Presenter's Notes

Title: Computerized Adaptive Testing: developments in theory and practice


1
Computerized Adaptive Testing developments in
theory and practice
Prof. dr. Cees Glas
2
Background information
  • Research line Computerized Adaptive Testing
  • Computerized Adaptive Testing Theory and
    Practice, Wim J. van der Linden Cees A.W. Glas
    (2002).
  • Linear models for optimal test design Wim J.
    van der Linden (2005)
  • Research line in Educational Surveys
  • PISA Cycle 2009, Core B Background questionnaires

3
Adaptive Testing
  • Target the difficulty of the items to the ability
    level of the students
  • Motives
  • Optimization of measurement precision
  • Shorter tests without loss of precision
  • More interesting tests for students
  • More informative tests for students and teachers
  • Flexibility testing on demand

4
Adaptive Testing is based on IRT(Item response
Theory Models)
5
Motive for using IRT
  • Possibility of using incomplete designs
  • Not everybody needs to respond to the same set of
    items
  • Yet measurement is on a common scale
  • Possibility of creating efficient statistically
    optimal designs

6
New developments in IRT
  • Response formats polytomous items, continuous
    responses
  • Multidimensional models
  • Item shells, item cloning
  • Modeling variability in item parameters
  • Cognitive models

7
Item and Test Information
  • Item and test information function
  • In CAT items are selected to maximize information
    at the estimated ability of examinee.

8
Adaptive Item Selection
Information
9
Adaptive Item Selection Contd
Information
Item 1
10
Adaptive Item Selection Contd
Test
Item 1
Item 2
Information
11
Adaptive Item Selection Contd
Test
Information
Item 3
Item 2
Item 1
12
Item and Test Information Contd
Test
Information
Items
Ability
13
Item parameters must be known through
pre-testing Item Bank Calibration
  • Problem how to collect proper pre-test data for
    precise estimation of item parameters
  • On-line calibration

14
CAT with Content Constraints
  • Adaptive individualized testing
  • Psychometrically optimal
  • Test content specifications
  • Psychometrically optimal within content
    constraints and practical constraints
  • Discrete optimization problem

15
CAT with Content Constraints
  • Law School Admission Test
  • content constraints
  • item type constraints
  • word count constraints
  • answer key constraints
  • gender / minority orientation
  • clusters of items (testlets)
  • some items contain clues to each other

16
CAT with Content Constraints
  • Constraints are imposed by Linear - Programming
    techniques
  • For every item i a variable is defined

17
Test assembly model
Item i is selected for the test or not.
18
Test assembly model
Item i is selected for the test or not.
At most 5 items on statistics
Items 12 and 35 contain clues to each other
Time available is 60 minutes
19
Test assembly model
Maximize information in the test
Item i is selected for the test or not.
At most 5 items on statistics
Items 12 and 35 contain clues to each other
Time available is 60 minutes
20
Exposure Control
  • For reasons of efficiency
  • For reasons of security

21
CAT without Item Exposure Control
Exposure Rate
rmax.25
Item
22
CAT with Sympson-Hetter
Exposure Rate
rmax.25
Item
23
Background information
  • Packages for IRT calibration
  • Bilog, Multilog, Parscale, Testfact
  • Conquest, OPLM
  • Commercial software packages for discrete
    optimization
  • CPLEX (ILOG)
  • AIMMS modeling software
  • OPL Studio

24
CAT in Practice
  • High stakes tests
  • ETS, ACT, LSAC
  • Problems with item bank security
  • Public item banks lead to undesirable strategies
  • New item types may be a solution

25
CAT in Practice
  • Low stakes tests
  • Selection and Placement, pupil monitoring
    systems less problematic
  • Cognitive and diagnostics models in development
  • Sub-optimal item selection
  • Cito CAT for young children

26
Conclusion
  • Computer adaptive testing gives a lot of
    possibilities of optimizing a survey study
  • But it is not simple
  • Logistic task software and expertise
  • Scientific task measurement and survey experts
Write a Comment
User Comments (0)
About PowerShow.com