Psychometric Properties of the AnswerUntilCorrect Item Format

1
Psychometric Properties of the Answer-Until-Correc
t Item Format

• José Muñiz, Fernando Menéndez, Eduardo
  García-Cueto, and Luis M. Lozano
• University of Oviedo, Spain

2
Items

• The basic building block of a good test
• Many types of items
• Influence of Information Technology

3
New Technologies impact(Bennet, 1999)

• Design of tests
• Generation of items
• Presentation of items
• Scoring of test
• Remote assessment

4
Dimensions of Item Innovations(Parshall et al.,
2002)

• Item format
• Response action
• Media inclusion
• Level of interactivity
• Scorig methods

5
Next decade hot issues

• Computerized testing
• New items (multimedia)
• Conventional versus Authentic evaluation
• Consequential validity
• Item banking
• Automatic item construction
• Global assessment Cross cultural issues
• Automatic subjects evaluation

6
Item Formats enhanced by the Computer

• Multiple correct answers
• Short answer
• Extended answer (essays)
• Highlighting text
• Ranking
• Numerical responses
• Drag and Drop format
• Answer-Until-Correct
• Over 50 formats have been identified in the
  literature (Zenisky and Sireci, 2002)

7
Multiple-Choice formatStill alive

• Strengts of the format
• Easy to score and simple to use Economic
• Adaptable to most assessment situations
• Flexible to be used in most content areas
• Criticisms
• Based on Selected-response
• Only one correct answer to every problem
• Inhibits creativity
• No valid to evaluate higher level thinking

8
Answer-Until-Correct variation

• Subjects keep answering until they get the right
  answer
• Two additional advantages
• Inmediate reinforcement
• Improve learning

9
Objectives

• Two main questions to be answered
• 1. Do examinees prefer this format to the
  classic Multiple-Choice format?
• 2. Does this format improve the psychometric
  properties of classical Multiple-Choice items?

10
Objective 1 Attitudes

• A questionnaire was developed in order to assess
  this objective
• 9 items Likert-type
• Likert-scale rank from 0 to 10
• Sample
• Secondary school students (Age 12-18)
• N 242
• Administration mode Computer

11
Questionnaire

• Receiving inmediate information about right and
  wrong answers of the items is pleasant
• The inmediate information about right and wrong
  answers makes easier the learning of the subject
• Receiving information inmediately about the right
  and wrong answers gives me greater calm
• This kind of tests generate less anxiety than the
  usual paper and pencil tests
• This kind of tests objectively assess the
  knowledge of the examinees

12
Questionnaire

• 6. I prefer this kind of tests to those classic
  ones which dont give you information about the
  right and wrong answers
• 7. If in the exams to enter the university I
  could choose, I would prefer this type of test
  better than the classic ones
• 8. I prefer computer testing to paper and pencil
  tests
• 9. I prefer Multiple-Choice tests to essay type
  tests

13
Results

• Items Mean SD
• 1 7.83 2.35
• 2 7.92 2.42
• 3 7.28 2.31
• 4 8.21 2.61
• 5 7.07 2.55
• 6 7.40 2.73
• 7 7.38 2.93
• 8 7.79 2.67
• 9 7.76 2.72
• -----------------
• Coefficient Alpha 0.74

14
Factor Analysis Scree plot
15
Factor Analysis

• Items Factor 1 Factor 2
• 1 --- 0.72
• 2 --- 0.63
• 3 --- 0.76
• 4 0.52 ---
• 5 0.47 ---
• 6 0.43 --- Var. Exp. 47.65
• 7 0.81 --- Rot Varimax
• 8 0.79 --- Ext. Kaiser
• 9 0.66 ---

16
General conclusions (Objective 1)

• Positive attitudes towards the Answer-Until-Correc
  t item format
• Students prefer computer testing to paper and
  pencil tests
• Multiple-choice items are preferred to
  constructed response formats

17
Objective 2

• Does the Answer-Until-Correct format improve the
  psychometric properties of classical dichotomous
  Multiple-Choice items?

18
Test used

• Subjects N 431
• Content area
• History
• Items
• 100 items
• Multiple-Choice five options
• Administration
• Computer

19
Scoring procedure

• Dichotomous Score
• Right 1.00
• Wrong 0.00
• Answer-Until-Correct
• First choice right 1.00
• Second choice right 0.75
• Third 0.50
• Fourth 0.25
• Fith 0.00

20
Test Difficulty
21
Reliability and Validity

• Alpha Factors Var.Ex.
• Dichotomous 0.88 39 63.3
• A-U-C 0.87 39 63.5
• -------------
• Eigenvalues gt 1

22
Correlations Dichotomous-AUC

• Subjects scores 0.96
• Items Difficulty 0.94

23
Future directions

• Test alternative scoring systems for AUC
• 1.00 0.75 0.50 0.25 0.00
• 1.00 0.50 0.25 0.00 0.00
• 1.00 0.25 0.00 0.00 0.00
• Improve unidimensionality of the test
• Increase the number of subjects
• Fit IRT Partial Credit Models
• Use CAT versions

24

• Thank you very much for your attention
• E-mail jmuniz_at_uniovi.es