The Mastery Level Judgment Consistency Rate of a Rasch Model Based Standard Setting Method for Classroom Achievement Tests

1
The Mastery Level Judgment Consistency Rate of a
Rasch Model Based Standard Setting Method for
Classroom Achievement Tests
13th Biennial International Objective Measurement
Workshops April 5th 7th , 2006 University of
California, Berkeley, California, USA
In-Hee Choi (Korea National Open University)
Sun-Geun Baek (Seoul National University)
2
Table of Presentation

• Introduction
• Standard Setting Methods
• Research Questions
• Methodology
• Research Procedure
• Results
• Summary and Discussion

3
Introduction

• Criterion-referenced Test
• - based on an appropriate standard (criterion)
• - guarantees what examinee has achieved and
  what he or she is able to do actually

• The most important and difficult step in
  criterion-referenced test
• - deciding the cutoff score that
  differentiates sufficiently
  competent candidates from those who
  are not
• - various studies about standard setting by
  notable scholars, Ebel(1979), Angoff(1971),
  Jeager(1989), Nedelsky(1954) and et al.

4
Introduction

• However in Korea
• - There have not been many researches related
  to standard setting method (Kim, Baek Chae,
  1998 Yang, 2000).
• - In order to promote the implementation of
  criterion-referenced tests, a variety of studies
  has to be conducted.
• Such as,

• researches about the applicability of the
  established standard
• setting methods (ex. Angoff method)
• the comparison studies between established
  methods and
• new methods (ex. Rasch method)

5
Introduction

• Currently at middle schools in Korea
• - raw score itself is used as a standard
• ex) 80 mastery level raw score 80 points
  out of 100 points
• - very simple to set up mastery level of
  tests
• - however, also easy to be changed according
  to the difficulty level of the test, so low
  judgment consistency rate

6
Introduction

• An appropriate standard setting method in this
  study
• - judge students achievement level
  consistently regardless of the difficulty level
  of tests

• investigate whether the mastery level judgment
• consistency rate of a Rasch method for
  classroom
• achievement tests is relatively high or not
  with a
• raw-score method and an Angoff method

7
Standard Setting Methods - 1

• Raw-score Method
• - raw score itself is set as a cutoff score
• ex) when raw score 80 points out of 100 is set as
  a criterion for mastery group
• ? Easy test ? more students will get score over
  80 points
• ? more students are
  regarded as mastery group
• ? Hard test ? less students will get score over
  80 points
• ? less students are
  regarded as mastery group
• ? judgment about a same student is changed
  according to the
• difficulty level of the test

8
Standard Setting Methods - 1
Mastery group
80 points
Non-mastery group
ltTest Agt - easy test
ltTest Bgt - hard test

• ? low mastery level judgment consistency rate!!

9
Standard Setting Methods - 2

• Angoff Method
• - based on judges judgment about each item of
  the test
• 1) judge participated in judge training
• 2) to decide how many (out of 100) prescribed
  minimally mastery level examinees would likely
  answer example items correctly
• 3) to set up the first standard from a summation
  and averaging of those predictions
• 4) to repeat the second step with referenced to
  the first result, and then the second standard is
  set up
• 5) after iterative ratings of total three times,
  the final standard is established
• The adequacy of a standard
• ? whether judges accurately conceptualize the
  minimal mastery level of candidates.

10
Standard Setting Methods - 3

• Rasch Method
• - based on Rasch model
• - based on judges judgment about each item of
  the test
• - recently developed method (Stone, 2001)
• ? necessary to examine usability at classroom
  achievement tests

11
Standard setting method - 1

• Procedure of Rasch Method
• 1) judge participated in judge training
• 2) to determine whether each item reflects
  essential content which minimally competent
  students should necessarily solve.
• ? essential item group
• 3) to set up a judges baseline criterion point
  mean item difficulty of the essential group
• 4) to reflect the mastery level, mastery level
  which transformed into a logit measure (i.e.
  801.4logit) was added to a base line
• A judges final standard a baseline
  criterion point mastery level
• Final standard of the test the average of
  each judges final standard

12
Research Questions

• Is there significant difference in the mastery
  level judgment consistency rate between Rasch
  method and raw-score method?
• Is there significant difference in the mastery
  level judgment consistency rate between Rasch
  method and Angoff method?

13
Methodology

• Subjects of the Study

• Achievement Tests Takers
• - 422 8th grade students at 12 classrooms in 4
  middle schools in Seoul, Korea

• Judge Panels
• - 7 science teachers (6 in middle school and
  1 in high school)

14
Methodology

• Instruments of the Study

• Two Science Achievement Tests (Test 1 and Test
  2)
• - developed with different difficulty levels
• (test 1 was intended to be easier that
  test 2 by and large)
• - based on Korean National Science Curriculum
  for 8th grade
• - composed of 25 multiple-choice items
  respectively
• - total test scores ranged between 0 and 100
  respectively
• - Crobachs alpha ? Test 1 0.84 , Test 2
  0.82

15
Methodology

• Item 12 in Test 1 (An example)
• 12. Which is the right match-up between the
  hormone that is released from the thyroid gland
  and its effect?
• ? Estrogen - promotes female secondary sex
  characteristics
• ? Thyroid Hormone - regulates metabolic
  rate
• ? Vasopressin - causes kidney to withhold
  water
• ? Epinephrine - regulates metabolic rate
• ? Insulin - facilitates to secrete
  Pancreatic Enzymes

16
Methodology

• Item 4 in Test 2 (An example)
• The below picture shows the endocrine glands
  in the human body. Answer the next question.

4. What is the endocrine gland of hormones which
does opposite effect? ? A, Pituitary ?
B, Thyroid ? C, Adrenal Medulla ? D,
Pancreas ? E, Ovaries
17
Methodology

• Unidimensionality Check
• - Test 1

18
Methodology

• Unidimensionality Check
• - Test 2

19
Research Procedure
Administered two science achievement tests to 8th
grade students
Judge training (Angoff method exercise Rasch
method exercise)
Standard setting (Raw-score method, Angoff method
Rasch method)
Grouped students according to the standards
established by three method in two
tests
Investigated the mastery level consistency rate
of those three methods
20

• Mastery Level Consistency Rate

Standard 2
Test 2
Standard 1
Test 1

• A, D Consistency Group
• B, C Inconsistency Group
• Consistency Rate ( A D ) / ( A B C D
  )?100

21
Results

• Descriptive Statistics

- The summary of the 422 students raw score
? Test 1 was much easier than Test 2
22
Results

• Established Standards by Three Methods

• Raw-score Method

23
Results
2) Angoff Method
24
Results

• Rasch Method
• 1-1. Selecting the essential items in
  Test 1

25
Results
1-2. Selecting the essential items in
Test 2
26
Results
2. Standard Setting
? 81.8
? 71.8
27
Results

• Grouped Students According to the Standards
• 1) Raw-score Method

80
Test 2
80
Test 1
Consistency Rate 54.7
28
Results

• 2) Angoff Method

71.23
Test 2
86.8
Test 1
Consistency Rate 75.4
29
Results

• 3) Rasch Method

1.03(logit)
Test 2
1.73(logit)
Test 1
Consistency Rate 71.3
30
Results

• Investigated the Mastery Level Consistency Rate

1) Rasch Method vs. Raw-score Method
Chi-square4.916, plt0.01

• There was statistically significant difference
  between
• Rasch methods consistency rate and
  raw-score methods rate.

31
Results
2) Rasch Method vs. Angoff Method
Chi-square1.751, pgt0.05

• There was no statistically significant
  difference between
• Rasch methods consistency rate and
  Angoff methods rate.

32
Summary and Discussion

• Summary
• - This study shows that the mastery level
  judgment consistency rate of Rasch method for
  classroom achievement tests is relatively higher
  than raw-score methods rate and as high as
  Angoff methods rate.
• - Moreover, there are additional advantages of
  Rasch method.
• ex) Item-person Map
• ? the difficulty of items and the
  distribution of persons
• can be represented visually.

33
Summary and Discussion
1) Item-person Map in Test 1
2) Item-person Map in Test 2
person
item
person
item
Cutoff score
Cutoff score
34
Summary and Discussion

• Discussion
• - Rasch method may supplement or replace both
  raw-score method and Angoff method for classroom
  achievement tests.
• - However, there are some restrictions because
  of the limited number of students and judges
  involved in the study and restricted items.
• - For further study,
• ? more examinees and judges are required.
• ? the comparison studies with other
  established standard setting
• methods, such as Ebel method, Jeager
  method and Nedelsky
• method as well as Angoff method and
  raw-score method should
• be conducted.

35
Thank you very much for your attention
! Sun-Geun Baek In-Hee Choi E-mail
dr100_at_snu.ac.kr Tel [phone] Address 599
Kwanak-Ro, Kwanak-Gu, Seoul, 151-748, Korea