The Item Bias Detection of the Reading Tests - PowerPoint PPT Presentation


Title: The Item Bias Detection of the Reading Tests


1
The Item Bias Detection of the Reading Tests and
the Development of the Item Bank Software Used
for English Reading Courses at King Mongkuts
University of Technology North Bangkok
Supalak Nakhornsri, Ph.D.
supalak_nak_at_yahoo.com
Department of Languages, King Mongkuts
University of Technology North Bangkok, Thailand
2
Outline of the Presentation
  • Background of the study
  • Objectives
  • Research Questions
  • Definitions of Terms
  • Methodology
  • Test Development
  • Item Bias Detection
  • Conclusion
  • Results
  • Item Bank Software

3
Background of the Study
  • Trying out a test
  • The results can be used to make changes to
    improve the potential usefulness of the test.
  • (Bachman, 2004)
  • For classroom teachers, it is not advisable to
    try out a test with the same students who will
    take it.
  • (Bachman, 2004)

4
Background of the Study
  • Item Analysis

The essential part of analyzing the results of
tests for improving their usefulness. (Brown,
2004)
  • Classical Item Analysis (IA) calculating
    descriptive statistics for individual items
  • Item Response Theory (IRT)
  • a more sophisticated procedure for estimating
    the statistical characteristics of item.
  • (Bachman, 2004)

5
Background of the Study
  • IA vs. IRT
  • IA
  • Subpopulation dependent
  • Implicitly averaged across all ability levels
  • (Bachman, 1990)
  • IRT
  • Independent of group of examinees used
  • Independently estimated test taker ability
  • Known precision of ability estimates

6
Background of the Study
  • Bias Vs. Unfairness
  • Unfairness Concerning with test administration
  • Bias A type of invalidity
  • (Shepard, 1982)

7
Background of the Study
  • Differences among three methods

IA IRT
Delta-plot
Chi-square
Three-parameter model
Transforming item difficulty into Delta value
and plot the value onto the Bivariate Graph
(Angoff, 1982)
Measuring the frequency of correct or incorrect
responses in each ability groups (Shepard,
1982)
The presence of some characteristic of an item
that results in differential performance for
individuals of the same ability but from
different ethnic, sex, cultural, or religious
groups. (Hambleton et.al, 1991.)
8
Background of the Study
  • Delta-plot

9
Background of the Study
  • Chi-square

Value df Asymp. Sig. (2-sided)
Pearson Chi-Square 5.799(b) 1 .016
Continuity Correction(a) 4.602 1 .032
Likelihood Ratio 5.597 1 .018
Fisher's Exact Test
Linear-by-Linear Association 5.729 1 .017
N of Valid Cases 83
10
Background of the Study
  • IRT Three-parameter model

11
Background of the Study
  • Sources of Bias
  • Gender
  • In some languages there are differences between
    the vocabulary items used by women and men.
  • Women tend to use more of the standard forms than
    men do.
  • (Homes, 2001)

12
Background of the Study
  • Background Knowledge
  • Schema Theory
  • Readers can fully comprehend what they read when
    their schema or prior knowledge, is activated by
    the words on the page.
  • (Chandavimol, 1998)

13
Background of the Study
  • Sources of Bias Studied
  • Gender male and female
  • Prior Knowledge Faculties they are studying.

14
Background of the Study
  • Problem of the Assessment of Reading Courses
  • Regarding Reading I course, it is an elective
    course for second fourth year undergraduate
    students, King Mongkuts University of Technology
    North Bangkok.
  • The same reading tests are used to all students
    to evaluate their achievement. Therefore, the
    important judgment on students grade is up to
    both midterm and final tests.
  • The quality of test is necessary to be evaluated.
    Although both Classical IA and IRT can help
    analyze the quality of test items, the issue of
    test bias cannot be neglected.
  • Since students who possesses different background
    knowledge or even different genders may perform
    differently in each test item.

15
Research Questions
  • 1. Are there any biased items in the reading
    tests when comparisons are made for Faculties and
    genders?
  • 2. What is the quality of the reading tests after
    being analyzed by the statistical procedures
    (classical item analysis and Item Response
    Theory)?
  • 3. What is the efficiency of the item bank
    software reported by the English instructors
    (both native and non-native)?

16
Research Objectives
  • 1. To detect item bias in the reading tests when
    comparisons are made for Faculties (Engineering,
    Applied Science, Technical Education and
    Industrial Technology and Management) and genders
    (males and females)
  • 2. To analyze the test items in the reading tests
    used for the English Reading course
  • 3. To develop the item bank software for managing
    the test items.

17
Significance
Regarding the analysis of test quality by both
classical IA and IRT, it can provide the
following advantages. 1. Diagnostic feedback to
test takers on how students performed on
individual test tasks can be illustrated. 2. The
findings can provide feedback to teachers and
course developers relevant to the improvement of
instruction. 3. Test developers and test writer
will obtain feedback to improve the usefulness of
the test, enabling the test developer to 3.1
Control the characteristics of the total score
distribution, specifically the level of
difficulty of the test and the dispersion of test
scores 3.2 Increase the internal consistency
reliability of test and 3.3 Diagnose why items
fail to function appropriately.
18
Significance
Since IRT can offer the advantages over
classical IA, the results obtained from IRT
analysis possess the following significances. 1.
Item parameter estimates are independent of the
group of examinees used. 2. Test takers ability
estimates are independent of the particular set
of test items used. 3. Precision of ability
estimates is known. In terms of the development
of the item bank software, the following results
will be gained. 1. Test developers can easily
build tests to measure objectives of interest. 2.
Test developers, within the limits of an item
bank, can produce tests with the desired number
of test items per objective. 3. If item banks
consist of content valid and technically sound
items, test quality will usually be better than
test developers could produce if they were to
prepare the test items themselves.
19
Significance
The item bias detection can lead to the
following advantages. 1. Having a test writer
provide the item containing content that is not
different or unfamiliar to all students. 2.
Having a student gets the item correct or
incorrect by his or her true ability. 3. Test
writers can provide the content of the item
reflecting information and/or skills within the
educational background of all students. 4. Test
writers can provide clues included in the item
that would not facilitate the performance of
students from one faculty over another. 5. Test
writers can provide adequacies or clarity in the
test instructions, item stem, keyed response, or
distractors.
20
Definitions of Terms
  • 1. Item bias detection Item bias detection in
    this study refers to the detection of the
    presence of some characteristic of an item that
    results in differential performance for
    individuals of the same ability when comparisons
    are made in the following conditions
  • 1.1 when comparisons are made for the four
    Faculties Engineering, Applied Science,
    Technical Education and Industrial Technology and
    Management
  • 1.2 when comparisons are made for gender males
    and females
  • 2. Item bias detection methods This study will
    detect the item bias according to both Classical
    IA and IRT.
  • 2.1 Regarding Classical IA, Delta Plot method
    and chi square technique are employed.
  • 2.2 In terms of IRT, three-parameter model is
    used. Accordingly, item parameters that vary
    across groups indicate Differential Item
    Functioning (DIF) or Item Bias.
  • 3. Item analysis This study will analyze the
    tests by being based on two main test theories,
    Classical Item Analysis (IA) and Item Response
    Theory (IRT). The indices obtained from the two
    theories are as follows
  • 3.1 The indices obtained from Classical IA are
    Item difficulty and Item discrimination.
  • 3.2 The indices obtained from IRT are
    a-parameter (discrimination), b-parameter
    (difficulty) and c-parameter (guessing).
  • 4. Text readability Readability index derived
    from Fry Readability formula and Flesch Reading
    Ease formula.
  • 5. The efficiency of the item bank software The
    efficiency of the item bank software is from the
    opinions of the English instructors who use the
    software. They are required to respond to the
    questionnaire. In order to triangulate the data,
    the retrospective method using the
    semi-structured interview will be conducted with
    them.

21
Methodology
Scope of the study 1. Population 1.1 In terms of
analyzing the quality of the reading tests, the
population is second-fourth year undergraduate
students at King Mongkuts University of
Technology North Bangkok in the first semester of
the academic year 2008. The students study in the
Faculties of Engineering, Applied Science,
Technical Education and Industrial Technology and
Management. Typically, most are male, aged
between 18 22. 1.2 Regarding the efficiency
of the item bank software, the population in this
study is all the English instructors (both native
and non-native instructors) from the Department
of Languages, Faculties of Applied Arts, King
Mongkuts University of Technology North
Bangkok. 2. Reading tests Reading tests used in
this study are the tests written for evaluating
students learning achievement for the Reading I
course. The tests are the ones used for both
midterm and final tests in the first semester of
the academic year 2008.
22
Methodology
  • Phase I The development and validation of the
    research instruments. The instruments used in
    this study can be divided into three main sets
    1) Reading Tests and 2) Item bank software 3) the
    questionnaire and the Retrospective Method.
  • Phase II The implementation of the study
  • Phase III Analysis of the data and report writing

23
Methodology
  • Test Development

Phase 1 The development and validation of the
research instruments. Stage 1 Design Stage 2
Operationalization Stage 3 Test administration
24
Blooms Taxonomy
Blooms Taxonomy (a hierarchy of abilities) Reading comprehension skills
Knowledge level Pronoun reference
Knowledge level Finding specific information
Comprehension level Understanding explicit and implied information
Comprehension level Interpret ideas
Analysis level Making inferences, identifying components
Synthesis level Finding main idea
Synthesis level Finding topic/title
Synthesis level Guessing word meaning
  • Course Objectives
  • Preview Previewing to get an idea of what you
    will find in the text
  • Scanning Looking for specific information
  • Skimming Getting the general sense of a passage
  • Using vocabulary knowledge for effective reading
    Guessing word meaning in context
  • Making inferences Guessing about the text or the
    writers idea when some ideas are not directly
    stated.
  • Finding topics Identifying topics

25
Table of Test Specification for Reading I
Midterm Test
Parts Texts Reading comprehension skills Item types Number of Items
1 Unit 1 (Preview) Unit 2 (Scanning) 3-4 very short paragraphs (assessing both previewing and scanning at the same time) - Previewing to get an idea of what you will find in the text - Looking for specific information Short answer 7 7
Total Total Total Total 14
2 Unit 3 (Skimming) Unit 6 (Finding topics) 4-5 very short paragraphs - Getting the general sense of a passage - Identifying topics Short answer Multiple choice (MC) 6 6
Total Total Total Total 12
3 Unit 5 (Making inferences) Unit 6 (Finding topics) Unit 4 (Using vocabulary knowledge for effective reading) A passage (350 400 words) - Guessing about the text or the writers idea when some ideas are not directly stated. - Identifying topics - Guessing word meaning in context MC Matching MC 6 4 5
Total Total Total Total 15
4 Unit 5 (Making inferences) Unit 4 (Using vocabulary knowledge for effective reading) A passage (350 400 words) - Guessing about the text or the writers idea when some ideas are not directly stated. - Guessing word meaning in context True/False MC 4 5
Total Total Total Total 9
Total Total Total 50 50
Midterm score (30) Raw score (Full point is
50) X 0.6
26
Proportion of the Test Items (based on Booms
Taxonomy)
Units Blooms Taxonomy (a hierarchy of abilities) Blooms Taxonomy (a hierarchy of abilities) Reading comprehension skills (According to Blooms Taxonomy)
1 Preview Previewing to get an idea of what you will find in the text Knowledge level 7 items Pronoun reference
2 Scanning Looking for specific information Knowledge level 7 items Finding specific information
3. Skimming Getting the general sense of a passage Comprehension level 6 items Understanding explicit and implied information
- Comprehension level - Interpret ideas
5 Making inferences Guessing about the text or the writers idea when some ideas are not directly stated. Analysis level 10 items Making inferences, identifying components
- Synthesis level - Finding main idea
6 Finding topics Identifying topics Synthesis level 10 items Finding topic/title
4 Using vocabulary knowledge for effective reading Guessing word meaning in context Synthesis level 10 items Guessing word meaning
4 Using vocabulary knowledge for effective reading Guessing word meaning in context 50 items Guessing word meaning
27
Phase II The implementation of the study 1.
Research design 1.1 This study is conducted
as a Research and Development (RD) project.
Hence, the findings, which will be obtained from
the systemic and objective analyzing method, can
lead to the development of the quality of the
language assessing instruments including an
advance technology like item bank software.
1.2 The population of this study consists of two
main groups 1.2.1 The population used for
trying out the tests in order to analyze the test
quality is second-fourth year undergraduate
students from the Faculties of Engineering,
Applied Science, Technical Education and
Industrial Technology and Management. . 1.2.2 The
population asked to evaluate the efficiency of
the item bank software is all the English
instructors (both native and non-native
instructors) from the Department of Languages,
Faculties of Applied Arts. 6 instructors will be
randomly selected. Each instructor will be
recorded about 1015 minutes. All the reports
will be transcribed. The data will be coded and
decoded by the researchers.
28
2. Research procedures 2.1 All the students who
enroll in the Reading course will be assigned to
have the reading tests written for midterm and
final tests. 2.2 The students answer sheets
will be collected to analyze the test quality.
The values obtained from the analysis are the
text readability, test reliability, item
difficulty and item discrimination (from
Classical IA), a-parameter, b-parameter and
c-parameter (from IRT), item bias values obtained
from Delta Plot method, Chi Square technique and
3-parameter model IRT. 2.3 The values obtained
from 2.2 will be used to create the item bank
software. 2.4 The English instructors will be
asked to use the item bank software. Then the
questionnaire will be given to evaluate its
efficiency. A week after trying out the software,
the retrospective method will be conducted in
order to collect information about the efficiency
of the software.
29
Phase III Data Analysis In order to answer the
research questions, the data will be analyzed
according to the following techniques. Research
questions 1. Are there any biased items in the
reading tests when comparisons are made for
Faculties and genders? 1. Mathcad will be used to
detect the item bias by means of Delta Plot
method and IRT. 2. SPSS will be implemented to
analyze the item bias by means of Chi Square
technique. Research questions 2. What is the
quality of the reading tests after being analyzed
by the statistical procedures (classical item
analysis and Item Response Theory)? 1. Regarding
Classical IA, SPSS will be used to analyze the
tests. Accordingly, test reliability (KR-20),
item difficulty and item discrimination will be
obtained. 2. In terms of IRT, XCALIBRE will be
used. The indices obtained are a-, b- and c-
parameters. Research questions 3. What is the
efficiency of the item bank software reported by
the English instructors (both native and
non-native)? 1. In terms of the questionnaire
used for evaluating the efficiency of the item
bank software, mean and standard deviation are
calculated. 2. Regarding the retrospective
interview, the mode is used as a descriptive
technique to examine the English instructors
opinions. The highest frequency of the opinions
will be reported.
30
Research questions 1. Are there any biased items
in the reading tests when comparisons are made
for Faculties and genders? 1. Mathcad will be
used to detect the item bias by means of Delta
Plot method and IRT.
Delta-plot
  • Major Axis Y bxa
  • slope b

31
Research questions 1. Are there any biased items
in the reading tests when comparisons are made
for Faculties and genders? 1. Mathcad will be
used to detect the item bias by means of Delta
Plot method and IRT.
Delta-plot
  • constant a

32
Research questions 1. Are there any biased items
in the reading tests when comparisons are made
for Faculties and genders? 1. Mathcad will be
used to detect the item bias by means of Delta
Plot method and IRT.
Three-Parameter Model
Analyze the test items in order to get the three
parameters
33
Research questions 1. Are there any biased items
in the reading tests when comparisons are made
for Faculties and genders? 1. Mathcad will be
used to detect the item bias by means of Delta
Plot method and IRT.
Three-Parameter Model
Calculate the area of Item Characteristic Curve
of each item.
34
Research questions 1. Are there any biased items
in the reading tests when comparisons are made
for Faculties and genders? 1. Mathcad will be
used to detect the item bias by means of Delta
Plot method and IRT.
Three-Parameter Model
Compare the area of Item Characteristic Curve of
the same item obtained from different groups.
35
Research questions 1. Are there any biased items
in the reading tests when comparisons are made
for Faculties and genders? 1. Mathcad will be
used to detect the item bias by means of Delta
Plot method and IRT.
Three-Parameter Model
Compare the area of Item Characteristic Curve of
the same item obtained from different groups.
36
Research questions 1. Are there any biased items
in the reading tests when comparisons are made
for Faculties and genders?2. SPSS will be
implemented to analyze the item bias by means of
Chi Square technique.
37
Research questions 1. Are there any biased items
in the reading tests when comparisons are made
for Faculties and genders?2. SPSS will be
implemented to analyze the item bias by means of
Chi Square technique.
38
Results
  • Test Quality IA Vs. IRT

IA Test Reliability .8893
IRT Test Reliability .884
IA Item Selecting Criteria Difficulty Level
0.2 -0.8 Discrimination Index .20 up
IRT Item Selecting Criteria a-parameter (Discrimi
nation) agt 0.50 b-parameter (Difficulty) b
3.0 c-parameter (Guessing) clt0.30
Number of good items 36 of 50 items Difficulty
Level ranged from .26 - .80. Discrimination Index
was between .21 - .54
Number of good items 49 of 50 items a-parameter (
Discrimination) .56 1.37 b-parameter (Difficul
ty) -2.41 3.00 c-parameter (Guessing) .17 -
.26
39
Results
  • Comparison was made for faculty

No. Delta Plot Chi Square IRT
1
2 2
3 3
4 4 4
5 5 5
6
7
8 8
40
Results
  • Comparison was made for faculty

Biased Items Content Measured Content Measured Content Measured
Biased Items Ability Measured Reading Topic Text Type
4 1. Knowledge Level Scanning - Looking for specific information The Bicycle Messenger and the Mysterious Synopsis
5 1. Knowledge Level Scanning - Looking for specific information The Bicycle Messenger and the Mysterious Synopsis
10 1. Knowledge Level Preview - Previewing to get an idea of what you will find in the text Fish Stories General Science Article
13 1. Knowledge Level Scanning - Looking for specific information Fish Stories General Science Article
22 4. Synthesis Level Finding topics - Identifying topics civilians Short News
24 4. Synthesis Level Finding topics - Identifying topics wildlife Tourism Advertisement
37 3. Analysis Level Making inferences - Guessing about the text or the writers idea when some ideas are not directly stated. The internet addiction Social Science Article
41
Results
  • Comparison was made for faculty

Biased Item Characteristics (faculty)
Ability Measured Reading Topic Text Type No. of Item No. of Biased Item
Level 1 Knowledge Level Preview - Previewing to get an idea of what you will find in the text The Bicycle Messenger and the Mysterious Synopsis 2 0 0
Level 1 Knowledge Level Preview - Previewing to get an idea of what you will find in the text Christmas Job Opportunities Job Advertisement 1 0 0
Level 1 Knowledge Level Preview - Previewing to get an idea of what you will find in the text Fish Stories General Science Article 2 1 50
Total Total Total 5 1 20
42
Results
  • Comparison was made for faculty

Biased Item Characteristics (faculty)
Level 1 Knowledge Level Scanning - Looking for specific information The Bicycle Messenger and the Mysterious Synopsis 3 2 66.67
Level 1 Knowledge Level Scanning - Looking for specific information Christmas Job Opportunities Job Advertisement 3 0 0
Level 1 Knowledge Level Scanning - Looking for specific information Fish Stories General Science Article 3 1 33.34
Total Total Total 9 3 33.34
43
Results
  • Comparison was made for faculty

Biased Item Characteristics (faculty)
Level2 Comprehension Level Skimming - Getting the general sense of a passage Suicide Blast Killed People Short News 1 0 0
Level2 Comprehension Level Skimming - Getting the general sense of a passage CEOs plan to develop the company Short News 1 0 0
Level2 Comprehension Level Skimming - Getting the general sense of a passage Environmental problem caused by some chemical Short News 1 0 0
Level2 Comprehension Level Skimming - Getting the general sense of a passage Visiting a place in America with a European Architecture Style Tourism Advertisement 1 0 0
Level2 Comprehension Level Skimming - Getting the general sense of a passage A New Type of Solar Energy Collector Science and Technology Article 1 0 0
Level2 Comprehension Level Skimming - Getting the general sense of a passage A New Natural Wonders Named Tourism Article 1 0 0
Total Total Total 6 0 0
44
Results
  • Comparison was made for faculty

Biased Item Characteristics (faculty)
Level 3 Analysis Level Making inferences - Guessing about the text or the writers idea when some ideas are not directly stated. The internet addiction Social Science Article 6 1 16,67
Level 3 Analysis Level Making inferences - Guessing about the text or the writers idea when some ideas are not directly stated. The Best Foods You Arent Eating Health Article from a Magazine 4 0 0
Total Total Total 10 1 10
45
Results
  • Comparison was made for faculty

Biased Item Characteristics (faculty)
Level 4 Synthesis Level Finding topics - Identifying topics a major restructuring of the company Short News 1 0 0
Level 4 Synthesis Level Finding topics - Identifying topics civilians Short News 1 1 100
Level 4 Synthesis Level Finding topics - Identifying topics Spanish-colonial architecture Tourism Advertisement 1 0 0
Level 4 Synthesis Level Finding topics - Identifying topics wildlife Tourism Advertisement 1 1 100
Level 4 Synthesis Level Finding topics - Identifying topics sport activities Short News 1 0 0
Level 4 Synthesis Level Finding topics - Identifying topics Eight new natural wonders Tourism Article 1 0 0
Level 4 Synthesis Level Finding topics - Identifying topics The internet addiction Social Science Article 4 0 0
Total Total Total 10 2 20
Level 4 Synthesis Level Using vocabulary knowledge for effective reading - Guessing word meaning in context The internet addiction Social Science Article 5 0 0
Level 4 Synthesis Level Using vocabulary knowledge for effective reading - Guessing word meaning in context The Best Foods You Arent Eating Health Article from a Magazine 5 0 0
Total Total Total 10 0 0
46
Results
  • Comparison was made for faculty

Reading Ability Levels (faculty)
Ability Measured No. of Items No. of Biased Items
Level 1 Knowledge Level Preview - Previewing to get an idea of what you will find in the text 5 1 20
Level 1 Knowledge Level Scanning - Looking for specific information 9 3 33.34
Level2 Comprehension Level Skimming - Getting the general sense of a passage 6 0 0
Level 3 Analysis Level Making inferences - Guessing about the text or the writers idea when some ideas are not directly stated. 10 1 10
Level 4 Synthesis Level Finding topics - Identifying topics 10 2 20
Level 4 Synthesis Level Using vocabulary knowledge for effective reading - Guessing word meaning in context 10 0 0
Total 50 7 14
47
Results
  • Comparison was made for faculty

Biased Reading Topics (faculty)
Reading Topic No. of Items No. of Biased Items
1. The Bicycle Messenger and the Mysterious 5 2 40
2. Christmas Job Opportunities 4 0 0
3. Fish Stories 5 2 40
4. Suicide Blast Killed People 1 0 0
5. CEOs plan to develop the company 1 0 0
6. Environmental problem caused by some chemical 1 0 0
7. Visiting a place in America with a European Architecture Style 1 0 0
8. A New Type of Solar Energy Collector 1 0 0
9. A New Natural Wonders Named 1 0 0
10. a major restructuring of the company 1 0 0
11. civilians 1 1 100
12. Spanish-colonial architecture 1 0 0
13. wildlife 1 1 100
14. sport activities 1 0 0
15. Eight new natural wonders 1 0 0
16. The internet addiction 15 1 100
17. The Best Foods You Arent Eating 9 0 0
Total 50 7 14
48
Results
  • Comparison was made for faculty

Biased Text Types (Faculty)
Reading Topic No. of Items No. of Biased Items
1. Synopsis 5 2 40
2. Job Advertisement 4 0 0
3. General Science Article 5 2 40
4. Science and Technology Article 1 0 0
5. Short News 6 1 16.67
6. Tourism Advertisement 3 1 33.34
7. Tourism Article 2 0 0
8. Social Science Article 15 1 6.67
9. Health Article from a Magazine 9 0 0
Total 50 7 14
49
Results
  • Comparison was made for gender

Compare Gender
Delta Plot Chi Square IRT
1
2
3
4 4
5
8
9
10
50
Results
  • Comparison was made for gender

Gender Bias Content Measured Content Measured Content Measured
Gender Bias Ability Measured Reading Topic Text Type
4 1. Knowledge Level Scanning - Looking for specific information The Bicycle Messenger and the Mysterious Synopsis
16 2. Comprehension Level Skimming - Getting the general sense of a passage CEOs plan to develop the company Short News
22 4. Synthesis Level Finding topics - Identifying topics civilians Short News
32 4. Synthesis Level Using vocabulary knowledge for effective reading - Guessing word meaning in context The internet addiction Social Science Article
39 3. Analysis Level Making inferences - Guessing about the text or the writers idea when some ideas are not directly stated. The internet addiction Social Science Article
41 3. Analysis Level Making inferences - Guessing about the text or the writers idea when some ideas are not directly stated. The internet addiction Social Science Article
42 3. Analysis Level Making inferences - Guessing about the text or the writers idea when some ideas are not directly stated. The Best Foods You Arent Eating Health Article from a Magazine
48 4. Synthesis Level Using vocabulary knowledge for effective reading - Guessing word meaning in context The Best Foods You Arent Eating Health Article from a Magazine
49 4. Synthesis Level Using vocabulary knowledge for effective reading - Guessing word meaning in context The Best Foods You Arent Eating Health Article from a Magazine
51
Results
  • Comparison was made for gender

Biased Item Characteristics (Gender)
Ability Measured Reading Topic Text Type No. of Item No. of Biased Item
Level 1 Knowledge Level Preview - Previewing to get an idea of what you will find in the text The Bicycle Messenger and the Mysterious Synopsis 2 0 0
Level 1 Knowledge Level Preview - Previewing to get an idea of what you will find in the text Christmas Job Opportunities Job Advertisement 1 0 0
Level 1 Knowledge Level Preview - Previewing to get an idea of what you will find in the text Fish Stories General Science Article 2 0 0
Total Total Total 5 0 0
Level 1 Knowledge Level Scanning - Looking for specific information The Bicycle Messenger and the Mysterious Synopsis 3 1 33.34
Level 1 Knowledge Level Scanning - Looking for specific information Christmas Job Opportunities Job Advertisement 3 0 0
Level 1 Knowledge Level Scanning - Looking for specific information Fish Stories General Science Article 3 0 0
Total Total Total 9 1 11.12
52
Results
  • Comparison was made for gender

Level2 Comprehension Level Skimming - Getting the general sense of a passage Suicide Blast Killed People Short News 1 0 0
Level2 Comprehension Level Skimming - Getting the general sense of a passage CEOs plan to develop the company Short News 1 1 100
Level2 Comprehension Level Skimming - Getting the general sense of a passage Environmental problem caused by some chemical Short News 1 0 0
Level2 Comprehension Level Skimming - Getting the general sense of a passage Visiting a place in America with a European Architecture Style Tourism Advertisement 1 0 0
Level2 Comprehension Level Skimming - Getting the general sense of a passage A New Type of Solar Energy Collector Science and Technology Article 1 0 0
Level2 Comprehension Level Skimming - Getting the general sense of a passage A New Natural Wonders Named Tourism Article 1 0 0
Total Total Total 6 1 16.67
Level 3 Analysis Level Making inferences - Guessing about the text or the writers idea when some ideas are not directly stated. The internet addiction Social Science Article 6 2 33.34
Level 3 Analysis Level Making inferences - Guessing about the text or the writers idea when some ideas are not directly stated. The Best Foods You Arent Eating Health Article from a Magazine 4 1 25
Total Total Total 10 3 30
53
Results
  • Comparison was made for gender

Level 4 Synthesis Level Finding topics - Identifying topics a major restructuring of the company Short News 1 0 0
Level 4 Synthesis Level Finding topics - Identifying topics civilians Short News 1 1 100
Level 4 Synthesis Level Finding topics - Identifying topics Spanish-colonial architecture Tourism Advertisement 1 0 0
Level 4 Synthesis Level Finding topics - Identifying topics wildlife Tourism Advertisement 1 0 0
Level 4 Synthesis Level Finding topics - Identifying topics sport activities Short News 1 0 0
Level 4 Synthesis Level Finding topics - Identifying topics Eight new natural wonders Tourism Article 1 0 0
Level 4 Synthesis Level Finding topics - Identifying topics The internet addiction Social Science Article 4 0 0
Total Total Total 10 1 10
Level 4 Synthesis Level Using vocabulary knowledge for effective reading - Guessing word meaning in context The internet addiction Social Science Article 5 1 20
Level 4 Synthesis Level Using vocabulary knowledge for effective reading - Guessing word meaning in context The Best Foods You Arent Eating Health Article from a Magazine 5 2 40
Total Total Total 10 3 30
54
Results
  • Comparison was made for gender

Reading Ability Levels (Gender)
Ability Measured No. of Items No. of Biased Items
Level 1 Knowledge Level Preview - Previewing to get an idea of what you will find in the text 5 0 0
Level 1 Knowledge Level Scanning - Looking for specific information 9 1 11.12
Level2 Comprehension Level Skimming - Getting the general sense of a passage 6 1 16.67
Level 3 Analysis Level Making inferences - Guessing about the text or the writers idea when some ideas are not directly stated. 10 3 30
Level 4 Synthesis Level Finding topics - Identifying topics 10 1 10
Level 4 Synthesis Level Using vocabulary knowledge for effective reading - Guessing word meaning in context 10 3 10
Total 50 9 18
55
Results
Reading Topic No. of Items No. of Biased Items
1. The Bicycle Messenger and the Mysterious 5 1 20
2. Christmas Job Opportunities 4 0 0
3. Fish Stories 5 0 0
4. Suicide Blast Killed People 1 0 0
5. CEOs plan to develop the company 1 1 100
6. Environmental problem caused by some chemical 1 0 0
7. Visiting a place in America with a European Architecture Style 1 0 0
8. A New Type of Solar Energy Collector 1 0 0
9. A New Natural Wonders Named 1 0 0
10. a major restructuring of the company 1 0 0
11. civilians 1 1 100
12. Spanish-colonial architecture 1 0 0
13. wildlife 1 0 0
14. sport activities 1 0 0
15. Eight new natural wonders 1 0 0
16. The internet addiction 15 3 20
17. The Best Foods You Arent Eating 9 3 33.34
Total 50 9 18
  • Comparison was made for gender

Biased Reading Topics (Gender)
56
Results
  • Comparison was made for gender

Biased Text Types (Gender)
Reading Topic No. of Items No. of Biased Items
1. Synopsis 5 1 20
2. Job Advertisement 4 0 0
3. General Science Article 5 0 0
4. Science and Technology Article 1 0 0
5. Short News 6 2 33.34
6. Tourism Advertisement 3 0 0
7. Tourism Article 2 0 0
8. Social Science Article 15 3 20
9. Health Article from a Magazine 9 3 33.34
Total 50 9 18
57
Conclusion
  • Prior Knowledge (Faculty) Bias

Level 1 Knowledge Level Preview - Previewing to
get an idea of what you will find in the text
Text Type General Science Article
Fish Stories
Level 1 Knowledge Level Scanning - Looking for
specific information
Synopsis
The Bicycle Messenger and the Mysterious
Level 4 Synthesis Level Finding topics -
Identifying topics
Civilians
Short News
Wild life
Tourism Advertisement
58
Conclusion
  • Gender Bias

Level 3 Analysis Level Making inferences -
Guessing about the text or the writers idea when
some ideas are not directly stated.
Social Science Articles
Internet Addiction
Level 4 Synthesis Level Using vocabulary
knowledge for effective reading - Guessing word
meaning in context
The Best Foods You Arent Eating
Health Article from Magazine
59
Ex.1 Internet Addicts
  • The US could be rife with "internet addicts" who
    are as clinically ill as alcoholics, according to
    psychiatrists involved in a nationwide study. The
    study, carried out by researchers at Stanford
    University School of Medicine in California, US,
    indicates that more than one in eight US
    residents show signs of "problematic internet
    use".
  • The Stanford researchers interviewed 2,513
    adults in a nationwide survey. According to the
    study's lead author Elias Aboujaoude, the most
    disturbing result is the discovery that some
    people hide their internet surfing, or go online
    to cure foul moods behavior that mirrors the
    way alcoholics behave. " And, obviously,
    something is wrong when people go out of their
    way to hide their internet activity," Aboujaoude
    says. .

60
Ex.2 The Best Foods You arent Eating
  • The Best Foods You Aren't Eating
  • By JONNY BOWDEN, Ph.D.
  • Although some guys aren't opposed to smoking
    some weed, most wouldn't think of eating one.
    It's a shame, really, since a succulent weed
    named purslane is not only delicious but also
    among the world's healthiest foods.
  • Guava beets and cinnamon are among the
    super-healthful foods
  • that you should probably be getting more of in
    your diet
  • Of course, there are many super foods that never
    see the inside of a shopping cart.
  • Some you've never heard of, and others you've
    simply forgotten about. That's why we've rounded
    up the best of the bunch. Make a place for them
    on your table and you'll instantly upgrade your
    health -- without a prescription. .

61
Item Bank Software
Software Structure
(item code assignments) (item instoration) (test
generation) (test utilization)
62
Methodology
  • Item Bank Software

Toolbar
  • Menu File

63
Methodology
  • Item Bank Software

Toolbar
  • Menu File
  • Create Exam

64
Methodology
  • Item Bank Software

Toolbar
  • Menu File
  • Create Exam
  • Manage Form

65
Methodology
  • Item Bank Software

Toolbar
  • Menu File
  • Create Exam
  • Manage Form
  • Help

66
Methodology
  • Item Bank Software
  • Create Exam

67
Methodology
  • Item Bank Software
  • Create Exam
  • View Report

68
Methodology
  • Item Bank Software
  • Create Exam
  • View Report
  • Exit

69
Conclusion
  • Efficiency of the Item Bank Software (1st Trial)
  • Interface design
  • Easy to read
  • Unpleasant screen design
  • Application
  • Very convenient to retrieve the test items
  • The storing process is complicated.

70
END OF THE PRESENTATION

SINCEREST THANKS
View by Category
About This Presentation
Title:

The Item Bias Detection of the Reading Tests

Description:

The Item Bias Detection of the Reading Tests and the Development of the Item Bank Software Used for English Reading Courses at King Mongkut s University of ... – PowerPoint PPT presentation

Number of Views:38
Avg rating:3.0/5.0
Slides: 71
Provided by: iLL74
Learn more at: http://www.teslontario.ca
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: The Item Bias Detection of the Reading Tests


1
The Item Bias Detection of the Reading Tests and
the Development of the Item Bank Software Used
for English Reading Courses at King Mongkuts
University of Technology North Bangkok
Supalak Nakhornsri, Ph.D.
supalak_nak_at_yahoo.com
Department of Languages, King Mongkuts
University of Technology North Bangkok, Thailand
2
Outline of the Presentation
  • Background of the study
  • Objectives
  • Research Questions
  • Definitions of Terms
  • Methodology
  • Test Development
  • Item Bias Detection
  • Conclusion
  • Results
  • Item Bank Software

3
Background of the Study
  • Trying out a test
  • The results can be used to make changes to
    improve the potential usefulness of the test.
  • (Bachman, 2004)
  • For classroom teachers, it is not advisable to
    try out a test with the same students who will
    take it.
  • (Bachman, 2004)

4
Background of the Study
  • Item Analysis

The essential part of analyzing the results of
tests for improving their usefulness. (Brown,
2004)
  • Classical Item Analysis (IA) calculating
    descriptive statistics for individual items
  • Item Response Theory (IRT)
  • a more sophisticated procedure for estimating
    the statistical characteristics of item.
  • (Bachman, 2004)

5
Background of the Study
  • IA vs. IRT
  • IA
  • Subpopulation dependent
  • Implicitly averaged across all ability levels
  • (Bachman, 1990)
  • IRT
  • Independent of group of examinees used
  • Independently estimated test taker ability
  • Known precision of ability estimates

6
Background of the Study
  • Bias Vs. Unfairness
  • Unfairness Concerning with test administration
  • Bias A type of invalidity
  • (Shepard, 1982)

7
Background of the Study
  • Differences among three methods

IA IRT
Delta-plot
Chi-square
Three-parameter model
Transforming item difficulty into Delta value
and plot the value onto the Bivariate Graph
(Angoff, 1982)
Measuring the frequency of correct or incorrect
responses in each ability groups (Shepard,
1982)
The presence of some characteristic of an item
that results in differential performance for
individuals of the same ability but from
different ethnic, sex, cultural, or religious
groups. (Hambleton et.al, 1991.)
8
Background of the Study
  • Delta-plot

9
Background of the Study
  • Chi-square

Value df Asymp. Sig. (2-sided)
Pearson Chi-Square 5.799(b) 1 .016
Continuity Correction(a) 4.602 1 .032
Likelihood Ratio 5.597 1 .018
Fisher's Exact Test
Linear-by-Linear Association 5.729 1 .017
N of Valid Cases 83
10
Background of the Study
  • IRT Three-parameter model

11
Background of the Study
  • Sources of Bias
  • Gender
  • In some languages there are differences between
    the vocabulary items used by women and men.
  • Women tend to use more of the standard forms than
    men do.
  • (Homes, 2001)

12
Background of the Study
  • Background Knowledge
  • Schema Theory
  • Readers can fully comprehend what they read when
    their schema or prior knowledge, is activated by
    the words on the page.
  • (Chandavimol, 1998)

13
Background of the Study
  • Sources of Bias Studied
  • Gender male and female
  • Prior Knowledge Faculties they are studying.

14
Background of the Study
  • Problem of the Assessment of Reading Courses
  • Regarding Reading I course, it is an elective
    course for second fourth year undergraduate
    students, King Mongkuts University of Technology
    North Bangkok.
  • The same reading tests are used to all students
    to evaluate their achievement. Therefore, the
    important judgment on students grade is up to
    both midterm and final tests.
  • The quality of test is necessary to be evaluated.
    Although both Classical IA and IRT can help
    analyze the quality of test items, the issue of
    test bias cannot be neglected.
  • Since students who possesses different background
    knowledge or even different genders may perform
    differently in each test item.

15
Research Questions
  • 1. Are there any biased items in the reading
    tests when comparisons are made for Faculties and
    genders?
  • 2. What is the quality of the reading tests after
    being analyzed by the statistical procedures
    (classical item analysis and Item Response
    Theory)?
  • 3. What is the efficiency of the item bank
    software reported by the English instructors
    (both native and non-native)?

16
Research Objectives
  • 1. To detect item bias in the reading tests when
    comparisons are made for Faculties (Engineering,
    Applied Science, Technical Education and
    Industrial Technology and Management) and genders
    (males and females)
  • 2. To analyze the test items in the reading tests
    used for the English Reading course
  • 3. To develop the item bank software for managing
    the test items.

17
Significance
Regarding the analysis of test quality by both
classical IA and IRT, it can provide the
following advantages. 1. Diagnostic feedback to
test takers on how students performed on
individual test tasks can be illustrated. 2. The
findings can provide feedback to teachers and
course developers relevant to the improvement of
instruction. 3. Test developers and test writer
will obtain feedback to improve the usefulness of
the test, enabling the test developer to 3.1
Control the characteristics of the total score
distribution, specifically the level of
difficulty of the test and the dispersion of test
scores 3.2 Increase the internal consistency
reliability of test and 3.3 Diagnose why items
fail to function appropriately.
18
Significance
Since IRT can offer the advantages over
classical IA, the results obtained from IRT
analysis possess the following significances. 1.
Item parameter estimates are independent of the
group of examinees used. 2. Test takers ability
estimates are independent of the particular set
of test items used. 3. Precision of ability
estimates is known. In terms of the development
of the item bank software, the following results
will be gained. 1. Test developers can easily
build tests to measure objectives of interest. 2.
Test developers, within the limits of an item
bank, can produce tests with the desired number
of test items per objective. 3. If item banks
consist of content valid and technically sound
items, test quality will usually be better than
test developers could produce if they were to
prepare the test items themselves.
19
Significance
The item bias detection can lead to the
following advantages. 1. Having a test writer
provide the item containing content that is not
different or unfamiliar to all students. 2.
Having a student gets the item correct or
incorrect by his or her true ability. 3. Test
writers can provide the content of the item
reflecting information and/or skills within the
educational background of all students. 4. Test
writers can provide clues included in the item
that would not facilitate the performance of
students from one faculty over another. 5. Test
writers can provide adequacies or clarity in the
test instructions, item stem, keyed response, or
distractors.
20
Definitions of Terms
  • 1. Item bias detection Item bias detection in
    this study refers to the detection of the
    presence of some characteristic of an item that
    results in differential performance for
    individuals of the same ability when comparisons
    are made in the following conditions
  • 1.1 when comparisons are made for the four
    Faculties Engineering, Applied Science,
    Technical Education and Industrial Technology and
    Management
  • 1.2 when comparisons are made for gender males
    and females
  • 2. Item bias detection methods This study will
    detect the item bias according to both Classical
    IA and IRT.
  • 2.1 Regarding Classical IA, Delta Plot method
    and chi square technique are employed.
  • 2.2 In terms of IRT, three-parameter model is
    used. Accordingly, item parameters that vary
    across groups indicate Differential Item
    Functioning (DIF) or Item Bias.
  • 3. Item analysis This study will analyze the
    tests by being based on two main test theories,
    Classical Item Analysis (IA) and Item Response
    Theory (IRT). The indices obtained from the two
    theories are as follows
  • 3.1 The indices obtained from Classical IA are
    Item difficulty and Item discrimination.
  • 3.2 The indices obtained from IRT are
    a-parameter (discrimination), b-parameter
    (difficulty) and c-parameter (guessing).
  • 4. Text readability Readability index derived
    from Fry Readability formula and Flesch Reading
    Ease formula.
  • 5. The efficiency of the item bank software The
    efficiency of the item bank software is from the
    opinions of the English instructors who use the
    software. They are required to respond to the
    questionnaire. In order to triangulate the data,
    the retrospective method using the
    semi-structured interview will be conducted with
    them.

21
Methodology
Scope of the study 1. Population 1.1 In terms of
analyzing the quality of the reading tests, the
population is second-fourth year undergraduate
students at King Mongkuts University of
Technology North Bangkok in the first semester of
the academic year 2008. The students study in the
Faculties of Engineering, Applied Science,
Technical Education and Industrial Technology and
Management. Typically, most are male, aged
between 18 22. 1.2 Regarding the efficiency
of the item bank software, the population in this
study is all the English instructors (both native
and non-native instructors) from the Department
of Languages, Faculties of Applied Arts, King
Mongkuts University of Technology North
Bangkok. 2. Reading tests Reading tests used in
this study are the tests written for evaluating
students learning achievement for the Reading I
course. The tests are the ones used for both
midterm and final tests in the first semester of
the academic year 2008.
22
Methodology
  • Phase I The development and validation of the
    research instruments. The instruments used in
    this study can be divided into three main sets
    1) Reading Tests and 2) Item bank software 3) the
    questionnaire and the Retrospective Method.
  • Phase II The implementation of the study
  • Phase III Analysis of the data and report writing

23
Methodology
  • Test Development

Phase 1 The development and validation of the
research instruments. Stage 1 Design Stage 2
Operationalization Stage 3 Test administration
24
Blooms Taxonomy
Blooms Taxonomy (a hierarchy of abilities) Reading comprehension skills
Knowledge level Pronoun reference
Knowledge level Finding specific information
Comprehension level Understanding explicit and implied information
Comprehension level Interpret ideas
Analysis level Making inferences, identifying components
Synthesis level Finding main idea
Synthesis level Finding topic/title
Synthesis level Guessing word meaning
  • Course Objectives
  • Preview Previewing to get an idea of what you
    will find in the text
  • Scanning Looking for specific information
  • Skimming Getting the general sense of a passage
  • Using vocabulary knowledge for effective reading
    Guessing word meaning in context
  • Making inferences Guessing about the text or the
    writers idea when some ideas are not directly
    stated.
  • Finding topics Identifying topics

25
Table of Test Specification for Reading I
Midterm Test
Parts Texts Reading comprehension skills Item types Number of Items
1 Unit 1 (Preview) Unit 2 (Scanning) 3-4 very short paragraphs (assessing both previewing and scanning at the same time) - Previewing to get an idea of what you will find in the text - Looking for specific information Short answer 7 7
Total Total Total Total 14
2 Unit 3 (Skimming) Unit 6 (Finding topics) 4-5 very short paragraphs - Getting the general sense of a passage - Identifying topics Short answer Multiple choice (MC) 6 6
Total Total Total Total 12
3 Unit 5 (Making inferences) Unit 6 (Finding topics) Unit 4 (Using vocabulary knowledge for effective reading) A passage (350 400 words) - Guessing about the text or the writers idea when some ideas are not directly stated. - Identifying topics - Guessing word meaning in context MC Matching MC 6 4 5
Total Total Total Total 15
4 Unit 5 (Making inferences) Unit 4 (Using vocabulary knowledge for effective reading) A passage (350 400 words) - Guessing about the text or the writers idea when some ideas are not directly stated. - Guessing word meaning in context True/False MC 4 5
Total Total Total Total 9
Total Total Total 50 50
Midterm score (30) Raw score (Full point is
50) X 0.6
26
Proportion of the Test Items (based on Booms
Taxonomy)
Units Blooms Taxonomy (a hierarchy of abilities) Blooms Taxonomy (a hierarchy of abilities) Reading comprehension skills (According to Blooms Taxonomy)
1 Preview Previewing to get an idea of what you will find in the text Knowledge level 7 items Pronoun reference
2 Scanning Looking for specific information Knowledge level 7 items Finding specific information
3. Skimming Getting the general sense of a passage Comprehension level 6 items Understanding explicit and implied information
- Comprehension level - Interpret ideas
5 Making inferences Guessing about the text or the writers idea when some ideas are not directly stated. Analysis level 10 items Making inferences, identifying components
- Synthesis level - Finding main idea
6 Finding topics Identifying topics Synthesis level 10 items Finding topic/title
4 Using vocabulary knowledge for effective reading Guessing word meaning in context Synthesis level 10 items Guessing word meaning
4 Using vocabulary knowledge for effective reading Guessing word meaning in context 50 items Guessing word meaning
27
Phase II The implementation of the study 1.
Research design 1.1 This study is conducted
as a Research and Development (RD) project.
Hence, the findings, which will be obtained from
the systemic and objective analyzing method, can
lead to the development of the quality of the
language assessing instruments including an
advance technology like item bank software.
1.2 The population of this study consists of two
main groups 1.2.1 The population used for
trying out the tests in order to analyze the test
quality is second-fourth year undergraduate
students from the Faculties of Engineering,
Applied Science, Technical Education and
Industrial Technology and Management. . 1.2.2 The
population asked to evaluate the efficiency of
the item bank software is all the English
instructors (both native and non-native
instructors) from the Department of Languages,
Faculties of Applied Arts. 6 instructors will be
randomly selected. Each instructor will be
recorded about 1015 minutes. All the reports
will be transcribed. The data will be coded and
decoded by the researchers.
28
2. Research procedures 2.1 All the students who
enroll in the Reading course will be assigned to
have the reading tests written for midterm and
final tests. 2.2 The students answer sheets
will be collected to analyze the test quality.
The values obtained from the analysis are the
text readability, test reliability, item
difficulty and item discrimination (from
Classical IA), a-parameter, b-parameter and
c-parameter (from IRT), item bias values obtained
from Delta Plot method, Chi Square technique and
3-parameter model IRT. 2.3 The values obtained
from 2.2 will be used to create the item bank
software. 2.4 The English instructors will be
asked to use the item bank software. Then the
questionnaire will be given to evaluate its
efficiency. A week after trying out the software,
the retrospective method will be conducted in
order to collect information about the efficiency
of the software.
29
Phase III Data Analysis In order to answer the
research questions, the data will be analyzed
according to the following techniques. Research
questions 1. Are there any biased items in the
reading tests when comparisons are made for
Faculties and genders? 1. Mathcad will be used to
detect the item bias by means of Delta Plot
method and IRT. 2. SPSS will be implemented to
analyze the item bias by means of Chi Square
technique. Research questions 2. What is the
quality of the reading tests after being analyzed
by the statistical procedures (classical item
analysis and Item Response Theory)? 1. Regarding
Classical IA, SPSS will be used to analyze the
tests. Accordingly, test reliability (KR-20),
item difficulty and item discrimination will be
obtained. 2. In terms of IRT, XCALIBRE will be
used. The indices obtained are a-, b- and c-
parameters. Research questions 3. What is the
efficiency of the item bank software reported by
the English instructors (both native and
non-native)? 1. In terms of the questionnaire
used for evaluating the efficiency of the item
bank software, mean and standard deviation are
calculated. 2. Regarding the retrospective
interview, the mode is used as a descriptive
technique to examine the English instructors
opinions. The highest frequency of the opinions
will be reported.
30
Research questions 1. Are there any biased items
in the reading tests when comparisons are made
for Faculties and genders? 1. Mathcad will be
used to detect the item bias by means of Delta
Plot method and IRT.
Delta-plot
  • Major Axis Y bxa
  • slope b

31
Research questions 1. Are there any biased items
in the reading tests when comparisons are made
for Faculties and genders? 1. Mathcad will be
used to detect the item bias by means of Delta
Plot method and IRT.
Delta-plot
  • constant a

32
Research questions 1. Are there any biased items
in the reading tests when comparisons are made
for Faculties and genders? 1. Mathcad will be
used to detect the item bias by means of Delta
Plot method and IRT.
Three-Parameter Model
Analyze the test items in order to get the three
parameters
33
Research questions 1. Are there any biased items
in the reading tests when comparisons are made
for Faculties and genders? 1. Mathcad will be
used to detect the item bias by means of Delta
Plot method and IRT.
Three-Parameter Model
Calculate the area of Item Characteristic Curve
of each item.
34
Research questions 1. Are there any biased items
in the reading tests when comparisons are made
for Faculties and genders? 1. Mathcad will be
used to detect the item bias by means of Delta
Plot method and IRT.
Three-Parameter Model
Compare the area of Item Characteristic Curve of
the same item obtained from different groups.
35
Research questions 1. Are there any biased items
in the reading tests when comparisons are made
for Faculties and genders? 1. Mathcad will be
used to detect the item bias by means of Delta
Plot method and IRT.
Three-Parameter Model
Compare the area of Item Characteristic Curve of
the same item obtained from different groups.
36
Research questions 1. Are there any biased items
in the reading tests when comparisons are made
for Faculties and genders?2. SPSS will be
implemented to analyze the item bias by means of
Chi Square technique.
37
Research questions 1. Are there any biased items
in the reading tests when comparisons are made
for Faculties and genders?2. SPSS will be
implemented to analyze the item bias by means of
Chi Square technique.
38
Results
  • Test Quality IA Vs. IRT

IA Test Reliability .8893
IRT Test Reliability .884
IA Item Selecting Criteria Difficulty Level
0.2 -0.8 Discrimination Index .20 up
IRT Item Selecting Criteria a-parameter (Discrimi
nation) agt 0.50 b-parameter (Difficulty) b
3.0 c-parameter (Guessing) clt0.30
Number of good items 36 of 50 items Difficulty
Level ranged from .26 - .80. Discrimination Index
was between .21 - .54
Number of good items 49 of 50 items a-parameter (
Discrimination) .56 1.37 b-parameter (Difficul
ty) -2.41 3.00 c-parameter (Guessing) .17 -
.26
39
Results
  • Comparison was made for faculty

No. Delta Plot Chi Square IRT
1
2 2
3 3
4 4 4
5 5 5
6
7
8 8
40
Results
  • Comparison was made for faculty

Biased Items Content Measured Content Measured Content Measured
Biased Items Ability Measured Reading Topic Text Type
4 1. Knowledge Level Scanning - Looking for specific information The Bicycle Messenger and the Mysterious Synopsis
5 1. Knowledge Level Scanning - Looking for specific information The Bicycle Messenger and the Mysterious Synopsis
10 1. Knowledge Level Preview - Previewing to get an idea of what you will find in the text Fish Stories General Science Article
13 1. Knowledge Level Scanning - Looking for specific information Fish Stories General Science Article
22 4. Synthesis Level Finding topics - Identifying topics civilians Short News
24 4. Synthesis Level Finding topics - Identifying topics wildlife Tourism Advertisement
37 3. Analysis Level Making inferences - Guessing about the text or the writers idea when some ideas are not directly stated. The internet addiction Social Science Article
41
Results
  • Comparison was made for faculty

Biased Item Characteristics (faculty)
Ability Measured Reading Topic Text Type No. of Item No. of Biased Item
Level 1 Knowledge Level Preview - Previewing to get an idea of what you will find in the text The Bicycle Messenger and the Mysterious Synopsis 2 0 0
Level 1 Knowledge Level Preview - Previewing to get an idea of what you will find in the text Christmas Job Opportunities Job Advertisement 1 0 0
Level 1 Knowledge Level Preview - Previewing to get an idea of what you will find in the text Fish Stories General Science Article 2 1 50
Total Total Total 5 1 20
42
Results
  • Comparison was made for faculty

Biased Item Characteristics (faculty)
Level 1 Knowledge Level Scanning - Looking for specific information The Bicycle Messenger and the Mysterious Synopsis 3 2 66.67
Level 1 Knowledge Level Scanning - Looking for specific information Christmas Job Opportunities Job Advertisement 3 0 0
Level 1 Knowledge Level Scanning - Looking for specific information Fish Stories General Science Article 3 1 33.34
Total Total Total 9 3 33.34
43
Results
  • Comparison was made for faculty

Biased Item Characteristics (faculty)
Level2 Comprehension Level Skimming - Getting the general sense of a passage Suicide Blast Killed People Short News 1 0 0
Level2 Comprehension Level Skimming - Getting the general sense of a passage CEOs plan to develop the company Short News 1 0 0
Level2 Comprehension Level Skimming - Getting the general sense of a passage Environmental problem caused by some chemical Short News 1 0 0
Level2 Comprehension Level Skimming - Getting the general sense of a passage Visiting a place in America with a European Architecture Style Tourism Advertisement 1 0 0
Level2 Comprehension Level Skimming - Getting the general sense of a passage A New Type of Solar Energy Collector Science and Technology Article 1 0 0
Level2 Comprehension Level Skimming - Getting the general sense of a passage A New Natural Wonders Named Tourism Article 1 0 0
Total Total Total 6 0 0
44
Results
  • Comparison was made for faculty

Biased Item Characteristics (faculty)
Level 3 Analysis Level Making inferences - Guessing about the text or the writers idea when some ideas are not directly stated. The internet addiction Social Science Article 6 1 16,67
Level 3 Analysis Level Making inferences - Guessing about the text or the writers idea when some ideas are not directly stated. The Best Foods You Arent Eating Health Article from a Magazine 4 0 0
Total Total Total 10 1 10
45
Results
  • Comparison was made for faculty

Biased Item Characteristics (faculty)
Level 4 Synthesis Level Finding topics - Identifying topics a major restructuring of the company Short News 1 0 0
Level 4 Synthesis Level Finding topics - Identifying topics civilians Short News 1 1 100
Level 4 Synthesis Level Finding topics - Identifying topics Spanish-colonial architecture Tourism Advertisement 1 0 0
Level 4 Synthesis Level Finding topics - Identifying topics wildlife Tourism Advertisement 1 1 100
Level 4 Synthesis Level Finding topics - Identifying topics sport activities Short News 1 0 0
Level 4 Synthesis Level Finding topics - Identifying topics Eight new natural wonders Tourism Article 1 0 0
Level 4 Synthesis Level Finding topics - Identifying topics The internet addiction Social Science Article 4 0 0
Total Total Total 10 2 20
Level 4 Synthesis Level Using vocabulary knowledge for effective reading - Guessing word meaning in context The internet addiction Social Science Article 5 0 0
Level 4 Synthesis Level Using vocabulary knowledge for effective reading - Guessing word meaning in context The Best Foods You Arent Eating Health Article from a Magazine 5 0 0
Total Total Total 10 0 0
46
Results
  • Comparison was made for faculty

Reading Ability Levels (faculty)
Ability Measured No. of Items No. of Biased Items
Level 1 Knowledge Level Preview - Previewing to get an idea of what you will find in the text 5 1 20
Level 1 Knowledge Level Scanning - Looking for specific information 9 3 33.34
Level2 Comprehension Level Skimming - Getting the general sense of a passage 6 0 0
Level 3 Analysis Level Making inferences - Guessing about the text or the writers idea when some ideas are not directly stated. 10 1 10
Level 4 Synthesis Level Finding topics - Identifying topics 10 2 20
Level 4 Synthesis Level Using vocabulary knowledge for effective reading - Guessing word meaning in context 10 0 0
Total 50 7 14
47
Results
  • Comparison was made for faculty

Biased Reading Topics (faculty)
Reading Topic No. of Items No. of Biased Items
1. The Bicycle Messenger and the Mysterious 5 2 40
2. Christmas Job Opportunities 4 0 0
3. Fish Stories 5 2 40
4. Suicide Blast Killed People 1 0 0
5. CEOs plan to develop the company 1 0 0
6. Environmental problem caused by some chemical 1 0 0
7. Visiting a place in America with a European Architecture Style 1 0 0
8. A New Type of Solar Energy Collector 1 0 0
9. A New Natural Wonders Named 1 0 0
10. a major restructuring of the company 1 0 0
11. civilians 1 1 100
12. Spanish-colonial architecture 1 0 0
13. wildlife 1 1 100
14. sport activities 1 0 0
15. Eight new natural wonders 1 0 0
16. The internet addiction 15 1 100
17. The Best Foods You Arent Eating 9 0 0
Total 50 7 14
48
Results
  • Comparison was made for faculty

Biased Text Types (Faculty)
Reading Topic No. of Items No. of Biased Items
1. Synopsis 5 2 40
2. Job Advertisement 4 0 0
3. General Science Article 5 2 40
4. Science and Technology Article 1 0 0
5. Short News 6 1 16.67
6. Tourism Advertisement 3 1 33.34
7. Tourism Article 2 0 0
8. Social Science Article 15 1 6.67
9. Health Article from a Magazine 9 0 0
Total 50 7 14
49
Results
  • Comparison was made for gender

Compare Gender
Delta Plot Chi Square IRT
1
2
3
4 4
5
8
9
10
50
Results
  • Comparison was made for gender

Gender Bias Content Measured Content Measured Content Measured
Gender Bias Ability Measured Reading Topic Text Type
4 1. Knowledge Level Scanning - Looking for specific information The Bicycle Messenger and the Mysterious Synopsis
16 2. Comprehension Level Skimming - Getting the general sense of a passage CEOs plan to develop the company Short News
22 4. Synthesis Level Finding topics - Identifying topics civilians Short News
32 4. Synthesis Level Using vocabulary knowledge for effective reading - Guessing word meaning in context The internet addiction Social Science Article
39 3. Analysis Level Making inferences - Guessing about the text or the writers idea when some ideas are not directly stated. The internet addiction Social Science Article
41 3. Analysis Level Making inferences - Guessing about the text or the writers idea when some ideas are not directly stated. The internet addiction Social Science Article
42 3. Analysis Level Making inferences - Guessing about the text or the writers idea when some ideas are not directly stated. The Best Foods You Arent Eating Health Article from a Magazine
48 4. Synthesis Level Using vocabulary knowledge for effective reading - Guessing word meaning in context The Best Foods You Arent Eating Health Article from a Magazine
49 4. Synthesis Level Using vocabulary knowledge for effective reading - Guessing word meaning in context The Best Foods You Arent Eating Health Article from a Magazine
51
Results
  • Comparison was made for gender

Biased Item Characteristics (Gender)
Ability Measured Reading Topic Text Type No. of Item No. of Biased Item
Level 1 Knowledge Level Preview - Previewing to get an idea of what you will find in the text The Bicycle Messenger and the Mysterious Synopsis 2 0 0
Level 1 Knowledge Level Preview - Previewing to get an idea of what you will find in the text Christmas Job Opportunities Job Advertisement 1 0 0
Level 1 Knowledge Level Preview - Previewing to get an idea of what you will find in the text Fish Stories General Science Article 2 0 0
Total Total Total 5 0 0
Level 1 Knowledge Level Scanning - Looking for specific information The Bicycle Messenger and the Mysterious Synopsis 3 1 33.34
Level 1 Knowledge Level Scanning - Looking for specific information Christmas Job Opportunities Job Advertisement 3 0 0
Level 1 Knowledge Level Scanning - Looking for specific information Fish Stories General Science Article 3 0 0
Total Total Total 9 1 11.12
52
Results
  • Comparison was made for gender

Level2 Comprehension Level Skimming - Getting the general sense of a passage Suicide Blast Killed People Short News 1 0 0
Level2 Comprehension Level Skimming - Getting the general sense of a passage CEOs plan to develop the company Short News 1 1 100
Level2 Comprehension Level Skimming - Getting the general sense of a passage Environmental problem caused by some chemical Short News 1 0 0
Level2 Comprehension Level Skimming - Getting the general sense of a passage Visiting a place in America with a European Architecture Style Tourism Advertisement 1 0 0
Level2 Comprehension Level Skimming - Getting the general sense of a passage A New Type of Solar Energy Collector Science and Technology Article 1 0 0
Level2 Comprehension Level Skimming - Getting the general sense of a passage A New Natural Wonders Named Tourism Article 1 0 0
Total Total Total 6 1 16.67
Level 3 Analysis Level Making inferences - Guessing about the text or the writers idea when some ideas are not directly stated. The internet addiction Social Science Article 6 2 33.34
Level 3 Analysis Level Making inferences - Guessing about the text or the writers idea when some ideas are not directly stated. The Best Foods You Arent Eating Health Article from a Magazine 4 1 25
Total Total Total 10 3 30
53
Results
  • Comparison was made for gender

Level 4 Synthesis Level Finding topics - Identifying topics a major restructuring of the company Short News 1 0 0
Level 4 Synthesis Level Finding topics - Identifying topics civilians Short News 1 1 100
Level 4 Synthesis Level Finding topics - Identifying topics Spanish-colonial architecture Tourism Advertisement 1 0 0
Level 4 Synthesis Level Finding topics - Identifying topics wildlife Tourism Advertisement 1 0 0
Level 4 Synthesis Level Finding topics - Identifying topics sport activities Short News 1 0 0
Level 4 Synthesis Level Finding topics - Identifying topics Eight new natural wonders Tourism Article 1 0 0
Level 4 Synthesis Level Finding topics - Identifying topics The internet addiction Social Science Article 4 0 0
Total Total Total 10 1 10
Level 4 Synthesis Level Using vocabulary knowledge for effective reading - Guessing word meaning in context The internet addiction Social Science Article 5 1 20
Level 4 Synthesis Level Using vocabulary knowledge for effective reading - Guessing word meaning in context The Best Foods You Arent Eating Health Article from a Magazine 5 2 40
Total Total Total 10 3 30
54
Results
  • Comparison was made for gender

Reading Ability Levels (Gender)
Ability Measured No. of Items No. of Biased Items
Level 1 Knowledge Level Preview - Previewing to get an idea of what you will find in the text 5 0 0
Level 1 Knowledge Level Scanning - Looking for specific information 9 1 11.12
Level2 Comprehension Level Skimming - Getting the general sense of a passage 6 1 16.67
Level 3 Analysis Level Making inferences - Guessing about the text or the writers idea when some ideas are not directly stated. 10 3 30
Level 4 Synthesis Level Finding topics - Identifying topics 10 1 10
Level 4 Synthesis Level Using vocabulary knowledge for effective reading - Guessing word meaning in context 10 3 10
Total 50 9 18
55
Results
Reading Topic No. of Items No. of Biased Items
1. The Bicycle Messenger and the Mysterious 5 1 20
2. Christmas Job Opportunities 4 0 0
3. Fish Stories 5 0 0
4. Suicide Blast Killed People 1 0 0
5. CEOs plan to develop the company 1 1 100
6. Environmental problem caused by some chemical 1 0 0
7. Visiting a place in America with a European Architecture Style 1 0 0
8. A New Type of Solar Energy Collector 1 0 0
9. A New Natural Wonders Named 1 0 0
10. a major restructuring of the company 1 0 0
11. civilians 1 1 100
12. Spanish-colonial architecture 1 0 0
13. wildlife 1 0 0
14. sport activities 1 0 0
15. Eight new natural wonders 1 0 0
16. The internet addiction 15 3 20
17. The Best Foods You Arent Eating 9 3 33.34
Total 50 9 18
  • Comparison was made for gender

Biased Reading Topics (Gender)
56
Results
  • Comparison was made for gender

Biased Text Types (Gender)
Reading Topic No. of Items No. of Biased Items
1. Synopsis 5 1 20
2. Job Advertisement 4 0 0
3. General Science Article 5 0 0
4. Science and Technology Article 1 0 0
5. Short News 6 2 33.34
6. Tourism Advertisement 3 0 0
7. Tourism Article 2 0 0
8. Social Science Article 15 3 20
9. Health Article from a Magazine 9 3 33.34
Total 50 9 18
57
Conclusion
  • Prior Knowledge (Faculty) Bias

Level 1 Knowledge Level Preview - Previewing to
get an idea of what you will find in the text
Text Type General Science Article
Fish Stories
Level 1 Knowledge Level Scanning - Looking for
specific information
Synopsis
The Bicycle Messenger and the Mysterious
Level 4 Synthesis Level Finding topics -
Identifying topics
Civilians
Short News
Wild life
Tourism Advertisement
58
Conclusion
  • Gender Bias

Level 3 Analysis Level Making inferences -
Guessing about the text or the writers idea when
some ideas are not directly stated.
Social Science Articles
Internet Addiction
Level 4 Synthesis Level Using vocabulary
knowledge for effective reading - Guessing word
meaning in context
The Best Foods You Arent Eating
Health Article from Magazine
59
Ex.1 Internet Addicts
  • The US could be rife with "internet addicts" who
    are as clinically ill as alcoholics, according to
    psychiatrists involved in a nationwide study. The
    study, carried out by researchers at Stanford
    University School of Medicine in California, US,
    indicates that more than one in eight US
    residents show signs of "problematic internet
    use".
  • The Stanford researchers interviewed 2,513
    adults in a nationwide survey. According to the
    study's lead author Elias Aboujaoude, the most
    disturbing result is the discovery that some
    people hide their internet surfing, or go online
    to cure foul moods behavior that mirrors the
    way alcoholics behave. " And, obviously,
    something is wrong when people go out of their
    way to hide their internet activity," Aboujaoude
    says. .

60
Ex.2 The Best Foods You arent Eating
  • The Best Foods You Aren't Eating
  • By JONNY BOWDEN, Ph.D.
  • Although some guys aren't opposed to smoking
    some weed, most wouldn't think of eating one.
    It's a shame, really, since a succulent weed
    named purslane is not only delicious but also
    among the world's healthiest foods.
  • Guava beets and cinnamon are among the
    super-healthful foods
  • that you should probably be getting more of in
    your diet
  • Of course, there are many super foods that never
    see the inside of a shopping cart.
  • Some you've never heard of, and others you've
    simply forgotten about. That's why we've rounded
    up the best of the bunch. Make a place for them
    on your table and you'll instantly upgrade your
    health -- without a prescription. .

61
Item Bank Software
Software Structure
(item code assignments) (item instoration) (test
generation) (test utilization)
62
Methodology
  • Item Bank Software

Toolbar
  • Menu File

63
Methodology
  • Item Bank Software

Toolbar
  • Menu File
  • Create Exam

64
Methodology
  • Item Bank Software

Toolbar
  • Menu File
  • Create Exam
  • Manage Form

65
Methodology
  • Item Bank Software

Toolbar
  • Menu File
  • Create Exam
  • Manage Form
  • Help

66
Methodology
  • Item Bank Software
  • Create Exam

67
Methodology
  • Item Bank Software
  • Create Exam
  • View Report

68
Methodology
  • Item Bank Software
  • Create Exam
  • View Report
  • Exit

69
Conclusion
  • Efficiency of the Item Bank Software (1st Trial)
  • Interface design
  • Easy to read
  • Unpleasant screen design
  • Application
  • Very convenient to retrieve the test items
  • The storing process is complicated.

70
END OF THE PRESENTATION

SINCEREST THANKS
About PowerShow.com