The IEA Civics Education Study: Assessing for DIF between two English-speaking Countries using a complex sample design.

1
The IEA Civics Education Study Assessing for DIF
between two English-speaking Countries using a
complex sample design.

• Paul Rock Krech, MSW
• Richard Rivera, BA

2
Items

• 38 Multiple Choice Items dichotomously Scored
• 0 Incorrect answer
• 1 Correct answer

3
Sampling design

• IEA Civic Education (CivEd) study used a
  two-stage stratified cluster approach to sample
  students.
• The schools were sampled in the first stage and
• classrooms were sampled in the second stage
  (Sibberns Foy, 2004a).

4
Why stratification?

• Improve efficiency of sample design making
  estimates more reliable.
• To apply different sample designs (i.e.
  disproportionate sample size) to different groups
  of schools.
• To ensure adequate representation in the sample
  of specific groups from the population.

5
Stages of Sampling

1. Creating stratification variables to determine
   the sampling frame.
2. One classroom (or homeroom) was drawn from each
   sampled school for a targeted grade (e.g., 8th,
   9th, or 10th). Grades were targeted that
   typically had 14-year-old students.

6
Complex sampling variables

• A weight variable (TOTWGT), across participants
  is an estimate of the population size.
• (HOUSEWGT) is equal to the raw weight divided by
  its mean and across students is equivalent to
  the sample size.
• (SCHOOLID) is the cluster variable.

7
Stage Number Weight variable
Stage 2 Cluster
House Weight
IDSCHOOL
Stage 1 Strata
School 1
Sampled Country
Students 14.0-14.11 YO
School 2
IDSTUDENT
IDCOUNTRY
. . . schoolj
Intact Classrooms
IDCLASS
Other schools (Could also be regions)
8
Item example 1 Domain I (Democracy), Sub-domain
B (Institutions Practices in Democracy).
Bs223 ELECTION LEAFLET HAS BEEN ISSUED BY
9
Item example 2 (Domain III Social Cohesion and
diversity).

• BS107Z HAVING MANY POLITICAL AND SOCIAL
  ORGANIZATIONS TO JOIN IS IMPORTANT IN A DEMOCRACY
  BECAUSE
• 1 They defend members who are arrested.
• 2 They are sources of taxes for the
  government.
• 3 They allow citizens to express different
  points of view.
• 4 They provide a way to tell citizens about
  new laws.

10
(No Transcript)
11
(No Transcript)
12
EFA results Approx. 5 factors
13
Goodness of Fit Summary
Table 7 Table 7 Table 7 Table 7 Table 7 Table 7 Table 7 Table 7 Table 7
Goodness of Fit Summary Goodness of Fit Summary Goodness of Fit Summary Goodness of Fit Summary Goodness of Fit Summary Goodness of Fit Summary Goodness of Fit Summary Goodness of Fit Summary Goodness of Fit Summary
Model Model Model ?2 df AIC CFI RMSEA
Single-Group Single-Group
USA 130.73 27 76.73 .94 .04
England 51.04 27 -2.96 .99 .02
Multiple Group Multiple Group
Baseline (Unconstrained) Baseline (Unconstrained) 198.22 54 93.77 .95 .03
Constrained 217.77 62 90.22 .96 .02
p lt .05 p lt .05 p lt .05 p lt .05 p lt .05 p lt .05 p lt .05 p lt .05 p lt .05
14
Two group baseline model
c2SB(54) 198.2151, p lt .0001 AIC
183.50 NNFI 0.95 CFI 0.96 RMSEA
0.03
15
Mantel-Haenszel Chi-Square Estimates

• Scores Right Wrong
• _________________
• US a b nUS
• _________________
• ENG c d nENG
• _________________
• Total n n nTOT

16
MH and IRT results (9-items)
17
DIF on Item BS231z WHICH IS AN OPINION?
1 Actions businesses solve environmental
problems. 2 Businesses contribute to the
pollution. 3 Businesses co-operate to
diminish acid rain. 4 Businesses add to
water pollution from different sources.
Chi-Square Results Focal Group ENG Reference
Group USA F-statistic p lt F
ChiSq p lt ChiSq N Wgt N 8.811
0.003 8.811 0.003 3011
579910
18
(No Transcript)