Using Structured Mixture IRT Models to Study Differential Item Functioning

1
Using Structured Mixture IRT Models to Study
Differential Item Functioning

• Yunyun Dai, Robert J. Mislevy
• Nov 20th, 2006
• The work was supported by the MCAT Graduate
  Student Research Program, as administered by
  Medical College Admission Test, Association of
  American Medical Colleges. The findings and
  opinions expressed in this report do not reflect
  the positions or policies of the Association of
  American Medical Colleges.

2
Outline

• Background Knowledge of DIF Analysis
• Framework of DIF Analysis
• Structured Mixture IRT Model
• WINBUGS Code
• Model Identification

3
Definition of DIF

• Differential item function (DIF)
• is defined as item performing differently for
  one group of examines from the way it performs
  for another group of examinees. (Embretson
  Reise, 2000)
• The presence of nuisance dimensions intrudes
  on the measurement of the ability of interest and
  leads to the differential item functioning.
  (Ackerman, 1992).

4
General Assumption on Examinees

• Homogenous Population
• Homogenous vs. Heterogeneous
• Cause of DIF

5
Potential Causes of DIF

• Demographic Characters
• (gender, ethnicity, etc.)
• Problem-solving Strategies
• (Mislevy Verhelst, 1990)
• Subcategory of Problem Types
• (Cohen Bolt, 2005)

6
Manifest Group DIF Analysis

• Manifest Group Characteristics
• Focal group vs. Reference group
• Q for Who the item function differently?

7
Latent Class DIF Analysis

• Grouping Variables are unobserved
• Group memberships are estimated based on
  examinees response patterns
• Q Why the item function differently?

8
Framework of DIF Analysis
9
Framework of DIF Analysis Manifest Group DIF
Analysis

• Politically oriented
• e.g., Maryland High School Assessment

10
Framework of DIF AnalysisLatent Class DIF
Analysis

• Exploratory Approach
• Confirmatory Approach

11
Structured Mixture IRT Model (1)
12
Structured Mixture IRT Model (2)

• Person covariate g
• (Math courses taken, gender group)
• Item covariate q
• (content areas, cognitive skills)

13
Structured Mixture IRT Model (3)

• Person covariate
• Q How persons covariate variables influence
  examinees probability belonging to certain
  latent group
• Item covariate
• Q How item covariates variables affect item
  difficulty

14
Specification of Persons Covariate (1)

• Gidcat( Pi, )
  (1)
• log(PHIi,j) lt- intj sljgenderi (2)
  
  
• Pi,jlt- PHIi,j / sum(PHIi,) (3)
  
• WINBUGS Constraint int10 sl10
  
  

15
Specification of Persons Covariate (2)

• Table 1 Relationship between parameters in
  logistic regression and proportion of persons
  covariate

16
Specification of Persons Covariate (3)
17
WINBUGS Specification

• Model
• latent group proportion and examinees' average
  ability fo each latent group
• for (j in 1J) P.totj lt-
  sum(P,j)/N mutj dnorm(0,1)
• intercept and slope in the logistic regression
  function
• for (j in 2J) intj
  dnorm(0,1) slj dnorm(0,1)
• int 1lt-0 sl1 lt- 0
• specification of item difficulty for each latent
  group
• for (m in 1M-1) b1,m
  dnorm(0,1)b2,m lt- b1,m difm difm
  dnorm(0,0.5)
• b1,Mlt- -1sum(b1,1(M-1))
  b2,M lt- -1sum(b2,1(M-1))
  difMlt-b2,M-b1,M
• latent group membership for every examinee
• for (i in 1N) Gidcat( Pi, )
  gmemcat1ilt-equals(Gi,1)
• latent ability distribution for every examinee
• for (i in 1N) thetaidnorm(mut
  Gi,1)
• logistic regression function

18
Model Identification (1)

• Identification is a challenge in IRT models,
• DIF models,
• LLTM models,
• mixture models,
• structured models with covariates
• and this research project is attempting to
  combine all of these modeling ideas at once.

19
Model Identification (2)

• Label switching problem
• Chung, Loken, and Schafer (2004) suggested
  assigning one observation to each component as
  prior may effectively eliminate the problem.

20
Model Identification (3)

• Graph1 MCMC runs for latent class proportion
  based on simulated data for stage 7 (50, 50)

21
Model Identification (4)

• Graph2 MCMC runs for latent class proportion
  based on simulated data for stage 2 (30, 70)

22
Model Identification (5)

• Graph3 MCMC runs with prior information added
  (the simple solution) for latent class proportion
  based on simulated data for stage 2 (30, 70)

23
Model Identification (6)

• Graph 4 MCMC runs for latent class proportion
  based on simulated data for stage 5 (30, 70)

24
Model Identification (7)

• Graph 5 MCMC runs for latent class proportion
  based on simulated data for stage 5 (30, 70)

25
References

• Ackerman, T. A. (1992). A didactic explanation of
  item bias, item impact, and item validity from a
  multidimensional perspective. Journal of
  Educational Measurement, 29, 67-91.
• Chung, H., Loken, E., and Schafer, J. L. (2004),
  "difficulties in drawing inferences with
  finite-mixture models a simple example with a
  simple solution," the American statistician, 58,
  152-158.
• Cohen, A.S., Bolt, D.M. (2005). A mixture
  model analysis of differential item functioning.
  Journal of Educational Measurement Sum 2005 Vol
  42(2) 133-148.
• Embretson, S.E., Reise, S.P. (2000). Item
  Response Theory for Psychologists. Mahwah, N.J.
  Lawrence Erlbaum.
• Fischer, G. H. (1973). The linear logistic test
  model as an instrument of educational research.
  Acta Psychologica, 37, 359-374.
• Mislevy, R. J., Verhelst, N. (1990). Modeling
  item responses when different subjects employ
  different solution strategies. Psychometrika, 55,
  195-215.