Application of Generalizability Theory to Concept-Map Assessment Research

1
Application of Generalizability Theory to
Concept-Map Assessment Research

• Yue Yin Richard J. Shavelson
• Stanford Educational Assessment Laboratory (SEAL)
• Stanford University
• CRESST
• AERA 2004, San Diego CA

2
Overview

• Part 1 Feasibility of applying G-theory to
  concept-map assessment (CMA) research
• - Examining the dependability of CMA scores
• - Designing a CMA for a particular
  application
• - Narrowing down alternatives
• Part 2 Empirical study of using G-theory to
  compare two CMAs
• - Construct-a-map with created linking
  phrases (C)
• - Construct-a-map with selected linking
  phrases (S)

3
A Concept-map
4
Variations in CMA
Components Variation Examples
Task -Topic only -Topic and concepts (C) -Topic, concepts and linking phrases (S) -Topic, incomplete concepts or incomplete linking phrases (fill-in-the-nodes or fill-in-the- lines)
Response -Computer -Paper-pencil
Scoring System -Link score -Concept score -Proposition score -Structure score
5
Part 1

• Feasibility of Applying
• G Theory to CMA Research

6
Viewing CMA with G theory

• Basic idea
• A particular type of score, given by a
  particular rater, based on a particular type of
  concept map, on a particular occasion, is a
  sample from a multifaceted universe.
• Object of measurement
• Peoplethe variation in students knowledge
  structure
• Facets
• Task (concept proposition), response format,
  scoring system, rater, occasion,

7
G theory vs. CTT
Similarity

• Concept-term sampling
• Proposition sampling
• Rater sampling
• Occasion sampling

• Equivalence of alternate forms
• Internal consistency
• Inter-rater reliability
• Stability over time

G Theorys Advantage

• Integrate conceptually and simultaneously
  evaluate all the technical properties above
• Estimate not only the effect of individual
  facets, but also interaction effects
• Permits us to optimize an assessments technical
  quality

8
Examining Technical Properties Designing
Assessments

• Examining dependability (G study)
• How well can a measure of students declarative
  knowledge structure be generalized across concept
  map tasks? scoring systems? occasions? raters?
  propositions? different concept samples?
• Designing an assessment (D study)
• How many concept map tasks, scoring systems,
  occasions, raters, propositions, and/or different
  concept samples will be needed to obtain a
  reliable measurement of students declarative
  knowledge structure?

9
Narrowing Down Alternatives

• Task
• - Which task type is more reliable over raters,
• occasions, propositions, concept samples?
• - Accordingly, this task needs fewer raters,
  occasions,
• propositions, and concept samples.
• Scoring system
• - Which scoring system is more reliable over
  raters,
• occasions, propositions, concept samples?
• - Accordingly, this scoring system needs fewer
  raters,
• occasions, propositions, and concept samples.

10
Part 2

• Empirical Study of Using
• G-theory to Compare
• Two CMAs

11
Two Frequently Used CMAs

• Construct-a-map with created linking phrases
  (C)--Provides a cognitively valid measure of
  knowledge structure (e.g., Ruiz-Primo et al.,
  2001 Yin et al., 2004)
• Construct-a-map with selected linking phrases
  (S)--Provides an efficient way to measure
  knowledge structure (e.g., Klein et al., 2001)

12
Method

• Concept-map task
• - 9 Concepts (for C S)
• water, volume, cubic centimeter, wood,
  density, mass, buoyancy, gram, and matter
• - 6 Linking phrases (for S only)
• is a measure of
• has a property of
• depends on
• is a form of
• is mass divided by
• divided by volume equals

• Participants
• - 92 eighth-graders
• - 46 girls
• - previously studied a
• related unit
• - no related instruction
• between two occasions
• Procedures
• C ? S (n 22)
• S ? C (n 23)
• C ? C (n 26)
• S ? S (n 21)

13
Criterion Map
14
Mandatory Propositions
15
Source of Variation

• CS SC
• Person (P)
• Proposition/Item (I)
• Format (F)
• P x F
• P x I
• F x I
• P x F x I, e

• CC SS
• Person (P)
• Proposition/Item (I)
• Occasion (O)
• P x O
• P x I
• O x I
• P x O x I, e

16
Variance Component Estimate
17
G Study in SC CS
18
G Study in CC SS
19
D Study for C CMA
20
D Study for S CMA
21
Conclusions

• G study pinpoints multiple sources of measurement
  error, thereby giving insight into how to improve
  the reliability and applicability of CMA via a D
  study
• C and S mapping tasks are not equivalent in their
  technical properties
• Fewer occasions and propositions are needed in S
  than C to get a reliable evaluation of students
  declarative knowledge structure

22
Thank You for Your Interest! ?

• To get the complete paper, please either
• contact Yue Yin at
• yyin_at_stanford.edu
• Or
• download the file directly at
• http//www.stanford.edu/dept/SUSE/SEAL/