Developing

1
Developing TestingConcept Inventories
David Klappholz (Comp. Sci.) davidk6_at_gmail.com) St
even J. Condly (Ed. Psych.) scondly_at_gmail.com)
2
Origin/History of CIs

• David Hestenes ASU physics
• Hestenes graduate student
• Quantitative/algorithmic questions
• Conceptual questions
• Hestenes response, etc.
• FCI Force Concept Inventory
• Eric Mazur Harvard physics

3
Example Test Items
CI-type Item
Traditional Item
A 1500 kg car, moving at 50 kph, collides with a
65,000 kg tree and comes to a stop in 0.2 sec.
During the collision, the magnitude of the force
of the tree on the car equals approximately
__. A. 105,000 B. -105,000 C. 4,550,000 D.
-4,550,000

• During the collision between the small
• car and the large tree,
• the car exerts a greater force on the tree than
  the tree exerts on the car.
• the tree exerts a greater force on the car than
  the car exerts on the tree.
• neither exerts a force on the other.
• the car exerts the same magnitude of force on the
  tree as the tree exerts on the car.

4
High-Frequency Response to Example CI-Type Item
the tree exerts a greater force on the car than
the car exerts on the treeas is evident from
the fact that the tree suffers a small bruise,
but the car suffers significant damage
5
Hestenes Vision of a Concept Inventory Tests
Purpose

• Covers aspects of a single concept. (Newtonian
  force)
• Intended to be used to inform faculty exactly
  where pedagogy should be changed so that students
  will correctly model the various aspects of the
  relevant concept. That is
• not to grade students (Hestenes)
• maybe to evaluate quality of instruction
  (Hestenes)
• to predict performance in advanced courses
  (Hestenes)
• If we understand the precise nature of the mental
  mis-modeling, we dont need radically different
  pedagogy rather, we can simply address the
  mis-modeling directly. (Others, including us)

6
Hestenes Vision of a Concept Inventory
TestsFunction

• Identifies
• mental mis-modelings
• of a single critical/fundamental concept
• doesnt test for
• ability to solve problems by plugging into
  formulas or applying algorithms
• mental mis-modelings over all the concepts in a
  complete course
• Newtons concept of force is only about 1/6 of
  the typical Intro to Physics
• FCI isnt concerned with momentum, energy,
• Cant both be of reasonable size and validatable
  for identifying specific mental mis-modelings if
  it covers an entire course.

7
Current STEM CIs

• Biology
• Calculus
• Chemical Equilibrium
• Chemistry
• Circuits
• Computer Engineering
• Discrete Mathematics
• Dynamics
• Electromagnetics
• Electronics
• Fluid Mechanics
• Geosciences
• Heat Transfer

• Lunar Phases
• Materials
• Nanotechnology
• Natural Selection
• Organic Chemistry
• Physics
• Signals Systems
• Statics
• Statistics
• Strength of Materials
• Thermal Transport Science
• Thermodynamics
• Waves

8
Many/Most Current CIs

• Test conceptual mis-modeling of all the concepts
  in an entire course, and, consequently
• cannot practically be validated to confirm
  identities of specific mental mis-modelings
• are essentially conceptual final exams
• some actually contain quantitative / algorithmic
  items
• Arent properly validated even as conceptual
  final exams, unless theyre to be used only in a
  single offering of the relevant course
• Test instruments to be used on a large
  population, e.g., SAT, require more than face and
  content validity.

9
What is a Concept?

• In the terminology of educational/cognitive
  psychology, a concept is denominated by a noun,
  possibly with modifiers, adjectival or otherwise.
  
• Thus, for example, F  ma is not a concept
  rather it is, in colloquial English, a rule or a
  law that governs the relationship among the
  concepts of force, mass, and acceleration or,
  simply, a relationship among the three concepts.
• In educational psychology, F ma is referred to
  as a proposition involving the concepts of force,
  mass, and acceleration.

10
Hestenes Modeling of Newtons Concept of Force

• The target concept for the FCI is thus force.
• The primitive associated concepts required to
  define Newtons concept of force and to work with
  it (and therefore concepts to which CI items
  relating to force must refer) are
• physical object
• mass of a physical object
• position in 2- or 3-dimensional space, of a
  physical object
• vector in 2 or 3 dimensions
• velocity of a physical object, as a vector
  quantity
• acceleration of a physical object as a vector
  quantity

11
Special Cases That Hestenes Thought to be
Important

• Specific instances of force about which Hestenes
  felt that questions would help reveal students
  mental (mis)models are
• Contact forces, further subdivided into
• solid contact force further subdivided into
• passive solid contact force
• impulsive solid contact force
• friction
• fluid contact force, further subdivided into
• air resistance
• buoyant fluid contact force, e.g., air pressure
• Non-contact forces, such as gravity

12
Hestenes Hypothesized Mental Mismodelings

• There are 30 of them
• They cover 2 pages
• We can provide details

13
Structure of Hestenes 29 FCI ItemsProblematic

• 14 items included at least one incorrect response
  with no identified mis-modelings, as follows
• 6 FCI questions have 1 incorrect response for
  which none of the 30 mis-modelings is identified
• 7 questions have 2 incorrect responses for which
  none of the 30 mismodelings is identified
• 1 question has 3 incorrect responses for which
  none of the 30 mismodelings is identified

14
Structure of Hestenes FCI Items (cont.)

• A total of 13 items included incorrect responses
  with multiple mis-modelings
• 12 incorrect responses have 2 mis-modelings
  identified with them
• 1 incorrect response has 3 mis-modelings
  identified with it

15
Structure of Hestenes FCI Items (cont.)

• A total of 18 items had multiple incorrect
  responses identified with the same mis-modeling
• 14 questions have 2 incorrect responses
  identified with the same mis-modeling
• 3 questions have 3 incorrect responses identified
  with the same mis-modeling
• 1 question has 4 incorrect responses identified
  with the same mis-modeling

16
Structure of Hestenes FCI Items (cont.)

• Only two of the FCIs 29 questions had four
  incorrect responses where each distractor was
  identified with exactly one of the 30
  mis-modelings

17
Proper CI Item Structure

• Item deals with only a single aspect of a single
  concept
• One correct answer
• Ideal Several wrong answers, each of which
  represents a different popular mis-modeling.

18
Ed. Psych. /Psychometric Issues

• Must be validated very differently from a test
  instrument that is to be used to grade students
• Classical Test Theory and Item Response Theory
  cant be used because they tell us to eliminate
  questions when
• item difficulty is too high or too low
• or when item discrimination is low or negative
  --
• even if they relate to modeling of critical /
  fundamental concepts
• Hestenes Halloun (con) argued with Huffman
  Heller (pro) about whether factor analysis is
  necessary to identify specific mental
  mis-modelings, but both were wrong
• Hestenes and Halloun about the lack of need for
  this level of validation to correctly identify
  mental mis-modelings
• Huffman Heller about using factor analysis for
  binary nominal variables

19
Proper CI Validation

• Content Validity SME agreement
• Latent Class Analysis to factor analyze CI
  results (nominal variables rather than ordinal,
  interval or ratio data)

20
Our Agenda

• Were interested in talking to you
• if youre seriously interested in constructing
  and validating (our vision of) a CI(s) in your
  field. (Doing so requires a large amount of work
  over multiple years)
• If theres a match between your interests and
  ours, funding may become available to support
  joint work with our group

21
Hypothesis re mental mis-modeling needs
validation
When the objects involved are every day objects
and no numbers are given, students use force in
its colloquial sense, not in its Newtonian sense.