Assessing Science Inquiry Skills

1
Interactive Computer Tasks in NAEP 2009 Mike
Timms, WestEd/CAESL CAESL2006 conference May 16,
2006
2
Why include ICTs in NAEP?

• Broad purpose is to tap performance expectations
  that are more advantageously assessed in this
  item format.
• ICTs complement the hands-on performance tasks,
  not replace them
• Committee was conscious that the 2009 Framework
  and Specifications will be in place until 2017
  and wanted to be forward-thinking

3
How will ICTs be used in NAEP?

• Interactive computer tasks (ICTs) will be
  administered to a sub-sample of students for
  every NAEP Science Assessment at grades 4, 8, and
  12.
• Tasks will be up to 30 minutes long.

4
What are the advantages of using ICTs?

• They circumvent the messiness and logistical
  challenges of hands-on performance tasks
• They can present extensive databases to assess
  students ability to select and evaluate
  information
• They overcome the difficulty of providing
  materials and training for complex tasks such as
  concept maps
• Can score electronically from student responses
  (e.g., using natural language processing)

5
When are ICTs advantageous?

• For scientific phenomena that cannot easily be
  observed in real time such as seeing things in
  slow-motion or speeded-up
• (e.g., slowing the motion of a wave or speeding
  the erosion caused by a river).
• For modeling scientific phenomena that are
  invisible to the naked eye
• (e.g., the movement of molecules in a gas).
• For working safely in lab-like simulations that
  would otherwise be hazardous or messy in a
  testing situation.
• (e.g., using dangerous chemicals)

6
When are ICTs advantageous?

• When several repetitions of an experiment in a
  limited testing time, while varying the
  parameters
• (e.g., rolling a ball down a slope while varying
  the mass, the angle of inclination, or the
  coefficient of friction of the surface).
• For searching the Internet and resource documents
  that provide high-fidelity real-world
• For manipulating objects in a facile manner
• (e.g., manipulating terms in a concept map)

7
Activity Selection Process
Presentation Process
Student
Administrator
Interactive Computer Tasks/ Evidence composite
database
Summary Report
Summary scoring process
Response processing
Adapted from Almond, Steinberg, Mislevy (2003)
Design of ICTs
8
Types of ICT items

• information search and analysis
• empirical investigation
• simulation
• concept mapping

9
Information search analysis example
Example from NCES study of Technology Rich
Environments - Hilary Persky and Randy Bennett at
ETS
10
Empirical Investigation Example
11
Simulation Example Procedural Knowledge
operate tools Access Organize search multiple
representational formats Analyze Interpret
Use visualization tools to investigate,

Source Quellmalz et al. (2004)
12
Example of student response to the simulation.
13
Example of Concept Mapping.
Adapted from Herl, ONeil, Chung, Schacter
(1999)
14
Challenges of ICTs

• Equipment costs/reliability
• Need to fit kids computer skills
• Content limits
• Psychometric new territory
• Not many existing examples

15
Interactive Computer Tasks in NAEP 2009 Mike
Timms, WestEd/CAESL mtimms_at_wested.org