Effective methods for the use, creation, analysis, and interpretation of short-answer student conceptual evaluations.

1
Effective methods for the use, creation,
analysis, and interpretation of short-answer
student conceptual evaluations.

• Ronald Thornton
• Professor of Physics and Education
• Center for Science Math Teaching
• Tufts University

2
What was I thinking?

• Ill paint your house and walk your dog as well.

3
In Defense of Thoughtful Multiple Choice
Conceptual Assessment

• Ronald Thornton
• Professor of Physics and Education
• Center for Science Math Teaching
• Tufts University

4
Modest Suggestions from a Chemically Illiterate
Physicist.

• Ronald Thornton
• Professor of Physics and Education
• Center for Science Math Teaching
• Tufts University

5

Center for Science and Math Teaching Tufts
University

6
Funding

• NSF
  National Science Foundation
• FIPSE Fund
  for the Improvement of Post Secondary Education
• US Department of Education

7
Wouldnt it be nice if teachers could understand
what students know from a simple conceptual
evaluation?

• and they knew what to do to help the student learn

8
What use might this talk be?

• If you intend to develop a chemistry concept
  inventory these suggestions may help you make it
  more useful.
• If you intend to use a chemistry concept
  inventory these ideas should help you pick a
  useful one.

9
We have spent years

• Creating effective learning environments for
  introductory science(physics) courses (curricula,
  tools, pedagogical methods, group structures)
• And developing methods of conceptual evaluation
  to measure student learning and guide our
  progress.

10
Why Multiple Choice?

• More easily administered to large numbers of
  students.
• Evaluation takes less time.
• Student responses can be reliably evaluated even
  by the inexperienced.
• Can be designed to guide instruction.
• With proper construction, student views can be
  evaluated from the pattern of answers, changes
  over time can be seen, frequency of student views
  can be measured.
• Multiple choice combined with open response can
  help the teacher/researcher explicate the
  students response.

11
Why not?

• Every good educator knows multiple choice
  questions are no good.
• Badly constructed multiple choice can give
  misleading results.
• Unless very carefully constructed, multiple
  choice will not identify student thinking.
• The choices may be inappropriate when used with
  different audiences

12
First steps

• Why do you want to make (use) a conceptual
  evaluation?
• In what conceptual area do you want to know how
  students think?

13
Why?

• There are pre-requisite areas of conceptual
  knowledge that students need to know to actually
  understand chemistry.

14
What? Three modest suggestions.

• Explore student beliefs in the atomic nature of
  matter. (students may say atoms exist but few
  believe it in any functional matter)
• Explore student beliefs the dynamic nature of
  equilibrium. (Most students seem to have a static
  model)
• Explore student beliefs about the difference
  between heat energy and temperature. (Most
  students do not clearly make this distinction.)

15
Our research has shown.

• Student conceptual responses can be context
  dependent.
• Student domains of applicability can be
  different from those of a scientist.
• Students (and scientists) can hold apparently
  inconsistent views simultaneously. (and it
  doesnt mean they are stupid.)
• Conceptual transitions are not instantaneous.
• There is statistical evidence of a hierarchy of
  student conceptual views.
• You can do more with large-scale conceptual
  evaluation than just generating a single number.

16
Good Practice for the Construction of Conceptual
Multiple Choice

• All answers, "right or wrong," should help
  evaluate student views.
• Derive the choices in the questions from from
  student answers to free response questions and
  from student interviews.
• Check to see students almost always find an
  answer that they are satisfied with. Random
  answers should be few.
• Ask similar questions in different
  representations.
• Check results with different student populations.
• (more)

17
Good Practice (continued)

• Look at correlations among questions and use
  patterns to understand student thinking.
• Understand the implications of correct and
  incorrect answers to their performance on other
  tasks.
• Check for gender differences
• Identify circumstances for false positive
  answers
• If at all possible, construct the evaluation so
  it is useful to guide instruction.

18
Multiple Choice Conceptual Evaluation

• Conceptual evaluation for
• kinematics (description of motion) and
• dynamics (force and motion which is well
  characterized by Newtons Laws).
• Force Motion Conceptual Evaluation (FMCE)
• Conceptual evaluation for heat energy and
  temperature
• The Heat and Temperature Conceptual Evaluation
  (HTCE)
• Both developed by the Center for Science and Math
  Teaching at Tufts

19
Using the FMCE as an example

• Student answers correlate well (well above 90)
  with written short answers in which students
  explain the reason for their choices
• Almost all students pick choices that we can
  associate with a relatively small number of
  student models. (Conceptual Dynamics, R.K.
  Thornton in ICUPE proceedings edited by Redish)
• Testing with smaller student samples shows that
  those who can pick the correct graph under
  these circumstances are almost equally successful
  at drawing the graph correctly without being
  presented with choices.

20
FMCE as example

• Because we are able to identify statistically
  most student views from the pattern of answers
  (and because there are very few random answers),
  we are also able to identify students with less
  common beliefs about motion and follow up with
  opportunities for interviews or open-ended
  responses to help us understand student thinking.
• The use of an easily administered and robust
  multiple choice test has also allowed us and
  others to track changes in student views of
  dynamics and to separate the effects of various
  curricular changes on student learning.

21
FMCE as example

• Use multiple representations
• The Force Graph questions require explicit
  knowledge of coordinate systems and graphs but
  require little reading.
• The Force Sled questions use natural language
  and make no explicit reference to a coordinate
  system or graphs.

22
(No Transcript)
23
(No Transcript)
24
Comparison with short answer

• As with all the questions on the test students
  who answered correctly were also able to describe
  in words why they picked the answers they did.
• Statistically one of the last questions to be
  answered in a Newtonian manner is the force on a
  cart rolling up a ramp as it reverses direction
  at the top (question 9).

25
Back to best practices. Consider

• All answers, "right or wrong," should help
  evaluate student views.
• Derive the choices in the questions from from
  student answers to free response questions and
  from student interviews.
• Check to see students almost always find an
  answer that they are satisfied with. Random
  answers should be few.
• Look at correlations among questions and use
  patterns to understand student thinking.

26
An example from the HT Conceptual Evaluation

• Distinguishes different student models for the
  relationship between heat and temperature.

27

28
Results by category

29
What about 1 number results

• Not my favorite, but useful in some situations
• Lets compare the performance of 350 RPI students
  in the beginning physics course on the FMCE and
  the FCI

30
(No Transcript)
31
Still one number

• Lets compare the performance of 350 RPI students
  in the beginning physics course on the FMCE and
  the FCI

32
Correlation Coefficient 0.791
33
Correlation Coefficient 0.8309
34
Are the evaluations the same?

• Yes? Very high correlations (about 0.8 pre and
  post with different instructional methods)
• Yes? A high score on one implies a high score on
  the other.
• No? FCI fractional scores are almost always
  higher than FMCE scores
• No? Evaluations are measuring different things
• No? A low score on the FMCE (non-Newtonian
  student) does not imply a low score on the FCI
• Lets look at a group of non-Newtonian students

35
(No Transcript)
36
The conceptual threshold effect(looking at
pre-post correlations)

37
Pre/Post Evaluation--The Threshold Effect
Tufts University Calculus-based Physics (N181)
FMCE Post vs. Pre
1.00
0.80
0.60
0.40
Spring 1994 (N48)
0.20
Spring 1995 (N37)
Spring 1997 (N43)
Spring 1998 (N53)
0.00
0.00
0.20
0.40
0.60
0.80
1.00
Before Instruction
38
University Physics Courses Before Instruction
Average College and University Results
Before Instruction
of Students Understanding Concepts
39

University Physics Courses After Normal
Instruction
Average College and University Results
After Traditional Instruction
Before Instruction
of Students Understanding Concepts
40

Physics Science Courses Using New Methods

• We have evidence of substantial, persistent
  learning of such physical concepts by a large
  number of students in varied contexts in courses
  and laboratories that use methods I am about to
  describe.
• Such methods also work for students who have
  traditionally had less success in physics and
  science courses women and girls, minority
  students, and those who are badly prepared.

41

University Physics Courses After New Methods
Average College and University Results
After New Methods
After Traditional Instruc.
Before Instruction
of Students Understanding Concepts
42
Our Instructional and Assessment Philosophy

• I still dont have all of the
• answers, but Im beginning to ask
• the right questions.