Computer Coaches for Problem Solving Skills in Introductory Physics: Initial Data Analysis

1
Computer Coaches for Problem Solving Skills in
Introductory Physics Initial Data Analysis

• Andrew Mason, Ph.D.
• Kenneth Heller, Leon Hsu, Anne Loyle-Langholz,
  Qing Xu
• University of Minnesota, Twin Cities
• MAAPT Spring 2011 Meeting
• St. Marys University, Winona, MN
• 4/30/2011

Supported by NSF DUE 0230830 and DUE 0715615
and by the University of Minnesota
2
Outline

• Background
• Demonstration
• Preliminary findings
• Initial calibration data
• Currently Scoring using Rubric for Problem
  Solving (Docktor 2009)
• Methodology for further study

3
Research Study - Basic Method

• Use coaches in a calculus-based physics course
  for scientists and engineers
• Students are asked to volunteer volunteers are
  paid for their participation
• Assign students into 2 matched groups
• Variables for matching background information,
  e.g. HS physics math level, FCI/CLASS/math
  pretests

4
Previous Study Overview

• Study 1 (Fall 2010)
• 40 students, 1 lecture session of intro
  calculus-based class (20 for each group)
• Subset of coaches available energy (8),
  momentum (7) done over 4 weeks (4 per week)

5
Preliminary Results, Fall 2010

• Retention rate high in fall (18 of 21) for 15
  coaches
• 2 others completed 12 of 15
• All students found them useful
• Does this hold for other sections?
• What will happen with a larger set of coaches?

6
Preliminary Results, Fall 2010

• Student preference for each type
• Faculty tend to disagree with students (found
  type 1 tedious)
• 1 student initially preferred type 1 but switched
  to type 3 after gaining familiarity with physics

Most useful 2nd most useful Least useful
Type 1 13 3 1
Type 2 0 9 9
Type 3 4 5 8
7
Time between questions

• Average time range to complete between coaches
  between 20 and 40 minutes
• Percentage of correct answers seems to correlate
  with background info
• 2 sampled A students 80-90
• 2 sampled C students 60-70
• Students tend to stay on task
• Only 2 of 18 had at least one break of more than
  5 minutes for a question

8
Sample of logging function data

• Time between questions(one type 1 coach)
• Distribution suggests students are taking the
  tutors seriously
• Median time 4.58 s

9
Evaluating Problem Solving(Docktor 2009 Docktor
and Heller 2009)

• Rubric developed to evaluate student problem
  solutions
• Validity, reliability have been tested
• 2 raters use, discuss with each other until gt90
  agreement
• Five rubric categories (established by research
  on expert and novice problem solvers)
• Useful Description
• Physics Approach
• Specific Application of Physics
• Mathematical Procedures
• Logical Progression

10
Current Study Establishing a Baseline

• Study 2 (Spring 2011)
• 9 students, 1 lecture session of intro
  calculus-based class
• Coaches available for 4 segments kinematics (3),
  dynamics (4), COE (8), COM (7)
• Good time to establish baseline of rubric scores
  for general class
• Eventual comparison with computer coach users
• 2 expert raters PER researchers
• 23 students (3 tiers according to pretests)
  eventually will expand to 40
• 13 problems (8 from quizzes, 5 from final)

11
(No Transcript)
12
Sample Problem from Q2
13
Sample Solution Rating(Before Discussion)
14
Baseline rubric scores
- Standard error bars are on the order of /- 1
to /- 2 out of 5
15
Baseline rubric scoring patterns

• Raters 2 experts/PER researchers

16
Baseline rubric scoring patterns
17
Plan for larger-scale study

• 90 students, 1 lecture session of intro
  calculus-based class (45 for each group)
• Comparison group given equal face time with
  problems used in coaches
• All coaches available (kinematics, dynamics,
  energy, momentum, rotational motion)
• 8 x 5 40 total

18
Questions to Be Addressed

• Short-term questions
• Will students use coaches?
• How will students use them? (keystroke function)
• Do they improve students problem solving skills
  with respect to baseline scores? (rubric scoring
  of quizzes)

19
Questions to Be Addressed

• Longer-term questions
• Are they adaptable to be used in teaching other
  physics courses?
• Possible software/AI development refine beta
  versions make it more able to follow student
  preferences
• Can this software be modified by instructors to
  fit a different problem solving framework?

20
Thanks!

• Summary
• Initial results seem to have much to say need to
  be expounded upon
• Currently examining a baseline of exam
  performance to compare to future data from
  computer coach users
• Website http//groups.physics.umn.edu/physed
• Can look at research, previous talks and
  publications
• Try out the coaches! Give us feedback!
• http//groups.physics.umn.edu/physed/prototypes.ht
  ml

21
Basic Method

• Treatment and Comparison groups
• Treatment group computer coaching (on Web,
  outside of class), 4 problems per week
• Comparison group normal class setting
• Data collection
• Diagnostic pretests and posttests
• Written solutions on quizzes final exam
• 24513 for each student
• Problem-solving interviews with students

22
Individual student data

• Can look at individual time on each question for
  each student
• A few questions take some time regardless of
  student
• Entering answer into calculator
• Can look for patterns in other questions

23
References

• Chi, Feltovich and Glaser, Categorization and
  representation of physics problems by experts and
  novices, 1981.
• Collins, Brown and Newman, Cognitive
  apprenticeship Teaching the crafts of reading,
  writing, and mathematics, 1989.
• Docktor and K. Heller, Robust assessment
  instrument for student problem solving, 2009
  also see J. Docktor, dissertation, 2009.
• P. Heller and K. Heller, The Competent Problem
  Solver A Strategy for Solving Problems in
  Physics, 1995.
• P. Heller and Hollabaugh, Teaching Problem
  Solving Through Cooperative Grouping. Part 2
  Designing Problems and Structuring Groups, 1992.
• Hsu, Heller, Mason, and Xu, Summer AAPT
  presentation, Portland, OR, 2010.

24
References

• Larkin, McDermott, D. Simon and H. Simon, Expert
  and Novice Performance in Solving Physics
  Problems, 1980.
• Newell and Simon, Human Problem Solving, 1972.
• Palincsar and Brown, Reciprocal teaching of
  comprehension-fostering and comprehension-monitori
  ng activities, 1984.
• Polya, How to Solve It, 1945 1957.
• Polya, Mathematical Discovery, 1962.
• Reif and Scott, Teaching Scientific Thinking
  Skills Students and Computers Coaching Each
  Other, 1999 also see L. Scott dissertation,
  2001.