Preparing for Success in College Science: The Dance of Mathematics, Misconceptions, Teacher Knowledge, and the Advanced Placement Program

1
Preparing for Success in College Science The
Dance of Mathematics, Misconceptions, Teacher
Knowledge, and the Advanced Placement Program

• Philip M. Sadler, Director
• Science Education Department
• Harvard-Smithsonian Center for Astrophysics,
  Cambridge, MA

2
Why someone from theHarvard-SmithsonianCenter
for Astrophysics?
3
Harvard-Smithsonian Center for Astrophysics

• Largest astronomical research institution in the
  world
• A partnership between
• Harvards Department of Astronomy
• Harvard College Observatory
• Smithsonian Astrophysical Observatory
• More than 250 scientists in a staff of over 800
• Telescopes on earth and in space
• Search for earth-like planets

4
Why listen?

• Education
• MIT B.S. in Physics
• Harvard Ed.M., Ed.D.92
• Teaching
• middle school science and math
• Harvard University
• Astronomy
• Ed Research
• Ed Methods
• 200 teachers
• Developer of
• Starlab Planetarium
• Project STAR
• MicroObservatory

• Publications
• 5 texts
• 38 papers and book chapters
• 4 award-winning videos
• Editorial Board
• 2 ed journals
• Sponsored research
• 56m, 5m/yr
• 40 staff
• Honors
• ASP Brennan Prize
• Project ASTRO Education Award
• 3 AIP Computers in Physics
• 1999 JRST Award

5
How do you rigorously measure the conceptual
understanding of teachers and students in science?
6
How do you rigorously measure the conceptual
understanding of teachers and students in science?
7
Psychological Foundations

• The unlearning of preconceptions might very well
  prove to be the most determinative single factor
  in the acquisition and retention of
  subject-matter knowledge.
• David Ausubel 1978

8
Psychological Foundations

• The unlearning of preconceptions might very well
  prove to be the most determinative single factor
  in the acquisition and retention of
  subject-matter knowledge.
• David Ausubel 1978

9
Clinical Interviews
Minds of Our Own consists of 3-one hour programs
broadcast on PBS in 1997-98. It explores the
ideas of students as they come to understand
scientific concepts
On-on-one with students
A Private Universe documents students ideas
through their own drawings and explanations
www.learner.org
10
Students and teachers have preconceptions

• Exist prior to instruction
• At odds with accepted scientific thought,
  misconceptions
• Commonly held, not idiosyncratic
• embedded in larger knowledge structures, not just
  an error
• resistant to change

11
MOSART Misconception Oriented Standards-based
Assessment Resource for Teachers

• Our criteria for conceptual understanding
• Students and teachers must
• Prefer accepted scientific explanations over
  widely-held misconceptions
• Apply their knowledge to make accurate predictions

12
Our criteria for conceptual understanding

• Students and teachers must
• Prefer accepted scientific explanations over
  widely-held misconceptions
• Apply their knowledge to make accurate
  predictions
• For assessments to do this, test items must
• Include the scientifically correct answer
• Include the most popular misconceptions
• Be easy to score and use
• Value predictive over why questions

13
5-8 Physical Science Motions and Forces

• The motion of an object can be described by its
  position, direction of motion, and speed. That
  motion can be measured and represented on a graph.

14
The Problem
108. Kevin starts walking from a store a certain
distance from his home. Which sentence is a
correct description of Kevins motion as shown on
the graph?
15
The Correct Answer
108. Kevin starts walking from a store a certain
distance from his home. Which sentence is a
correct description of Kevins motion as shown on
the graph?
b. He walks toward home, stops for a while,
then walks away from home.
16
The Fraction Who Choose Correctly
108. Kevin starts walking from a store a certain
distance from his home. Which sentence is a
correct description of Kevins motion as shown on
the graph?
b. He walks toward home, stops for a while,
then walks away from home. 30
17
Other answers that students give?
108. Kevin starts walking from a store a certain
distance from his home. Which sentence is a
correct description of Kevins motion as shown on
the graph?
a. He walks toward home down a hill, then walks
along a level path, then walks up a hill.
b. He walks toward home, stops for a while,
then walks away from home. 30 c. He walks away
from home, stops for a while, then walks toward
home. d. He walks toward home down a hill, stops
for a while, then walks up a hill. e. He walks
down a hill and gets trapped in a valley.
18
Which answers do your students give?
108. Kevin starts walking from a store a certain
distance from his home. Which sentence is a
correct description of Kevins motion as shown on
the graph?
a. He walks toward home down a hill, then walks
along a level path, then walks up a hill.
28 b. He walks toward home, stops for a while,
then walks away from home. 30 c. He walks away
from home, stops for a while, then walks toward
home. 18 d. He walks toward home down a hill,
stops for a while, then walks up a hill.
20 e. He walks down a hill and gets trapped in a
valley. 5
19
Research Questions

• To what degree have students who completed
  science courses mastered the NRC standards?
• At grade level
• At prior grade levels
• Are there patterns of strength and weakness?

20
Research Questions

• To what degree have students who completed
  science courses mastered the NRC standards?
• At grade level
• At prior grade levels
• Are there patterns of strength and weakness?
• Have primary, middle, and high school science
  teachers mastered the standards that they teach?
• How well can teachers predict the knowledge state
  of their students (including misconceptions)?
• What is the impact of professional development
  activities on teacher content knowledge?

21
Mining the Research Literature
76. An electric cord runs from a wall outlet
along the floor to a lamp. The lamps light is
on. You carefully stack books, one at a time, on
top of each other on the wire until you have 100
pounds of books. Assuming the wire does not
break, what do you think would happen to the
brightness of the light?

1. The brightness of the light would decrease
   gradually as more books were added to the stack.
2. The light would dim all at once at some point,
   then remain dim.
3. The light would go out as soon as the first book
   was placed on the wire.
4. The light would flicker or briefly dim as each
   book was added, then return to normal.
5. The light would not change in brightness. 14

22
Mining the Research Literature
76. An electric cord runs from a wall outlet
along the floor to a lamp. The lamps light is
on. You carefully stack books, one at a time, on
top of each other on the wire until you have 100
pounds of books. Assuming the wire does not
break, what do you think would happen to the
brightness of the light?

1. The brightness of the light would decrease
   gradually as more books were added to the stack.
2. The light would dim all at once at some point,
   then remain dim.
3. The light would go out as soon as the first book
   was placed on the wire.
4. The light would flicker or briefly dim as each
   book was added, then return to normal.
5. The light would not change in brightness. 14

23
Student Preference
76. An electric cord runs from a wall outlet
along the floor to a lamp. The lamps light is
on. You carefully stack books, one at a time, on
top of each other on the wire until you have 100
pounds of books. Assuming the wire does not
break, what do you think would happen to the
brightness of the light?

1. The brightness of the light would decrease
   gradually as more books were added to the stack.
   39
2. The light would dim all at once at some point,
   then remain dim. 10
3. The light would go out as soon as the first book
   was placed on the wire. 6
4. The light would flicker or briefly dim as each
   book was added, then return to normal. 11
5. The light would not change in brightness. 14

24
Test Item Development

• Breakdown of the NRCs
• What concepts are the standards really asking
  kids to know?
• What are the relevant misconceptions reported in
  the literature?
• Item Construction
• Items (M/C for ease) that represents the
  standard and captures kids knowledge based on
  research protocols.
• Validation
• Are the questions accurate in terms of the
  science? Readable?
• Pilot Testing (N100/item)
• selection of core items that represent the most
  variance
• Large scale sample (Physical Science Example,
  N1000/item)
• Item characteristics for 100-200 items/domain
• Characterization of domains 7,000 students,
  50 teachers
• Finalization of Instruments
• Made available for evaluation of programs like
  yours

25
National Data
26
Grade 7/8 Physical Science StudentsAfter Taking
a Year of Physical Science
27
Adding HS Chemistry and Physics Students
28
Adding HS Science Teachers
29
Adding MS Physical Science Teachers
30
Teacher Content and Predictive Knowledge
31
Patterns in Classroom Data
32
Comparison of Item Formats
33
Teacher Content and Predictive Knowledge Across
38 Classrooms
34
Teacher Content Knowledge and Teacher Prediction
Accuracy across 38 Classrooms
35
Patterns

• For each standard at each level
• Students have not achieved mastery
• Teachers generally overestimate student
  knowledge.
• Teachers know far more than their students
• Teacher knowledge is a not a guarantee of student
  knowledge
• Subject do much better on items if misconceptions
  are not a choice
• Teachers knowledge of student ideas is
  associated with higher performance than content
  knowledge

36
Patterns in Professional Development Data
37
Which factors predict teacher content knowledge
of the curriculum concepts?

1. Grade level
2. Gender
3. Years Teaching
4. Years Teaching science subject
5. Certification in the science subject
6. Degrees (BS, BA, MS, PhD)
7. Grad Courses taken in domain
8. Professional development in science
   teaching/content

38
Predicting Teacher Masterylinear models with
significant factors

• Model B 36 of variance
• Source df ?sq F-ratio Prob
• Const 1 57.56 6836.20 0.0001
• Grade band 2 0.14 8.70 0.0004
• Gender 1 0.06 8.29 0.0052
• Years Teaching subject 1 0.08 10.58 0.0017
• Certification 1 0.03 4.49 0.0374
• Yrs Teaching subject Cert 1 0.06 7.90 0.0063
• Error 73 0.61
• Total 79 0.95

39
Which factors predict teacher content knowledge
of the curriculum concepts?

1. Grade level
2. Gender
3. Years Teaching
4. Years Teaching science subject
5. Certification in the science subject
6. Degrees (BS, BA, MS, PhD)
7. Grad Courses taken in domain
8. Professional development in science
   teaching/content

40
Interaction of Years Teaching Subject and
Certification
41
2-Week Astronomy Institute

• Basics
• To boost astronomy background
• General astronomy test
• Speakers
• Activities
• Observing

42
2-Week Astronomy Institute

• Basics
• To boost astronomy background
• General astronomy test
• Speakers
• Activities
• Observing

43
2-Week Astronomy Institute

• Moderate initial knowledge
• Gains at all levels of teacher knowledge
• Few teachers with no or negative growth

44
1-Week Astronomy Institute

• Instrumentation
• Earth-Sun connection only
• Only relevant items
• Speakers
• Activities
• Observing

45
1-Week Astronomy Institute

• Learn to use professional instrumentation
• Disciplinary domain focus
• Speakers

46
1-Week Astronomy Institute

• High initial knowledge
• No gains at highest level of teacher knowledge
• Many teachers with no or negative growth

47
Comparison of 2 MSP Institutes
48
Patterns

• Some teacher content weakness at all grade
  levels weakest at MS levels
• Content knowledge grows very slowly for the
  non-certified teacher
• Professional development can make a difference in
  teacher content knowledge
• Length of program
• Focus on content knowledge at grade level vs.
  science apprenticeships
• Must evaluate the fulfillment of goals
• Content knowledge at higher levels does not
  translate to knowledge at lower levels

49
Seeking Research Partners

• Professional Development
• Increase in teacher content knowledge
• Increase in teacher pedagogical content knowledge
• Customized assessment instruments
• Linking to Student Pre-Post Assessment
• Curricular and Pedagogical Innovation
• Impact of professional development
• Teacher Subject Matter Knowledge
• Accuracy of Teacher Prediction
• Breakout session tomorrow

50
Overview of Research

• 3M, 4-year IERI study to investigate the kinds
  of high school courses that best prepare college
  students for
• introductory courses in biology, chemistry, or
  physics

51
Overview of Research

• 3M, 4-year IERI study to investigate the kinds
  of high school courses that best prepare college
  students for
• introductory courses in biology, chemistry, or
  physics
• Drawing hypotheses from
• research literature
• high school teachers
• Professors

52
Overview of Research

• 3M, 4-year IERI study to investigate the kinds
  of high school courses that best prepare college
  students for
• introductory courses in biology, chemistry, or
  physics
• Drawing hypotheses from
• research literature
• high school teachers
• Professors
• 67 items survey, sample of
• 18,000 college students
• 1st and 2nd semester
• 63 randomly-chosen colleges

53
FICSS Study Goals

• Identify the HS pedagogy and curriculum that
  prepare students for college science
• From HS science teachers
• From college professors
• From educational researchers

54
FICSS Study Goals

• Identify the HS pedagogy and curriculum that
  prepare students for college science
• From HS science teachers
• From college professors
• From educational researchers
• Collect evidence that supports or refutes these
  beliefs concerning
• Physics First
• Block Scheduling
• Advanced Placement

• Labs and Demos
• Mathematics
• Project Work

55
Comparison of Teacher and Professor Views of
Factors Predicting Success in College Science
56
Comparison of Teacher and Professor Views of
Factors Predicting Success in College Science
57
Comparison of Teacher and Professor Views of
Factors Predicting Success in College Science
58
Comparison of Teacher and Professor Views of
Factors Predicting Success in College Science
59
Predictor Categories

• Background
• Parents Ed
• SES
• Type of physics
• School
• Class attributes
• Course choice
• Grades
• SATs

60
Predictor Categories

• Background
• Parents Ed
• SES
• Type of physics
• School
• Class attributes
• Course choice
• Grades
• SATs

• Pedagogy
• Instructional approach
• Demos
• Labs
• Autonomy
• Technology
• Homework/text
• Teacher
• Tests/assignments
• Discipline

61
Predictor Categories

• Background
• Parents Ed
• SES
• Type of physics
• School
• Class attributes
• Course choice
• Grades
• SATs

• Pedagogy
• Instructional approach
• Demos
• Labs
• Autonomy
• Technology
• Homework/text
• Teacher
• Tests/assignments
• Discipline

• Content
• Facts
• Concepts
• Skills
• Mechanics
• Electricity
• Stoichiometry
• Periodic table
• Genetics
• Evolution
• Dissection

62
Views on Factors
Block Scheduling
Teacher Quality
Mathematics
Graphing by Hand
63
our most recent findings.
64
(No Transcript)
65
(No Transcript)
66
(No Transcript)
67
Mathematics Preparation
68
Student Comments math

• Fewer topics, more in-depth. Make honors physics
  calculus based. I was in honors physics in HS and
  it was hardly math-based at all, much less
  calculus-based.
• The high school course I took gave me a good
  conceptual basis, but the mathematics was not
  stressed as much as in college.
• More focus on the mathematical side of physics
• My high school teacher taught us step by step
  methods to obtaining the answers mathematically,
  this was very beneficial when doing word problems
  in college.
• High school students should be learning to think
  about physical situations mathematically, and
  gaining familiarity with the kinds of problems we
  would do in college.
• More of the mathematical transformations needed
  to properly do physics at the college level is
  required.

69
High School Science Laboratory Experiences

• Some examples of predictors used in this
  analysis
• Full Understanding (5) versus Memorization (1)
• Labs Frequently Addressed Students Beliefs
• Labs for Improving Conceptual Understanding
• Time Discussing Labs
• Analyzing Pictures or Illustrations
• Draw/Interpret Graphs by Hand
• Student-Designed Projects
• Read Discuss Labs a Day Before
• Labs Frequently Built Upon Previous Experience
• Understanding of Lab Procedure
• Freedom in Designing Conducting Labs
• Use of Computer Simulations
• 30 Variables were studied in this analysis

70
What Appears to

• Help
• Often Analyzed Pictures or Illustrations
• Often Draw/Interpret Graphs by Hand
• Quantitative problems
• Labs Addressed Students Beliefs
• More Freedom in Designing Conducting Labs (high
  math achievers)
• Testing for facts
• Mastery of select foundational concepts
• Physics
• more mechanics
• more history of physics
• less relativity
• More prediction, less demo discussion
• Chemistry
• More stoichiometry
• Less nuclear chemistry

71
What Appears to

• Help
• Often Analyzed Pictures or Illustrations
• Often Draw/Interpret Graphs by Hand
• Quantitative problems
• Labs Addressed Students Beliefs
• More Freedom in Designing Conducting Labs (high
  math achievers)
• Testing for facts
• Mastery of select foundational concepts
• Physics
• more mechanics
• more history of physics
• less relativity
• More prediction, less demo discussion
• Chemistry
• More stoichiometry
• Less nuclear chemistry

• Hinder
• Read Discuss Labs a Day Before
• Greater Understanding of Lab Procedure
• Student-Designed Projects
• More Freedom in Designing Conducting Labs (low
  math achievers)
• Coverage of entire domain
• Standardized exam prep
• Testing on labs
• Using class time to teach facts and vocabulary
• Reading the textbook

72
Lab Experience
73
Student Comments labs

• In a basic high school physics course I would
  advise a lot of hands on activities and labs to
  help students understand the basic concepts of
  kinematics which tend to hinder a lot of students
  at the college level.
• I would change the labs. Although the labs
  completed were excellent in high school, detailed
  lab reports were not required and did not prepare
  me for college physics lab reports.
• Also, less physics labs in high school would be
  better and more focus on the math and free body
  diagram aspect of physics.
• I would suggest that the labs should be more
  challenging and less emphasis should be placed on
  memorization and more emphasis on comprehension.

74
Physics First

• Leon Ledermans Project ARISE
• A physics-chemistry-biology sequence leads the
  student from the simple to the complex, an
  approach which is in harmony with current
  understanding of how the brain learns.
• Understanding modern biology, for example the
  function of DNA, requires a background in
  chemistry, physics, and mathematics.
• Moreover, chemistry is based upon the charge
  structure of atoms and the forces between these
  charges, concepts learned in physics.

75
Testing Physics First Hypotheses

1. Taking HS physics will have a positive impact on
   chemistry performance
2. Taking HS chemistry will have a positive effect
   on college biology
3. Students who take HS physics before HS chemistry
   (2) will perform better in college chemistry
   (4)
4. Students who now take HS chemistry before HS
   biology (6) will perform better in college
   biology

76
College Performance in Biology, Chemistry and
Physics Based on HS Coursework
High School Biology
High School Chemistry
High School Physics
90
85
College
Biology
College Grade
College
Chemistry
80
College
Physics
75
none
Reg
AP
Reg
none
Reg
AP
Reg
none
Reg
AP
Reg
only
only
AP
only
only
AP
only
only
AP
77
College Performance in Biologybased on high
school coursework
High School Biology
High School Chemistry
High School Physics
90
85
College Grade
College
Biology
80
75
none
Reg
AP
Reg
none
Reg
AP
Reg
none
Reg
AP
Reg
only
only
AP
only
only
AP
only
only
AP
78
College Performance in Biology and
ChemistryBased on Amount of HS Coursework
High School Biology
High School Chemistry
High School Physics
90
85
College
Biology
College Grade
80
College
Chemistry
75
none
Reg
AP
Reg
none
Reg
AP
Reg
none
Reg
AP
Reg
only
only
AP
only
only
AP
only
only
AP
79
The Advanced Placement Program

• AP began as a way for exceptional students at
  elite private schools
• To take rigorous courses in HS
• No planned impact on college admissions (1952)
• No planned impact on GPA

80
The Advanced Placement Program

• AP began as a way for exceptional students at
  elite private schools
• To take rigorous courses in HS
• No planned impact on college admissions (1952)
• No planned impact on GPA
• Expanded to gt2.1M exams/yr in 35 subjects

81
The Advanced Placement Program

• AP began as a way for exceptional students at
  elite private schools
• To take rigorous courses in HS
• No planned impact on college admissions (1952)
• No planned impact on GPA
• Expanded to gt2.1M exams/yr in 35 subjects
• Benefits to the student (other than learning)
• Higher HS Grade Point Average
• Taking college courses in High School
• High probability of getting into college and
  financial aid
• Higher college grades if repeated
• College credit (advanced standing), cost savings

82
What the public hears

• It is better to take a tougher course and get a
  low grade than to take an easy course and get a
  high grade.
• Clifford Adelman, Senior Research Analyst, U.S.
  Dept. of Ed.

83
Our Research Questions

• For students taking introductory college biology,
  chemistry, and physics
• What grades do students earn based on high school
  AP performance?
• What is the predicted advantage taking AP when
  controlling for student background, preparation,
  and SES?

84
College Performance in Introductory Science
Courses
85
Covariates with AP Score The need for regression
models to model the unique contribution of AP
courses.
86
Modeling the Impact of AP CoursesAfter
controlling for covariates
87
Modeling the Impact of AP CoursesAfter
controlling for covariates
88
Modeling the Impact of AP CoursesStudents who do
not take the exam perform at the same level as
those earning a 3
89
Earning a 5 predicts increasing college grade
by 5 points over honors
90
Earning a 4 predicts increasing college grade
by 4 points over honors
91
Modeling the Impact of AP Courses
92
Conclusions
93
Conclusions

• AP students do earn somewhat higher grades in
  college science
• Partial proxy for demographic, general scholastic
  performance, math preparation
• and performance in high school science courses
  that are prerequisites to AP in most schools

94
Conclusions

• AP students do earn somewhat higher grades in
  college science
• Partial proxy for demographic, general scholastic
  performance, math preparation
• and performance in high school science courses
  that are prerequisites to AP in most schools
• Course order is unimportant

95
Conclusions

• AP students do earn somewhat higher grades in
  college science
• Partial proxy for demographic, general scholastic
  performance, math preparation
• and performance in high school science courses
  that are prerequisites to AP in most schools
• Course order is unimportant, amount is
• The best preparation comes from HS courses that
• Use lots of math
• Concentrate on key concepts, not coverage
• Use labs judiciously to change misconceptions

96
What do we know now?

• Misconceptions often unchanged after taking
  science.
• Necessary step in learning
• The standards are hard to master.
• Teachers are knowledgeable, but does not assure
  student learning.
• Teachers do not know their students
  misconceptions, but should.
• Teacher knowledge builds slowly.
• Professional development must be
• targeted to specific standards at grade levels
• evaluated with relevant tools.
• AP courses help the most if they focus on
  quantitative science, conceptual labs,
  fundamentals.

97
Acknowledgments

• Co-investigators
• Robert Tai, University of Virginia, Matthew
  Schneps,
• Project Managers
• Hal Coyle, Michael Filisky
• Survey Staff
• Jamie Miller, Nancy Cook Smith, Cynthia Crockett,
  Marc Schwartz (McGill), Annette Trenga, Bruce
  Ward
• Video Staff
• Yael Bowman, Toby McElheny, Nancy Finkelstein,
  Alexia Prichard, Alex Griswold
• Graduate Students
• Zahra Hazari, John Loehr

• Advice
• NSF Janice Earle, Barry Sloane, Elizabeth
  VanderPutten, Larry Suter
• Board of Advisors
• Joel Mintzes, Mary Atwater
• Brian Alters, Lillian McDermott
• Eric Mazur, James Wandersee
• Dudley Herschbach
• Financial Support
• NSF DoEd
• Annenberg/CPB NIH
• Center for Astrophysics
• Irwin Shapiro, Judith Peritz.

98
Any opinions, findings and conclusions or
recommendations expressed in this material are
those of the authors and do not necessarily
reflect the views of the National Science
Foundation, National Institutes of Health, U.S.
Department of Education
99
Harvard-Smithsonian Center for
AstrophysicsScience Education Department

• 60 Garden Street, MS-71
• Cambridge, MA 02138
• Phone [phone]
• Fax [phone]
• Email psadler_at_cfa.harvard.edu

100
Leo Tolstoy
"I know that most men, including those at ease
with problems of the greatest complexity, can
seldom accept even the simplest and most obvious
truth if it be such as would oblige them to admit
the falsity of conclusions which they have
delighted in explaining to colleagues, which they
have proudly taught to others, and which they
have woven, thread by thread, into the fabric of
their lives."