Thinking and Acting Like a Scientist: Investigating the Outcomes of Introductory Science and Math Courses

1
Thinking and Acting Like a Scientist
Investigating the Outcomes of Introductory
Science and Math Courses

• Kevin Eagan
• Jessica Sharkness
• Sylvia Hurtado
• Higher Education Research Institute, UCLA
• Association for Institutional Research 49th
  Annual Forum
• May 30 - June 3, 2009
• Atlanta, Georgia

2
Background

• Relatively few students earn degrees in natural
  science or engineering in the U.S.
• 15 of U.S. BA degrees are in science/engineering
• Compared to 67 in Singapore, 50 in China, 47
  in France, 38 in South Korea
• U.S. needs more undergraduate science majors to
  maintain achievement and innovation in science
  and engineering
• Also need to diversify the scientific workforce
  and increase representation of women and
  minorities

3
Background

• To graduate more bachelors degrees in science,
  U.S. needs students to choose science majors and
  to maintain interest in science majors
• National increases in proportion of freshmen
  indicating interest in science, technology,
  engineering and math (STEM) majors
• However, low proportion of students who intend to
  major in STEM actually graduate with STEM majors
• One obstacle to STEM major completion
  Introductory gatekeeper courses
• Mechanism for sorting students
• What is rewarded?

4
Introductory Gatekeeper Courses

• First course in a series of courses in which
  knowledge is cumulative
• In science relatively high drop-out and failure
  rates in gatekeeper courses
• Large lectures
• Un-engaging
• Highly competitive
• Grading on a curve

5
Classroom Environments Instructor Pedagogies

• Classroom climates have an impact on learning and
  performance
• Competitive environments have negative impact on
  learning, performance, retention, self-confidence
• Collaborative environments that emphasize group
  work can mitigate negative effects of large
  lectures and competitive environments
• Can also promote critical thinking about
  scientific concepts and their applications

6
Supportive Learning Environments and the Skills
Needed for Scientific Success

• Six necessary conditions for a supportive
  learning environment
• Quality of instruction, Teachers interest,
  Social relatedness, Support of competence,
  Support of autonomy
• Engender greater self-motivation, encourages
  self-directed learning
• Two primary pedagogical techniques in science
• Domain-specific learning memorization of facts
  and causal relationships
• Domain-general learning reasoning strategies
  and critical thinking skills

7
Additional influences on student success in STEM
courses

• Experiences external to the classroom environment
• Participation in research projects
• Peer Tutoring
• Prior academic achievement and preparation
• Most significant influence on outcome of
  introductory courses?

8
Goals of Current Study

• Untangle effect of prior preparation and
  background factors from performance assessment in
  introductory STEM courses
• Identify how students develop in introductory
  courses the critical thinking dispositions
  necessary for science careers, and whether these
  dispositions are reflected in student grades

9
Conceptual model
High School Science Achievement
Tutoring Research Participation
Course Grade
Amount of student effort expended on course
Demographic variables
Course Learning Environment Pedagogy
Ability to act and think like a scientist
(Post-test)
Ability to act and think like a scientist
(Pre-Test)
Critical thinking dispositions
10
Data Sample

• Data collected via online survey from students in
  12 introductory science and math courses at 5
  institutions
• Two surveys one at beginning of course
  (pre-survey) and one at end (post-survey) final
  analytic sample 255
• Final longitudinal sample
• 70 female
• 34 White, 43 Asian, 8 Black, 13 Latino
• 86 majoring in STEM field
• 64 first-year students, 27 second-year

11
Thinking and Acting like a Scientist
Loadings Loadings
Factor Factor Items Pre-test Post-test
Thinking like a scientist pre-test Thinking like a scientist pre-test Thinking like a scientist pre-test
See connections between different areas of science math See connections between different areas of science math 0.78 0.73
Understand scientific concepts Understand scientific concepts 0.77 0.80
Identify what is known and not known in a problem Identify what is known and not known in a problem 0.72 0.76
Ask relevant questions Ask relevant questions 0.72 0.72
Draw a picture to represent a problem or concept Draw a picture to represent a problem or concept 0.66 0.58
Make predictions based on existing knowledge Make predictions based on existing knowledge 0.75 0.82
Come up with solutions and explain them to others Come up with solutions and explain them to others 0.71 0.81
Investigate alternative solutions to a problem Investigate alternative solutions to a problem 0.71 0.73
Understand/translate scientific terminology into non-scientific language Understand/translate scientific terminology into non-scientific language 0.74 0.72
Acting like a scientist pre-test Acting like a scientist pre-test Acting like a scientist pre-test
Relate scientific concepts to real-world problems Relate scientific concepts to real-world problems 0.77 0.80
Synthesize or comprehend several sources of information Synthesize or comprehend several sources of information 0.85 0.78
Conduct an experiment Conduct an experiment 0.64 0.71
Look up scientific research articles and resources Look up scientific research articles and resources 0.60 0.62
  Memorize large quantities of information Memorize large quantities of information 0.61 0.53
Measurement model fit statistics ?2300.69 (305,
N255), NNFI 0.98, CFI 0.98, RMSEA 0.03,
reliability 0.82.
All items were asked as part of a questions stem
that read, Rate your ability in the following
areas as it pertains to your academic learning in
the sciences. Response options were Major
Strength (5), Above Average (4), Average (3),
Below Average (2), Major Weakness (1)
12
Variables

• Independent Variables
• Demographic characteristics
• Prior preparation
• Course pedagogy
• Classroom environment
• College experiences
• Critical thinking dispositions (CCTDI subscales)
• Dependent Variables
• Final course grade
• Dispositions toward science

13
Analysis Plan

• Identification of latent constructs (factors),
  representing acting like a scientist and thinking
  like a scientist
• Exploratory Factory Analysis ? Confirmatory
  Factor Analysis (measurement model in SEM)
• Structural Equation Modeling (SEM) to model how
  student experiences in introductory courses
  affect three outcomes

14
Structural Modeling Procedure

• Added hypothesized predictors and paths to
  structural equation model
• Used prior research, theory and Wald and LaGrange
  multiplier tests to add and remove paths that did
  not contribute to model fit
• When all paths were deleted from a variable,
  variable was removed from the model

15
Results (non-significant paths not shown)
External Support/Science Experiences
Course Learning Environment
Student Effort
Felt competition in course
HPW Engage in lab activities
Consistently got support needed
Course Pedagogy
Felt Overwhelmed
Crammed for exams
HPW profs research project (college)
Course employed group activities
Sought tutoring on campus
Format primarily lecture
Grade in intro course
Participated in HS research program
Think like a scientistPost-test
Demographics
Pretests
Income
Act like a scientistPost-test
Avg HS GPA in STEM courses
URM Student
BBS Major
Female
Act like a scientistPre-test
AP Chemistry Score
Tutored Student in HS
Think like a scientistPre-test
Open-Mindedness
Critical Thinking Confidence
Analyticity
Critical thinking dispositions
16
Discussion of findings Thinking and Acting Like
a Scientist

• CCTDI Openmindedness negatively predicted both
  dispositions
• Relativism in conflict with objective, empirical
  perspective of science
• CCTDI Critical thinking self-confidence
  positively predicted both dispositions
• Indicative of students development of
  domain-general science skills
• CCTDI Analyticity positively predicted thinking
  like a scientist
• Indicative of students ability to think more
  carefully and critically during problem-solving
  activities
• Students who felt overwhelmed scored lower on
  both dispositions
• May be indicative of naïveté about what is
  required of science majors

17
Discussion of findings Final Course Grade

• Thinking and acting like a scientist and CCTDI
  subscales unrelated to final grade
• Suggests that introductory courses focus too much
  on the acquisition of knowledge rather than
  development of higher-order thinking skills
• Final grade in large part predicted by prior
  preparation (high school grades, research)
• Suggests that failure to earn top grades may
  merely lack the preparation necessary for success
  in these courses rather than science-related
  skills

18
Discussion of findings Final Course Grade

• Cramming for exams positively predicted course
  grades
• Underscores that grades reward rote memorization
  rather than development of higher-order thinking
  skills
• Group work positively predicted final course
  grade
• Supports prior research that concluded that peer
  learning provides learning reinforcement, which
  may have longer-term benefits
• Tutoring (receiving and providing) positively
  predicted final course grade
• Reinforces the benefits of peer-to-peer teaching
  and learning

19
Implications and Conclusions

• Can we afford to cram content into courses at the
  expense of development of scientific skills and
  thinking?
• Adjust grading practices so that they more
  accurately reflect learning rather than prior
  preparation
• Further examination of pedagogical practices and
  interventions in large, lecture-based gatekeeper
  courses is needed

20
Contact Information
Faculty and Co-PIs Sylvia Hurtado Mitchell
Chang Graduate Research Assistants Kevin
Eagan Lorelle Espinsoa Christopher Newman
Administrative Staff Aaron Pearl
Jessica Sharkness Minh Tran Paolo Velasco
Papers and reports are available for
download from project website http//heri.ucla.e
du/nih Project e-mail herinih_at_ucla.edu

• Acknowledgments This study was made possible by
  the support of the National Institute of General
  Medical Sciences, NIH Grant Numbers 1 R01
  GMO71968-01 and R01 GMO71968-05 as well as the
  National Science Foundation, NSF Grant Number
  0757076. This independent research and the views
  expressed here do not indicate endorsement by the
  sponsors.