Research Strategies for Assessment of Learning in Large Classes

1
Research Strategies for Assessment of Learning in
Large Classes
Diane Ebert-May Department of Plant Biology
Michigan State University ebertmay_at_msu.edu http
//first2.org http//www.msu.edu/course/isb/202/ebe
rtmay/2004/home.html
2
(No Transcript)
3
(No Transcript)
4
(No Transcript)
5
Anonymous (Change Magazine, 2001)

• I believe we would all agree that the absolute
  best teaching learning-assessment model is the
  one-on-one Socratic apprenticeship model with
  unlimited time with the student. But ever since
  Socrates took on two students rather than only
  one (to double his income), teachers have had to
  make compromises in teaching.

6
Question 1
Please respond on a scale if 0-100 in increments
of 10

• How important is it to use multiple kinds of
  data to assess student learning?

7
How important is it to use multiple forms of data
to assess student learning?

Relative Importance
n127
8
Question 2
Please respond on a scale of 0 - 100 in
increments of 10

• How often do you use data to make instructional
  decisions?

9
How often do you use data to make instructional
decisions?

Frequency
n127
10
True or False?

• Assessing student learning in science parallels
  what scientists do as researchers.

11
Parallel ask questions

• Description
• -What is happening?
• Cause
• -Does x (teaching strategy) affect y
  (understanding)?
• Process or mechanism
• -Why or how does x cause y?

12
Parallel collect data

• We collect data to find out what our students
  know.
• Data helps us understand student thinking about
  concepts and content.
• We use data to guide decisions about
  course/curriculum/innovative instruction

13
Parallel analyze data

• Quantitative data - statistical analysis
• Qualitative data
• break into manageable units and define coding
  categories
• search for patterns, quantify
• interpret and synthesize
• Valid and repeatable measures

14
Parallel peer review

• Ideas and results are peer reviewed - formally
  and/or informally.

15
Q3. What is assessment?

• Data collection with the purpose of answering
  questions about
• students understanding
• students attitudes
• students skills
• instructional design and implementation
• curricular reform (at multiple grainsizes)

16
Research Methods
17
Why do assessment?
Improve student learning and development.
Provides students and faculty substantive
feedback about student understanding.
Challenge to use disciplinary research strategies
to assess learning. (Ebert-May et al 2003)
18
Data collection approaches
19
(No Transcript)
20
(No Transcript)
21
Tools for Data Collection
Items Students Purchase Course Pack - at Ned's
and the College Store on Hagadorn ISB202 -
Section 7 Drs. Ebert-May and Linton Carbonless
Lab Notebook - 8½" x 11" - at the College Store
and the Student Bookstore can be graph paper or
lined Student Response Pad - 6 - only at Ned's
Register your transmitter online _at_
www.einstruction.com Cost 15 - you'll need a
credit card You need this number to register
Class Key S2349Q361 Stapler (mini or regular
sized) - Bring to class each day  
22
Research Question Do students perceptions
of the value of cooperative learning change
throughout this course? Design Experiment
Course over time (15 weeks) Pre-post
in-class inquiries/exam questions
Data type Indirect self-report from
clickers Direct individual /group assessment
23
How often did you work in cooperative groups in
high school? 01.26.04
24
How often have you worked in cooperative groups
in college? 01.26.04
25
How much do you think cooperative learning will
improve your understanding?01.26.04
26
How much do you think cooperative learning has
improved your understanding thus far? 03.29.04
27
How much do you think cooperative learning
improved your understanding about the frog
deformity problem? 03.31.04
28
(No Transcript)
29

• Tool Concept Maps
• Reliability
• test-retest reliability not a concern
• inter-rater reliability (do people scoring maps
  agree)
• Validity
• face validity - represent meaningful learning
  progressive differentiation, integration
• construct validity - do maps correlate with other
  measures of learning?

30

• C-TOOLS
• http//ctools.msu.edu/

31
Tool Blooms Rankings
Use to classify assessment questions
(1) Knowledge (2) Comprehension (3) Application
(4) Analysis (5) Evaluation (6) Synthesis
32
Concept Carbon Cycle

• Goals Students will be able to
• Trace the pathway of a carbon atom through
• living and nonliving components of an
• ecosystem, explaining both the processes and
• types of organisms involved.

33
Pre-test

• Which of the following substances has mass?
  Answer yes or no in the blank to the left.
• H20 vapor _______
• CO2 _______
• Glucose (C6H12O6) _______
• O2 _______

34
Pre-test

• What are the different pathways that carbon can
  take once it is inside a plant?
• Select as many as apply
• a. it can exit the plant as CO2
• b. it can become part of the plants cell walls,
  protein, fat, DNA
• c. it can be consumed by an insect feeding on
  the plant and become part of the insects body
• d. as a plant decomposes in the soil, carbon can
  exit the soil as CO2

35
Pre-test

• The majority of the actual weight (dry biomass)
  gained by plants as they progress from seed to
  adult plant comes from which of the following
  substances?
• a. molecules in the air that enter through the
  leaves
• b. particle substances in the soil taken up by
  the roots
• c. substances dissolved in water that are taken
  up by roots
• d. energy from the sun

36
Radish Experiment

• Experimental Setup
• Weighed out 3 batches of radish seeds each
  weighing 1.5 g.
• Experimental treatments
• 1. Seeds placed on moistened paper towels in
  LIGHT
• 2. Seeds placed on moistened paper towels in
  DARK
• 3. Seeds not moistened (left DRY) placed in light
• After 2 weeks, all plant material was dried in
  an oven overnight (no water left) and plant
  biomass was measured in grams. Predict the
  biomass of each treatment.

37
Results
1.46 g 1.63 g 1.20 g
Write an explanation of the results.
38
Grandma Johnson Problem

• Hypothetical Scenario Grandma Johnson had very
  sentimental feelings toward Johnson Canyon,Utah
  where she and her late husband had honeymooned
  long ago. Her feelings toward this spot were such
  that upon her death she requested to be buried
  under a creosote bush overlooking the canyon. She
  loved the idea that she'd become part of the
  wonderful wilderness and live on through the
  wildlife that lived there. Think to yourself and
  begin to trace the path of a carbon atom from
  Grandma Johnson's (GJ) remains to where it could
  become part of a coyote (NOTE the coyote WILL
  NOT dig up Grandma and consume any of her
  remains). What fundamental pathways and processes
  of biology will be involved in the transit of
  GJ's carbon atoms to that of the wild coyote in
  Utah?

39
In Groups of 3 or 4

• Select a question you want to answer in related
  to the CCLI innovations in your project .
• ...research design?
• ...data collection tools?
• ...analysis?
• ...journal?

40
Question
Does an active, inquiry-based instructional
design affect students understanding of
evolution and natural selection?
Method of analysis Pre-post test
41
From literature and our research Alternative
Conceptions about Natural Selection
Changes in a population occur through a gradual
change in individual members of a population.
New traits in species are developed in response
to need. All members of a population are
genetically equivalent, variation and fitness are
not considered. Traits acquired during an
individuals lifetime will be inherited by
offspring.
42
Instructional Design
Cooperative groups in class
Guppy Problem sexual vs. natural
selection http//www.first2.org/resources/inquiry_
activities/guppy_activity.htm
-PBS film -Simulation -Analyze data -Written
explanation
43
(No Transcript)
44
(No Transcript)
45
(No Transcript)
46
Explain the changes that occurred in the tree and
animal. Use your current understanding of
evolution by natural selection.
(AAAS 1999)
47
Misconception individuals evolve new traits
n80 plt.01
of Students
48
Misconception evolution is driven by need
n80 plt.01
of Students
49
In guppy populations, what are the primary
changes that occur gradually over time?

• a. The traits of each individual guppy within a
  population gradually change. b. The proportions
  of guppies having different traits within a
  population change. c. Successful behaviors
  learned by certain guppies are passed on to
  offspring. d. Mutations occur to meet the needs
  of the guppies as the environment changes.

Anderson et al 2002
50
Posttest Student responses to mc
n171

of Students
51
Animal/Tree Posttest Gain in student
understanding of fitness
n80 plt.01
of Students
52

• Quantitative Data
• Qualitative Data

Design Experiment
Ebert-May et al. 2003 Bioscience
53
Question
How do assessment questions help us determine
students prior understanding and progressive
thinking about the carbon cycle.
54
Instructional Design

• Two class meetings on carbon cycle (160 minutes)
• Active, inquiry-based learning
• Cooperative groups
• Questions, group processing, large lecture
  sections, small discussion sections, multi-week
  laboratory investigation
• Homework problems including web-based modules
• Different faculty for each course
• One graduate/8-10 undergraduate TAs per course

55
Experimental Design

• Two introductory courses for majors
• Bio 1 - organismal/population biology (faculty A)
• Bio 2 - cell and molecular biology (faculty B)
• Three cohorts
• Cohort 1 Bio 1 (n141)
• Cohort 2 Bio1/Bio2 (n63)
• Cohort 3 Other/Bio2 (n40)

56
Assessment Design

• Multiple iterations/versions of the carbon cycle
  problem
• Pretest, midterm, final with additional formative
  assessments during class
• Administered during instruction
• Semester 1 - pretest, midterm, final exam
• Semester 2 - final exam

57
Grandma Johnson Problem

• Hypothetical scenario Grandma Johnson had very
  sentimental feelings toward Johnson Canyon, Utah,
  where she and her late husband had honeymooned
  long ago. Her feelings toward this spot were
  such that upon her death she requested to be
  buried under a creosote bush overlooking the
  canyon. Trace the path of a carbon atom from
  Grandma Johnsons remains to where it could
  become part of a coyote. NOTE the coyote will
  not dig up Grandma Johnson and consume any of her
  remains.

58
Analysis of Responses

• Used same scoring rubric (coding scheme) for all
  three problems - calibrated by adding additional
  criteria when necessary, rescoring
• Examined two major concepts
• Concept 1 Decomposers respire CO2
• Concept 2 Plants uptake of CO2
• Explanations categorized into two groups
• Organisms (trophic levels)
• Processes (metabolic)

59
Coding Scheme
60
Cellular Respiration by Decomposers
Correct Student Responses ()
Bio1/Bio2
Other/Bio2
Friedmans, plt0.01
61
Pathway of Carbon in Photosynthesis
Correct Student Responses ()
Bio1/Bio2
Other/Bio2
Friedmans, plt0.05
62
IRD Team at MSU

• Janet Batzli - Plant Biology U of Wisconsin
• Doug Luckie - Physiology
• Scott Harrison - Microbiology (grad student)
• Tammy Long - Plant Biology
• Jim Smith - Zoology
• Deb Linton - Plant Biology (postdoc)
• Heejun Lim - Chemistry Education
• Duncan Sibley - Geology
• National Science Foundation

63
What evidence will we accept?

• What is the question?
• What research and instructional designs?
• What data collection methods?
• How to analyze and interpret data?
• Are findings valid and generalizable?
• What are the next questions?
• WHO?