Though this be madness, there is a method in it: Using methodological exemplars to improve pedagogical research

1
Though this be madness, there is a method in
itUsing methodological exemplars to improve
pedagogical research

• Andrew N. Christopher
• Albion College

Jordan D. Troisi Widener University
Presented at the 12th Annual Society for the
Teaching of Psychology Best Practices
Conference October 11, 2013
2
Though this be madness, there is a method in it

• A brief summary of the beginning of Hamlet
• Hamlet is visited by his fathers ghost (The
  King)
• The ghost says that his own brother (Claudius)
  has killed him
• 1-2 months later, Claudius has married Gertrude,
  Hamlets mother
• The ghost says kill Claudius
• Hamlet says sure, and that hes going to act
  mad for awhile, presumably to cover his tracks
• Perhaps there is a method in it?

3
Though this be madness, there is a method in it

• Hamlet is acting(?) mad, and a bunch of
  characters develop hypotheses as to why
• Gertrude hypothesizes hes mad because his father
  has died and she married his brother in 1-2
  months
• Claudius hypothesizes hes acting mad because he
  wants to steal the throne he has recently assumed
• Polonius hypothesizes hes mad in love with his
  daughter Ophelia and that he just wants to sleep
  with her
• All valid hypotheses, which they each test
  (poorly)
• But they ignore a crucial confound. . .
• A freaking ghost is telling Hamlet what to do!

? FAIR
? FAIR
? FAIR
4
Guidance in pedagogical research from Hamlet

• Theres a lot of madness out there, and its
  often hard to understand
• Luckily, psychologists are trained to determine
  the reasons why behavior occurs
• Also luckily, such behavior is rarely guided by
  ghosts
• What happens in the classroom may seem like
  madness, but theres a method in it
• We need the methodological tools to make sense of
  the madness

5
Overview

• Scholarly teaching, the scholarship of teaching
  and learning (SoTL), and the learning sciences
• Issues of control in pedagogical research
• Effective methodology exemplars from SoTL and the
  learning sciences
• Take-home messages for effective pedagogical
  research

6
SoTL Learning SciencesDaniel Chew (in press)
SoTL
Learning Sciences

• Extension of higher education
• Evaluates specific pedagogical methods
• Application-driven
• Constructs may lack theoretical precision

• Extension of basic research in psychology
• Findings may not translate across teaching
  contexts
• Theory-driven
• Construct operationalizations tend to be
  widely-accepted

7
SoTL Learning SciencesDaniel Chew (in press)
SoTL
Learning Sciences

• Predicting the weather

• Knowing the laws of thermodynamics

8
Research on Note-Taking

• Williams and Eggert (2002) examined the notes of
  125 students in an undergraduate human
  development course
• Students took notes in a 125-page study guide on
  both their
• Class notes
• Textbook readings

9
Research on Note-Taking

• Three blind raters coded these notes for
• Completeness
• Length
• Accuracy

10
Research on Note-Taking

• Completeness, length, and accuracy used to
  predict
• Brief essay quizzes linked to the notes
• Exam questions from text only
• Exam questions from lecture only
• Exam questions from both text and lecture

11
Research on Note-Taking

• Peverly et al. (2013) randomly assigned 200
  undergraduates in an educational psychology
  course to either
• Outline provided
• No outline provided
• while taking notes on a 23 min video-taped
  lecture

12
Research on Note-Taking

• Predictors
• Handwriting speed
• Wrote letters of alphabet horizontally in orders
  as many times as possible in 45 sec
• Language comprehension
• 7 reading passages with questions about each
• Verbal working memory
• Listening span test
• Attention

13
Research on Note-Taking

• Dependent Variables
• Quality of notes
• Written summary of the lecture

14
Research on Note-Taking

• Used structural equation modeling to assess
  predictors relationships with quality of notes
  and written summary of the lecture

15
Overview

• Scholarly teaching, the scholarship of teaching
  and learning (SoTL), and the learning sciences
• Issues of control in pedagogical research
• Effective methodology exemplars from SoTL and the
  learning sciences
• Take-home messages for effective pedagogical
  research

16
SoTL The Early Days

• Focused on student attitudes (i.e., Did students
  like it?)
• e.g., Christopher and colleagues (2004)
• Wesp and Meile (2008, p. 362)
• It appears that student opinions about the
  effectiveness of teaching techniques are
  inaccurate.
• Researchers should prefer direct measures (of
  student learning) because they provide a more
  accurate assessment of pedagogical
  effectiveness.

17
Learning Might not Equal Liking

• Attitudes about courses and topics may not equate
  to learning, and vice versa
• After a course on research and statistics,
  students reported increases in knowledge but no
  changes in favorable attitudes toward the subject
  (Sizemore Lewandowski, 2009)

18
SoTL The Early Days A Vicious Cycle

• Psychologists are control freaks
• Correlation does not equal causation
• equates to
• Correlational research is not as valuable as
  experimental research

19
SoTL Why is it so hard to do?

• Special challenges in doing SoTL
• How often you teach a particular class
• Number of students in the class
• Time of day a class (or sections of a class) is
  offered
• Lack of control relative to lab-based research

20
Overcoming the Control Issue in SoTL

• As psychologists, we love control
• As teachers, we want to do whatever we can to
  maximize student learning and sometimes produce
  attitude change
• But, classrooms are very noisy places

21
Finding Appropriate Comparisons

• Smith (2008)
• Use a previous class
• Compare and (ideally) statistically control for
  ACT/SAT scores
• Use an in-tact class
• Divide the class into 2 equal groups
• Group 1 attends the first half of class Group 2
  attends the second half of class
• Use your intervention on one of the 2 groups
• Measure outcome variable on the same day

22
More Methods to Use Within 1 Class

• Bartsch et al. (2008)
• Typical pre/post contains same questions
• Using different tests at pre and post
  instrumentation confound
• Focus on using a 1-group pre/post with alternate
  forms

23
1-Group Pre/Post with Alternate Forms

• Prepare 2 versions of an assessment instrument
  (i.e., Version A Version B)
• Give half the students Version A at pretest and
  Version B at posttest
• Switch for the other half of the students

24
1-Group Pre/Post with Alternate Forms

• 2 (Version A first or Version B first) x 2 (pre
  and post) mixed ANOVA
• Allows us to learn if
• Groups were equivalent to start with
• Version A was equivalent to Version B
• The intervention does or does not facilitate
  student learning

25
1-Group Pre/Post with Alternate Forms

• Use random assignment to
• Form your 2 groups of students
• Put questions on the 2 versions of the assessment
  instrument

26
Overview

• Scholarly teaching, the scholarship of teaching
  and learning (SoTL), and the learning sciences
• Issues of control in pedagogical research
• Effective methodology exemplars from SoTL and the
  learning sciences
• Take-home messages for effective pedagogical
  research

27
Rapport, Student Motivation, and Course Attitudes

• Wed all like good rapport, motivated students,
  and favorable attitudes toward our courses
• Legg and Wilson (2009) examined how sending a
  welcome email to students before a course began
  could create these outcomes
• Building rapport before the first day of class!

28
Rapport, Student Motivation, and Course Attitudes

• Half of an Introduction to Psychology courses
  students (n 66) received a welcome email one
  week prior to the beginning of the semester
• Randomly assigned, instructor blind to
  participant condition
• Surveyed rapport, student motivation, and course
  attitudes at
• End of 1st class period
• At mid-term
• After the final exam

29
Rapport, Student Motivation, and Course Attitudes

• Most favorable ratings for email group during the
  first class period
• Fewer effects for subsequent measurement periods
• Some interactions with gender of student
• Students in the email group were less likely to
  drop the course

30
Kudos to Legg and Wilson (2009)!

• A welcome email is a low-tech, time-efficient way
  to contact students and start building rapport
  and motivation
• Authors randomly assigned students within one
  class and were blind to which students received
  the email
• Measured multiple, relevant DVs, at 3 time periods

31
Student Attitude Change in a Prejudice Course

• Reducing prejudice is often a goal of psychology
  courses
• Reducing prejudice should be assessed over time,
  as done superbly by Kernahan and Davis (2010)
• The utility of the classroom for ideas in basic
  research

32
Student Attitude Change in a Prejudice Course

• Compared a Psychology of Prejudice and Racism
  course a Statistics course
• Utilized a pretest-posttest design with a control
  group, plus a one-year follow-up
• Design allows for examination and comparison of
  attitude changes from beginning to end of
  semester, and beyond

33
Student Attitude Change in a Prejudice Course

• Results showed that by the end of the semester,
  those in the Prejudice and Racism course showed
  greater
• Awareness of white privilege, white guilt,
  noticing of racism, and responsibility for taking
  action
• At one year follow-up, some effects
• Plateaued (e.g., awareness of white privilege)
• Waned (e.g., responsibility for taking action)
• Increased (e.g., comfortability in mixed-race
  interactions)

34
Kudos to Kernahan and Davis (2010)!

• Comparison between a group with an anticipated
  effect and a control group
• Effects examined across 3 time periods, including
  one year later
• High external validity, and the target outcome of
  interest for basic and pedagogical research!

35
Learning or Living-Learning Communities

• Designed to create a coherent educational
  experience for a group of students centered on a
  particular topic
• Can improve student performance and retention
  while fostering relationships
• Buch and Spaulding (2008) spectacularly examined
  these important outcomes utilizing a longitudinal
  design with a matched comparison group

36
Learning or Living-Learning Communities

• Compared GPA, retention, co-curricular
  involvement in psychology courses/activities over
  7 semesters of college
• Utilized a longitudinal design with a control
  course with students matched (n total 40) on
  important variables (SAT score, gender,
  ethnicity, generation status)

37
Learning or Living-Learning Communities

• Cumulative GPA better in the Learning Community
  group during the 1st, 2nd, and 4th semester of
  college
• Retention at the school and progress within the
  major were better among Learning Community group
  (some significant findings, some non-significant
  but in the expected direction)
• Co-curricular involvement higher in the Learning
  Community group
• Involvement in psychology club, internship,
  research

38
Kudos to Buch and Spaulding (2008)!

• Longitudinal design (see also Buch Spaulding,
  2011), with a comparison group matched on
  important variables
• On a hot topic in higher education
• 7
• Semesters
• Of
• Data
• !

39
Laptop Multitasking and Performance

• Multitasking divides attention and can lead to
  poor memory of course material and poor
  performance
• Laptops in the classroom can provide distractions
  for the users and others in view of laptop
  screens
• Sana and colleagues (2013) put distractions from
  laptop use to the test (literally!)

40
Laptop Multitasking and Performance

• Two studies on how well students retained
  information during lectures for a multiple choice
  quiz
• One study examined student performance when they
  were randomly assigned to multitask during the
  lecture (given some online tasks to perform)
• A second study examined student performance when
  they were in view of confederates laptop screens
  on which they were multitasking

41
Laptop Multitasking and Performance

• In the first study, students assigned to
  multitask performed worse on a quiz of the
  information of the lecture (with no difference
  between simple fact-based items or application
  items)
• In the second study, students who were in view of
  a research confederate who was multitasking
  performed worse on the quiz

42
Kudos to Sana and Colleagues (2013)!

• Examining a topic extremely relevant to teaching
  and learning
• Including two studies, one on distraction from
  the self and one on distraction from others
• Using an experimental design that simulated
  classroom experiences
• High external validity and mundane realism

43
Time-of-Day Preference and Grade-Point-Average
(GPA)

• Preckel et al. (2013) examined whether high
  school students (N 272) time-of-day preference
  was predictive of their GPAs in a variety of
  subjects
• At the college level, we discuss the best times
  to offer classes from a students perspective

44
Time-of-Day Preference and Grade-Point-Average
(GPA)

• Preckel et al. (2013) statistically controlled
  for variables suggested to be predictive of
  academic performance (e.g., conscientiousness,
  cognitive ability, achievement motivation,
  gender)
• Included measures of other-report as well as
  self-report

45
Time-of-Day Preference and Grade-Point-Average
(GPA)

• Entered time-of-day preference on the last step
  of three hierarchical linear regressions
• Used three criterion
• overall GPA
• math-science GPA
• language GPA

46
Time-of-Day Preference and Grade-Point-Average
(GPA)

• Found that students with an evening preference
  had a lower overall GPA, lower math-science GPA,
  and language GPA than students with a morning
  preference

47
Time-of-Day Preference and Grade-Point-Average
(GPA)

• Use of statistical control, a commonly-used
  technique in personality research
• Use of multiple predictors, including self- and
  other-report data

48
Academic Dishonesty and Narcissism

• Much has been made about the rise of narcissism
  in Western society during the past 2-3 decades
• Teachers have lamented (likely since the
  beginning of time) academic dishonesty among
  students

49
Academic Dishonesty and Narcissism

• Brunnell et al. (2011) randomly assigned 199
  Introductory Psychology students to one of two
  questionnaire conditions
• Questions referred to the Self
• Questions referred to the Typical Student on
  Campus

50
Academic Dishonesty and Narcissism

• Questions referred to
• Guilt experienced for cheating if
• Exam was overly difficult
• Classmates did not help them study
• Friends pressured them to cheat
• Prevalence of Academic Dishonesty
• The number of times they (others) cheated on a
  test in the past 12 months
• Predicted how many times the typical student on
  campus would cheat on a test

51
Academic Dishonesty and Narcissism

• 40-item Narcissism Personality Inventory (Raskin
  Terry, 1988)
• Contains 3 subscales
• Power
• Exhibitionism
• Special Person

52
Academic Dishonesty and Narcissism

• Low exhibitionists displayed less guilt when they
  cheated than did high exhibitionists no
  differences were found when judging others who
  cheated
• High exhibitionists reported more academic
  dishonesty when judging themselves than others
  no differences were found when judging others

53
Academic Dishonesty and Narcissism

• Within the self condition, mediational analyses
  revealed that the relationship between
  exhibitionism and reported academic dishonesty
  was fully mediated by guilt

54
Academic Dishonesty and Narcissism

• Use of both experimental and nonexperimental
  measures
• Analysis of narcissism measure at both the factor
  and subscale levels
• Examined interactive effects of experimental
  condition and narcissism
• Mediational analyses to hint at the underlying
  mechanism

55
Overview

• Scholarly teaching, the scholarship of teaching
  and learning (SoTL), and the learning sciences
• Issues of control in pedagogical research
• Effective methodology exemplars from SoTL and the
  learning sciences
• Take-home messages for effective pedagogical
  research

56
A happy medium?

• SoTL and the learning sciences have somewhat
  different approaches to studying similar outcomes
• Effective pedagogical research utilizes the
  strengths of both areas

57
Highlighted strengths of my research (as a
gracious author)

• Strengths highlighted in reviews of my work
• Novelty of topic and research design
• Building in theory to support findings in an
  intuitive way
• Meaningful, stringent control/comparison groups
• Using established materials from basic and
  pedagogical research
• Large-scale studies
• Pressing, relevant, generalizable, and current
  topics
• Clear, logical writing

58
Though this be madness, there is a method in it

• Instead of ghosts of kings past, we have ghosts
  in the form of confounds and noise
• But fear not! Though this may be madness, there
  is a method in it
• We just need to use the right tools to separate
  the method from the madness
  (or the
  meaningful effects
  from the confounds and
  noise)

59

• Thank you for your attention! Questions please!