Evaluating Student and Teacher Performance in a Teaching American History Program

1
Evaluating Student and Teacher Performance in a
Teaching American History Program

• September 26, 2006
• 415 515 PM
• Albuquerque, New Mexico

2
American History Education Enhancement Project

• Marc Cooper, Cindy Mais,
• La Reva Newcomer, Reed Olsen

3
A project of Missouri State University in
partnership with a consortium of public school
districts includingRepublic, Marshfield, Nixa,
Ozark, and Springfield funded by the U.S.
Department of Educationthrough Teaching American
History Grants
4
Table of Contents

• Program Overview
• Discussion Pods
• Experimental Design
• Statistical Evaluation
• Conclusions
• Internet URL for this presentation
• Appendix Statistical Methods

5
Program Overview and History

• Dr. Marc Cooper, Director
• Missouri State University

6
Cohorts

• First Cohort Spring 2002 Summer 2004
• Second Cohort Spring 2005 Summer 2007

7
Goal To improve Middle and High School student
performance in American history

• To improve the American history content knowledge
  of social studies teachers
  
• To improve teaching skills in history

8
Secondary Goals

• To form a partnership between the Missouri State
  University History department and local public
  schools
  
• To develop Internet courses designed for
  teachers
  
• To focus the MSU history department on improving
  American history in the public schools

9
Strategy

• Offer 32 credit hour MA program to 25
  social-studies teachers
  
• Offer 1 Internet course per semester for 5
  consecutive semesters (3-ch)
  
• Offer pod groups coordinated with Internet
  courses (1-ch)
  
• Offer 4-week summer institutes during 3
  consecutive summer sessions (4-ch)

10
Internet Courses

• Courses offered
• American Revolution
• American Social History
• Civil War in Missouri
• American Education
• Women in American History
• Faculty collaborated with AHEEP administrators
  and teacher liaisons in improving the program

11
Pod Groups

• School centered
• Offered as HST 597 (graded) for 1 credit hour
• 3-5 teachers
• La Reva Newcomer leads groups
• Focused on applying content to the
  classroom More detail later

12
Summer Institutes

• 2002/2005 Historical methodology
• 2003/2006 Sources for local history
• 2004/2007 Writing local history

13
Overall Experiences Using the Online Format
14
To what degree have you applied the material and
objectives to your own classroom curriculum?
15
Evaluating Improvement in Teacher Content
Knowledge

• Grades in content courses
• Major Field Achievement Test (ETS)
• Pre-test during first course
• Post-test during last course

16
MFAT limitations

• Strongest students in 2002 already better than
  90th percentile
  
• Internet courses do not fit MFAT areas except
  Assessment Area 1 equivalent to Colonial/Early
  Federal course
  
• Assessment Area 1 raw scores
• Spring 2002 50.7 correct
• Spring 2004 73.6 correct

17
MFAT Results

• Spring of 2002
• American History 61st Percentile
• European History 47th Percentile
• Spring of 2004
• American History 75th Percentile
• European History 53rd Percentile

18
Participant comments 2002

• I teach government, so I have incorporated
  various documents that pertain to that subject.
  I have also incorporated the general knowledge
  gained overall. The course has also given me
  homework assignment ideas.
• I have used some of the documents and assignments
  in my classes.
  
• I have been too busy to utilize these strategies
  at the present time. I do plan on using them as
  time allows.

19
Participant comments 2004

• I have already included the role of women in
  history and their contributions. I also included
  society's expectations of women in different eras
  and how some women tried to change them, for
  example Stanton and Paul.
• I use a lot of primary documents in my class.
  The kids love trivia, so I actively search for
  that as an add-in. Any time there's a famous
  woman mentioned in one or two sentences, I look
  for more information online that I can share. I
  write my own tests worksheets to make the
  questions more like the MAP questions.

20
Cohort 1 Conclusions

• Teachers improved computer skills
• Teachers improved in content knowledge
• Teachers began using new methods in class
• 17 teachers completed MA coursework
• 11 teachers completed comps and research paper
  obtaining the MA
  

21
Discussion Pods Improvement and Evaluation of
Pedagogy

• Ms. La Reva Newcomer
• Assistant Director, AHEEP
• Missouri State University

22
Discussion Pod Overview

• Generalizations
• Academics and Application
• Configurations

23
Cohort Activities

• Course Material Applications
• Strategy Sessions
• Peer Critiques

24
Semester Project

• American History Lesson Plan
• Cohort Critiques
• Mega-pod Critique Sessions

25
Impact of Pod Discussions

• Cohort Relationships
• Requests from Principals
• Assistance for Teachers outside AHEEP

26
Experimental Design Cohort 2

• Mrs. Cindy Mais, Director
• Republic R-III

27
Testing

• Pre-test post-test model
• Instrument developed from National Assessment of
  Educational Progress (NAEP) released questions
  
• 30 questions
• Mix of 8th grade and 12th grade questions
• Variable difficulty
• Entirely objective
• Treatment and control groups

28
Data Collection

• Student characteristics Middle/High School
• Collected by districts
• Questionnaire part of test
• Teacher characteristics
• Questionnaire
• Grades
• Procedures approved by MSU Human Subjects
  internal review board and district administrators

29
Statistical Evaluation

• Principles and Preliminary Results
• Dr. Reed Olsen
• Missouri State University

30
Statistical Design

• Control for variables that affect student
  achievement but that are outside the control of
  classroom teachers.
  
• Random assignment of participants and control
  groups were not possible.
  
• Multiple Regression Analysis is used instead to
  control for relevant variables.

31
What is Multiple Regression Analysis?

• Controls for Other Variables
• Also estimates the impact of these control
  variables
  
• Estimates the impact of Middle and High School
  Teacher Participation and Performance in AHEEP
  Program upon their middle and high school
  students performance on a standardized exam.

32
Empirical Estimation Gather Data

• Use a standardized exam given twice to each group
  of students.
  
• Pre-Test given at the beginning of school year.
• Post-Test given at the end of school year.

33
Student Characteristics (SC)

• Sex
• School Attendance
• Race
• Student On Free/Reduced Lunch?
• Student with Limited English Proficiency?
• Grade
• Home Living Situation
• Student with Individual Education Plan?
• Is the Student Gifted?

34
Teacher Characteristics (TC)

• Is the Teacher an AHEEP Participant?
• Teachers Performance in AHEEP Program GPA
• Overall
• in methods courses
• in content courses
• for other courses
• Education
• Experience
• Certification
• Personal Characteristics
• Marital Status, Sex, Children, Race, etc.
• Class Preparations
• Outside classroom activities

35
Interpreting Multiple Regression Results
36
A positive and statistically significant
coefficient

• Estimate for a control variable indicates that
  the variable increases student performance on
  exam.
  

37
A negative and statistically significant
coefficient

• Estimate for a control variable indicates that
  the variable decreases student performance on
  exam.
  

38
Statistical significance

• Indicates that the estimated impact of the
  control variable could only have occurred
  randomly 10 percent (or less) of the time.
  
• Therefore, conclude the variable likely has an
  impact on student performance.
  

39
Statistical insignificance

• Indicates that one cannot reject the possibility
  that the variable has no impact on student
  performance.
  

40
General Results

• How does AHEEP effect high school and middle
  school student achievement?
  
• Test credibility
• Higher attendance, higher Pre-test scores, gifted
  students, etc. score better on the exam.
  
• Students with teachers who have more education,
  experience, that are married, without small
  children at home, etc. score better on exams.

41
What results do we expect for AHEEP participation?

• Two kinds of AHEEP participants.
• Those who remained in the program and those who
  dropped out of the program.
  
• All else equal, expect that students of those
  participants who dropped out of AHEEP will do
  better.
  
• Expect students of those participants with higher
  grades (GPA) in AHEEP to do better.

42
AHEEP Results - Participation

• For all participants, AHEEP participation is not
  found to have a statistically significant impact
  on middle and high school student exam scores.
  
• For those participants who remained in the
  program, AHEEP participation is not found to
  have a statistically significant impact on middle
  and high school student exam scores.
• However, for those who participated in the
  previous year but dropped out AHEEP participation
  is found to have a positive and statistically
  significant impact on middle and high school
  student exam scores.
• Student scores increase by approximately 1 exam
  question (3.8 ), controlling for other factors.

43
AHEEP Results Participation and Performance

• AHEEP Participation, controlling for GPA, is
  found to significantly decrease their students
  exam scores by 2.8 (9.9 ) to 5.1 (18.2)
  questions.
• These numbers estimate the impact that AHEEP
  participation has upon student performance for
  those AHEEP teachers with a 0 (F) GPA.
  
• The lowest GPA for an AHEEP participant is 2.81.

44
(No Transcript)
45

• AHEEP participants earned either B or A
  grades in their classes
  
• AHEEP participants with a B average are estimated
  to have no significant impact on student
  performance.
  
• AHEEP participants with an A average are
  estimated to positively affect their students
  performance.

46
Results

• Strong grades in AHEEP courses improved Middle
  and High School student performance by
  
• 1 to 1.5 questions
• 3.8 5.4 percent

47
Preliminary Conclusions

• When teachers successfully pursue rigorous
  masters level course work in history, they will
  improve their students performance.
  
• Pod Group work is crucial in increasing student
  performance.
  
• Graduate work stresses teachers requiring post
  program testing to reveal the true magnitude of
  the results.

48
Appendix Statistical Methods
49
Combine Data and Estimate Multiple Regression
Equation

• Post-testi ? ?Pre-testi ?SCi ?TCi
  ?AHEEPi ?GPAiAHEEPi ?i
  
• i is an index number referring to an individual
  observation a middle or high school student
  taking American History.

50

• Post-test, Pre-test, SC, TC, GPA, and AHEEP are
  as defined above.
  
• ?i is an error term.
• ?, ?, ?, ?, ?, and ? are parameters (coefficient
  estimates) to be estimated.

51
(No Transcript)
52
(No Transcript)