Using Student Ratings to Assess Student Learning and Motivation in General Education Courses

1
Using Student Ratings to Assess Student Learning
and Motivation in General Education Courses

• AACU Network for Academic Renewal Conference
• March 11, 2006
• Phoenix, AZ
• Bill Pallett
• The IDEA Center
• pallett_at_ksu.edu

www.idea.ksu.edu
2
The IDEA Student Ratings of Instruction System
3
Student Learning Model

• Specific teaching methods influence certain types
  of student progress (learning) under certain
  circumstances.

4
Student Survey - Diagnostic Form

• Teaching Methods Items 1-20
• Learning Objectives Items 21-32
• Student and Course
• Student Characteristics Items 36-39, 43
• Course Management/Content Items 33-35
• Global Summary Items 40-42
• Experimental Items Items 44-47
• Extra Questions Items 48-66
• Comments

5
Teaching Method Categories

• Stimulating Student Interest
• Items 4, 8, 13, 15
• Fostering Student Collaboration
• Items 5, 16, 18
• Establishing Rapport
• Items 1, 2, 7, 20
• Encouraging Student Involvement
• Items 9, 11, 14, 19
• Structuring Classroom Experiences
• Items 3, 6, 10, 12, 17

6
Faculty Information Form

• 12 Learning Objectives
• Basic Cognitive Background (items 1, 2)
• Applications of Learning (items 3, 4)
• Expressiveness (items 6, 8)
• Intellectual Development (items 7, 10, 11)
• Lifelong Learning (items 9, 12)
• Team Skills (item 5)
• Rated by the instructor
• Essential
• Important
• Of minor or no importance
• Essential and Important objectives considered
  Relevant

7
Faculty Information Form

• Primary Intent of Course
• First year/Sophomore Gen. Ed./Distribution
• First year/Sophomore Specialization
• Upperclassmen Gen. Ed./Distribution
• Upperclassmen Specialization
• Graduate/Professional
• Combination of two or more above

8
Research General Education Courses

• 2002 IDEA Research Report 5 Are Student
  Ratings of Courses and Instructors Fair to
  Faculty Teaching General/Liberal Education
  Classes?
• 2006 Follow-up Research

9
IDEA Research Report 5 - 2002

• Donald P. Hoyt and Subashan Perera
• Classes administered during 1998-99 academic year
• 6,013 classes included
• 80 percent identified type of class on FIF
• Exclusions
• Enrollments lt 10
• Response rate gt75
• Short Form classes

10
Follow-up in 2006

• Classes administered during 2004-2005 academic
  year
• Examined undergraduate classes only
• 40,976 classes included
• Exclusions
• Short Form classes

11
Number of Classes by Emphasis and Student
Characteristics
12
Questions Addressed

• Do objectives stressed by teachers of
  professional-oriented classes differ from those
  of general/liberal education classes?
• Do student ratings of progress on
  instructor-chosen objectives differ depending on
  the intended audience?
• After differences in student motivation and other
  extraneous circumstances are taken into account,
  do teaching effectiveness ratings differ
  depending on the type of student enrolled?

13
Objectives Selected

• Are they different for general/liberal education
  and professional-oriented classes?

14
2002 - Objectives Selected (percent of classes)
15
2006 - Objectives Selected (percent of classes)
16
Student Progress

• Do ratings of progress differ depending on
  student type?

17
2002 Differences in Reported Learning

• Student Self-report of Progress
• Four Objectives Emphasized in General Education
  Classes
• 3.79 on five point scale
• Three Objectives Emphasized in Professional
  Classes
• 4.02 on five point scale

18
2002 Highest Levels of Progress

• Highest for graduate classes
• Higher for upper division than lower division
  classes
• Higher for professional than general education
• Graduate 55.8
• Upper Division 54.2
• Lower Division 52.2
• Professional 54.0
• General Education 52.0

T Score (Average of 50, standard deviation of 10)
19
2006 Progress on Relevant Objectives

• General Education
• Lower Division 50.8
• Upper Division 52.0
• Professional Orientation
• Lower Division 52.7
• Upper Division 53.7

20
2002 Highest Levels of Progress

• Upper Division General Education
• Broad liberal education

• Upper Division Professional
• Professional skill/views
• Second highest on 7 of 12 objectives
• Graduate Professional
• Highest on 8 of 12 Objectives

21
2002 Lowest Levels of Progress

• Lower Division General Education
• Factual knowledge
• Principle/theories
• Application
• Professional skill/views
• Team skills
• Interest learning

• Lower Division Professional
• Creative capacities
• Broad/liberal education
• Communication skills
• Find/use resources
• Values development
• Critical analysis

22
Why the differences?

• Teacher Effectiveness
• Student Characteristics
• Other factors????

23
Influence of Extraneous Circumstances Type of
Student

• After accounting for extraneous influences, are
  there differences by type of student?

24
IDEA Adjusted Scores

• Adjust for Influences Beyond Instructors Control
• 2002
• Student Motivation
• Class Size
• Student Effort
• Course Difficulty

25
2002 Results

• Adjusted scores higher than raw scores for both
  lower and upper division general education
  courses
• Adjusted scores lower than raw scores for both
  Graduate and Upper Division Professional-Oriented
  Classes
• Adjusted Scores similar to Raw Scores for Lower
  Division Professional Courses

26
2002 Conclusion

• Ratings of Teaching Effectiveness were adversely
  impacted by extraneous circumstances
• However, differences still existed with General
  Education Classes typically receiving lower
  ratings than Professional Oriented Classes.

27
Additional Findings

• Student Course Characteristics
• Teaching Method Use
• Critical Thinking

28
Student Course Characteristics (2006)
29
Teaching Method Use

• Should different teaching methods be employed for
  lower division general education classes?
• Regression models created for each of 12 learning
  objectives
• Student/Course Type included with each of the 20
  IDEA teaching methods
• Student/Course Type added little to any model
  (less than 2 of variance)

30
Conclusion

• While IDEA Research has found that different
  teaching methods are important in supporting
  learning for the 12 IDEA learning
  objectivesadding information about
  Course/Student Type does not change what teaching
  methods are most important to employ.

31
Critical Thinking Skills

• Learning to analyze and critically evaluate
  ideas, arguments, and points of view
• Selected Important or Essential in 47 of all
  classes
• Lower Division General Education 42
• Upper Division General Education 10
• Lower Division Professional 18
• Upper Division Professional 30

32
Critical Thinking Class Size
33
Questions/Discussion