Measuring and Evaluating Student Motivation and Effort Related to the SROI Assessment Process at NDS

1
Measuring and Evaluating Student Motivation and
Effort Related to the SROI Assessment Process at
NDSU
NDSU
Rhonda Magel Charles McIntyre
November 16, 2005
2
Background

• Student assessment of courses and instruction has
  become standard practice at virtually all
  institutions of higher learning.
• Data collected from these surveys can be used for
  both formative and summative assessment.
• Many educators place great faith in reliability
  of student assessment, while others are skeptical
  of the data and the process.
• A fundamental concern of student assessment is
  the validity of the data.
• The underlying assumption is that students
  possess the motivation and expend the required
  level of effort to provide valid data for the
  assessments.
• One could argue that in order to increase the
  validity of the student responses, an increase in
  student motivation and effort are required.

3
Expectancy Theory

• Expectancy Theory was originally developed by
  Vroom and provides a model for individual
  motivation. It has served as a theoretical
  foundation for a large body of studies in
  psychology, organizational behavior, consumer
  behavior, and management accounting.
• It is hypothesized that beliefs and attitudes do
  not directly influence actual behavior, but are
  moderated by the strength of intention to perform
  specific behaviors (behavioral intentions).
• For student assessment, it is assumed that
  students complete the assessment process to the
  best of their ability.
• However, Expectancy Theory predicts that student
  motivation to participate in the assessment is a
  better predictor of the success of the assessment
  process, than student attitudes and beliefs
  related to the process.

4
The Model

• An individual is considered to be an active,
  thinking, and predicting creature within their
  environment and they chose behaviors (and levels
  of effort) that are based on expected outcomes.
• The choice of the amount of effort that they
  exert on a particular task, F, is modeled by the
  equation
• F is the motivational force to participate in a
  task
• E is the expectancy (probability) that a great
  deal of participation (or effort) will affect the
  success of the task and
• V is the valence (overall attractiveness), which
  results from a successfully completed task.

5
The Model

• The valence can be further modeled with the
  equation
• Vk is the valence or attractiveness of a specific
  outcome from a task, k, and
• Ik is the perceived probability that a particular
  task will lead to outcome k.

6
NDSU Study

• A study was conducted on the campus of North
  Dakota State University during the 2005 Spring
  Semester in order to collect data related to
  student motivation and effort related to the SROI
  process.
• Administration and faculty were surveyed during
  the 2005 Fall Semester.

7
RESULTS OF NDSU STUDY

• Charlie McIntyre
• Rhonda Magel

8
YEARS IN MAJOR

• First year - 64
• Second year - 38
• Third year - 25
• Fourth year - 18
• Fifth year - 24

9
Breakdown of GPAs
10
College of their Majors

• Arts, Hum, Social Science - 10
• Agriculture -
  13
• Business - 16
• Engineering - 66
• Hum Dev Education - 19
• Pharmacy - 4
• Science Math - 25

11
Combining All Students-Model 1

• Students selected instructor improvement as their
  greatest motivating factor
• Students selected course improvement as their
  second greatest motivating factor
• Students selected sharing information with other
  students as their third choice
• Students selected using the information for
  salary, tenure, and promotion decisions as their
  least motivating factor

12
MODEL 2

• Students felt that the intended use of student
  evaluations was more important than whether or
  not the evaluations were actually interpreted
  correctly when determining effort level

13
Comparisons Among Genders

• Males had a definite ranking as to how they would
  like to see the evaluations used. The top reason
  was instructor improvement of teaching followed
  by course improvement, giving the information to
  other students, and then for promotion, tenure,
  and salary decisions.

14
Gender Comparisons cont.

• Females did not have as definitive of ranking.
  They ranked using the information for promotion,
  tenure, and salary decisions lower than the
  others. There were no other significant
  differences.
• Both males and females put a higher importance on
  the intended use of student evaluations rather
  than whether or not they were interpreted
  correctly when determining level of effort.

15
Gender Comparison (cont.)

• Males were more consistent in their responses in
  the first model. This difference disappeared
  once college and year in class were taken into
  consideration. (determining motivation)
• Females were more consistent in their responses
  in the second model. (determining effort)

16
Gender Comparisons (cont.)

• There was not a significant difference in the
  mean ratings that both males and females gave the
  current SROI process. (females 4.4167
  males3.8689) scale0 to 9
• There was not a significant difference in the
  mean rating that males and females gave their
  professors overall at NDSU. (males 6.1707
  females6.1489) scale 0 to 9

17
RESULTS (colleges)

• Students from hard science colleges
  (agriculture, engineering, science and math,
  pharmacy) had more consistent responses in the
  first model than students in the soft science
  colleges (arts, hum soc sci, business, and hum
  dev. and ed.)- 43 of the variation of student
  responses explained by variables in the model for
  the soft science students and 56 of the
  variation was explained by variables in the model
  for the hard science students.(motivation)

18
RESULTS (colleges)

• The same thing was true for students in the
  second model (44 for soft and 57 for hard)
  (effort)

19
RESULTS (First Year Students)

• Students in the first year had instructor
  improvement significantly different than giving
  the information to other students and using the
  information for promotion and raises. Using the
  information for promotion and raises was
  significantly lower than any of the other reasons.

20
Results (2nd, 3rd,4th year students)

• Students in the second year had using the
  information for promotion and raises
  significantly lower than the other reasons. This
  was also true for students in their 3rd and 4th
  years. No other significant differences were
  found.

21
RESULTS (5TH YEAR)

• Students in their fifth year ranked using the
  information to give other students as their last
  priority for doing student evaluations.
• Students in the upper years of their major were
  generally more consistent in their responses.

22
Correlations

• There was some relationship between a students
  year in major versus the rating the student
  assigned to the current SROI system (negative)
• There was no relationship between the students
  year in major and the rating the student gave
  their overall professors at NDSU.

23
Correlations (cont.)

• The correlation coefficient was calculated
  between a students current GPA and how the
  student rating the current SROI system. It was
  not found to be significant. (p-value 0.228)
• The correlation was found to be significant as to
  a students current GPA and the overall ratings
  they gave their professors. The correlation was
  positive. (p-value 0.006)

24
Chairs

• Question In general, on a scale of 0 to 9 with
  0Very Bad and 9Very Good, how would you
  describe the quality of instruction at this
  institution?
• Chairs 6.90
• Students 6.16
• Difference found to be significant (p-value
  0.003)

25
Impression about current SROI

• Question What is your general impression about
  the current SROI course/instructor evaluation
  system used at NDSU?
• Chairs 4.45
• Students 4.02
• No significant difference was found
  (p-value0.280)

26
Correlations

• Correlation coefficient between the rating that a
  chair gave the current SROI system and length of
  time the chair had been at the University
• -0.317 (p-value 0.082)

27
Chairs - Results

• Chairs gave the following rankings as to how they
  would like to see student evaluations used
• 1. Improve teaching (1.29)
• 2. Improve course content (2.32)
• 3. Promotion, tenure, and salary (2.77)
• 4. Giving the information to other students
  (3.87)
• (All significantly different)

28
Ratings Chairs Gave Current SROI System By
College

• Agriculture (5.0)
• Arts, Humanities and Social Sciences (5.6)
• Business (4.5)
• Engineering (2.83)
• Human Development and Education(3.0)
• Pharmacy (5.5)
• Science and Math (4.57)
• These are not significantly different

29
Comments by Chairs

• Current questions are vague
• Calculate separate statistics for 100 level
  classes- dont lump all classes together and
  calculate an average
• Encourage student comments

30
Chairs Comments (continued)

• Use more than student evaluations to evaluate
  teaching
• Easy teachers get high ratings on SROIs- teachers
  in rigorous courses get low ratings
• Get rid of SROIs
• Maybe separate student satisfaction with the
  class from the evaluation of the class

31
Comments (continued)

• Avoid questions that ask students to rate how
  much they like the instructor
• Students need to be better informed as to how
  SROIs are used
• No need to evaluate every class
• Need more student comments (several)
• SROIs not a valid measurement of the quality of
  teaching