Student Surveys and the Student Learning Experience

1
Student Surveys and the Student Learning
Experience

• Michael ProsserHigher Education Academy

2
Academy Mission

• The Academys mission is to help institutions,
  discipline groups and all staff to provide the
  best possible learning experience for their
  students.

3

• Presentation based upon
• Over 20 years research into the student learning
  experience in higher education in the United
  Kingdom, Australia, Sweden and Hong Kong
• Experience in Australia with the Course
  Experience (Perceptions) Questionnaire
• The interpretation of the results of student
  surveys is not value or theory free
• Interpretations in terms of student satisfaction
  and rankings
• Interpretations in terms of student perceptions
  relating to student learning

4
  Overview of the student learning perspective
Figure 1 Model of Student Learning
5

• STUDENT APPROACHES TO LEARNING
•  
• Surface Approach
• Intention to reproduce
• - rote memorise information needed for assessment
• - failure to distinguish principles from examples
• - treat tasks as external impositions
• - focus on discrete elements without integration
•  
• Deep Approach
• Intention to understand
• meaningfully memorise information for later use
• - relate new ideas to previous knowledge
• - relate concepts to everyday experiences
• relate evidence to conclusions 

6
Approach to learning represents the relationship
between the student and the task or
subject Approaches vary between
subjects Approaches vary between tasks within
subjects Approaches not stable but
variable Surface and deep approaches describe a
fundamental variation in the way students
approach learning they do not describe the rich
detail in the way students approach their learning
7
STUDENTS PERCEPTIONS OF THE LEARNING
CONTEXT Research by Entwistle and Ramsden (1983)
using interviews and questionnaires identified a
number of student perceptions relating to the way
they approached their studies Student perceptions
of Quality of teaching including quality of
feedback (NSS Teaching, Assessment and Feedback,
Academic Support)) Clearness of goals of course
and standards of assessment (NSS Assessment and
feedback) Workload so high that it was not
possible to understand everything Assessment
measuring reproduction and not understanding were
found to relate to how they approach their
studies and to learning outcomes (exam results
and other indicators)
8

• RELATIONSHIP BETWEEN PERCEPTIONS AND APPROACHES
• Study of 8837 students in 51 first year subjects
  around Australia
• Amongst the data collected were students
  responses to
• contextualised Ramsden's Course Experience
  Questionnaire and a
• contextualised Biggs Study Process Questionnaire
• 1994-1996 Australian Research Council Academic
  Departments and the Quality of Teaching and
  Learning Paul Ramsden, Griffith University,
  Elaine Martin, RMIT, Michael Prosser, La Trobe
  University, Keith Trigwell, UTS

9
Approaches to Study Surface Approach 32. Although
I generally remember facts and details, I find it
difficult to fit them together into an overall
picture 35. The best way for me to understand
what technical terms mean is to remember the
textbook definitions   Deep Approach 28. I try to
relate ideas in this subject to those in other
subjects, wherever possible 34. In trying to
understand new ideas, I often try to relate them
to real life situations to which they might
apply.   Biggs Study Process Questionnaire
10
Student Perceptions Of Learning Context Good
Teaching 15. The staff made a real effort to
understand difficulties students might be having
with their work. Clear Goals and Standards 1. It
was always easy to know the standard of work
expected 6. I usually had a clear idea of where I
was going and what was expected of me in this
subject. Appropriate Workload 25. The sheer
volume of work in this subject meant that it
couldn't all be thoroughly comprehended
(-). Appropriate Assessment 8. To do well in this
subject, all you really need is a good memory
(-). Ramsdens Course Experience Questionnaire
11

• Analysis conducted using
• Individual student as unit of analysis across all
  first year subjects
• Individual student as unit of analysis within
  first year subjects subjects
• First year subjects as unit of analysis

12
Factor Analysis of Perceptions of T L Context
and Approach to Study - individual _______________
__________________________________________________
Scale Factors _______________________
_____ 1 2 _________________________________
________________________________ Perceptions of
Context Good teaching .80 Clear Goals and
Standards .67 Appropriate Workload -.69 Appr
opriate Assessment -.65 Approach to
Study Surface Approach .81 Deep
Approach .73 ___________________________________
______________________________ Principal
Components, Varimax Rotation, n8837
13
 At individual student level   1.   A deep
approach is associated with perceptions that the
teaching is good and the goals and standards are
clear (NSS Teaching, Assessment and
Feedback)   2.   A surface approach is associated
with perceptions that the workload is too high
and assessment tests reproduction That is,
variation in individual students perceptions of
the learning context across all subjects is
associated with the approaches to study (and
learning outcome) 1994-1996 Australian Research
Council Academic Departments and the Quality of
Teaching and Learning Paul Ramsden, Griffith
University, Elaine Martin, RMIT, Michael
Prosser, La Trobe University, Keith Trigwell, UTS
14
Factor Analysis of Perceptions of T L Context
and Approach to Study within first year medical
subjects in different universities _______________
__________________________________________________
Scale Subject A Subject B Subject
C ________ ________ ________ 1 2 1 2 1 2
_________________________________________________
________________ Perceptions of Context Good
teaching .84 .77 .79 Clear Goals and
Standards .68 .70 .69 Appropriate
Workload -.73 -.68 -.69 Appropriate
Assessment -.70 -.60 -.60 Approach to
Study Surface Approach .75 .84 .80 Deep
Approach .62 .64 .72 __________________________
________________________________________
15
 Within each subject   1.   A deep approach is
associated with perceptions that the teaching is
good and the goals and standards are clear (NSS
Teaching, Assessment and Feedback)   2.   A
surface approach is associated with perceptions
that the workload is too high and assessment
tests reproduction That is, variation in
students perceptions of the learning context
within subjects is associated with the approaches
to study within subjects within subject
variation in perception is not measurement
error. 1994-1996 Australian Research Council
Academic Departments and the Quality of Teaching
and Learning Paul Ramsden, Griffith University,
Elaine Martin, RMIT, Michael Prosser, La Trobe
University, Keith Trigwell, UTS
16
Factor Analysis of Perceptions of T L Context
and Approach to Study - subject __________________
_______________________________________________ Sc
ale Factors __________________________
1 2 ______________________________________
___________________________ Perceptions of
Context Good teaching .81 -.34 Clear Goals
and Standards .78 Appropriate
Workload -.75 Appropriate Assessment -.87
Approach to Study Surface Approach
.91 Deep Approach .82 _________________________
________________________________________ Principal
Components, Variamax Rotation, n51
17
 At subject level   1.   A deep approach is
associated with perceptions that the teaching is
good and the goals and standards are clear (NSS
Teaching, Assessment and Feedback)   2.   A
surface approach is associated with perceptions
that the teaching is not so good, workload is too
high and assessment tests reproduction That is,
variation in students perceptions of the
learning context between subjects is associated
with the approaches to study 1994-1996
Australian Research Council Academic
Departments and the Quality of Teaching and
Learning Paul Ramsden, Griffith University,
Elaine Martin, RMIT, Michael Prosser, La Trobe
University, Keith Trigwell, UTS
18

• Example for student learning in first year
  mathematics
• In a study of first year university mathematics
  students in a large research intensive
  university, analyses of short open ended written
  statements by 236 students identified
• Conceptions of Mathematics
• Fragmented Conceptions
• Maths as numbers, rules and formulae
• Maths as numbers etc with applications to
  problems
• Cohesive Conceptions
• C. Maths as a way of thinking
• D. Maths as a way of thinking for complex problem
  solving
• E. Maths provides insights for understanding the
  world

19

• Fragmented Conceptions
• Maths as numbers, rules and formulae
• Maths is the study of numbers, and the
  application of various methods to changing
  numbers
• Maths as numbers etc with applications to
  problems
• Maths is the study of numbers and their
  applications in other subjects and the physical
  world

20

• Cohesive Conceptions
• Maths as a way of thinking
• Maths is the study of logic. Numbers and symbols
  are used to study life in a systematic
  perspective and requires the mind to think in a
  logical and often precise manner
• Maths as a way of thinking for complex problem
  solving
• Maths is an abstract reasoning process which can
  be utilized to explore and solve problems
• Maths provides insights for understanding the
  world
• Techniques for thinking about observable,
  physical phenomena in a quantitative way and also
  for thinking more abstractly with little or no
  relation to the directly observable universe.

21

• Approaches to Studying Mathematics
• Surface approach focus on reproducing
• Learning by rote memorisation to reproduce
  knowledge and procedures
• Learning by doing lots of examples to reproduce
  knowledge and procedures
• Deep approach focus on understanding
• Learning by doing lots of examples to
  understand theory and concepts
• Learning by doing difficult problems to
  understand and to relate to other knowledge
• Learning by doing difficult problems and studying
  theory to look for situations that theory may
  apply.

22

• Approaches to Studying Mathematics
• Surface approach focus on reproducing
• Learning by rote memorisation to reproduce
  knowledge and procedures
• I liked calculus because I could remember
  formulas which is how I used to study. I would
  rote learn all the formulas and summarize all my
  theoretical notes
• Learning by doing lots of examples to reproduce
  knowledge and procedures
• The way I go about studying for mathematics is by
  doing lots of examples and questions. Firstly I
  would study the notes and learn the formulas,
  then I put all of that to use by doing heaps of
  examples

23

• Deep approach focus on understanding
• Learning by doing difficult problems to
  understand and to relate to other knowledge
• After listening to explanation of how a
  particular maths works the most essential
  features a repetition to develop speed (this
  usually consists of boring menial tasks) and an
  equal component of very difficult problems which
  require a great deal of thought to explore that
  area and its various properties and their
  consequences
• Learning by doing difficult problems and studying
  theory to look for situations that theory may
  apply.
• Read the relevant theory and try to get on the
  same wavelength as the person who actually
  discovered it. Before I attempt any problems I
  try to think where you can use the concept i.e.
  what the concept was invented for. Then I attempt
  problems (on my own).

24
Relationship between Conception of Mathematics
and Approach to Studying Mathematics  ____________
________________________________________________ C
onception Approach Total ________________
_ Surface Deep ______________________________
______________________________ Fragmented (A
B) 179 17 196 Cohesive (C, D, E) 4 36
40   Total 183 53 236 __________________
__________________________________________ Chi-squ
are126, plt.001 Phi.57 ES2.1 Crawford, K.,
Gordon, S., Nicholas, J. and Prosser, M. (1998)
Qualitatively different experiences of learning
mathematics at university. Learning and
Instruction, 8, 455-468.
25
Defining Items from the Conceptions of
Mathematics Questionnaire
26
Cluster Analysis of Prior Experiences and
Understandings, Perceptions and Approaches and
Post Experiences and Understandings Subscales
27
__________________________________________________
_________________________ Variables Mean
Surface Mean Deep (n147) (n127) ________
__________________________________________________
_________ Prior Experiences and Understandings
Prior Fragmented Conception of Mathematics
0.31 -0.46 Prior Cohesive Conception of
Mathematics -0.39 0.45 Prior Surface
Orientation 0.34 -0.44 Prior Deep
Orientation -0.56 0.59 Prior Academic
Ranking (TER) -0.29 0.35 Perceptions and
Approaches Good Teaching -0.37 0.37
Clear Goals and Standards -0.23 0.24
Appropriate Workload -0.30 0.32
Appropriate Assessment -0.12 0.12 Surface
Approach 0.44 -0.45 Deep
Approach -0.54 0.57 Post Experiences and
Understanding Post Fragmented Conception
0.34 -0.41 Post Cohesive Conception -0.27
0.27 Achievement (Final Mark in
Mathematics) -0.34 0.40 ______________________
_____________________________________________
28

• Conclusions
• The Table shows clear relations between prior
  experiences, perceptions of teaching and
  approaches and post experience and understanding.
• The analysis identified 2 groups of students
• In particular, those students
• with more fragmented and less cohesive
  conceptions on entry, describe having adopted
  more of a surface approach to studying at high
  school and less of a deep approach,
• perceive the teaching to be poorer, to adopt more
  of a surface and less of a deep approach to
  studying at university, and
• have more fragmented and less cohesive
  conceptions after studying and poorer achievement
• The opposite was found for the other group

29
  Overview of the student learning perspective
Figure 1 Model of Student Learning
30
Substantial amount of research linking results on
surveys such as the NSS with student approaches
to study and learning outcomes Student
perceptions are a function of both their prior
experiences and understandings and the course
design and teaching Do not improve student
satisfaction by focusing on satisfaction - the
individual items or scales need to better
understand why they responded the way they
Spread of results proportions responding in
certain ways better than mean of
responses Substantial disciplinary variation
unreasonable to compare between disciplines Can
only expect small effect sizes in changes in
scores over time - .2 of a SD if over 3 - 5
years changes of the order of .1 to .2 points
(3.5 to 3.6 or 3.7).
31

• Focus on the student perception rather than
  satisfaction better understand their
  perceptions why they respond the way they have
• Comprehensive and aligned set of student
  evaluation instrument designed and interpreted in
  terms of student perceptions
• Institutions quality assurance and academic
  development and CPD activities aligned with the
  underlying model
• Treat data as indicators requiring further
  investigation focus groups, open ended
  responses etc

32

• Academy Response
• Conferences relating to student experiences
• Workshops and seminars to interested HEIS and
  groups of staff
• Work with interested institutions on ways to
  improve their students experiences as indicated
  by the NSS
• Suggesting ways of further developing and / or
  improving future surveys
• Review of dissemination site
• Development of a bank of optional items, a
  selection of which individual institutions may
  wish to add to their NSS survey

33
Correlation matrix of NSS institutional