Student surveys and quality enhancement

1
Student surveys and quality enhancement

• Dr. Paul Bennett, Head of Surveys, Higher
  Education Academy

2
The HEAs surveys role

• Deliver national post-graduate surveys (PRES and
  PTES)
• NSS further analysis and best practice sharing
• Support and consultancy around using survey data
  for enhancement
• Annual Surveys for Enhancement conference
• Survey related research and policy advice

www.heacademy.ac.uk/surveys
3
Outline

• The HEAs surveys role
• Student surveys
• The NSS and quality enhancement- Communication-
  Staff attitudes- Analysis and exploration-
  Staff-student partnerships
• Surveying student engagement
• Surveying the experience of postgraduates

4
Student surveys
Love? Hate?
   
Efficient Overused
Democratic Consumerist
Digestible Shallow
Comparable Misleading
Reliable Invalid
Useful Motivating Dangerous Demotivating
SURVEYS
Student Extract
5
Student surveys

• Research method providing a partial but often
  useful representation of experience
• Need to be triangulated with other information
  especially qualitative
• Comparing results useful but comes with health
  warnings
• Surveys can be a good starting point for student
  engagement not the end point

6
The NSS and quality enhancement

• NSS intended to inform student choice and quality
  assurance
• HE White Paper (England 2011) Well informed
  students driving teaching excellence
• Publication of NSS results has increased focus on
  learning and teaching in many institutions and a
  powerful lever for change
• though questions remain over the focus of the
  survey and validity of making comparisons
• NSS becoming better used in stimulating
  enhancement

7

• HEAs NSS Institutional Strategy Working Group
• Making it Count Reflecting on the National
  Student Survey in the process of enhancement
  (October 2012)
• Available viawww.heacademy.ac.uk/nss

8
The NSS and quality enhancement

• Progress in communication of NSS results
• - Top-down dissemination to staff
• - Dissemination of results to students unions
• - You said, we did
• - Staff-student discussions in formal structures
• - You said, we didnt and wider dialogue
• - Staff-student partnerships at all stages
  promotion, analysis, dissemination, further
  research, discussions, decision-making

9
The NSS and quality enhancement

• Staff attitudes
• Important to mitigate perception that the NSS is
  a stick to beat us with
• Deficit model focusing on poor performance,
  statistical rankings and red lights can be
  counter-productive with academic staff
• More effective use focuses on sharing best
  practice, case-studies, mutual support and use of
  qualitative information (including free-text
  comments)
• Being open about the strengths and limitations of
  the survey

10
The NSS and quality enhancement

• Analysis and exploration
• NSS limitations include detail, focus and
  one-method
• Vital to triangulate with other information,
  e.g.
• more detailed internal and module surveys
• information about other (non-satisfaction)
  aspects of experience e.g. engagement
• information beyond experience, including
  grades, employability, retention, examiners
  reports
• non-survey methods, including qualitative
  information for depth

11
The NSS and quality enhancement

• Staff-student partnerships
• Student analyses can add interpretations and
  find solutions that staff committees cant do
  on their own
• Student unions, reps and other students need
  support in analysing, understanding and using NSS
  data
• Vital for effective discussion of NSS and
  enhancement in system of student representation
• Some institutions have used NSS results as a
  springboard for staff-student workshops and
  conferences

12
Using NSS data

• Partnership approaches

13
The NSS and quality enhancement
Scale Rank Beta
The teaching on my course 1 0.326
Personal development 2 0.211
Organisation and management 2 0.209
Academic support 4 0.156
Assessment and feedback 5 0.082
Learning resources 6 0.027
Impact of learning experience on overall
satisfaction Multiple regression of National
Student Survey 2011 dataset
14
The NSS and quality enhancement

• Are the questions valid measures of quality (and
  does that matter)?
• Graham Gibbs (2010) Dimensions of Quality
• Valid, comparable measures of educational quality
  in institutions should relate to educational gain
• Measures of effective practice (e.g. student
  engagement, intellectual challenge, deep
  learning) are good predictors of educational gain
• Such measures can be found in the US National
  Survey of Student Engagement (NSSE) but not, on
  the whole, in the UKs NSS

15
The NSS and quality enhancement

• CEQ research suggested some items indicate deep
  vs. surface learning
• A perception that workload is too high is
  strongly related to a surface approach
• A perception that assessment is focused on
  memorisation and reproduction is strongly related
  to a surface approach
• A perception that teachers are enthusiastic,
  give good feedback, make the subject interesting
  and communicate well is partly related to a deep
  approach
• Questions in the NSS optional bank are relevant
  for enhancement

16
The NSS and quality enhancement
17
Surveying student engagement

• Amongst other things, engagement surveys like
  NSSE and AUSSE ask about
• - learning interaction with peers (in and
  outside class)
• - learning interaction with staff (in and
  outside class)
• - engagement in own study self-directed
  learning and effort
• - course challenge / depth of learning
• Research suggest these predict educational
  gain, especially depth of learning
• Less clear to what extent they predict NSS
  results and less researched in a UK context

18
Surveying student engagement
19
Surveying student engagement
20
Surveying student engagement

• HEAs UK Engagement Survey Pilot
• 14 items from NSSE adapted for UK use in internal
  surveys in 9 institutions (plus 2 more running
  variants of the whole NSSE/AUSSE)
• HEA analysing pooled data, providing national
  aggregate for participants, testing reliability
  and commissioning cognitive testing
• Institutions will produce case-studies on use of
  results for enhancement
• Year 2 of pilot will expand the number of
  institutions involved and may test relationship
  with NSS items

21
Surveying the experience of postgraduates

• Postgraduate Research Experience Survey (PRES) -
  launched 2007, runs every two years
• Postgraduate Taught Experience Survey (PTES) -
  launched 2009, runs annually
• Aim to inform enhancements to learning and
  teaching and advise national policy
• Not intended to inform student choice
• Results are confidential no league tables
• Benchmarking clubs facilitate comparisons
• Allow for inclusion of locally specific questions

22
Surveying the experience of postgraduates
Reason for using PRES
Identify specific areas for enhancement 94
Assess perceptions of quality of degree programmes 67
Assess the equality of experience and/or opportunities 58
Benchmark experience nationally or with comparator institutions 56
Evaluate consistency of experience across disciplines/departments 52
Help engage relevant staff groups in enhancement 29
Demonstrate to potential PGR students the quality of the research training environment 17
Demonstrate to funders the institution's commitment to PGR support 8
Other 8
23
Surveying the experience of postgraduates

• Significant redevelopment of PRES for 2013
• Focus on enhancement priorities in Researcher
  Development Framework and Quality Code (B11)
• Greater emphasis on research skills and
  professional development including supervisors
  role
• Also importance of Research community for PGRs
• More detailed than NSS, but same principles apply
  in use for enhancement
• PTES being redeveloped for 2014

24
(No Transcript)
25
(No Transcript)
26
PDF of questionnaire available online via
www.heacademy.ac.uk/pres
27

• Vitae study for HEA
• Using PRES to enhance the experience of
  postgraduate researchers(September, 2012)
• Available viawww.heacademy.ac.uk/pres