Reading the (survey) runes Students

1
Reading the (survey) runesStudents experience
of Art Design

• Mantz Yorke
• Lancaster University
• mantzyorke_at_mantzyorke.plus.com

Expectations, Experiences and Encounters in HE
and HE in FE The 3Es The University of
Bolton 12 June 2009
2
Whats the story?

• There is not a problem with student commitment
• Organisation and Management in AD consistently
  
• show weaknesses relative to other subject
  areas
• As does Overall Satisfaction on the NSS
• Creative Arts Design do generally poorly on
  the NSS
• Feedback the picture from different surveys is
• inconsistent
• Some HEIs do consistently better and some
• consistently worse across the NSS scales
• NSS scales vary re discrimination between HEIs

3
The general plan

• What surveys can and cant tell
• The NSS what it says, generally, about Art
  Design
• What some other surveys have shown
• Some pointers from this collection of evidence

4
What surveys can and cant tell
5
Uses

• Measuring
• Public information
• Measuring/judging
• Auditing institutional performance
• Enhancing institutional performance

Technical requirements vary with purpose
6
Robustness

• Population gt Sample gt Methodology gt Responses
• Biases
• Statistical testing
• What is good enough?
• Raw data doesnt necessarily tell the story
• NSS analyses of residuals have their uses

7
The NSS is particularly useful for highlighting
areas of concern, in other words it is good at
identifying the where in student experience
where there are differences in experiences as
rated by students. It is not, however, designed
to answer the why of student
experience Surridge 2009, p.29
The same goes for quantitative surveys in general
8
(No Transcript)
9
(No Transcript)
10
Art Design in the NSS
11
The NSS
2005 2006 2007 AD All AD All
AD All TL 3.93 3.96 3.94 3.98 3.9
4 4.00 AF 3.48 3.47 3.50 3.49 3.54 3.52 AcSup 3.6
0 3.71 3.65 3.76 3.70 3.82 OM 3.39 3.69 3.39 3.7
2 3.45 3.79 LRes 3.86 3.96 3.92 4.02 3.96 4.08 PDe
v 3.93 3.98 3.93 3.99 3.94 4.02 OvSatis 3.70 3.96
3.71 3.96 3.76 4.01 This figure is wrongly
given as 3.90 in the report of the 2006 NSS and
in the Trends 2005-07
12
(No Transcript)
13
(No Transcript)
14
Subject area median percentage agreement 2007,
ranked for each scale
Bottom 4
Top 4
15
(No Transcript)
16
(No Transcript)
17
JACS Subject TL AF AS OM LR PD OvS
Nursing 6 2 12 20 9 1 13
Biology 3 22 1 1 2 14 1
SportSci 9 6 7 9 6 3 9
Psych 10 13 17 2 10 19 6
CompSci 21 15 8 7 3 16 15
ElecEE 16 14 2 10 1 12 5
Architec 14 20 14 22 14 5 17
Sociology 11 9 15 6 22 18 7
SocWork 17 16 22 21 18 4 18
Law 7 11 13 5 8 7 3
BusStuds 19 17 16 8 4 8 12
MediaStd 18 18 18 14 15 20 19
English 2 3 5 4 16 13 4
History 1 1 3 3 19 10 2
FineArt 22 12 21 17 7 21 21
DesignSt 20 8 20 19 11 17 22
Music 8 10 4 12 12 15 10
Drama 4 19 10 15 20 9 14
Dance 5 21 9 13 5 11 16
CinePho 15 7 19 16 13 22 20
TeachrTr 12 5 6 18 17 2 8
AcStEduc 13 4 11 11 21 6 11
Top 5
18
Where the truth lies
A paradox Raw NSS ratings dont tell the whole
story The residuals make allowances for
various background characteristics of HEIs and
hence provide some evidence of institutional
effect Joe and Jill Public wont appreciate
this With residuals theres no Lake Wobegon
effect
19
Organisation Management, Creative Arts Design
2007
Residual
20
Overall Satisfaction, Creative Arts Design 2007
21
TL
Asst Fdbk
Org Man
Acad Sup
22
Learn Res
Pers Dev
23
Institutional size

• Smaller institutions (Nlt10,000) consistently get
• higher scores on the following scales
• Teaching Learning
• Assessment Feedback
• Academic Support
• Personal Development
• Overall Satisfaction

So how do you make the big seem small?
24
Three strikes
Surridge (2009, p.18)
Reference subject area Law
25
Art Design in some other surveys
26
Interpreting surveys of early leavers
Self-selected respondents, hence possible
bias Possibility of blaming programme gt
self Unclear which had done foundation
programme Items are selected, mainly focusing on
programme Responses from AD compared with the
rest high scores (red) are worse low (blue)
are better Scores are percentages indicating
moderate or high influence on departure
27
Reactions from early leavers ( responses)
2007 1997 AD
Rest AD Rest Nmax 60 267
105 1881 Programme not as expected
52 45 48 36 Chose wrong field of study
30 44 26 40 Lack of commitment to programme
23 35 30 39 Contact with academics 68 33
29 24 Teaching quality 60 24 39 22 Lack
of staff support 52 30 33 24 Feedback
quality 52 20 - - Programme
organisation 60 30 47 26 Lack of academic
progress 48 32 38 30 Workload 13 21
10 18 Provision of specialist equipment 39
5 16 7 Financial problems 25 33 44 36
28
Selected comments from 1st year leavers, 2007
Did foundation year in art design which I
really enjoyed but when started 1st year
it was completely different, hardly any contact
with staff or students, lost motivation
interest
I was at college for 3 years and enjoy my time
there. When I went to univ to study Graphic
communication it was not what I expected. The
programme was unorganised and the teaching was
at a poor standard. I work hard on this course
and give my all. For one year I just hated
univ.
Badly run course with staff totally
disinterested in their students.
29
Selected comments from 1st year leavers, 2007
There was very little actual teaching, promised
classes did not materialise until after students
complained (i.e. life drawing). Quite often we
were kept waiting for staff to arrive on time and
often what was said by one was repeated by
another, usually with a slighter different
version which was confusing. The equipment was
more in line for a 6th form class rather than a
degree course and often space was at a minimum,
especially in the printshop and ceramics rooms.
The course was very disorganised. The
content of the course was not what I expected
and at times seemed irrelevant. Constant room
changes early on was very off putting and tutors
failing to attend lectures consistently.
30
Selected comments from 1st year leavers, 2007
I felt univ graphic arts and design course was
overcrowded , the tutors were largely
unhelpful and arrogant. There was little course
structure and no group spirit or suitable work
space. I am now attending a nearby FEC and
loving every minute.
The main reason I left HEI was because of the
organisation of the course and the projects. The
projects were so open you could film anything
and say it has something to do with the subject
matter. I did not see one project that I thought
was art when I attended.
I love photography but my experience at univ
has put me off. The lack of support from
staff was a big part. Just finding them took 2
hours. Then they went on strike
31
The first-year experience and part-time student
surveys same caveats as before
Self-selected respondents, hence possible
bias Possibility of blaming programme gt
self Unclear which had done foundation
programme Items are selected, mainly focusing on
programme Responses from AD compared with the
rest low scores (red) are worse high (blue)
are better Scale is 1 to 5, with high ratings
indicating positivity to first-year experience
32
Reactions from the first year (scale means)
2006 AD
Rest Nmax 957
6076 Programme not as expected 3.09
3.15 Choice of subject 4.35 4.19
Motivation 3.20 2.97 Contact with
academics 3.39 3.74 Teaching quality
3.56 3.69 Tutorial support 3.38
3.54 Feedback quality 3.44 3.14
Programme organisation 3.28
3.67 Keeping up with workload 3.19
3.14 Access to institutional computing
3.88 4.03 Financial worries 2.36 2.46
33
Selected best comments from first-year
students, 2006
The tutors - great members of staff and a really
interesting vibrant course.
The tutors are great and always available if I'm
having trouble with my work.
Being able to chat to the tutors casually about
your work etc.
The structure of the course is very well set up,
I really like the fact that we have specific
projects and deadlines for those projects and I
find the workshop sessions very enjoyable.
The projects that we work on have been exciting.
34
Selected worst comments from first-year
students, 2006
Teaching! I want to learn please!

Dont feel I'm being taught enough.

Wish I had more 1 on 1 tutorials as I have no
clue what is expected of me or
how to do better!
Having no feedback on work. It's needed.

Feedback takes too long.

Not receiving enough feedback on work and staff
not being present.
Lack of support, lack of contact with tutors,
having to wait 6 months before having any
feedback in the form of an assessment. Last
minute changes to timetable.
35
Selected worst comments from first-year
students, 2006
Organisation (not impressed).

The organisation of the course has been poor in
respect to sessions and lectures being changed
at short notice or even cancelled.
Not well organised course, very few teachers, no
help to develop computer skills.
Poor organisation of projects, workshops,
teaching.
Confusion due to teachers saying different things
for 1 brief.
36
Issues for practice
37
Issues for practice

• Organisation and management of programmes,
• and Feedback, are two aspects that seem to
  need
• the greatest attention
• Can programmes be structured to fit more
  closely
• the current situation of students?
• (E.g. commuter students, students with PT
  jobs)
• The general message seems to be about being
• as considerate as possible about the whole of
  the
• student experience

38
Two messages to academic leaders

• 1 Target your efforts on the student experience,
  and
• not merely on getting higher NSS scores
• Dont naively demand 4.1 (or whatever) across
• the board
• Look carefully at where the subject
  performances
• in your institution stand in relation to
  those in
• comparable institutions, and act appropriately