Enhancing assessment and feedback: an evidencebased response

1
Enhancing assessment and feedback an
evidence-based response

• Berry ODonovan Chris Rust
• Deputy Directors
• ASKe Centre for Excellence in Teaching and
  Learning
• (Assessment Standards Knowledge exchange)
• Oxford Brookes University
• bodonovan_at_brookes.ac.uk, crust_at_brookes.ac.uk
• ASKe Directorate
• Margaret Price, Jude Carroll, Berry ODonovan and
  Chris Rust

2
Programme

• 10.30 Welcome and introduction
• 10.50 Activity
• 11.15 Tacit knowledge the social-constructivist
  assessment
• process model
• 11.50 Student engagement with criteria - 4 case
  study examples
• 12.30 Lunch
• 1.30 Feedback - the issues
• 1.45 Activity - 7 conditions
• 1.55 Improving feedback - preparing students
• 2.05 Activity
• 2.20 Improving feedback - making it fit for
  purpose
• 2.40 Activity
• 3.10 The rest of the cycle
• 3.20 Summation evaluation
• 3.30 End

3
Three main points from the literature

• Assessment and feedback are key drivers of
  student learning

4
Assessment a key driver of student learning

• Assessment is at the heart of the student
  experience
• (Brown, S Knight, P., 1994)
• From our students point of view, assessment
  always defines the actual curriculum
• (Ramsden, P.,1992)
• Assessment defines what students regard as
  important, how they spend their time and how they
  come to see themselves as students and then as
  graduates.........If you want to change student
  learning then change the methods of assessment
• (Brown, G et al, 1997)

5
Three main points from the literature

• Assessment and feedback are key drivers of
  student learning
• Current assessment and feedback processes are
  often not effective

6
But there are problems

• QAA subject reviews
• National Student Satisfaction Survey
• the Achilles heel of quality (Knight 2002a, p.
  107)
• Summative assessment practices in disarray
  (Knight 2002b, p. 275
• Broken (Race 2003, p. 5)
• There is considerable scope for professional
  development in the area of assessment (Yorke et
  al, 2000, p7)
• Rising concern about cheating and plagiarism

7
Problems contd.

• The types of assessment we currently use do not
  promote conceptual understanding and do not
  encourage a deep approach to learningOur means
  of assessing them seems to do little to encourage
  them to adopt anything other than a strategic or
  mechanical approach to their studies.
• (Newstead 2002, p3)
• students become more interested in the mark
  and less interested in the subject over the
  course of their studies. (Ibid, p2)

8
Three main points from the literature

• Assessment and feedback are key drivers of
  student learning
• Current assessment and feedback processes are
  often not effective
• There is lots of useful and evidence-based
  meta-level advice in the literature plus a
  plethora of articles on successful practice.

9
ASKes focus

• Communicating and engaging students with
  assessment requirements standards, criteria and
  feedback
• A key issue in assessment is that students often
  do not understand what is a better piece of work
  and do not understand what is being asked of them
  particularly in terms of standards and criteria
  (ODonovan et al., 2001)
• Feedback is also often not understood (Lea and
  Street, 1998), often considered too vague
  (Higgins, 2000) and the language complex and
  subject to interpretation (Lea and Street, 1998
  Ridsdale 2000)

10

Activity

• Two students are given the same written
  assessment task (e.g. an essay in your
  discipline). In the tutor feedback on the task
  both students are congratulated on producing a
  highly analytical piece of work. One student
  is a postgraduate, the other a first year
  undergrad.
• Does the term highly analytical mean something
  different at these two levels? And, if so, what
  is the difference? How do you explain to
  students the standard of work expected?

11
Making meaning requires explicit and tacit
knowledge

• Meaningful understanding of standards requires
  both tacit and explicit knowledge (ODonovan, B.,
  Price, M., Rust, C., 2004)
• we can know more than we can tell (Polanyi,
  reprinted 1998, p.136).
• Verbal level descriptors are inevitably fuzzy
  (Sadler 1987)
• There is a cost (in terms of time and resources)
  to codifying knowledge which increases the more
  diverse an audiences experience and language
  (Snowdon, 2002).
• Tacit knowledge is experience-based and can only
  be revealed through the sharing of experience
  socialisation processes involving observation,
  imitation and practice (Nonaka, 1991)
• making sense of the world is seen as a social
  and collaborative activity (Vygotsky, 1978).

12
Social-constructivist view of assessment
ASkes Focus

• the social-constructivist view of learning
  argues that knowledge is shaped and evolves
  through increasing participation within different
  communities of practice

the social-constructivist process model of
assessment argues that students should be
actively engaged with every stage of the
assessment process in order that they truly
understand the requirements of the process, and
the criteria and standards being applied, and
should subsequently produce better work (Rust C.,
ODonovan, B., Price, M., 2005)
13
Explicit Criteria
Students
14
Explicit Criteria
Active engagement with criteria
Students
15
Marking exercise
Immediate results participants av. mk non
participants av. mk. Cohort 1 (99/00) 59.78 54.
12 Cohort 2 (00/01) 59.86 52.86 Cohort 3
(01/02 55.7 49.7 Results 1 year later Cohort
1 57.91 51.3 Cohort 2 56.4 51.7 Rust,
C., Price, M ODonovan, B.(2003) "Improving
students learning by developing their
understanding of assessment criteria and
processes Assessment and Evaluation in Higher
Education, Vol. 28, No. 2
16
Peer marking using model answers (Forbes
Spence, 1991)

• Scenario
• Engineering students had weekly maths problem
  sheets marked and problem classes
• Increased student numbers meant marking
  impossible and problem classes big enough to hide
  in
• Students stopped doing problems
• Exam marks declined (Average 55gt45)
• Solution
• Course requirement to complete 50 problem sheets
• Peer assessed at six lecture sessions but marks
  do not count
• Exams and teaching unchanged
• Outcome Exam marks increased (Av. 45gt80)

17
Peer feedback - Geography (Rust, 2001)

• Scenario
• Geography students did two essays but no apparent
  improvement from one to the other despite lots of
  tutor time writing feedback
• Increased student numbers made tutor workload
  impossible
• Solution
• Only one essay but first draft required part way
  through course
• Students read and give each other feedback on
  their draft essays
• Students rewrite the essay in the light of the
  feedback
• In addition to the final draft, students also
  submit a summary of how the 2nd draft has been
  altered from the1st in the light of the feedback
• Outcome Much better essays

18
Peer feedback - Computing (Zeller, 2000)
The Praktomat system allows students to read,
review, and assess each others programs in order
to improve quality and style. After a successful
submission, the student can retrieve and review a
program of some fellow student selected by
Praktomat. After the review is complete, the
student may obtain reviews and re-submit improved
versions of his program. The reviewing process is
independent of grading the risk of plagiarism is
narrowed by personalized assignments and
automatic testing of submitted programs. In a
survey, more than two thirds of the students
affirmed that reading each others programs
improved their program quality this is also
confirmed by statistical data. An evaluation
shows that program readability improved
significantly for students that had written or
received reviews. Available at
http//www.infosun.fim.unipassau.de
/st/papers/iticse2000/iticse2000.pdf
19
Explicit Criteria
Active engagement with
feedback
Students
20
Potential of feedback

• Feedback is the most powerful single influence
  that makes a difference to student achievement
• Hattie (1987) - in a comprehensive review of 87
  meta-analyses of studies
• Feedback has extraordinarily large and
  consistently positive effects on learning
  compared with other aspects of teaching or other
  interventions designed to improve learning
• Black and Wiliam (1998) - in a comprehensive
  review of formative assessment
• Students are hungry for feedback to develop
  their learning
• (Higgins et al, 2002)

21
Retention and feedback

• The number of opportunities available for
  feedback is an important variable in
  non-completion of students in the early years of
  study
• (Yorke, 1999)
• Where students are uncertain about their ability
  to succeed, formative feedback is of particular
  importance
• (Yorke Longden, 2004)

22
Feedback problems

• Unhelpful feedback (Maclellan, 2001)
• Too vague (Higgins, 2000)
• Subject to interpretation (Ridsdale, 2003)
• Not understood (e.g. Lea and Street, 1998)
• Dont read it (Hounsell, 1987)
• Damage self-efficacy (Wotjas, 1998)
• Has no effect (Fritz et al, 2000)
• Seen to be too subjective (Holmes Smith, 2003)

23
11 conditions under which assessment supports
learning 1 (Gibbs and Simpson, 2002)

• Sufficient assessed tasks are provided for
  students to capture sufficient study time
  (motivation)
• These tasks are engaged with by students,
  orienting them to allocate appropriate amounts of
  time and effort to the most important aspects of
  the course (motivation)
• Tackling the assessed task engages students in
  productive learning activity of an appropriate
  kind (learning activity)
• Assessment communicates clear and high
  expectations (motivation)

24
11 conditions under which assessment supports
learning 2 (Gibbs and Simpson, 2002)

• 5 Sufficient feedback is provided, both often
  enough and in enough detail
• 6 The feedback focuses on students performance,
  on their learning and on actions under the
  students control, rather than on the students
  themselves and on their characteristics
• 7 The feedback is timely in that it is received
  by students while it still matters to them and in
  time for them to pay attention to further
  learning or receive further assistance
• 8 Feedback is appropriate to the purpose of the
  assignment and to its criteria for success.
• 9 Feedback is appropriate, in relation to
  students understanding of what they are supposed
  to be doing.
• 10 Feedback is received and attended to.
• 11 Feedback is acted upon by the student

25
Improving feedback - prepare students (in Yr 1
esp.)

• Aligning expectations (of staff students,
  between teams of markers)
• - often a mismatch of expectations e.g correcting
  errors, advice for the future, diagnosis of
  general problems, comments specific only to that
  piece of work. These mismatches occur frequently
  with no particular pattern about who holds which
  view/perspective but problems arise when the the
  two don't coincide. Purpose of feedback may vary
  from assignment to assignment so would need to be
  clarified each time. (Freeman Lewis, 1998)
• Identifying all feedback available
• Model the application of feedback
• - e.g. using previously-marked assignments to
  show how feedback was used to improve later
  assignments
• Encourage the application of feedback
• - e.g. in a subsequent piece of work the student
  is required to show how they have used prior
  feedback to try to improve their work and some
  marks allocated for this.
• Require and develop self-assessment

it is the interaction between both believing in
self-responsibility and using assessment
formatively that leads to greater educational
achievements (Brown Hirschfeld, 2008)
26
Activity

• In 3s, discuss
• How do you currently prepare students to
  understand and engage with feedback?
• Which of these ideas could you introduce, or
  develop further, and how?

27
Improving feedback - ensuring engagement

• Ensure students have MOM - Motive, Opportunity,
  Means (Angelo, 2007)
• Draft-plus-rework - feedback effort (for markers
  and students) is located at the draft stage, and
  possibly only a summative grade is given for the
  final submission
• Improve the linkage of assessment strategies
  across programmes and between modules/units
• Increase student engagement and understanding
  through dialogue - in-class discussion of
  exemplars, peer-review discussions supported by
  tutors, learning-sets, etc.
• Identify what is feasible in a given assessment
  context - written feedback can often do little
  more than diagnose development issues and then
  direct students to other resources for help and
  support
• Ensure it is timely - quick and dirty generic
  feedback, feedback on a draft, MCQs quizzes,
  etc. (using technology may help)
• Consider the role of marks - they obscure
  feedback
• Reduce over-emphasis on written feedback - oral
  can be more effective (McCune, 2004). But
  individual F2F can be resource intensive
• Review resource allocations (N.B. OU 60)

28
Figure 1 Peer-review as a method of encouraging
students to discuss and compare their
understanding of assessment criteria
29
Figure 2 the use of 'exemplars' as amechanism
for encouraging dialogue about assessment
criteria
30
Figure 3 Generic feedback and self critique
31
Activity

• Individually
• Choose one or more specific ideas to improve
  feedback that you think you could use. In as
  much detail as possible, identify how you would
  put the idea/s into practice.
• In pairs
• Take it in turns to explain your plans to your
  partner. The job for the listener is to be a
  friendly and constructive critic

32
7 principles of good feedback practice (Nicol
and Macfarlane-Dick, 2006)

• Good feedback practice
• helps clarify what good performance is (goals,
  criteria, expected standards)
• facilitates the development of reflection and
  self-assessment in learning
• delivers high-quality information to students
  about their learning
• encourages teacher and peer dialogue around
  learning
• encourages positive motivational beliefs and
  self-esteem
• provides opportunities to close the gap between
  current and desired performance
• provides information to teachers that can be used
  to help shape the teaching

33
Tutor discussion of criteria
Staff
Assessment guidance to staff
Marking and moderation
Explicit Criteria
Students
34
Rust C.,ODonovan B Price., M (2005)
Tutor discussion of criteria
Staff
Assessment guidance to staff
Marking and moderation
Explicit Criteria
Students