Title: Enhancing Assessment and Feedback
1- Enhancing Assessment and Feedback
- Principles and Practice
- David Nicol
- Professor of Higher Education
- Centre for Academic Practice and Learning
Enhancement (CAPLE - Director of REAP and PEER projects
(www.reap.ac.uk) - University of Strathclyde, Scotland
- ATN Australia 20th October 2011
2National Student Survey (UK) Assessment
feedback (2008)
No Survey Statement England Scotland Northern Ireland
5. The criteria used in marking have been clear in advance 69 69 69
6. Assessment arrangements and marking have been fair 74 74 73
7. Feedback on my work has been prompt 56 51 53
8. I received detailed comments on my work 61 52 52
9. Feedback on my work has helped clarify things I did not understand 56 51 50
22 Overall, I am satisfied with the quality of the course 82 86 83
3Plan
- Background
- Re-engineering Assessment Practices (REAP)
project 1m - Concepts and example of practice
- An institutional viewpoint
- PEER project
4Background
- Departments and faculties supporting educational
improvement projects. - REAP project big implementation across 3
universities - Policy/strategy led development of policy in
assessment and feedback based on REAP - Students Feedback as dialogue campaign
- PEER project JISC funded 50k
- HE Sector Project facilitator for QAA Scotland
on AF - Research/publications assessment, learning,
change -
- See www.reap.ac.uk
5REAP Re-engineering Assessment Practices
- Scottish Funding Council for Universities (1m
project) - 3 Universities - Strathclyde, Glasgow Glasgow
Caledonian - Large 1st year classes (160-600 students)
- A range of disciplines (19 modules 6000
students) - Many technologies online tests, simulations,
discussion boards, e-portfolios, e-voting,
peer/feedback software, VLE, online-offline - Learning quality and teaching efficiencies
- Assessment for learner self-regulation
- www.reap.ac.uk
6Background (1)
- Gibbs, G. Simpson, C (2004) Conditions under
which assessment supports students learning,
Learning and Teaching in Higher Education, 1,
3-31. - See
- Formative Assessment in Science Teaching (FAST)
project at http//www.open.ac.uk/science/fdtl
7Gibbs and Simpson (2004)
- Assessment tasks Conditions 1-4
- Capture sufficient study time (in and out of
class) - Are spread out evenly across timeline of study
- Lead to productive activity (deep vs surface)
- Communicate clear and high expectations
- i.e concern here is with time on task how much
work students do - their active engagement in
study
8Background (2)
- Literature Review
- Nicol, D. Macfarlane-Dick, D. (2006). Formative
assessment and self-regulated learning A model
and seven principles of good feedback practice.
Studies in Higher Education, 34 (1), 199-218 - Background
- Student Enhanced Learning through Effective
Feedback SENLEF project funded by HE Academy - REAP project www.reap.ac.uk
9Rethinking assessment and feedback
- 1. Consider self and peers as much as the teacher
as sources of assessment and feedback - Tap into different qualities than teacher can
provide - Saves time
- Provides considerable learning benefits (lifelong
learning)
- 2. Focus on every step of the cycle
- Understanding the task criteria
- Applying what was learned in action
- 3. Not just written feedback
- Also verbal, computer, vicarious, formal and
informal
10Seven principles of good feedback
- Good feedback
- Clarifies what good performance is (goals,
criteria, standards). - Facilitates the development of reflection and
self-assessment in learning - Delivers high quality information to students
that enables them to self-correct - Encourages student-teacher and peer dialogue
around learning - Encourages positive motivational beliefs self
esteem - Provides opportunities to act on feedback
- Provides information to teachers that can be used
to help shape their teaching (making learning
visible) -
- Source Nicol and Macfarlane-Dick (2006)
11Principle 1 Clarify what good performance is
(the context of dialogue)
EMPOWERMENT/ SELF-REGULATION
- Students create criteria
- Students add own criteria
- Students identify criteria from samples of work
- Exemplars of different performance levels
provided - Students rephrase criteria in own words
- Provide document with criteria
ENGAGEMENT
12Two meta principles
- 1. Meta-PRINCIPLE 1 time and effort on task
(structured engagement) i.e. steers on how much
work to do and when Gibbs and Simpson 4
conditions - 2. Meta-PRINCIPLE 2 developing learner
self-regulation (empowerment/self-regulation) i.e
steers to encourage ownership of learning the
seven principles discussed above. -
- Key task for teacher is to balance 1 and 2
-
13 14Psychology
- 560 first year students
- 6 topic areas (e.g. personality, classical
conditioning), 48 lectures, 4 tutorials, 12
practicals - Assessment 2 x MCQs (25), tutorial attendance
(4), taking part in experiment (5), essay exam
(66)
15Problems identified
- No practice in writing skills but required in the
exam - More detail provided in lectures than mentioned
in exams (not enough independent reading) - No feedback except on Multiple Choice Questions
(percent correct) - Didnt want to increase staff workload
16Psychology Redesign
- Discussion board in Learning Management System
- Students in 85 discussion groups of 7-8, same
groups throughout year - Also open discussion board for class
- Friday lectures cancelled discover material
themselves - Series of online tasks
17Structure of group tasks
- 6 cycles of 3 weeks (one cycle x major course
topic) - First week light written task (e.g. define
terms) 7 short answers (all answer) - Second week guided reading
- Week three deep written task students
collaborate in writing a 700-800 word essay on
the same topic. - Within each week
- The Monday lecture introducing material
- Immediately after lecture, task posted online
for delivery the following Monday - Model answers (selected from students) posted for
previous weeks task
18The teaching role
- Participation in the discussions was compulsory
but not marked (in subsequent years there was 2
mark for participation) - Course leader provided general feedback to the
whole class often motivational - He encouraged students to give each other
feedback - The group discussions were not moderated but
monitored for participation
19An example of deep task
- The Task 800 word essay
- Assess the strengths and weaknesses of Freuds
and Eysencks theories of personality. Are the
theories incompatible? -
- readings suggested
- questions provided which all students should try
-
20Relation to the Gibbs Simpsons four
assessment conditions
- Tasks require significant study out of class
(condition 1) - Tasks are distributed across topics and weeks
(condition 2) - They move students progressively to deeper levels
of understanding (condition 3) - There are explicit goals and progressive increase
in challenge (condition 4)
21Relation to 7 feedback principles
- Standard format and model answers provide
progressive clarification of expectations
(principle 1) - Students encouraged to self-assess against model
answer (principle 2) - Course leader provides motivational and
meta-level feedback and selects model answers
(principle 3) - Online peer discussion aimed at reaching
consensus is core feature of design about
response (principle 4) - Focus on learning not just marks, sense of
control/challenge enhanced motivation
(principle5) - Repeated cycle of topics and tasks provide
opportunities to act on feedback (principle 6) - VLE captures all interactions allowing course
leader to monitor progress and adapt teaching
(principle 7)
22Benefits
- Students worked exceptionally hard
- Written responses of exceedingly high standard
- Students took responsibility for learning
- High levels of motivation atmosphere in class
improved - Online interactions showed powerful scaffolding
and community building - Feedback with 560 students through peer and
self-feedback (model answers) - Easy for tutors to monitor participation
- Improved mean exam performance (up from 51-59,
plt0.01) weaker students benefit most
23Online postings/interaction
- 24,362 messages posted by groups for essay tasks
- Average number of postings per student 44.3
- 1067 postings to general open discussion forum
- Students set up online study groups for other
subjects - Structured tasks online triggered important
socio-cognitive processes
24Has it worked?
25(No Transcript)
26(No Transcript)
27 28Why use principles
- Provides a framework for operationalizing the big
idea the development of learner self-regulation - Helps translate the research into accessible
guidelines for teaching practice - Ensures change is educationally driven
- Enables connections to be made across innovations
in different disciplines - Provides a common language to talk about
innovation and for dissemination - Can add to the evaluation process measures
- Helps identify where technology can leverage
benefits
29Guidelines for Implementation
- A single principle or many?
- Tight-loose maintain fidelity to the principles
(tight) but encourage disciplines to develop
their own techniques of implementation (loose) - Balance teacher feedback with peer and
self-generated feedback - The more actively engaged students are, the
better the course design
30 Developments since REAP
- Principles of Assessment and Feedback approved by
University Senate and embedded in policy (2008) - Use of principles to inform curriculum renewal
and Quality Assurance processes - Feedback as Dialogue campaign to gain
commitment of students - PEER Project (Peer Evaluation in Education Review)
31(No Transcript)
32(No Transcript)
33(No Transcript)
34 Peer Review in Education Evaluation PEER
- The aims of the PEER project are to
- Review evidence base for peer review
- Develop educational designs for peer review (and
self-review) - Identify software support for peer review
- Pilot implementations of peer review with large
student numbers - Produce guidelines for higher education why do
it, how to do it, pitfalls and solutions and
software possibilities. - see http//www.reap.ac.uk/peer.aspx
35Peer feedback augmenting teacher feedback
- Increasing quantity and variety of feedback
- No extra workload on teacher with software (e.g.
PeerMark, Aropa) - More timely e.g. collaborative projects
- Simulates professional life reconciling
different feedback perspectives
36The argument
-
- Not enough attention has been focused on the
potential of peer feedback not just as a way of
increasing the quantity and quality of the
feedback students receive, but as a way of giving
students practice in constructing feedback.
37The focus
- Scenarios where students make evaluative
judgements about the work of peers and provide a
feedback commentary, usually written - Not talking about scenarios involving
- ...informal feedback in collaborative tasks
- .....students evaluating each others
contribution to group working - .....students grading/marking each others work,
although some rating might be part of peer
design
38Benefits of feedback construction (1)
- Constructivist rather than telling paradigm
- High-level cognitive activity students cannot
easily be passive - Students actively exercise assessment criteria
from many perspectives - Writing commentaries develops deep disciplinary
expertise - See many approaches and learn that quality can be
produced in different ways - Shifts responsibility to student puts them in
role of assessor exercising critical judgement
39Benefits of feedback construction (2)
- Learn to assess own work as exactly the same
skills are involved - Develops capacity to make evaluative judgements
- a fundamental requirement for life beyond
university. Also, this capacity underpins all
graduate attribute development (Nicol, 2010) - Nicol, D (2011) Developing students ability to
construct feedback, Published by QAA for Higher
Education, UK
40Example 1Peer feedback
- Laboratory work Gibbs 2011 www.testa.ac.uk/resour
ces - Reason poor quality of lab reports in science
- Strategy students organised in groups and
produce poster to represent their lab report - Peer process hang poster in class and all
students individually walk round analyse posters
and write feedback on them (e.g post-its)
questions, suggestions, inaccuracies etc - Result significant improvements in lab work and
reporting, positive competition in class,
students did not want to look bad.
41Example 2 Engineering Design
- Peer Project case study
- DM 100 Design 1 first-year class
- Dr Avril Thomson, Course Leader, Design
Manufacturing and Engineering Management (DMEM),
University of Strathclyde - avril.thomson_at_strath.ac.uk
- Caroline Breslin, Learning Technology Adviser,
University of Srathclyde - caroline.breslin_at_strath.ac.uk
42Example 2 Design 1
- 82 first-year students
- Design a product theme eating and resting in
the city - Research in groups (in city, in library etc.)
- Individually produce a Product Design
Specification (PDS) detailed requirements for
and constraints on design (rationale,
performance,standards, manufacturing etc) - Given a PDS exemplar from another domain to show
whats required (stainless steel hot water
cylinder) - Online learning environment Moodle and PeerMark
part of Turnitin suite
43Product Design SpecificationTypical Headings
Rationale Performance Environment Packaging Maintenance Weight and size Disposal Shipping Processes Time Scale Patents Product costs Aesthetics Ergonomics Materials Testing Quality/reliability Competition Marketing Life in service
44DM 100 Design 1
- Peer review task
- Individually, each student peer-reviewed and
provided feedback on the draft PDS of two other
students - Criteria completeness, convincingness of
rationale, specificity of values (performance)
and one main suggestion for improvements with
reasons - Students used experience, giving and receiving
feedback, to update PDS which comprises part of a
Folio - Students self-review Folio and meet/discuss with
tutor - Peer review not assessed directly but 10 marks
for professionalism which included participation
in peer review.
45Peer review rubric DM 100
- Do you feel the PDS is complete in the range of
headings covered? If no, can you suggest any
headings that would contribute towards the
completeness of the PDS and explain why they are
important? - Is the PDS specific enough? Does it specify
appropriate target values or ranges of values?
Please suggest aspects that would benefit from
further detail and explain. - To what extent do you think the rationale is
convincing for the PDS? Can you make any
suggestions as to how it might be more
convincing? Please explain. - Can you identify one main improvement that could
be made to the PDS? Provide reason(s) for your
answer.
46Evaluation
- Online survey completed by 64 students
- Course work marks compared to previous years
- Focus group interviews
- Peer review comments recorded online
47Results 1
- Which aspects of the peer review did you learn
from? - Giving feedback 10.9
- Receiving feedback 26.6
- Giving and receiving feedback 54.7
- Neither giving or receiving 7.8
48Results 2
Did you modify your initial submission as a
result of the peer review activity?
Yes, as a result of the peer review given 23.4
Yes, as a result of the peer review received 25.0
Yes as a result of the peer review given AND received 28.1
No 21.9
N/A 1.6
49Results student comments
- If yes, please give specific examples of
modifications (n41) - I added a couple of paragraphs and improved
existing paragraphs, this added two full A4 pages
to my work - I included specific materials as changed the
formatting of the document so it looked more
professional - I provided more specific numeric values and
expanded my rationale after seeing someone elses
PDS and after receiving feedback
50Results RECEIVING feedback
- Please give examples of what you learned from
RECEIVING peer reviews from other students (n54) - Parts that I had previously missed were brought
to eye such as market competition (noticing) - Receiving peer reviews gave me insight into what
others thought of my work and gave me a direction
to improve (reader response) - Where the PDS was confusing to understand (reader
response) - I found out how good mine was (motivational)
- The person who peer reviewed my PDS gave me
positive feedback which helped me a lot
(motivational) - Not much, they werent very good
51Results PROVIDING feedback
- Please give examples of what you learned from
PROVIDING peer reviews of others work (n47) - When giving advice to people on theirs, it gave
me greater perception when reviewing my own work
by listening to my own advice for example
transfer - I had a chance to see other peoples work and
aspects of their work that I felt were lacking in
my work, this helped me to improve my work
transfer - I was given a greater understanding of the level
of the work the course may be demanding
standards - Allowed me to see from an assessors perspective
- Thinking from a critical point of view critical
judgement - How to look at work critically that isnt your
own
52Results How you carried out peer review
- Could you make any comments about how you
carried out the peer review? How did you evaluate
the quality of the work to provide a response to
the peer review questions? (n37) - I compared it to mine and the ideal PDS and said
how I would improve it - Partly by comparing my work to theirs
- I tried to think about what I wrote and whether
this PDS was better or worse - Looked through the headings first and noted any
that should have been included then read more in
detail about each individually and compared it
with my own PDS
53Focus groups
- How did you go about reviewing?
- I read it through and compared it with what I
had done to see if they had put something I had
not done and then I added it in if they hadnt.
The four questions were useful as they provided a
framework for the review. If we hadnt had the
questions it would have been difficult. I did the
reviews separately and then answered one then the
other. The first was a better standard than the
other so I used the ideas from the better one
to comment on the weaker one. I also read the
guidelines in class when I did the peer review.
There were ideas from the good one that I hadnt
even thought of in mine
54Focus groups
- What do you think is best for learning giving or
receiving feedback? - For me it would probably be to give feedback
because I think seeing what other people have
done is more helpful than getting other peoples
comments on what you have already done. By
looking at other peoples work you can see for
yourself what you have forgotten or not even
thought about. When people give feedback on yours
they generally just talk about what is there.
They dont say, well I did this on mine and you
could put that in yours.
55Results
Would you choose to participate in a peer review
exercise in the future?
-
- Yes 76.6
- No 3.1
- Maybe 18.8
- Dont know 1.6
56Some tentative conclusions
- Giving and receiving feedback are qualitatively
different - Feedback receipt helps bring deficiencies in own
work to students attention, can be motivational
and gives a experience of different audiences
(readers) - Feedback production appears to be better for
developing judgement critical thinking, using
criteria and standards, being more objective
about own work - Fortunately receiving and giving usually occur in
the same domain of assignment production hence
double duty. - More research required